From 64769883ae232520987a9d1cc40d039288a1689a Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 13 Oct 2026 00:33:21 +0000 Subject: [PATCH] Draft smoke test capability proposal --- docs/roadmap/smoke-tests.md | 179 ++++++++++++++++++++++++++++++++++++ 1 file changed, 179 insertions(+) create mode 100644 docs/roadmap/smoke-tests.md diff --git a/docs/roadmap/smoke-tests.md b/docs/roadmap/smoke-tests.md new file mode 100644 index 0000000..ade4839 --- /dev/null +++ b/docs/roadmap/smoke-tests.md @@ -0,0 +1,179 @@ +# Smoke Test Capability Proposal + +Status: proposed for design review; implementation has not started. + +## Purpose + +Provide reusable diagnostic operations for endpoint reachability, +authentication, plain-text inference, and structured-output inference. +Applications building setup, diagnostic, or preflight workflows would otherwise +repeat similar prompts, schemas, provider requests, and result interpretation. +Promptkit should standardize those mechanisms while applications decide which +checks to request and how their outcomes affect application behavior. + +This proposal captures the direction discussed with the maintainer. The API, +configuration shape, defaults, and delivery scope remain subject to review. +It is not an implementation plan or a new public contract. + +## Relationship To Existing Operations + +[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request) +checks whether a concrete request passes preparation without model generation. +`Run` prepares and executes the actual application request. Smoke tests would +instead exercise selected diagnostic operations using synthetic input or a +provider-specific probe. They need not load the application's prompt or inputs, +or call `ValidateRequest` on a future request. + +A consumer could combine these operations into a preflight stage. Promptkit +would provide smoke test evidence, not decide that the application is ready. +Results describe the tested operation at the time of the check; they do not +reserve capacity or guarantee that a subsequent request will succeed. + +The main value is shared diagnostics before real requests exist or when testing +a small synthetic request is useful. Consumers about to execute an ordinary +request may prefer to run it directly and handle its errors. + +## Ownership + +| Owner | Responsibility | +| --- | --- | +| Backend configuration | Declares available diagnostic operations and endpoints, including authentication or model-status endpoints where supported, and configuration needed by supported probe implementations. | +| Promptkit | Owns probe execution, default smoke test prompts and schemas, available criteria, their evaluators, and structured observations and outcomes. | +| Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. | + +An endpoint address alone does not define a probe's meaning. A supported probe +implementation must know the request method, authentication requirements, and +how to interpret the response. A candidate design is for configuration to +identify a supported probe kind plus its endpoint settings. A general request +templating and response-expression language is not proposed initially. + +Configuration changes must retain the existing +[ConfigKit ownership boundary](../adr/0001-configuration-boundary.md). +The [architecture policy](../policy/architecture.md) continues to govern the +library and downstream application boundary. + +## Candidate Smoke Tests + +These are independently selectable operations, not increasingly strict levels +that require every preceding test to pass. + +| Test | Intended evidence | Important distinction | +| --- | --- | --- | +| HTTP/HTTPS reachability | An HTTP response was obtained from the selected destination. | A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available. | +| Authentication | A designated provider operation accepted the effective credentials. | Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract. | +| Plain-text inference | The selected model completed a small text-generation request. | Successful generation and compliance with an exact expected response are separate criteria. | +| Structured-output inference | The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria. | Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally. | + +A backend model-status endpoint could provide additional provider-reported +evidence. It would not substitute for actually completing inference. Whether +to include model-status probes in the first implementation remains open. + +Inference tests should select the effective profile or execution target the +caller wants to test. Automatically substituting a cheap unrelated model is +not part of this proposal. Backend-only operations should not require unrelated +prompt data or a runnable model selection. + +## Default Test Content And Execution Options + +Promptkit would maintain a short plain-text prompt asking for the word +`success`, and a separate structured-output prompt with a simple schema, such +as an object with a required boolean `success` property constrained to true. +Exact wording and schema are design choices still to be selected. + +The defaults provide convenient repeatable tests. Applications select which +criteria to evaluate; an exact-text match need not be required merely to +establish that inference completed. A request to respond without thinking is +prompt content, not a substitute for explicit execution settings. + +Callers should have options for relevant generation settings, timeouts, retries, +and repair. The design must specify which settings inherit from the selected +profile and which a smoke test overrides, and make the effective choices +inspectable without exposing credentials. No retry, repair, reasoning, or token +limit defaults are settled here. A short prompt alone does not guarantee low +cost. Non-inference probes should use endpoints documented for that purpose; +billing properties must not be inferred merely from an HTTP method. + +## Probes, Criteria, And Results + +A probe performs an operation. Criteria evaluate observations from that +operation. Several criteria can use one response without repeating the request. +Candidate criteria include receiving an HTTP response, authentication +acceptance, successful generation, returned text, expected-text matching, JSON +parsing, and schema conformance. Each needs an explicit definition and a clear +set of applicable probes. + +For example, a model returning `Success!` could satisfy successful generation +and returned-text criteria while failing an exact match against `success`. +The application could treat the mismatch as a warning or a failure. Promptkit +would implement the comparison rather than requiring every application to +reimplement it. Matching and normalization rules remain to be specified. + +Candidate criterion outcomes are passed, failed, unsupported, and inconclusive, +with structured reasons. An unavailable authentication probe must not become an +invalid-key verdict. A timeout during authentication may prevent determining +credential acceptance even though the probe itself failed. Criteria that could +not be evaluated must not be confused with assertions evaluated as false. + +A report should identify the tested operation and effective target, applicable +criteria and outcomes, and useful evidence such as HTTP status, elapsed time, +attempts, and reported usage. Unknown measurements remain unknown. Inference +retries or repair should not conceal initial failures behind a final success. +Exact fields, observation retention, and the relationship between reports and +returned Go errors remain open. Credential redaction and disclosure of provider +diagnostics need explicit treatment consistent with existing contracts. + +## Implementation Direction + +Reuse ordinary generation, decoding, validation, and error handling for +inference tests wherever their behavior fits. Do not build a second inference +stack. Existing behavior is described by the +[runner](../internal/runner.md), [generation coordinator](../internal/generation.md), +and [model-client integration contract](../integrations/openai-compatible-chat.md). + +Reachability and authentication probes need a separate, narrow provider-facing +capability. How injected clients participate must be decided without assuming +that every custom client uses the built-in HTTP transport. Existing clients +should not acquire new required methods merely to continue ordinary execution. + +## Decisions Needed Before Implementation + +1. **Public operation shape and first delivery scope.** Choose individual + methods, a typed single-probe operation, or a convenience batch operation; + define target selection for backend, profile, and endpoint-only cases. Select + initial provider adapters and decide whether model-status checks belong in + the first delivery. Go names and configuration keys are not yet proposed. +2. **Probe configuration and routing.** Define supported probe kinds and their + response semantics. Resolve endpoint overrides, relative paths, separate + diagnostic hosts, redirects, and credential destinations deliberately so + the report describes the destination actually tested. +3. **Criteria and result contract.** Specify applicability, exact matching, + unsupported and inconclusive outcomes, errors versus diagnostic findings, + and what remains available after cancellation or partial completion. +4. **Execution policy.** Choose default test content, profile inheritance and + overrides, retry and repair defaults, capacity participation, observation, + and cancellation behavior. Decide whether custom test content is needed + initially; ordinary application-authored prompts remain available via `Run`. + +## Suggested Next Step + +Review this proposal's scope and ownership first. Then write concrete request +and result examples for three cases: an HTTP response with rejected +authentication, successful inference with an unexpected text response, and +schema-valid structured output. Include a backend without an authentication +probe to expose capability and outcome handling. + +Use those examples to settle the operation and result shape, then specify the +smallest backend configuration needed by the initial adapters. Convert the +agreed design into a focused implementation plan with validation proportionate +to the selected contracts under the [testing policy](../policy/testing.md). +Default tests should remain offline using controlled transports and injected +clients; live smoke testing would be an explicit consumer operation. + +## Scope Boundaries + +This proposal does not include Scriptorium commands, application readiness +policy, continuous monitoring, automatic probes before ordinary runs, a +general-purpose diagnostic scripting language, or a guarantee about future +generation. It does not change `ValidateRequest` or ordinary execution. +Published API and format documentation should be updated when implementation +lands, not to present this proposal as current behavior.