promptkit/docs/roadmap/smoke-tests.md

Smoke Test Capability Proposal

Status: proposed for design review; implementation has not started.

Purpose

Provide reusable diagnostic operations for endpoint reachability, authentication, plain-text inference, and structured-output inference. Applications building setup, diagnostic, or preflight workflows would otherwise repeat similar prompts, schemas, provider requests, and result interpretation. Promptkit should standardize those mechanisms while applications decide which checks to request and how their outcomes affect application behavior.

This proposal captures the direction discussed with the maintainer. The API, configuration shape, defaults, and delivery scope remain subject to review. It is not an implementation plan or a new public contract.

Relationship To Existing Operations

ValidateRequest checks whether a concrete request passes preparation without model generation. Run prepares and executes the actual application request. Smoke tests would instead exercise selected diagnostic operations using synthetic input or a provider-specific probe. They need not load the application's prompt or inputs, or call ValidateRequest on a future request.

A consumer could combine these operations into a preflight stage. Promptkit would provide smoke test evidence, not decide that the application is ready. Results describe the tested operation at the time of the check; they do not reserve capacity or guarantee that a subsequent request will succeed.

The main value is shared diagnostics before real requests exist or when testing a small synthetic request is useful. Consumers about to execute an ordinary request may prefer to run it directly and handle its errors.

Ownership

Owner	Responsibility
Backend configuration	Declares available diagnostic operations and endpoints, including authentication or model-status endpoints where supported, and configuration needed by supported probe implementations.
Promptkit	Owns probe execution, default smoke test prompts and schemas, available criteria, their evaluators, and structured observations and outcomes.
Consuming application	Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses.

An endpoint address alone does not define a probe's meaning. A supported probe implementation must know the request method, authentication requirements, and how to interpret the response. A candidate design is for configuration to identify a supported probe kind plus its endpoint settings. A general request templating and response-expression language is not proposed initially.

Configuration changes must retain the existing ConfigKit ownership boundary. The architecture policy continues to govern the library and downstream application boundary.

Candidate Smoke Tests

These are independently selectable operations, not increasingly strict levels that require every preceding test to pass.

Test	Intended evidence	Important distinction
HTTP/HTTPS reachability	An HTTP response was obtained from the selected destination.	A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available.
Authentication	A designated provider operation accepted the effective credentials.	Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract.
Plain-text inference	The selected model completed a small text-generation request.	Successful generation and compliance with an exact expected response are separate criteria.
Structured-output inference	The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria.	Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally.

A backend model-status endpoint could provide additional provider-reported evidence. It would not substitute for actually completing inference. Whether to include model-status probes in the first implementation remains open.

Inference tests should select the effective profile or execution target the caller wants to test. Automatically substituting a cheap unrelated model is not part of this proposal. Backend-only operations should not require unrelated prompt data or a runnable model selection.

Default Test Content And Execution Options

Promptkit would maintain a short plain-text prompt asking for the word success, and a separate structured-output prompt with a simple schema, such as an object with a required boolean success property constrained to true. Exact wording and schema are design choices still to be selected.

The defaults provide convenient repeatable tests. Applications select which criteria to evaluate; an exact-text match need not be required merely to establish that inference completed. A request to respond without thinking is prompt content, not a substitute for explicit execution settings.

Callers should have options for relevant generation settings, timeouts, retries, and repair. The design must specify which settings inherit from the selected profile and which a smoke test overrides, and make the effective choices inspectable without exposing credentials. No retry, repair, reasoning, or token limit defaults are settled here. A short prompt alone does not guarantee low cost. Non-inference probes should use endpoints documented for that purpose; billing properties must not be inferred merely from an HTTP method.

Probes, Criteria, And Results

A probe performs an operation. Criteria evaluate observations from that operation. Several criteria can use one response without repeating the request. Candidate criteria include receiving an HTTP response, authentication acceptance, successful generation, returned text, expected-text matching, JSON parsing, and schema conformance. Each needs an explicit definition and a clear set of applicable probes.

For example, a model returning Success! could satisfy successful generation and returned-text criteria while failing an exact match against success. The application could treat the mismatch as a warning or a failure. Promptkit would implement the comparison rather than requiring every application to reimplement it. Matching and normalization rules remain to be specified.

Candidate criterion outcomes are passed, failed, unsupported, and inconclusive, with structured reasons. An unavailable authentication probe must not become an invalid-key verdict. A timeout during authentication may prevent determining credential acceptance even though the probe itself failed. Criteria that could not be evaluated must not be confused with assertions evaluated as false.

A report should identify the tested operation and effective target, applicable criteria and outcomes, and useful evidence such as HTTP status, elapsed time, attempts, and reported usage. Unknown measurements remain unknown. Inference retries or repair should not conceal initial failures behind a final success. Exact fields, observation retention, and the relationship between reports and returned Go errors remain open. Credential redaction and disclosure of provider diagnostics need explicit treatment consistent with existing contracts.

Implementation Direction

Reuse ordinary generation, decoding, validation, and error handling for inference tests wherever their behavior fits. Do not build a second inference stack. Existing behavior is described by the runner, generation coordinator, and model-client integration contract.

Reachability and authentication probes need a separate, narrow provider-facing capability. How injected clients participate must be decided without assuming that every custom client uses the built-in HTTP transport. Existing clients should not acquire new required methods merely to continue ordinary execution.

Decisions Needed Before Implementation

1. Public operation shape and first delivery scope. Choose individual methods, a typed single-probe operation, or a convenience batch operation; define target selection for backend, profile, and endpoint-only cases. Select initial provider adapters and decide whether model-status checks belong in the first delivery. Go names and configuration keys are not yet proposed.
2. Probe configuration and routing. Define supported probe kinds and their response semantics. Resolve endpoint overrides, relative paths, separate diagnostic hosts, redirects, and credential destinations deliberately so the report describes the destination actually tested.
3. Criteria and result contract. Specify applicability, exact matching, unsupported and inconclusive outcomes, errors versus diagnostic findings, and what remains available after cancellation or partial completion.
4. Execution policy. Choose default test content, profile inheritance and overrides, retry and repair defaults, capacity participation, observation, and cancellation behavior. Decide whether custom test content is needed initially; ordinary application-authored prompts remain available via Run.

Suggested Next Step

Review this proposal's scope and ownership first. Then write concrete request and result examples for three cases: an HTTP response with rejected authentication, successful inference with an unexpected text response, and schema-valid structured output. Include a backend without an authentication probe to expose capability and outcome handling.

Use those examples to settle the operation and result shape, then specify the smallest backend configuration needed by the initial adapters. Convert the agreed design into a focused implementation plan with validation proportionate to the selected contracts under the testing policy. Default tests should remain offline using controlled transports and injected clients; live smoke testing would be an explicit consumer operation.

Scope Boundaries

This proposal does not include Scriptorium commands, application readiness policy, continuous monitoring, automatic probes before ordinary runs, a general-purpose diagnostic scripting language, or a guarantee about future generation. It does not change ValidateRequest or ordinary execution. Published API and format documentation should be updated when implementation lands, not to present this proposal as current behavior.