promptkit/docs/roadmap/smoke-tests.md

Smoke Test Capability Proposal

Status: core design decisions accepted; API details and delivery scope remain under review. Implementation has not started.

Purpose

Provide reusable diagnostic operations for endpoint reachability, authentication, plain-text inference, and structured-output inference. Applications building setup, diagnostic, or preflight workflows would otherwise repeat similar prompts, schemas, provider requests, and result interpretation. Promptkit should standardize those mechanisms while applications decide which checks to request and how their outcomes affect application behavior.

This proposal records the agreed diagnostic boundary, private test content, and execution-profile selection rules. The concrete API, configuration shape, remaining execution defaults, and delivery scope remain subject to review. It is not an implementation plan or a description of current public behavior.

Relationship To Existing Operations

ValidateRequest checks whether a concrete request passes preparation without model generation. Run prepares and executes the actual application request. Smoke tests would instead exercise selected diagnostic operations using synthetic input or a provider-specific probe. An application prompt may be selected to obtain its default profile, but its messages are not executed and its variables and inputs are not required. Smoke tests do not need to call ValidateRequest on a future request.

A consumer could combine these operations into a preflight stage. Promptkit would provide smoke test evidence, not decide that the application is ready. Results describe the tested operation at the time of the check; they do not reserve capacity or guarantee that a subsequent request will succeed.

The main value is shared diagnostics before real requests exist or when testing a small synthetic request is useful. Consumers about to execute an ordinary request may prefer to run it directly and handle its errors.

Ownership

Owner	Responsibility
Backend configuration	Declares available diagnostic operations and endpoints, configuration needed by supported probe implementations, and the backend's default smoke-test execution profile.
Promptkit	Owns probe execution, private fixed smoke-test prompts and schemas, available criteria, their evaluators, and neutral diagnostic reports.
Consuming application	Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses.

An endpoint address alone does not define a probe's meaning. A supported probe implementation must know the request method, authentication requirements, and how to interpret the response. A candidate design is for configuration to identify a supported probe kind plus its endpoint settings. A general request templating and response-expression language is not proposed initially.

Configuration changes must retain the existing ConfigKit ownership boundary. The architecture policy continues to govern the library and downstream application boundary.

Candidate Smoke Tests

These are independently selectable operations, not increasingly strict levels that require every preceding test to pass.

Test	Intended evidence	Important distinction
HTTP/HTTPS reachability	An HTTP response was obtained from the selected destination.	A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available.
Authentication	A designated provider operation accepted the effective credentials.	Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract.
Plain-text inference	The selected model completed a small text-generation request.	Successful generation and compliance with an exact expected response are separate criteria.
Structured-output inference	The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria.	Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally.

A backend model-status endpoint could provide additional provider-reported evidence. It would not substitute for actually completing inference. Whether to include model-status probes in the first implementation remains open.

Backend-only operations should not require unrelated prompt data or a runnable model selection.

Inference Profile Selection

Callers can override the backend's default smoke-test execution profile through the following selections:

Caller supplies	Execution profile
Prompt ID without a profile override	The selected prompt definition's default profile.
Profile ID without a prompt ID	The explicitly selected profile.
Both prompt and profile IDs	The supplied profile overrides the prompt's default under ordinary Promptkit selection rules.
Neither prompt nor profile ID	The selected backend's default smoke-test profile.

Reuse existing prompt/profile selection semantics, including exact selection and omitted-selector handling; see the consumer guide and request GoDoc. The backend default applies when neither selector is supplied, not as a replacement for an invalid explicit selection. How callers identify the backend in that case, and behavior when its default smoke-test profile is absent, remain to be specified.

Selecting an application prompt supplies execution-profile selection only. It does not replace the synthetic smoke-test messages or import the application's output contract. An explicit profile selection must not be silently replaced with a cheaper model. Reports should identify the selection source and effective execution settings, excluding credential values.

Private Test Content And Execution Options

Promptkit owns fixed, private prompt definitions for plain-text and structured-output smoke tests, including the latter's small schema. Their exact language is an implementation detail. Callers cannot override the prompt language, in whole or in part, or retrieve it through public prompt inspection, configuration inspection, prepared snapshots, or diagnostic reports. These definitions are not caller-configurable prompt catalog entries.

Injected model clients necessarily receive the messages they execute. That execution boundary does not make the smoke-test prompt a public inspection or customization feature, nor does private implementation content imply secrecy from the executing client or provider.

Applications select criteria independently of the fixed test content. An exact-text match need not be required merely to establish that inference completed. Profile selection changes execution settings, not prompt language. The supported per-call generation overrides and inheritance from the selected profile remain to be specified, including retry, repair, reasoning, and token limit defaults. Effective settings must be inspectable without revealing credentials or private prompt messages.

Promptkit reports timing without prescribing an acceptable latency threshold or a universal diagnostic deadline. Caller cancellation and configured execution timeouts still govern the operation. Applications decide whether the measured timing is acceptable and whether to impose additional deadlines.

Inference smoke tests make a small inference request and are likely to incur a small, nonzero cost. Promptkit does not estimate prices or guarantee a cost ceiling; prices and execution settings can change the actual cost. Report usage when available. Non-inference probes should use endpoints documented for that purpose; billing properties must not be inferred merely from an HTTP method.

Probes, Criteria, And Results

A probe performs an operation. Criteria evaluate observations from that operation. Several criteria can use one response without repeating the request. Candidate criteria include receiving an HTTP response, authentication acceptance, successful generation, returned text, expected-text matching, JSON parsing, and schema conformance. Each needs an explicit definition and a clear set of applicable probes.

For example, a response could satisfy successful generation and returned-text criteria while failing the built-in expected-text comparison. Promptkit reports those findings without assigning an overall application-readiness verdict. The application decides whether the mismatch warrants a warning, a failure, or no action. Matching and normalization rules remain to be specified.

Candidate criterion outcomes are passed, failed, unsupported, and inconclusive, with structured reasons. An unavailable authentication probe must not become an invalid-key verdict. A timeout during authentication may prevent determining credential acceptance even though the probe itself failed. Criteria that could not be evaluated must not be confused with assertions evaluated as false.

A report should identify the tested operation and effective target, applicable criteria and outcomes, and useful evidence such as HTTP status, elapsed time, attempts, and reported usage. Unknown measurements remain unknown. Inference retries or repair should not conceal initial failures behind a final success. Exact fields, observation retention, and the relationship between reports and returned Go errors remain open. Credential redaction and disclosure of provider diagnostics need explicit treatment consistent with existing contracts.

Implementation Direction

Reuse ordinary generation, decoding, validation, and error handling for inference tests wherever their behavior fits. Do not build a second inference stack. Private built-in prompt definitions can supply synthetic content through that machinery without exposing them through the public prompt catalog or configuration view. Existing behavior is described by the runner, generation coordinator, and model-client integration contract.

Reachability and authentication probes need a separate, narrow provider-facing capability. How injected clients participate must be decided without assuming that every custom client uses the built-in HTTP transport. Existing clients should not acquire new required methods merely to continue ordinary execution.

Decisions Needed Before Implementation

1. Public operation shape and first delivery scope. Choose individual methods, a typed single-probe operation, or a convenience batch operation; express the agreed profile-selection rules and define backend identification, absent backend defaults, prompt-version selection, and endpoint-only cases. Select initial provider adapters and decide whether model-status checks belong in the first delivery. Go names and configuration keys are not yet proposed.
2. Probe configuration and routing. Define supported probe kinds and their response semantics. Resolve endpoint overrides, relative paths, separate diagnostic hosts, redirects, and credential destinations deliberately so the report describes the destination actually tested.
3. Criteria and result contract. Specify applicability, exact matching, unsupported and inconclusive outcomes, errors versus diagnostic findings, and what remains available after cancellation or partial completion.
4. Execution mechanics. Choose private test content, supported setting overrides and inheritance, retry and repair defaults, capacity participation, observation, and cancellation behavior within the agreed diagnostic boundary. Custom smoke-test prompt language is excluded; application-authored tests remain available via ordinary Run.

Suggested Next Step

Write concrete request and result examples under the agreed boundaries for three cases: an HTTP response with rejected authentication, successful inference with an unexpected text response, and schema-valid structured output. Include a backend without an authentication probe to expose capability and outcome handling. Show prompt-based selection, an explicit profile overriding a prompt default, and selection through a backend's default smoke-test profile without disclosing private test language.

Use those examples to settle the operation and result shape, then specify the smallest backend configuration needed by the initial adapters. Convert the agreed design into a focused implementation plan with validation proportionate to the selected contracts under the testing policy. Default tests should remain offline using controlled transports and injected clients; live smoke testing would be an explicit consumer operation.

Scope Boundaries

This proposal does not include Scriptorium commands, application readiness policy, continuous monitoring, automatic probes before ordinary runs, a general-purpose diagnostic scripting language, public or customizable smoke-test prompt language, pricing estimates, application latency thresholds, or a guarantee about future generation. It does not change ValidateRequest or ordinary execution. Published API and format documentation should be updated when implementation lands, not to present this proposal as current behavior.