promptkit/docs/roadmap/smoke-tests.md

# Smoke Test Capability Proposal

Status: core design decisions accepted; API details and delivery scope remain
under review. Implementation has not started.

## Purpose

Provide reusable diagnostic operations for endpoint reachability,
authentication, plain-text inference, and structured-output inference.
Applications building setup, diagnostic, or preflight workflows would otherwise
repeat similar prompts, schemas, provider requests, and result interpretation.
Promptkit should standardize those mechanisms while applications decide which
checks to request and how their outcomes affect application behavior.

This proposal records the agreed diagnostic boundary, private test content, and
execution-profile selection rules. The concrete API, configuration shape,
remaining execution defaults, and delivery scope remain subject to review.
It is not an implementation plan or a description of current public behavior.

## Relationship To Existing Operations

[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
checks whether a concrete request passes preparation without model generation.
`Run` prepares and executes the actual application request. Smoke tests would
instead exercise selected diagnostic operations using synthetic input or a
provider-specific probe. An application prompt may be selected to obtain its
default profile, but its messages are not executed and its variables and inputs
are not required. Smoke tests do not need to call `ValidateRequest` on a future
request.

A consumer could combine these operations into a preflight stage. Promptkit
would provide smoke test evidence, not decide that the application is ready.
Results describe the tested operation at the time of the check; they do not
reserve capacity or guarantee that a subsequent request will succeed.

The main value is shared diagnostics before real requests exist or when testing
a small synthetic request is useful. Consumers about to execute an ordinary
request may prefer to run it directly and handle its errors.

## Ownership

| Owner | Responsibility |
| --- | --- |
| Backend configuration | Declares available diagnostic operations and endpoints, configuration needed by supported probe implementations, and the backend's default smoke-test execution profile. |
| Promptkit | Owns probe execution, private fixed smoke-test prompts and schemas, available criteria, their evaluators, and neutral diagnostic reports. |
| Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. |

An endpoint address alone does not define a probe's meaning. A supported probe
implementation must know the request method, authentication requirements, and
how to interpret the response. A candidate design is for configuration to
identify a supported probe kind plus its endpoint settings. A general request
templating and response-expression language is not proposed initially.

Configuration changes must retain the existing
[ConfigKit ownership boundary](../adr/0001-configuration-boundary.md).
The [architecture policy](../policy/architecture.md) continues to govern the
library and downstream application boundary.

## Candidate Smoke Tests

These are independently selectable operations, not increasingly strict levels
that require every preceding test to pass.

| Test | Intended evidence | Important distinction |
| --- | --- | --- |
| HTTP/HTTPS reachability | An HTTP response was obtained from the selected destination. | A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available. |
| Authentication | A designated provider operation accepted the effective credentials. | Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract. |
| Plain-text inference | The selected model completed a small text-generation request. | Successful generation and compliance with an exact expected response are separate criteria. |
| Structured-output inference | The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria. | Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally. |

A backend model-status endpoint could provide additional provider-reported
evidence. It would not substitute for actually completing inference. Whether
to include model-status probes in the first implementation remains open.

Backend-only operations should not require unrelated prompt data or a runnable
model selection.

## Inference Profile Selection

Callers can override the backend's default smoke-test execution profile through
the following selections:

| Caller supplies | Execution profile |
| --- | --- |
| Prompt ID without a profile override | The selected prompt definition's default profile. |
| Profile ID without a prompt ID | The explicitly selected profile. |
| Both prompt and profile IDs | The supplied profile overrides the prompt's default under ordinary Promptkit selection rules. |
| Neither prompt nor profile ID | The selected backend's default smoke-test profile. |

Reuse existing prompt/profile selection semantics, including exact selection
and omitted-selector handling; see the [consumer guide](../consumers/pkg-promptkit.md)
and [request GoDoc](../../types.go). The backend default applies when neither
selector is supplied, not as a replacement for an invalid explicit selection.
How callers identify the backend in that case, and behavior when its default
smoke-test profile is absent, remain to be specified.

Selecting an application prompt supplies execution-profile selection only. It
does not replace the synthetic smoke-test messages or import the application's
output contract. An explicit profile selection must not be silently replaced
with a cheaper model. Reports should identify the selection source and effective
execution settings, excluding credential values.

## Private Test Content And Execution Options

Promptkit owns fixed, private prompt definitions for plain-text and
structured-output smoke tests, including the latter's small schema. Their exact
language is an implementation detail. Callers cannot override the prompt
language, in whole or in part, or retrieve it through public prompt inspection,
configuration inspection, prepared snapshots, or diagnostic reports. These
definitions are not caller-configurable prompt catalog entries.

Injected model clients necessarily receive the messages they execute. That
execution boundary does not make the smoke-test prompt a public inspection or
customization feature, nor does private implementation content imply secrecy
from the executing client or provider.

Applications select criteria independently of the fixed test content. An
exact-text match need not be required merely to establish that inference
completed. Profile selection changes execution settings, not prompt language.
The supported per-call generation overrides and inheritance from the selected
profile remain to be specified, including retry, repair, reasoning, and token
limit defaults. Effective settings must be inspectable without revealing
credentials or private prompt messages.

Promptkit reports timing without prescribing an acceptable latency threshold
or a universal diagnostic deadline. Caller cancellation and configured execution
timeouts still govern the operation. Applications decide whether the measured
timing is acceptable and whether to impose additional deadlines.

Inference smoke tests make a small inference request and are likely to incur a
small, nonzero cost. Promptkit does not estimate prices or guarantee a cost
ceiling; prices and execution settings can change the actual cost. Report usage
when available. Non-inference probes should use endpoints documented for that
purpose; billing properties must not be inferred merely from an HTTP method.

## Probes, Criteria, And Results

A probe performs an operation. Criteria evaluate observations from that
operation. Several criteria can use one response without repeating the request.
Candidate criteria include receiving an HTTP response, authentication
acceptance, successful generation, returned text, expected-text matching, JSON
parsing, and schema conformance. Each needs an explicit definition and a clear
set of applicable probes.

For example, a response could satisfy successful generation and returned-text
criteria while failing the built-in expected-text comparison. Promptkit reports
those findings without assigning an overall application-readiness verdict. The
application decides whether the mismatch warrants a warning, a failure, or no
action. Matching and normalization rules remain to be specified.

Candidate criterion outcomes are passed, failed, unsupported, and inconclusive,
with structured reasons. An unavailable authentication probe must not become an
invalid-key verdict. A timeout during authentication may prevent determining
credential acceptance even though the probe itself failed. Criteria that could
not be evaluated must not be confused with assertions evaluated as false.

A report should identify the tested operation and effective target, applicable
criteria and outcomes, and useful evidence such as HTTP status, elapsed time,
attempts, and reported usage. Unknown measurements remain unknown. Inference
retries or repair should not conceal initial failures behind a final success.
Exact fields, observation retention, and the relationship between reports and
returned Go errors remain open. Credential redaction and disclosure of provider
diagnostics need explicit treatment consistent with existing contracts.

## Implementation Direction

Reuse ordinary generation, decoding, validation, and error handling for
inference tests wherever their behavior fits. Do not build a second inference
stack. Private built-in prompt definitions can supply synthetic content through
that machinery without exposing them through the public prompt catalog or
configuration view. Existing behavior is described by the
[runner](../internal/runner.md), [generation coordinator](../internal/generation.md),
and [model-client integration contract](../integrations/openai-compatible-chat.md).

Reachability and authentication probes need a separate, narrow provider-facing
capability. How injected clients participate must be decided without assuming
that every custom client uses the built-in HTTP transport. Existing clients
should not acquire new required methods merely to continue ordinary execution.

## Decisions Needed Before Implementation

1. **Public operation shape and first delivery scope.** Choose individual
   methods, a typed single-probe operation, or a convenience batch operation;
   express the agreed profile-selection rules and define backend identification,
   absent backend defaults, prompt-version selection, and endpoint-only cases.
   Select initial provider adapters and decide whether model-status checks
   belong in the first delivery. Go names and configuration keys are not yet
   proposed.
2. **Probe configuration and routing.** Define supported probe kinds and their
   response semantics. Resolve endpoint overrides, relative paths, separate
   diagnostic hosts, redirects, and credential destinations deliberately so
   the report describes the destination actually tested.
3. **Criteria and result contract.** Specify applicability, exact matching,
   unsupported and inconclusive outcomes, errors versus diagnostic findings,
   and what remains available after cancellation or partial completion.
4. **Execution mechanics.** Choose private test content, supported setting
   overrides and inheritance, retry and repair defaults, capacity participation,
   observation, and cancellation behavior within the agreed diagnostic boundary.
   Custom smoke-test prompt language is excluded; application-authored tests
   remain available via ordinary `Run`.

## Suggested Next Step

Write concrete request and result examples under the agreed boundaries for
three cases: an HTTP response with rejected
authentication, successful inference with an unexpected text response, and
schema-valid structured output. Include a backend without an authentication
probe to expose capability and outcome handling. Show prompt-based selection,
an explicit profile overriding a prompt default, and selection through a
backend's default smoke-test profile without disclosing private test language.

Use those examples to settle the operation and result shape, then specify the
smallest backend configuration needed by the initial adapters. Convert the
agreed design into a focused implementation plan with validation proportionate
to the selected contracts under the [testing policy](../policy/testing.md).
Default tests should remain offline using controlled transports and injected
clients; live smoke testing would be an explicit consumer operation.

## Scope Boundaries

This proposal does not include Scriptorium commands, application readiness
policy, continuous monitoring, automatic probes before ordinary runs, a
general-purpose diagnostic scripting language, public or customizable smoke-test
prompt language, pricing estimates, application latency thresholds, or a
guarantee about future generation. It does not change `ValidateRequest` or
ordinary execution.
Published API and format documentation should be updated when implementation
lands, not to present this proposal as current behavior.