eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Draft smoke test capability proposal

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-13 00:33:21 +00:00
parent 635674c066
commit 64769883ae
1 changed files with 179 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

179
docs/roadmap/smoke-tests.md Normal file
View File

@@ -0,0 +1,179 @@
# Smoke Test Capability Proposal
Status: proposed for design review; implementation has not started.
## Purpose
Provide reusable diagnostic operations for endpoint reachability,
authentication, plain-text inference, and structured-output inference.
Applications building setup, diagnostic, or preflight workflows would otherwise
repeat similar prompts, schemas, provider requests, and result interpretation.
Promptkit should standardize those mechanisms while applications decide which
checks to request and how their outcomes affect application behavior.
This proposal captures the direction discussed with the maintainer. The API,
configuration shape, defaults, and delivery scope remain subject to review.
It is not an implementation plan or a new public contract.
## Relationship To Existing Operations
[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
checks whether a concrete request passes preparation without model generation.
`Run` prepares and executes the actual application request. Smoke tests would
instead exercise selected diagnostic operations using synthetic input or a
provider-specific probe. They need not load the application's prompt or inputs,
or call `ValidateRequest` on a future request.
A consumer could combine these operations into a preflight stage. Promptkit
would provide smoke test evidence, not decide that the application is ready.
Results describe the tested operation at the time of the check; they do not
reserve capacity or guarantee that a subsequent request will succeed.
The main value is shared diagnostics before real requests exist or when testing
a small synthetic request is useful. Consumers about to execute an ordinary
request may prefer to run it directly and handle its errors.
## Ownership
| Owner | Responsibility |
| --- | --- |
| Backend configuration | Declares available diagnostic operations and endpoints, including authentication or model-status endpoints where supported, and configuration needed by supported probe implementations. |
| Promptkit | Owns probe execution, default smoke test prompts and schemas, available criteria, their evaluators, and structured observations and outcomes. |
| Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. |
An endpoint address alone does not define a probe's meaning. A supported probe
implementation must know the request method, authentication requirements, and
how to interpret the response. A candidate design is for configuration to
identify a supported probe kind plus its endpoint settings. A general request
templating and response-expression language is not proposed initially.
Configuration changes must retain the existing
[ConfigKit ownership boundary](../adr/0001-configuration-boundary.md).
The [architecture policy](../policy/architecture.md) continues to govern the
library and downstream application boundary.
## Candidate Smoke Tests
These are independently selectable operations, not increasingly strict levels
that require every preceding test to pass.
| Test | Intended evidence | Important distinction |
| --- | --- | --- |
| HTTP/HTTPS reachability | An HTTP response was obtained from the selected destination. | A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available. |
| Authentication | A designated provider operation accepted the effective credentials. | Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract. |
| Plain-text inference | The selected model completed a small text-generation request. | Successful generation and compliance with an exact expected response are separate criteria. |
| Structured-output inference | The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria. | Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally. |
A backend model-status endpoint could provide additional provider-reported
evidence. It would not substitute for actually completing inference. Whether
to include model-status probes in the first implementation remains open.
Inference tests should select the effective profile or execution target the
caller wants to test. Automatically substituting a cheap unrelated model is
not part of this proposal. Backend-only operations should not require unrelated
prompt data or a runnable model selection.
## Default Test Content And Execution Options
Promptkit would maintain a short plain-text prompt asking for the word
`success`, and a separate structured-output prompt with a simple schema, such
as an object with a required boolean `success` property constrained to true.
Exact wording and schema are design choices still to be selected.
The defaults provide convenient repeatable tests. Applications select which
criteria to evaluate; an exact-text match need not be required merely to
establish that inference completed. A request to respond without thinking is
prompt content, not a substitute for explicit execution settings.
Callers should have options for relevant generation settings, timeouts, retries,
and repair. The design must specify which settings inherit from the selected
profile and which a smoke test overrides, and make the effective choices
inspectable without exposing credentials. No retry, repair, reasoning, or token
limit defaults are settled here. A short prompt alone does not guarantee low
cost. Non-inference probes should use endpoints documented for that purpose;
billing properties must not be inferred merely from an HTTP method.
## Probes, Criteria, And Results
A probe performs an operation. Criteria evaluate observations from that
operation. Several criteria can use one response without repeating the request.
Candidate criteria include receiving an HTTP response, authentication
acceptance, successful generation, returned text, expected-text matching, JSON
parsing, and schema conformance. Each needs an explicit definition and a clear
set of applicable probes.
For example, a model returning `Success!` could satisfy successful generation
and returned-text criteria while failing an exact match against `success`.
The application could treat the mismatch as a warning or a failure. Promptkit
would implement the comparison rather than requiring every application to
reimplement it. Matching and normalization rules remain to be specified.
Candidate criterion outcomes are passed, failed, unsupported, and inconclusive,
with structured reasons. An unavailable authentication probe must not become an
invalid-key verdict. A timeout during authentication may prevent determining
credential acceptance even though the probe itself failed. Criteria that could
not be evaluated must not be confused with assertions evaluated as false.
A report should identify the tested operation and effective target, applicable
criteria and outcomes, and useful evidence such as HTTP status, elapsed time,
attempts, and reported usage. Unknown measurements remain unknown. Inference
retries or repair should not conceal initial failures behind a final success.
Exact fields, observation retention, and the relationship between reports and
returned Go errors remain open. Credential redaction and disclosure of provider
diagnostics need explicit treatment consistent with existing contracts.
## Implementation Direction
Reuse ordinary generation, decoding, validation, and error handling for
inference tests wherever their behavior fits. Do not build a second inference
stack. Existing behavior is described by the
[runner](../internal/runner.md), [generation coordinator](../internal/generation.md),
and [model-client integration contract](../integrations/openai-compatible-chat.md).
Reachability and authentication probes need a separate, narrow provider-facing
capability. How injected clients participate must be decided without assuming
that every custom client uses the built-in HTTP transport. Existing clients
should not acquire new required methods merely to continue ordinary execution.
## Decisions Needed Before Implementation
1. **Public operation shape and first delivery scope.** Choose individual
methods, a typed single-probe operation, or a convenience batch operation;
define target selection for backend, profile, and endpoint-only cases. Select
initial provider adapters and decide whether model-status checks belong in
the first delivery. Go names and configuration keys are not yet proposed.
2. **Probe configuration and routing.** Define supported probe kinds and their
response semantics. Resolve endpoint overrides, relative paths, separate
diagnostic hosts, redirects, and credential destinations deliberately so
the report describes the destination actually tested.
3. **Criteria and result contract.** Specify applicability, exact matching,
unsupported and inconclusive outcomes, errors versus diagnostic findings,
and what remains available after cancellation or partial completion.
4. **Execution policy.** Choose default test content, profile inheritance and
overrides, retry and repair defaults, capacity participation, observation,
and cancellation behavior. Decide whether custom test content is needed
initially; ordinary application-authored prompts remain available via `Run`.
## Suggested Next Step
Review this proposal's scope and ownership first. Then write concrete request
and result examples for three cases: an HTTP response with rejected
authentication, successful inference with an unexpected text response, and
schema-valid structured output. Include a backend without an authentication
probe to expose capability and outcome handling.
Use those examples to settle the operation and result shape, then specify the
smallest backend configuration needed by the initial adapters. Convert the
agreed design into a focused implementation plan with validation proportionate
to the selected contracts under the [testing policy](../policy/testing.md).
Default tests should remain offline using controlled transports and injected
clients; live smoke testing would be an explicit consumer operation.
## Scope Boundaries
This proposal does not include Scriptorium commands, application readiness
policy, continuous monitoring, automatic probes before ordinary runs, a
general-purpose diagnostic scripting language, or a guarantee about future
generation. It does not change `ValidateRequest` or ordinary execution.
Published API and format documentation should be updated when implementation
lands, not to present this proposal as current behavior.