180 lines
11 KiB
Markdown
180 lines
11 KiB
Markdown
# Smoke Test Capability Proposal
|
|
|
|
Status: proposed for design review; implementation has not started.
|
|
|
|
## Purpose
|
|
|
|
Provide reusable diagnostic operations for endpoint reachability,
|
|
authentication, plain-text inference, and structured-output inference.
|
|
Applications building setup, diagnostic, or preflight workflows would otherwise
|
|
repeat similar prompts, schemas, provider requests, and result interpretation.
|
|
Promptkit should standardize those mechanisms while applications decide which
|
|
checks to request and how their outcomes affect application behavior.
|
|
|
|
This proposal captures the direction discussed with the maintainer. The API,
|
|
configuration shape, defaults, and delivery scope remain subject to review.
|
|
It is not an implementation plan or a new public contract.
|
|
|
|
## Relationship To Existing Operations
|
|
|
|
[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
|
|
checks whether a concrete request passes preparation without model generation.
|
|
`Run` prepares and executes the actual application request. Smoke tests would
|
|
instead exercise selected diagnostic operations using synthetic input or a
|
|
provider-specific probe. They need not load the application's prompt or inputs,
|
|
or call `ValidateRequest` on a future request.
|
|
|
|
A consumer could combine these operations into a preflight stage. Promptkit
|
|
would provide smoke test evidence, not decide that the application is ready.
|
|
Results describe the tested operation at the time of the check; they do not
|
|
reserve capacity or guarantee that a subsequent request will succeed.
|
|
|
|
The main value is shared diagnostics before real requests exist or when testing
|
|
a small synthetic request is useful. Consumers about to execute an ordinary
|
|
request may prefer to run it directly and handle its errors.
|
|
|
|
## Ownership
|
|
|
|
| Owner | Responsibility |
|
|
| --- | --- |
|
|
| Backend configuration | Declares available diagnostic operations and endpoints, including authentication or model-status endpoints where supported, and configuration needed by supported probe implementations. |
|
|
| Promptkit | Owns probe execution, default smoke test prompts and schemas, available criteria, their evaluators, and structured observations and outcomes. |
|
|
| Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. |
|
|
|
|
An endpoint address alone does not define a probe's meaning. A supported probe
|
|
implementation must know the request method, authentication requirements, and
|
|
how to interpret the response. A candidate design is for configuration to
|
|
identify a supported probe kind plus its endpoint settings. A general request
|
|
templating and response-expression language is not proposed initially.
|
|
|
|
Configuration changes must retain the existing
|
|
[ConfigKit ownership boundary](../adr/0001-configuration-boundary.md).
|
|
The [architecture policy](../policy/architecture.md) continues to govern the
|
|
library and downstream application boundary.
|
|
|
|
## Candidate Smoke Tests
|
|
|
|
These are independently selectable operations, not increasingly strict levels
|
|
that require every preceding test to pass.
|
|
|
|
| Test | Intended evidence | Important distinction |
|
|
| --- | --- | --- |
|
|
| HTTP/HTTPS reachability | An HTTP response was obtained from the selected destination. | A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available. |
|
|
| Authentication | A designated provider operation accepted the effective credentials. | Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract. |
|
|
| Plain-text inference | The selected model completed a small text-generation request. | Successful generation and compliance with an exact expected response are separate criteria. |
|
|
| Structured-output inference | The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria. | Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally. |
|
|
|
|
A backend model-status endpoint could provide additional provider-reported
|
|
evidence. It would not substitute for actually completing inference. Whether
|
|
to include model-status probes in the first implementation remains open.
|
|
|
|
Inference tests should select the effective profile or execution target the
|
|
caller wants to test. Automatically substituting a cheap unrelated model is
|
|
not part of this proposal. Backend-only operations should not require unrelated
|
|
prompt data or a runnable model selection.
|
|
|
|
## Default Test Content And Execution Options
|
|
|
|
Promptkit would maintain a short plain-text prompt asking for the word
|
|
`success`, and a separate structured-output prompt with a simple schema, such
|
|
as an object with a required boolean `success` property constrained to true.
|
|
Exact wording and schema are design choices still to be selected.
|
|
|
|
The defaults provide convenient repeatable tests. Applications select which
|
|
criteria to evaluate; an exact-text match need not be required merely to
|
|
establish that inference completed. A request to respond without thinking is
|
|
prompt content, not a substitute for explicit execution settings.
|
|
|
|
Callers should have options for relevant generation settings, timeouts, retries,
|
|
and repair. The design must specify which settings inherit from the selected
|
|
profile and which a smoke test overrides, and make the effective choices
|
|
inspectable without exposing credentials. No retry, repair, reasoning, or token
|
|
limit defaults are settled here. A short prompt alone does not guarantee low
|
|
cost. Non-inference probes should use endpoints documented for that purpose;
|
|
billing properties must not be inferred merely from an HTTP method.
|
|
|
|
## Probes, Criteria, And Results
|
|
|
|
A probe performs an operation. Criteria evaluate observations from that
|
|
operation. Several criteria can use one response without repeating the request.
|
|
Candidate criteria include receiving an HTTP response, authentication
|
|
acceptance, successful generation, returned text, expected-text matching, JSON
|
|
parsing, and schema conformance. Each needs an explicit definition and a clear
|
|
set of applicable probes.
|
|
|
|
For example, a model returning `Success!` could satisfy successful generation
|
|
and returned-text criteria while failing an exact match against `success`.
|
|
The application could treat the mismatch as a warning or a failure. Promptkit
|
|
would implement the comparison rather than requiring every application to
|
|
reimplement it. Matching and normalization rules remain to be specified.
|
|
|
|
Candidate criterion outcomes are passed, failed, unsupported, and inconclusive,
|
|
with structured reasons. An unavailable authentication probe must not become an
|
|
invalid-key verdict. A timeout during authentication may prevent determining
|
|
credential acceptance even though the probe itself failed. Criteria that could
|
|
not be evaluated must not be confused with assertions evaluated as false.
|
|
|
|
A report should identify the tested operation and effective target, applicable
|
|
criteria and outcomes, and useful evidence such as HTTP status, elapsed time,
|
|
attempts, and reported usage. Unknown measurements remain unknown. Inference
|
|
retries or repair should not conceal initial failures behind a final success.
|
|
Exact fields, observation retention, and the relationship between reports and
|
|
returned Go errors remain open. Credential redaction and disclosure of provider
|
|
diagnostics need explicit treatment consistent with existing contracts.
|
|
|
|
## Implementation Direction
|
|
|
|
Reuse ordinary generation, decoding, validation, and error handling for
|
|
inference tests wherever their behavior fits. Do not build a second inference
|
|
stack. Existing behavior is described by the
|
|
[runner](../internal/runner.md), [generation coordinator](../internal/generation.md),
|
|
and [model-client integration contract](../integrations/openai-compatible-chat.md).
|
|
|
|
Reachability and authentication probes need a separate, narrow provider-facing
|
|
capability. How injected clients participate must be decided without assuming
|
|
that every custom client uses the built-in HTTP transport. Existing clients
|
|
should not acquire new required methods merely to continue ordinary execution.
|
|
|
|
## Decisions Needed Before Implementation
|
|
|
|
1. **Public operation shape and first delivery scope.** Choose individual
|
|
methods, a typed single-probe operation, or a convenience batch operation;
|
|
define target selection for backend, profile, and endpoint-only cases. Select
|
|
initial provider adapters and decide whether model-status checks belong in
|
|
the first delivery. Go names and configuration keys are not yet proposed.
|
|
2. **Probe configuration and routing.** Define supported probe kinds and their
|
|
response semantics. Resolve endpoint overrides, relative paths, separate
|
|
diagnostic hosts, redirects, and credential destinations deliberately so
|
|
the report describes the destination actually tested.
|
|
3. **Criteria and result contract.** Specify applicability, exact matching,
|
|
unsupported and inconclusive outcomes, errors versus diagnostic findings,
|
|
and what remains available after cancellation or partial completion.
|
|
4. **Execution policy.** Choose default test content, profile inheritance and
|
|
overrides, retry and repair defaults, capacity participation, observation,
|
|
and cancellation behavior. Decide whether custom test content is needed
|
|
initially; ordinary application-authored prompts remain available via `Run`.
|
|
|
|
## Suggested Next Step
|
|
|
|
Review this proposal's scope and ownership first. Then write concrete request
|
|
and result examples for three cases: an HTTP response with rejected
|
|
authentication, successful inference with an unexpected text response, and
|
|
schema-valid structured output. Include a backend without an authentication
|
|
probe to expose capability and outcome handling.
|
|
|
|
Use those examples to settle the operation and result shape, then specify the
|
|
smallest backend configuration needed by the initial adapters. Convert the
|
|
agreed design into a focused implementation plan with validation proportionate
|
|
to the selected contracts under the [testing policy](../policy/testing.md).
|
|
Default tests should remain offline using controlled transports and injected
|
|
clients; live smoke testing would be an explicit consumer operation.
|
|
|
|
## Scope Boundaries
|
|
|
|
This proposal does not include Scriptorium commands, application readiness
|
|
policy, continuous monitoring, automatic probes before ordinary runs, a
|
|
general-purpose diagnostic scripting language, or a guarantee about future
|
|
generation. It does not change `ValidateRequest` or ordinary execution.
|
|
Published API and format documentation should be updated when implementation
|
|
lands, not to present this proposal as current behavior.
|