eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Record smoke test design decisions and profile precedence

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-13 13:33:12 +00:00
parent 64769883ae
commit d4676bc870
1 changed files with 95 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

141
docs/roadmap/smoke-tests.md
View File

@@ -1,6 +1,7 @@
# Smoke Test Capability Proposal # Smoke Test Capability Proposal
Status: proposed for design review; implementation has not started. Status: core design decisions accepted; API details and delivery scope remain
under review. Implementation has not started.
## Purpose ## Purpose
@@ -11,9 +12,10 @@ repeat similar prompts, schemas, provider requests, and result interpretation.
Promptkit should standardize those mechanisms while applications decide which Promptkit should standardize those mechanisms while applications decide which
checks to request and how their outcomes affect application behavior. checks to request and how their outcomes affect application behavior.
This proposal captures the direction discussed with the maintainer. The API, This proposal records the agreed diagnostic boundary, private test content, and
configuration shape, defaults, and delivery scope remain subject to review. execution-profile selection rules. The concrete API, configuration shape,
It is not an implementation plan or a new public contract. remaining execution defaults, and delivery scope remain subject to review.
It is not an implementation plan or a description of current public behavior.
## Relationship To Existing Operations ## Relationship To Existing Operations
@@ -21,8 +23,10 @@ It is not an implementation plan or a new public contract.
checks whether a concrete request passes preparation without model generation. checks whether a concrete request passes preparation without model generation.
`Run` prepares and executes the actual application request. Smoke tests would `Run` prepares and executes the actual application request. Smoke tests would
instead exercise selected diagnostic operations using synthetic input or a instead exercise selected diagnostic operations using synthetic input or a
provider-specific probe. They need not load the application's prompt or inputs, provider-specific probe. An application prompt may be selected to obtain its
or call `ValidateRequest` on a future request. default profile, but its messages are not executed and its variables and inputs
are not required. Smoke tests do not need to call `ValidateRequest` on a future
request.
A consumer could combine these operations into a preflight stage. Promptkit A consumer could combine these operations into a preflight stage. Promptkit
would provide smoke test evidence, not decide that the application is ready. would provide smoke test evidence, not decide that the application is ready.
@@ -37,8 +41,8 @@ request may prefer to run it directly and handle its errors.
| Owner | Responsibility | | Owner | Responsibility |
| --- | --- | | --- | --- |
| Backend configuration | Declares available diagnostic operations and endpoints, including authentication or model-status endpoints where supported, and configuration needed by supported probe implementations. | | Backend configuration | Declares available diagnostic operations and endpoints, configuration needed by supported probe implementations, and the backend's default smoke-test execution profile. |
| Promptkit | Owns probe execution, default smoke test prompts and schemas, available criteria, their evaluators, and structured observations and outcomes. | | Promptkit | Owns probe execution, private fixed smoke-test prompts and schemas, available criteria, their evaluators, and neutral diagnostic reports. |
| Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. | | Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. |
An endpoint address alone does not define a probe's meaning. A supported probe An endpoint address alone does not define a probe's meaning. A supported probe
@@ -68,30 +72,66 @@ A backend model-status endpoint could provide additional provider-reported
evidence. It would not substitute for actually completing inference. Whether evidence. It would not substitute for actually completing inference. Whether
to include model-status probes in the first implementation remains open. to include model-status probes in the first implementation remains open.
Inference tests should select the effective profile or execution target the Backend-only operations should not require unrelated prompt data or a runnable
caller wants to test. Automatically substituting a cheap unrelated model is model selection.
not part of this proposal. Backend-only operations should not require unrelated
prompt data or a runnable model selection.
## Default Test Content And Execution Options ## Inference Profile Selection
Promptkit would maintain a short plain-text prompt asking for the word Callers can override the backend's default smoke-test execution profile through
`success`, and a separate structured-output prompt with a simple schema, such the following selections:
as an object with a required boolean `success` property constrained to true.
Exact wording and schema are design choices still to be selected.
The defaults provide convenient repeatable tests. Applications select which | Caller supplies | Execution profile |
criteria to evaluate; an exact-text match need not be required merely to | --- | --- |
establish that inference completed. A request to respond without thinking is | Prompt ID without a profile override | The selected prompt definition's default profile. |
prompt content, not a substitute for explicit execution settings. | Profile ID without a prompt ID | The explicitly selected profile. |
| Both prompt and profile IDs | The supplied profile overrides the prompt's default under ordinary Promptkit selection rules. |
| Neither prompt nor profile ID | The selected backend's default smoke-test profile. |
Callers should have options for relevant generation settings, timeouts, retries, Reuse existing prompt/profile selection semantics, including exact selection
and repair. The design must specify which settings inherit from the selected and omitted-selector handling; see the [consumer guide](../consumers/pkg-promptkit.md)
profile and which a smoke test overrides, and make the effective choices and [request GoDoc](../../types.go). The backend default applies when neither
inspectable without exposing credentials. No retry, repair, reasoning, or token selector is supplied, not as a replacement for an invalid explicit selection.
limit defaults are settled here. A short prompt alone does not guarantee low How callers identify the backend in that case, and behavior when its default
cost. Non-inference probes should use endpoints documented for that purpose; smoke-test profile is absent, remain to be specified.
billing properties must not be inferred merely from an HTTP method.
Selecting an application prompt supplies execution-profile selection only. It
does not replace the synthetic smoke-test messages or import the application's
output contract. An explicit profile selection must not be silently replaced
with a cheaper model. Reports should identify the selection source and effective
execution settings, excluding credential values.
## Private Test Content And Execution Options
Promptkit owns fixed, private prompt definitions for plain-text and
structured-output smoke tests, including the latter's small schema. Their exact
language is an implementation detail. Callers cannot override the prompt
language, in whole or in part, or retrieve it through public prompt inspection,
configuration inspection, prepared snapshots, or diagnostic reports. These
definitions are not caller-configurable prompt catalog entries.
Injected model clients necessarily receive the messages they execute. That
execution boundary does not make the smoke-test prompt a public inspection or
customization feature, nor does private implementation content imply secrecy
from the executing client or provider.
Applications select criteria independently of the fixed test content. An
exact-text match need not be required merely to establish that inference
completed. Profile selection changes execution settings, not prompt language.
The supported per-call generation overrides and inheritance from the selected
profile remain to be specified, including retry, repair, reasoning, and token
limit defaults. Effective settings must be inspectable without revealing
credentials or private prompt messages.
Promptkit reports timing without prescribing an acceptable latency threshold
or a universal diagnostic deadline. Caller cancellation and configured execution
timeouts still govern the operation. Applications decide whether the measured
timing is acceptable and whether to impose additional deadlines.
Inference smoke tests make a small inference request and are likely to incur a
small, nonzero cost. Promptkit does not estimate prices or guarantee a cost
ceiling; prices and execution settings can change the actual cost. Report usage
when available. Non-inference probes should use endpoints documented for that
purpose; billing properties must not be inferred merely from an HTTP method.
## Probes, Criteria, And Results ## Probes, Criteria, And Results
@@ -102,11 +142,11 @@ acceptance, successful generation, returned text, expected-text matching, JSON
parsing, and schema conformance. Each needs an explicit definition and a clear parsing, and schema conformance. Each needs an explicit definition and a clear
set of applicable probes. set of applicable probes.
For example, a model returning `Success!` could satisfy successful generation For example, a response could satisfy successful generation and returned-text
and returned-text criteria while failing an exact match against `success`. criteria while failing the built-in expected-text comparison. Promptkit reports
The application could treat the mismatch as a warning or a failure. Promptkit those findings without assigning an overall application-readiness verdict. The
would implement the comparison rather than requiring every application to application decides whether the mismatch warrants a warning, a failure, or no
reimplement it. Matching and normalization rules remain to be specified. action. Matching and normalization rules remain to be specified.
Candidate criterion outcomes are passed, failed, unsupported, and inconclusive, Candidate criterion outcomes are passed, failed, unsupported, and inconclusive,
with structured reasons. An unavailable authentication probe must not become an with structured reasons. An unavailable authentication probe must not become an
@@ -126,7 +166,9 @@ diagnostics need explicit treatment consistent with existing contracts.
Reuse ordinary generation, decoding, validation, and error handling for Reuse ordinary generation, decoding, validation, and error handling for
inference tests wherever their behavior fits. Do not build a second inference inference tests wherever their behavior fits. Do not build a second inference
stack. Existing behavior is described by the stack. Private built-in prompt definitions can supply synthetic content through
that machinery without exposing them through the public prompt catalog or
configuration view. Existing behavior is described by the
[runner](../internal/runner.md), [generation coordinator](../internal/generation.md), [runner](../internal/runner.md), [generation coordinator](../internal/generation.md),
and [model-client integration contract](../integrations/openai-compatible-chat.md). and [model-client integration contract](../integrations/openai-compatible-chat.md).
@@ -139,9 +181,11 @@ should not acquire new required methods merely to continue ordinary execution.
1. **Public operation shape and first delivery scope.** Choose individual 1. **Public operation shape and first delivery scope.** Choose individual
methods, a typed single-probe operation, or a convenience batch operation; methods, a typed single-probe operation, or a convenience batch operation;
define target selection for backend, profile, and endpoint-only cases. Select express the agreed profile-selection rules and define backend identification,
initial provider adapters and decide whether model-status checks belong in absent backend defaults, prompt-version selection, and endpoint-only cases.
the first delivery. Go names and configuration keys are not yet proposed. Select initial provider adapters and decide whether model-status checks
belong in the first delivery. Go names and configuration keys are not yet
proposed.
2. **Probe configuration and routing.** Define supported probe kinds and their 2. **Probe configuration and routing.** Define supported probe kinds and their
response semantics. Resolve endpoint overrides, relative paths, separate response semantics. Resolve endpoint overrides, relative paths, separate
diagnostic hosts, redirects, and credential destinations deliberately so diagnostic hosts, redirects, and credential destinations deliberately so
@@ -149,18 +193,21 @@ should not acquire new required methods merely to continue ordinary execution.
3. **Criteria and result contract.** Specify applicability, exact matching, 3. **Criteria and result contract.** Specify applicability, exact matching,
unsupported and inconclusive outcomes, errors versus diagnostic findings, unsupported and inconclusive outcomes, errors versus diagnostic findings,
and what remains available after cancellation or partial completion. and what remains available after cancellation or partial completion.
4. **Execution policy.** Choose default test content, profile inheritance and 4. **Execution mechanics.** Choose private test content, supported setting
overrides, retry and repair defaults, capacity participation, observation, overrides and inheritance, retry and repair defaults, capacity participation,
and cancellation behavior. Decide whether custom test content is needed observation, and cancellation behavior within the agreed diagnostic boundary.
initially; ordinary application-authored prompts remain available via `Run`. Custom smoke-test prompt language is excluded; application-authored tests
remain available via ordinary `Run`.
## Suggested Next Step ## Suggested Next Step
Review this proposal's scope and ownership first. Then write concrete request Write concrete request and result examples under the agreed boundaries for
and result examples for three cases: an HTTP response with rejected three cases: an HTTP response with rejected
authentication, successful inference with an unexpected text response, and authentication, successful inference with an unexpected text response, and
schema-valid structured output. Include a backend without an authentication schema-valid structured output. Include a backend without an authentication
probe to expose capability and outcome handling. probe to expose capability and outcome handling. Show prompt-based selection,
an explicit profile overriding a prompt default, and selection through a
backend's default smoke-test profile without disclosing private test language.
Use those examples to settle the operation and result shape, then specify the Use those examples to settle the operation and result shape, then specify the
smallest backend configuration needed by the initial adapters. Convert the smallest backend configuration needed by the initial adapters. Convert the
@@ -173,7 +220,9 @@ clients; live smoke testing would be an explicit consumer operation.
This proposal does not include Scriptorium commands, application readiness This proposal does not include Scriptorium commands, application readiness
policy, continuous monitoring, automatic probes before ordinary runs, a policy, continuous monitoring, automatic probes before ordinary runs, a
general-purpose diagnostic scripting language, or a guarantee about future general-purpose diagnostic scripting language, public or customizable smoke-test
generation. It does not change `ValidateRequest` or ordinary execution. prompt language, pricing estimates, application latency thresholds, or a
guarantee about future generation. It does not change `ValidateRequest` or
ordinary execution.
Published API and format documentation should be updated when implementation Published API and format documentation should be updated when implementation
lands, not to present this proposal as current behavior. lands, not to present this proposal as current behavior.