From d4676bc870c1d1ec0a4d163a62e2f58493309483 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 13 Oct 2026 13:33:12 +0000 Subject: [PATCH] Record smoke test design decisions and profile precedence --- docs/roadmap/smoke-tests.md | 141 ++++++++++++++++++++++++------------ 1 file changed, 95 insertions(+), 46 deletions(-) diff --git a/docs/roadmap/smoke-tests.md b/docs/roadmap/smoke-tests.md index ade4839..fa0295b 100644 --- a/docs/roadmap/smoke-tests.md +++ b/docs/roadmap/smoke-tests.md @@ -1,6 +1,7 @@ # Smoke Test Capability Proposal -Status: proposed for design review; implementation has not started. +Status: core design decisions accepted; API details and delivery scope remain +under review. Implementation has not started. ## Purpose @@ -11,9 +12,10 @@ repeat similar prompts, schemas, provider requests, and result interpretation. Promptkit should standardize those mechanisms while applications decide which checks to request and how their outcomes affect application behavior. -This proposal captures the direction discussed with the maintainer. The API, -configuration shape, defaults, and delivery scope remain subject to review. -It is not an implementation plan or a new public contract. +This proposal records the agreed diagnostic boundary, private test content, and +execution-profile selection rules. The concrete API, configuration shape, +remaining execution defaults, and delivery scope remain subject to review. +It is not an implementation plan or a description of current public behavior. ## Relationship To Existing Operations @@ -21,8 +23,10 @@ It is not an implementation plan or a new public contract. checks whether a concrete request passes preparation without model generation. `Run` prepares and executes the actual application request. Smoke tests would instead exercise selected diagnostic operations using synthetic input or a -provider-specific probe. They need not load the application's prompt or inputs, -or call `ValidateRequest` on a future request. +provider-specific probe. An application prompt may be selected to obtain its +default profile, but its messages are not executed and its variables and inputs +are not required. Smoke tests do not need to call `ValidateRequest` on a future +request. A consumer could combine these operations into a preflight stage. Promptkit would provide smoke test evidence, not decide that the application is ready. @@ -37,8 +41,8 @@ request may prefer to run it directly and handle its errors. | Owner | Responsibility | | --- | --- | -| Backend configuration | Declares available diagnostic operations and endpoints, including authentication or model-status endpoints where supported, and configuration needed by supported probe implementations. | -| Promptkit | Owns probe execution, default smoke test prompts and schemas, available criteria, their evaluators, and structured observations and outcomes. | +| Backend configuration | Declares available diagnostic operations and endpoints, configuration needed by supported probe implementations, and the backend's default smoke-test execution profile. | +| Promptkit | Owns probe execution, private fixed smoke-test prompts and schemas, available criteria, their evaluators, and neutral diagnostic reports. | | Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. | An endpoint address alone does not define a probe's meaning. A supported probe @@ -68,30 +72,66 @@ A backend model-status endpoint could provide additional provider-reported evidence. It would not substitute for actually completing inference. Whether to include model-status probes in the first implementation remains open. -Inference tests should select the effective profile or execution target the -caller wants to test. Automatically substituting a cheap unrelated model is -not part of this proposal. Backend-only operations should not require unrelated -prompt data or a runnable model selection. +Backend-only operations should not require unrelated prompt data or a runnable +model selection. -## Default Test Content And Execution Options +## Inference Profile Selection -Promptkit would maintain a short plain-text prompt asking for the word -`success`, and a separate structured-output prompt with a simple schema, such -as an object with a required boolean `success` property constrained to true. -Exact wording and schema are design choices still to be selected. +Callers can override the backend's default smoke-test execution profile through +the following selections: -The defaults provide convenient repeatable tests. Applications select which -criteria to evaluate; an exact-text match need not be required merely to -establish that inference completed. A request to respond without thinking is -prompt content, not a substitute for explicit execution settings. +| Caller supplies | Execution profile | +| --- | --- | +| Prompt ID without a profile override | The selected prompt definition's default profile. | +| Profile ID without a prompt ID | The explicitly selected profile. | +| Both prompt and profile IDs | The supplied profile overrides the prompt's default under ordinary Promptkit selection rules. | +| Neither prompt nor profile ID | The selected backend's default smoke-test profile. | -Callers should have options for relevant generation settings, timeouts, retries, -and repair. The design must specify which settings inherit from the selected -profile and which a smoke test overrides, and make the effective choices -inspectable without exposing credentials. No retry, repair, reasoning, or token -limit defaults are settled here. A short prompt alone does not guarantee low -cost. Non-inference probes should use endpoints documented for that purpose; -billing properties must not be inferred merely from an HTTP method. +Reuse existing prompt/profile selection semantics, including exact selection +and omitted-selector handling; see the [consumer guide](../consumers/pkg-promptkit.md) +and [request GoDoc](../../types.go). The backend default applies when neither +selector is supplied, not as a replacement for an invalid explicit selection. +How callers identify the backend in that case, and behavior when its default +smoke-test profile is absent, remain to be specified. + +Selecting an application prompt supplies execution-profile selection only. It +does not replace the synthetic smoke-test messages or import the application's +output contract. An explicit profile selection must not be silently replaced +with a cheaper model. Reports should identify the selection source and effective +execution settings, excluding credential values. + +## Private Test Content And Execution Options + +Promptkit owns fixed, private prompt definitions for plain-text and +structured-output smoke tests, including the latter's small schema. Their exact +language is an implementation detail. Callers cannot override the prompt +language, in whole or in part, or retrieve it through public prompt inspection, +configuration inspection, prepared snapshots, or diagnostic reports. These +definitions are not caller-configurable prompt catalog entries. + +Injected model clients necessarily receive the messages they execute. That +execution boundary does not make the smoke-test prompt a public inspection or +customization feature, nor does private implementation content imply secrecy +from the executing client or provider. + +Applications select criteria independently of the fixed test content. An +exact-text match need not be required merely to establish that inference +completed. Profile selection changes execution settings, not prompt language. +The supported per-call generation overrides and inheritance from the selected +profile remain to be specified, including retry, repair, reasoning, and token +limit defaults. Effective settings must be inspectable without revealing +credentials or private prompt messages. + +Promptkit reports timing without prescribing an acceptable latency threshold +or a universal diagnostic deadline. Caller cancellation and configured execution +timeouts still govern the operation. Applications decide whether the measured +timing is acceptable and whether to impose additional deadlines. + +Inference smoke tests make a small inference request and are likely to incur a +small, nonzero cost. Promptkit does not estimate prices or guarantee a cost +ceiling; prices and execution settings can change the actual cost. Report usage +when available. Non-inference probes should use endpoints documented for that +purpose; billing properties must not be inferred merely from an HTTP method. ## Probes, Criteria, And Results @@ -102,11 +142,11 @@ acceptance, successful generation, returned text, expected-text matching, JSON parsing, and schema conformance. Each needs an explicit definition and a clear set of applicable probes. -For example, a model returning `Success!` could satisfy successful generation -and returned-text criteria while failing an exact match against `success`. -The application could treat the mismatch as a warning or a failure. Promptkit -would implement the comparison rather than requiring every application to -reimplement it. Matching and normalization rules remain to be specified. +For example, a response could satisfy successful generation and returned-text +criteria while failing the built-in expected-text comparison. Promptkit reports +those findings without assigning an overall application-readiness verdict. The +application decides whether the mismatch warrants a warning, a failure, or no +action. Matching and normalization rules remain to be specified. Candidate criterion outcomes are passed, failed, unsupported, and inconclusive, with structured reasons. An unavailable authentication probe must not become an @@ -126,7 +166,9 @@ diagnostics need explicit treatment consistent with existing contracts. Reuse ordinary generation, decoding, validation, and error handling for inference tests wherever their behavior fits. Do not build a second inference -stack. Existing behavior is described by the +stack. Private built-in prompt definitions can supply synthetic content through +that machinery without exposing them through the public prompt catalog or +configuration view. Existing behavior is described by the [runner](../internal/runner.md), [generation coordinator](../internal/generation.md), and [model-client integration contract](../integrations/openai-compatible-chat.md). @@ -139,9 +181,11 @@ should not acquire new required methods merely to continue ordinary execution. 1. **Public operation shape and first delivery scope.** Choose individual methods, a typed single-probe operation, or a convenience batch operation; - define target selection for backend, profile, and endpoint-only cases. Select - initial provider adapters and decide whether model-status checks belong in - the first delivery. Go names and configuration keys are not yet proposed. + express the agreed profile-selection rules and define backend identification, + absent backend defaults, prompt-version selection, and endpoint-only cases. + Select initial provider adapters and decide whether model-status checks + belong in the first delivery. Go names and configuration keys are not yet + proposed. 2. **Probe configuration and routing.** Define supported probe kinds and their response semantics. Resolve endpoint overrides, relative paths, separate diagnostic hosts, redirects, and credential destinations deliberately so @@ -149,18 +193,21 @@ should not acquire new required methods merely to continue ordinary execution. 3. **Criteria and result contract.** Specify applicability, exact matching, unsupported and inconclusive outcomes, errors versus diagnostic findings, and what remains available after cancellation or partial completion. -4. **Execution policy.** Choose default test content, profile inheritance and - overrides, retry and repair defaults, capacity participation, observation, - and cancellation behavior. Decide whether custom test content is needed - initially; ordinary application-authored prompts remain available via `Run`. +4. **Execution mechanics.** Choose private test content, supported setting + overrides and inheritance, retry and repair defaults, capacity participation, + observation, and cancellation behavior within the agreed diagnostic boundary. + Custom smoke-test prompt language is excluded; application-authored tests + remain available via ordinary `Run`. ## Suggested Next Step -Review this proposal's scope and ownership first. Then write concrete request -and result examples for three cases: an HTTP response with rejected +Write concrete request and result examples under the agreed boundaries for +three cases: an HTTP response with rejected authentication, successful inference with an unexpected text response, and schema-valid structured output. Include a backend without an authentication -probe to expose capability and outcome handling. +probe to expose capability and outcome handling. Show prompt-based selection, +an explicit profile overriding a prompt default, and selection through a +backend's default smoke-test profile without disclosing private test language. Use those examples to settle the operation and result shape, then specify the smallest backend configuration needed by the initial adapters. Convert the @@ -173,7 +220,9 @@ clients; live smoke testing would be an explicit consumer operation. This proposal does not include Scriptorium commands, application readiness policy, continuous monitoring, automatic probes before ordinary runs, a -general-purpose diagnostic scripting language, or a guarantee about future -generation. It does not change `ValidateRequest` or ordinary execution. +general-purpose diagnostic scripting language, public or customizable smoke-test +prompt language, pricing estimates, application latency thresholds, or a +guarantee about future generation. It does not change `ValidateRequest` or +ordinary execution. Published API and format documentation should be updated when implementation lands, not to present this proposal as current behavior.