Record smoke test design decisions and profile precedence

docs/roadmap/smoke-tests.md

@@ -1,6 +1,7 @@
 # Smoke Test Capability Proposal
 
-Status: proposed for design review; implementation has not started.
+Status: core design decisions accepted; API details and delivery scope remain
+under review. Implementation has not started.
 
 ## Purpose
 
@@ -11,9 +12,10 @@ repeat similar prompts, schemas, provider requests, and result interpretation.
 Promptkit should standardize those mechanisms while applications decide which
 checks to request and how their outcomes affect application behavior.
 
-This proposal captures the direction discussed with the maintainer. The API,
-configuration shape, defaults, and delivery scope remain subject to review.
-It is not an implementation plan or a new public contract.
+This proposal records the agreed diagnostic boundary, private test content, and
+execution-profile selection rules. The concrete API, configuration shape,
+remaining execution defaults, and delivery scope remain subject to review.
+It is not an implementation plan or a description of current public behavior.
 
 ## Relationship To Existing Operations
 
@@ -21,8 +23,10 @@ It is not an implementation plan or a new public contract.
 checks whether a concrete request passes preparation without model generation.
 `Run` prepares and executes the actual application request. Smoke tests would
 instead exercise selected diagnostic operations using synthetic input or a
-provider-specific probe. They need not load the application's prompt or inputs,
-or call `ValidateRequest` on a future request.
+provider-specific probe. An application prompt may be selected to obtain its
+default profile, but its messages are not executed and its variables and inputs
+are not required. Smoke tests do not need to call `ValidateRequest` on a future
+request.
 
 A consumer could combine these operations into a preflight stage. Promptkit
 would provide smoke test evidence, not decide that the application is ready.
@@ -37,8 +41,8 @@ request may prefer to run it directly and handle its errors.
 
 | Owner | Responsibility |
 | --- | --- |
-| Backend configuration | Declares available diagnostic operations and endpoints, including authentication or model-status endpoints where supported, and configuration needed by supported probe implementations. |
-| Promptkit | Owns probe execution, default smoke test prompts and schemas, available criteria, their evaluators, and structured observations and outcomes. |
+| Backend configuration | Declares available diagnostic operations and endpoints, configuration needed by supported probe implementations, and the backend's default smoke-test execution profile. |
+| Promptkit | Owns probe execution, private fixed smoke-test prompts and schemas, available criteria, their evaluators, and neutral diagnostic reports. |
 | Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. |
 
 An endpoint address alone does not define a probe's meaning. A supported probe
@@ -68,30 +72,66 @@ A backend model-status endpoint could provide additional provider-reported
 evidence. It would not substitute for actually completing inference. Whether
 to include model-status probes in the first implementation remains open.
 
-Inference tests should select the effective profile or execution target the
-caller wants to test. Automatically substituting a cheap unrelated model is
-not part of this proposal. Backend-only operations should not require unrelated
-prompt data or a runnable model selection.
+Backend-only operations should not require unrelated prompt data or a runnable
+model selection.
 
-## Default Test Content And Execution Options
+## Inference Profile Selection
 
-Promptkit would maintain a short plain-text prompt asking for the word
-`success`, and a separate structured-output prompt with a simple schema, such
-as an object with a required boolean `success` property constrained to true.
-Exact wording and schema are design choices still to be selected.
+Callers can override the backend's default smoke-test execution profile through
+the following selections:
 
-The defaults provide convenient repeatable tests. Applications select which
-criteria to evaluate; an exact-text match need not be required merely to
-establish that inference completed. A request to respond without thinking is
-prompt content, not a substitute for explicit execution settings.
+| Caller supplies | Execution profile |
+| --- | --- |
+| Prompt ID without a profile override | The selected prompt definition's default profile. |
+| Profile ID without a prompt ID | The explicitly selected profile. |
+| Both prompt and profile IDs | The supplied profile overrides the prompt's default under ordinary Promptkit selection rules. |
+| Neither prompt nor profile ID | The selected backend's default smoke-test profile. |
 
-Callers should have options for relevant generation settings, timeouts, retries,
-and repair. The design must specify which settings inherit from the selected
-profile and which a smoke test overrides, and make the effective choices
-inspectable without exposing credentials. No retry, repair, reasoning, or token
-limit defaults are settled here. A short prompt alone does not guarantee low
-cost. Non-inference probes should use endpoints documented for that purpose;
-billing properties must not be inferred merely from an HTTP method.
+Reuse existing prompt/profile selection semantics, including exact selection
+and omitted-selector handling; see the [consumer guide](../consumers/pkg-promptkit.md)
+and [request GoDoc](../../types.go). The backend default applies when neither
+selector is supplied, not as a replacement for an invalid explicit selection.
+How callers identify the backend in that case, and behavior when its default
+smoke-test profile is absent, remain to be specified.
+
+Selecting an application prompt supplies execution-profile selection only. It
+does not replace the synthetic smoke-test messages or import the application's
+output contract. An explicit profile selection must not be silently replaced
+with a cheaper model. Reports should identify the selection source and effective
+execution settings, excluding credential values.
+
+## Private Test Content And Execution Options
+
+Promptkit owns fixed, private prompt definitions for plain-text and
+structured-output smoke tests, including the latter's small schema. Their exact
+language is an implementation detail. Callers cannot override the prompt
+language, in whole or in part, or retrieve it through public prompt inspection,
+configuration inspection, prepared snapshots, or diagnostic reports. These
+definitions are not caller-configurable prompt catalog entries.
+
+Injected model clients necessarily receive the messages they execute. That
+execution boundary does not make the smoke-test prompt a public inspection or
+customization feature, nor does private implementation content imply secrecy
+from the executing client or provider.
+
+Applications select criteria independently of the fixed test content. An
+exact-text match need not be required merely to establish that inference
+completed. Profile selection changes execution settings, not prompt language.
+The supported per-call generation overrides and inheritance from the selected
+profile remain to be specified, including retry, repair, reasoning, and token
+limit defaults. Effective settings must be inspectable without revealing
+credentials or private prompt messages.
+
+Promptkit reports timing without prescribing an acceptable latency threshold
+or a universal diagnostic deadline. Caller cancellation and configured execution
+timeouts still govern the operation. Applications decide whether the measured
+timing is acceptable and whether to impose additional deadlines.
+
+Inference smoke tests make a small inference request and are likely to incur a
+small, nonzero cost. Promptkit does not estimate prices or guarantee a cost
+ceiling; prices and execution settings can change the actual cost. Report usage
+when available. Non-inference probes should use endpoints documented for that
+purpose; billing properties must not be inferred merely from an HTTP method.
 
 ## Probes, Criteria, And Results
 
@@ -102,11 +142,11 @@ acceptance, successful generation, returned text, expected-text matching, JSON
 parsing, and schema conformance. Each needs an explicit definition and a clear
 set of applicable probes.
 
-For example, a model returning `Success!` could satisfy successful generation
-and returned-text criteria while failing an exact match against `success`.
-The application could treat the mismatch as a warning or a failure. Promptkit
-would implement the comparison rather than requiring every application to
-reimplement it. Matching and normalization rules remain to be specified.
+For example, a response could satisfy successful generation and returned-text
+criteria while failing the built-in expected-text comparison. Promptkit reports
+those findings without assigning an overall application-readiness verdict. The
+application decides whether the mismatch warrants a warning, a failure, or no
+action. Matching and normalization rules remain to be specified.
 
 Candidate criterion outcomes are passed, failed, unsupported, and inconclusive,
 with structured reasons. An unavailable authentication probe must not become an
@@ -126,7 +166,9 @@ diagnostics need explicit treatment consistent with existing contracts.
 
 Reuse ordinary generation, decoding, validation, and error handling for
 inference tests wherever their behavior fits. Do not build a second inference
-stack. Existing behavior is described by the
+stack. Private built-in prompt definitions can supply synthetic content through
+that machinery without exposing them through the public prompt catalog or
+configuration view. Existing behavior is described by the
 [runner](../internal/runner.md), [generation coordinator](../internal/generation.md),
 and [model-client integration contract](../integrations/openai-compatible-chat.md).
 
@@ -139,9 +181,11 @@ should not acquire new required methods merely to continue ordinary execution.
 
 1. **Public operation shape and first delivery scope.** Choose individual
    methods, a typed single-probe operation, or a convenience batch operation;
-   define target selection for backend, profile, and endpoint-only cases. Select
-   initial provider adapters and decide whether model-status checks belong in
-   the first delivery. Go names and configuration keys are not yet proposed.
+   express the agreed profile-selection rules and define backend identification,
+   absent backend defaults, prompt-version selection, and endpoint-only cases.
+   Select initial provider adapters and decide whether model-status checks
+   belong in the first delivery. Go names and configuration keys are not yet
+   proposed.
 2. **Probe configuration and routing.** Define supported probe kinds and their
    response semantics. Resolve endpoint overrides, relative paths, separate
    diagnostic hosts, redirects, and credential destinations deliberately so
@@ -149,18 +193,21 @@ should not acquire new required methods merely to continue ordinary execution.
 3. **Criteria and result contract.** Specify applicability, exact matching,
    unsupported and inconclusive outcomes, errors versus diagnostic findings,
    and what remains available after cancellation or partial completion.
-4. **Execution policy.** Choose default test content, profile inheritance and
-   overrides, retry and repair defaults, capacity participation, observation,
-   and cancellation behavior. Decide whether custom test content is needed
-   initially; ordinary application-authored prompts remain available via `Run`.
+4. **Execution mechanics.** Choose private test content, supported setting
+   overrides and inheritance, retry and repair defaults, capacity participation,
+   observation, and cancellation behavior within the agreed diagnostic boundary.
+   Custom smoke-test prompt language is excluded; application-authored tests
+   remain available via ordinary `Run`.
 
 ## Suggested Next Step
 
-Review this proposal's scope and ownership first. Then write concrete request
-and result examples for three cases: an HTTP response with rejected
+Write concrete request and result examples under the agreed boundaries for
+three cases: an HTTP response with rejected
 authentication, successful inference with an unexpected text response, and
 schema-valid structured output. Include a backend without an authentication
-probe to expose capability and outcome handling.
+probe to expose capability and outcome handling. Show prompt-based selection,
+an explicit profile overriding a prompt default, and selection through a
+backend's default smoke-test profile without disclosing private test language.
 
 Use those examples to settle the operation and result shape, then specify the
 smallest backend configuration needed by the initial adapters. Convert the
@@ -173,7 +220,9 @@ clients; live smoke testing would be an explicit consumer operation.
 
 This proposal does not include Scriptorium commands, application readiness
 policy, continuous monitoring, automatic probes before ordinary runs, a
-general-purpose diagnostic scripting language, or a guarantee about future
-generation. It does not change `ValidateRequest` or ordinary execution.
+general-purpose diagnostic scripting language, public or customizable smoke-test
+prompt language, pricing estimates, application latency thresholds, or a
+guarantee about future generation. It does not change `ValidateRequest` or
+ordinary execution.
 Published API and format documentation should be updated when implementation
 lands, not to present this proposal as current behavior.