diff --git a/docs/roadmap/smoke-tests.md b/docs/roadmap/smoke-tests.md index fa0295b..820787c 100644 --- a/docs/roadmap/smoke-tests.md +++ b/docs/roadmap/smoke-tests.md @@ -1,228 +1,510 @@ -# Smoke Test Capability Proposal +# Explicit Smoke Test Diagnostics -Status: core design decisions accepted; API details and delivery scope remain -under review. Implementation has not started. +Status: purpose and ownership boundary accepted; the concrete API and delivery +proposals below remain subject to review. Implementation has not started. -## Purpose +This roadmap replaces the earlier capability proposal and incorporates the +subsequent discussion. It defines the target feature, not a staged implementation +plan or current public API. Significant remaining choices appear under Open Decisions. -Provide reusable diagnostic operations for endpoint reachability, -authentication, plain-text inference, and structured-output inference. -Applications building setup, diagnostic, or preflight workflows would otherwise -repeat similar prompts, schemas, provider requests, and result interpretation. -Promptkit should standardize those mechanisms while applications decide which -checks to request and how their outcomes affect application behavior. +## Purpose And Ownership -This proposal records the agreed diagnostic boundary, private test content, and -execution-profile selection rules. The concrete API, configuration shape, -remaining execution defaults, and delivery scope remain subject to review. -It is not an implementation plan or a description of current public behavior. +Provide reusable, explicitly invoked diagnostics for endpoint reachability, +authentication, plain-text inference, and structured-output inference. Consumers +can build setup and diagnostic workflows without maintaining their own synthetic +prompts, schemas, and provider probes. -## Relationship To Existing Operations +PromptKit is opinionated about what each test checks and how it operates. It owns +fixed content, provider adapters, evaluation rules, and neutral factual reports. +Consumers select tests, targets, and supported execution settings; they interpret +the results and decide what action to take. There is no overall success, +application-readiness verdict, or configurable criteria framework. +For structured output, report whether structured output was requested, a response +was received, generated content was valid JSON, and that JSON matched the fixed +test schema. Four positive observations do not prove provider enforcement; no +field attempts to infer it. + +For authentication, distinguish credentials supplied with the dispatched request, +a reachable authentication endpoint, and credentials accepted by that operation. +A reachable endpoint can reject credentials, and a transport failure can leave +acceptance undetermined. Authentication is inapplicable on the maintainer's +credential-free RakestrawHome deployment; that is neither an error nor a failure. + +## Current Baseline + +At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation. [`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request) -checks whether a concrete request passes preparation without model generation. -`Run` prepares and executes the actual application request. Smoke tests would -instead exercise selected diagnostic operations using synthetic input or a -provider-specific probe. An application prompt may be selected to obtain its -default profile, but its messages are not executed and its variables and inputs -are not required. Smoke tests do not need to call `ValidateRequest` on a future -request. +checks a concrete application request without model generation. `Run` executes +that request and may structurally validate and repair its output. These operations +remain unchanged; smoke tests need not validate an intended application request. -A consumer could combine these operations into a preflight stage. Promptkit -would provide smoke test evidence, not decide that the application is ready. -Results describe the tested operation at the time of the check; they do not -reserve capacity or guarantee that a subsequent request will succeed. +The [runner](../internal/runner.md) shares captured prompt/profile selection and +execution-target resolution. The [generation coordinator](../internal/generation.md) +handles attempts, retries, usage, and active backend permits; the +[model client](../internal/llm.md) handles transport and decoding. Ordinary `Run` +requires an application prompt and returns no partial result on operational +failure, so calling it directly is not sufficient for the proposed diagnostic API. -The main value is shared diagnostics before real requests exist or when testing -a small synthetic request is useful. Consumers about to execute an ordinary -request may prefer to run it directly and handle its errors. +Backend configuration has no diagnostic endpoints, capabilities, or default test +profile. New fields must follow [ADR 0001](../adr/0001-configuration-boundary.md): +ConfigKit owns loading and composition; PromptKit interprets and validates final +effective values. Provenance stays available for explicit inspection and is not +an execution-time resolution or validation mechanism. -## Ownership +## Proposed Public Operation And Request -| Owner | Responsibility | +One synchronous call runs one independently selected operation: + +```go +func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error) + +type SmokeTestKind string + +const ( + SmokeTestReachability SmokeTestKind = "reachability" + SmokeTestAuthentication SmokeTestKind = "authentication" + SmokeTestTextInference SmokeTestKind = "text_inference" + SmokeTestStructuredOutput SmokeTestKind = "structured_output" +) + +type SmokeTestRequest struct { + Kind SmokeTestKind + BackendID string + PromptID string + PromptVersion string + ProfileID string + Execution *ExecutionTargetOverride + APIKey string `json:"-"` + TransientRetries *int +} +``` + +The request supplies no variables, inputs, appended messages, schema, or output +contract. An application prompt is a profile-selection aid only: its messages, +session template, input requirements, and output contract are not executed. +Consumers can compose several calls themselves; custom application tests remain +available through ordinary `Run`. + +Reuse existing `ExecutionTargetOverride` semantics. Inference uses its endpoint, +model, numeric, reasoning, timeout, credential-source, and provider-extra controls. +Backend-only probes use endpoint, credential-source, and timeout settings as +applicable; they do not require model completeness or execute generation merely +because a model setting was supplied. Profile settings remain the inference +baseline unless overridden; no silent cheaper-model substitution is proposed. + +Direct credentials take precedence over environment credentials and follow +existing request-scoped ownership and redacted-formatting conventions. Optional +missing/blank credentials are omitted from the upstream request. Authentication +can test the resulting unauthenticated request rather than reject missing keys +locally. If an explicitly required credential prevents inference dispatch, report +that local diagnostic finding without pretending the provider rejected it. +Ordinary execution's required-credential behavior remains unchanged. + +## Selection And Routing + +Preserve the accepted inference selection order: + +| Caller selection | Execution profile | | --- | --- | -| Backend configuration | Declares available diagnostic operations and endpoints, configuration needed by supported probe implementations, and the backend's default smoke-test execution profile. | -| Promptkit | Owns probe execution, private fixed smoke-test prompts and schemas, available criteria, their evaluators, and neutral diagnostic reports. | -| Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. | +| Prompt ID only | Selected prompt version's default profile | +| Profile ID only | Explicit profile | +| Both | Explicit profile overrides the prompt default | +| Neither | Explicit backend's configured default smoke-test profile | -An endpoint address alone does not define a probe's meaning. A supported probe -implementation must know the request method, authentication requirements, and -how to interpret the response. A candidate design is for configuration to -identify a supported probe kind plus its endpoint settings. A general request -templating and response-expression language is not proposed initially. +Reuse exact identity and omitted-version semantics. An invalid explicit selection +does not fall back to a backend default. Missing backend/default selection is an +error when inference needs it, not a reason to substitute an arbitrary profile. +Backend-only probes require neither a default profile nor unrelated prompt data. -Configuration changes must retain the existing -[ConfigKit ownership boundary](../adr/0001-configuration-boundary.md). -The [architecture policy](../policy/architecture.md) continues to govern the -library and downstream application boundary. +Proposed routing: when a prompt or profile is selected, the effective profile +supplies the backend identity and endpoint. Otherwise `BackendID` selects the +backend. If both are provided, profile selection wins, including over a different +backend hint; report the actual choice. This is proposed precedence, not an +accepted new prohibition on selector combinations; see Open Decisions. -## Candidate Smoke Tests +An endpoint-only profile remains usable for inference and reachability without +an inferred backend identity or provider-specific authentication adapter. +Endpoint overrides retain the effective backend's ordinary capacity identity. +Backend-only probes can resolve profile endpoint/credential metadata without +requiring its model. Prompt selection uses captured identity/default-profile +metadata without rendering or loading the application's artifacts or schemas. -These are independently selectable operations, not increasingly strict levels -that require every preceding test to pass. +## Proposed Report Shapes -| Test | Intended evidence | Important distinction | +Use factual Boolean observations with presence, rather than a caller-configurable +list of criteria. These Go-facing values need no stable JSON contract initially. + +```go +type SmokeTestReport struct { + Kind SmokeTestKind + Supported bool + Applicable *bool + Completed bool + Reason string + Target SmokeTestTarget + Facts SmokeTestFacts + Timing SmokeTestTiming + Attempts []SmokeTestAttempt + Usage ObservedTokenUsage + Failure ExecutionFailureKind +} + +type SmokeTestTarget struct { + BackendID, ProfileID, PromptID, PromptVersion string + SelectionSource string // "backend_default", "profile", "prompt", or "backend" + Endpoint string // effective diagnostic destination + EffectiveSettings ExecutionTarget + SettingsPresence ExecutionTargetPresence +} + +type SmokeTestFacts struct { + CredentialsSupplied *bool + EndpointReachable *bool + CredentialsAccepted *bool + ResponseReceived *bool + GenerationCompleted *bool + TextReturned *bool + ExpectedTextMatched *bool + StructuredOutputRequested *bool + ValidJSON *bool + SchemaMatched *bool +} + +type SmokeTestTiming struct { + StartedAt time.Time + Duration time.Duration + CapacityWait *time.Duration + RequestDuration *time.Duration + EvaluationDuration *time.Duration +} + +type SmokeTestAttempt struct { + Dispatched bool + Endpoint string + HTTPStatus *int + Facts SmokeTestFacts + Timing SmokeTestTiming + Failure ExecutionFailureKind + Usage ObservedTokenUsage +} +``` + +`Supported` records whether an implementation is available for the target and +selected client. `Applicable=false` records known inapplicability; nil leaves +applicability undetermined, for example when no authentication probe is configured. +Missing probe metadata does not imply that a provider accepts anonymous access. +`Reason` is a short factual code such as `authentication_not_applicable`, +`probe_not_configured`, or `http_diagnostics_unavailable`, not an application +severity or verdict. + +`Completed` means the diagnostic concluded, not that checks passed. A rejected, +failed, unsupported, or inapplicable diagnostic can conclude with nil Go error. +Caller cancellation before completion leaves it false and returns available facts. + +A fact is true or false only when established; nil means not evaluated, +undetermined, or inapplicable. Operation metadata and failure classification +provide context. In particular: + +- `CredentialsSupplied`: a nonempty credential was supplied with the dispatched + request, not merely named in configuration. No dispatch leaves this fact absent. + For injected clients, do not claim upstream transmission PromptKit cannot observe. +- `EndpointReachable`: an HTTP response was received from the tested destination, + regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation + ending without a response yields false. An injected generation result alone + does not establish HTTP reachability. +- `CredentialsAccepted`: the designated authentication adapter recognized acceptance + or rejection. A rejection yields false; transport failure, server error, or an + uninterpretable response can leave it nil. Acceptance does not certify balance, + model permissions, or future access. Absence of credentials is not an invalid-key + assertion. +- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a + response was returned by the injected generation boundary. A non-success or + malformed HTTP response still counts as received. +- `GenerationCompleted`: generation decoded/completed successfully, separately from + generated-content checks. Empty decoded content can complete generation. +- `TextReturned`: generated content is non-whitespace. `ExpectedTextMatched` compares + against the private expected text using trimming only, without case folding or + semantic matching. +- `StructuredOutputRequested`: the fixed native schema constraint was supplied to + the built-in provider request or injected generation boundary. This is not proof + of upstream transmission by an injected client or provider enforcement. +- `ValidJSON`: the generated content, not the HTTP envelope, parsed as JSON. + `SchemaMatched` evaluates parsed JSON against the fixed test schema and remains + absent when parsing or response acquisition prevents evaluation. + +Known inapplicable authentication performs no dispatch and leaves request and +acceptance facts absent. Unknown measurements remain unknown, including phase +measurements for unperformed work. Total duration includes waiting and evaluation; +request duration excludes capacity waiting. Timings impose no application latency +threshold or universal deadline. + +If retries are enabled, top-level facts describe the terminal attempt and attempt +records preserve earlier failures. Usage reuses presence-aware accounting without +claiming complete provider billing. No facts are inferred from arbitrary error text. +`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt +observations; it is not proof that packets were sent or billable work occurred. + +Reports omit private prompts, expected text, schemas, generated content, credentials, +raw bodies, and headers. Use content-free failure classifications and HTTP status, +not arbitrary provider messages, to preserve this exposure boundary. Effective +settings, provider-extra values, endpoints, and identities may still be sensitive +operator data; applications own their disclosure. Apply existing credential +exclusion and formatting rules rather than promise that all configuration is safe +for unrestricted logging. + +## Diagnostic Procedures And Capability Configuration + +### Reachability + +Propose one unauthenticated HTTP GET to the configured diagnostic endpoint, or +the effective base endpoint when no separate endpoint is supplied. Any HTTP +response establishes reachability, including 401, 404, or 405. It does not establish +that the completion route, credentials, or a model work. Authentication is tested +separately. Classify transport failure where available without promising exhaustive +DNS/TLS diagnosis. + +### Authentication + +Metadata names a supported provider adapter or explicitly marks authentication +inapplicable. The proposed first adapter is `openrouter_key`, using GET on the +selected base endpoint's `key` path with ordinary bearer credential resolution. +OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key) +documents this authenticated operation and its 200 and 401 responses. Recognize +the documented successful shape and unauthorized response; leave acceptance +unknown on other outcomes instead of guessing. Returned account/key details +are not exposed. + +Configure RakestrawHome authentication as `none`, pursuant to the maintainer's +stated deployment behavior. Other backends can declare supported adapters. +An arbitrary models-list or health endpoint is not credential-acceptance evidence. + +### Inference + +Plain-text inference uses a small fixed private prompt and expected answer. +Structured-output inference uses a fixed private prompt and self-contained schema, +requests native JSON Schema output through the existing client path, and evaluates +JSON and schema conformance locally. It imports no application output contract. + +Exact content is an implementation detail and does not enter public prompt listings, +inspection, configuration inspection, prepared snapshots, or reports. Injected +clients and providers necessarily receive the content they execute: privacy here +is an API-exposure boundary, not secrecy from executing collaborators. There is +no persistent diagnostic prompt or session state. + +### Proposed Backend Metadata + +Extend optional backend configuration through the final-value loader and the +`BackendConfig`/`BackendSource` convenience. Illustrative shape: + +```yaml +backends: + openrouter: + smoke_tests: + default_profile: promptkit-openrouter-standard + reachability: + endpoint: https://openrouter.ai/api/v1 + authentication: + kind: openrouter_key + rakestrawhome: + smoke_tests: + authentication: + kind: none +``` + +Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted +reachability metadata uses the effective base for the generic HTTP probe; +omitted authentication metadata makes that operation unsupported, with applicability +undetermined. Missing default-profile selection matters when inference selects it, +not for backend-only checks or an explicit profile. Retain normal effective +presence/null and layering semantics. Validate syntax and recognized configuration +at the final handoff, without contacting providers or validating intermediate layers. + +Provider paths follow the selected endpoint override, not silently the catalog's +original endpoint. Explicitly configured diagnostic destinations can be different; +report the destination actually tested. Exact endpoint options remain a proposal +under Open Decisions. + +Catalogs or caller overlays can supply backend-specific metadata. New catalog +fields need a coordinated compatible loader release: older strict PromptKit +loaders may reject them. A catalog default is replaceable metadata, not a promise +of upstream availability or a stable model choice. + +## Execution And Error Boundary + +Reuse target resolution, generation coordination, HTTP decoding, structural +validation, and usage accounting internally. A separate diagnostic orchestration +entry point may retain evidence that ordinary `Run` intentionally discards. +Do not duplicate inference serialization or build a second inference stack. + +Inference participates in existing engine-local admission and active backend +limits. HTTP-only probes need no generation permit or unrelated model selection. +Caller context and configured timeouts govern operations. The proposed inference +baseline is a single attempt, no structural repair, and otherwise the selected +profile plus explicit overrides. Retry choices below remain proposals; ordinary +engine defaults and repair behavior do not change. Small test content does not +establish small output, a price estimate, or a cost ceiling. + +Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without +new required methods. HTTP diagnostics must describe an explicitly selected +transport, not secretly probe a different transport in an injected-client engine. +The proposed initial behavior is to report HTTP diagnostics unavailable unless +that transport is supported or explicitly supplied; an additional injection option +is a delivery choice below. + +Proposed return conventions: + +- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections + return ordinary public errors before probe work. Unsupported capability is not + the same as invalid selection. +- Concluded diagnostics return reports with nil Go error, including credential + rejection, provider rejection, transport or configured request-timeout failure, + invalid JSON, schema mismatch, unsupported capability, or inapplicability. + Local capacity exhaustion is a diagnostic execution finding, not provider failure; + request facts remain absent if nothing was dispatched. +- Caller cancellation or caller-context deadline expiration returns accumulated + observations with an error recognizable through `errors.Is` against + `context.Canceled` or `context.DeadlineExceeded`. An already-done context can yield + a report with no observations. Do not dispatch again merely to fill missing facts. + As elsewhere, arbitrary synchronous injected code must return before cancellation + can be observed. + +The cancellation contract applies to this new operation, not to existing +preparation APIs. Internal faults preventing meaningful diagnostic execution may +still return an ordinary error; the implementation plan should identify those +boundaries without promising that every conceivable failure becomes a report. + +## Representative Requests And Results + +Request examples use ordinary exact selections, without private test language: + +```go +// Select the application's default profile, but do not run its prompt. +req := promptkit.SmokeTestRequest{ + Kind: promptkit.SmokeTestTextInference, + PromptID: "meeting.summary", +} +// An explicit profile overrides that default. +req.ProfileID = "promptkit-openrouter-standard" + +// Select a backend's configured default for structured inference. +structured := promptkit.SmokeTestRequest{ + Kind: promptkit.SmokeTestStructuredOutput, + BackendID: "openrouter", +} +``` + +Expected report examples: + +| Operation/outcome | Observations | Go error | | --- | --- | --- | -| HTTP/HTTPS reachability | An HTTP response was obtained from the selected destination. | A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available. | -| Authentication | A designated provider operation accepted the effective credentials. | Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract. | -| Plain-text inference | The selected model completed a small text-generation request. | Successful generation and compliance with an exact expected response are separate criteria. | -| Structured-output inference | The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria. | Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally. | +| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil | +| RakestrawHome authentication | Applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil | +| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil | +| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil | +| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil | +| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil | +| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error | -A backend model-status endpoint could provide additional provider-reported -evidence. It would not substitute for actually completing inference. Whether -to include model-status probes in the first implementation remains open. +Consumers decide whether any of these findings warrant success, a warning, or a +failure in their own application. They own CLI diagnostics, exit statuses, +scheduling, and any combination with `ValidateRequest` or normal execution. -Backend-only operations should not require unrelated prompt data or a runnable -model selection. +## Work Needed And Target Completion -## Inference Profile Selection +- Add the public request/report operation and GoDoc, keeping business logic internal. + Resolve significant Open Decisions before writing the staged implementation plan. +- Extend final backend metadata interpretation and captured lookup for capabilities + and defaults. Preserve explicit configuration inspection without exposing test + content or using source history for validation. +- Connect private inference content to shared selection, execution resolution, + capacity, generation, decoding, and structural evaluation without preparing + application inputs or messages. +- Add narrow HTTP probes, credential handling, evidence retention, factual results, + and partial cancellation reporting while preserving existing client contracts. +- Supply agreed built-in metadata through compatible catalog releases or PromptKit + underlays, with normal caller overlays and no provider-availability guarantee. +- Add focused offline tests with controlled transports and injected clients for + selection, rejected/absent credentials, applicability/support, transport/malformed + responses, content/JSON/schema facts, cancellation, shared inference capacity, + and credential/private-content exposure. Assign distinct test owners; do not + repeat existing transport, capacity, or validation matrices or snapshot private + test language. +- Update canonical consumer, backend-format, integration, and internal documentation + when implementation lands, with release notes explaining the optional addition. + Live diagnostics are explicit consumer operations, not default tests or new + release gates. -Callers can override the backend's default smoke-test execution profile through -the following selections: +Completion means consumers can select the four kinds, obtain supported factual +observations, distinguish false from unknown/inapplicable, and retain evidence +on cancellation. Existing preparation and execution continue unchanged. Initial +scope excludes application CLI/service work, readiness policy, continuous monitoring, +model-status tests, batch orchestration, customizable test language, cost estimates, +and guarantees about later generation. -| Caller supplies | Execution profile | -| --- | --- | -| Prompt ID without a profile override | The selected prompt definition's default profile. | -| Profile ID without a prompt ID | The explicitly selected profile. | -| Both prompt and profile IDs | The supplied profile overrides the prompt's default under ordinary Promptkit selection rules. | -| Neither prompt nor profile ID | The selected backend's default smoke-test profile. | +## Open Decisions -Reuse existing prompt/profile selection semantics, including exact selection -and omitted-selector handling; see the [consumer guide](../consumers/pkg-promptkit.md) -and [request GoDoc](../../types.go). The backend default applies when neither -selector is supplied, not as a replacement for an invalid explicit selection. -How callers identify the backend in that case, and behavior when its default -smoke-test profile is absent, remain to be specified. +### 1. Backend Selection Alongside A Profile -Selecting an application prompt supplies execution-profile selection only. It -does not replace the synthetic smoke-test messages or import the application's -output contract. An explicit profile selection must not be silently replaced -with a cheaper model. Reports should identify the selection source and effective -execution settings, excluding credential values. +**Recommendation:** adopt the proposed precedence: explicit or prompt-selected +profiles supply routing; `BackendID` supplies backend/default selection otherwise. +Report the effective choice. This follows familiar profile precedence without +inventing a mismatch prohibition or rebinding capacity identity. -## Private Test Content And Execution Options +**Alternative:** treat `BackendID` as a constraint when combined with profile/prompt +selection and reject mismatches. That catches mistakes and makes every selector +operative, but adds a diagnostic-specific restriction and makes profile changes +more disruptive. Choose it only if consumers want that constraint. -Promptkit owns fixed, private prompt definitions for plain-text and -structured-output smoke tests, including the latter's small schema. Their exact -language is an implementation detail. Callers cannot override the prompt -language, in whole or in part, or retrieve it through public prompt inspection, -configuration inspection, prepared snapshots, or diagnostic reports. These -definitions are not caller-configurable prompt catalog entries. +### 2. Initial HTTP Adapters, Endpoints, Redirects, And Injection -Injected model clients necessarily receive the messages they execute. That -execution boundary does not make the smoke-test prompt a public inspection or -customization feature, nor does private implementation content imply secrecy -from the executing client or provider. +**Recommendation:** start with generic unauthenticated HTTP-response reachability, +the OpenRouter current-key adapter, and explicit `none` authentication. Derive the +key route from the effective base, while allowing explicitly configured absolute +diagnostic endpoints for separate health/auth routes. Reuse the configured HTTP +client and its standard redirect handling, record the final observed destination, +and avoid custom redirect credential forwarding. A separate diagnostic host is +explicit configuration, not an inference from an overridden completion endpoint. +Keep configuration limited to named adapters and endpoints, not arbitrary methods +or response predicates. For injected generation clients, initially report HTTP +diagnostics unavailable unless a separate transport is explicitly supported. -Applications select criteria independently of the fixed test content. An -exact-text match need not be required merely to establish that inference -completed. Profile selection changes execution settings, not prompt language. -The supported per-call generation overrides and inheritance from the selected -profile remain to be specified, including retry, repair, reasoning, and token -limit defaults. Effective settings must be inspectable without revealing -credentials or private prompt messages. +This serves the maintained backends with a small adapter surface. Planning must +settle concrete endpoint fields and how request/credential observations reflect +redirects: a final response is not proof of acceptance at every redirect hop. -Promptkit reports timing without prescribing an acceptable latency threshold -or a universal diagnostic deadline. Caller cancellation and configured execution -timeouts still govern the operation. Applications decide whether the measured -timing is acceptable and whether to impose additional deadlines. +**Alternative:** initially support base-relative routes only, with no separate +host or additional transport option. That simplifies routing but excludes dedicated +diagnostic hosts. Another viable addition is an explicit diagnostic HTTP-client +option for injected-client engines; it improves custom-client coverage but adds a +supported injection surface. Neither choice requires expanding `LLMClient`. -Inference smoke tests make a small inference request and are likely to incur a -small, nonzero cost. Promptkit does not estimate prices or guarantee a cost -ceiling; prices and execution settings can change the actual cost. Report usage -when available. Non-inference probes should use endpoints documented for that -purpose; billing properties must not be inferred merely from an HTTP method. +### 3. Retry And Repair Controls -## Probes, Criteria, And Results +**Recommendation:** retain `TransientRetries`, with nil meaning zero for this +operation. Positive values reuse built-in inference status retries and preserve +attempt records; injected generation remains single-dispatch. Keep reachability +and authentication single-attempt and structural repair outside the initial test +procedure. This gives direct default evidence while preserving caller choice for +transient inference failures, without changing ordinary execution policy. -A probe performs an operation. Criteria evaluate observations from that -operation. Several criteria can use one response without repeating the request. -Candidate criteria include receiving an HTTP response, authentication -acceptance, successful generation, returned text, expected-text matching, JSON -parsing, and schema conformance. Each needs an explicit definition and a clear -set of applicable probes. +**Alternative:** omit the retry field and keep every diagnostic single-attempt. +That is the smallest API, but applications must repeat calls themselves. Inheriting +the engine retry budget is also viable: it standardizes defaults but makes cost +and recovery less obvious. Automatic structural repair would test recovery as +well as first-response conformance and warrants a separate explicit decision. -For example, a response could satisfy successful generation and returned-text -criteria while failing the built-in expected-text comparison. Promptkit reports -those findings without assigning an overall application-readiness verdict. The -application decides whether the mismatch warrants a warning, a failure, or no -action. Matching and normalization rules remain to be specified. +### 4. Built-In Metadata Delivery And Default Profiles -Candidate criterion outcomes are passed, failed, unsupported, and inconclusive, -with structured reasons. An unavailable authentication probe must not become an -invalid-key verdict. A timeout during authentication may prevent determining -credential acceptance even though the probe itself failed. Criteria that could -not be evaluated must not be confused with assertions evaluated as false. +**Recommendation:** coordinate compatible catalog releases containing capability +metadata and backend defaults. Use `promptkit-openrouter-standard` for OpenRouter; +have the RakestrawHome catalog maintainer select an explicit current profile for +its diagnostic default. Include OpenRouter's adapter and RakestrawHome's `none` +setting. Caller overlays remain available. These are replaceable defaults, not +new promises about model names, availability, or cheap inference. -A report should identify the tested operation and effective target, applicable -criteria and outcomes, and useful evidence such as HTTP status, elapsed time, -attempts, and reported usage. Unknown measurements remain unknown. Inference -retries or repair should not conceal initial failures behind a final success. -Exact fields, observation retention, and the relationship between reports and -returned Go errors remain open. Credential redaction and disclosure of provider -diagnostics need explicit treatment consistent with existing contracts. - -## Implementation Direction - -Reuse ordinary generation, decoding, validation, and error handling for -inference tests wherever their behavior fits. Do not build a second inference -stack. Private built-in prompt definitions can supply synthetic content through -that machinery without exposing them through the public prompt catalog or -configuration view. Existing behavior is described by the -[runner](../internal/runner.md), [generation coordinator](../internal/generation.md), -and [model-client integration contract](../integrations/openai-compatible-chat.md). - -Reachability and authentication probes need a separate, narrow provider-facing -capability. How injected clients participate must be decided without assuming -that every custom client uses the built-in HTTP transport. Existing clients -should not acquire new required methods merely to continue ordinary execution. - -## Decisions Needed Before Implementation - -1. **Public operation shape and first delivery scope.** Choose individual - methods, a typed single-probe operation, or a convenience batch operation; - express the agreed profile-selection rules and define backend identification, - absent backend defaults, prompt-version selection, and endpoint-only cases. - Select initial provider adapters and decide whether model-status checks - belong in the first delivery. Go names and configuration keys are not yet - proposed. -2. **Probe configuration and routing.** Define supported probe kinds and their - response semantics. Resolve endpoint overrides, relative paths, separate - diagnostic hosts, redirects, and credential destinations deliberately so - the report describes the destination actually tested. -3. **Criteria and result contract.** Specify applicability, exact matching, - unsupported and inconclusive outcomes, errors versus diagnostic findings, - and what remains available after cancellation or partial completion. -4. **Execution mechanics.** Choose private test content, supported setting - overrides and inheritance, retry and repair defaults, capacity participation, - observation, and cancellation behavior within the agreed diagnostic boundary. - Custom smoke-test prompt language is excluded; application-authored tests - remain available via ordinary `Run`. - -## Suggested Next Step - -Write concrete request and result examples under the agreed boundaries for -three cases: an HTTP response with rejected -authentication, successful inference with an unexpected text response, and -schema-valid structured output. Include a backend without an authentication -probe to expose capability and outcome handling. Show prompt-based selection, -an explicit profile overriding a prompt default, and selection through a -backend's default smoke-test profile without disclosing private test language. - -Use those examples to settle the operation and result shape, then specify the -smallest backend configuration needed by the initial adapters. Convert the -agreed design into a focused implementation plan with validation proportionate -to the selected contracts under the [testing policy](../policy/testing.md). -Default tests should remain offline using controlled transports and injected -clients; live smoke testing would be an explicit consumer operation. - -## Scope Boundaries - -This proposal does not include Scriptorium commands, application readiness -policy, continuous monitoring, automatic probes before ordinary runs, a -general-purpose diagnostic scripting language, public or customizable smoke-test -prompt language, pricing estimates, application latency thresholds, or a -guarantee about future generation. It does not change `ValidateRequest` or -ordinary execution. -Published API and format documentation should be updated when implementation -lands, not to present this proposal as current behavior. +**Alternative:** contribute metadata through PromptKit-owned underlays initially, +or require caller configuration. Underlays avoid waiting for catalog releases but +move provider configuration into core. Caller-only metadata avoids coupled releases +but loses much of the convenience motivating adoption. No new stable profile-name +promise is necessary for the RakestrawHome default.