From 52cd94fcd126ad6a08e6f2945ef7ea21f5c7b4b7 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 13 Oct 2026 14:42:24 +0000 Subject: [PATCH] Finalize smoke test roadmap decisions and evidence contracts --- docs/roadmap/smoke-tests.md | 279 ++++++++++++++++++++++-------------- 1 file changed, 168 insertions(+), 111 deletions(-) diff --git a/docs/roadmap/smoke-tests.md b/docs/roadmap/smoke-tests.md index 820787c..996e703 100644 --- a/docs/roadmap/smoke-tests.md +++ b/docs/roadmap/smoke-tests.md @@ -1,11 +1,11 @@ # Explicit Smoke Test Diagnostics -Status: purpose and ownership boundary accepted; the concrete API and delivery -proposals below remain subject to review. Implementation has not started. +Status: feature design and delivery direction accepted; remaining implementation +details are listed below. Implementation has not started. This roadmap replaces the earlier capability proposal and incorporates the subsequent discussion. It defines the target feature, not a staged implementation -plan or current public API. Significant remaining choices appear under Open Decisions. +plan or current public API. Remaining questions appear under Remaining Decisions. ## Purpose And Ownership @@ -52,7 +52,7 @@ ConfigKit owns loading and composition; PromptKit interprets and validates final effective values. Provenance stays available for explicit inspection and is not an execution-time resolution or validation mechanism. -## Proposed Public Operation And Request +## Public Operation And Request One synchronous call runs one independently selected operation: @@ -117,11 +117,23 @@ does not fall back to a backend default. Missing backend/default selection is an error when inference needs it, not a reason to substitute an arbitrary profile. Backend-only probes require neither a default profile nor unrelated prompt data. -Proposed routing: when a prompt or profile is selected, the effective profile -supplies the backend identity and endpoint. Otherwise `BackendID` selects the -backend. If both are provided, profile selection wins, including over a different -backend hint; report the actual choice. This is proposed precedence, not an -accepted new prohibition on selector combinations; see Open Decisions. +Whenever inference selects a profile, that profile supplies the resolved endpoint, +credential settings, and backend/capacity identity, subject to ordinary execution +overrides. This applies equally to explicit profiles, prompt defaults, and +backend-default profiles. A supplied `BackendID` is a selection fallback, not a +constraint or a request to rebind the selected profile's backend identity. + +For example, if backend A names a default profile that resolves to backend B, +inference uses B's resolved target and capacity identity. If that default profile +is endpoint-only, inference retains an empty backend identity. Report A as the +default-selection source and B (or the empty identity) as the effective backend. +No recursive backend-default lookup or mismatch rejection is introduced. + +For backend-only probes, an explicit or prompt-selected profile likewise supplies +the target and its backend's diagnostic metadata. Otherwise `BackendID` selects +the backend directly, without consulting its default inference profile. The +profile wins when both selectors are supplied, even when they name different +backends. An endpoint-only profile remains usable for inference and reachability without an inferred backend identity or provider-specific authentication adapter. @@ -130,7 +142,7 @@ Backend-only probes can resolve profile endpoint/credential metadata without requiring its model. Prompt selection uses captured identity/default-profile metadata without rendering or loading the application's artifacts or schemas. -## Proposed Report Shapes +## Report Shapes Use factual Boolean observations with presence, rather than a caller-configurable list of criteria. These Go-facing values need no stable JSON contract initially. @@ -141,7 +153,7 @@ type SmokeTestReport struct { Supported bool Applicable *bool Completed bool - Reason string + Reason SmokeTestReason Target SmokeTestTarget Facts SmokeTestFacts Timing SmokeTestTiming @@ -152,8 +164,9 @@ type SmokeTestReport struct { type SmokeTestTarget struct { BackendID, ProfileID, PromptID, PromptVersion string + DefaultSourceBackendID string // backend whose default selected the profile, if any SelectionSource string // "backend_default", "profile", "prompt", or "backend" - Endpoint string // effective diagnostic destination + Endpoint string // initially resolved diagnostic destination EffectiveSettings ExecutionTarget SettingsPresence ExecutionTargetPresence } @@ -176,39 +189,60 @@ type SmokeTestTiming struct { Duration time.Duration CapacityWait *time.Duration RequestDuration *time.Duration + RetryDelay *time.Duration EvaluationDuration *time.Duration } type SmokeTestAttempt struct { Dispatched bool - Endpoint string + InitialEndpoint string // resolved destination before redirects + Endpoint string // last observed request destination; facts refer to this destination HTTPStatus *int Facts SmokeTestFacts Timing SmokeTestTiming Failure ExecutionFailureKind Usage ObservedTokenUsage } + +type SmokeTestReason string + +const ( + SmokeTestReasonNone SmokeTestReason = "" + SmokeTestReasonAuthenticationNotApplicable SmokeTestReason = "authentication_not_applicable" + SmokeTestReasonProbeNotConfigured SmokeTestReason = "probe_not_configured" + SmokeTestReasonHTTPDiagnosticsUnavailable SmokeTestReason = "http_diagnostics_unavailable" + SmokeTestReasonRequiredCredentialMissing SmokeTestReason = "required_credential_missing" + SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized" +) ``` `Supported` records whether an implementation is available for the target and selected client. `Applicable=false` records known inapplicability; nil leaves applicability undetermined, for example when no authentication probe is configured. Missing probe metadata does not imply that a provider accepts anonymous access. -`Reason` is a short factual code such as `authentication_not_applicable`, -`probe_not_configured`, or `http_diagnostics_unavailable`, not an application -severity or verdict. +`Reason` is a named factual code, not application severity or free-form diagnostic +text. The constants distinguish, respectively, no additional reason, explicitly +inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics +for the selected client, a required credential missing before inference dispatch, +and an authentication response whose documented meaning cannot be established. +HTTP status and `Failure` classify ordinary rejection and transport outcomes; +they do not need duplicate reason codes. `Completed` means the diagnostic concluded, not that checks passed. A rejected, failed, unsupported, or inapplicable diagnostic can conclude with nil Go error. Caller cancellation before completion leaves it false and returns available facts. +**A nil Go error means the diagnostic concluded, not that the tested operation +succeeded.** Make this distinction prominent in GoDoc and consumer examples. A fact is true or false only when established; nil means not evaluated, undetermined, or inapplicable. Operation metadata and failure classification provide context. In particular: -- `CredentialsSupplied`: a nonempty credential was supplied with the dispatched - request, not merely named in configuration. No dispatch leaves this fact absent. - For injected clients, do not claim upstream transmission PromptKit cannot observe. +- `CredentialsSupplied`: a nonempty credential was attached to the observed + dispatched request for the attempt's `Endpoint`, not merely named in configuration + or attached to an earlier redirect hop. No observed request leaves this fact + absent. For injected clients, do not claim upstream transmission PromptKit cannot + observe. Observation at the transport boundary is not proof of wire transmission. - `EndpointReachable`: an HTTP response was received from the tested destination, regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation ending without a response yields false. An injected generation result alone @@ -217,7 +251,8 @@ provide context. In particular: or rejection. A rejection yields false; transport failure, server error, or an uninterpretable response can leave it nil. Acceptance does not certify balance, model permissions, or future access. Absence of credentials is not an invalid-key - assertion. + assertion: a documented 401 with `CredentialsSupplied=false` records rejection + of an unauthenticated request, not rejection of a supplied key. - `ResponseReceived`: an HTTP response was obtained for built-in operations, or a response was returned by the injected generation boundary. A non-success or malformed HTTP response still counts as received. @@ -235,13 +270,24 @@ provide context. In particular: Known inapplicable authentication performs no dispatch and leaves request and acceptance facts absent. Unknown measurements remain unknown, including phase -measurements for unperformed work. Total duration includes waiting and evaluation; -request duration excludes capacity waiting. Timings impose no application latency -threshold or universal deadline. +measurements for unperformed work. Report-level capacity wait, request duration, +evaluation duration, and retry delay sum the corresponding measured work across +the diagnostic. Request duration excludes capacity waiting, evaluation, and retry +backoff. Retry delay includes completed and interrupted backoff. Per-attempt +timings retain their own measurements; backoff is reported at the operation level. +Total duration is wall time, including selection and other orchestration overhead, +so it need not equal the phase sum. Timings impose no application latency threshold +or universal deadline. -If retries are enabled, top-level facts describe the terminal attempt and attempt -records preserve earlier failures. Usage reuses presence-aware accounting without -claiming complete provider billing. No facts are inferred from arbitrary error text. +If retries are enabled, top-level facts describe the most recent dispatched +attempt. Cancellation during backoff or a later capacity wait does not replace +those facts with an undispatched attempt's empty facts. The operation's `Failure` +and `Completed` still describe the interruption; earlier attempt records retain +their own outcomes. A canceled capacity acquisition can add a `Dispatched=false` +record, but no synthetic attempt is added for cancellation during backoff. With +no dispatched attempt, request facts remain absent. Usage reuses presence-aware +accounting without claiming complete provider billing. No facts are inferred from +arbitrary error text. `Dispatched` means the HTTP/client boundary was invoked, as in existing attempt observations; it is not proof that packets were sent or billable work occurred. @@ -257,7 +303,7 @@ for unrestricted logging. ### Reachability -Propose one unauthenticated HTTP GET to the configured diagnostic endpoint, or +Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or the effective base endpoint when no separate endpoint is supplied. Any HTTP response establishes reachability, including 401, 404, or 405. It does not establish that the completion route, credentials, or a model work. Authentication is tested @@ -267,7 +313,7 @@ DNS/TLS diagnosis. ### Authentication Metadata names a supported provider adapter or explicitly marks authentication -inapplicable. The proposed first adapter is `openrouter_key`, using GET on the +inapplicable. The first adapter is `openrouter_key`, using GET on the selected base endpoint's `key` path with ordinary bearer credential resolution. OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key) documents this authenticated operation and its 200 and 401 responses. Recognize @@ -292,7 +338,7 @@ clients and providers necessarily receive the content they execute: privacy here is an API-exposure boundary, not secrecy from executing collaborators. There is no persistent diagnostic prompt or session state. -### Proposed Backend Metadata +### Backend Metadata Extend optional backend configuration through the final-value loader and the `BackendConfig`/`BackendSource` convenience. Illustrative shape: @@ -321,14 +367,39 @@ presence/null and layering semantics. Validate syntax and recognized configurati at the final handoff, without contacting providers or validating intermediate layers. Provider paths follow the selected endpoint override, not silently the catalog's -original endpoint. Explicitly configured diagnostic destinations can be different; -report the destination actually tested. Exact endpoint options remain a proposal -under Open Decisions. +original endpoint. Explicit absolute diagnostic endpoints are supported for separate +health/auth routes and take precedence for their operation. Configuration stays +limited to named adapters and endpoint settings, not arbitrary methods or response +predicates. Concrete field names and presence/null rules remain to be specified. -Catalogs or caller overlays can supply backend-specific metadata. New catalog -fields need a coordinated compatible loader release: older strict PromptKit -loaders may reject them. A catalog default is replaceable metadata, not a promise -of upstream availability or a stable model choice. +Reuse the configured HTTP client and its redirect handling, without adding custom +credential forwarding. `Target.Endpoint` records the resolved starting destination. +Each HTTP attempt records `InitialEndpoint` and the last observed dispatched +request destination as `Endpoint`; its credential, reachability, response, and +acceptance facts describe that destination. If a redirect leads to a failed request, +do not attribute an earlier hop's response or credentials to the later destination. +When a client prevents observing the relevant request, leave the affected evidence +unknown. Injected generation alone supplies no HTTP destination evidence. + +Authentication acceptance is interpreted from the response for the observed +destination under the selected adapter's contract. It does not assert acceptance +at the initial destination or at every redirect hop. These endpoint fields provide +attribution without introducing a public redirect trace. + +Deliver built-in metadata through coordinated compatible catalog releases; caller +overlays remain available. First release loader support while retaining compatible +existing catalog dependencies, then release catalogs containing the new metadata, +then adopt those versions in PromptKit. Older strict loaders may reject new fields, +so do not direct their consumers to upgrade catalogs independently. Select actual +versions and compatibility notes during release preparation. + +Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key` +adapter, and RakestrawHome's `none` authentication setting. The RakestrawHome catalog +maintainer must select its current default profile. Verify each shipped default +resolves in the supported assembled effective configuration, including any +PromptKit-owned profiles. This is final-value integration validation, not a second +validation pass over intermediate sources. A catalog default is replaceable metadata, +not a promise of upstream availability, cheap inference, or a stable model choice. ## Execution And Error Boundary @@ -337,22 +408,34 @@ validation, and usage accounting internally. A separate diagnostic orchestration entry point may retain evidence that ordinary `Run` intentionally discards. Do not duplicate inference serialization or build a second inference stack. +Add a narrow internal evidence path at the transport/coordinator boundary. Existing +execution events do not carry redirect destinations or dispatched credential +presence, and the coordinator can discard a completed response when cancellation +is observed before return. Capture available facts before that discard and retain +them for diagnostics. Keep ordinary execution's error/result contracts unchanged; +do not expand public execution events just to reconstruct smoke-test evidence. + Inference participates in existing engine-local admission and active backend limits. HTTP-only probes need no generation permit or unrelated model selection. -Caller context and configured timeouts govern operations. The proposed inference -baseline is a single attempt, no structural repair, and otherwise the selected -profile plus explicit overrides. Retry choices below remain proposals; ordinary +Caller context and configured timeouts govern operations. Inference defaults to +one attempt: nil `TransientRetries` means zero for this operation rather than +inheritance from the engine retry budget. Positive values reuse existing built-in +inference status retries and retain attempt records; injected generation remains +single-dispatch. Reachability and authentication remain single-attempt regardless +of a valid inference retry setting. Structural repair is excluded from the initial +procedure. Otherwise use the selected profile plus explicit overrides. Ordinary engine defaults and repair behavior do not change. Small test content does not establish small output, a price estimate, or a cost ceiling. Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without new required methods. HTTP diagnostics must describe an explicitly selected transport, not secretly probe a different transport in an injected-client engine. -The proposed initial behavior is to report HTTP diagnostics unavailable unless +The initial behavior is to report HTTP diagnostics unavailable unless that transport is supported or explicitly supplied; an additional injection option -is a delivery choice below. +remains a delivery detail below. Do not silently fall back to a default HTTP client +to diagnose an injected generation client. -Proposed return conventions: +Return conventions: - Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections return ordinary public errors before probe work. Unsupported capability is not @@ -399,11 +482,20 @@ Expected report examples: | Operation/outcome | Observations | Go error | | --- | --- | --- | | Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil | -| RakestrawHome authentication | Applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil | +| RakestrawHome authentication | Supported true, applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil | | Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil | +| HTTP diagnostics unavailable for injected client | Supported false, applicability unknown, completed, `http_diagnostics_unavailable`; no dispatch | nil | +| Authentication without credentials receives documented 401 | Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request | nil | +| Required credential absent before inference | Supported/applicable/completed; `required_credential_missing`, failure `invalid_request`; no dispatch and no request facts | nil | +| Authentication receives uninterpretable HTTP 200 | Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; `authentication_response_unrecognized`, failure `malformed_response` | nil | | Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil | | Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil | | Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil | +| Configured request timeout while caller context remains active | Supported/applicable/completed; failure `deadline`; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts | nil | +| Local capacity exhausted | Supported/applicable/completed; failure `capacity`; no dispatch or request facts | nil | +| Caller cancels during initial capacity wait | Supported/applicable, completed false; failure `canceled`, measured capacity wait, undispatched attempt; request facts absent | `context.Canceled` identity | +| Caller deadline expires during a dispatched request | Supported/applicable, completed false; failure `deadline`; retain available attempt evidence | `context.DeadlineExceeded` identity | +| Caller cancels during retry backoff after HTTP 503 | Completed false, failure `canceled`; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt | `context.Canceled` identity | | Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error | Consumers decide whether any of these findings warrant success, a warning, or a @@ -413,7 +505,7 @@ scheduling, and any combination with `ValidateRequest` or normal execution. ## Work Needed And Target Completion - Add the public request/report operation and GoDoc, keeping business logic internal. - Resolve significant Open Decisions before writing the staged implementation plan. + Resolve the remaining delivery details when writing the staged implementation plan. - Extend final backend metadata interpretation and captured lookup for capabilities and defaults. Preserve explicit configuration inspection without exposing test content or using source history for validation. @@ -421,15 +513,21 @@ scheduling, and any combination with `ValidateRequest` or normal execution. capacity, generation, decoding, and structural evaluation without preparing application inputs or messages. - Add narrow HTTP probes, credential handling, evidence retention, factual results, - and partial cancellation reporting while preserving existing client contracts. -- Supply agreed built-in metadata through compatible catalog releases or PromptKit - underlays, with normal caller overlays and no provider-availability guarantee. + and partial cancellation reporting through the internal evidence path while + preserving existing client contracts. Specify destination/credential attribution + without exposing headers or a redirect trace. +- Supply agreed built-in metadata through compatible catalog releases in the + documented order, with normal caller overlays and no provider-availability + guarantee. Check default-profile resolution in the assembled final configuration. - Add focused offline tests with controlled transports and injected clients for selection, rejected/absent credentials, applicability/support, transport/malformed responses, content/JSON/schema facts, cancellation, shared inference capacity, and credential/private-content exposure. Assign distinct test owners; do not repeat existing transport, capacity, or validation matrices or snapshot private test language. +- Include the report-state matrix, cross-backend and endpoint-only default profiles, + redirect attribution, cancellation after a response, and interrupted retry waits + in focused contract coverage. Reuse existing lower-level checks where sufficient. - Update canonical consumer, backend-format, integration, and internal documentation when implementation lands, with release notes explaining the optional addition. Live diagnostics are explicit consumer operations, not default tests or new @@ -442,69 +540,28 @@ scope excludes application CLI/service work, readiness policy, continuous monito model-status tests, batch orchestration, customizable test language, cost estimates, and guarantees about later generation. -## Open Decisions +## Remaining Decisions And Delivery Dependencies -### 1. Backend Selection Alongside A Profile +The operation, routing precedence, report/error boundary, initial adapters, +retry/repair policy, and catalog delivery direction are settled. The remaining +items should be resolved in the focused implementation plan: -**Recommendation:** adopt the proposed precedence: explicit or prompt-selected -profiles supply routing; `BackendID` supplies backend/default selection otherwise. -Report the effective choice. This follows familiar profile precedence without -inventing a mismatch prohibition or rebinding capacity identity. - -**Alternative:** treat `BackendID` as a constraint when combined with profile/prompt -selection and reject mismatches. That catches mistakes and makes every selector -operative, but adds a diagnostic-specific restriction and makes profile changes -more disruptive. Choose it only if consumers want that constraint. - -### 2. Initial HTTP Adapters, Endpoints, Redirects, And Injection - -**Recommendation:** start with generic unauthenticated HTTP-response reachability, -the OpenRouter current-key adapter, and explicit `none` authentication. Derive the -key route from the effective base, while allowing explicitly configured absolute -diagnostic endpoints for separate health/auth routes. Reuse the configured HTTP -client and its standard redirect handling, record the final observed destination, -and avoid custom redirect credential forwarding. A separate diagnostic host is -explicit configuration, not an inference from an overridden completion endpoint. -Keep configuration limited to named adapters and endpoints, not arbitrary methods -or response predicates. For injected generation clients, initially report HTTP -diagnostics unavailable unless a separate transport is explicitly supported. - -This serves the maintained backends with a small adapter surface. Planning must -settle concrete endpoint fields and how request/credential observations reflect -redirects: a final response is not proof of acceptance at every redirect hop. - -**Alternative:** initially support base-relative routes only, with no separate -host or additional transport option. That simplifies routing but excludes dedicated -diagnostic hosts. Another viable addition is an explicit diagnostic HTTP-client -option for injected-client engines; it improves custom-client coverage but adds a -supported injection surface. Neither choice requires expanding `LLMClient`. - -### 3. Retry And Repair Controls - -**Recommendation:** retain `TransientRetries`, with nil meaning zero for this -operation. Positive values reuse built-in inference status retries and preserve -attempt records; injected generation remains single-dispatch. Keep reachability -and authentication single-attempt and structural repair outside the initial test -procedure. This gives direct default evidence while preserving caller choice for -transient inference failures, without changing ordinary execution policy. - -**Alternative:** omit the retry field and keep every diagnostic single-attempt. -That is the smallest API, but applications must repeat calls themselves. Inheriting -the engine retry budget is also viable: it standardizes defaults but makes cost -and recovery less obvious. Automatic structural repair would test recovery as -well as first-response conformance and warrants a separate explicit decision. - -### 4. Built-In Metadata Delivery And Default Profiles - -**Recommendation:** coordinate compatible catalog releases containing capability -metadata and backend defaults. Use `promptkit-openrouter-standard` for OpenRouter; -have the RakestrawHome catalog maintainer select an explicit current profile for -its diagnostic default. Include OpenRouter's adapter and RakestrawHome's `none` -setting. Caller overlays remain available. These are replaceable defaults, not -new promises about model names, availability, or cheap inference. - -**Alternative:** contribute metadata through PromptKit-owned underlays initially, -or require caller configuration. Underlays avoid waiting for catalog releases but -move provider configuration into core. Caller-only metadata avoids coupled releases -but loses much of the convenience motivating adoption. No new stable profile-name -promise is necessary for the RakestrawHome default. +1. **Configuration representation.** Finalize the Go convenience types and exact + endpoint fields, including omission/null/overlay behavior and interpretation of + configured default-profile references at construction versus selection. Preserve + final-value ownership and verify shipped defaults in assembled configuration; + do not introduce source-history validation or change ordinary profile selection. +2. **Explicit diagnostic transport for injected clients.** Decide whether the first + delivery exposes a separate diagnostic HTTP-client option or reports HTTP + diagnostics unavailable for injected-client engines until such an option exists. + Inference remains available through `LLMClient` in either case. No hidden + transport fallback or new required client methods are permitted. +3. **Catalog coordination.** Obtain the RakestrawHome maintainer's default-profile + choice and select compatible loader/catalog versions in the documented release + order. These are delivery dependencies, not reasons to reopen routing policy. +4. **Implementation details.** Choose private content and schema, the narrow evidence + handoff, and content-free classifications for internal faults that prevent a + meaningful report. Define ownership and copying for reports and request values. + Verify the state matrix, timeout/cancellation distinction, and redirect evidence + against actual transport behavior before finalizing GoDoc. Keep public execution + events and ordinary result/error semantics unchanged.