From 8fbd058b905e79a964a3e8799c7870b8282dc2c7 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 13 Oct 2026 15:21:58 +0000 Subject: [PATCH] Plan smoke test diagnostics implementation --- docs/roadmap/implementation.md | 644 +++++++++++++++++++++++++++++++++ docs/roadmap/smoke-tests.md | 103 ++---- 2 files changed, 679 insertions(+), 68 deletions(-) create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..b2459f3 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,644 @@ +# Smoke Test Diagnostics Implementation Plan + +Status: ready for implementation. + +Implement the accepted [feature roadmap](smoke-tests.md). This document owns the +implementation sequence; the feature roadmap owns the requested API, factual +reporting semantics, selection rules, and scope. Each numbered stage is a single +bounded prompt for a Luna coding agent. Execute stages in order within one sprint. +There are no intermediate human-review, external-publication, or live-provider +gates. Focused automated validation and local commits are ordinary development +steps, not pauses awaiting approval. + +## Baseline And Working Rules + +The reviewed PromptKit baseline is `c8f9a4def9f84301bf534a331320ea9fb798ac24`. +It selects both `/v3` catalogs at `v3.20260930.0` and ConfigKit at `v0.7.1`. +The adjacent checkouts are: + +- `../promptkit-backend-openrouter`, reviewed at + `f17d208cb7ad1fb043cda2266cd3011d11b40aa0`; +- `../promptkit-backend-rakestrawhome`, reviewed at + `993443d88c451901f4c917fa00c204fd3b54d28d`. + +Recheck repository status before editing and preserve unrelated work. Begin with +[development orientation](../development.md), then the applicable +[agent](../policy/agents.md), [architecture](../policy/architecture.md), +[testing](../policy/testing.md), [documentation](../policy/documentation.md), +[OSS](../policy/oss.md), and [release](../policy/release.md) policies. Read each +catalog's own instructions and release procedure before its stages. Reuse +unchanged policy readings during the sprint. + +Read focused internal documents and owning tests before modifying a component. +Use existing code-discovery guidance. Stage and commit each completed, nonempty +unit with a short plain-English message after proportionate checks and staged-diff +review. Record stage completion and relevant validation here. Catalog stages +commit in their own repositories; PromptKit records their tag and full commit +identities here. Do not push any repository or publish a PromptKit release during +this implementation sprint. + +Only final effective ConfigKit values are interpreted or validated by PromptKit, +per [ADR 0001](../adr/0001-configuration-boundary.md). Do not inspect raw catalog +documents or source history to enforce domain validity, even in PromptKit tests. +The catalog repositories may test the assets they themselves own. Operator +inspection continues to delegate to ConfigKit. + +Keep existing `Run`, `Prepare`, `ValidateRequest`, repair, observation, and client +contracts unchanged. Add no application CLI, readiness verdict, custom test +criteria, batch operation, live release gate, or new public transport injection +option. Do not add catalog exclusion controls absent from the current API. + +## Implementation Decisions + +These decisions resolve the feature roadmap's delivery details. Internal names +below identify a concrete implementation path, not additional public contracts. + +### Configuration And Catalog Assets + +Add optional `smoke_tests` to final backend definitions: + +```yaml +backends: + openrouter: + smoke_tests: + default_profile: promptkit-openrouter-standard + authentication: + kind: openrouter_key + rakestrawhome: + smoke_tests: + default_profile: rakestrawhome-gemma-4-31b + authentication: + kind: none +``` + +Both catalogs omit a separate reachability endpoint so ordinary endpoint overrides +remain effective. Caller definitions may set `reachability.endpoint` or +`authentication.endpoint` to an absolute diagnostic destination. Authentication +otherwise joins `key` to the effective base; reachability otherwise uses that base. + +The public convenience surface adds `SmokeTests *BackendSmokeTestConfig` to +`BackendConfig`, with: + +- `BackendSmokeTestConfig`: `DefaultProfile *string`, + `Reachability *ReachabilityProbeConfig`, + `Authentication *AuthenticationProbeConfig`; +- `ReachabilityProbeConfig`: `Endpoint *string`; +- `AuthenticationProbeConfig`: `Kind *string`, `Endpoint *string`. + +`BackendSource` omits nil convenience fields, as it does today. Explicit nulls +remain expressible with ordinary ConfigKit mappings/YAML. In final values, +omitted/null `smoke_tests` disables its defaults and authentication declaration; +omitted/null `default_profile` means no inference default; omitted/null +`reachability` or its endpoint means generic reachability at the effective base; +omitted/null `authentication` means unsupported authentication; a non-null +authentication object requires `kind`. Recognize `none` and `openrouter_key`. +Normalize surrounding whitespace/case for adapter kinds and record normalization +through the existing ConfigKit resolution boundary. Preserve profile IDs exactly; +a supplied default-profile string must not be blank. Resolve its existence only +when that default is selected, not eagerly at engine construction. + +Diagnostic endpoint strings are trimmed and parsed as absolute HTTP(S) URLs with +a host. Use standard URL/request validation, allowing paths and queries; do not +reuse the base-endpoint validator's query restriction for these full URLs. Null +means fallback, whereas an explicit blank endpoint is invalid. Retain existing +base-endpoint rules for ordinary target overrides. Reachability sends no +Authorization header or URL user credentials. Public endpoint fields omit URL +user information; other URL components and provider settings remain potentially +sensitive operator data, as specified in the feature roadmap. + +Preserve the existing `/v3` catalog asset format: do not place new fields under +the existing `Root` where older strict loaders would reject them. In each module, +add `smoke-tests/backend.yml` and the additive exported constant +`SmokeTestsRoot = "smoke-tests"`; extend `FS()` embedding to include that root. +Keep `Root = "catalog"` and its existing assets unchanged. New PromptKit explicitly +selects both roots through `internal/catalog`; diagnostic data remains +catalog-owned. This is a small opt-in asset surface, not another runtime library +or a PromptKit-owned metadata underlay. + +### Private Fixtures And Evaluation + +Put fixed inference messages and a self-contained embedded schema in +`internal/usecase` alongside diagnostic orchestration. Use an ordinary user +message requesting the exact token `PROMPTKIT_OK` for text inference. Structured +inference requests `{"status":"ok"}` with an object schema requiring that single +string property with `const: "ok"` and `additionalProperties: false`. +Use native schema name `promptkit_smoke`, strict mode, and the existing structured +output serializer. These literal choices are private and may change; do not +duplicate them across public tests or documentation. + +Use `internal/validate.NewFSValidator` with this private schema filesystem, not +the application's schema source. Prepare validation once per diagnostic call and +reuse its schema document for native output. Use existing JSON and schema +validation mechanisms for distinct `ValidJSON` and `SchemaMatched` facts. Do not +repair content. Text matching trims whitespace only. JSON/schema failures use +`ExecutionFailureValidation`; an unexpected text answer is expressed by the +text facts without inventing a domain-validity verdict. + +### Shared Execution And Evidence + +Keep source-neutral diagnostic values under `internal/domain`. Add a +`usecase.SmokeTester` assembled with the existing runner's captured selection +collaborators, the same generation coordinator, admission manager, and a narrow +built-in HTTP probe collaborator. Share existing private selection and override +helpers within `internal/usecase`; do not implement diagnostics by calling `Run` +or `Prepare`, and do not render an application's prompt. + +Add a per-invocation internal evidence path to `internal/llm` and +`internal/generation`. A diagnostic `GenerateWithEvidence` entry point and +ordinary `Generate` share the same private generation implementation. Ordinary +calls need not allocate diagnostic records. Evidence retains dispatch state, +final observed destination, HTTP status, credential presence, decoded-response +availability, timings, and presence-aware usage before existing cancellation +checks can discard a result. Private response content exists only long enough +for local evaluation and is not copied into reports. + +Observe HTTP requests with an invocation-local wrapper around the selected +client's RoundTripper, on a shallow client copy. Preserve its timeout, redirects, +cookie jar, and underlying transport; do not mutate a caller's client or forward +credentials specially. A nil transport uses `http.DefaultTransport`. Record only +the most recent observed request and response, resetting destination-specific +response evidence on each new hop. This reports what PromptKit can observe at +its boundary, not wire transmission or any hidden dispatch inside a custom +transport. Do not retain headers, credentials, raw bodies, or a redirect trace. + +Reuse the coordinator's failure classification and extract its existing +presence-aware usage aggregation into a small internal reusable accumulator if +needed. Do not duplicate that arithmetic or expand public `ExecutionEvent`. +Mask enclosing execution observation with `observation.WithoutExecutionState` +so diagnostic traffic does not become an application's ordinary execution events. + +### Result And Error Decisions + +Use the request/report declarations and reason constants in the feature roadmap +without adding status/severity or a stable JSON guarantee. Reports own their +pointer values, attempt slices, effective-setting maps, and usage values. Copy +mutable request overrides using existing ownership helpers before handing them +to collaborators. Apply existing request credential redaction to formatting and +exclude `APIKey` from JSON. Do not store request credentials on an engine. + +Invalid selections, kinds, overrides, or negative retry settings are public Go +errors before dispatch. Validate numeric/settings bounds even when a supplied +setting is unused by an HTTP operation; do not require a model for HTTP probes. +Concluded diagnostic execution findings become reports, not raw provider errors. +Required inference credentials missing locally produce +`required_credential_missing` and `invalid_request`. HTTP authentication may +dispatch without credentials even if inference would require them. + +Reachability records any observed HTTP response as reachable and received; +non-2xx status alone is not failure of this probe. Authentication records +documented 401 rejection and recognized 200 acceptance. For its 200 recognition, +require one JSON object with a `data` object, string `label`, Boolean +`is_free_tier`, and numeric `usage`; tolerate unrelated fields and ignore account +limits. These are a minimal identifying subset of the provider's documented +[current-key response](https://github.com/OpenRouterTeam/docs/blob/main/api_reference/limits.mdx#checking-your-limits), +not a validation of the full account document. An unrecognized 200 yields the +accepted reason and `malformed_response`; other statuses retain acceptance nil +and use ordinary provider-status classification. Never infer acceptance from +arbitrary provider text. + +For inference, generation/transport failures retain their existing classifications. +Set content facts only when evaluation actually occurs. Preserve decoded +generation and usage evidence even when subsequent caller cancellation interrupts +evaluation. A malformed HTTP envelope still counts as an observed response. +An injected client establishes response availability and generation facts, not +HTTP reachability, destinations, status, or upstream credential supply. + +Use the configured client timeout and selected target timeout, as the existing +built-in client does. When the outer caller context remains active, an elapsed +request timeout concludes a report with `deadline` and nil Go error. If caller +cancellation/deadline interrupts the diagnostic, return the partial report with +`ctx.Err()` identity, `Completed=false`, and corresponding failure. Check context +before dispatch and between phases; synchronous injected code must return before +PromptKit can observe cancellation. Preserve a diagnostic already completed +before later cancellation; do not promise new behavior for preparation APIs. + +Report-level facts come from the last dispatched attempt, then its local +evaluation. A later undispatched wait cannot erase them. An interrupted backoff +adds elapsed retry delay, not an attempt. Total timing includes orchestration; +phase sums need not equal wall time. Request time includes transport/body/envelope +work but excludes capacity wait, retry delay, and generated-content evaluation. +Absent phases stay nil; performed work can measure zero. + +Internal fixture loading/compilation, request construction, or assembly faults +that prevent meaningful diagnostic execution may return a content-free Go error +through the existing facade boundary. Do not recover arbitrary panics or promise +that every internal fault becomes a report. Do not expose raw provider or injected +error text in a diagnostic report or its operational-error return. + +## Stage 1: Add Internal Diagnostic Values + +Scope: `internal/domain`, with focused ownership helpers only. + +1. Add request, report, target, facts, timing, attempt, kind, and reason values + corresponding to the feature roadmap. Reuse existing execution settings, + failure kinds, and observed usage types. +2. Add source-neutral backend smoke-test metadata values to `domain.Backend`. + Keep ConfigKit and HTTP transport types out of domain values. +3. Implement copying needed for nested optional metadata, report values, and + provider maps with existing JSON-value helpers. Separate private decoded + response evidence from report values so conversion cannot expose content. + +Validate: `go test ./internal/domain ./internal/backend`. Add ownership tests +only for mutable structures whose accidental sharing would affect calls. +No public `SmokeTest` method or temporary not-implemented API is introduced yet. + +## Stage 2: Load Final Backend Diagnostic Metadata + +Scope: `internal/configuration`, `internal/backend`, and `backends.go`. + +1. Extend the final typed configuration with the decided fields, normalize and + validate through existing resolution/validation callbacks, and capture them + in the immutable backend index. Deep-copy optional nested values on lookup. +2. Implement the public convenience types and `BackendSource` translation. + Preserve existing convenience omission rules; do not add a parallel loader. +3. Test absent/null blocks, effective nested overlays, explicit endpoint fallback, + invalid final adapter/URL/default syntax, and owned convenience input values. + Show that an invalid lower contribution replaced by a valid final value is + not separately domain-validated. +4. Confirm a dangling default-profile reference is retained until selected and + does not invalidate ordinary engine construction. No provider contact. + +Validate: `go test ./internal/configuration ./internal/backend` and focused root +backend-source tests. Update backend-field GoDoc and `docs/formats.md` as these +configuration fields land; runtime smoke-test guidance waits for Stage 18. + +## Stage 3: Resolve Diagnostic Targets Without Preparing Applications + +Scope: selection helpers in `internal/usecase`; no transport. + +1. Add the diagnostic selector using `resolvePromptInspection`, captured profile + selection, execution override merging, and captured backend lookup. +2. Implement the roadmap's prompt/profile/backend-default precedence exactly. + An explicit profile wins over a backend hint; selecting A's default may route + to B or to an endpoint-only profile without recursive default lookup. +3. Split model completeness from shared settings/base-endpoint validation so + HTTP probes can use profile metadata without a model. Backend-only selection + never loads a default inference profile. Resolve exact diagnostic destinations + and retain the base execution settings separately in the target report. +4. Validate kinds/retry overrides and selections without resolving application + artifacts, session templates, inputs, messages, or output schemas. + +Validate: focused `internal/usecase` tests for the selection table, explicit +selection failure, default-reference failure only on use, cross-backend/default +identity, endpoint-only profiles, and HTTP model incompleteness. Use real captured +configuration and failing external readers to prove no application preparation. + +## Stage 4: Add OpenRouter Catalog Diagnostic Assets + +Scope: `../promptkit-backend-openrouter` only, plus this plan's completion record. + +1. Add the opt-in embedded root/constant and keyed backend metadata decided above. + Leave existing `Root` assets and model inventory unchanged. +2. Add a small asset-surface test for the new root and its shipped declaration. + Reuse existing YAML test helpers; do not add runtime adapters or PromptKit + imports to the catalog. +3. Document the opt-in root and its relation to existing `/v3` consumers in the + README. Clarify that selecting new metadata requires a compatible loader. +4. Run the catalog's local checks with `GOWORK=off`; review and commit. Create + an annotated, unused local candidate tag using the catalog version-selection + rule described under Stage 6. Record the full peeled commit and tag here. + Do not require remote synchronization, publication, or a public fetch. + +Validate: offline tests, race tests, vet, build, formatting, and local hygiene +from the catalog runbook. Its remote-publication checks belong to later release +preparation, not this local-candidate step. + +## Stage 5: Add RakestrawHome Catalog Diagnostic Assets + +Scope: `../promptkit-backend-rakestrawhome` only, plus this plan's completion record. + +1. Add the same opt-in root/constant with its existing Gemma profile as the + diagnostic default and authentication `kind: none`. +2. Leave existing base endpoint, credential environment name, capacity limits, + profile data, and ordinary execution credential policy unchanged. Declaring + authentication inapplicable does not alter generation's credential settings. +3. Add the focused asset test and README explanation as in Stage 4. Validate, + commit, and locally tag independently; record the tag and full commit here. + +Validate: the same local catalog checks, without provider access or publication. +Both catalog changes must be complete before PromptKit selects their new roots. + +## Stage 6: Adopt Exact Catalog Candidates Through Local Module Resolution + +Scope: `internal/catalog`, module pins/checksums, attribution, integration tests; +temporary tooling stays outside all three repositories. + +1. Choose candidate versions independently as `v3..`: + use today's UTC date and the first unused local revision, monotonically above + the selected version. These are local candidate identities, not claims of + publication. Do not overwrite existing tags. If tagged contents need changes + later, commit them and choose a new candidate revision rather than reuse its + already-adopted identity. +2. A `mktemp -d` workspace may contain an uncommitted `go.work` selecting all + three checkouts for development. Set `GOWORK` explicitly when using it; never + commit workspace files or filesystem `replace` directives. +3. Build a temporary filesystem module proxy from the exact peeled catalog tag + commits. A small scratch Go helper can use the already-selected + `golang.org/x/mod/module` escaping helpers and + [`zip.CreateFromVCS`](https://pkg.go.dev/golang.org/x/mod/zip#CreateFromVCS) + to generate compliant archives. Do not add a production proxy tool or forge + module-cache entries. Supply each escaped module/version's `.info`, `.mod`, + `.zip`, and version list under `@v`, following the + [Go proxy protocol](https://go.dev/ref/mod#goproxy-protocol). + `.mod` comes from that commit; `.info` uses its version and commit UTC time. +4. Cache existing dependencies before switching resolution. For this workflow + only, use `GOWORK=off`, `GOPROXY=file://`, + `GONOPROXY=none`, and `GONOSUMDB` scoped to the two catalog module paths. + Existing dependencies can use the existing download cache as a second local + filesystem proxy. Do not change global Go environment configuration. +5. Update both ordinary `/v3` requirements to the exact candidates with the Go + tool; let it generate `go.sum`. Include both exported roots in the existing + strict catalog composition group, ahead of existing PromptKit-owned profiles + and caller overlays. Do not hard-code provider metadata in PromptKit. +6. Update selected module/version and commit-pinned notices in attribution after + checking each candidate's notices. Keep the existing licenses unchanged unless + the actual upstream material changed. Update stale selected-catalog references + in format/internal docs where needed. +7. Test shipped defaults through the final assembled configuration, including + `promptkit-openrouter-standard`, and test caller replacement/null behavior. + Do not separately parse or audit source contributions in PromptKit. + +Validate: `GOWORK=off go test ./internal/catalog ./internal/configuration` plus +focused assembled-construction tests under the temporary proxy environment. +Verify `go list -m -json` reports the exact versions and no replacement. Record +the tag-to-commit mapping here, not temporary paths. Later publication must use +the actual UTC publication date; if new version names are needed, retain the +validated contents and update PromptKit's pins/checksums before its release. +Publication and public module availability are not sprint completion gates. + +## Stage 7: Capture Destination-Specific HTTP Evidence + +Scope: shared transport helpers in `internal/llm`, independent of generation. + +1. Add an invocation-local evidence collector and client-copy execution helper. + Wrap the actual selected RoundTripper and preserve standard redirect policy. +2. Record each observed outgoing request's destination and credential presence, + retaining only the last hop; clear previous response/status facts when a new + request is observed. Record response availability/status before body decoding + or caller-context handling can discard it. +3. Keep dispatch invocation distinct from observed transport evidence. Handle + redirect-policy termination and response-plus-error according to what was + actually observed; do not infer a later dispatch from a Location header. +4. Exclude URL user information from exposed endpoints and keep credential/body + values out of collector records. Provide plain GET support reusing the built-in + configured client and target timeout without introducing public methods. + +Validate: focused adapter tests with `httptest` or controlled transports for a +response, no response, credential-stripping redirect, failed redirected hop, and +redirect-policy termination. Verify the supplied client is not mutated and +evidence is isolated across concurrent calls. Test facts, not wrapper choreography. + +## Stage 8: Retain Generation Evidence Before Result Discard + +Scope: `PreparedGeneration` execution in `internal/llm`. + +1. Add an evidence-enabled internal execution path sharing the existing frozen + payload, credential resolution, transport, size handling, envelope decoding, + usage decoding, and context error logic. Preserve ordinary `Execute` behavior. +2. Attach Stage 7 evidence to attempt outcomes. Retain decoded-response success + and usage before context checks overwrite an error/result. +3. Distinguish HTTP response acquisition from successful envelope decoding and + generated-content availability, including empty decoded content. Native-schema + request presence is known at dispatch; no provider-enforcement inference. + +Validate: focused `internal/llm` tests for valid/empty/malformed generation, +non-success status, and a decoded response followed by cancellation. Reuse +existing envelope/usage/size tests rather than reproduce their full matrices. + +## Stage 9: Carry Evidence Through Built-In Retry Coordination + +Scope: `internal/generation` and shared observation usage accounting. + +1. Add `GenerateWithEvidence` using the same private built-in attempt/retry loop + as `Generate`. Collect attempts before ordinary result-discard checks. +2. Preserve status retry eligibility, frozen payload/credentials, active-permit + lifetime, jitter, and Retry-After handling. Add no network/validation retries. +3. Collect capacity waits and elapsed backoffs, including interrupted backoffs. + A canceled acquisition can produce an undispatched record; backoff alone does + not produce an attempt. Preserve last dispatched evidence independently. +4. Make the existing failure classifier internally reusable. Share existing + presence-aware usage aggregation with diagnostics, extracting a small + accumulator from `internal/observation` if needed; preserve its existing tests + and no-dispatch known/complete-zero semantics. + +Validate: coordinator tests using its controlled clock/wait seams for one +attempt, retry-success, exhausted retries, cancellation during backoff, and +canceled reacquisition. Check evidence retention and timing/accounting; existing +retry tests own unchanged retry policy. Run observation tests after extraction. + +## Stage 10: Retain Injected-Client Evidence Without HTTP Claims + +Scope: the injected branch of `internal/generation`. + +1. Extend the shared evidence path to one injected dispatch using the same active + manager. Preserve the existing no-automatic-retry rule. +2. Capture returned response availability, successful generation/usage, and + native-schema request presence. Retain successful decoded content privately + if cancellation is observed immediately after the client returns. +3. Keep destinations, HTTP status/reachability, and upstream credential-supply + facts unknown. A response returned alongside an error establishes receipt, + not successful generation or content conformance; retain existing injected + usage policy rather than inventing billing completeness on errors. + +Validate: injected coordinator tests for success, nil response, opaque failure, +response-plus-error, and success followed by caller cancellation. Verify one +dispatch despite a positive retry allowance and ownership at the existing client +adapter boundary. Do not add methods to public `LLMClient`. + +## Stage 11: Implement The Reachability Probe + +Scope: internal HTTP probe collaborator and focused use-case support decisions. + +1. Resolve the chosen full diagnostic endpoint or effective base and perform + exactly one unauthenticated GET with the selected built-in HTTP client. +2. Report any observed HTTP response as reachable/received, independently of its + status. Report transport failure without guessing DNS/TLS subcategories. +3. Ignore inference retries for this operation after validating the option. + Never acquire a generation permit or require credentials/model completeness. +4. For injected-client engines, return the unsupported transport outcome without + falling back to another client. Preserve target selection errors as Go errors. + +Validate: internal probe tests for 2xx/401/404/405, transport failure, timeout, +and unsupported injected transport. Derive evidence from Stage 7 rather than +repeat redirect matrices here. Close response bodies; do not retain them. + +## Stage 12: Implement Authentication Applicability And OpenRouter Probe + +Scope: named HTTP authentication adapter; no inference orchestration. + +1. Evaluate captured metadata first: absent/null means unsupported/unknown + applicability; `none` means supported, inapplicable, completed, no dispatch. + Known inapplicability also works in injected-client engines. +2. For `openrouter_key`, use the selected full endpoint or base-relative `key` + path. Resolve direct/environment credentials consistently with generation, + permitting a missing key for this probe. Send at most one GET. +3. Implement the decided 200 recognition and 401 rejection. Preserve unknown + acceptance for other responses/transport failure, correct reason/failure + classification, and last-observed destination evidence. Discard account data. +4. Keep adapter choice fixed by metadata, not inferred from hostname, HTTP status, + or a models-list response. Reuse configured timeout and redirect behavior. + +Validate: focused adapter tests with synthetic current-key responses for accepted, +rejected, anonymous rejection, malformed/unrecognized 200, other status, and +transport failure. Add applicability/support tests without duplicating HTTP +client tests. Synthetic credentials must not appear in report-facing values. + +## Stage 13: Build And Evaluate Private Inference Cases + +Scope: private fixtures and validation collaborators in `internal/usecase`. + +1. Add the fixed messages/schema described above, independent of application + prompt/profile listings and schema options. Create ordinary generation requests + from selected settings, using zero repairs and the diagnostic retry allowance. +2. Reuse prepared private schema validation for native schema request and local + conformance. Evaluate JSON syntax separately before evaluating the schema. +3. Populate text/JSON/schema facts only when evaluated; whitespace text returns + false, case changes do not match, invalid JSON leaves schema matching nil. + Keep malformed generation separate from content validation failure. + +Validate: focused case tests for match/mismatch/empty text, JSON parse failure, +schema failure, and success using real validators. Derive generated success +fixtures from the private case rather than spread private literals through tests. +No domain criteria, configurable prompts, public fixtures, or repair loop. + +## Stage 14: Orchestrate HTTP Diagnostic Reports + +Scope: `internal/usecase.SmokeTester` and its HTTP operations. + +1. Connect selection, support/applicability, and HTTP probes into a synchronous + operation. Start total timing before selection; HTTP probes bypass generation + capacity. Keep this internal while inference and engine wiring are completed. +2. Assemble HTTP attempts, measured phases, facts, reasons, and failure kinds. + Preserve nil unknown measurements and no-dispatch usage semantics. +3. Apply the decided Go-error/report boundary and mask inherited observation. +4. Distinguish outer cancellation from configured request timeouts; retain + previously established facts and finalize total timing on every report return. + +Validate: use-case tests for representative HTTP report-state rows, invalid +selection before dispatch, canceled-before-dispatch, timeout with live caller, +and available evidence after cancellation. Use real internal components with +controlled HTTP effects, not a new mock framework. + +## Stage 15: Integrate Inference Diagnostics And Engine Assembly + +Scope: inference orchestration in `internal/usecase` and `internal/engine` wiring. + +1. Connect private inference cases to the shared coordinator and existing + admission manager. Admit one logical diagnostic until it concludes; active + permits remain coordinator-owned and release during backoff/local evaluation. + Use the effective profile backend identity, including an empty identity. +2. Nil diagnostic retries means zero, independent of the engine default. Retain + all actual attempts, summed measured phases, aggregate usage, and last-dispatched + facts followed by generated-content evaluation. Keep interrupted waits from + replacing those facts. Do not run structural repair. +3. Turn local capacity exhaustion and missing required inference credentials into + concluded findings without dispatch. Apply the shared report/error and context + boundary, retaining decoded evidence without evaluating canceled content. +4. Assemble the engine's ordinary runner and smoke tester with the same snapshot, + coordinator, client, and capacity manager. Built-in engines supply the shared + HTTP collaborator; injected engines supply none. Do not change construction + options or public client methods. + +Validate: focused use-case/engine tests for text/schema results, retry-default +isolation, injected generation, required-credential/capacity findings, and evidence +after cancellation. Stage 17 owns broader facade-level capacity/observer regressions. + +## Stage 16: Expose The Public SmokeTest Facade + +Scope: root declarations, GoDoc, conversions, and external-package API tests. + +1. Add the exact request/report method and public values from the feature roadmap + in a focused root file. Preserve the facade/internal ownership boundary and + existing public error translation. +2. Translate owned request values and reports, including a non-nil partial + report when a recognizable context error is returned. Add request formatting + redaction using established conventions; `APIKey` stays excluded from JSON. +3. Write complete GoDoc: nil Go error means concluded, not successful; tri-state + facts; selection precedence; HTTP transport support; retry/repair semantics; + configured timeout versus caller cancellation; timing and disclosure boundaries. +4. Cover nil engine, invalid kind, detached mutable values, concluded diagnostic + failure with nil Go error, and partial cancellation identity through the API. + +Validate: focused root `go test` cases plus internal engine tests. Do not copy all +internal selection/HTTP matrices into external-package tests or promise stable +JSON output for the new report types. + +## Stage 17: Verify Assembled Capacity, Cancellation, And Privacy Boundaries + +Scope: a small number of consequential integration regressions. + +1. Prove ordinary `Run` and inference `SmokeTest` on the same engine share a + test-controlled backend limit/admission bound. Verify permits release after + failure/cancellation and a later call proceeds. Avoid scheduler-order/fairness + guarantees and sleep-based synchronization. +2. Verify canceled retry backoff and later capacity wait preserve last-dispatched + facts/status/usage, report interruption separately, and do not add a synthetic + backoff attempt. Reuse lower-level tests for mechanical retry behavior. +3. Verify application prompt messages, inputs, session templates, artifacts, and + schema sources are unused, and private cases do not enter listings/inspection. +4. Verify reports/operational-error returns omit synthetic credential values, + generated text, provider messages, raw HTTP bodies, and private fixture content. + Verify diagnostic traffic does not enter an inherited execution observer. +5. Test a representative cross-backend default and endpoint override through the + facade with a controlled transport to prove selection/capacity/probe wiring. + +Validate: focused race-enabled tests for these assembled paths. Keep cases small +and each risk owned once; do not introduce a second end-to-end state-matrix suite. + +## Stage 18: Complete Canonical Documentation + +Scope: GoDoc and existing consumer, format, integration, and internal owners. + +1. Add a consumer smoke-test section with minimal request/report handling, + explicit interpretation of nil error, optional facts, and a partial cancellation + example. Link to GoDoc for exact declarations rather than duplicate them. +2. Finish the backend-format reference for defaults, named adapters, explicit + endpoints, null/overlay behavior, and opt-in catalog assets. Link the selected + catalog versions rather than promise provider availability or free tests. +3. Update integration/internal documents for observed HTTP evidence, private + fixtures, shared coordinator/capacity, operator-only configuration inspection, + and injected-client limitations. Update the internal inventory only for + components actually introduced. Add a brief README navigation link if useful. +4. Describe diagnostics as explicit live consumer operations, separate from the + offline default suite and release checks. Do not add a live example/check to + existing release tooling or make an extra maintained executable necessary. +5. Record the later release-note need and unpublished catalog handoff here. + Do not rewrite historical release notes or publish a release in this sprint. + +Validate: local documentation paths/anchors, GoDoc against implemented behavior, +and any changed example snippets through focused offline tests. No online link +availability or human sign-off gate. + +## Stage 19: Validate The Integrated Candidate And Record Handoff + +Scope: final verification, completion records, and unintended-file cleanup. + +1. Rebuild the temporary proxy from the recorded exact tags if necessary. From + an isolated temporary module/cache, resolve both selected catalog versions + with `GOWORK=off` and that proxy, verify module identities/versions and no + replacements, and compare Go-generated sums with the adopted requirements. + Do not use local checkout substitution for this proof. Other dependencies can + resolve from the prepared existing local download-cache proxy. +2. Run the integrated PromptKit suite with `GOWORK=off` and local proxy settings: + `go test -race ./...`, `go vet ./...`, and + `python3 scripts/check-examples.py`. Check Go formatting, `git diff --check`, + local documentation links, and absence of committed workspace/replacement or + proxy/cache artifacts. These are sprint checks, not additions to release gates. +3. Reuse successful catalog checks for unchanged tagged candidates. Ensure all + three repositories' intended changes are committed; preserve and report any + unrelated work. Correct discovered defects with focused tests/commits and + rerun only checks invalidated by those changes. +4. Mark completed stages and record final catalog versions/full commits, + validation results, and the publication handoff below. Explain that catalog + tags/pins have been validated locally but not necessarily published, and that + publication-date re-pinning may be needed before a distributable release. +5. Report completion without waiting for public module fetches, provider access, + CI, human review, or separate loader/catalog releases. Publication of catalogs + and PromptKit follows their documented procedures under separate authority. + +## Completion And Publication Handoff + +To be filled as stages complete: catalog candidate tags and full peeled commits, +completed stage markers, final validation results, and any remaining publication +preparation. Temporary environment paths are not durable integration metadata. +There are no remaining product decisions required to start this plan. diff --git a/docs/roadmap/smoke-tests.md b/docs/roadmap/smoke-tests.md index 25d9219..c374748 100644 --- a/docs/roadmap/smoke-tests.md +++ b/docs/roadmap/smoke-tests.md @@ -1,11 +1,11 @@ # Explicit Smoke Test Diagnostics -Status: feature design and delivery direction accepted; remaining implementation -details are listed below. Implementation has not started. +Status: accepted target state. Implementation has not started. This roadmap replaces the earlier capability proposal and incorporates the subsequent discussion. It defines the target feature, not a staged implementation -plan or current public API. Remaining implementation details are listed at the end. +plan or current public API. The companion [implementation plan](implementation.md) +owns the ordered work and delivery mechanics. ## Purpose And Ownership @@ -39,19 +39,17 @@ intermediate human-review approvals, upstream releases, external publication, or live-provider access to progress or establish completion. Automated checks and focused local commits remain part of ordinary implementation. -Update the local OpenRouter and RakestrawHome catalog checkouts, validate and -commit their changes, and identify the resulting revisions with local candidate -tags. Built-in diagnostic metadata belongs in those catalogs, not in an -intermediate PromptKit-owned metadata underlay. This supersedes both the earlier -loader-release/catalog-release/adoption sequence and the temporary core-owned -underlay proposal. +Built-in diagnostic metadata belongs in the local OpenRouter and RakestrawHome +catalogs, with their validated revisions identified by local candidate tags. +There is no intermediate PromptKit-owned metadata underlay or requirement for +separate loader/catalog publication before integration. A temporary, uncommitted Go workspace may support coordinated development. -Final sprint validation must also resolve the exact locally tagged catalog -revisions through a temporary filesystem module proxy, using normal module -requirements and checksums with `GOWORK=off` and no filesystem replacements. -Local tags alone do not make Go discover adjacent checkouts. The proxy supplies -that connection without public HTTPS availability or upstream publication. +The completed integration uses normal module requirements and Go-generated +checksums and is validated with `GOWORK=off`, without filesystem replacements. +Exact local tag contents can be supplied through a temporary filesystem module +proxy, without public HTTPS availability or upstream publication. Local tags +alone do not make Go discover adjacent checkouts. Keep workspace/proxy artifacts outside committed repository contents, following the [repository-independence policy](../policy/architecture.md#repository-independence). @@ -382,8 +380,6 @@ backends: openrouter: smoke_tests: default_profile: promptkit-openrouter-standard - reachability: - endpoint: https://openrouter.ai/api/v1 authentication: kind: openrouter_key rakestrawhome: @@ -405,7 +401,8 @@ Provider paths follow the selected endpoint override, not silently the catalog's original endpoint. Explicit absolute diagnostic endpoints are supported for separate health/auth routes and take precedence for their operation. Configuration stays limited to named adapters and endpoint settings, not arbitrary methods or response -predicates. Concrete field names and presence/null rules remain to be specified. +predicates. The companion implementation plan specifies convenience types and +presence/null interpretation for these fields. Reuse the configured HTTP client and its redirect handling, without adding custom credential forwarding. `Target.Endpoint` records the resolved starting destination. @@ -423,27 +420,25 @@ attribution without introducing a public redirect trace. Deliver built-in metadata in the local catalog repositories at `../promptkit-backend-openrouter` and `../promptkit-backend-rakestrawhome`. -Catalog changes, local validation, commits, candidate tags, and PromptKit adoption -are all included in the sprint; publication and remote module availability are -not intermediate dependencies. Caller overlays remain available through ordinary -composition, and excluded catalogs remain excluded. Interpret and validate only -the assembled final configuration. +Catalog changes and PromptKit adoption are included in the sprint; publication +and remote module availability are not intermediate dependencies. Caller +overlays remain available through ordinary composition. Interpret and validate +only the assembled final configuration. Built-in reachability follows the +effective endpoint rather than pinning a separate catalog endpoint. -Use standards-compliant module archives made from the exact tagged commits for -the temporary filesystem proxy, not mutable working-tree contents or manually -edited module-cache entries. Update normal version requirements, let Go generate -checksums, and validate the integrated feature without workspace substitution. -Scope any temporary proxy/checksum-database settings to this local workflow; -do not impose them on consumers or commit machine-specific module resolution. +The catalogs expose diagnostic metadata through an additive, opt-in embedded +asset root. Existing `/v3` catalog roots remain usable by older strict loaders; +new PromptKit explicitly selects the additional assets. Provider metadata stays +owned by its catalog rather than being duplicated in PromptKit. Local candidate tags identify validated contents; they are not published releases. -Specify candidate version selection and reconcile the catalogs' publication-date -versioning convention during separate publication preparation. If that requires -new final version names, update PromptKit's pins and checksums accordingly; retain -the validated catalog contents and never move a published tag. Older strict -PromptKit loaders may reject the new metadata fields, so describe the compatible -consumer versions when publishing. No intervening PromptKit loader release or -external module-download verification is required to finish implementation. +The implementation plan owns candidate selection and local proxy mechanics. +The catalogs' publication-date versioning convention still applies to later +publication. If that requires new final version names, update PromptKit's pins +and checksums accordingly; retain the validated catalog contents and never move +a published tag. Selecting the new metadata requires a compatible PromptKit +loader. No intervening PromptKit loader release or external module-download +verification is required to finish implementation. Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key` adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication @@ -560,7 +555,6 @@ scheduling, and any combination with `ValidateRequest` or normal execution. ## Work Needed And Target Completion - Add the public request/report operation and GoDoc, keeping business logic internal. - Resolve the remaining delivery details when writing the staged implementation plan. - Extend final backend metadata interpretation and captured lookup for capabilities and defaults. Preserve explicit configuration inspection without exposing test content or using source history for validation. @@ -571,13 +565,11 @@ scheduling, and any combination with `ValidateRequest` or normal execution. and partial cancellation reporting through the internal evidence path while preserving existing client contracts. Specify destination/credential attribution without exposing headers or a redirect trace. -- Update and validate both local catalogs with the agreed metadata, then commit - and locally tag their candidate revisions. Adopt those versions in PromptKit - with ordinary catalog/caller precedence and no provider-availability guarantee. - Check default-profile resolution in the assembled final configuration and final - local-proxy module resolution without workspace/replacement substitution. - Complete all work without intermediate review or publication gates, as required - by the single-sprint objective. +- Deliver the agreed metadata through both local catalogs with ordinary + catalog/caller precedence and no provider-availability guarantee. Shipped + defaults resolve in the assembled final configuration, and selected versions + work without workspace/replacement substitution. The single-sprint objective + applies to the complete integration. - Add focused offline tests with controlled transports and injected clients for selection, rejected/absent credentials, applicability/support, transport/malformed responses, content/JSON/schema facts, cancellation, shared inference capacity, @@ -598,28 +590,3 @@ on cancellation. Existing preparation and execution continue unchanged. Initial scope excludes application CLI/service work, readiness policy, continuous monitoring, model-status tests, batch orchestration, customizable test language, cost estimates, and guarantees about later generation. - -## Remaining Implementation Details - -The operation, routing precedence, report/error boundary, initial adapters, -retry/repair policy, default profiles, injected-client scope, and single-sprint -delivery approach are settled. The remaining items should be resolved in the -focused implementation plan: - -1. **Configuration representation.** Finalize the Go convenience types and exact - endpoint fields, including omission/null/overlay behavior and interpretation of - configured default-profile references at construction versus selection. Preserve - final-value ownership and verify shipped defaults in assembled configuration; - do not introduce source-history validation or change ordinary profile selection. -2. **Local catalog integration.** Specify local candidate versions/tags and temporary - workspace/proxy setup for the two catalog checkouts, with standards-compliant - archives of the tagged contents and ordinary Go checksum generation. Define - final validation without workspace/replacement substitution and keep later - publication separate. There is no external catalog publication gate or - PromptKit-owned diagnostic metadata underlay. -3. **Internal implementation.** Choose private content and schema, the narrow evidence - handoff, and content-free classifications for internal faults that prevent a - meaningful report. Define ownership and copying for reports and request values. - Verify the state matrix, timeout/cancellation distinction, and redirect evidence - against actual transport behavior before finalizing GoDoc. Keep public execution - events and ordinary result/error semantics unchanged.