promptkit/docs/roadmap/implementation.md

Smoke Test Diagnostics Implementation Plan

Status: ready for implementation.

Implement the accepted feature roadmap. This document owns the implementation sequence; the feature roadmap owns the requested API, factual reporting semantics, selection rules, and scope. Each numbered stage is a single bounded prompt for a Luna coding agent. Execute stages in order within one sprint. There are no intermediate human-review, external-publication, or live-provider gates. Focused automated validation and local commits are ordinary development steps, not pauses awaiting approval.

Baseline And Working Rules

The reviewed PromptKit baseline is c8f9a4def9f84301bf534a331320ea9fb798ac24. It selects both /v3 catalogs at v3.20260930.0 and ConfigKit at v0.7.1. The adjacent checkouts are:

• ../promptkit-backend-openrouter, reviewed at f17d208cb7ad1fb043cda2266cd3011d11b40aa0;
• ../promptkit-backend-rakestrawhome, reviewed at 993443d88c451901f4c917fa00c204fd3b54d28d.

Recheck repository status before editing and preserve unrelated work. Begin with development orientation, then the applicable agent, architecture, testing, documentation, OSS, and release policies. Read each catalog's own instructions and release procedure before its stages. Reuse unchanged policy readings during the sprint.

Read focused internal documents and owning tests before modifying a component. Use existing code-discovery guidance. Stage and commit each completed, nonempty unit with a short plain-English message after proportionate checks and staged-diff review. Record stage completion and relevant validation here. Catalog stages commit in their own repositories; PromptKit records their tag and full commit identities here. Do not push any repository or publish a PromptKit release during this implementation sprint.

Only final effective ConfigKit values are interpreted or validated by PromptKit, per ADR 0001. Do not inspect raw catalog documents or source history to enforce domain validity, even in PromptKit tests. The catalog repositories may test the assets they themselves own. Operator inspection continues to delegate to ConfigKit.

Keep existing Run, Prepare, ValidateRequest, repair, observation, and client contracts unchanged. Add no application CLI, readiness verdict, custom test criteria, batch operation, live release gate, or new public transport injection option. Do not add catalog exclusion controls absent from the current API.

Implementation Decisions

These decisions resolve the feature roadmap's delivery details. Internal names below identify a concrete implementation path, not additional public contracts.

Configuration And Catalog Assets

Add optional smoke_tests to final backend definitions:

backends:
  openrouter:
    smoke_tests:
      default_profile: promptkit-openrouter-standard
      authentication:
        kind: openrouter_key
  rakestrawhome:
    smoke_tests:
      default_profile: rakestrawhome-gemma-4-31b
      authentication:
        kind: none

Both catalogs omit a separate reachability endpoint so ordinary endpoint overrides remain effective. Caller definitions may set reachability.endpoint or authentication.endpoint to an absolute diagnostic destination. Authentication otherwise joins key to the effective base; reachability otherwise uses that base.

The public convenience surface adds SmokeTests *BackendSmokeTestConfig to BackendConfig, with:

• BackendSmokeTestConfig: DefaultProfile *string, Reachability *ReachabilityProbeConfig, Authentication *AuthenticationProbeConfig;
• ReachabilityProbeConfig: Endpoint *string;
• AuthenticationProbeConfig: Kind *string, Endpoint *string.

BackendSource omits nil convenience fields, as it does today. Explicit nulls remain expressible with ordinary ConfigKit mappings/YAML. In final values, omitted/null smoke_tests disables its defaults and authentication declaration; omitted/null default_profile means no inference default; omitted/null reachability or its endpoint means generic reachability at the effective base; omitted/null authentication means unsupported authentication; a non-null authentication object requires kind. Recognize none and openrouter_key. Normalize surrounding whitespace/case for adapter kinds and record normalization through the existing ConfigKit resolution boundary. Preserve profile IDs exactly; a supplied default-profile string must not be blank. Resolve its existence only when that default is selected, not eagerly at engine construction.

Diagnostic endpoint strings are trimmed and parsed as absolute HTTP(S) URLs with a host. Use standard URL/request validation, allowing paths and queries; do not reuse the base-endpoint validator's query restriction for these full URLs. Null means fallback, whereas an explicit blank endpoint is invalid. Retain existing base-endpoint rules for ordinary target overrides. Reachability sends no Authorization header or URL user credentials. Public endpoint fields omit URL user information; other URL components and provider settings remain potentially sensitive operator data, as specified in the feature roadmap.

Preserve the existing /v3 catalog asset format: do not place new fields under the existing Root where older strict loaders would reject them. In each module, add smoke-tests/backend.yml and the additive exported constant SmokeTestsRoot = "smoke-tests"; extend FS() embedding to include that root. Keep Root = "catalog" and its existing assets unchanged. New PromptKit explicitly selects both roots through internal/catalog; diagnostic data remains catalog-owned. This is a small opt-in asset surface, not another runtime library or a PromptKit-owned metadata underlay.

Private Fixtures And Evaluation

Put fixed inference messages and a self-contained embedded schema in internal/usecase alongside diagnostic orchestration. Use an ordinary user message requesting the exact token PROMPTKIT_OK for text inference. Structured inference requests {"status":"ok"} with an object schema requiring that single string property with const: "ok" and additionalProperties: false. Use native schema name promptkit_smoke, strict mode, and the existing structured output serializer. These literal choices are private and may change; do not duplicate them across public tests or documentation.

Use internal/validate.NewFSValidator with this private schema filesystem, not the application's schema source. Prepare validation once per diagnostic call and reuse its schema document for native output. Use existing JSON and schema validation mechanisms for distinct ValidJSON and SchemaMatched facts. Do not repair content. Text matching trims whitespace only. JSON/schema failures use ExecutionFailureValidation; an unexpected text answer is expressed by the text facts without inventing a domain-validity verdict.

Shared Execution And Evidence

Keep source-neutral diagnostic values under internal/domain. Add a usecase.SmokeTester assembled with the existing runner's captured selection collaborators, the same generation coordinator, admission manager, and a narrow built-in HTTP probe collaborator. Share existing private selection and override helpers within internal/usecase; do not implement diagnostics by calling Run or Prepare, and do not render an application's prompt.

Add a per-invocation internal evidence path to internal/llm and internal/generation. A diagnostic GenerateWithEvidence entry point and ordinary Generate share the same private generation implementation. Ordinary calls need not allocate diagnostic records. Evidence retains dispatch state, final observed destination, HTTP status, credential presence, decoded-response availability, timings, and presence-aware usage before existing cancellation checks can discard a result. Private response content exists only long enough for local evaluation and is not copied into reports.

Observe HTTP requests with an invocation-local wrapper around the selected client's RoundTripper, on a shallow client copy. Preserve its timeout, redirects, cookie jar, and underlying transport; do not mutate a caller's client or forward credentials specially. A nil transport uses http.DefaultTransport. Record only the most recent observed request and response, resetting destination-specific response evidence on each new hop. This reports what PromptKit can observe at its boundary, not wire transmission or any hidden dispatch inside a custom transport. Do not retain headers, credentials, raw bodies, or a redirect trace.

Reuse the coordinator's failure classification and extract its existing presence-aware usage aggregation into a small internal reusable accumulator if needed. Do not duplicate that arithmetic or expand public ExecutionEvent. Mask enclosing execution observation with observation.WithoutExecutionState so diagnostic traffic does not become an application's ordinary execution events.

Result And Error Decisions

Use the request/report declarations and reason constants in the feature roadmap without adding status/severity or a stable JSON guarantee. Reports own their pointer values, attempt slices, effective-setting maps, and usage values. Copy mutable request overrides using existing ownership helpers before handing them to collaborators. Apply existing request credential redaction to formatting and exclude APIKey from JSON. Do not store request credentials on an engine.

Invalid selections, kinds, overrides, or negative retry settings are public Go errors before dispatch. Validate numeric/settings bounds even when a supplied setting is unused by an HTTP operation; do not require a model for HTTP probes. Concluded diagnostic execution findings become reports, not raw provider errors. Required inference credentials missing locally produce required_credential_missing and invalid_request. HTTP authentication may dispatch without credentials even if inference would require them.

Reachability records any observed HTTP response as reachable and received; non-2xx status alone is not failure of this probe. Authentication records documented 401 rejection and recognized 200 acceptance. For its 200 recognition, require one JSON object with a data object, string label, Boolean is_free_tier, and numeric usage; tolerate unrelated fields and ignore account limits. These are a minimal identifying subset of the provider's documented current-key response, not a validation of the full account document. An unrecognized 200 yields the accepted reason and malformed_response; other statuses retain acceptance nil and use ordinary provider-status classification. Never infer acceptance from arbitrary provider text.

For inference, generation/transport failures retain their existing classifications. Set content facts only when evaluation actually occurs. Preserve decoded generation and usage evidence even when subsequent caller cancellation interrupts evaluation. A malformed HTTP envelope still counts as an observed response. An injected client establishes response availability and generation facts, not HTTP reachability, destinations, status, or upstream credential supply.

Use the configured client timeout and selected target timeout, as the existing built-in client does. When the outer caller context remains active, an elapsed request timeout concludes a report with deadline and nil Go error. If caller cancellation/deadline interrupts the diagnostic, return the partial report with ctx.Err() identity, Completed=false, and corresponding failure. Check context before dispatch and between phases; synchronous injected code must return before PromptKit can observe cancellation. Preserve a diagnostic already completed before later cancellation; do not promise new behavior for preparation APIs.

Report-level facts come from the last dispatched attempt, then its local evaluation. A later undispatched wait cannot erase them. An interrupted backoff adds elapsed retry delay, not an attempt. Total timing includes orchestration; phase sums need not equal wall time. Request time includes transport/body/envelope work but excludes capacity wait, retry delay, and generated-content evaluation. Absent phases stay nil; performed work can measure zero.

Internal fixture loading/compilation, request construction, or assembly faults that prevent meaningful diagnostic execution may return a content-free Go error through the existing facade boundary. Do not recover arbitrary panics or promise that every internal fault becomes a report. Do not expose raw provider or injected error text in a diagnostic report or its operational-error return.

Stage 1: Add Internal Diagnostic Values

Scope: internal/domain, with focused ownership helpers only.

1. Add request, report, target, facts, timing, attempt, kind, and reason values corresponding to the feature roadmap. Reuse existing execution settings, failure kinds, and observed usage types.
2. Add source-neutral backend smoke-test metadata values to domain.Backend. Keep ConfigKit and HTTP transport types out of domain values.
3. Implement copying needed for nested optional metadata, report values, and provider maps with existing JSON-value helpers. Separate private decoded response evidence from report values so conversion cannot expose content.

Validate: go test ./internal/domain ./internal/backend. Add ownership tests only for mutable structures whose accidental sharing would affect calls. No public SmokeTest method or temporary not-implemented API is introduced yet.

Completed: 2026-10-13.

Stage 2: Load Final Backend Diagnostic Metadata

Scope: internal/configuration, internal/backend, and backends.go.

1. Extend the final typed configuration with the decided fields, normalize and validate through existing resolution/validation callbacks, and capture them in the immutable backend index. Deep-copy optional nested values on lookup.
2. Implement the public convenience types and BackendSource translation. Preserve existing convenience omission rules; do not add a parallel loader.
3. Test absent/null blocks, effective nested overlays, explicit endpoint fallback, invalid final adapter/URL/default syntax, and owned convenience input values. Show that an invalid lower contribution replaced by a valid final value is not separately domain-validated.
4. Confirm a dangling default-profile reference is retained until selected and does not invalidate ordinary engine construction. No provider contact.

Validate: go test ./internal/configuration ./internal/backend and focused root backend-source tests. Update backend-field GoDoc and docs/formats.md as these configuration fields land; runtime smoke-test guidance waits for Stage 18.

Completed: 2026-10-13. Validation passed: go test ./internal/configuration ./internal/backend, go test ., and git diff --check.

Stage 3: Resolve Diagnostic Targets Without Preparing Applications

Scope: selection helpers in internal/usecase; no transport.

1. Add the diagnostic selector using resolvePromptInspection, captured profile selection, execution override merging, and captured backend lookup.
2. Implement the roadmap's prompt/profile/backend-default precedence exactly. An explicit profile wins over a backend hint; selecting A's default may route to B or to an endpoint-only profile without recursive default lookup.
3. Split model completeness from shared settings/base-endpoint validation so HTTP probes can use profile metadata without a model. Backend-only selection never loads a default inference profile. Resolve exact diagnostic destinations and retain the base execution settings separately in the target report.
4. Validate kinds/retry overrides and selections without resolving application artifacts, session templates, inputs, messages, or output schemas.

Validate: focused internal/usecase tests for the selection table, explicit selection failure, default-reference failure only on use, cross-backend/default identity, endpoint-only profiles, and HTTP model incompleteness. Use real captured configuration and failing external readers to prove no application preparation.

Completed: 2026-10-13. Validation passed: go test ./..., go vet ./..., and git diff --check.

Stage 4: Add OpenRouter Catalog Diagnostic Assets

Scope: ../promptkit-backend-openrouter only, plus this plan's completion record.

1. Add the opt-in embedded root/constant and keyed backend metadata decided above. Leave existing Root assets and model inventory unchanged.
2. Add a small asset-surface test for the new root and its shipped declaration. Reuse existing YAML test helpers; do not add runtime adapters or PromptKit imports to the catalog.
3. Document the opt-in root and its relation to existing /v3 consumers in the README. Clarify that selecting new metadata requires a compatible loader.
4. Run the catalog's local checks with GOWORK=off; review and commit. Create an annotated, unused local candidate tag using the catalog version-selection rule described under Stage 6. Record the full peeled commit and tag here. Do not require remote synchronization, publication, or a public fetch.

Validate: offline tests, race tests, vet, build, formatting, and local hygiene from the catalog runbook. Its remote-publication checks belong to later release preparation, not this local-candidate step.

Completed: 2026-10-13. OpenRouter candidate v3.20261013.0 points to peeled commit 2a69672b1bb220f1a730153341470f5ac67e15f4. Validation passed: GOWORK=off GOPROXY=off go test ./..., go test -race ./..., go vet ./..., go build ./..., formatting, git diff --check, and module identity via go list -m. The annotated tag is local only; no publication or push was made.

Stage 5: Add RakestrawHome Catalog Diagnostic Assets

Scope: ../promptkit-backend-rakestrawhome only, plus this plan's completion record.

1. Add the same opt-in root/constant with its existing Gemma profile as the diagnostic default and authentication kind: none.
2. Leave existing base endpoint, credential environment name, capacity limits, profile data, and ordinary execution credential policy unchanged. Declaring authentication inapplicable does not alter generation's credential settings.
3. Add the focused asset test and README explanation as in Stage 4. Validate, commit, and locally tag independently; record the tag and full commit here.

Validate: the same local catalog checks, without provider access or publication. Both catalog changes must be complete before PromptKit selects their new roots.

Completed: 2026-10-13. RakestrawHome candidate v3.20261013.0 points to peeled commit d40a8aad50111c15d279469170f7b5032ac22529. Validation passed: GOWORK=off GOPROXY=off go test ./..., go test -race ./..., go vet ./..., go build ./..., formatting, git diff --check, and module identity via go list -m. The annotated tag is local only; no publication or push was made.

Stage 6: Adopt Exact Catalog Candidates Through Local Module Resolution

Scope: internal/catalog, module pins/checksums, attribution, integration tests; temporary tooling stays outside all three repositories.

1. Choose candidate versions independently as v3.<UTC preparation date>.<N>: use today's UTC date and the first unused local revision, monotonically above the selected version. These are local candidate identities, not claims of publication. Do not overwrite existing tags. If tagged contents need changes later, commit them and choose a new candidate revision rather than reuse its already-adopted identity.
2. A mktemp -d workspace may contain an uncommitted go.work selecting all three checkouts for development. Set GOWORK explicitly when using it; never commit workspace files or filesystem replace directives.
3. Build a temporary filesystem module proxy from the exact peeled catalog tag commits. A small scratch Go helper can use the already-selected golang.org/x/mod/module escaping helpers and zip.CreateFromVCS to generate compliant archives. Do not add a production proxy tool or forge module-cache entries. Supply each escaped module/version's .info, .mod, .zip, and version list under @v, following the Go proxy protocol. .mod comes from that commit; .info uses its version and commit UTC time.
4. Cache existing dependencies before switching resolution. For this workflow only, use GOWORK=off, GOPROXY=file://<temporary-proxy>, GONOPROXY=none, and GONOSUMDB scoped to the two catalog module paths. Existing dependencies can use the existing download cache as a second local filesystem proxy. Do not change global Go environment configuration.
5. Update both ordinary /v3 requirements to the exact candidates with the Go tool; let it generate go.sum. Include both exported roots in the existing strict catalog composition group, ahead of existing PromptKit-owned profiles and caller overlays. Do not hard-code provider metadata in PromptKit.
6. Update selected module/version and commit-pinned notices in attribution after checking each candidate's notices. Keep the existing licenses unchanged unless the actual upstream material changed. Update stale selected-catalog references in format/internal docs where needed.
7. Test shipped defaults through the final assembled configuration, including promptkit-openrouter-standard, and test caller replacement/null behavior. Do not separately parse or audit source contributions in PromptKit.

Validate: GOWORK=off go test ./internal/catalog ./internal/configuration plus focused assembled-construction tests under the temporary proxy environment. Verify go list -m -json reports the exact versions and no replacement. Record the tag-to-commit mapping here, not temporary paths. Later publication must use the actual UTC publication date; if new version names are needed, retain the validated contents and update PromptKit's pins/checksums before its release. Publication and public module availability are not sprint completion gates.

Completed: 2026-10-13. Adopted the local OpenRouter candidate v3.20261013.0 at 2a69672b1bb220f1a730153341470f5ac67e15f4 and RakestrawHome candidate v3.20261013.0 at d40a8aad50111c15d279469170f7b5032ac22529. Validation passed through a temporary filesystem module proxy: focused catalog/configuration tests, go test ./..., go vet ./..., go mod verify, exact module identity and version checks with no replacements, and git diff --check. No workspace or proxy artifacts were committed; the candidate tags remain local and unpublished.

Stage 7: Capture Destination-Specific HTTP Evidence

Scope: shared transport helpers in internal/llm, independent of generation.

1. Add an invocation-local evidence collector and client-copy execution helper. Wrap the actual selected RoundTripper and preserve standard redirect policy.
2. Record each observed outgoing request's destination and credential presence, retaining only the last hop; clear previous response/status facts when a new request is observed. Record response availability/status before body decoding or caller-context handling can discard it.
3. Keep dispatch invocation distinct from observed transport evidence. Handle redirect-policy termination and response-plus-error according to what was actually observed; do not infer a later dispatch from a Location header.
4. Exclude URL user information from exposed endpoints and keep credential/body values out of collector records. Provide plain GET support reusing the built-in configured client and target timeout without introducing public methods.

Validate: focused adapter tests with httptest or controlled transports for a response, no response, credential-stripping redirect, failed redirected hop, and redirect-policy termination. Verify the supplied client is not mutated and evidence is isolated across concurrent calls. Test facts, not wrapper choreography.

Stage 8: Retain Generation Evidence Before Result Discard

Scope: PreparedGeneration execution in internal/llm.

1. Add an evidence-enabled internal execution path sharing the existing frozen payload, credential resolution, transport, size handling, envelope decoding, usage decoding, and context error logic. Preserve ordinary Execute behavior.
2. Attach Stage 7 evidence to attempt outcomes. Retain decoded-response success and usage before context checks overwrite an error/result.
3. Distinguish HTTP response acquisition from successful envelope decoding and generated-content availability, including empty decoded content. Native-schema request presence is known at dispatch; no provider-enforcement inference.

Validate: focused internal/llm tests for valid/empty/malformed generation, non-success status, and a decoded response followed by cancellation. Reuse existing envelope/usage/size tests rather than reproduce their full matrices.

Stage 9: Carry Evidence Through Built-In Retry Coordination

Scope: internal/generation and shared observation usage accounting.

1. Add GenerateWithEvidence using the same private built-in attempt/retry loop as Generate. Collect attempts before ordinary result-discard checks.
2. Preserve status retry eligibility, frozen payload/credentials, active-permit lifetime, jitter, and Retry-After handling. Add no network/validation retries.
3. Collect capacity waits and elapsed backoffs, including interrupted backoffs. A canceled acquisition can produce an undispatched record; backoff alone does not produce an attempt. Preserve last dispatched evidence independently.
4. Make the existing failure classifier internally reusable. Share existing presence-aware usage aggregation with diagnostics, extracting a small accumulator from internal/observation if needed; preserve its existing tests and no-dispatch known/complete-zero semantics.

Validate: coordinator tests using its controlled clock/wait seams for one attempt, retry-success, exhausted retries, cancellation during backoff, and canceled reacquisition. Check evidence retention and timing/accounting; existing retry tests own unchanged retry policy. Run observation tests after extraction.

Stage 10: Retain Injected-Client Evidence Without HTTP Claims

Scope: the injected branch of internal/generation.

1. Extend the shared evidence path to one injected dispatch using the same active manager. Preserve the existing no-automatic-retry rule.
2. Capture returned response availability, successful generation/usage, and native-schema request presence. Retain successful decoded content privately if cancellation is observed immediately after the client returns.
3. Keep destinations, HTTP status/reachability, and upstream credential-supply facts unknown. A response returned alongside an error establishes receipt, not successful generation or content conformance; retain existing injected usage policy rather than inventing billing completeness on errors.

Validate: injected coordinator tests for success, nil response, opaque failure, response-plus-error, and success followed by caller cancellation. Verify one dispatch despite a positive retry allowance and ownership at the existing client adapter boundary. Do not add methods to public LLMClient.

Stage 11: Implement The Reachability Probe

Scope: internal HTTP probe collaborator and focused use-case support decisions.

1. Resolve the chosen full diagnostic endpoint or effective base and perform exactly one unauthenticated GET with the selected built-in HTTP client.
2. Report any observed HTTP response as reachable/received, independently of its status. Report transport failure without guessing DNS/TLS subcategories.
3. Ignore inference retries for this operation after validating the option. Never acquire a generation permit or require credentials/model completeness.
4. For injected-client engines, return the unsupported transport outcome without falling back to another client. Preserve target selection errors as Go errors.

Validate: internal probe tests for 2xx/401/404/405, transport failure, timeout, and unsupported injected transport. Derive evidence from Stage 7 rather than repeat redirect matrices here. Close response bodies; do not retain them.

Stage 12: Implement Authentication Applicability And OpenRouter Probe

Scope: named HTTP authentication adapter; no inference orchestration.

1. Evaluate captured metadata first: absent/null means unsupported/unknown applicability; none means supported, inapplicable, completed, no dispatch. Known inapplicability also works in injected-client engines.
2. For openrouter_key, use the selected full endpoint or base-relative key path. Resolve direct/environment credentials consistently with generation, permitting a missing key for this probe. Send at most one GET.
3. Implement the decided 200 recognition and 401 rejection. Preserve unknown acceptance for other responses/transport failure, correct reason/failure classification, and last-observed destination evidence. Discard account data.
4. Keep adapter choice fixed by metadata, not inferred from hostname, HTTP status, or a models-list response. Reuse configured timeout and redirect behavior.

Validate: focused adapter tests with synthetic current-key responses for accepted, rejected, anonymous rejection, malformed/unrecognized 200, other status, and transport failure. Add applicability/support tests without duplicating HTTP client tests. Synthetic credentials must not appear in report-facing values.

Stage 13: Build And Evaluate Private Inference Cases

Scope: private fixtures and validation collaborators in internal/usecase.

1. Add the fixed messages/schema described above, independent of application prompt/profile listings and schema options. Create ordinary generation requests from selected settings, using zero repairs and the diagnostic retry allowance.
2. Reuse prepared private schema validation for native schema request and local conformance. Evaluate JSON syntax separately before evaluating the schema.
3. Populate text/JSON/schema facts only when evaluated; whitespace text returns false, case changes do not match, invalid JSON leaves schema matching nil. Keep malformed generation separate from content validation failure.

Validate: focused case tests for match/mismatch/empty text, JSON parse failure, schema failure, and success using real validators. Derive generated success fixtures from the private case rather than spread private literals through tests. No domain criteria, configurable prompts, public fixtures, or repair loop.

Stage 14: Orchestrate HTTP Diagnostic Reports

Scope: internal/usecase.SmokeTester and its HTTP operations.

1. Connect selection, support/applicability, and HTTP probes into a synchronous operation. Start total timing before selection; HTTP probes bypass generation capacity. Keep this internal while inference and engine wiring are completed.
2. Assemble HTTP attempts, measured phases, facts, reasons, and failure kinds. Preserve nil unknown measurements and no-dispatch usage semantics.
3. Apply the decided Go-error/report boundary and mask inherited observation.
4. Distinguish outer cancellation from configured request timeouts; retain previously established facts and finalize total timing on every report return.

Validate: use-case tests for representative HTTP report-state rows, invalid selection before dispatch, canceled-before-dispatch, timeout with live caller, and available evidence after cancellation. Use real internal components with controlled HTTP effects, not a new mock framework.

Stage 15: Integrate Inference Diagnostics And Engine Assembly

Scope: inference orchestration in internal/usecase and internal/engine wiring.

1. Connect private inference cases to the shared coordinator and existing admission manager. Admit one logical diagnostic until it concludes; active permits remain coordinator-owned and release during backoff/local evaluation. Use the effective profile backend identity, including an empty identity.
2. Nil diagnostic retries means zero, independent of the engine default. Retain all actual attempts, summed measured phases, aggregate usage, and last-dispatched facts followed by generated-content evaluation. Keep interrupted waits from replacing those facts. Do not run structural repair.
3. Turn local capacity exhaustion and missing required inference credentials into concluded findings without dispatch. Apply the shared report/error and context boundary, retaining decoded evidence without evaluating canceled content.
4. Assemble the engine's ordinary runner and smoke tester with the same snapshot, coordinator, client, and capacity manager. Built-in engines supply the shared HTTP collaborator; injected engines supply none. Do not change construction options or public client methods.

Validate: focused use-case/engine tests for text/schema results, retry-default isolation, injected generation, required-credential/capacity findings, and evidence after cancellation. Stage 17 owns broader facade-level capacity/observer regressions.

Stage 16: Expose The Public SmokeTest Facade

Scope: root declarations, GoDoc, conversions, and external-package API tests.

1. Add the exact request/report method and public values from the feature roadmap in a focused root file. Preserve the facade/internal ownership boundary and existing public error translation.
2. Translate owned request values and reports, including a non-nil partial report when a recognizable context error is returned. Add request formatting redaction using established conventions; APIKey stays excluded from JSON.
3. Write complete GoDoc: nil Go error means concluded, not successful; tri-state facts; selection precedence; HTTP transport support; retry/repair semantics; configured timeout versus caller cancellation; timing and disclosure boundaries.
4. Cover nil engine, invalid kind, detached mutable values, concluded diagnostic failure with nil Go error, and partial cancellation identity through the API.

Validate: focused root go test cases plus internal engine tests. Do not copy all internal selection/HTTP matrices into external-package tests or promise stable JSON output for the new report types.

Stage 17: Verify Assembled Capacity, Cancellation, And Privacy Boundaries

Scope: a small number of consequential integration regressions.

1. Prove ordinary Run and inference SmokeTest on the same engine share a test-controlled backend limit/admission bound. Verify permits release after failure/cancellation and a later call proceeds. Avoid scheduler-order/fairness guarantees and sleep-based synchronization.
2. Verify canceled retry backoff and later capacity wait preserve last-dispatched facts/status/usage, report interruption separately, and do not add a synthetic backoff attempt. Reuse lower-level tests for mechanical retry behavior.
3. Verify application prompt messages, inputs, session templates, artifacts, and schema sources are unused, and private cases do not enter listings/inspection.
4. Verify reports/operational-error returns omit synthetic credential values, generated text, provider messages, raw HTTP bodies, and private fixture content. Verify diagnostic traffic does not enter an inherited execution observer.
5. Test a representative cross-backend default and endpoint override through the facade with a controlled transport to prove selection/capacity/probe wiring.

Validate: focused race-enabled tests for these assembled paths. Keep cases small and each risk owned once; do not introduce a second end-to-end state-matrix suite.

Stage 18: Complete Canonical Documentation

Scope: GoDoc and existing consumer, format, integration, and internal owners.

1. Add a consumer smoke-test section with minimal request/report handling, explicit interpretation of nil error, optional facts, and a partial cancellation example. Link to GoDoc for exact declarations rather than duplicate them.
2. Finish the backend-format reference for defaults, named adapters, explicit endpoints, null/overlay behavior, and opt-in catalog assets. Link the selected catalog versions rather than promise provider availability or free tests.
3. Update integration/internal documents for observed HTTP evidence, private fixtures, shared coordinator/capacity, operator-only configuration inspection, and injected-client limitations. Update the internal inventory only for components actually introduced. Add a brief README navigation link if useful.
4. Describe diagnostics as explicit live consumer operations, separate from the offline default suite and release checks. Do not add a live example/check to existing release tooling or make an extra maintained executable necessary.
5. Record the later release-note need and unpublished catalog handoff here. Do not rewrite historical release notes or publish a release in this sprint.

Validate: local documentation paths/anchors, GoDoc against implemented behavior, and any changed example snippets through focused offline tests. No online link availability or human sign-off gate.

Stage 19: Validate The Integrated Candidate And Record Handoff

Scope: final verification, completion records, and unintended-file cleanup.

1. Rebuild the temporary proxy from the recorded exact tags if necessary. From an isolated temporary module/cache, resolve both selected catalog versions with GOWORK=off and that proxy, verify module identities/versions and no replacements, and compare Go-generated sums with the adopted requirements. Do not use local checkout substitution for this proof. Other dependencies can resolve from the prepared existing local download-cache proxy.
2. Run the integrated PromptKit suite with GOWORK=off and local proxy settings: go test -race ./..., go vet ./..., and python3 scripts/check-examples.py. Check Go formatting, git diff --check, local documentation links, and absence of committed workspace/replacement or proxy/cache artifacts. These are sprint checks, not additions to release gates.
3. Reuse successful catalog checks for unchanged tagged candidates. Ensure all three repositories' intended changes are committed; preserve and report any unrelated work. Correct discovered defects with focused tests/commits and rerun only checks invalidated by those changes.
4. Mark completed stages and record final catalog versions/full commits, validation results, and the publication handoff below. Explain that catalog tags/pins have been validated locally but not necessarily published, and that publication-date re-pinning may be needed before a distributable release.
5. Report completion without waiting for public module fetches, provider access, CI, human review, or separate loader/catalog releases. Publication of catalogs and PromptKit follows their documented procedures under separate authority.

Completion And Publication Handoff

To be filled as stages complete: catalog candidate tags and full peeled commits, completed stage markers, final validation results, and any remaining publication preparation. Temporary environment paths are not durable integration metadata. There are no remaining product decisions required to start this plan.