Plan smoke test diagnostics implementation

This commit is contained in:
2026-10-13 15:21:58 +00:00
parent c8f9a4def9
commit 8fbd058b90
2 changed files with 679 additions and 68 deletions

View File

@@ -1,11 +1,11 @@
# Explicit Smoke Test Diagnostics
Status: feature design and delivery direction accepted; remaining implementation
details are listed below. Implementation has not started.
Status: accepted target state. Implementation has not started.
This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. Remaining implementation details are listed at the end.
plan or current public API. The companion [implementation plan](implementation.md)
owns the ordered work and delivery mechanics.
## Purpose And Ownership
@@ -39,19 +39,17 @@ intermediate human-review approvals, upstream releases, external publication, or
live-provider access to progress or establish completion. Automated checks and
focused local commits remain part of ordinary implementation.
Update the local OpenRouter and RakestrawHome catalog checkouts, validate and
commit their changes, and identify the resulting revisions with local candidate
tags. Built-in diagnostic metadata belongs in those catalogs, not in an
intermediate PromptKit-owned metadata underlay. This supersedes both the earlier
loader-release/catalog-release/adoption sequence and the temporary core-owned
underlay proposal.
Built-in diagnostic metadata belongs in the local OpenRouter and RakestrawHome
catalogs, with their validated revisions identified by local candidate tags.
There is no intermediate PromptKit-owned metadata underlay or requirement for
separate loader/catalog publication before integration.
A temporary, uncommitted Go workspace may support coordinated development.
Final sprint validation must also resolve the exact locally tagged catalog
revisions through a temporary filesystem module proxy, using normal module
requirements and checksums with `GOWORK=off` and no filesystem replacements.
Local tags alone do not make Go discover adjacent checkouts. The proxy supplies
that connection without public HTTPS availability or upstream publication.
The completed integration uses normal module requirements and Go-generated
checksums and is validated with `GOWORK=off`, without filesystem replacements.
Exact local tag contents can be supplied through a temporary filesystem module
proxy, without public HTTPS availability or upstream publication. Local tags
alone do not make Go discover adjacent checkouts.
Keep workspace/proxy artifacts outside committed repository contents, following
the [repository-independence policy](../policy/architecture.md#repository-independence).
@@ -382,8 +380,6 @@ backends:
openrouter:
smoke_tests:
default_profile: promptkit-openrouter-standard
reachability:
endpoint: https://openrouter.ai/api/v1
authentication:
kind: openrouter_key
rakestrawhome:
@@ -405,7 +401,8 @@ Provider paths follow the selected endpoint override, not silently the catalog's
original endpoint. Explicit absolute diagnostic endpoints are supported for separate
health/auth routes and take precedence for their operation. Configuration stays
limited to named adapters and endpoint settings, not arbitrary methods or response
predicates. Concrete field names and presence/null rules remain to be specified.
predicates. The companion implementation plan specifies convenience types and
presence/null interpretation for these fields.
Reuse the configured HTTP client and its redirect handling, without adding custom
credential forwarding. `Target.Endpoint` records the resolved starting destination.
@@ -423,27 +420,25 @@ attribution without introducing a public redirect trace.
Deliver built-in metadata in the local catalog repositories at
`../promptkit-backend-openrouter` and `../promptkit-backend-rakestrawhome`.
Catalog changes, local validation, commits, candidate tags, and PromptKit adoption
are all included in the sprint; publication and remote module availability are
not intermediate dependencies. Caller overlays remain available through ordinary
composition, and excluded catalogs remain excluded. Interpret and validate only
the assembled final configuration.
Catalog changes and PromptKit adoption are included in the sprint; publication
and remote module availability are not intermediate dependencies. Caller
overlays remain available through ordinary composition. Interpret and validate
only the assembled final configuration. Built-in reachability follows the
effective endpoint rather than pinning a separate catalog endpoint.
Use standards-compliant module archives made from the exact tagged commits for
the temporary filesystem proxy, not mutable working-tree contents or manually
edited module-cache entries. Update normal version requirements, let Go generate
checksums, and validate the integrated feature without workspace substitution.
Scope any temporary proxy/checksum-database settings to this local workflow;
do not impose them on consumers or commit machine-specific module resolution.
The catalogs expose diagnostic metadata through an additive, opt-in embedded
asset root. Existing `/v3` catalog roots remain usable by older strict loaders;
new PromptKit explicitly selects the additional assets. Provider metadata stays
owned by its catalog rather than being duplicated in PromptKit.
Local candidate tags identify validated contents; they are not published releases.
Specify candidate version selection and reconcile the catalogs' publication-date
versioning convention during separate publication preparation. If that requires
new final version names, update PromptKit's pins and checksums accordingly; retain
the validated catalog contents and never move a published tag. Older strict
PromptKit loaders may reject the new metadata fields, so describe the compatible
consumer versions when publishing. No intervening PromptKit loader release or
external module-download verification is required to finish implementation.
The implementation plan owns candidate selection and local proxy mechanics.
The catalogs' publication-date versioning convention still applies to later
publication. If that requires new final version names, update PromptKit's pins
and checksums accordingly; retain the validated catalog contents and never move
a published tag. Selecting the new metadata requires a compatible PromptKit
loader. No intervening PromptKit loader release or external module-download
verification is required to finish implementation.
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
@@ -560,7 +555,6 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
## Work Needed And Target Completion
- Add the public request/report operation and GoDoc, keeping business logic internal.
Resolve the remaining delivery details when writing the staged implementation plan.
- Extend final backend metadata interpretation and captured lookup for capabilities
and defaults. Preserve explicit configuration inspection without exposing test
content or using source history for validation.
@@ -571,13 +565,11 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
and partial cancellation reporting through the internal evidence path while
preserving existing client contracts. Specify destination/credential attribution
without exposing headers or a redirect trace.
- Update and validate both local catalogs with the agreed metadata, then commit
and locally tag their candidate revisions. Adopt those versions in PromptKit
with ordinary catalog/caller precedence and no provider-availability guarantee.
Check default-profile resolution in the assembled final configuration and final
local-proxy module resolution without workspace/replacement substitution.
Complete all work without intermediate review or publication gates, as required
by the single-sprint objective.
- Deliver the agreed metadata through both local catalogs with ordinary
catalog/caller precedence and no provider-availability guarantee. Shipped
defaults resolve in the assembled final configuration, and selected versions
work without workspace/replacement substitution. The single-sprint objective
applies to the complete integration.
- Add focused offline tests with controlled transports and injected clients for
selection, rejected/absent credentials, applicability/support, transport/malformed
responses, content/JSON/schema facts, cancellation, shared inference capacity,
@@ -598,28 +590,3 @@ on cancellation. Existing preparation and execution continue unchanged. Initial
scope excludes application CLI/service work, readiness policy, continuous monitoring,
model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation.
## Remaining Implementation Details
The operation, routing precedence, report/error boundary, initial adapters,
retry/repair policy, default profiles, injected-client scope, and single-sprint
delivery approach are settled. The remaining items should be resolved in the
focused implementation plan:
1. **Configuration representation.** Finalize the Go convenience types and exact
endpoint fields, including omission/null/overlay behavior and interpretation of
configured default-profile references at construction versus selection. Preserve
final-value ownership and verify shipped defaults in assembled configuration;
do not introduce source-history validation or change ordinary profile selection.
2. **Local catalog integration.** Specify local candidate versions/tags and temporary
workspace/proxy setup for the two catalog checkouts, with standards-compliant
archives of the tagged contents and ordinary Go checksum generation. Define
final validation without workspace/replacement substitution and keep later
publication separate. There is no external catalog publication gate or
PromptKit-owned diagnostic metadata underlay.
3. **Internal implementation.** Choose private content and schema, the narrow evidence
handoff, and content-free classifications for internal faults that prevent a
meaningful report. Define ownership and copying for reports and request values.
Verify the state matrix, timeout/cancellation distinction, and redirect evidence
against actual transport behavior before finalizing GoDoc. Keep public execution
events and ordinary result/error semantics unchanged.