Plan smoke test diagnostics implementation
This commit is contained in:
@@ -1,11 +1,11 @@
|
||||
# Explicit Smoke Test Diagnostics
|
||||
|
||||
Status: feature design and delivery direction accepted; remaining implementation
|
||||
details are listed below. Implementation has not started.
|
||||
Status: accepted target state. Implementation has not started.
|
||||
|
||||
This roadmap replaces the earlier capability proposal and incorporates the
|
||||
subsequent discussion. It defines the target feature, not a staged implementation
|
||||
plan or current public API. Remaining implementation details are listed at the end.
|
||||
plan or current public API. The companion [implementation plan](implementation.md)
|
||||
owns the ordered work and delivery mechanics.
|
||||
|
||||
## Purpose And Ownership
|
||||
|
||||
@@ -39,19 +39,17 @@ intermediate human-review approvals, upstream releases, external publication, or
|
||||
live-provider access to progress or establish completion. Automated checks and
|
||||
focused local commits remain part of ordinary implementation.
|
||||
|
||||
Update the local OpenRouter and RakestrawHome catalog checkouts, validate and
|
||||
commit their changes, and identify the resulting revisions with local candidate
|
||||
tags. Built-in diagnostic metadata belongs in those catalogs, not in an
|
||||
intermediate PromptKit-owned metadata underlay. This supersedes both the earlier
|
||||
loader-release/catalog-release/adoption sequence and the temporary core-owned
|
||||
underlay proposal.
|
||||
Built-in diagnostic metadata belongs in the local OpenRouter and RakestrawHome
|
||||
catalogs, with their validated revisions identified by local candidate tags.
|
||||
There is no intermediate PromptKit-owned metadata underlay or requirement for
|
||||
separate loader/catalog publication before integration.
|
||||
|
||||
A temporary, uncommitted Go workspace may support coordinated development.
|
||||
Final sprint validation must also resolve the exact locally tagged catalog
|
||||
revisions through a temporary filesystem module proxy, using normal module
|
||||
requirements and checksums with `GOWORK=off` and no filesystem replacements.
|
||||
Local tags alone do not make Go discover adjacent checkouts. The proxy supplies
|
||||
that connection without public HTTPS availability or upstream publication.
|
||||
The completed integration uses normal module requirements and Go-generated
|
||||
checksums and is validated with `GOWORK=off`, without filesystem replacements.
|
||||
Exact local tag contents can be supplied through a temporary filesystem module
|
||||
proxy, without public HTTPS availability or upstream publication. Local tags
|
||||
alone do not make Go discover adjacent checkouts.
|
||||
Keep workspace/proxy artifacts outside committed repository contents, following
|
||||
the [repository-independence policy](../policy/architecture.md#repository-independence).
|
||||
|
||||
@@ -382,8 +380,6 @@ backends:
|
||||
openrouter:
|
||||
smoke_tests:
|
||||
default_profile: promptkit-openrouter-standard
|
||||
reachability:
|
||||
endpoint: https://openrouter.ai/api/v1
|
||||
authentication:
|
||||
kind: openrouter_key
|
||||
rakestrawhome:
|
||||
@@ -405,7 +401,8 @@ Provider paths follow the selected endpoint override, not silently the catalog's
|
||||
original endpoint. Explicit absolute diagnostic endpoints are supported for separate
|
||||
health/auth routes and take precedence for their operation. Configuration stays
|
||||
limited to named adapters and endpoint settings, not arbitrary methods or response
|
||||
predicates. Concrete field names and presence/null rules remain to be specified.
|
||||
predicates. The companion implementation plan specifies convenience types and
|
||||
presence/null interpretation for these fields.
|
||||
|
||||
Reuse the configured HTTP client and its redirect handling, without adding custom
|
||||
credential forwarding. `Target.Endpoint` records the resolved starting destination.
|
||||
@@ -423,27 +420,25 @@ attribution without introducing a public redirect trace.
|
||||
|
||||
Deliver built-in metadata in the local catalog repositories at
|
||||
`../promptkit-backend-openrouter` and `../promptkit-backend-rakestrawhome`.
|
||||
Catalog changes, local validation, commits, candidate tags, and PromptKit adoption
|
||||
are all included in the sprint; publication and remote module availability are
|
||||
not intermediate dependencies. Caller overlays remain available through ordinary
|
||||
composition, and excluded catalogs remain excluded. Interpret and validate only
|
||||
the assembled final configuration.
|
||||
Catalog changes and PromptKit adoption are included in the sprint; publication
|
||||
and remote module availability are not intermediate dependencies. Caller
|
||||
overlays remain available through ordinary composition. Interpret and validate
|
||||
only the assembled final configuration. Built-in reachability follows the
|
||||
effective endpoint rather than pinning a separate catalog endpoint.
|
||||
|
||||
Use standards-compliant module archives made from the exact tagged commits for
|
||||
the temporary filesystem proxy, not mutable working-tree contents or manually
|
||||
edited module-cache entries. Update normal version requirements, let Go generate
|
||||
checksums, and validate the integrated feature without workspace substitution.
|
||||
Scope any temporary proxy/checksum-database settings to this local workflow;
|
||||
do not impose them on consumers or commit machine-specific module resolution.
|
||||
The catalogs expose diagnostic metadata through an additive, opt-in embedded
|
||||
asset root. Existing `/v3` catalog roots remain usable by older strict loaders;
|
||||
new PromptKit explicitly selects the additional assets. Provider metadata stays
|
||||
owned by its catalog rather than being duplicated in PromptKit.
|
||||
|
||||
Local candidate tags identify validated contents; they are not published releases.
|
||||
Specify candidate version selection and reconcile the catalogs' publication-date
|
||||
versioning convention during separate publication preparation. If that requires
|
||||
new final version names, update PromptKit's pins and checksums accordingly; retain
|
||||
the validated catalog contents and never move a published tag. Older strict
|
||||
PromptKit loaders may reject the new metadata fields, so describe the compatible
|
||||
consumer versions when publishing. No intervening PromptKit loader release or
|
||||
external module-download verification is required to finish implementation.
|
||||
The implementation plan owns candidate selection and local proxy mechanics.
|
||||
The catalogs' publication-date versioning convention still applies to later
|
||||
publication. If that requires new final version names, update PromptKit's pins
|
||||
and checksums accordingly; retain the validated catalog contents and never move
|
||||
a published tag. Selecting the new metadata requires a compatible PromptKit
|
||||
loader. No intervening PromptKit loader release or external module-download
|
||||
verification is required to finish implementation.
|
||||
|
||||
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
|
||||
adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
|
||||
@@ -560,7 +555,6 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
|
||||
## Work Needed And Target Completion
|
||||
|
||||
- Add the public request/report operation and GoDoc, keeping business logic internal.
|
||||
Resolve the remaining delivery details when writing the staged implementation plan.
|
||||
- Extend final backend metadata interpretation and captured lookup for capabilities
|
||||
and defaults. Preserve explicit configuration inspection without exposing test
|
||||
content or using source history for validation.
|
||||
@@ -571,13 +565,11 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
|
||||
and partial cancellation reporting through the internal evidence path while
|
||||
preserving existing client contracts. Specify destination/credential attribution
|
||||
without exposing headers or a redirect trace.
|
||||
- Update and validate both local catalogs with the agreed metadata, then commit
|
||||
and locally tag their candidate revisions. Adopt those versions in PromptKit
|
||||
with ordinary catalog/caller precedence and no provider-availability guarantee.
|
||||
Check default-profile resolution in the assembled final configuration and final
|
||||
local-proxy module resolution without workspace/replacement substitution.
|
||||
Complete all work without intermediate review or publication gates, as required
|
||||
by the single-sprint objective.
|
||||
- Deliver the agreed metadata through both local catalogs with ordinary
|
||||
catalog/caller precedence and no provider-availability guarantee. Shipped
|
||||
defaults resolve in the assembled final configuration, and selected versions
|
||||
work without workspace/replacement substitution. The single-sprint objective
|
||||
applies to the complete integration.
|
||||
- Add focused offline tests with controlled transports and injected clients for
|
||||
selection, rejected/absent credentials, applicability/support, transport/malformed
|
||||
responses, content/JSON/schema facts, cancellation, shared inference capacity,
|
||||
@@ -598,28 +590,3 @@ on cancellation. Existing preparation and execution continue unchanged. Initial
|
||||
scope excludes application CLI/service work, readiness policy, continuous monitoring,
|
||||
model-status tests, batch orchestration, customizable test language, cost estimates,
|
||||
and guarantees about later generation.
|
||||
|
||||
## Remaining Implementation Details
|
||||
|
||||
The operation, routing precedence, report/error boundary, initial adapters,
|
||||
retry/repair policy, default profiles, injected-client scope, and single-sprint
|
||||
delivery approach are settled. The remaining items should be resolved in the
|
||||
focused implementation plan:
|
||||
|
||||
1. **Configuration representation.** Finalize the Go convenience types and exact
|
||||
endpoint fields, including omission/null/overlay behavior and interpretation of
|
||||
configured default-profile references at construction versus selection. Preserve
|
||||
final-value ownership and verify shipped defaults in assembled configuration;
|
||||
do not introduce source-history validation or change ordinary profile selection.
|
||||
2. **Local catalog integration.** Specify local candidate versions/tags and temporary
|
||||
workspace/proxy setup for the two catalog checkouts, with standards-compliant
|
||||
archives of the tagged contents and ordinary Go checksum generation. Define
|
||||
final validation without workspace/replacement substitution and keep later
|
||||
publication separate. There is no external catalog publication gate or
|
||||
PromptKit-owned diagnostic metadata underlay.
|
||||
3. **Internal implementation.** Choose private content and schema, the narrow evidence
|
||||
handoff, and content-free classifications for internal faults that prevent a
|
||||
meaningful report. Define ownership and copying for reports and request values.
|
||||
Verify the state matrix, timeout/cancellation distinction, and redirect evidence
|
||||
against actual transport behavior before finalizing GoDoc. Keep public execution
|
||||
events and ordinary result/error semantics unchanged.
|
||||
|
||||
Reference in New Issue
Block a user