From d8b6c0b471556a4b9619ce05d81237fc2b0ac1eb Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 13 Oct 2026 14:48:47 +0000 Subject: [PATCH] Make smoke test delivery self-contained in one sprint --- docs/roadmap/smoke-tests.md | 82 ++++++++++++++++++++++++------------- 1 file changed, 53 insertions(+), 29 deletions(-) diff --git a/docs/roadmap/smoke-tests.md b/docs/roadmap/smoke-tests.md index 996e703..62d392c 100644 --- a/docs/roadmap/smoke-tests.md +++ b/docs/roadmap/smoke-tests.md @@ -5,7 +5,7 @@ details are listed below. Implementation has not started. This roadmap replaces the earlier capability proposal and incorporates the subsequent discussion. It defines the target feature, not a staged implementation -plan or current public API. Remaining questions appear under Remaining Decisions. +plan or current public API. Remaining implementation details are listed at the end. ## Purpose And Ownership @@ -31,6 +31,25 @@ A reachable endpoint can reject credentials, and a transport failure can leave acceptance undetermined. Authentication is inapplicable on the maintainer's credential-free RakestrawHome deployment; that is neither an error nor a failure. +## Single-Sprint Delivery Objective + +The staged implementation must be completable from start to finish in one sprint, +including built-in defaults, tests, and documentation. It must not require +intermediate human-review approvals, upstream releases, external publication, or +live-provider access to progress or establish completion. Automated checks and +focused local commits remain part of ordinary implementation. + +Use the currently released catalog dependencies and PromptKit-owned diagnostic +metadata underlays for the initial delivery. This supersedes the earlier +loader-release/catalog-release/adoption sequence. The feature must be complete +and usable without unpublished dependencies, committed workspace files, or local +filesystem replacements. Any later move of metadata into catalogs is optional +follow-up work, not an unfinished stage or a prerequisite for this sprint. + +Normal release preparation and publication follow implementation separately; +this objective does not bypass the repository's release process or authorize +publication during the sprint. + ## Current Baseline At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation. @@ -354,6 +373,7 @@ backends: kind: openrouter_key rakestrawhome: smoke_tests: + default_profile: rakestrawhome-gemma-4-31b authentication: kind: none ``` @@ -386,17 +406,19 @@ destination under the selected adapter's contract. It does not assert acceptance at the initial destination or at every redirect hop. These endpoint fields provide attribution without introducing a public redirect trace. -Deliver built-in metadata through coordinated compatible catalog releases; caller -overlays remain available. First release loader support while retaining compatible -existing catalog dependencies, then release catalogs containing the new metadata, -then adopt those versions in PromptKit. Older strict loaders may reject new fields, -so do not direct their consumers to upgrade catalogs independently. Select actual -versions and compatibility notes during release preparation. +Deliver initial built-in metadata through PromptKit-owned underlays composed with +the currently released catalog assets. Arrange these defaults so catalog metadata +and caller overlays can replace them through ordinary composition. Supply defaults +for enabled maintained catalogs without reintroducing a backend whose catalog the +caller excluded. Interpret and validate only the assembled final configuration. +No catalog edits, dependency publication, or intervening PromptKit release are +required for initial completion. Later catalog adoption, if pursued, must account +for older strict loaders that reject new fields; it is outside this sprint. Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key` -adapter, and RakestrawHome's `none` authentication setting. The RakestrawHome catalog -maintainer must select its current default profile. Verify each shipped default -resolves in the supported assembled effective configuration, including any +adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication +setting. Verify each shipped default resolves in the supported assembled effective +configuration, including any PromptKit-owned profiles. This is final-value integration validation, not a second validation pass over intermediate sources. A catalog default is replaceable metadata, not a promise of upstream availability, cheap inference, or a stable model choice. @@ -430,10 +452,13 @@ establish small output, a price estimate, or a cost ceiling. Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without new required methods. HTTP diagnostics must describe an explicitly selected transport, not secretly probe a different transport in an injected-client engine. -The initial behavior is to report HTTP diagnostics unavailable unless -that transport is supported or explicitly supplied; an additional injection option -remains a delivery detail below. Do not silently fall back to a default HTTP client -to diagnose an injected generation client. +For injected-client engines, the initial delivery reports HTTP diagnostics +unavailable and adds no separate diagnostic HTTP-client option. Inference +diagnostics remain available through the injected client. Known inapplicability, +such as authentication configured as `none`, can be reported without any transport. +Do not silently fall back to a default HTTP client to diagnose an injected +generation client. A future explicit diagnostic transport option remains possible +but is not needed to complete this feature. Return conventions: @@ -516,9 +541,11 @@ scheduling, and any combination with `ValidateRequest` or normal execution. and partial cancellation reporting through the internal evidence path while preserving existing client contracts. Specify destination/credential attribution without exposing headers or a redirect trace. -- Supply agreed built-in metadata through compatible catalog releases in the - documented order, with normal caller overlays and no provider-availability - guarantee. Check default-profile resolution in the assembled final configuration. +- Supply agreed built-in metadata through PromptKit-owned underlays over existing + released dependencies, with ordinary catalog/caller precedence and no + provider-availability guarantee. Check default-profile resolution in the assembled + final configuration. Complete all work without intermediate review or publication + gates, as required by the single-sprint objective. - Add focused offline tests with controlled transports and injected clients for selection, rejected/absent credentials, applicability/support, transport/malformed responses, content/JSON/schema facts, cancellation, shared inference capacity, @@ -540,26 +567,23 @@ scope excludes application CLI/service work, readiness policy, continuous monito model-status tests, batch orchestration, customizable test language, cost estimates, and guarantees about later generation. -## Remaining Decisions And Delivery Dependencies +## Remaining Implementation Details The operation, routing precedence, report/error boundary, initial adapters, -retry/repair policy, and catalog delivery direction are settled. The remaining -items should be resolved in the focused implementation plan: +retry/repair policy, default profiles, injected-client scope, and single-sprint +delivery approach are settled. The remaining items should be resolved in the +focused implementation plan: 1. **Configuration representation.** Finalize the Go convenience types and exact endpoint fields, including omission/null/overlay behavior and interpretation of configured default-profile references at construction versus selection. Preserve final-value ownership and verify shipped defaults in assembled configuration; do not introduce source-history validation or change ordinary profile selection. -2. **Explicit diagnostic transport for injected clients.** Decide whether the first - delivery exposes a separate diagnostic HTTP-client option or reports HTTP - diagnostics unavailable for injected-client engines until such an option exists. - Inference remains available through `LLMClient` in either case. No hidden - transport fallback or new required client methods are permitted. -3. **Catalog coordination.** Obtain the RakestrawHome maintainer's default-profile - choice and select compatible loader/catalog versions in the documented release - order. These are delivery dependencies, not reasons to reopen routing policy. -4. **Implementation details.** Choose private content and schema, the narrow evidence +2. **Default metadata composition.** Choose the underlay representation and placement + that lets later catalog metadata and caller overlays replace the defaults while + respecting catalog selection. Keep released dependencies usable throughout; + there is no external catalog coordination gate. +3. **Internal implementation.** Choose private content and schema, the narrow evidence handoff, and content-free classifications for internal faults that prevent a meaningful report. Define ownership and copying for reports and request values. Verify the state matrix, timeout/cancellation distinction, and redirect evidence