Make smoke test delivery self-contained in one sprint

This commit is contained in:
2026-10-13 14:48:47 +00:00
parent 52cd94fcd1
commit d8b6c0b471

View File

@@ -5,7 +5,7 @@ details are listed below. Implementation has not started.
This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. Remaining questions appear under Remaining Decisions.
plan or current public API. Remaining implementation details are listed at the end.
## Purpose And Ownership
@@ -31,6 +31,25 @@ A reachable endpoint can reject credentials, and a transport failure can leave
acceptance undetermined. Authentication is inapplicable on the maintainer's
credential-free RakestrawHome deployment; that is neither an error nor a failure.
## Single-Sprint Delivery Objective
The staged implementation must be completable from start to finish in one sprint,
including built-in defaults, tests, and documentation. It must not require
intermediate human-review approvals, upstream releases, external publication, or
live-provider access to progress or establish completion. Automated checks and
focused local commits remain part of ordinary implementation.
Use the currently released catalog dependencies and PromptKit-owned diagnostic
metadata underlays for the initial delivery. This supersedes the earlier
loader-release/catalog-release/adoption sequence. The feature must be complete
and usable without unpublished dependencies, committed workspace files, or local
filesystem replacements. Any later move of metadata into catalogs is optional
follow-up work, not an unfinished stage or a prerequisite for this sprint.
Normal release preparation and publication follow implementation separately;
this objective does not bypass the repository's release process or authorize
publication during the sprint.
## Current Baseline
At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
@@ -354,6 +373,7 @@ backends:
kind: openrouter_key
rakestrawhome:
smoke_tests:
default_profile: rakestrawhome-gemma-4-31b
authentication:
kind: none
```
@@ -386,17 +406,19 @@ destination under the selected adapter's contract. It does not assert acceptance
at the initial destination or at every redirect hop. These endpoint fields provide
attribution without introducing a public redirect trace.
Deliver built-in metadata through coordinated compatible catalog releases; caller
overlays remain available. First release loader support while retaining compatible
existing catalog dependencies, then release catalogs containing the new metadata,
then adopt those versions in PromptKit. Older strict loaders may reject new fields,
so do not direct their consumers to upgrade catalogs independently. Select actual
versions and compatibility notes during release preparation.
Deliver initial built-in metadata through PromptKit-owned underlays composed with
the currently released catalog assets. Arrange these defaults so catalog metadata
and caller overlays can replace them through ordinary composition. Supply defaults
for enabled maintained catalogs without reintroducing a backend whose catalog the
caller excluded. Interpret and validate only the assembled final configuration.
No catalog edits, dependency publication, or intervening PromptKit release are
required for initial completion. Later catalog adoption, if pursued, must account
for older strict loaders that reject new fields; it is outside this sprint.
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
adapter, and RakestrawHome's `none` authentication setting. The RakestrawHome catalog
maintainer must select its current default profile. Verify each shipped default
resolves in the supported assembled effective configuration, including any
adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
setting. Verify each shipped default resolves in the supported assembled effective
configuration, including any
PromptKit-owned profiles. This is final-value integration validation, not a second
validation pass over intermediate sources. A catalog default is replaceable metadata,
not a promise of upstream availability, cheap inference, or a stable model choice.
@@ -430,10 +452,13 @@ establish small output, a price estimate, or a cost ceiling.
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
new required methods. HTTP diagnostics must describe an explicitly selected
transport, not secretly probe a different transport in an injected-client engine.
The initial behavior is to report HTTP diagnostics unavailable unless
that transport is supported or explicitly supplied; an additional injection option
remains a delivery detail below. Do not silently fall back to a default HTTP client
to diagnose an injected generation client.
For injected-client engines, the initial delivery reports HTTP diagnostics
unavailable and adds no separate diagnostic HTTP-client option. Inference
diagnostics remain available through the injected client. Known inapplicability,
such as authentication configured as `none`, can be reported without any transport.
Do not silently fall back to a default HTTP client to diagnose an injected
generation client. A future explicit diagnostic transport option remains possible
but is not needed to complete this feature.
Return conventions:
@@ -516,9 +541,11 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
and partial cancellation reporting through the internal evidence path while
preserving existing client contracts. Specify destination/credential attribution
without exposing headers or a redirect trace.
- Supply agreed built-in metadata through compatible catalog releases in the
documented order, with normal caller overlays and no provider-availability
guarantee. Check default-profile resolution in the assembled final configuration.
- Supply agreed built-in metadata through PromptKit-owned underlays over existing
released dependencies, with ordinary catalog/caller precedence and no
provider-availability guarantee. Check default-profile resolution in the assembled
final configuration. Complete all work without intermediate review or publication
gates, as required by the single-sprint objective.
- Add focused offline tests with controlled transports and injected clients for
selection, rejected/absent credentials, applicability/support, transport/malformed
responses, content/JSON/schema facts, cancellation, shared inference capacity,
@@ -540,26 +567,23 @@ scope excludes application CLI/service work, readiness policy, continuous monito
model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation.
## Remaining Decisions And Delivery Dependencies
## Remaining Implementation Details
The operation, routing precedence, report/error boundary, initial adapters,
retry/repair policy, and catalog delivery direction are settled. The remaining
items should be resolved in the focused implementation plan:
retry/repair policy, default profiles, injected-client scope, and single-sprint
delivery approach are settled. The remaining items should be resolved in the
focused implementation plan:
1. **Configuration representation.** Finalize the Go convenience types and exact
endpoint fields, including omission/null/overlay behavior and interpretation of
configured default-profile references at construction versus selection. Preserve
final-value ownership and verify shipped defaults in assembled configuration;
do not introduce source-history validation or change ordinary profile selection.
2. **Explicit diagnostic transport for injected clients.** Decide whether the first
delivery exposes a separate diagnostic HTTP-client option or reports HTTP
diagnostics unavailable for injected-client engines until such an option exists.
Inference remains available through `LLMClient` in either case. No hidden
transport fallback or new required client methods are permitted.
3. **Catalog coordination.** Obtain the RakestrawHome maintainer's default-profile
choice and select compatible loader/catalog versions in the documented release
order. These are delivery dependencies, not reasons to reopen routing policy.
4. **Implementation details.** Choose private content and schema, the narrow evidence
2. **Default metadata composition.** Choose the underlay representation and placement
that lets later catalog metadata and caller overlays replace the defaults while
respecting catalog selection. Keep released dependencies usable throughout;
there is no external catalog coordination gate.
3. **Internal implementation.** Choose private content and schema, the narrow evidence
handoff, and content-free classifications for internal faults that prevent a
meaningful report. Define ownership and copying for reports and request values.
Verify the state matrix, timeout/cancellation distinction, and redirect evidence