eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Make smoke test delivery self-contained in one sprint

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-13 14:48:47 +00:00
parent 52cd94fcd1
commit d8b6c0b471
1 changed files with 53 additions and 29 deletions
Download Patch File Download Diff File Expand all files Collapse all files

82
docs/roadmap/smoke-tests.md
View File

@@ -5,7 +5,7 @@ details are listed below. Implementation has not started.
This roadmap replaces the earlier capability proposal and incorporates the This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. Remaining questions appear under Remaining Decisions. plan or current public API. Remaining implementation details are listed at the end.
## Purpose And Ownership ## Purpose And Ownership
@@ -31,6 +31,25 @@ A reachable endpoint can reject credentials, and a transport failure can leave
acceptance undetermined. Authentication is inapplicable on the maintainer's acceptance undetermined. Authentication is inapplicable on the maintainer's
credential-free RakestrawHome deployment; that is neither an error nor a failure. credential-free RakestrawHome deployment; that is neither an error nor a failure.
## Single-Sprint Delivery Objective
The staged implementation must be completable from start to finish in one sprint,
including built-in defaults, tests, and documentation. It must not require
intermediate human-review approvals, upstream releases, external publication, or
live-provider access to progress or establish completion. Automated checks and
focused local commits remain part of ordinary implementation.
Use the currently released catalog dependencies and PromptKit-owned diagnostic
metadata underlays for the initial delivery. This supersedes the earlier
loader-release/catalog-release/adoption sequence. The feature must be complete
and usable without unpublished dependencies, committed workspace files, or local
filesystem replacements. Any later move of metadata into catalogs is optional
follow-up work, not an unfinished stage or a prerequisite for this sprint.
Normal release preparation and publication follow implementation separately;
this objective does not bypass the repository's release process or authorize
publication during the sprint.
## Current Baseline ## Current Baseline
At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation. At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
@@ -354,6 +373,7 @@ backends:
kind: openrouter_key kind: openrouter_key
rakestrawhome: rakestrawhome:
smoke_tests: smoke_tests:
default_profile: rakestrawhome-gemma-4-31b
authentication: authentication:
kind: none kind: none
``` ```
@@ -386,17 +406,19 @@ destination under the selected adapter's contract. It does not assert acceptance
at the initial destination or at every redirect hop. These endpoint fields provide at the initial destination or at every redirect hop. These endpoint fields provide
attribution without introducing a public redirect trace. attribution without introducing a public redirect trace.
Deliver built-in metadata through coordinated compatible catalog releases; caller Deliver initial built-in metadata through PromptKit-owned underlays composed with
overlays remain available. First release loader support while retaining compatible the currently released catalog assets. Arrange these defaults so catalog metadata
existing catalog dependencies, then release catalogs containing the new metadata, and caller overlays can replace them through ordinary composition. Supply defaults
then adopt those versions in PromptKit. Older strict loaders may reject new fields, for enabled maintained catalogs without reintroducing a backend whose catalog the
so do not direct their consumers to upgrade catalogs independently. Select actual caller excluded. Interpret and validate only the assembled final configuration.
versions and compatibility notes during release preparation. No catalog edits, dependency publication, or intervening PromptKit release are
required for initial completion. Later catalog adoption, if pursued, must account
for older strict loaders that reject new fields; it is outside this sprint.
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key` Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
adapter, and RakestrawHome's `none` authentication setting. The RakestrawHome catalog adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
maintainer must select its current default profile. Verify each shipped default setting. Verify each shipped default resolves in the supported assembled effective
resolves in the supported assembled effective configuration, including any configuration, including any
PromptKit-owned profiles. This is final-value integration validation, not a second PromptKit-owned profiles. This is final-value integration validation, not a second
validation pass over intermediate sources. A catalog default is replaceable metadata, validation pass over intermediate sources. A catalog default is replaceable metadata,
not a promise of upstream availability, cheap inference, or a stable model choice. not a promise of upstream availability, cheap inference, or a stable model choice.
@@ -430,10 +452,13 @@ establish small output, a price estimate, or a cost ceiling.
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
new required methods. HTTP diagnostics must describe an explicitly selected new required methods. HTTP diagnostics must describe an explicitly selected
transport, not secretly probe a different transport in an injected-client engine. transport, not secretly probe a different transport in an injected-client engine.
The initial behavior is to report HTTP diagnostics unavailable unless For injected-client engines, the initial delivery reports HTTP diagnostics
that transport is supported or explicitly supplied; an additional injection option unavailable and adds no separate diagnostic HTTP-client option. Inference
remains a delivery detail below. Do not silently fall back to a default HTTP client diagnostics remain available through the injected client. Known inapplicability,
to diagnose an injected generation client. such as authentication configured as `none`, can be reported without any transport.
Do not silently fall back to a default HTTP client to diagnose an injected
generation client. A future explicit diagnostic transport option remains possible
but is not needed to complete this feature.
Return conventions: Return conventions:
@@ -516,9 +541,11 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
and partial cancellation reporting through the internal evidence path while and partial cancellation reporting through the internal evidence path while
preserving existing client contracts. Specify destination/credential attribution preserving existing client contracts. Specify destination/credential attribution
without exposing headers or a redirect trace. without exposing headers or a redirect trace.
- Supply agreed built-in metadata through compatible catalog releases in the - Supply agreed built-in metadata through PromptKit-owned underlays over existing
documented order, with normal caller overlays and no provider-availability released dependencies, with ordinary catalog/caller precedence and no
guarantee. Check default-profile resolution in the assembled final configuration. provider-availability guarantee. Check default-profile resolution in the assembled
final configuration. Complete all work without intermediate review or publication
gates, as required by the single-sprint objective.
- Add focused offline tests with controlled transports and injected clients for - Add focused offline tests with controlled transports and injected clients for
selection, rejected/absent credentials, applicability/support, transport/malformed selection, rejected/absent credentials, applicability/support, transport/malformed
responses, content/JSON/schema facts, cancellation, shared inference capacity, responses, content/JSON/schema facts, cancellation, shared inference capacity,
@@ -540,26 +567,23 @@ scope excludes application CLI/service work, readiness policy, continuous monito
model-status tests, batch orchestration, customizable test language, cost estimates, model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation. and guarantees about later generation.
## Remaining Decisions And Delivery Dependencies ## Remaining Implementation Details
The operation, routing precedence, report/error boundary, initial adapters, The operation, routing precedence, report/error boundary, initial adapters,
retry/repair policy, and catalog delivery direction are settled. The remaining retry/repair policy, default profiles, injected-client scope, and single-sprint
items should be resolved in the focused implementation plan: delivery approach are settled. The remaining items should be resolved in the
focused implementation plan:
1. **Configuration representation.** Finalize the Go convenience types and exact 1. **Configuration representation.** Finalize the Go convenience types and exact
endpoint fields, including omission/null/overlay behavior and interpretation of endpoint fields, including omission/null/overlay behavior and interpretation of
configured default-profile references at construction versus selection. Preserve configured default-profile references at construction versus selection. Preserve
final-value ownership and verify shipped defaults in assembled configuration; final-value ownership and verify shipped defaults in assembled configuration;
do not introduce source-history validation or change ordinary profile selection. do not introduce source-history validation or change ordinary profile selection.
2. **Explicit diagnostic transport for injected clients.** Decide whether the first 2. **Default metadata composition.** Choose the underlay representation and placement
delivery exposes a separate diagnostic HTTP-client option or reports HTTP that lets later catalog metadata and caller overlays replace the defaults while
diagnostics unavailable for injected-client engines until such an option exists. respecting catalog selection. Keep released dependencies usable throughout;
Inference remains available through `LLMClient` in either case. No hidden there is no external catalog coordination gate.
transport fallback or new required client methods are permitted. 3. **Internal implementation.** Choose private content and schema, the narrow evidence
3. **Catalog coordination.** Obtain the RakestrawHome maintainer's default-profile
choice and select compatible loader/catalog versions in the documented release
order. These are delivery dependencies, not reasons to reopen routing policy.
4. **Implementation details.** Choose private content and schema, the narrow evidence
handoff, and content-free classifications for internal faults that prevent a handoff, and content-free classifications for internal faults that prevent a
meaningful report. Define ownership and copying for reports and request values. meaningful report. Define ownership and copying for reports and request values.
Verify the state matrix, timeout/cancellation distinction, and redirect evidence Verify the state matrix, timeout/cancellation distinction, and redirect evidence