Make smoke test delivery self-contained in one sprint
This commit is contained in:
@@ -5,7 +5,7 @@ details are listed below. Implementation has not started.
|
||||
|
||||
This roadmap replaces the earlier capability proposal and incorporates the
|
||||
subsequent discussion. It defines the target feature, not a staged implementation
|
||||
plan or current public API. Remaining questions appear under Remaining Decisions.
|
||||
plan or current public API. Remaining implementation details are listed at the end.
|
||||
|
||||
## Purpose And Ownership
|
||||
|
||||
@@ -31,6 +31,25 @@ A reachable endpoint can reject credentials, and a transport failure can leave
|
||||
acceptance undetermined. Authentication is inapplicable on the maintainer's
|
||||
credential-free RakestrawHome deployment; that is neither an error nor a failure.
|
||||
|
||||
## Single-Sprint Delivery Objective
|
||||
|
||||
The staged implementation must be completable from start to finish in one sprint,
|
||||
including built-in defaults, tests, and documentation. It must not require
|
||||
intermediate human-review approvals, upstream releases, external publication, or
|
||||
live-provider access to progress or establish completion. Automated checks and
|
||||
focused local commits remain part of ordinary implementation.
|
||||
|
||||
Use the currently released catalog dependencies and PromptKit-owned diagnostic
|
||||
metadata underlays for the initial delivery. This supersedes the earlier
|
||||
loader-release/catalog-release/adoption sequence. The feature must be complete
|
||||
and usable without unpublished dependencies, committed workspace files, or local
|
||||
filesystem replacements. Any later move of metadata into catalogs is optional
|
||||
follow-up work, not an unfinished stage or a prerequisite for this sprint.
|
||||
|
||||
Normal release preparation and publication follow implementation separately;
|
||||
this objective does not bypass the repository's release process or authorize
|
||||
publication during the sprint.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
|
||||
@@ -354,6 +373,7 @@ backends:
|
||||
kind: openrouter_key
|
||||
rakestrawhome:
|
||||
smoke_tests:
|
||||
default_profile: rakestrawhome-gemma-4-31b
|
||||
authentication:
|
||||
kind: none
|
||||
```
|
||||
@@ -386,17 +406,19 @@ destination under the selected adapter's contract. It does not assert acceptance
|
||||
at the initial destination or at every redirect hop. These endpoint fields provide
|
||||
attribution without introducing a public redirect trace.
|
||||
|
||||
Deliver built-in metadata through coordinated compatible catalog releases; caller
|
||||
overlays remain available. First release loader support while retaining compatible
|
||||
existing catalog dependencies, then release catalogs containing the new metadata,
|
||||
then adopt those versions in PromptKit. Older strict loaders may reject new fields,
|
||||
so do not direct their consumers to upgrade catalogs independently. Select actual
|
||||
versions and compatibility notes during release preparation.
|
||||
Deliver initial built-in metadata through PromptKit-owned underlays composed with
|
||||
the currently released catalog assets. Arrange these defaults so catalog metadata
|
||||
and caller overlays can replace them through ordinary composition. Supply defaults
|
||||
for enabled maintained catalogs without reintroducing a backend whose catalog the
|
||||
caller excluded. Interpret and validate only the assembled final configuration.
|
||||
No catalog edits, dependency publication, or intervening PromptKit release are
|
||||
required for initial completion. Later catalog adoption, if pursued, must account
|
||||
for older strict loaders that reject new fields; it is outside this sprint.
|
||||
|
||||
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
|
||||
adapter, and RakestrawHome's `none` authentication setting. The RakestrawHome catalog
|
||||
maintainer must select its current default profile. Verify each shipped default
|
||||
resolves in the supported assembled effective configuration, including any
|
||||
adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
|
||||
setting. Verify each shipped default resolves in the supported assembled effective
|
||||
configuration, including any
|
||||
PromptKit-owned profiles. This is final-value integration validation, not a second
|
||||
validation pass over intermediate sources. A catalog default is replaceable metadata,
|
||||
not a promise of upstream availability, cheap inference, or a stable model choice.
|
||||
@@ -430,10 +452,13 @@ establish small output, a price estimate, or a cost ceiling.
|
||||
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
|
||||
new required methods. HTTP diagnostics must describe an explicitly selected
|
||||
transport, not secretly probe a different transport in an injected-client engine.
|
||||
The initial behavior is to report HTTP diagnostics unavailable unless
|
||||
that transport is supported or explicitly supplied; an additional injection option
|
||||
remains a delivery detail below. Do not silently fall back to a default HTTP client
|
||||
to diagnose an injected generation client.
|
||||
For injected-client engines, the initial delivery reports HTTP diagnostics
|
||||
unavailable and adds no separate diagnostic HTTP-client option. Inference
|
||||
diagnostics remain available through the injected client. Known inapplicability,
|
||||
such as authentication configured as `none`, can be reported without any transport.
|
||||
Do not silently fall back to a default HTTP client to diagnose an injected
|
||||
generation client. A future explicit diagnostic transport option remains possible
|
||||
but is not needed to complete this feature.
|
||||
|
||||
Return conventions:
|
||||
|
||||
@@ -516,9 +541,11 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
|
||||
and partial cancellation reporting through the internal evidence path while
|
||||
preserving existing client contracts. Specify destination/credential attribution
|
||||
without exposing headers or a redirect trace.
|
||||
- Supply agreed built-in metadata through compatible catalog releases in the
|
||||
documented order, with normal caller overlays and no provider-availability
|
||||
guarantee. Check default-profile resolution in the assembled final configuration.
|
||||
- Supply agreed built-in metadata through PromptKit-owned underlays over existing
|
||||
released dependencies, with ordinary catalog/caller precedence and no
|
||||
provider-availability guarantee. Check default-profile resolution in the assembled
|
||||
final configuration. Complete all work without intermediate review or publication
|
||||
gates, as required by the single-sprint objective.
|
||||
- Add focused offline tests with controlled transports and injected clients for
|
||||
selection, rejected/absent credentials, applicability/support, transport/malformed
|
||||
responses, content/JSON/schema facts, cancellation, shared inference capacity,
|
||||
@@ -540,26 +567,23 @@ scope excludes application CLI/service work, readiness policy, continuous monito
|
||||
model-status tests, batch orchestration, customizable test language, cost estimates,
|
||||
and guarantees about later generation.
|
||||
|
||||
## Remaining Decisions And Delivery Dependencies
|
||||
## Remaining Implementation Details
|
||||
|
||||
The operation, routing precedence, report/error boundary, initial adapters,
|
||||
retry/repair policy, and catalog delivery direction are settled. The remaining
|
||||
items should be resolved in the focused implementation plan:
|
||||
retry/repair policy, default profiles, injected-client scope, and single-sprint
|
||||
delivery approach are settled. The remaining items should be resolved in the
|
||||
focused implementation plan:
|
||||
|
||||
1. **Configuration representation.** Finalize the Go convenience types and exact
|
||||
endpoint fields, including omission/null/overlay behavior and interpretation of
|
||||
configured default-profile references at construction versus selection. Preserve
|
||||
final-value ownership and verify shipped defaults in assembled configuration;
|
||||
do not introduce source-history validation or change ordinary profile selection.
|
||||
2. **Explicit diagnostic transport for injected clients.** Decide whether the first
|
||||
delivery exposes a separate diagnostic HTTP-client option or reports HTTP
|
||||
diagnostics unavailable for injected-client engines until such an option exists.
|
||||
Inference remains available through `LLMClient` in either case. No hidden
|
||||
transport fallback or new required client methods are permitted.
|
||||
3. **Catalog coordination.** Obtain the RakestrawHome maintainer's default-profile
|
||||
choice and select compatible loader/catalog versions in the documented release
|
||||
order. These are delivery dependencies, not reasons to reopen routing policy.
|
||||
4. **Implementation details.** Choose private content and schema, the narrow evidence
|
||||
2. **Default metadata composition.** Choose the underlay representation and placement
|
||||
that lets later catalog metadata and caller overlays replace the defaults while
|
||||
respecting catalog selection. Keep released dependencies usable throughout;
|
||||
there is no external catalog coordination gate.
|
||||
3. **Internal implementation.** Choose private content and schema, the narrow evidence
|
||||
handoff, and content-free classifications for internal faults that prevent a
|
||||
meaningful report. Define ownership and copying for reports and request values.
|
||||
Verify the state matrix, timeout/cancellation distinction, and redirect evidence
|
||||
|
||||
Reference in New Issue
Block a user