eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Finalize smoke test roadmap decisions and evidence contracts

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-13 14:42:24 +00:00
parent f1f72dfe8f
commit 52cd94fcd1
1 changed files with 168 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files

279
docs/roadmap/smoke-tests.md
View File

@@ -1,11 +1,11 @@
# Explicit Smoke Test Diagnostics # Explicit Smoke Test Diagnostics
Status: purpose and ownership boundary accepted; the concrete API and delivery Status: feature design and delivery direction accepted; remaining implementation
proposals below remain subject to review. Implementation has not started. details are listed below. Implementation has not started.
This roadmap replaces the earlier capability proposal and incorporates the This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. Significant remaining choices appear under Open Decisions. plan or current public API. Remaining questions appear under Remaining Decisions.
## Purpose And Ownership ## Purpose And Ownership
@@ -52,7 +52,7 @@ ConfigKit owns loading and composition; PromptKit interprets and validates final
effective values. Provenance stays available for explicit inspection and is not effective values. Provenance stays available for explicit inspection and is not
an execution-time resolution or validation mechanism. an execution-time resolution or validation mechanism.
## Proposed Public Operation And Request ## Public Operation And Request
One synchronous call runs one independently selected operation: One synchronous call runs one independently selected operation:
@@ -117,11 +117,23 @@ does not fall back to a backend default. Missing backend/default selection is an
error when inference needs it, not a reason to substitute an arbitrary profile. error when inference needs it, not a reason to substitute an arbitrary profile.
Backend-only probes require neither a default profile nor unrelated prompt data. Backend-only probes require neither a default profile nor unrelated prompt data.
Proposed routing: when a prompt or profile is selected, the effective profile Whenever inference selects a profile, that profile supplies the resolved endpoint,
supplies the backend identity and endpoint. Otherwise `BackendID` selects the credential settings, and backend/capacity identity, subject to ordinary execution
backend. If both are provided, profile selection wins, including over a different overrides. This applies equally to explicit profiles, prompt defaults, and
backend hint; report the actual choice. This is proposed precedence, not an backend-default profiles. A supplied `BackendID` is a selection fallback, not a
accepted new prohibition on selector combinations; see Open Decisions. constraint or a request to rebind the selected profile's backend identity.
For example, if backend A names a default profile that resolves to backend B,
inference uses B's resolved target and capacity identity. If that default profile
is endpoint-only, inference retains an empty backend identity. Report A as the
default-selection source and B (or the empty identity) as the effective backend.
No recursive backend-default lookup or mismatch rejection is introduced.
For backend-only probes, an explicit or prompt-selected profile likewise supplies
the target and its backend's diagnostic metadata. Otherwise `BackendID` selects
the backend directly, without consulting its default inference profile. The
profile wins when both selectors are supplied, even when they name different
backends.
An endpoint-only profile remains usable for inference and reachability without An endpoint-only profile remains usable for inference and reachability without
an inferred backend identity or provider-specific authentication adapter. an inferred backend identity or provider-specific authentication adapter.
@@ -130,7 +142,7 @@ Backend-only probes can resolve profile endpoint/credential metadata without
requiring its model. Prompt selection uses captured identity/default-profile requiring its model. Prompt selection uses captured identity/default-profile
metadata without rendering or loading the application's artifacts or schemas. metadata without rendering or loading the application's artifacts or schemas.
## Proposed Report Shapes ## Report Shapes
Use factual Boolean observations with presence, rather than a caller-configurable Use factual Boolean observations with presence, rather than a caller-configurable
list of criteria. These Go-facing values need no stable JSON contract initially. list of criteria. These Go-facing values need no stable JSON contract initially.
@@ -141,7 +153,7 @@ type SmokeTestReport struct {
Supported bool Supported bool
Applicable *bool Applicable *bool
Completed bool Completed bool
Reason string Reason SmokeTestReason
Target SmokeTestTarget Target SmokeTestTarget
Facts SmokeTestFacts Facts SmokeTestFacts
Timing SmokeTestTiming Timing SmokeTestTiming
@@ -152,8 +164,9 @@ type SmokeTestReport struct {
type SmokeTestTarget struct { type SmokeTestTarget struct {
BackendID, ProfileID, PromptID, PromptVersion string BackendID, ProfileID, PromptID, PromptVersion string
DefaultSourceBackendID string // backend whose default selected the profile, if any
SelectionSource string // "backend_default", "profile", "prompt", or "backend" SelectionSource string // "backend_default", "profile", "prompt", or "backend"
Endpoint string // effective diagnostic destination Endpoint string // initially resolved diagnostic destination
EffectiveSettings ExecutionTarget EffectiveSettings ExecutionTarget
SettingsPresence ExecutionTargetPresence SettingsPresence ExecutionTargetPresence
} }
@@ -176,39 +189,60 @@ type SmokeTestTiming struct {
Duration time.Duration Duration time.Duration
CapacityWait *time.Duration CapacityWait *time.Duration
RequestDuration *time.Duration RequestDuration *time.Duration
RetryDelay *time.Duration
EvaluationDuration *time.Duration EvaluationDuration *time.Duration
} }
type SmokeTestAttempt struct { type SmokeTestAttempt struct {
Dispatched bool Dispatched bool
Endpoint string InitialEndpoint string // resolved destination before redirects
Endpoint string // last observed request destination; facts refer to this destination
HTTPStatus *int HTTPStatus *int
Facts SmokeTestFacts Facts SmokeTestFacts
Timing SmokeTestTiming Timing SmokeTestTiming
Failure ExecutionFailureKind Failure ExecutionFailureKind
Usage ObservedTokenUsage Usage ObservedTokenUsage
} }
type SmokeTestReason string
const (
SmokeTestReasonNone SmokeTestReason = ""
SmokeTestReasonAuthenticationNotApplicable SmokeTestReason = "authentication_not_applicable"
SmokeTestReasonProbeNotConfigured SmokeTestReason = "probe_not_configured"
SmokeTestReasonHTTPDiagnosticsUnavailable SmokeTestReason = "http_diagnostics_unavailable"
SmokeTestReasonRequiredCredentialMissing SmokeTestReason = "required_credential_missing"
SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized"
)
``` ```
`Supported` records whether an implementation is available for the target and `Supported` records whether an implementation is available for the target and
selected client. `Applicable=false` records known inapplicability; nil leaves selected client. `Applicable=false` records known inapplicability; nil leaves
applicability undetermined, for example when no authentication probe is configured. applicability undetermined, for example when no authentication probe is configured.
Missing probe metadata does not imply that a provider accepts anonymous access. Missing probe metadata does not imply that a provider accepts anonymous access.
`Reason` is a short factual code such as `authentication_not_applicable`, `Reason` is a named factual code, not application severity or free-form diagnostic
`probe_not_configured`, or `http_diagnostics_unavailable`, not an application text. The constants distinguish, respectively, no additional reason, explicitly
severity or verdict. inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics
for the selected client, a required credential missing before inference dispatch,
and an authentication response whose documented meaning cannot be established.
HTTP status and `Failure` classify ordinary rejection and transport outcomes;
they do not need duplicate reason codes.
`Completed` means the diagnostic concluded, not that checks passed. A rejected, `Completed` means the diagnostic concluded, not that checks passed. A rejected,
failed, unsupported, or inapplicable diagnostic can conclude with nil Go error. failed, unsupported, or inapplicable diagnostic can conclude with nil Go error.
Caller cancellation before completion leaves it false and returns available facts. Caller cancellation before completion leaves it false and returns available facts.
**A nil Go error means the diagnostic concluded, not that the tested operation
succeeded.** Make this distinction prominent in GoDoc and consumer examples.
A fact is true or false only when established; nil means not evaluated, A fact is true or false only when established; nil means not evaluated,
undetermined, or inapplicable. Operation metadata and failure classification undetermined, or inapplicable. Operation metadata and failure classification
provide context. In particular: provide context. In particular:
- `CredentialsSupplied`: a nonempty credential was supplied with the dispatched - `CredentialsSupplied`: a nonempty credential was attached to the observed
request, not merely named in configuration. No dispatch leaves this fact absent. dispatched request for the attempt's `Endpoint`, not merely named in configuration
For injected clients, do not claim upstream transmission PromptKit cannot observe. or attached to an earlier redirect hop. No observed request leaves this fact
absent. For injected clients, do not claim upstream transmission PromptKit cannot
observe. Observation at the transport boundary is not proof of wire transmission.
- `EndpointReachable`: an HTTP response was received from the tested destination, - `EndpointReachable`: an HTTP response was received from the tested destination,
regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation
ending without a response yields false. An injected generation result alone ending without a response yields false. An injected generation result alone
@@ -217,7 +251,8 @@ provide context. In particular:
or rejection. A rejection yields false; transport failure, server error, or an or rejection. A rejection yields false; transport failure, server error, or an
uninterpretable response can leave it nil. Acceptance does not certify balance, uninterpretable response can leave it nil. Acceptance does not certify balance,
model permissions, or future access. Absence of credentials is not an invalid-key model permissions, or future access. Absence of credentials is not an invalid-key
assertion. assertion: a documented 401 with `CredentialsSupplied=false` records rejection
of an unauthenticated request, not rejection of a supplied key.
- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a - `ResponseReceived`: an HTTP response was obtained for built-in operations, or a
response was returned by the injected generation boundary. A non-success or response was returned by the injected generation boundary. A non-success or
malformed HTTP response still counts as received. malformed HTTP response still counts as received.
@@ -235,13 +270,24 @@ provide context. In particular:
Known inapplicable authentication performs no dispatch and leaves request and Known inapplicable authentication performs no dispatch and leaves request and
acceptance facts absent. Unknown measurements remain unknown, including phase acceptance facts absent. Unknown measurements remain unknown, including phase
measurements for unperformed work. Total duration includes waiting and evaluation; measurements for unperformed work. Report-level capacity wait, request duration,
request duration excludes capacity waiting. Timings impose no application latency evaluation duration, and retry delay sum the corresponding measured work across
threshold or universal deadline. the diagnostic. Request duration excludes capacity waiting, evaluation, and retry
backoff. Retry delay includes completed and interrupted backoff. Per-attempt
timings retain their own measurements; backoff is reported at the operation level.
Total duration is wall time, including selection and other orchestration overhead,
so it need not equal the phase sum. Timings impose no application latency threshold
or universal deadline.
If retries are enabled, top-level facts describe the terminal attempt and attempt If retries are enabled, top-level facts describe the most recent dispatched
records preserve earlier failures. Usage reuses presence-aware accounting without attempt. Cancellation during backoff or a later capacity wait does not replace
claiming complete provider billing. No facts are inferred from arbitrary error text. those facts with an undispatched attempt's empty facts. The operation's `Failure`
and `Completed` still describe the interruption; earlier attempt records retain
their own outcomes. A canceled capacity acquisition can add a `Dispatched=false`
record, but no synthetic attempt is added for cancellation during backoff. With
no dispatched attempt, request facts remain absent. Usage reuses presence-aware
accounting without claiming complete provider billing. No facts are inferred from
arbitrary error text.
`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt `Dispatched` means the HTTP/client boundary was invoked, as in existing attempt
observations; it is not proof that packets were sent or billable work occurred. observations; it is not proof that packets were sent or billable work occurred.
@@ -257,7 +303,7 @@ for unrestricted logging.
### Reachability ### Reachability
Propose one unauthenticated HTTP GET to the configured diagnostic endpoint, or Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or
the effective base endpoint when no separate endpoint is supplied. Any HTTP the effective base endpoint when no separate endpoint is supplied. Any HTTP
response establishes reachability, including 401, 404, or 405. It does not establish response establishes reachability, including 401, 404, or 405. It does not establish
that the completion route, credentials, or a model work. Authentication is tested that the completion route, credentials, or a model work. Authentication is tested
@@ -267,7 +313,7 @@ DNS/TLS diagnosis.
### Authentication ### Authentication
Metadata names a supported provider adapter or explicitly marks authentication Metadata names a supported provider adapter or explicitly marks authentication
inapplicable. The proposed first adapter is `openrouter_key`, using GET on the inapplicable. The first adapter is `openrouter_key`, using GET on the
selected base endpoint's `key` path with ordinary bearer credential resolution. selected base endpoint's `key` path with ordinary bearer credential resolution.
OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key) OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key)
documents this authenticated operation and its 200 and 401 responses. Recognize documents this authenticated operation and its 200 and 401 responses. Recognize
@@ -292,7 +338,7 @@ clients and providers necessarily receive the content they execute: privacy here
is an API-exposure boundary, not secrecy from executing collaborators. There is is an API-exposure boundary, not secrecy from executing collaborators. There is
no persistent diagnostic prompt or session state. no persistent diagnostic prompt or session state.
### Proposed Backend Metadata ### Backend Metadata
Extend optional backend configuration through the final-value loader and the Extend optional backend configuration through the final-value loader and the
`BackendConfig`/`BackendSource` convenience. Illustrative shape: `BackendConfig`/`BackendSource` convenience. Illustrative shape:
@@ -321,14 +367,39 @@ presence/null and layering semantics. Validate syntax and recognized configurati
at the final handoff, without contacting providers or validating intermediate layers. at the final handoff, without contacting providers or validating intermediate layers.
Provider paths follow the selected endpoint override, not silently the catalog's Provider paths follow the selected endpoint override, not silently the catalog's
original endpoint. Explicitly configured diagnostic destinations can be different; original endpoint. Explicit absolute diagnostic endpoints are supported for separate
report the destination actually tested. Exact endpoint options remain a proposal health/auth routes and take precedence for their operation. Configuration stays
under Open Decisions. limited to named adapters and endpoint settings, not arbitrary methods or response
predicates. Concrete field names and presence/null rules remain to be specified.
Catalogs or caller overlays can supply backend-specific metadata. New catalog Reuse the configured HTTP client and its redirect handling, without adding custom
fields need a coordinated compatible loader release: older strict PromptKit credential forwarding. `Target.Endpoint` records the resolved starting destination.
loaders may reject them. A catalog default is replaceable metadata, not a promise Each HTTP attempt records `InitialEndpoint` and the last observed dispatched
of upstream availability or a stable model choice. request destination as `Endpoint`; its credential, reachability, response, and
acceptance facts describe that destination. If a redirect leads to a failed request,
do not attribute an earlier hop's response or credentials to the later destination.
When a client prevents observing the relevant request, leave the affected evidence
unknown. Injected generation alone supplies no HTTP destination evidence.
Authentication acceptance is interpreted from the response for the observed
destination under the selected adapter's contract. It does not assert acceptance
at the initial destination or at every redirect hop. These endpoint fields provide
attribution without introducing a public redirect trace.
Deliver built-in metadata through coordinated compatible catalog releases; caller
overlays remain available. First release loader support while retaining compatible
existing catalog dependencies, then release catalogs containing the new metadata,
then adopt those versions in PromptKit. Older strict loaders may reject new fields,
so do not direct their consumers to upgrade catalogs independently. Select actual
versions and compatibility notes during release preparation.
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
adapter, and RakestrawHome's `none` authentication setting. The RakestrawHome catalog
maintainer must select its current default profile. Verify each shipped default
resolves in the supported assembled effective configuration, including any
PromptKit-owned profiles. This is final-value integration validation, not a second
validation pass over intermediate sources. A catalog default is replaceable metadata,
not a promise of upstream availability, cheap inference, or a stable model choice.
## Execution And Error Boundary ## Execution And Error Boundary
@@ -337,22 +408,34 @@ validation, and usage accounting internally. A separate diagnostic orchestration
entry point may retain evidence that ordinary `Run` intentionally discards. entry point may retain evidence that ordinary `Run` intentionally discards.
Do not duplicate inference serialization or build a second inference stack. Do not duplicate inference serialization or build a second inference stack.
Add a narrow internal evidence path at the transport/coordinator boundary. Existing
execution events do not carry redirect destinations or dispatched credential
presence, and the coordinator can discard a completed response when cancellation
is observed before return. Capture available facts before that discard and retain
them for diagnostics. Keep ordinary execution's error/result contracts unchanged;
do not expand public execution events just to reconstruct smoke-test evidence.
Inference participates in existing engine-local admission and active backend Inference participates in existing engine-local admission and active backend
limits. HTTP-only probes need no generation permit or unrelated model selection. limits. HTTP-only probes need no generation permit or unrelated model selection.
Caller context and configured timeouts govern operations. The proposed inference Caller context and configured timeouts govern operations. Inference defaults to
baseline is a single attempt, no structural repair, and otherwise the selected one attempt: nil `TransientRetries` means zero for this operation rather than
profile plus explicit overrides. Retry choices below remain proposals; ordinary inheritance from the engine retry budget. Positive values reuse existing built-in
inference status retries and retain attempt records; injected generation remains
single-dispatch. Reachability and authentication remain single-attempt regardless
of a valid inference retry setting. Structural repair is excluded from the initial
procedure. Otherwise use the selected profile plus explicit overrides. Ordinary
engine defaults and repair behavior do not change. Small test content does not engine defaults and repair behavior do not change. Small test content does not
establish small output, a price estimate, or a cost ceiling. establish small output, a price estimate, or a cost ceiling.
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
new required methods. HTTP diagnostics must describe an explicitly selected new required methods. HTTP diagnostics must describe an explicitly selected
transport, not secretly probe a different transport in an injected-client engine. transport, not secretly probe a different transport in an injected-client engine.
The proposed initial behavior is to report HTTP diagnostics unavailable unless The initial behavior is to report HTTP diagnostics unavailable unless
that transport is supported or explicitly supplied; an additional injection option that transport is supported or explicitly supplied; an additional injection option
is a delivery choice below. remains a delivery detail below. Do not silently fall back to a default HTTP client
to diagnose an injected generation client.
Proposed return conventions: Return conventions:
- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections - Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections
return ordinary public errors before probe work. Unsupported capability is not return ordinary public errors before probe work. Unsupported capability is not
@@ -399,11 +482,20 @@ Expected report examples:
| Operation/outcome | Observations | Go error | | Operation/outcome | Observations | Go error |
| --- | --- | --- | | --- | --- | --- |
| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil | | Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil |
| RakestrawHome authentication | Applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil | | RakestrawHome authentication | Supported true, applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil | | Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil |
| HTTP diagnostics unavailable for injected client | Supported false, applicability unknown, completed, `http_diagnostics_unavailable`; no dispatch | nil |
| Authentication without credentials receives documented 401 | Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request | nil |
| Required credential absent before inference | Supported/applicable/completed; `required_credential_missing`, failure `invalid_request`; no dispatch and no request facts | nil |
| Authentication receives uninterpretable HTTP 200 | Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; `authentication_response_unrecognized`, failure `malformed_response` | nil |
| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil | | Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil |
| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil | | Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil |
| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil | | Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil |
| Configured request timeout while caller context remains active | Supported/applicable/completed; failure `deadline`; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts | nil |
| Local capacity exhausted | Supported/applicable/completed; failure `capacity`; no dispatch or request facts | nil |
| Caller cancels during initial capacity wait | Supported/applicable, completed false; failure `canceled`, measured capacity wait, undispatched attempt; request facts absent | `context.Canceled` identity |
| Caller deadline expires during a dispatched request | Supported/applicable, completed false; failure `deadline`; retain available attempt evidence | `context.DeadlineExceeded` identity |
| Caller cancels during retry backoff after HTTP 503 | Completed false, failure `canceled`; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt | `context.Canceled` identity |
| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error | | Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error |
Consumers decide whether any of these findings warrant success, a warning, or a Consumers decide whether any of these findings warrant success, a warning, or a
@@ -413,7 +505,7 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
## Work Needed And Target Completion ## Work Needed And Target Completion
- Add the public request/report operation and GoDoc, keeping business logic internal. - Add the public request/report operation and GoDoc, keeping business logic internal.
Resolve significant Open Decisions before writing the staged implementation plan. Resolve the remaining delivery details when writing the staged implementation plan.
- Extend final backend metadata interpretation and captured lookup for capabilities - Extend final backend metadata interpretation and captured lookup for capabilities
and defaults. Preserve explicit configuration inspection without exposing test and defaults. Preserve explicit configuration inspection without exposing test
content or using source history for validation. content or using source history for validation.
@@ -421,15 +513,21 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
capacity, generation, decoding, and structural evaluation without preparing capacity, generation, decoding, and structural evaluation without preparing
application inputs or messages. application inputs or messages.
- Add narrow HTTP probes, credential handling, evidence retention, factual results, - Add narrow HTTP probes, credential handling, evidence retention, factual results,
and partial cancellation reporting while preserving existing client contracts. and partial cancellation reporting through the internal evidence path while
- Supply agreed built-in metadata through compatible catalog releases or PromptKit preserving existing client contracts. Specify destination/credential attribution
underlays, with normal caller overlays and no provider-availability guarantee. without exposing headers or a redirect trace.
- Supply agreed built-in metadata through compatible catalog releases in the
documented order, with normal caller overlays and no provider-availability
guarantee. Check default-profile resolution in the assembled final configuration.
- Add focused offline tests with controlled transports and injected clients for - Add focused offline tests with controlled transports and injected clients for
selection, rejected/absent credentials, applicability/support, transport/malformed selection, rejected/absent credentials, applicability/support, transport/malformed
responses, content/JSON/schema facts, cancellation, shared inference capacity, responses, content/JSON/schema facts, cancellation, shared inference capacity,
and credential/private-content exposure. Assign distinct test owners; do not and credential/private-content exposure. Assign distinct test owners; do not
repeat existing transport, capacity, or validation matrices or snapshot private repeat existing transport, capacity, or validation matrices or snapshot private
test language. test language.
- Include the report-state matrix, cross-backend and endpoint-only default profiles,
redirect attribution, cancellation after a response, and interrupted retry waits
in focused contract coverage. Reuse existing lower-level checks where sufficient.
- Update canonical consumer, backend-format, integration, and internal documentation - Update canonical consumer, backend-format, integration, and internal documentation
when implementation lands, with release notes explaining the optional addition. when implementation lands, with release notes explaining the optional addition.
Live diagnostics are explicit consumer operations, not default tests or new Live diagnostics are explicit consumer operations, not default tests or new
@@ -442,69 +540,28 @@ scope excludes application CLI/service work, readiness policy, continuous monito
model-status tests, batch orchestration, customizable test language, cost estimates, model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation. and guarantees about later generation.
## Open Decisions ## Remaining Decisions And Delivery Dependencies
### 1. Backend Selection Alongside A Profile The operation, routing precedence, report/error boundary, initial adapters,
retry/repair policy, and catalog delivery direction are settled. The remaining
items should be resolved in the focused implementation plan:
**Recommendation:** adopt the proposed precedence: explicit or prompt-selected 1. **Configuration representation.** Finalize the Go convenience types and exact
profiles supply routing; `BackendID` supplies backend/default selection otherwise. endpoint fields, including omission/null/overlay behavior and interpretation of
Report the effective choice. This follows familiar profile precedence without configured default-profile references at construction versus selection. Preserve
inventing a mismatch prohibition or rebinding capacity identity. final-value ownership and verify shipped defaults in assembled configuration;
do not introduce source-history validation or change ordinary profile selection.
**Alternative:** treat `BackendID` as a constraint when combined with profile/prompt 2. **Explicit diagnostic transport for injected clients.** Decide whether the first
selection and reject mismatches. That catches mistakes and makes every selector delivery exposes a separate diagnostic HTTP-client option or reports HTTP
operative, but adds a diagnostic-specific restriction and makes profile changes diagnostics unavailable for injected-client engines until such an option exists.
more disruptive. Choose it only if consumers want that constraint. Inference remains available through `LLMClient` in either case. No hidden
transport fallback or new required client methods are permitted.
### 2. Initial HTTP Adapters, Endpoints, Redirects, And Injection 3. **Catalog coordination.** Obtain the RakestrawHome maintainer's default-profile
choice and select compatible loader/catalog versions in the documented release
**Recommendation:** start with generic unauthenticated HTTP-response reachability, order. These are delivery dependencies, not reasons to reopen routing policy.
the OpenRouter current-key adapter, and explicit `none` authentication. Derive the 4. **Implementation details.** Choose private content and schema, the narrow evidence
key route from the effective base, while allowing explicitly configured absolute handoff, and content-free classifications for internal faults that prevent a
diagnostic endpoints for separate health/auth routes. Reuse the configured HTTP meaningful report. Define ownership and copying for reports and request values.
client and its standard redirect handling, record the final observed destination, Verify the state matrix, timeout/cancellation distinction, and redirect evidence
and avoid custom redirect credential forwarding. A separate diagnostic host is against actual transport behavior before finalizing GoDoc. Keep public execution
explicit configuration, not an inference from an overridden completion endpoint. events and ordinary result/error semantics unchanged.
Keep configuration limited to named adapters and endpoints, not arbitrary methods
or response predicates. For injected generation clients, initially report HTTP
diagnostics unavailable unless a separate transport is explicitly supported.
This serves the maintained backends with a small adapter surface. Planning must
settle concrete endpoint fields and how request/credential observations reflect
redirects: a final response is not proof of acceptance at every redirect hop.
**Alternative:** initially support base-relative routes only, with no separate
host or additional transport option. That simplifies routing but excludes dedicated
diagnostic hosts. Another viable addition is an explicit diagnostic HTTP-client
option for injected-client engines; it improves custom-client coverage but adds a
supported injection surface. Neither choice requires expanding `LLMClient`.
### 3. Retry And Repair Controls
**Recommendation:** retain `TransientRetries`, with nil meaning zero for this
operation. Positive values reuse built-in inference status retries and preserve
attempt records; injected generation remains single-dispatch. Keep reachability
and authentication single-attempt and structural repair outside the initial test
procedure. This gives direct default evidence while preserving caller choice for
transient inference failures, without changing ordinary execution policy.
**Alternative:** omit the retry field and keep every diagnostic single-attempt.
That is the smallest API, but applications must repeat calls themselves. Inheriting
the engine retry budget is also viable: it standardizes defaults but makes cost
and recovery less obvious. Automatic structural repair would test recovery as
well as first-response conformance and warrants a separate explicit decision.
### 4. Built-In Metadata Delivery And Default Profiles
**Recommendation:** coordinate compatible catalog releases containing capability
metadata and backend defaults. Use `promptkit-openrouter-standard` for OpenRouter;
have the RakestrawHome catalog maintainer select an explicit current profile for
its diagnostic default. Include OpenRouter's adapter and RakestrawHome's `none`
setting. Caller overlays remain available. These are replaceable defaults, not
new promises about model names, availability, or cheap inference.
**Alternative:** contribute metadata through PromptKit-owned underlays initially,
or require caller configuration. Underlays avoid waiting for catalog releases but
move provider configuration into core. Caller-only metadata avoids coupled releases
but loses much of the convenience motivating adoption. No new stable profile-name
promise is necessary for the RakestrawHome default.