Finalize smoke test roadmap decisions and evidence contracts
This commit is contained in:
@@ -1,11 +1,11 @@
|
||||
# Explicit Smoke Test Diagnostics
|
||||
|
||||
Status: purpose and ownership boundary accepted; the concrete API and delivery
|
||||
proposals below remain subject to review. Implementation has not started.
|
||||
Status: feature design and delivery direction accepted; remaining implementation
|
||||
details are listed below. Implementation has not started.
|
||||
|
||||
This roadmap replaces the earlier capability proposal and incorporates the
|
||||
subsequent discussion. It defines the target feature, not a staged implementation
|
||||
plan or current public API. Significant remaining choices appear under Open Decisions.
|
||||
plan or current public API. Remaining questions appear under Remaining Decisions.
|
||||
|
||||
## Purpose And Ownership
|
||||
|
||||
@@ -52,7 +52,7 @@ ConfigKit owns loading and composition; PromptKit interprets and validates final
|
||||
effective values. Provenance stays available for explicit inspection and is not
|
||||
an execution-time resolution or validation mechanism.
|
||||
|
||||
## Proposed Public Operation And Request
|
||||
## Public Operation And Request
|
||||
|
||||
One synchronous call runs one independently selected operation:
|
||||
|
||||
@@ -117,11 +117,23 @@ does not fall back to a backend default. Missing backend/default selection is an
|
||||
error when inference needs it, not a reason to substitute an arbitrary profile.
|
||||
Backend-only probes require neither a default profile nor unrelated prompt data.
|
||||
|
||||
Proposed routing: when a prompt or profile is selected, the effective profile
|
||||
supplies the backend identity and endpoint. Otherwise `BackendID` selects the
|
||||
backend. If both are provided, profile selection wins, including over a different
|
||||
backend hint; report the actual choice. This is proposed precedence, not an
|
||||
accepted new prohibition on selector combinations; see Open Decisions.
|
||||
Whenever inference selects a profile, that profile supplies the resolved endpoint,
|
||||
credential settings, and backend/capacity identity, subject to ordinary execution
|
||||
overrides. This applies equally to explicit profiles, prompt defaults, and
|
||||
backend-default profiles. A supplied `BackendID` is a selection fallback, not a
|
||||
constraint or a request to rebind the selected profile's backend identity.
|
||||
|
||||
For example, if backend A names a default profile that resolves to backend B,
|
||||
inference uses B's resolved target and capacity identity. If that default profile
|
||||
is endpoint-only, inference retains an empty backend identity. Report A as the
|
||||
default-selection source and B (or the empty identity) as the effective backend.
|
||||
No recursive backend-default lookup or mismatch rejection is introduced.
|
||||
|
||||
For backend-only probes, an explicit or prompt-selected profile likewise supplies
|
||||
the target and its backend's diagnostic metadata. Otherwise `BackendID` selects
|
||||
the backend directly, without consulting its default inference profile. The
|
||||
profile wins when both selectors are supplied, even when they name different
|
||||
backends.
|
||||
|
||||
An endpoint-only profile remains usable for inference and reachability without
|
||||
an inferred backend identity or provider-specific authentication adapter.
|
||||
@@ -130,7 +142,7 @@ Backend-only probes can resolve profile endpoint/credential metadata without
|
||||
requiring its model. Prompt selection uses captured identity/default-profile
|
||||
metadata without rendering or loading the application's artifacts or schemas.
|
||||
|
||||
## Proposed Report Shapes
|
||||
## Report Shapes
|
||||
|
||||
Use factual Boolean observations with presence, rather than a caller-configurable
|
||||
list of criteria. These Go-facing values need no stable JSON contract initially.
|
||||
@@ -141,7 +153,7 @@ type SmokeTestReport struct {
|
||||
Supported bool
|
||||
Applicable *bool
|
||||
Completed bool
|
||||
Reason string
|
||||
Reason SmokeTestReason
|
||||
Target SmokeTestTarget
|
||||
Facts SmokeTestFacts
|
||||
Timing SmokeTestTiming
|
||||
@@ -152,8 +164,9 @@ type SmokeTestReport struct {
|
||||
|
||||
type SmokeTestTarget struct {
|
||||
BackendID, ProfileID, PromptID, PromptVersion string
|
||||
DefaultSourceBackendID string // backend whose default selected the profile, if any
|
||||
SelectionSource string // "backend_default", "profile", "prompt", or "backend"
|
||||
Endpoint string // effective diagnostic destination
|
||||
Endpoint string // initially resolved diagnostic destination
|
||||
EffectiveSettings ExecutionTarget
|
||||
SettingsPresence ExecutionTargetPresence
|
||||
}
|
||||
@@ -176,39 +189,60 @@ type SmokeTestTiming struct {
|
||||
Duration time.Duration
|
||||
CapacityWait *time.Duration
|
||||
RequestDuration *time.Duration
|
||||
RetryDelay *time.Duration
|
||||
EvaluationDuration *time.Duration
|
||||
}
|
||||
|
||||
type SmokeTestAttempt struct {
|
||||
Dispatched bool
|
||||
Endpoint string
|
||||
InitialEndpoint string // resolved destination before redirects
|
||||
Endpoint string // last observed request destination; facts refer to this destination
|
||||
HTTPStatus *int
|
||||
Facts SmokeTestFacts
|
||||
Timing SmokeTestTiming
|
||||
Failure ExecutionFailureKind
|
||||
Usage ObservedTokenUsage
|
||||
}
|
||||
|
||||
type SmokeTestReason string
|
||||
|
||||
const (
|
||||
SmokeTestReasonNone SmokeTestReason = ""
|
||||
SmokeTestReasonAuthenticationNotApplicable SmokeTestReason = "authentication_not_applicable"
|
||||
SmokeTestReasonProbeNotConfigured SmokeTestReason = "probe_not_configured"
|
||||
SmokeTestReasonHTTPDiagnosticsUnavailable SmokeTestReason = "http_diagnostics_unavailable"
|
||||
SmokeTestReasonRequiredCredentialMissing SmokeTestReason = "required_credential_missing"
|
||||
SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized"
|
||||
)
|
||||
```
|
||||
|
||||
`Supported` records whether an implementation is available for the target and
|
||||
selected client. `Applicable=false` records known inapplicability; nil leaves
|
||||
applicability undetermined, for example when no authentication probe is configured.
|
||||
Missing probe metadata does not imply that a provider accepts anonymous access.
|
||||
`Reason` is a short factual code such as `authentication_not_applicable`,
|
||||
`probe_not_configured`, or `http_diagnostics_unavailable`, not an application
|
||||
severity or verdict.
|
||||
`Reason` is a named factual code, not application severity or free-form diagnostic
|
||||
text. The constants distinguish, respectively, no additional reason, explicitly
|
||||
inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics
|
||||
for the selected client, a required credential missing before inference dispatch,
|
||||
and an authentication response whose documented meaning cannot be established.
|
||||
HTTP status and `Failure` classify ordinary rejection and transport outcomes;
|
||||
they do not need duplicate reason codes.
|
||||
|
||||
`Completed` means the diagnostic concluded, not that checks passed. A rejected,
|
||||
failed, unsupported, or inapplicable diagnostic can conclude with nil Go error.
|
||||
Caller cancellation before completion leaves it false and returns available facts.
|
||||
**A nil Go error means the diagnostic concluded, not that the tested operation
|
||||
succeeded.** Make this distinction prominent in GoDoc and consumer examples.
|
||||
|
||||
A fact is true or false only when established; nil means not evaluated,
|
||||
undetermined, or inapplicable. Operation metadata and failure classification
|
||||
provide context. In particular:
|
||||
|
||||
- `CredentialsSupplied`: a nonempty credential was supplied with the dispatched
|
||||
request, not merely named in configuration. No dispatch leaves this fact absent.
|
||||
For injected clients, do not claim upstream transmission PromptKit cannot observe.
|
||||
- `CredentialsSupplied`: a nonempty credential was attached to the observed
|
||||
dispatched request for the attempt's `Endpoint`, not merely named in configuration
|
||||
or attached to an earlier redirect hop. No observed request leaves this fact
|
||||
absent. For injected clients, do not claim upstream transmission PromptKit cannot
|
||||
observe. Observation at the transport boundary is not proof of wire transmission.
|
||||
- `EndpointReachable`: an HTTP response was received from the tested destination,
|
||||
regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation
|
||||
ending without a response yields false. An injected generation result alone
|
||||
@@ -217,7 +251,8 @@ provide context. In particular:
|
||||
or rejection. A rejection yields false; transport failure, server error, or an
|
||||
uninterpretable response can leave it nil. Acceptance does not certify balance,
|
||||
model permissions, or future access. Absence of credentials is not an invalid-key
|
||||
assertion.
|
||||
assertion: a documented 401 with `CredentialsSupplied=false` records rejection
|
||||
of an unauthenticated request, not rejection of a supplied key.
|
||||
- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a
|
||||
response was returned by the injected generation boundary. A non-success or
|
||||
malformed HTTP response still counts as received.
|
||||
@@ -235,13 +270,24 @@ provide context. In particular:
|
||||
|
||||
Known inapplicable authentication performs no dispatch and leaves request and
|
||||
acceptance facts absent. Unknown measurements remain unknown, including phase
|
||||
measurements for unperformed work. Total duration includes waiting and evaluation;
|
||||
request duration excludes capacity waiting. Timings impose no application latency
|
||||
threshold or universal deadline.
|
||||
measurements for unperformed work. Report-level capacity wait, request duration,
|
||||
evaluation duration, and retry delay sum the corresponding measured work across
|
||||
the diagnostic. Request duration excludes capacity waiting, evaluation, and retry
|
||||
backoff. Retry delay includes completed and interrupted backoff. Per-attempt
|
||||
timings retain their own measurements; backoff is reported at the operation level.
|
||||
Total duration is wall time, including selection and other orchestration overhead,
|
||||
so it need not equal the phase sum. Timings impose no application latency threshold
|
||||
or universal deadline.
|
||||
|
||||
If retries are enabled, top-level facts describe the terminal attempt and attempt
|
||||
records preserve earlier failures. Usage reuses presence-aware accounting without
|
||||
claiming complete provider billing. No facts are inferred from arbitrary error text.
|
||||
If retries are enabled, top-level facts describe the most recent dispatched
|
||||
attempt. Cancellation during backoff or a later capacity wait does not replace
|
||||
those facts with an undispatched attempt's empty facts. The operation's `Failure`
|
||||
and `Completed` still describe the interruption; earlier attempt records retain
|
||||
their own outcomes. A canceled capacity acquisition can add a `Dispatched=false`
|
||||
record, but no synthetic attempt is added for cancellation during backoff. With
|
||||
no dispatched attempt, request facts remain absent. Usage reuses presence-aware
|
||||
accounting without claiming complete provider billing. No facts are inferred from
|
||||
arbitrary error text.
|
||||
`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt
|
||||
observations; it is not proof that packets were sent or billable work occurred.
|
||||
|
||||
@@ -257,7 +303,7 @@ for unrestricted logging.
|
||||
|
||||
### Reachability
|
||||
|
||||
Propose one unauthenticated HTTP GET to the configured diagnostic endpoint, or
|
||||
Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or
|
||||
the effective base endpoint when no separate endpoint is supplied. Any HTTP
|
||||
response establishes reachability, including 401, 404, or 405. It does not establish
|
||||
that the completion route, credentials, or a model work. Authentication is tested
|
||||
@@ -267,7 +313,7 @@ DNS/TLS diagnosis.
|
||||
### Authentication
|
||||
|
||||
Metadata names a supported provider adapter or explicitly marks authentication
|
||||
inapplicable. The proposed first adapter is `openrouter_key`, using GET on the
|
||||
inapplicable. The first adapter is `openrouter_key`, using GET on the
|
||||
selected base endpoint's `key` path with ordinary bearer credential resolution.
|
||||
OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key)
|
||||
documents this authenticated operation and its 200 and 401 responses. Recognize
|
||||
@@ -292,7 +338,7 @@ clients and providers necessarily receive the content they execute: privacy here
|
||||
is an API-exposure boundary, not secrecy from executing collaborators. There is
|
||||
no persistent diagnostic prompt or session state.
|
||||
|
||||
### Proposed Backend Metadata
|
||||
### Backend Metadata
|
||||
|
||||
Extend optional backend configuration through the final-value loader and the
|
||||
`BackendConfig`/`BackendSource` convenience. Illustrative shape:
|
||||
@@ -321,14 +367,39 @@ presence/null and layering semantics. Validate syntax and recognized configurati
|
||||
at the final handoff, without contacting providers or validating intermediate layers.
|
||||
|
||||
Provider paths follow the selected endpoint override, not silently the catalog's
|
||||
original endpoint. Explicitly configured diagnostic destinations can be different;
|
||||
report the destination actually tested. Exact endpoint options remain a proposal
|
||||
under Open Decisions.
|
||||
original endpoint. Explicit absolute diagnostic endpoints are supported for separate
|
||||
health/auth routes and take precedence for their operation. Configuration stays
|
||||
limited to named adapters and endpoint settings, not arbitrary methods or response
|
||||
predicates. Concrete field names and presence/null rules remain to be specified.
|
||||
|
||||
Catalogs or caller overlays can supply backend-specific metadata. New catalog
|
||||
fields need a coordinated compatible loader release: older strict PromptKit
|
||||
loaders may reject them. A catalog default is replaceable metadata, not a promise
|
||||
of upstream availability or a stable model choice.
|
||||
Reuse the configured HTTP client and its redirect handling, without adding custom
|
||||
credential forwarding. `Target.Endpoint` records the resolved starting destination.
|
||||
Each HTTP attempt records `InitialEndpoint` and the last observed dispatched
|
||||
request destination as `Endpoint`; its credential, reachability, response, and
|
||||
acceptance facts describe that destination. If a redirect leads to a failed request,
|
||||
do not attribute an earlier hop's response or credentials to the later destination.
|
||||
When a client prevents observing the relevant request, leave the affected evidence
|
||||
unknown. Injected generation alone supplies no HTTP destination evidence.
|
||||
|
||||
Authentication acceptance is interpreted from the response for the observed
|
||||
destination under the selected adapter's contract. It does not assert acceptance
|
||||
at the initial destination or at every redirect hop. These endpoint fields provide
|
||||
attribution without introducing a public redirect trace.
|
||||
|
||||
Deliver built-in metadata through coordinated compatible catalog releases; caller
|
||||
overlays remain available. First release loader support while retaining compatible
|
||||
existing catalog dependencies, then release catalogs containing the new metadata,
|
||||
then adopt those versions in PromptKit. Older strict loaders may reject new fields,
|
||||
so do not direct their consumers to upgrade catalogs independently. Select actual
|
||||
versions and compatibility notes during release preparation.
|
||||
|
||||
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
|
||||
adapter, and RakestrawHome's `none` authentication setting. The RakestrawHome catalog
|
||||
maintainer must select its current default profile. Verify each shipped default
|
||||
resolves in the supported assembled effective configuration, including any
|
||||
PromptKit-owned profiles. This is final-value integration validation, not a second
|
||||
validation pass over intermediate sources. A catalog default is replaceable metadata,
|
||||
not a promise of upstream availability, cheap inference, or a stable model choice.
|
||||
|
||||
## Execution And Error Boundary
|
||||
|
||||
@@ -337,22 +408,34 @@ validation, and usage accounting internally. A separate diagnostic orchestration
|
||||
entry point may retain evidence that ordinary `Run` intentionally discards.
|
||||
Do not duplicate inference serialization or build a second inference stack.
|
||||
|
||||
Add a narrow internal evidence path at the transport/coordinator boundary. Existing
|
||||
execution events do not carry redirect destinations or dispatched credential
|
||||
presence, and the coordinator can discard a completed response when cancellation
|
||||
is observed before return. Capture available facts before that discard and retain
|
||||
them for diagnostics. Keep ordinary execution's error/result contracts unchanged;
|
||||
do not expand public execution events just to reconstruct smoke-test evidence.
|
||||
|
||||
Inference participates in existing engine-local admission and active backend
|
||||
limits. HTTP-only probes need no generation permit or unrelated model selection.
|
||||
Caller context and configured timeouts govern operations. The proposed inference
|
||||
baseline is a single attempt, no structural repair, and otherwise the selected
|
||||
profile plus explicit overrides. Retry choices below remain proposals; ordinary
|
||||
Caller context and configured timeouts govern operations. Inference defaults to
|
||||
one attempt: nil `TransientRetries` means zero for this operation rather than
|
||||
inheritance from the engine retry budget. Positive values reuse existing built-in
|
||||
inference status retries and retain attempt records; injected generation remains
|
||||
single-dispatch. Reachability and authentication remain single-attempt regardless
|
||||
of a valid inference retry setting. Structural repair is excluded from the initial
|
||||
procedure. Otherwise use the selected profile plus explicit overrides. Ordinary
|
||||
engine defaults and repair behavior do not change. Small test content does not
|
||||
establish small output, a price estimate, or a cost ceiling.
|
||||
|
||||
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
|
||||
new required methods. HTTP diagnostics must describe an explicitly selected
|
||||
transport, not secretly probe a different transport in an injected-client engine.
|
||||
The proposed initial behavior is to report HTTP diagnostics unavailable unless
|
||||
The initial behavior is to report HTTP diagnostics unavailable unless
|
||||
that transport is supported or explicitly supplied; an additional injection option
|
||||
is a delivery choice below.
|
||||
remains a delivery detail below. Do not silently fall back to a default HTTP client
|
||||
to diagnose an injected generation client.
|
||||
|
||||
Proposed return conventions:
|
||||
Return conventions:
|
||||
|
||||
- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections
|
||||
return ordinary public errors before probe work. Unsupported capability is not
|
||||
@@ -399,11 +482,20 @@ Expected report examples:
|
||||
| Operation/outcome | Observations | Go error |
|
||||
| --- | --- | --- |
|
||||
| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil |
|
||||
| RakestrawHome authentication | Applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
|
||||
| RakestrawHome authentication | Supported true, applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
|
||||
| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil |
|
||||
| HTTP diagnostics unavailable for injected client | Supported false, applicability unknown, completed, `http_diagnostics_unavailable`; no dispatch | nil |
|
||||
| Authentication without credentials receives documented 401 | Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request | nil |
|
||||
| Required credential absent before inference | Supported/applicable/completed; `required_credential_missing`, failure `invalid_request`; no dispatch and no request facts | nil |
|
||||
| Authentication receives uninterpretable HTTP 200 | Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; `authentication_response_unrecognized`, failure `malformed_response` | nil |
|
||||
| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil |
|
||||
| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil |
|
||||
| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil |
|
||||
| Configured request timeout while caller context remains active | Supported/applicable/completed; failure `deadline`; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts | nil |
|
||||
| Local capacity exhausted | Supported/applicable/completed; failure `capacity`; no dispatch or request facts | nil |
|
||||
| Caller cancels during initial capacity wait | Supported/applicable, completed false; failure `canceled`, measured capacity wait, undispatched attempt; request facts absent | `context.Canceled` identity |
|
||||
| Caller deadline expires during a dispatched request | Supported/applicable, completed false; failure `deadline`; retain available attempt evidence | `context.DeadlineExceeded` identity |
|
||||
| Caller cancels during retry backoff after HTTP 503 | Completed false, failure `canceled`; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt | `context.Canceled` identity |
|
||||
| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error |
|
||||
|
||||
Consumers decide whether any of these findings warrant success, a warning, or a
|
||||
@@ -413,7 +505,7 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
|
||||
## Work Needed And Target Completion
|
||||
|
||||
- Add the public request/report operation and GoDoc, keeping business logic internal.
|
||||
Resolve significant Open Decisions before writing the staged implementation plan.
|
||||
Resolve the remaining delivery details when writing the staged implementation plan.
|
||||
- Extend final backend metadata interpretation and captured lookup for capabilities
|
||||
and defaults. Preserve explicit configuration inspection without exposing test
|
||||
content or using source history for validation.
|
||||
@@ -421,15 +513,21 @@ scheduling, and any combination with `ValidateRequest` or normal execution.
|
||||
capacity, generation, decoding, and structural evaluation without preparing
|
||||
application inputs or messages.
|
||||
- Add narrow HTTP probes, credential handling, evidence retention, factual results,
|
||||
and partial cancellation reporting while preserving existing client contracts.
|
||||
- Supply agreed built-in metadata through compatible catalog releases or PromptKit
|
||||
underlays, with normal caller overlays and no provider-availability guarantee.
|
||||
and partial cancellation reporting through the internal evidence path while
|
||||
preserving existing client contracts. Specify destination/credential attribution
|
||||
without exposing headers or a redirect trace.
|
||||
- Supply agreed built-in metadata through compatible catalog releases in the
|
||||
documented order, with normal caller overlays and no provider-availability
|
||||
guarantee. Check default-profile resolution in the assembled final configuration.
|
||||
- Add focused offline tests with controlled transports and injected clients for
|
||||
selection, rejected/absent credentials, applicability/support, transport/malformed
|
||||
responses, content/JSON/schema facts, cancellation, shared inference capacity,
|
||||
and credential/private-content exposure. Assign distinct test owners; do not
|
||||
repeat existing transport, capacity, or validation matrices or snapshot private
|
||||
test language.
|
||||
- Include the report-state matrix, cross-backend and endpoint-only default profiles,
|
||||
redirect attribution, cancellation after a response, and interrupted retry waits
|
||||
in focused contract coverage. Reuse existing lower-level checks where sufficient.
|
||||
- Update canonical consumer, backend-format, integration, and internal documentation
|
||||
when implementation lands, with release notes explaining the optional addition.
|
||||
Live diagnostics are explicit consumer operations, not default tests or new
|
||||
@@ -442,69 +540,28 @@ scope excludes application CLI/service work, readiness policy, continuous monito
|
||||
model-status tests, batch orchestration, customizable test language, cost estimates,
|
||||
and guarantees about later generation.
|
||||
|
||||
## Open Decisions
|
||||
## Remaining Decisions And Delivery Dependencies
|
||||
|
||||
### 1. Backend Selection Alongside A Profile
|
||||
The operation, routing precedence, report/error boundary, initial adapters,
|
||||
retry/repair policy, and catalog delivery direction are settled. The remaining
|
||||
items should be resolved in the focused implementation plan:
|
||||
|
||||
**Recommendation:** adopt the proposed precedence: explicit or prompt-selected
|
||||
profiles supply routing; `BackendID` supplies backend/default selection otherwise.
|
||||
Report the effective choice. This follows familiar profile precedence without
|
||||
inventing a mismatch prohibition or rebinding capacity identity.
|
||||
|
||||
**Alternative:** treat `BackendID` as a constraint when combined with profile/prompt
|
||||
selection and reject mismatches. That catches mistakes and makes every selector
|
||||
operative, but adds a diagnostic-specific restriction and makes profile changes
|
||||
more disruptive. Choose it only if consumers want that constraint.
|
||||
|
||||
### 2. Initial HTTP Adapters, Endpoints, Redirects, And Injection
|
||||
|
||||
**Recommendation:** start with generic unauthenticated HTTP-response reachability,
|
||||
the OpenRouter current-key adapter, and explicit `none` authentication. Derive the
|
||||
key route from the effective base, while allowing explicitly configured absolute
|
||||
diagnostic endpoints for separate health/auth routes. Reuse the configured HTTP
|
||||
client and its standard redirect handling, record the final observed destination,
|
||||
and avoid custom redirect credential forwarding. A separate diagnostic host is
|
||||
explicit configuration, not an inference from an overridden completion endpoint.
|
||||
Keep configuration limited to named adapters and endpoints, not arbitrary methods
|
||||
or response predicates. For injected generation clients, initially report HTTP
|
||||
diagnostics unavailable unless a separate transport is explicitly supported.
|
||||
|
||||
This serves the maintained backends with a small adapter surface. Planning must
|
||||
settle concrete endpoint fields and how request/credential observations reflect
|
||||
redirects: a final response is not proof of acceptance at every redirect hop.
|
||||
|
||||
**Alternative:** initially support base-relative routes only, with no separate
|
||||
host or additional transport option. That simplifies routing but excludes dedicated
|
||||
diagnostic hosts. Another viable addition is an explicit diagnostic HTTP-client
|
||||
option for injected-client engines; it improves custom-client coverage but adds a
|
||||
supported injection surface. Neither choice requires expanding `LLMClient`.
|
||||
|
||||
### 3. Retry And Repair Controls
|
||||
|
||||
**Recommendation:** retain `TransientRetries`, with nil meaning zero for this
|
||||
operation. Positive values reuse built-in inference status retries and preserve
|
||||
attempt records; injected generation remains single-dispatch. Keep reachability
|
||||
and authentication single-attempt and structural repair outside the initial test
|
||||
procedure. This gives direct default evidence while preserving caller choice for
|
||||
transient inference failures, without changing ordinary execution policy.
|
||||
|
||||
**Alternative:** omit the retry field and keep every diagnostic single-attempt.
|
||||
That is the smallest API, but applications must repeat calls themselves. Inheriting
|
||||
the engine retry budget is also viable: it standardizes defaults but makes cost
|
||||
and recovery less obvious. Automatic structural repair would test recovery as
|
||||
well as first-response conformance and warrants a separate explicit decision.
|
||||
|
||||
### 4. Built-In Metadata Delivery And Default Profiles
|
||||
|
||||
**Recommendation:** coordinate compatible catalog releases containing capability
|
||||
metadata and backend defaults. Use `promptkit-openrouter-standard` for OpenRouter;
|
||||
have the RakestrawHome catalog maintainer select an explicit current profile for
|
||||
its diagnostic default. Include OpenRouter's adapter and RakestrawHome's `none`
|
||||
setting. Caller overlays remain available. These are replaceable defaults, not
|
||||
new promises about model names, availability, or cheap inference.
|
||||
|
||||
**Alternative:** contribute metadata through PromptKit-owned underlays initially,
|
||||
or require caller configuration. Underlays avoid waiting for catalog releases but
|
||||
move provider configuration into core. Caller-only metadata avoids coupled releases
|
||||
but loses much of the convenience motivating adoption. No new stable profile-name
|
||||
promise is necessary for the RakestrawHome default.
|
||||
1. **Configuration representation.** Finalize the Go convenience types and exact
|
||||
endpoint fields, including omission/null/overlay behavior and interpretation of
|
||||
configured default-profile references at construction versus selection. Preserve
|
||||
final-value ownership and verify shipped defaults in assembled configuration;
|
||||
do not introduce source-history validation or change ordinary profile selection.
|
||||
2. **Explicit diagnostic transport for injected clients.** Decide whether the first
|
||||
delivery exposes a separate diagnostic HTTP-client option or reports HTTP
|
||||
diagnostics unavailable for injected-client engines until such an option exists.
|
||||
Inference remains available through `LLMClient` in either case. No hidden
|
||||
transport fallback or new required client methods are permitted.
|
||||
3. **Catalog coordination.** Obtain the RakestrawHome maintainer's default-profile
|
||||
choice and select compatible loader/catalog versions in the documented release
|
||||
order. These are delivery dependencies, not reasons to reopen routing policy.
|
||||
4. **Implementation details.** Choose private content and schema, the narrow evidence
|
||||
handoff, and content-free classifications for internal faults that prevent a
|
||||
meaningful report. Define ownership and copying for reports and request values.
|
||||
Verify the state matrix, timeout/cancellation distinction, and redirect evidence
|
||||
against actual transport behavior before finalizing GoDoc. Keep public execution
|
||||
events and ordinary result/error semantics unchanged.
|
||||
|
||||
Reference in New Issue
Block a user