eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Define factual smoke test diagnostics roadmap

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-13 14:22:35 +00:00
parent d4676bc870
commit f1f72dfe8f
1 changed files with 474 additions and 192 deletions
Download Patch File Download Diff File Expand all files Collapse all files

666
docs/roadmap/smoke-tests.md
View File

@@ -1,228 +1,510 @@
# Smoke Test Capability Proposal
# Explicit Smoke Test Diagnostics
Status: core design decisions accepted; API details and delivery scope remain
under review. Implementation has not started.
Status: purpose and ownership boundary accepted; the concrete API and delivery
proposals below remain subject to review. Implementation has not started.
## Purpose
This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. Significant remaining choices appear under Open Decisions.
Provide reusable diagnostic operations for endpoint reachability,
authentication, plain-text inference, and structured-output inference.
Applications building setup, diagnostic, or preflight workflows would otherwise
repeat similar prompts, schemas, provider requests, and result interpretation.
Promptkit should standardize those mechanisms while applications decide which
checks to request and how their outcomes affect application behavior.
## Purpose And Ownership
This proposal records the agreed diagnostic boundary, private test content, and
execution-profile selection rules. The concrete API, configuration shape,
remaining execution defaults, and delivery scope remain subject to review.
It is not an implementation plan or a description of current public behavior.
Provide reusable, explicitly invoked diagnostics for endpoint reachability,
authentication, plain-text inference, and structured-output inference. Consumers
can build setup and diagnostic workflows without maintaining their own synthetic
prompts, schemas, and provider probes.
## Relationship To Existing Operations
PromptKit is opinionated about what each test checks and how it operates. It owns
fixed content, provider adapters, evaluation rules, and neutral factual reports.
Consumers select tests, targets, and supported execution settings; they interpret
the results and decide what action to take. There is no overall success,
application-readiness verdict, or configurable criteria framework.
For structured output, report whether structured output was requested, a response
was received, generated content was valid JSON, and that JSON matched the fixed
test schema. Four positive observations do not prove provider enforcement; no
field attempts to infer it.
For authentication, distinguish credentials supplied with the dispatched request,
a reachable authentication endpoint, and credentials accepted by that operation.
A reachable endpoint can reject credentials, and a transport failure can leave
acceptance undetermined. Authentication is inapplicable on the maintainer's
credential-free RakestrawHome deployment; that is neither an error nor a failure.
## Current Baseline
At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
checks whether a concrete request passes preparation without model generation.
`Run` prepares and executes the actual application request. Smoke tests would
instead exercise selected diagnostic operations using synthetic input or a
provider-specific probe. An application prompt may be selected to obtain its
default profile, but its messages are not executed and its variables and inputs
are not required. Smoke tests do not need to call `ValidateRequest` on a future
request.
checks a concrete application request without model generation. `Run` executes
that request and may structurally validate and repair its output. These operations
remain unchanged; smoke tests need not validate an intended application request.
A consumer could combine these operations into a preflight stage. Promptkit
would provide smoke test evidence, not decide that the application is ready.
Results describe the tested operation at the time of the check; they do not
reserve capacity or guarantee that a subsequent request will succeed.
The [runner](../internal/runner.md) shares captured prompt/profile selection and
execution-target resolution. The [generation coordinator](../internal/generation.md)
handles attempts, retries, usage, and active backend permits; the
[model client](../internal/llm.md) handles transport and decoding. Ordinary `Run`
requires an application prompt and returns no partial result on operational
failure, so calling it directly is not sufficient for the proposed diagnostic API.
The main value is shared diagnostics before real requests exist or when testing
a small synthetic request is useful. Consumers about to execute an ordinary
request may prefer to run it directly and handle its errors.
Backend configuration has no diagnostic endpoints, capabilities, or default test
profile. New fields must follow [ADR 0001](../adr/0001-configuration-boundary.md):
ConfigKit owns loading and composition; PromptKit interprets and validates final
effective values. Provenance stays available for explicit inspection and is not
an execution-time resolution or validation mechanism.
## Ownership
## Proposed Public Operation And Request
| Owner | Responsibility |
One synchronous call runs one independently selected operation:
```go
func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error)
type SmokeTestKind string
const (
SmokeTestReachability SmokeTestKind = "reachability"
SmokeTestAuthentication SmokeTestKind = "authentication"
SmokeTestTextInference SmokeTestKind = "text_inference"
SmokeTestStructuredOutput SmokeTestKind = "structured_output"
)
type SmokeTestRequest struct {
Kind SmokeTestKind
BackendID string
PromptID string
PromptVersion string
ProfileID string
Execution *ExecutionTargetOverride
APIKey string `json:"-"`
TransientRetries *int
}
```
The request supplies no variables, inputs, appended messages, schema, or output
contract. An application prompt is a profile-selection aid only: its messages,
session template, input requirements, and output contract are not executed.
Consumers can compose several calls themselves; custom application tests remain
available through ordinary `Run`.
Reuse existing `ExecutionTargetOverride` semantics. Inference uses its endpoint,
model, numeric, reasoning, timeout, credential-source, and provider-extra controls.
Backend-only probes use endpoint, credential-source, and timeout settings as
applicable; they do not require model completeness or execute generation merely
because a model setting was supplied. Profile settings remain the inference
baseline unless overridden; no silent cheaper-model substitution is proposed.
Direct credentials take precedence over environment credentials and follow
existing request-scoped ownership and redacted-formatting conventions. Optional
missing/blank credentials are omitted from the upstream request. Authentication
can test the resulting unauthenticated request rather than reject missing keys
locally. If an explicitly required credential prevents inference dispatch, report
that local diagnostic finding without pretending the provider rejected it.
Ordinary execution's required-credential behavior remains unchanged.
## Selection And Routing
Preserve the accepted inference selection order:
| Caller selection | Execution profile |
| --- | --- |
| Backend configuration | Declares available diagnostic operations and endpoints, configuration needed by supported probe implementations, and the backend's default smoke-test execution profile. |
| Promptkit | Owns probe execution, private fixed smoke-test prompts and schemas, available criteria, their evaluators, and neutral diagnostic reports. |
| Consuming application | Selects tests, targets, criteria, and execution options; decides whether results mean success, a warning, or a reason to stop; owns scheduling, presentation, and exit statuses. |
| Prompt ID only | Selected prompt version's default profile |
| Profile ID only | Explicit profile |
| Both | Explicit profile overrides the prompt default |
| Neither | Explicit backend's configured default smoke-test profile |
An endpoint address alone does not define a probe's meaning. A supported probe
implementation must know the request method, authentication requirements, and
how to interpret the response. A candidate design is for configuration to
identify a supported probe kind plus its endpoint settings. A general request
templating and response-expression language is not proposed initially.
Reuse exact identity and omitted-version semantics. An invalid explicit selection
does not fall back to a backend default. Missing backend/default selection is an
error when inference needs it, not a reason to substitute an arbitrary profile.
Backend-only probes require neither a default profile nor unrelated prompt data.
Configuration changes must retain the existing
[ConfigKit ownership boundary](../adr/0001-configuration-boundary.md).
The [architecture policy](../policy/architecture.md) continues to govern the
library and downstream application boundary.
Proposed routing: when a prompt or profile is selected, the effective profile
supplies the backend identity and endpoint. Otherwise `BackendID` selects the
backend. If both are provided, profile selection wins, including over a different
backend hint; report the actual choice. This is proposed precedence, not an
accepted new prohibition on selector combinations; see Open Decisions.
## Candidate Smoke Tests
An endpoint-only profile remains usable for inference and reachability without
an inferred backend identity or provider-specific authentication adapter.
Endpoint overrides retain the effective backend's ordinary capacity identity.
Backend-only probes can resolve profile endpoint/credential metadata without
requiring its model. Prompt selection uses captured identity/default-profile
metadata without rendering or loading the application's artifacts or schemas.
These are independently selectable operations, not increasingly strict levels
that require every preceding test to pass.
## Proposed Report Shapes
| Test | Intended evidence | Important distinction |
Use factual Boolean observations with presence, rather than a caller-configurable
list of criteria. These Go-facing values need no stable JSON contract initially.
```go
type SmokeTestReport struct {
Kind SmokeTestKind
Supported bool
Applicable *bool
Completed bool
Reason string
Target SmokeTestTarget
Facts SmokeTestFacts
Timing SmokeTestTiming
Attempts []SmokeTestAttempt
Usage ObservedTokenUsage
Failure ExecutionFailureKind
}
type SmokeTestTarget struct {
BackendID, ProfileID, PromptID, PromptVersion string
SelectionSource string // "backend_default", "profile", "prompt", or "backend"
Endpoint string // effective diagnostic destination
EffectiveSettings ExecutionTarget
SettingsPresence ExecutionTargetPresence
}
type SmokeTestFacts struct {
CredentialsSupplied *bool
EndpointReachable *bool
CredentialsAccepted *bool
ResponseReceived *bool
GenerationCompleted *bool
TextReturned *bool
ExpectedTextMatched *bool
StructuredOutputRequested *bool
ValidJSON *bool
SchemaMatched *bool
}
type SmokeTestTiming struct {
StartedAt time.Time
Duration time.Duration
CapacityWait *time.Duration
RequestDuration *time.Duration
EvaluationDuration *time.Duration
}
type SmokeTestAttempt struct {
Dispatched bool
Endpoint string
HTTPStatus *int
Facts SmokeTestFacts
Timing SmokeTestTiming
Failure ExecutionFailureKind
Usage ObservedTokenUsage
}
```
`Supported` records whether an implementation is available for the target and
selected client. `Applicable=false` records known inapplicability; nil leaves
applicability undetermined, for example when no authentication probe is configured.
Missing probe metadata does not imply that a provider accepts anonymous access.
`Reason` is a short factual code such as `authentication_not_applicable`,
`probe_not_configured`, or `http_diagnostics_unavailable`, not an application
severity or verdict.
`Completed` means the diagnostic concluded, not that checks passed. A rejected,
failed, unsupported, or inapplicable diagnostic can conclude with nil Go error.
Caller cancellation before completion leaves it false and returns available facts.
A fact is true or false only when established; nil means not evaluated,
undetermined, or inapplicable. Operation metadata and failure classification
provide context. In particular:
- `CredentialsSupplied`: a nonempty credential was supplied with the dispatched
request, not merely named in configuration. No dispatch leaves this fact absent.
For injected clients, do not claim upstream transmission PromptKit cannot observe.
- `EndpointReachable`: an HTTP response was received from the tested destination,
regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation
ending without a response yields false. An injected generation result alone
does not establish HTTP reachability.
- `CredentialsAccepted`: the designated authentication adapter recognized acceptance
or rejection. A rejection yields false; transport failure, server error, or an
uninterpretable response can leave it nil. Acceptance does not certify balance,
model permissions, or future access. Absence of credentials is not an invalid-key
assertion.
- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a
response was returned by the injected generation boundary. A non-success or
malformed HTTP response still counts as received.
- `GenerationCompleted`: generation decoded/completed successfully, separately from
generated-content checks. Empty decoded content can complete generation.
- `TextReturned`: generated content is non-whitespace. `ExpectedTextMatched` compares
against the private expected text using trimming only, without case folding or
semantic matching.
- `StructuredOutputRequested`: the fixed native schema constraint was supplied to
the built-in provider request or injected generation boundary. This is not proof
of upstream transmission by an injected client or provider enforcement.
- `ValidJSON`: the generated content, not the HTTP envelope, parsed as JSON.
`SchemaMatched` evaluates parsed JSON against the fixed test schema and remains
absent when parsing or response acquisition prevents evaluation.
Known inapplicable authentication performs no dispatch and leaves request and
acceptance facts absent. Unknown measurements remain unknown, including phase
measurements for unperformed work. Total duration includes waiting and evaluation;
request duration excludes capacity waiting. Timings impose no application latency
threshold or universal deadline.
If retries are enabled, top-level facts describe the terminal attempt and attempt
records preserve earlier failures. Usage reuses presence-aware accounting without
claiming complete provider billing. No facts are inferred from arbitrary error text.
`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt
observations; it is not proof that packets were sent or billable work occurred.
Reports omit private prompts, expected text, schemas, generated content, credentials,
raw bodies, and headers. Use content-free failure classifications and HTTP status,
not arbitrary provider messages, to preserve this exposure boundary. Effective
settings, provider-extra values, endpoints, and identities may still be sensitive
operator data; applications own their disclosure. Apply existing credential
exclusion and formatting rules rather than promise that all configuration is safe
for unrestricted logging.
## Diagnostic Procedures And Capability Configuration
### Reachability
Propose one unauthenticated HTTP GET to the configured diagnostic endpoint, or
the effective base endpoint when no separate endpoint is supplied. Any HTTP
response establishes reachability, including 401, 404, or 405. It does not establish
that the completion route, credentials, or a model work. Authentication is tested
separately. Classify transport failure where available without promising exhaustive
DNS/TLS diagnosis.
### Authentication
Metadata names a supported provider adapter or explicitly marks authentication
inapplicable. The proposed first adapter is `openrouter_key`, using GET on the
selected base endpoint's `key` path with ordinary bearer credential resolution.
OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key)
documents this authenticated operation and its 200 and 401 responses. Recognize
the documented successful shape and unauthorized response; leave acceptance
unknown on other outcomes instead of guessing. Returned account/key details
are not exposed.
Configure RakestrawHome authentication as `none`, pursuant to the maintainer's
stated deployment behavior. Other backends can declare supported adapters.
An arbitrary models-list or health endpoint is not credential-acceptance evidence.
### Inference
Plain-text inference uses a small fixed private prompt and expected answer.
Structured-output inference uses a fixed private prompt and self-contained schema,
requests native JSON Schema output through the existing client path, and evaluates
JSON and schema conformance locally. It imports no application output contract.
Exact content is an implementation detail and does not enter public prompt listings,
inspection, configuration inspection, prepared snapshots, or reports. Injected
clients and providers necessarily receive the content they execute: privacy here
is an API-exposure boundary, not secrecy from executing collaborators. There is
no persistent diagnostic prompt or session state.
### Proposed Backend Metadata
Extend optional backend configuration through the final-value loader and the
`BackendConfig`/`BackendSource` convenience. Illustrative shape:
```yaml
backends:
openrouter:
smoke_tests:
default_profile: promptkit-openrouter-standard
reachability:
endpoint: https://openrouter.ai/api/v1
authentication:
kind: openrouter_key
rakestrawhome:
smoke_tests:
authentication:
kind: none
```
Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted
reachability metadata uses the effective base for the generic HTTP probe;
omitted authentication metadata makes that operation unsupported, with applicability
undetermined. Missing default-profile selection matters when inference selects it,
not for backend-only checks or an explicit profile. Retain normal effective
presence/null and layering semantics. Validate syntax and recognized configuration
at the final handoff, without contacting providers or validating intermediate layers.
Provider paths follow the selected endpoint override, not silently the catalog's
original endpoint. Explicitly configured diagnostic destinations can be different;
report the destination actually tested. Exact endpoint options remain a proposal
under Open Decisions.
Catalogs or caller overlays can supply backend-specific metadata. New catalog
fields need a coordinated compatible loader release: older strict PromptKit
loaders may reject them. A catalog default is replaceable metadata, not a promise
of upstream availability or a stable model choice.
## Execution And Error Boundary
Reuse target resolution, generation coordination, HTTP decoding, structural
validation, and usage accounting internally. A separate diagnostic orchestration
entry point may retain evidence that ordinary `Run` intentionally discards.
Do not duplicate inference serialization or build a second inference stack.
Inference participates in existing engine-local admission and active backend
limits. HTTP-only probes need no generation permit or unrelated model selection.
Caller context and configured timeouts govern operations. The proposed inference
baseline is a single attempt, no structural repair, and otherwise the selected
profile plus explicit overrides. Retry choices below remain proposals; ordinary
engine defaults and repair behavior do not change. Small test content does not
establish small output, a price estimate, or a cost ceiling.
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
new required methods. HTTP diagnostics must describe an explicitly selected
transport, not secretly probe a different transport in an injected-client engine.
The proposed initial behavior is to report HTTP diagnostics unavailable unless
that transport is supported or explicitly supplied; an additional injection option
is a delivery choice below.
Proposed return conventions:
- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections
return ordinary public errors before probe work. Unsupported capability is not
the same as invalid selection.
- Concluded diagnostics return reports with nil Go error, including credential
rejection, provider rejection, transport or configured request-timeout failure,
invalid JSON, schema mismatch, unsupported capability, or inapplicability.
Local capacity exhaustion is a diagnostic execution finding, not provider failure;
request facts remain absent if nothing was dispatched.
- Caller cancellation or caller-context deadline expiration returns accumulated
observations with an error recognizable through `errors.Is` against
`context.Canceled` or `context.DeadlineExceeded`. An already-done context can yield
a report with no observations. Do not dispatch again merely to fill missing facts.
As elsewhere, arbitrary synchronous injected code must return before cancellation
can be observed.
The cancellation contract applies to this new operation, not to existing
preparation APIs. Internal faults preventing meaningful diagnostic execution may
still return an ordinary error; the implementation plan should identify those
boundaries without promising that every conceivable failure becomes a report.
## Representative Requests And Results
Request examples use ordinary exact selections, without private test language:
```go
// Select the application's default profile, but do not run its prompt.
req := promptkit.SmokeTestRequest{
Kind: promptkit.SmokeTestTextInference,
PromptID: "meeting.summary",
}
// An explicit profile overrides that default.
req.ProfileID = "promptkit-openrouter-standard"
// Select a backend's configured default for structured inference.
structured := promptkit.SmokeTestRequest{
Kind: promptkit.SmokeTestStructuredOutput,
BackendID: "openrouter",
}
```
Expected report examples:
| Operation/outcome | Observations | Go error |
| --- | --- | --- |
| HTTP/HTTPS reachability | An HTTP response was obtained from the selected destination. | A response such as HTTP 401 establishes reachability without establishing authentication. A failure should retain enough classification to distinguish connection, TLS, and timeout problems where available. |
| Authentication | A designated provider operation accepted the effective credentials. | Sending an Authorization header to an endpoint that ignores it does not prove acceptance. The operation needs a documented authentication contract. |
| Plain-text inference | The selected model completed a small text-generation request. | Successful generation and compliance with an exact expected response are separate criteria. |
| Structured-output inference | The selected model completed a small structured-output request, with separately evaluated JSON and schema criteria. | Provider-enforced structured output must be distinguished from prompting for JSON and validating it locally. |
| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil |
| RakestrawHome authentication | Applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil |
| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil |
| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil |
| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil |
| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error |
A backend model-status endpoint could provide additional provider-reported
evidence. It would not substitute for actually completing inference. Whether
to include model-status probes in the first implementation remains open.
Consumers decide whether any of these findings warrant success, a warning, or a
failure in their own application. They own CLI diagnostics, exit statuses,
scheduling, and any combination with `ValidateRequest` or normal execution.
Backend-only operations should not require unrelated prompt data or a runnable
model selection.
## Work Needed And Target Completion
## Inference Profile Selection
- Add the public request/report operation and GoDoc, keeping business logic internal.
Resolve significant Open Decisions before writing the staged implementation plan.
- Extend final backend metadata interpretation and captured lookup for capabilities
and defaults. Preserve explicit configuration inspection without exposing test
content or using source history for validation.
- Connect private inference content to shared selection, execution resolution,
capacity, generation, decoding, and structural evaluation without preparing
application inputs or messages.
- Add narrow HTTP probes, credential handling, evidence retention, factual results,
and partial cancellation reporting while preserving existing client contracts.
- Supply agreed built-in metadata through compatible catalog releases or PromptKit
underlays, with normal caller overlays and no provider-availability guarantee.
- Add focused offline tests with controlled transports and injected clients for
selection, rejected/absent credentials, applicability/support, transport/malformed
responses, content/JSON/schema facts, cancellation, shared inference capacity,
and credential/private-content exposure. Assign distinct test owners; do not
repeat existing transport, capacity, or validation matrices or snapshot private
test language.
- Update canonical consumer, backend-format, integration, and internal documentation
when implementation lands, with release notes explaining the optional addition.
Live diagnostics are explicit consumer operations, not default tests or new
release gates.
Callers can override the backend's default smoke-test execution profile through
the following selections:
Completion means consumers can select the four kinds, obtain supported factual
observations, distinguish false from unknown/inapplicable, and retain evidence
on cancellation. Existing preparation and execution continue unchanged. Initial
scope excludes application CLI/service work, readiness policy, continuous monitoring,
model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation.
| Caller supplies | Execution profile |
| --- | --- |
| Prompt ID without a profile override | The selected prompt definition's default profile. |
| Profile ID without a prompt ID | The explicitly selected profile. |
| Both prompt and profile IDs | The supplied profile overrides the prompt's default under ordinary Promptkit selection rules. |
| Neither prompt nor profile ID | The selected backend's default smoke-test profile. |
## Open Decisions
Reuse existing prompt/profile selection semantics, including exact selection
and omitted-selector handling; see the [consumer guide](../consumers/pkg-promptkit.md)
and [request GoDoc](../../types.go). The backend default applies when neither
selector is supplied, not as a replacement for an invalid explicit selection.
How callers identify the backend in that case, and behavior when its default
smoke-test profile is absent, remain to be specified.
### 1. Backend Selection Alongside A Profile
Selecting an application prompt supplies execution-profile selection only. It
does not replace the synthetic smoke-test messages or import the application's
output contract. An explicit profile selection must not be silently replaced
with a cheaper model. Reports should identify the selection source and effective
execution settings, excluding credential values.
**Recommendation:** adopt the proposed precedence: explicit or prompt-selected
profiles supply routing; `BackendID` supplies backend/default selection otherwise.
Report the effective choice. This follows familiar profile precedence without
inventing a mismatch prohibition or rebinding capacity identity.
## Private Test Content And Execution Options
**Alternative:** treat `BackendID` as a constraint when combined with profile/prompt
selection and reject mismatches. That catches mistakes and makes every selector
operative, but adds a diagnostic-specific restriction and makes profile changes
more disruptive. Choose it only if consumers want that constraint.
Promptkit owns fixed, private prompt definitions for plain-text and
structured-output smoke tests, including the latter's small schema. Their exact
language is an implementation detail. Callers cannot override the prompt
language, in whole or in part, or retrieve it through public prompt inspection,
configuration inspection, prepared snapshots, or diagnostic reports. These
definitions are not caller-configurable prompt catalog entries.
### 2. Initial HTTP Adapters, Endpoints, Redirects, And Injection
Injected model clients necessarily receive the messages they execute. That
execution boundary does not make the smoke-test prompt a public inspection or
customization feature, nor does private implementation content imply secrecy
from the executing client or provider.
**Recommendation:** start with generic unauthenticated HTTP-response reachability,
the OpenRouter current-key adapter, and explicit `none` authentication. Derive the
key route from the effective base, while allowing explicitly configured absolute
diagnostic endpoints for separate health/auth routes. Reuse the configured HTTP
client and its standard redirect handling, record the final observed destination,
and avoid custom redirect credential forwarding. A separate diagnostic host is
explicit configuration, not an inference from an overridden completion endpoint.
Keep configuration limited to named adapters and endpoints, not arbitrary methods
or response predicates. For injected generation clients, initially report HTTP
diagnostics unavailable unless a separate transport is explicitly supported.
Applications select criteria independently of the fixed test content. An
exact-text match need not be required merely to establish that inference
completed. Profile selection changes execution settings, not prompt language.
The supported per-call generation overrides and inheritance from the selected
profile remain to be specified, including retry, repair, reasoning, and token
limit defaults. Effective settings must be inspectable without revealing
credentials or private prompt messages.
This serves the maintained backends with a small adapter surface. Planning must
settle concrete endpoint fields and how request/credential observations reflect
redirects: a final response is not proof of acceptance at every redirect hop.
Promptkit reports timing without prescribing an acceptable latency threshold
or a universal diagnostic deadline. Caller cancellation and configured execution
timeouts still govern the operation. Applications decide whether the measured
timing is acceptable and whether to impose additional deadlines.
**Alternative:** initially support base-relative routes only, with no separate
host or additional transport option. That simplifies routing but excludes dedicated
diagnostic hosts. Another viable addition is an explicit diagnostic HTTP-client
option for injected-client engines; it improves custom-client coverage but adds a
supported injection surface. Neither choice requires expanding `LLMClient`.
Inference smoke tests make a small inference request and are likely to incur a
small, nonzero cost. Promptkit does not estimate prices or guarantee a cost
ceiling; prices and execution settings can change the actual cost. Report usage
when available. Non-inference probes should use endpoints documented for that
purpose; billing properties must not be inferred merely from an HTTP method.
### 3. Retry And Repair Controls
## Probes, Criteria, And Results
**Recommendation:** retain `TransientRetries`, with nil meaning zero for this
operation. Positive values reuse built-in inference status retries and preserve
attempt records; injected generation remains single-dispatch. Keep reachability
and authentication single-attempt and structural repair outside the initial test
procedure. This gives direct default evidence while preserving caller choice for
transient inference failures, without changing ordinary execution policy.
A probe performs an operation. Criteria evaluate observations from that
operation. Several criteria can use one response without repeating the request.
Candidate criteria include receiving an HTTP response, authentication
acceptance, successful generation, returned text, expected-text matching, JSON
parsing, and schema conformance. Each needs an explicit definition and a clear
set of applicable probes.
**Alternative:** omit the retry field and keep every diagnostic single-attempt.
That is the smallest API, but applications must repeat calls themselves. Inheriting
the engine retry budget is also viable: it standardizes defaults but makes cost
and recovery less obvious. Automatic structural repair would test recovery as
well as first-response conformance and warrants a separate explicit decision.
For example, a response could satisfy successful generation and returned-text
criteria while failing the built-in expected-text comparison. Promptkit reports
those findings without assigning an overall application-readiness verdict. The
application decides whether the mismatch warrants a warning, a failure, or no
action. Matching and normalization rules remain to be specified.
### 4. Built-In Metadata Delivery And Default Profiles
Candidate criterion outcomes are passed, failed, unsupported, and inconclusive,
with structured reasons. An unavailable authentication probe must not become an
invalid-key verdict. A timeout during authentication may prevent determining
credential acceptance even though the probe itself failed. Criteria that could
not be evaluated must not be confused with assertions evaluated as false.
**Recommendation:** coordinate compatible catalog releases containing capability
metadata and backend defaults. Use `promptkit-openrouter-standard` for OpenRouter;
have the RakestrawHome catalog maintainer select an explicit current profile for
its diagnostic default. Include OpenRouter's adapter and RakestrawHome's `none`
setting. Caller overlays remain available. These are replaceable defaults, not
new promises about model names, availability, or cheap inference.
A report should identify the tested operation and effective target, applicable
criteria and outcomes, and useful evidence such as HTTP status, elapsed time,
attempts, and reported usage. Unknown measurements remain unknown. Inference
retries or repair should not conceal initial failures behind a final success.
Exact fields, observation retention, and the relationship between reports and
returned Go errors remain open. Credential redaction and disclosure of provider
diagnostics need explicit treatment consistent with existing contracts.
## Implementation Direction
Reuse ordinary generation, decoding, validation, and error handling for
inference tests wherever their behavior fits. Do not build a second inference
stack. Private built-in prompt definitions can supply synthetic content through
that machinery without exposing them through the public prompt catalog or
configuration view. Existing behavior is described by the
[runner](../internal/runner.md), [generation coordinator](../internal/generation.md),
and [model-client integration contract](../integrations/openai-compatible-chat.md).
Reachability and authentication probes need a separate, narrow provider-facing
capability. How injected clients participate must be decided without assuming
that every custom client uses the built-in HTTP transport. Existing clients
should not acquire new required methods merely to continue ordinary execution.
## Decisions Needed Before Implementation
1. **Public operation shape and first delivery scope.** Choose individual
methods, a typed single-probe operation, or a convenience batch operation;
express the agreed profile-selection rules and define backend identification,
absent backend defaults, prompt-version selection, and endpoint-only cases.
Select initial provider adapters and decide whether model-status checks
belong in the first delivery. Go names and configuration keys are not yet
proposed.
2. **Probe configuration and routing.** Define supported probe kinds and their
response semantics. Resolve endpoint overrides, relative paths, separate
diagnostic hosts, redirects, and credential destinations deliberately so
the report describes the destination actually tested.
3. **Criteria and result contract.** Specify applicability, exact matching,
unsupported and inconclusive outcomes, errors versus diagnostic findings,
and what remains available after cancellation or partial completion.
4. **Execution mechanics.** Choose private test content, supported setting
overrides and inheritance, retry and repair defaults, capacity participation,
observation, and cancellation behavior within the agreed diagnostic boundary.
Custom smoke-test prompt language is excluded; application-authored tests
remain available via ordinary `Run`.
## Suggested Next Step
Write concrete request and result examples under the agreed boundaries for
three cases: an HTTP response with rejected
authentication, successful inference with an unexpected text response, and
schema-valid structured output. Include a backend without an authentication
probe to expose capability and outcome handling. Show prompt-based selection,
an explicit profile overriding a prompt default, and selection through a
backend's default smoke-test profile without disclosing private test language.
Use those examples to settle the operation and result shape, then specify the
smallest backend configuration needed by the initial adapters. Convert the
agreed design into a focused implementation plan with validation proportionate
to the selected contracts under the [testing policy](../policy/testing.md).
Default tests should remain offline using controlled transports and injected
clients; live smoke testing would be an explicit consumer operation.
## Scope Boundaries
This proposal does not include Scriptorium commands, application readiness
policy, continuous monitoring, automatic probes before ordinary runs, a
general-purpose diagnostic scripting language, public or customizable smoke-test
prompt language, pricing estimates, application latency thresholds, or a
guarantee about future generation. It does not change `ValidateRequest` or
ordinary execution.
Published API and format documentation should be updated when implementation
lands, not to present this proposal as current behavior.
**Alternative:** contribute metadata through PromptKit-owned underlays initially,
or require caller configuration. Underlays avoid waiting for catalog releases but
move provider configuration into core. Caller-only metadata avoids coupled releases
but loses much of the convenience motivating adoption. No new stable profile-name
promise is necessary for the RakestrawHome default.