promptkit/docs/roadmap/smoke-tests.md

# Explicit Smoke Test Diagnostics

Status: accepted target state. Implementation has not started.

This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. The companion [implementation plan](implementation.md)
owns the ordered work and delivery mechanics.

## Purpose And Ownership

Provide reusable, explicitly invoked diagnostics for endpoint reachability,
authentication, plain-text inference, and structured-output inference. Consumers
can build setup and diagnostic workflows without maintaining their own synthetic
prompts, schemas, and provider probes.

PromptKit is opinionated about what each test checks and how it operates. It owns
fixed content, provider adapters, evaluation rules, and neutral factual reports.
Consumers select tests, targets, and supported execution settings; they interpret
the results and decide what action to take. There is no overall success,
application-readiness verdict, or configurable criteria framework.

For structured output, report whether structured output was requested, a response
was received, generated content was valid JSON, and that JSON matched the fixed
test schema. Four positive observations do not prove provider enforcement; no
field attempts to infer it.

For authentication, distinguish credentials supplied with the dispatched request,
a reachable authentication endpoint, and credentials accepted by that operation.
A reachable endpoint can reject credentials, and a transport failure can leave
acceptance undetermined. Authentication is inapplicable on the maintainer's
credential-free RakestrawHome deployment; that is neither an error nor a failure.

## Single-Sprint Delivery Objective

The staged implementation must be completable from start to finish in one sprint,
including built-in defaults, tests, and documentation. It must not require
intermediate human-review approvals, upstream releases, external publication, or
live-provider access to progress or establish completion. Automated checks and
focused local commits remain part of ordinary implementation.

Built-in diagnostic metadata belongs in the local OpenRouter and RakestrawHome
catalogs, with their validated revisions identified by local candidate tags.
There is no intermediate PromptKit-owned metadata underlay or requirement for
separate loader/catalog publication before integration.

A temporary, uncommitted Go workspace may support coordinated development.
The completed integration uses normal module requirements and Go-generated
checksums and is validated with `GOWORK=off`, without filesystem replacements.
Exact local tag contents can be supplied through a temporary filesystem module
proxy, without public HTTPS availability or upstream publication. Local tags
alone do not make Go discover adjacent checkouts.
Keep workspace/proxy artifacts outside committed repository contents, following
the [repository-independence policy](../policy/architecture.md#repository-independence).

PromptKit's completed sprint commits may reference those not-yet-published
catalog versions in `go.mod` and their corresponding Go-generated checksums in
`go.sum`. Implementation completion and local validation do not claim that other
machines can fetch them. Public distribution is separate: consumers eventually
need those exact contents available through their configured module sources.

Normal release preparation and publication follow implementation separately;
this objective does not bypass the repository's release process or authorize
publication during the sprint.

## Current Baseline

At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
checks a concrete application request without model generation. `Run` executes
that request and may structurally validate and repair its output. These operations
remain unchanged; smoke tests need not validate an intended application request.

The [runner](../internal/runner.md) shares captured prompt/profile selection and
execution-target resolution. The [generation coordinator](../internal/generation.md)
handles attempts, retries, usage, and active backend permits; the
[model client](../internal/llm.md) handles transport and decoding. Ordinary `Run`
requires an application prompt and returns no partial result on operational
failure, so calling it directly is not sufficient for the proposed diagnostic API.

Backend configuration has no diagnostic endpoints, capabilities, or default test
profile. New fields must follow [ADR 0001](../adr/0001-configuration-boundary.md):
ConfigKit owns loading and composition; PromptKit interprets and validates final
effective values. Provenance stays available for explicit inspection and is not
an execution-time resolution or validation mechanism.

## Public Operation And Request

One synchronous call runs one independently selected operation:

```go
func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error)

type SmokeTestKind string

const (
    SmokeTestReachability     SmokeTestKind = "reachability"
    SmokeTestAuthentication   SmokeTestKind = "authentication"
    SmokeTestTextInference    SmokeTestKind = "text_inference"
    SmokeTestStructuredOutput SmokeTestKind = "structured_output"
)

type SmokeTestRequest struct {
    Kind          SmokeTestKind
    BackendID     string
    PromptID      string
    PromptVersion string
    ProfileID     string
    Execution     *ExecutionTargetOverride
    APIKey        string `json:"-"`
    TransientRetries *int
}
```

The request supplies no variables, inputs, appended messages, schema, or output
contract. An application prompt is a profile-selection aid only: its messages,
session template, input requirements, and output contract are not executed.
Consumers can compose several calls themselves; custom application tests remain
available through ordinary `Run`.

Reuse existing `ExecutionTargetOverride` semantics. Inference uses its endpoint,
model, numeric, reasoning, timeout, credential-source, and provider-extra controls.
Backend-only probes use endpoint, credential-source, and timeout settings as
applicable; they do not require model completeness or execute generation merely
because a model setting was supplied. Profile settings remain the inference
baseline unless overridden; no silent cheaper-model substitution is proposed.

Direct credentials take precedence over environment credentials and follow
existing request-scoped ownership and redacted-formatting conventions. Optional
missing/blank credentials are omitted from the upstream request. Authentication
can test the resulting unauthenticated request rather than reject missing keys
locally. If an explicitly required credential prevents inference dispatch, report
that local diagnostic finding without pretending the provider rejected it.
Ordinary execution's required-credential behavior remains unchanged.

## Selection And Routing

Preserve the accepted inference selection order:

| Caller selection | Execution profile |
| --- | --- |
| Prompt ID only | Selected prompt version's default profile |
| Profile ID only | Explicit profile |
| Both | Explicit profile overrides the prompt default |
| Neither | Explicit backend's configured default smoke-test profile |

Reuse exact identity and omitted-version semantics. An invalid explicit selection
does not fall back to a backend default. Missing backend/default selection is an
error when inference needs it, not a reason to substitute an arbitrary profile.
Backend-only probes require neither a default profile nor unrelated prompt data.

Whenever inference selects a profile, that profile supplies the resolved endpoint,
credential settings, and backend/capacity identity, subject to ordinary execution
overrides. This applies equally to explicit profiles, prompt defaults, and
backend-default profiles. A supplied `BackendID` is a selection fallback, not a
constraint or a request to rebind the selected profile's backend identity.

For example, if backend A names a default profile that resolves to backend B,
inference uses B's resolved target and capacity identity. If that default profile
is endpoint-only, inference retains an empty backend identity. Report A as the
default-selection source and B (or the empty identity) as the effective backend.
No recursive backend-default lookup or mismatch rejection is introduced.

For backend-only probes, an explicit or prompt-selected profile likewise supplies
the target and its backend's diagnostic metadata. Otherwise `BackendID` selects
the backend directly, without consulting its default inference profile. The
profile wins when both selectors are supplied, even when they name different
backends.

An endpoint-only profile remains usable for inference and reachability without
an inferred backend identity or provider-specific authentication adapter.
Endpoint overrides retain the effective backend's ordinary capacity identity.
Backend-only probes can resolve profile endpoint/credential metadata without
requiring its model. Prompt selection uses captured identity/default-profile
metadata without rendering or loading the application's artifacts or schemas.

## Report Shapes

Use factual Boolean observations with presence, rather than a caller-configurable
list of criteria. These Go-facing values need no stable JSON contract initially.

```go
type SmokeTestReport struct {
    Kind       SmokeTestKind
    Supported  bool
    Applicable *bool
    Completed  bool
    Reason     SmokeTestReason
    Target     SmokeTestTarget
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Attempts   []SmokeTestAttempt
    Usage      ObservedTokenUsage
    Failure    ExecutionFailureKind
}

type SmokeTestTarget struct {
    BackendID, ProfileID, PromptID, PromptVersion string
    DefaultSourceBackendID string // backend whose default selected the profile, if any
    SelectionSource string // "backend_default", "profile", "prompt", or "backend"
    Endpoint        string // initially resolved diagnostic destination
    EffectiveSettings ExecutionTarget
    SettingsPresence ExecutionTargetPresence
}

type SmokeTestFacts struct {
    CredentialsSupplied       *bool
    EndpointReachable         *bool
    CredentialsAccepted       *bool
    ResponseReceived          *bool
    GenerationCompleted       *bool
    TextReturned              *bool
    ExpectedTextMatched       *bool
    StructuredOutputRequested *bool
    ValidJSON                 *bool
    SchemaMatched             *bool
}

type SmokeTestTiming struct {
    StartedAt           time.Time
    Duration            time.Duration
    CapacityWait        *time.Duration
    RequestDuration     *time.Duration
    RetryDelay          *time.Duration
    EvaluationDuration  *time.Duration
}

type SmokeTestAttempt struct {
    Dispatched bool
    InitialEndpoint string // resolved destination before redirects
    Endpoint   string // last observed request destination; facts refer to this destination
    HTTPStatus *int
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Failure    ExecutionFailureKind
    Usage      ObservedTokenUsage
}

type SmokeTestReason string

const (
    SmokeTestReasonNone                              SmokeTestReason = ""
    SmokeTestReasonAuthenticationNotApplicable       SmokeTestReason = "authentication_not_applicable"
    SmokeTestReasonProbeNotConfigured                SmokeTestReason = "probe_not_configured"
    SmokeTestReasonHTTPDiagnosticsUnavailable        SmokeTestReason = "http_diagnostics_unavailable"
    SmokeTestReasonRequiredCredentialMissing         SmokeTestReason = "required_credential_missing"
    SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized"
)
```

`Supported` records whether an implementation is available for the target and
selected client. `Applicable=false` records known inapplicability; nil leaves
applicability undetermined, for example when no authentication probe is configured.
Missing probe metadata does not imply that a provider accepts anonymous access.
`Reason` is a named factual code, not application severity or free-form diagnostic
text. The constants distinguish, respectively, no additional reason, explicitly
inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics
for the selected client, a required credential missing before inference dispatch,
and an authentication response whose documented meaning cannot be established.
HTTP status and `Failure` classify ordinary rejection and transport outcomes;
they do not need duplicate reason codes.

`Completed` means the diagnostic concluded, not that checks passed. A rejected,
failed, unsupported, or inapplicable diagnostic can conclude with nil Go error.
Caller cancellation before completion leaves it false and returns available facts.
**A nil Go error means the diagnostic concluded, not that the tested operation
succeeded.** Make this distinction prominent in GoDoc and consumer examples.

A fact is true or false only when established; nil means not evaluated,
undetermined, or inapplicable. Operation metadata and failure classification
provide context. In particular:

- `CredentialsSupplied`: a nonempty credential was attached to the observed
  dispatched request for the attempt's `Endpoint`, not merely named in configuration
  or attached to an earlier redirect hop. No observed request leaves this fact
  absent. For injected clients, do not claim upstream transmission PromptKit cannot
  observe. Observation at the transport boundary is not proof of wire transmission.
- `EndpointReachable`: an HTTP response was received from the tested destination,
  regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation
  ending without a response yields false. An injected generation result alone
  does not establish HTTP reachability.
- `CredentialsAccepted`: the designated authentication adapter recognized acceptance
  or rejection. A rejection yields false; transport failure, server error, or an
  uninterpretable response can leave it nil. Acceptance does not certify balance,
  model permissions, or future access. Absence of credentials is not an invalid-key
  assertion: a documented 401 with `CredentialsSupplied=false` records rejection
  of an unauthenticated request, not rejection of a supplied key.
- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a
  response was returned by the injected generation boundary. A non-success or
  malformed HTTP response still counts as received.
- `GenerationCompleted`: generation decoded/completed successfully, separately from
  generated-content checks. Empty decoded content can complete generation.
- `TextReturned`: generated content is non-whitespace. `ExpectedTextMatched` compares
  against the private expected text using trimming only, without case folding or
  semantic matching.
- `StructuredOutputRequested`: the fixed native schema constraint was supplied to
  the built-in provider request or injected generation boundary. This is not proof
  of upstream transmission by an injected client or provider enforcement.
- `ValidJSON`: the generated content, not the HTTP envelope, parsed as JSON.
  `SchemaMatched` evaluates parsed JSON against the fixed test schema and remains
  absent when parsing or response acquisition prevents evaluation.

Known inapplicable authentication performs no dispatch and leaves request and
acceptance facts absent. Unknown measurements remain unknown, including phase
measurements for unperformed work. Report-level capacity wait, request duration,
evaluation duration, and retry delay sum the corresponding measured work across
the diagnostic. Request duration excludes capacity waiting, evaluation, and retry
backoff. Retry delay includes completed and interrupted backoff. Per-attempt
timings retain their own measurements; backoff is reported at the operation level.
Total duration is wall time, including selection and other orchestration overhead,
so it need not equal the phase sum. Timings impose no application latency threshold
or universal deadline.

If retries are enabled, top-level facts describe the most recent dispatched
attempt. Cancellation during backoff or a later capacity wait does not replace
those facts with an undispatched attempt's empty facts. The operation's `Failure`
and `Completed` still describe the interruption; earlier attempt records retain
their own outcomes. A canceled capacity acquisition can add a `Dispatched=false`
record, but no synthetic attempt is added for cancellation during backoff. With
no dispatched attempt, request facts remain absent. Usage reuses presence-aware
accounting without claiming complete provider billing. No facts are inferred from
arbitrary error text.
`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt
observations; it is not proof that packets were sent or billable work occurred.

Reports omit private prompts, expected text, schemas, generated content, credentials,
raw bodies, and headers. Use content-free failure classifications and HTTP status,
not arbitrary provider messages, to preserve this exposure boundary. Effective
settings, provider-extra values, endpoints, and identities may still be sensitive
operator data; applications own their disclosure. Apply existing credential
exclusion and formatting rules rather than promise that all configuration is safe
for unrestricted logging.

## Diagnostic Procedures And Capability Configuration

### Reachability

Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or
the effective base endpoint when no separate endpoint is supplied. Any HTTP
response establishes reachability, including 401, 404, or 405. It does not establish
that the completion route, credentials, or a model work. Authentication is tested
separately. Classify transport failure where available without promising exhaustive
DNS/TLS diagnosis.

### Authentication

Metadata names a supported provider adapter or explicitly marks authentication
inapplicable. The first adapter is `openrouter_key`, using GET on the
selected base endpoint's `key` path with ordinary bearer credential resolution.
OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key)
documents this authenticated operation and its 200 and 401 responses. Recognize
the documented successful shape and unauthorized response; leave acceptance
unknown on other outcomes instead of guessing. Returned account/key details
are not exposed.

Configure RakestrawHome authentication as `none`, pursuant to the maintainer's
stated deployment behavior. Other backends can declare supported adapters.
An arbitrary models-list or health endpoint is not credential-acceptance evidence.

### Inference

Plain-text inference uses a small fixed private prompt and expected answer.
Structured-output inference uses a fixed private prompt and self-contained schema,
requests native JSON Schema output through the existing client path, and evaluates
JSON and schema conformance locally. It imports no application output contract.

Exact content is an implementation detail and does not enter public prompt listings,
inspection, configuration inspection, prepared snapshots, or reports. Injected
clients and providers necessarily receive the content they execute: privacy here
is an API-exposure boundary, not secrecy from executing collaborators. There is
no persistent diagnostic prompt or session state.

### Backend Metadata

Extend optional backend configuration through the final-value loader and the
`BackendConfig`/`BackendSource` convenience. Illustrative shape:

```yaml
backends:
  openrouter:
    smoke_tests:
      default_profile: promptkit-openrouter-standard
      authentication:
        kind: openrouter_key
  rakestrawhome:
    smoke_tests:
      default_profile: rakestrawhome-gemma-4-31b
      authentication:
        kind: none
```

Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted
reachability metadata uses the effective base for the generic HTTP probe;
omitted authentication metadata makes that operation unsupported, with applicability
undetermined. Missing default-profile selection matters when inference selects it,
not for backend-only checks or an explicit profile. Retain normal effective
presence/null and layering semantics. Validate syntax and recognized configuration
at the final handoff, without contacting providers or validating intermediate layers.

Provider paths follow the selected endpoint override, not silently the catalog's
original endpoint. Explicit absolute diagnostic endpoints are supported for separate
health/auth routes and take precedence for their operation. Configuration stays
limited to named adapters and endpoint settings, not arbitrary methods or response
predicates. The companion implementation plan specifies convenience types and
presence/null interpretation for these fields.

Reuse the configured HTTP client and its redirect handling, without adding custom
credential forwarding. `Target.Endpoint` records the resolved starting destination.
Each HTTP attempt records `InitialEndpoint` and the last observed dispatched
request destination as `Endpoint`; its credential, reachability, response, and
acceptance facts describe that destination. If a redirect leads to a failed request,
do not attribute an earlier hop's response or credentials to the later destination.
When a client prevents observing the relevant request, leave the affected evidence
unknown. Injected generation alone supplies no HTTP destination evidence.

Authentication acceptance is interpreted from the response for the observed
destination under the selected adapter's contract. It does not assert acceptance
at the initial destination or at every redirect hop. These endpoint fields provide
attribution without introducing a public redirect trace.

Deliver built-in metadata in the local catalog repositories at
`../promptkit-backend-openrouter` and `../promptkit-backend-rakestrawhome`.
Catalog changes and PromptKit adoption are included in the sprint; publication
and remote module availability are not intermediate dependencies. Caller
overlays remain available through ordinary composition. Interpret and validate
only the assembled final configuration. Built-in reachability follows the
effective endpoint rather than pinning a separate catalog endpoint.

The catalogs expose diagnostic metadata through an additive, opt-in embedded
asset root. Existing `/v3` catalog roots remain usable by older strict loaders;
new PromptKit explicitly selects the additional assets. Provider metadata stays
owned by its catalog rather than being duplicated in PromptKit.

Local candidate tags identify validated contents; they are not published releases.
The implementation plan owns candidate selection and local proxy mechanics.
The catalogs' publication-date versioning convention still applies to later
publication. If that requires new final version names, update PromptKit's pins
and checksums accordingly; retain the validated catalog contents and never move
a published tag. Selecting the new metadata requires a compatible PromptKit
loader. No intervening PromptKit loader release or external module-download
verification is required to finish implementation.

Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
setting. Verify each shipped default resolves in the supported assembled effective
configuration, including any PromptKit-owned profiles. This is final-value
integration validation, not a second validation pass over intermediate sources.
A catalog default is replaceable metadata,
not a promise of upstream availability, cheap inference, or a stable model choice.

## Execution And Error Boundary

Reuse target resolution, generation coordination, HTTP decoding, structural
validation, and usage accounting internally. A separate diagnostic orchestration
entry point may retain evidence that ordinary `Run` intentionally discards.
Do not duplicate inference serialization or build a second inference stack.

Add a narrow internal evidence path at the transport/coordinator boundary. Existing
execution events do not carry redirect destinations or dispatched credential
presence, and the coordinator can discard a completed response when cancellation
is observed before return. Capture available facts before that discard and retain
them for diagnostics. Keep ordinary execution's error/result contracts unchanged;
do not expand public execution events just to reconstruct smoke-test evidence.

Inference participates in existing engine-local admission and active backend
limits. HTTP-only probes need no generation permit or unrelated model selection.
Caller context and configured timeouts govern operations. Inference defaults to
one attempt: nil `TransientRetries` means zero for this operation rather than
inheritance from the engine retry budget. Positive values reuse existing built-in
inference status retries and retain attempt records; injected generation remains
single-dispatch. Reachability and authentication remain single-attempt regardless
of a valid inference retry setting. Structural repair is excluded from the initial
procedure. Otherwise use the selected profile plus explicit overrides. Ordinary
engine defaults and repair behavior do not change. Small test content does not
establish small output, a price estimate, or a cost ceiling.

Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
new required methods. HTTP diagnostics must describe an explicitly selected
transport, not secretly probe a different transport in an injected-client engine.
For injected-client engines, the initial delivery reports HTTP diagnostics
unavailable and adds no separate diagnostic HTTP-client option. Inference
diagnostics remain available through the injected client. Known inapplicability,
such as authentication configured as `none`, can be reported without any transport.
Do not silently fall back to a default HTTP client to diagnose an injected
generation client. A future explicit diagnostic transport option remains possible
but is not needed to complete this feature.

Return conventions:

- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections
  return ordinary public errors before probe work. Unsupported capability is not
  the same as invalid selection.
- Concluded diagnostics return reports with nil Go error, including credential
  rejection, provider rejection, transport or configured request-timeout failure,
  invalid JSON, schema mismatch, unsupported capability, or inapplicability.
  Local capacity exhaustion is a diagnostic execution finding, not provider failure;
  request facts remain absent if nothing was dispatched.
- Caller cancellation or caller-context deadline expiration returns accumulated
  observations with an error recognizable through `errors.Is` against
  `context.Canceled` or `context.DeadlineExceeded`. An already-done context can yield
  a report with no observations. Do not dispatch again merely to fill missing facts.
  As elsewhere, arbitrary synchronous injected code must return before cancellation
  can be observed.

The cancellation contract applies to this new operation, not to existing
preparation APIs. Internal faults preventing meaningful diagnostic execution may
still return an ordinary error; the implementation plan should identify those
boundaries without promising that every conceivable failure becomes a report.

## Representative Requests And Results

Request examples use ordinary exact selections, without private test language:

```go
// Select the application's default profile, but do not run its prompt.
req := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestTextInference,
    PromptID: "meeting.summary",
}
// An explicit profile overrides that default.
req.ProfileID = "promptkit-openrouter-standard"

// Select a backend's configured default for structured inference.
structured := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestStructuredOutput,
    BackendID: "openrouter",
}
```

Expected report examples:

| Operation/outcome | Observations | Go error |
| --- | --- | --- |
| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil |
| RakestrawHome authentication | Supported true, applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil |
| HTTP diagnostics unavailable for injected client | Supported false, applicability unknown, completed, `http_diagnostics_unavailable`; no dispatch | nil |
| Authentication without credentials receives documented 401 | Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request | nil |
| Required credential absent before inference | Supported/applicable/completed; `required_credential_missing`, failure `invalid_request`; no dispatch and no request facts | nil |
| Authentication receives uninterpretable HTTP 200 | Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; `authentication_response_unrecognized`, failure `malformed_response` | nil |
| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil |
| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil |
| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil |
| Configured request timeout while caller context remains active | Supported/applicable/completed; failure `deadline`; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts | nil |
| Local capacity exhausted | Supported/applicable/completed; failure `capacity`; no dispatch or request facts | nil |
| Caller cancels during initial capacity wait | Supported/applicable, completed false; failure `canceled`, measured capacity wait, undispatched attempt; request facts absent | `context.Canceled` identity |
| Caller deadline expires during a dispatched request | Supported/applicable, completed false; failure `deadline`; retain available attempt evidence | `context.DeadlineExceeded` identity |
| Caller cancels during retry backoff after HTTP 503 | Completed false, failure `canceled`; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt | `context.Canceled` identity |
| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error |

Consumers decide whether any of these findings warrant success, a warning, or a
failure in their own application. They own CLI diagnostics, exit statuses,
scheduling, and any combination with `ValidateRequest` or normal execution.

## Work Needed And Target Completion

- Add the public request/report operation and GoDoc, keeping business logic internal.
- Extend final backend metadata interpretation and captured lookup for capabilities
  and defaults. Preserve explicit configuration inspection without exposing test
  content or using source history for validation.
- Connect private inference content to shared selection, execution resolution,
  capacity, generation, decoding, and structural evaluation without preparing
  application inputs or messages.
- Add narrow HTTP probes, credential handling, evidence retention, factual results,
  and partial cancellation reporting through the internal evidence path while
  preserving existing client contracts. Specify destination/credential attribution
  without exposing headers or a redirect trace.
- Deliver the agreed metadata through both local catalogs with ordinary
  catalog/caller precedence and no provider-availability guarantee. Shipped
  defaults resolve in the assembled final configuration, and selected versions
  work without workspace/replacement substitution. The single-sprint objective
  applies to the complete integration.
- Add focused offline tests with controlled transports and injected clients for
  selection, rejected/absent credentials, applicability/support, transport/malformed
  responses, content/JSON/schema facts, cancellation, shared inference capacity,
  and credential/private-content exposure. Assign distinct test owners; do not
  repeat existing transport, capacity, or validation matrices or snapshot private
  test language.
- Include the report-state matrix, cross-backend and endpoint-only default profiles,
  redirect attribution, cancellation after a response, and interrupted retry waits
  in focused contract coverage. Reuse existing lower-level checks where sufficient.
- Update canonical consumer, backend-format, integration, and internal documentation
  when implementation lands, with release notes explaining the optional addition.
  Live diagnostics are explicit consumer operations, not default tests or new
  release gates.

Completion means consumers can select the four kinds, obtain supported factual
observations, distinguish false from unknown/inapplicable, and retain evidence
on cancellation. Existing preparation and execution continue unchanged. Initial
scope excludes application CLI/service work, readiness policy, continuous monitoring,
model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation.