promptkit/docs/roadmap/smoke-tests.md

# Explicit Smoke Test Diagnostics

Status: purpose and ownership boundary accepted; the concrete API and delivery
proposals below remain subject to review. Implementation has not started.

This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. Significant remaining choices appear under Open Decisions.

## Purpose And Ownership

Provide reusable, explicitly invoked diagnostics for endpoint reachability,
authentication, plain-text inference, and structured-output inference. Consumers
can build setup and diagnostic workflows without maintaining their own synthetic
prompts, schemas, and provider probes.

PromptKit is opinionated about what each test checks and how it operates. It owns
fixed content, provider adapters, evaluation rules, and neutral factual reports.
Consumers select tests, targets, and supported execution settings; they interpret
the results and decide what action to take. There is no overall success,
application-readiness verdict, or configurable criteria framework.

For structured output, report whether structured output was requested, a response
was received, generated content was valid JSON, and that JSON matched the fixed
test schema. Four positive observations do not prove provider enforcement; no
field attempts to infer it.

For authentication, distinguish credentials supplied with the dispatched request,
a reachable authentication endpoint, and credentials accepted by that operation.
A reachable endpoint can reject credentials, and a transport failure can leave
acceptance undetermined. Authentication is inapplicable on the maintainer's
credential-free RakestrawHome deployment; that is neither an error nor a failure.

## Current Baseline

At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
checks a concrete application request without model generation. `Run` executes
that request and may structurally validate and repair its output. These operations
remain unchanged; smoke tests need not validate an intended application request.

The [runner](../internal/runner.md) shares captured prompt/profile selection and
execution-target resolution. The [generation coordinator](../internal/generation.md)
handles attempts, retries, usage, and active backend permits; the
[model client](../internal/llm.md) handles transport and decoding. Ordinary `Run`
requires an application prompt and returns no partial result on operational
failure, so calling it directly is not sufficient for the proposed diagnostic API.

Backend configuration has no diagnostic endpoints, capabilities, or default test
profile. New fields must follow [ADR 0001](../adr/0001-configuration-boundary.md):
ConfigKit owns loading and composition; PromptKit interprets and validates final
effective values. Provenance stays available for explicit inspection and is not
an execution-time resolution or validation mechanism.

## Proposed Public Operation And Request

One synchronous call runs one independently selected operation:

```go
func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error)

type SmokeTestKind string

const (
    SmokeTestReachability     SmokeTestKind = "reachability"
    SmokeTestAuthentication   SmokeTestKind = "authentication"
    SmokeTestTextInference    SmokeTestKind = "text_inference"
    SmokeTestStructuredOutput SmokeTestKind = "structured_output"
)

type SmokeTestRequest struct {
    Kind          SmokeTestKind
    BackendID     string
    PromptID      string
    PromptVersion string
    ProfileID     string
    Execution     *ExecutionTargetOverride
    APIKey        string `json:"-"`
    TransientRetries *int
}
```

The request supplies no variables, inputs, appended messages, schema, or output
contract. An application prompt is a profile-selection aid only: its messages,
session template, input requirements, and output contract are not executed.
Consumers can compose several calls themselves; custom application tests remain
available through ordinary `Run`.

Reuse existing `ExecutionTargetOverride` semantics. Inference uses its endpoint,
model, numeric, reasoning, timeout, credential-source, and provider-extra controls.
Backend-only probes use endpoint, credential-source, and timeout settings as
applicable; they do not require model completeness or execute generation merely
because a model setting was supplied. Profile settings remain the inference
baseline unless overridden; no silent cheaper-model substitution is proposed.

Direct credentials take precedence over environment credentials and follow
existing request-scoped ownership and redacted-formatting conventions. Optional
missing/blank credentials are omitted from the upstream request. Authentication
can test the resulting unauthenticated request rather than reject missing keys
locally. If an explicitly required credential prevents inference dispatch, report
that local diagnostic finding without pretending the provider rejected it.
Ordinary execution's required-credential behavior remains unchanged.

## Selection And Routing

Preserve the accepted inference selection order:

| Caller selection | Execution profile |
| --- | --- |
| Prompt ID only | Selected prompt version's default profile |
| Profile ID only | Explicit profile |
| Both | Explicit profile overrides the prompt default |
| Neither | Explicit backend's configured default smoke-test profile |

Reuse exact identity and omitted-version semantics. An invalid explicit selection
does not fall back to a backend default. Missing backend/default selection is an
error when inference needs it, not a reason to substitute an arbitrary profile.
Backend-only probes require neither a default profile nor unrelated prompt data.

Proposed routing: when a prompt or profile is selected, the effective profile
supplies the backend identity and endpoint. Otherwise `BackendID` selects the
backend. If both are provided, profile selection wins, including over a different
backend hint; report the actual choice. This is proposed precedence, not an
accepted new prohibition on selector combinations; see Open Decisions.

An endpoint-only profile remains usable for inference and reachability without
an inferred backend identity or provider-specific authentication adapter.
Endpoint overrides retain the effective backend's ordinary capacity identity.
Backend-only probes can resolve profile endpoint/credential metadata without
requiring its model. Prompt selection uses captured identity/default-profile
metadata without rendering or loading the application's artifacts or schemas.

## Proposed Report Shapes

Use factual Boolean observations with presence, rather than a caller-configurable
list of criteria. These Go-facing values need no stable JSON contract initially.

```go
type SmokeTestReport struct {
    Kind       SmokeTestKind
    Supported  bool
    Applicable *bool
    Completed  bool
    Reason     string
    Target     SmokeTestTarget
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Attempts   []SmokeTestAttempt
    Usage      ObservedTokenUsage
    Failure    ExecutionFailureKind
}

type SmokeTestTarget struct {
    BackendID, ProfileID, PromptID, PromptVersion string
    SelectionSource string // "backend_default", "profile", "prompt", or "backend"
    Endpoint        string // effective diagnostic destination
    EffectiveSettings ExecutionTarget
    SettingsPresence ExecutionTargetPresence
}

type SmokeTestFacts struct {
    CredentialsSupplied       *bool
    EndpointReachable         *bool
    CredentialsAccepted       *bool
    ResponseReceived          *bool
    GenerationCompleted       *bool
    TextReturned              *bool
    ExpectedTextMatched       *bool
    StructuredOutputRequested *bool
    ValidJSON                 *bool
    SchemaMatched             *bool
}

type SmokeTestTiming struct {
    StartedAt           time.Time
    Duration            time.Duration
    CapacityWait        *time.Duration
    RequestDuration     *time.Duration
    EvaluationDuration  *time.Duration
}

type SmokeTestAttempt struct {
    Dispatched bool
    Endpoint   string
    HTTPStatus *int
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Failure    ExecutionFailureKind
    Usage      ObservedTokenUsage
}
```

`Supported` records whether an implementation is available for the target and
selected client. `Applicable=false` records known inapplicability; nil leaves
applicability undetermined, for example when no authentication probe is configured.
Missing probe metadata does not imply that a provider accepts anonymous access.
`Reason` is a short factual code such as `authentication_not_applicable`,
`probe_not_configured`, or `http_diagnostics_unavailable`, not an application
severity or verdict.

`Completed` means the diagnostic concluded, not that checks passed. A rejected,
failed, unsupported, or inapplicable diagnostic can conclude with nil Go error.
Caller cancellation before completion leaves it false and returns available facts.

A fact is true or false only when established; nil means not evaluated,
undetermined, or inapplicable. Operation metadata and failure classification
provide context. In particular:

- `CredentialsSupplied`: a nonempty credential was supplied with the dispatched
  request, not merely named in configuration. No dispatch leaves this fact absent.
  For injected clients, do not claim upstream transmission PromptKit cannot observe.
- `EndpointReachable`: an HTTP response was received from the tested destination,
  regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation
  ending without a response yields false. An injected generation result alone
  does not establish HTTP reachability.
- `CredentialsAccepted`: the designated authentication adapter recognized acceptance
  or rejection. A rejection yields false; transport failure, server error, or an
  uninterpretable response can leave it nil. Acceptance does not certify balance,
  model permissions, or future access. Absence of credentials is not an invalid-key
  assertion.
- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a
  response was returned by the injected generation boundary. A non-success or
  malformed HTTP response still counts as received.
- `GenerationCompleted`: generation decoded/completed successfully, separately from
  generated-content checks. Empty decoded content can complete generation.
- `TextReturned`: generated content is non-whitespace. `ExpectedTextMatched` compares
  against the private expected text using trimming only, without case folding or
  semantic matching.
- `StructuredOutputRequested`: the fixed native schema constraint was supplied to
  the built-in provider request or injected generation boundary. This is not proof
  of upstream transmission by an injected client or provider enforcement.
- `ValidJSON`: the generated content, not the HTTP envelope, parsed as JSON.
  `SchemaMatched` evaluates parsed JSON against the fixed test schema and remains
  absent when parsing or response acquisition prevents evaluation.

Known inapplicable authentication performs no dispatch and leaves request and
acceptance facts absent. Unknown measurements remain unknown, including phase
measurements for unperformed work. Total duration includes waiting and evaluation;
request duration excludes capacity waiting. Timings impose no application latency
threshold or universal deadline.

If retries are enabled, top-level facts describe the terminal attempt and attempt
records preserve earlier failures. Usage reuses presence-aware accounting without
claiming complete provider billing. No facts are inferred from arbitrary error text.
`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt
observations; it is not proof that packets were sent or billable work occurred.

Reports omit private prompts, expected text, schemas, generated content, credentials,
raw bodies, and headers. Use content-free failure classifications and HTTP status,
not arbitrary provider messages, to preserve this exposure boundary. Effective
settings, provider-extra values, endpoints, and identities may still be sensitive
operator data; applications own their disclosure. Apply existing credential
exclusion and formatting rules rather than promise that all configuration is safe
for unrestricted logging.

## Diagnostic Procedures And Capability Configuration

### Reachability

Propose one unauthenticated HTTP GET to the configured diagnostic endpoint, or
the effective base endpoint when no separate endpoint is supplied. Any HTTP
response establishes reachability, including 401, 404, or 405. It does not establish
that the completion route, credentials, or a model work. Authentication is tested
separately. Classify transport failure where available without promising exhaustive
DNS/TLS diagnosis.

### Authentication

Metadata names a supported provider adapter or explicitly marks authentication
inapplicable. The proposed first adapter is `openrouter_key`, using GET on the
selected base endpoint's `key` path with ordinary bearer credential resolution.
OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key)
documents this authenticated operation and its 200 and 401 responses. Recognize
the documented successful shape and unauthorized response; leave acceptance
unknown on other outcomes instead of guessing. Returned account/key details
are not exposed.

Configure RakestrawHome authentication as `none`, pursuant to the maintainer's
stated deployment behavior. Other backends can declare supported adapters.
An arbitrary models-list or health endpoint is not credential-acceptance evidence.

### Inference

Plain-text inference uses a small fixed private prompt and expected answer.
Structured-output inference uses a fixed private prompt and self-contained schema,
requests native JSON Schema output through the existing client path, and evaluates
JSON and schema conformance locally. It imports no application output contract.

Exact content is an implementation detail and does not enter public prompt listings,
inspection, configuration inspection, prepared snapshots, or reports. Injected
clients and providers necessarily receive the content they execute: privacy here
is an API-exposure boundary, not secrecy from executing collaborators. There is
no persistent diagnostic prompt or session state.

### Proposed Backend Metadata

Extend optional backend configuration through the final-value loader and the
`BackendConfig`/`BackendSource` convenience. Illustrative shape:

```yaml
backends:
  openrouter:
    smoke_tests:
      default_profile: promptkit-openrouter-standard
      reachability:
        endpoint: https://openrouter.ai/api/v1
      authentication:
        kind: openrouter_key
  rakestrawhome:
    smoke_tests:
      authentication:
        kind: none
```

Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted
reachability metadata uses the effective base for the generic HTTP probe;
omitted authentication metadata makes that operation unsupported, with applicability
undetermined. Missing default-profile selection matters when inference selects it,
not for backend-only checks or an explicit profile. Retain normal effective
presence/null and layering semantics. Validate syntax and recognized configuration
at the final handoff, without contacting providers or validating intermediate layers.

Provider paths follow the selected endpoint override, not silently the catalog's
original endpoint. Explicitly configured diagnostic destinations can be different;
report the destination actually tested. Exact endpoint options remain a proposal
under Open Decisions.

Catalogs or caller overlays can supply backend-specific metadata. New catalog
fields need a coordinated compatible loader release: older strict PromptKit
loaders may reject them. A catalog default is replaceable metadata, not a promise
of upstream availability or a stable model choice.

## Execution And Error Boundary

Reuse target resolution, generation coordination, HTTP decoding, structural
validation, and usage accounting internally. A separate diagnostic orchestration
entry point may retain evidence that ordinary `Run` intentionally discards.
Do not duplicate inference serialization or build a second inference stack.

Inference participates in existing engine-local admission and active backend
limits. HTTP-only probes need no generation permit or unrelated model selection.
Caller context and configured timeouts govern operations. The proposed inference
baseline is a single attempt, no structural repair, and otherwise the selected
profile plus explicit overrides. Retry choices below remain proposals; ordinary
engine defaults and repair behavior do not change. Small test content does not
establish small output, a price estimate, or a cost ceiling.

Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
new required methods. HTTP diagnostics must describe an explicitly selected
transport, not secretly probe a different transport in an injected-client engine.
The proposed initial behavior is to report HTTP diagnostics unavailable unless
that transport is supported or explicitly supplied; an additional injection option
is a delivery choice below.

Proposed return conventions:

- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections
  return ordinary public errors before probe work. Unsupported capability is not
  the same as invalid selection.
- Concluded diagnostics return reports with nil Go error, including credential
  rejection, provider rejection, transport or configured request-timeout failure,
  invalid JSON, schema mismatch, unsupported capability, or inapplicability.
  Local capacity exhaustion is a diagnostic execution finding, not provider failure;
  request facts remain absent if nothing was dispatched.
- Caller cancellation or caller-context deadline expiration returns accumulated
  observations with an error recognizable through `errors.Is` against
  `context.Canceled` or `context.DeadlineExceeded`. An already-done context can yield
  a report with no observations. Do not dispatch again merely to fill missing facts.
  As elsewhere, arbitrary synchronous injected code must return before cancellation
  can be observed.

The cancellation contract applies to this new operation, not to existing
preparation APIs. Internal faults preventing meaningful diagnostic execution may
still return an ordinary error; the implementation plan should identify those
boundaries without promising that every conceivable failure becomes a report.

## Representative Requests And Results

Request examples use ordinary exact selections, without private test language:

```go
// Select the application's default profile, but do not run its prompt.
req := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestTextInference,
    PromptID: "meeting.summary",
}
// An explicit profile overrides that default.
req.ProfileID = "promptkit-openrouter-standard"

// Select a backend's configured default for structured inference.
structured := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestStructuredOutput,
    BackendID: "openrouter",
}
```

Expected report examples:

| Operation/outcome | Observations | Go error |
| --- | --- | --- |
| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil |
| RakestrawHome authentication | Applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil |
| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil |
| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil |
| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil |
| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error |

Consumers decide whether any of these findings warrant success, a warning, or a
failure in their own application. They own CLI diagnostics, exit statuses,
scheduling, and any combination with `ValidateRequest` or normal execution.

## Work Needed And Target Completion

- Add the public request/report operation and GoDoc, keeping business logic internal.
  Resolve significant Open Decisions before writing the staged implementation plan.
- Extend final backend metadata interpretation and captured lookup for capabilities
  and defaults. Preserve explicit configuration inspection without exposing test
  content or using source history for validation.
- Connect private inference content to shared selection, execution resolution,
  capacity, generation, decoding, and structural evaluation without preparing
  application inputs or messages.
- Add narrow HTTP probes, credential handling, evidence retention, factual results,
  and partial cancellation reporting while preserving existing client contracts.
- Supply agreed built-in metadata through compatible catalog releases or PromptKit
  underlays, with normal caller overlays and no provider-availability guarantee.
- Add focused offline tests with controlled transports and injected clients for
  selection, rejected/absent credentials, applicability/support, transport/malformed
  responses, content/JSON/schema facts, cancellation, shared inference capacity,
  and credential/private-content exposure. Assign distinct test owners; do not
  repeat existing transport, capacity, or validation matrices or snapshot private
  test language.
- Update canonical consumer, backend-format, integration, and internal documentation
  when implementation lands, with release notes explaining the optional addition.
  Live diagnostics are explicit consumer operations, not default tests or new
  release gates.

Completion means consumers can select the four kinds, obtain supported factual
observations, distinguish false from unknown/inapplicable, and retain evidence
on cancellation. Existing preparation and execution continue unchanged. Initial
scope excludes application CLI/service work, readiness policy, continuous monitoring,
model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation.

## Open Decisions

### 1. Backend Selection Alongside A Profile

**Recommendation:** adopt the proposed precedence: explicit or prompt-selected
profiles supply routing; `BackendID` supplies backend/default selection otherwise.
Report the effective choice. This follows familiar profile precedence without
inventing a mismatch prohibition or rebinding capacity identity.

**Alternative:** treat `BackendID` as a constraint when combined with profile/prompt
selection and reject mismatches. That catches mistakes and makes every selector
operative, but adds a diagnostic-specific restriction and makes profile changes
more disruptive. Choose it only if consumers want that constraint.

### 2. Initial HTTP Adapters, Endpoints, Redirects, And Injection

**Recommendation:** start with generic unauthenticated HTTP-response reachability,
the OpenRouter current-key adapter, and explicit `none` authentication. Derive the
key route from the effective base, while allowing explicitly configured absolute
diagnostic endpoints for separate health/auth routes. Reuse the configured HTTP
client and its standard redirect handling, record the final observed destination,
and avoid custom redirect credential forwarding. A separate diagnostic host is
explicit configuration, not an inference from an overridden completion endpoint.
Keep configuration limited to named adapters and endpoints, not arbitrary methods
or response predicates. For injected generation clients, initially report HTTP
diagnostics unavailable unless a separate transport is explicitly supported.

This serves the maintained backends with a small adapter surface. Planning must
settle concrete endpoint fields and how request/credential observations reflect
redirects: a final response is not proof of acceptance at every redirect hop.

**Alternative:** initially support base-relative routes only, with no separate
host or additional transport option. That simplifies routing but excludes dedicated
diagnostic hosts. Another viable addition is an explicit diagnostic HTTP-client
option for injected-client engines; it improves custom-client coverage but adds a
supported injection surface. Neither choice requires expanding `LLMClient`.

### 3. Retry And Repair Controls

**Recommendation:** retain `TransientRetries`, with nil meaning zero for this
operation. Positive values reuse built-in inference status retries and preserve
attempt records; injected generation remains single-dispatch. Keep reachability
and authentication single-attempt and structural repair outside the initial test
procedure. This gives direct default evidence while preserving caller choice for
transient inference failures, without changing ordinary execution policy.

**Alternative:** omit the retry field and keep every diagnostic single-attempt.
That is the smallest API, but applications must repeat calls themselves. Inheriting
the engine retry budget is also viable: it standardizes defaults but makes cost
and recovery less obvious. Automatic structural repair would test recovery as
well as first-response conformance and warrants a separate explicit decision.

### 4. Built-In Metadata Delivery And Default Profiles

**Recommendation:** coordinate compatible catalog releases containing capability
metadata and backend defaults. Use `promptkit-openrouter-standard` for OpenRouter;
have the RakestrawHome catalog maintainer select an explicit current profile for
its diagnostic default. Include OpenRouter's adapter and RakestrawHome's `none`
setting. Caller overlays remain available. These are replaceable defaults, not
new promises about model names, availability, or cheap inference.

**Alternative:** contribute metadata through PromptKit-owned underlays initially,
or require caller configuration. Underlays avoid waiting for catalog releases but
move provider configuration into core. Caller-only metadata avoids coupled releases
but loses much of the convenience motivating adoption. No new stable profile-name
promise is necessary for the RakestrawHome default.