promptkit/docs/roadmap/smoke-tests.md

Explicit Smoke Test Diagnostics

Status: feature design and delivery direction accepted; remaining implementation details are listed below. Implementation has not started.

This roadmap replaces the earlier capability proposal and incorporates the subsequent discussion. It defines the target feature, not a staged implementation plan or current public API. Remaining questions appear under Remaining Decisions.

Purpose And Ownership

Provide reusable, explicitly invoked diagnostics for endpoint reachability, authentication, plain-text inference, and structured-output inference. Consumers can build setup and diagnostic workflows without maintaining their own synthetic prompts, schemas, and provider probes.

PromptKit is opinionated about what each test checks and how it operates. It owns fixed content, provider adapters, evaluation rules, and neutral factual reports. Consumers select tests, targets, and supported execution settings; they interpret the results and decide what action to take. There is no overall success, application-readiness verdict, or configurable criteria framework.

For structured output, report whether structured output was requested, a response was received, generated content was valid JSON, and that JSON matched the fixed test schema. Four positive observations do not prove provider enforcement; no field attempts to infer it.

For authentication, distinguish credentials supplied with the dispatched request, a reachable authentication endpoint, and credentials accepted by that operation. A reachable endpoint can reject credentials, and a transport failure can leave acceptance undetermined. Authentication is inapplicable on the maintainer's credential-free RakestrawHome deployment; that is neither an error nor a failure.

Current Baseline

At commit d4676bc, PromptKit has no dedicated live diagnostic operation. ValidateRequest checks a concrete application request without model generation. Run executes that request and may structurally validate and repair its output. These operations remain unchanged; smoke tests need not validate an intended application request.

The runner shares captured prompt/profile selection and execution-target resolution. The generation coordinator handles attempts, retries, usage, and active backend permits; the model client handles transport and decoding. Ordinary Run requires an application prompt and returns no partial result on operational failure, so calling it directly is not sufficient for the proposed diagnostic API.

Backend configuration has no diagnostic endpoints, capabilities, or default test profile. New fields must follow ADR 0001: ConfigKit owns loading and composition; PromptKit interprets and validates final effective values. Provenance stays available for explicit inspection and is not an execution-time resolution or validation mechanism.

Public Operation And Request

One synchronous call runs one independently selected operation:

func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error)

type SmokeTestKind string

const (
    SmokeTestReachability     SmokeTestKind = "reachability"
    SmokeTestAuthentication   SmokeTestKind = "authentication"
    SmokeTestTextInference    SmokeTestKind = "text_inference"
    SmokeTestStructuredOutput SmokeTestKind = "structured_output"
)

type SmokeTestRequest struct {
    Kind          SmokeTestKind
    BackendID     string
    PromptID      string
    PromptVersion string
    ProfileID     string
    Execution     *ExecutionTargetOverride
    APIKey        string `json:"-"`
    TransientRetries *int
}

The request supplies no variables, inputs, appended messages, schema, or output contract. An application prompt is a profile-selection aid only: its messages, session template, input requirements, and output contract are not executed. Consumers can compose several calls themselves; custom application tests remain available through ordinary Run.

Reuse existing ExecutionTargetOverride semantics. Inference uses its endpoint, model, numeric, reasoning, timeout, credential-source, and provider-extra controls. Backend-only probes use endpoint, credential-source, and timeout settings as applicable; they do not require model completeness or execute generation merely because a model setting was supplied. Profile settings remain the inference baseline unless overridden; no silent cheaper-model substitution is proposed.

Direct credentials take precedence over environment credentials and follow existing request-scoped ownership and redacted-formatting conventions. Optional missing/blank credentials are omitted from the upstream request. Authentication can test the resulting unauthenticated request rather than reject missing keys locally. If an explicitly required credential prevents inference dispatch, report that local diagnostic finding without pretending the provider rejected it. Ordinary execution's required-credential behavior remains unchanged.

Selection And Routing

Preserve the accepted inference selection order:

Caller selection	Execution profile
Prompt ID only	Selected prompt version's default profile
Profile ID only	Explicit profile
Both	Explicit profile overrides the prompt default
Neither	Explicit backend's configured default smoke-test profile

Reuse exact identity and omitted-version semantics. An invalid explicit selection does not fall back to a backend default. Missing backend/default selection is an error when inference needs it, not a reason to substitute an arbitrary profile. Backend-only probes require neither a default profile nor unrelated prompt data.

Whenever inference selects a profile, that profile supplies the resolved endpoint, credential settings, and backend/capacity identity, subject to ordinary execution overrides. This applies equally to explicit profiles, prompt defaults, and backend-default profiles. A supplied BackendID is a selection fallback, not a constraint or a request to rebind the selected profile's backend identity.

For example, if backend A names a default profile that resolves to backend B, inference uses B's resolved target and capacity identity. If that default profile is endpoint-only, inference retains an empty backend identity. Report A as the default-selection source and B (or the empty identity) as the effective backend. No recursive backend-default lookup or mismatch rejection is introduced.

For backend-only probes, an explicit or prompt-selected profile likewise supplies the target and its backend's diagnostic metadata. Otherwise BackendID selects the backend directly, without consulting its default inference profile. The profile wins when both selectors are supplied, even when they name different backends.

An endpoint-only profile remains usable for inference and reachability without an inferred backend identity or provider-specific authentication adapter. Endpoint overrides retain the effective backend's ordinary capacity identity. Backend-only probes can resolve profile endpoint/credential metadata without requiring its model. Prompt selection uses captured identity/default-profile metadata without rendering or loading the application's artifacts or schemas.

Report Shapes

Use factual Boolean observations with presence, rather than a caller-configurable list of criteria. These Go-facing values need no stable JSON contract initially.

type SmokeTestReport struct {
    Kind       SmokeTestKind
    Supported  bool
    Applicable *bool
    Completed  bool
    Reason     SmokeTestReason
    Target     SmokeTestTarget
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Attempts   []SmokeTestAttempt
    Usage      ObservedTokenUsage
    Failure    ExecutionFailureKind
}

type SmokeTestTarget struct {
    BackendID, ProfileID, PromptID, PromptVersion string
    DefaultSourceBackendID string // backend whose default selected the profile, if any
    SelectionSource string // "backend_default", "profile", "prompt", or "backend"
    Endpoint        string // initially resolved diagnostic destination
    EffectiveSettings ExecutionTarget
    SettingsPresence ExecutionTargetPresence
}

type SmokeTestFacts struct {
    CredentialsSupplied       *bool
    EndpointReachable         *bool
    CredentialsAccepted       *bool
    ResponseReceived          *bool
    GenerationCompleted       *bool
    TextReturned              *bool
    ExpectedTextMatched       *bool
    StructuredOutputRequested *bool
    ValidJSON                 *bool
    SchemaMatched             *bool
}

type SmokeTestTiming struct {
    StartedAt           time.Time
    Duration            time.Duration
    CapacityWait        *time.Duration
    RequestDuration     *time.Duration
    RetryDelay          *time.Duration
    EvaluationDuration  *time.Duration
}

type SmokeTestAttempt struct {
    Dispatched bool
    InitialEndpoint string // resolved destination before redirects
    Endpoint   string // last observed request destination; facts refer to this destination
    HTTPStatus *int
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Failure    ExecutionFailureKind
    Usage      ObservedTokenUsage
}

type SmokeTestReason string

const (
    SmokeTestReasonNone                              SmokeTestReason = ""
    SmokeTestReasonAuthenticationNotApplicable       SmokeTestReason = "authentication_not_applicable"
    SmokeTestReasonProbeNotConfigured                SmokeTestReason = "probe_not_configured"
    SmokeTestReasonHTTPDiagnosticsUnavailable        SmokeTestReason = "http_diagnostics_unavailable"
    SmokeTestReasonRequiredCredentialMissing         SmokeTestReason = "required_credential_missing"
    SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized"
)

Supported records whether an implementation is available for the target and selected client. Applicable=false records known inapplicability; nil leaves applicability undetermined, for example when no authentication probe is configured. Missing probe metadata does not imply that a provider accepts anonymous access. Reason is a named factual code, not application severity or free-form diagnostic text. The constants distinguish, respectively, no additional reason, explicitly inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics for the selected client, a required credential missing before inference dispatch, and an authentication response whose documented meaning cannot be established. HTTP status and Failure classify ordinary rejection and transport outcomes; they do not need duplicate reason codes.

Completed means the diagnostic concluded, not that checks passed. A rejected, failed, unsupported, or inapplicable diagnostic can conclude with nil Go error. Caller cancellation before completion leaves it false and returns available facts. A nil Go error means the diagnostic concluded, not that the tested operation succeeded. Make this distinction prominent in GoDoc and consumer examples.

A fact is true or false only when established; nil means not evaluated, undetermined, or inapplicable. Operation metadata and failure classification provide context. In particular:

• CredentialsSupplied: a nonempty credential was attached to the observed dispatched request for the attempt's Endpoint, not merely named in configuration or attached to an earlier redirect hop. No observed request leaves this fact absent. For injected clients, do not claim upstream transmission PromptKit cannot observe. Observation at the transport boundary is not proof of wire transmission.
• EndpointReachable: an HTTP response was received from the tested destination, regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation ending without a response yields false. An injected generation result alone does not establish HTTP reachability.
• CredentialsAccepted: the designated authentication adapter recognized acceptance or rejection. A rejection yields false; transport failure, server error, or an uninterpretable response can leave it nil. Acceptance does not certify balance, model permissions, or future access. Absence of credentials is not an invalid-key assertion: a documented 401 with CredentialsSupplied=false records rejection of an unauthenticated request, not rejection of a supplied key.
• ResponseReceived: an HTTP response was obtained for built-in operations, or a response was returned by the injected generation boundary. A non-success or malformed HTTP response still counts as received.
• GenerationCompleted: generation decoded/completed successfully, separately from generated-content checks. Empty decoded content can complete generation.
• TextReturned: generated content is non-whitespace. ExpectedTextMatched compares against the private expected text using trimming only, without case folding or semantic matching.
• StructuredOutputRequested: the fixed native schema constraint was supplied to the built-in provider request or injected generation boundary. This is not proof of upstream transmission by an injected client or provider enforcement.
• ValidJSON: the generated content, not the HTTP envelope, parsed as JSON. SchemaMatched evaluates parsed JSON against the fixed test schema and remains absent when parsing or response acquisition prevents evaluation.

Known inapplicable authentication performs no dispatch and leaves request and acceptance facts absent. Unknown measurements remain unknown, including phase measurements for unperformed work. Report-level capacity wait, request duration, evaluation duration, and retry delay sum the corresponding measured work across the diagnostic. Request duration excludes capacity waiting, evaluation, and retry backoff. Retry delay includes completed and interrupted backoff. Per-attempt timings retain their own measurements; backoff is reported at the operation level. Total duration is wall time, including selection and other orchestration overhead, so it need not equal the phase sum. Timings impose no application latency threshold or universal deadline.

If retries are enabled, top-level facts describe the most recent dispatched attempt. Cancellation during backoff or a later capacity wait does not replace those facts with an undispatched attempt's empty facts. The operation's Failure and Completed still describe the interruption; earlier attempt records retain their own outcomes. A canceled capacity acquisition can add a Dispatched=false record, but no synthetic attempt is added for cancellation during backoff. With no dispatched attempt, request facts remain absent. Usage reuses presence-aware accounting without claiming complete provider billing. No facts are inferred from arbitrary error text. Dispatched means the HTTP/client boundary was invoked, as in existing attempt observations; it is not proof that packets were sent or billable work occurred.

Reports omit private prompts, expected text, schemas, generated content, credentials, raw bodies, and headers. Use content-free failure classifications and HTTP status, not arbitrary provider messages, to preserve this exposure boundary. Effective settings, provider-extra values, endpoints, and identities may still be sensitive operator data; applications own their disclosure. Apply existing credential exclusion and formatting rules rather than promise that all configuration is safe for unrestricted logging.

Diagnostic Procedures And Capability Configuration

Reachability

Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or the effective base endpoint when no separate endpoint is supplied. Any HTTP response establishes reachability, including 401, 404, or 405. It does not establish that the completion route, credentials, or a model work. Authentication is tested separately. Classify transport failure where available without promising exhaustive DNS/TLS diagnosis.

Authentication

Metadata names a supported provider adapter or explicitly marks authentication inapplicable. The first adapter is openrouter_key, using GET on the selected base endpoint's key path with ordinary bearer credential resolution. OpenRouter's current-key API documents this authenticated operation and its 200 and 401 responses. Recognize the documented successful shape and unauthorized response; leave acceptance unknown on other outcomes instead of guessing. Returned account/key details are not exposed.

Configure RakestrawHome authentication as none, pursuant to the maintainer's stated deployment behavior. Other backends can declare supported adapters. An arbitrary models-list or health endpoint is not credential-acceptance evidence.

Inference

Plain-text inference uses a small fixed private prompt and expected answer. Structured-output inference uses a fixed private prompt and self-contained schema, requests native JSON Schema output through the existing client path, and evaluates JSON and schema conformance locally. It imports no application output contract.

Exact content is an implementation detail and does not enter public prompt listings, inspection, configuration inspection, prepared snapshots, or reports. Injected clients and providers necessarily receive the content they execute: privacy here is an API-exposure boundary, not secrecy from executing collaborators. There is no persistent diagnostic prompt or session state.

Backend Metadata

Extend optional backend configuration through the final-value loader and the BackendConfig/BackendSource convenience. Illustrative shape:

backends:
  openrouter:
    smoke_tests:
      default_profile: promptkit-openrouter-standard
      reachability:
        endpoint: https://openrouter.ai/api/v1
      authentication:
        kind: openrouter_key
  rakestrawhome:
    smoke_tests:
      authentication:
        kind: none

Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted reachability metadata uses the effective base for the generic HTTP probe; omitted authentication metadata makes that operation unsupported, with applicability undetermined. Missing default-profile selection matters when inference selects it, not for backend-only checks or an explicit profile. Retain normal effective presence/null and layering semantics. Validate syntax and recognized configuration at the final handoff, without contacting providers or validating intermediate layers.

Provider paths follow the selected endpoint override, not silently the catalog's original endpoint. Explicit absolute diagnostic endpoints are supported for separate health/auth routes and take precedence for their operation. Configuration stays limited to named adapters and endpoint settings, not arbitrary methods or response predicates. Concrete field names and presence/null rules remain to be specified.

Reuse the configured HTTP client and its redirect handling, without adding custom credential forwarding. Target.Endpoint records the resolved starting destination. Each HTTP attempt records InitialEndpoint and the last observed dispatched request destination as Endpoint; its credential, reachability, response, and acceptance facts describe that destination. If a redirect leads to a failed request, do not attribute an earlier hop's response or credentials to the later destination. When a client prevents observing the relevant request, leave the affected evidence unknown. Injected generation alone supplies no HTTP destination evidence.

Authentication acceptance is interpreted from the response for the observed destination under the selected adapter's contract. It does not assert acceptance at the initial destination or at every redirect hop. These endpoint fields provide attribution without introducing a public redirect trace.

Deliver built-in metadata through coordinated compatible catalog releases; caller overlays remain available. First release loader support while retaining compatible existing catalog dependencies, then release catalogs containing the new metadata, then adopt those versions in PromptKit. Older strict loaders may reject new fields, so do not direct their consumers to upgrade catalogs independently. Select actual versions and compatibility notes during release preparation.

Use promptkit-openrouter-standard for OpenRouter's default, its openrouter_key adapter, and RakestrawHome's none authentication setting. The RakestrawHome catalog maintainer must select its current default profile. Verify each shipped default resolves in the supported assembled effective configuration, including any PromptKit-owned profiles. This is final-value integration validation, not a second validation pass over intermediate sources. A catalog default is replaceable metadata, not a promise of upstream availability, cheap inference, or a stable model choice.

Execution And Error Boundary

Reuse target resolution, generation coordination, HTTP decoding, structural validation, and usage accounting internally. A separate diagnostic orchestration entry point may retain evidence that ordinary Run intentionally discards. Do not duplicate inference serialization or build a second inference stack.

Add a narrow internal evidence path at the transport/coordinator boundary. Existing execution events do not carry redirect destinations or dispatched credential presence, and the coordinator can discard a completed response when cancellation is observed before return. Capture available facts before that discard and retain them for diagnostics. Keep ordinary execution's error/result contracts unchanged; do not expand public execution events just to reconstruct smoke-test evidence.

Inference participates in existing engine-local admission and active backend limits. HTTP-only probes need no generation permit or unrelated model selection. Caller context and configured timeouts govern operations. Inference defaults to one attempt: nil TransientRetries means zero for this operation rather than inheritance from the engine retry budget. Positive values reuse existing built-in inference status retries and retain attempt records; injected generation remains single-dispatch. Reachability and authentication remain single-attempt regardless of a valid inference retry setting. Structural repair is excluded from the initial procedure. Otherwise use the selected profile plus explicit overrides. Ordinary engine defaults and repair behavior do not change. Small test content does not establish small output, a price estimate, or a cost ceiling.

Keep LLMClient unchanged. Injected clients serve inference smoke tests without new required methods. HTTP diagnostics must describe an explicitly selected transport, not secretly probe a different transport in an injected-client engine. The initial behavior is to report HTTP diagnostics unavailable unless that transport is supported or explicitly supplied; an additional injection option remains a delivery detail below. Do not silently fall back to a default HTTP client to diagnose an injected generation client.

Return conventions:

• Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections return ordinary public errors before probe work. Unsupported capability is not the same as invalid selection.
• Concluded diagnostics return reports with nil Go error, including credential rejection, provider rejection, transport or configured request-timeout failure, invalid JSON, schema mismatch, unsupported capability, or inapplicability. Local capacity exhaustion is a diagnostic execution finding, not provider failure; request facts remain absent if nothing was dispatched.
• Caller cancellation or caller-context deadline expiration returns accumulated observations with an error recognizable through errors.Is against context.Canceled or context.DeadlineExceeded. An already-done context can yield a report with no observations. Do not dispatch again merely to fill missing facts. As elsewhere, arbitrary synchronous injected code must return before cancellation can be observed.

The cancellation contract applies to this new operation, not to existing preparation APIs. Internal faults preventing meaningful diagnostic execution may still return an ordinary error; the implementation plan should identify those boundaries without promising that every conceivable failure becomes a report.

Representative Requests And Results

Request examples use ordinary exact selections, without private test language:

// Select the application's default profile, but do not run its prompt.
req := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestTextInference,
    PromptID: "meeting.summary",
}
// An explicit profile overrides that default.
req.ProfileID = "promptkit-openrouter-standard"

// Select a backend's configured default for structured inference.
structured := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestStructuredOutput,
    BackendID: "openrouter",
}

Expected report examples:

Operation/outcome	Observations	Go error
Authentication rejected	Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401	nil
RakestrawHome authentication	Supported true, applicable false, completed, authentication_not_applicable; no dispatch/acceptance facts	nil
Unconfigured authentication	Supported false, applicability unknown, completed, probe_not_configured	nil
HTTP diagnostics unavailable for injected client	Supported false, applicability unknown, completed, http_diagnostics_unavailable; no dispatch	nil
Authentication without credentials receives documented 401	Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request	nil
Required credential absent before inference	Supported/applicable/completed; required_credential_missing, failure invalid_request; no dispatch and no request facts	nil
Authentication receives uninterpretable HTTP 200	Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; authentication_response_unrecognized, failure malformed_response	nil
Text inference returns unexpected nonempty text	Generation completed and text returned true; expected-text match false	nil
Structured inference matches schema	Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion	nil
Authentication transport fails	Endpoint reachable/response received false; credential supply recorded when known; acceptance nil	nil
Configured request timeout while caller context remains active	Supported/applicable/completed; failure deadline; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts	nil
Local capacity exhausted	Supported/applicable/completed; failure capacity; no dispatch or request facts	nil
Caller cancels during initial capacity wait	Supported/applicable, completed false; failure canceled, measured capacity wait, undispatched attempt; request facts absent	context.Canceled identity
Caller deadline expires during a dispatched request	Supported/applicable, completed false; failure deadline; retain available attempt evidence	context.DeadlineExceeded identity
Caller cancels during retry backoff after HTTP 503	Completed false, failure canceled; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt	context.Canceled identity
Caller cancels	Available observations retained; completed false when interrupted	Recognizable context error

Consumers decide whether any of these findings warrant success, a warning, or a failure in their own application. They own CLI diagnostics, exit statuses, scheduling, and any combination with ValidateRequest or normal execution.

Work Needed And Target Completion

• Add the public request/report operation and GoDoc, keeping business logic internal. Resolve the remaining delivery details when writing the staged implementation plan.
• Extend final backend metadata interpretation and captured lookup for capabilities and defaults. Preserve explicit configuration inspection without exposing test content or using source history for validation.
• Connect private inference content to shared selection, execution resolution, capacity, generation, decoding, and structural evaluation without preparing application inputs or messages.
• Add narrow HTTP probes, credential handling, evidence retention, factual results, and partial cancellation reporting through the internal evidence path while preserving existing client contracts. Specify destination/credential attribution without exposing headers or a redirect trace.
• Supply agreed built-in metadata through compatible catalog releases in the documented order, with normal caller overlays and no provider-availability guarantee. Check default-profile resolution in the assembled final configuration.
• Add focused offline tests with controlled transports and injected clients for selection, rejected/absent credentials, applicability/support, transport/malformed responses, content/JSON/schema facts, cancellation, shared inference capacity, and credential/private-content exposure. Assign distinct test owners; do not repeat existing transport, capacity, or validation matrices or snapshot private test language.
• Include the report-state matrix, cross-backend and endpoint-only default profiles, redirect attribution, cancellation after a response, and interrupted retry waits in focused contract coverage. Reuse existing lower-level checks where sufficient.
• Update canonical consumer, backend-format, integration, and internal documentation when implementation lands, with release notes explaining the optional addition. Live diagnostics are explicit consumer operations, not default tests or new release gates.

Completion means consumers can select the four kinds, obtain supported factual observations, distinguish false from unknown/inapplicable, and retain evidence on cancellation. Existing preparation and execution continue unchanged. Initial scope excludes application CLI/service work, readiness policy, continuous monitoring, model-status tests, batch orchestration, customizable test language, cost estimates, and guarantees about later generation.

Remaining Decisions And Delivery Dependencies

The operation, routing precedence, report/error boundary, initial adapters, retry/repair policy, and catalog delivery direction are settled. The remaining items should be resolved in the focused implementation plan:

1. Configuration representation. Finalize the Go convenience types and exact endpoint fields, including omission/null/overlay behavior and interpretation of configured default-profile references at construction versus selection. Preserve final-value ownership and verify shipped defaults in assembled configuration; do not introduce source-history validation or change ordinary profile selection.
2. Explicit diagnostic transport for injected clients. Decide whether the first delivery exposes a separate diagnostic HTTP-client option or reports HTTP diagnostics unavailable for injected-client engines until such an option exists. Inference remains available through LLMClient in either case. No hidden transport fallback or new required client methods are permitted.
3. Catalog coordination. Obtain the RakestrawHome maintainer's default-profile choice and select compatible loader/catalog versions in the documented release order. These are delivery dependencies, not reasons to reopen routing policy.
4. Implementation details. Choose private content and schema, the narrow evidence handoff, and content-free classifications for internal faults that prevent a meaningful report. Define ownership and copying for reports and request values. Verify the state matrix, timeout/cancellation distinction, and redirect evidence against actual transport behavior before finalizing GoDoc. Keep public execution events and ordinary result/error semantics unchanged.