promptkit/docs/roadmap/smoke-tests.md

Explicit Smoke Test Diagnostics

Status: purpose and ownership boundary accepted; the concrete API and delivery proposals below remain subject to review. Implementation has not started.

This roadmap replaces the earlier capability proposal and incorporates the subsequent discussion. It defines the target feature, not a staged implementation plan or current public API. Significant remaining choices appear under Open Decisions.

Purpose And Ownership

Provide reusable, explicitly invoked diagnostics for endpoint reachability, authentication, plain-text inference, and structured-output inference. Consumers can build setup and diagnostic workflows without maintaining their own synthetic prompts, schemas, and provider probes.

PromptKit is opinionated about what each test checks and how it operates. It owns fixed content, provider adapters, evaluation rules, and neutral factual reports. Consumers select tests, targets, and supported execution settings; they interpret the results and decide what action to take. There is no overall success, application-readiness verdict, or configurable criteria framework.

For structured output, report whether structured output was requested, a response was received, generated content was valid JSON, and that JSON matched the fixed test schema. Four positive observations do not prove provider enforcement; no field attempts to infer it.

For authentication, distinguish credentials supplied with the dispatched request, a reachable authentication endpoint, and credentials accepted by that operation. A reachable endpoint can reject credentials, and a transport failure can leave acceptance undetermined. Authentication is inapplicable on the maintainer's credential-free RakestrawHome deployment; that is neither an error nor a failure.

Current Baseline

At commit d4676bc, PromptKit has no dedicated live diagnostic operation. ValidateRequest checks a concrete application request without model generation. Run executes that request and may structurally validate and repair its output. These operations remain unchanged; smoke tests need not validate an intended application request.

The runner shares captured prompt/profile selection and execution-target resolution. The generation coordinator handles attempts, retries, usage, and active backend permits; the model client handles transport and decoding. Ordinary Run requires an application prompt and returns no partial result on operational failure, so calling it directly is not sufficient for the proposed diagnostic API.

Backend configuration has no diagnostic endpoints, capabilities, or default test profile. New fields must follow ADR 0001: ConfigKit owns loading and composition; PromptKit interprets and validates final effective values. Provenance stays available for explicit inspection and is not an execution-time resolution or validation mechanism.

Proposed Public Operation And Request

One synchronous call runs one independently selected operation:

func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error)

type SmokeTestKind string

const (
    SmokeTestReachability     SmokeTestKind = "reachability"
    SmokeTestAuthentication   SmokeTestKind = "authentication"
    SmokeTestTextInference    SmokeTestKind = "text_inference"
    SmokeTestStructuredOutput SmokeTestKind = "structured_output"
)

type SmokeTestRequest struct {
    Kind          SmokeTestKind
    BackendID     string
    PromptID      string
    PromptVersion string
    ProfileID     string
    Execution     *ExecutionTargetOverride
    APIKey        string `json:"-"`
    TransientRetries *int
}

The request supplies no variables, inputs, appended messages, schema, or output contract. An application prompt is a profile-selection aid only: its messages, session template, input requirements, and output contract are not executed. Consumers can compose several calls themselves; custom application tests remain available through ordinary Run.

Reuse existing ExecutionTargetOverride semantics. Inference uses its endpoint, model, numeric, reasoning, timeout, credential-source, and provider-extra controls. Backend-only probes use endpoint, credential-source, and timeout settings as applicable; they do not require model completeness or execute generation merely because a model setting was supplied. Profile settings remain the inference baseline unless overridden; no silent cheaper-model substitution is proposed.

Direct credentials take precedence over environment credentials and follow existing request-scoped ownership and redacted-formatting conventions. Optional missing/blank credentials are omitted from the upstream request. Authentication can test the resulting unauthenticated request rather than reject missing keys locally. If an explicitly required credential prevents inference dispatch, report that local diagnostic finding without pretending the provider rejected it. Ordinary execution's required-credential behavior remains unchanged.

Selection And Routing

Preserve the accepted inference selection order:

Caller selection	Execution profile
Prompt ID only	Selected prompt version's default profile
Profile ID only	Explicit profile
Both	Explicit profile overrides the prompt default
Neither	Explicit backend's configured default smoke-test profile

Reuse exact identity and omitted-version semantics. An invalid explicit selection does not fall back to a backend default. Missing backend/default selection is an error when inference needs it, not a reason to substitute an arbitrary profile. Backend-only probes require neither a default profile nor unrelated prompt data.

Proposed routing: when a prompt or profile is selected, the effective profile supplies the backend identity and endpoint. Otherwise BackendID selects the backend. If both are provided, profile selection wins, including over a different backend hint; report the actual choice. This is proposed precedence, not an accepted new prohibition on selector combinations; see Open Decisions.

An endpoint-only profile remains usable for inference and reachability without an inferred backend identity or provider-specific authentication adapter. Endpoint overrides retain the effective backend's ordinary capacity identity. Backend-only probes can resolve profile endpoint/credential metadata without requiring its model. Prompt selection uses captured identity/default-profile metadata without rendering or loading the application's artifacts or schemas.

Proposed Report Shapes

Use factual Boolean observations with presence, rather than a caller-configurable list of criteria. These Go-facing values need no stable JSON contract initially.

type SmokeTestReport struct {
    Kind       SmokeTestKind
    Supported  bool
    Applicable *bool
    Completed  bool
    Reason     string
    Target     SmokeTestTarget
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Attempts   []SmokeTestAttempt
    Usage      ObservedTokenUsage
    Failure    ExecutionFailureKind
}

type SmokeTestTarget struct {
    BackendID, ProfileID, PromptID, PromptVersion string
    SelectionSource string // "backend_default", "profile", "prompt", or "backend"
    Endpoint        string // effective diagnostic destination
    EffectiveSettings ExecutionTarget
    SettingsPresence ExecutionTargetPresence
}

type SmokeTestFacts struct {
    CredentialsSupplied       *bool
    EndpointReachable         *bool
    CredentialsAccepted       *bool
    ResponseReceived          *bool
    GenerationCompleted       *bool
    TextReturned              *bool
    ExpectedTextMatched       *bool
    StructuredOutputRequested *bool
    ValidJSON                 *bool
    SchemaMatched             *bool
}

type SmokeTestTiming struct {
    StartedAt           time.Time
    Duration            time.Duration
    CapacityWait        *time.Duration
    RequestDuration     *time.Duration
    EvaluationDuration  *time.Duration
}

type SmokeTestAttempt struct {
    Dispatched bool
    Endpoint   string
    HTTPStatus *int
    Facts      SmokeTestFacts
    Timing     SmokeTestTiming
    Failure    ExecutionFailureKind
    Usage      ObservedTokenUsage
}

Supported records whether an implementation is available for the target and selected client. Applicable=false records known inapplicability; nil leaves applicability undetermined, for example when no authentication probe is configured. Missing probe metadata does not imply that a provider accepts anonymous access. Reason is a short factual code such as authentication_not_applicable, probe_not_configured, or http_diagnostics_unavailable, not an application severity or verdict.

Completed means the diagnostic concluded, not that checks passed. A rejected, failed, unsupported, or inapplicable diagnostic can conclude with nil Go error. Caller cancellation before completion leaves it false and returns available facts.

A fact is true or false only when established; nil means not evaluated, undetermined, or inapplicable. Operation metadata and failure classification provide context. In particular:

• CredentialsSupplied: a nonempty credential was supplied with the dispatched request, not merely named in configuration. No dispatch leaves this fact absent. For injected clients, do not claim upstream transmission PromptKit cannot observe.
• EndpointReachable: an HTTP response was received from the tested destination, regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation ending without a response yields false. An injected generation result alone does not establish HTTP reachability.
• CredentialsAccepted: the designated authentication adapter recognized acceptance or rejection. A rejection yields false; transport failure, server error, or an uninterpretable response can leave it nil. Acceptance does not certify balance, model permissions, or future access. Absence of credentials is not an invalid-key assertion.
• ResponseReceived: an HTTP response was obtained for built-in operations, or a response was returned by the injected generation boundary. A non-success or malformed HTTP response still counts as received.
• GenerationCompleted: generation decoded/completed successfully, separately from generated-content checks. Empty decoded content can complete generation.
• TextReturned: generated content is non-whitespace. ExpectedTextMatched compares against the private expected text using trimming only, without case folding or semantic matching.
• StructuredOutputRequested: the fixed native schema constraint was supplied to the built-in provider request or injected generation boundary. This is not proof of upstream transmission by an injected client or provider enforcement.
• ValidJSON: the generated content, not the HTTP envelope, parsed as JSON. SchemaMatched evaluates parsed JSON against the fixed test schema and remains absent when parsing or response acquisition prevents evaluation.

Known inapplicable authentication performs no dispatch and leaves request and acceptance facts absent. Unknown measurements remain unknown, including phase measurements for unperformed work. Total duration includes waiting and evaluation; request duration excludes capacity waiting. Timings impose no application latency threshold or universal deadline.

If retries are enabled, top-level facts describe the terminal attempt and attempt records preserve earlier failures. Usage reuses presence-aware accounting without claiming complete provider billing. No facts are inferred from arbitrary error text. Dispatched means the HTTP/client boundary was invoked, as in existing attempt observations; it is not proof that packets were sent or billable work occurred.

Reports omit private prompts, expected text, schemas, generated content, credentials, raw bodies, and headers. Use content-free failure classifications and HTTP status, not arbitrary provider messages, to preserve this exposure boundary. Effective settings, provider-extra values, endpoints, and identities may still be sensitive operator data; applications own their disclosure. Apply existing credential exclusion and formatting rules rather than promise that all configuration is safe for unrestricted logging.

Diagnostic Procedures And Capability Configuration

Reachability

Propose one unauthenticated HTTP GET to the configured diagnostic endpoint, or the effective base endpoint when no separate endpoint is supplied. Any HTTP response establishes reachability, including 401, 404, or 405. It does not establish that the completion route, credentials, or a model work. Authentication is tested separately. Classify transport failure where available without promising exhaustive DNS/TLS diagnosis.

Authentication

Metadata names a supported provider adapter or explicitly marks authentication inapplicable. The proposed first adapter is openrouter_key, using GET on the selected base endpoint's key path with ordinary bearer credential resolution. OpenRouter's current-key API documents this authenticated operation and its 200 and 401 responses. Recognize the documented successful shape and unauthorized response; leave acceptance unknown on other outcomes instead of guessing. Returned account/key details are not exposed.

Configure RakestrawHome authentication as none, pursuant to the maintainer's stated deployment behavior. Other backends can declare supported adapters. An arbitrary models-list or health endpoint is not credential-acceptance evidence.

Inference

Plain-text inference uses a small fixed private prompt and expected answer. Structured-output inference uses a fixed private prompt and self-contained schema, requests native JSON Schema output through the existing client path, and evaluates JSON and schema conformance locally. It imports no application output contract.

Exact content is an implementation detail and does not enter public prompt listings, inspection, configuration inspection, prepared snapshots, or reports. Injected clients and providers necessarily receive the content they execute: privacy here is an API-exposure boundary, not secrecy from executing collaborators. There is no persistent diagnostic prompt or session state.

Proposed Backend Metadata

Extend optional backend configuration through the final-value loader and the BackendConfig/BackendSource convenience. Illustrative shape:

backends:
  openrouter:
    smoke_tests:
      default_profile: promptkit-openrouter-standard
      reachability:
        endpoint: https://openrouter.ai/api/v1
      authentication:
        kind: openrouter_key
  rakestrawhome:
    smoke_tests:
      authentication:
        kind: none

Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted reachability metadata uses the effective base for the generic HTTP probe; omitted authentication metadata makes that operation unsupported, with applicability undetermined. Missing default-profile selection matters when inference selects it, not for backend-only checks or an explicit profile. Retain normal effective presence/null and layering semantics. Validate syntax and recognized configuration at the final handoff, without contacting providers or validating intermediate layers.

Provider paths follow the selected endpoint override, not silently the catalog's original endpoint. Explicitly configured diagnostic destinations can be different; report the destination actually tested. Exact endpoint options remain a proposal under Open Decisions.

Catalogs or caller overlays can supply backend-specific metadata. New catalog fields need a coordinated compatible loader release: older strict PromptKit loaders may reject them. A catalog default is replaceable metadata, not a promise of upstream availability or a stable model choice.

Execution And Error Boundary

Reuse target resolution, generation coordination, HTTP decoding, structural validation, and usage accounting internally. A separate diagnostic orchestration entry point may retain evidence that ordinary Run intentionally discards. Do not duplicate inference serialization or build a second inference stack.

Inference participates in existing engine-local admission and active backend limits. HTTP-only probes need no generation permit or unrelated model selection. Caller context and configured timeouts govern operations. The proposed inference baseline is a single attempt, no structural repair, and otherwise the selected profile plus explicit overrides. Retry choices below remain proposals; ordinary engine defaults and repair behavior do not change. Small test content does not establish small output, a price estimate, or a cost ceiling.

Keep LLMClient unchanged. Injected clients serve inference smoke tests without new required methods. HTTP diagnostics must describe an explicitly selected transport, not secretly probe a different transport in an injected-client engine. The proposed initial behavior is to report HTTP diagnostics unavailable unless that transport is supported or explicitly supplied; an additional injection option is a delivery choice below.

Proposed return conventions:

• Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections return ordinary public errors before probe work. Unsupported capability is not the same as invalid selection.
• Concluded diagnostics return reports with nil Go error, including credential rejection, provider rejection, transport or configured request-timeout failure, invalid JSON, schema mismatch, unsupported capability, or inapplicability. Local capacity exhaustion is a diagnostic execution finding, not provider failure; request facts remain absent if nothing was dispatched.
• Caller cancellation or caller-context deadline expiration returns accumulated observations with an error recognizable through errors.Is against context.Canceled or context.DeadlineExceeded. An already-done context can yield a report with no observations. Do not dispatch again merely to fill missing facts. As elsewhere, arbitrary synchronous injected code must return before cancellation can be observed.

The cancellation contract applies to this new operation, not to existing preparation APIs. Internal faults preventing meaningful diagnostic execution may still return an ordinary error; the implementation plan should identify those boundaries without promising that every conceivable failure becomes a report.

Representative Requests And Results

Request examples use ordinary exact selections, without private test language:

// Select the application's default profile, but do not run its prompt.
req := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestTextInference,
    PromptID: "meeting.summary",
}
// An explicit profile overrides that default.
req.ProfileID = "promptkit-openrouter-standard"

// Select a backend's configured default for structured inference.
structured := promptkit.SmokeTestRequest{
    Kind: promptkit.SmokeTestStructuredOutput,
    BackendID: "openrouter",
}

Expected report examples:

Operation/outcome	Observations	Go error
Authentication rejected	Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401	nil
RakestrawHome authentication	Applicable false, completed, authentication_not_applicable; no dispatch/acceptance facts	nil
Unconfigured authentication	Supported false, applicability unknown, completed, probe_not_configured	nil
Text inference returns unexpected nonempty text	Generation completed and text returned true; expected-text match false	nil
Structured inference matches schema	Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion	nil
Authentication transport fails	Endpoint reachable/response received false; credential supply recorded when known; acceptance nil	nil
Caller cancels	Available observations retained; completed false when interrupted	Recognizable context error

Consumers decide whether any of these findings warrant success, a warning, or a failure in their own application. They own CLI diagnostics, exit statuses, scheduling, and any combination with ValidateRequest or normal execution.

Work Needed And Target Completion

• Add the public request/report operation and GoDoc, keeping business logic internal. Resolve significant Open Decisions before writing the staged implementation plan.
• Extend final backend metadata interpretation and captured lookup for capabilities and defaults. Preserve explicit configuration inspection without exposing test content or using source history for validation.
• Connect private inference content to shared selection, execution resolution, capacity, generation, decoding, and structural evaluation without preparing application inputs or messages.
• Add narrow HTTP probes, credential handling, evidence retention, factual results, and partial cancellation reporting while preserving existing client contracts.
• Supply agreed built-in metadata through compatible catalog releases or PromptKit underlays, with normal caller overlays and no provider-availability guarantee.
• Add focused offline tests with controlled transports and injected clients for selection, rejected/absent credentials, applicability/support, transport/malformed responses, content/JSON/schema facts, cancellation, shared inference capacity, and credential/private-content exposure. Assign distinct test owners; do not repeat existing transport, capacity, or validation matrices or snapshot private test language.
• Update canonical consumer, backend-format, integration, and internal documentation when implementation lands, with release notes explaining the optional addition. Live diagnostics are explicit consumer operations, not default tests or new release gates.

Completion means consumers can select the four kinds, obtain supported factual observations, distinguish false from unknown/inapplicable, and retain evidence on cancellation. Existing preparation and execution continue unchanged. Initial scope excludes application CLI/service work, readiness policy, continuous monitoring, model-status tests, batch orchestration, customizable test language, cost estimates, and guarantees about later generation.

Open Decisions

1. Backend Selection Alongside A Profile

Recommendation: adopt the proposed precedence: explicit or prompt-selected profiles supply routing; BackendID supplies backend/default selection otherwise. Report the effective choice. This follows familiar profile precedence without inventing a mismatch prohibition or rebinding capacity identity.

Alternative: treat BackendID as a constraint when combined with profile/prompt selection and reject mismatches. That catches mistakes and makes every selector operative, but adds a diagnostic-specific restriction and makes profile changes more disruptive. Choose it only if consumers want that constraint.

2. Initial HTTP Adapters, Endpoints, Redirects, And Injection

Recommendation: start with generic unauthenticated HTTP-response reachability, the OpenRouter current-key adapter, and explicit none authentication. Derive the key route from the effective base, while allowing explicitly configured absolute diagnostic endpoints for separate health/auth routes. Reuse the configured HTTP client and its standard redirect handling, record the final observed destination, and avoid custom redirect credential forwarding. A separate diagnostic host is explicit configuration, not an inference from an overridden completion endpoint. Keep configuration limited to named adapters and endpoints, not arbitrary methods or response predicates. For injected generation clients, initially report HTTP diagnostics unavailable unless a separate transport is explicitly supported.

This serves the maintained backends with a small adapter surface. Planning must settle concrete endpoint fields and how request/credential observations reflect redirects: a final response is not proof of acceptance at every redirect hop.

Alternative: initially support base-relative routes only, with no separate host or additional transport option. That simplifies routing but excludes dedicated diagnostic hosts. Another viable addition is an explicit diagnostic HTTP-client option for injected-client engines; it improves custom-client coverage but adds a supported injection surface. Neither choice requires expanding LLMClient.

3. Retry And Repair Controls

Recommendation: retain TransientRetries, with nil meaning zero for this operation. Positive values reuse built-in inference status retries and preserve attempt records; injected generation remains single-dispatch. Keep reachability and authentication single-attempt and structural repair outside the initial test procedure. This gives direct default evidence while preserving caller choice for transient inference failures, without changing ordinary execution policy.

Alternative: omit the retry field and keep every diagnostic single-attempt. That is the smallest API, but applications must repeat calls themselves. Inheriting the engine retry budget is also viable: it standardizes defaults but makes cost and recovery less obvious. Automatic structural repair would test recovery as well as first-response conformance and warrants a separate explicit decision.

4. Built-In Metadata Delivery And Default Profiles

Recommendation: coordinate compatible catalog releases containing capability metadata and backend defaults. Use promptkit-openrouter-standard for OpenRouter; have the RakestrawHome catalog maintainer select an explicit current profile for its diagnostic default. Include OpenRouter's adapter and RakestrawHome's none setting. Caller overlays remain available. These are replaceable defaults, not new promises about model names, availability, or cheap inference.

Alternative: contribute metadata through PromptKit-owned underlays initially, or require caller configuration. Underlays avoid waiting for catalog releases but move provider configuration into core. Caller-only metadata avoids coupled releases but loses much of the convenience motivating adoption. No new stable profile-name promise is necessary for the RakestrawHome default.