Document smoke-test diagnostics

docs/internal/capacity.md

@@ -88,6 +88,17 @@ release, including canceled acquisition without dispatch. A retry wait holds
 the logical admission lease but no active permit. Observer callbacks must not
 block on work that requires the same run's admission slot.
 
+## Smoke-Test Capacity
+
+The engine's smoke tester shares this manager and generation coordinator with
+the ordinary runner. Inference diagnostics reserve the same logical admission
+slot and acquire the same backend-scoped active permits for actual model
+attempts, including retries. They therefore participate in the same configured
+backend limits as `Run`. Reachability and authentication HTTP probes do not
+enter generation admission or consume active permits. The public
+[`Engine.SmokeTest` GoDoc](../../smoke_diagnostics.go) owns the caller-visible
+operation and injected-client limits.
+
 ## Cancellation And Release
 
 Admission checks the caller context before reserving a slot. A call canceled
@@ -127,4 +138,7 @@ cover shared and independent backend pools during retry and repair. The
 own the public prepared-capacity boundary. The
 [root error-boundary tests](../../errors_internal_test.go) own preservation of
 the public generation category and context identity when generation is
-canceled.
+canceled. The assembled
+[smoke diagnostic contract tests](../../smoke_integration_contract_test.go)
+verify that inference diagnostics share the engine's Run admission and recover
+capacity after cancellation or failure.

docs/internal/generation.md

@@ -41,9 +41,20 @@ original error chain. This keeps injected attempt classifications consistent
 with terminal observations without parsing diagnostic text or retrying injected
 clients.
 
+The diagnostic path uses the same private attempt and retry implementation as
+ordinary generation. It retains invocation-local transport, response,
+generation, usage, capacity-wait, and retry-delay evidence before normal
+result-discard and cancellation checks. This evidence is returned to diagnostic
+orchestration; it is not added to public `ExecutionEvent` values or ordinary
+observer accounting. The [model-client document](llm.md) describes the
+transport-observation boundary, and the root
+[`SmokeTestReport` GoDoc](../../smoke_diagnostics.go) owns report semantics.
+
 Focused coordinator tests use controlled transport and wait functions. They
 cover eligibility, exhaustion, cancellation, backoff, request freezing, permit
-release, and the injected-client path. The capacity manager tests own permit
+release, and the injected-client path. The
+[evidence tests](../../internal/generation/evidence_test.go) own diagnostic
+attempt retention and retry evidence. The capacity manager tests own permit
 acquisition and idempotent release. Public
 [integration tests](../../reliable_execution_contract_test.go) cover retry,
 repair, observation, frozen prepared execution, and shared-backend behavior

docs/internal/llm.md

@@ -59,6 +59,33 @@ This client does not schedule retries; the [generation coordinator](generation.m
 does so around its prepared attempts. The client has no tool-call support,
 provider catalog, inbound HTTP behavior, or durable session store.
 
+## Diagnostic HTTP Evidence
+
+Built-in diagnostics wrap the selected HTTP client's `RoundTripper` for one
+invocation using a shallow client copy, preserving its timeout, redirect policy,
+cookie jar, and underlying transport without mutating the configured client.
+Evidence distinguishes a diagnostic dispatch invocation from an observed
+request and response. A newly observed request replaces the previous
+destination-specific response evidence, so redirects retain only the last
+observed hop. A response is recorded before envelope decoding or later context
+checks can discard a generation result.
+
+The collector keeps only the last observed destination (without URL user
+information), status, credential-presence, response-availability, decoded
+generation, and presence-aware usage facts needed by the diagnostic. It does
+not retain headers, credentials, raw request or response bodies, or a redirect
+trace. These are observations at Promptkit's RoundTripper boundary, not proof
+that bytes reached a peer or that a custom transport did no hidden dispatch.
+Reachability and named authentication probes use the same configured built-in
+client but discard their response bodies after local recognition. The
+[integration contract](../integrations/openai-compatible-chat.md#explicit-smoke-diagnostics)
+owns their observable HTTP semantics.
+
+An injected `LLMClient` does not pass through this HTTP boundary. Its returned
+response can establish response and inference facts, but not an HTTP
+destination, status, reachability, or upstream credential-supply fact. The
+facade does not silently install a built-in transport for those diagnostics.
+
 ## Prepared Generation
 
 For [`RunPrepared`](../../engine.go), the runner supplies the model client with
@@ -137,7 +164,10 @@ own configuration, client cloning, deterministic deadline precedence,
 authentication, request and response mapping, malformed data, error identity,
 cancellation, endpoint selection and composition, pre-transport rejection, and
 bounded single-document successful-response framing, closure, and
-response-body suppression. The focused
+response-body suppression. The [HTTP evidence tests](../../internal/llm/http_evidence_test.go)
+own invocation-local last-hop request/response facts, while the
+[prepared-generation evidence tests](../../internal/llm/prepared_generation_evidence_test.go)
+own retention across decoding and cancellation. The focused
 [provider HTTP error tests](../../internal/llm/provider_http_error_test.go)
 own envelope parsing, normalization, and bounded-reader cases; their
 [transport tests](../../internal/llm/provider_http_error_transport_test.go)

docs/internal/observation.md

@@ -41,6 +41,11 @@ attempts above the first in a generation, and repair generations with at least
 one dispatch. A terminal event is an aggregate and must not be added to its
 attempt events.
 
+Smoke diagnostics mask inherited execution-recorder state while invoking
+generation or HTTP collaborators. Their private attempt evidence is assembled
+into a `SmokeTestReport`; it does not create ordinary run events or enter an
+enclosing run's observation totals.
+
 The [recorder tests](../../internal/observation/recorder_test.go) own usage
 presence, phase sums, event ordering, empty-run accounting, identity snapshots,
 and callback panic behavior. The [generation coordinator](generation.md) owns

docs/internal/overview.md

@@ -11,17 +11,17 @@ contributor workflow and validation.
 
 | Component | Implemented responsibility | References |
 | --- | --- | --- |
-| Root `promptkit` package | Provides the supported engine facade, ConfigKit source helpers, profile and backend mapping conveniences, injection options, public requests and results, prompt/profile/requirement inspection and identity listing, concrete-request validation, observation values, opaque prepared-execution handles, value conversion, redacted formatting, and typed capacity and generation error mapping. | [Package GoDoc](../../doc.go), [observations](../../observations.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine facade](../../engine.go), [identity listing](../../listing.go), [requirement facade](../../requirements.go) |
+| Root `promptkit` package | Provides the supported engine facade, ConfigKit source helpers, profile and backend mapping conveniences, injection options, public requests and results, prompt/profile/requirement inspection and identity listing, concrete-request validation, explicit smoke diagnostics, observation values, opaque prepared-execution handles, value conversion, redacted formatting, and typed capacity and generation error mapping. | [Package GoDoc](../../doc.go), [observations](../../observations.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine facade](../../engine.go), [smoke diagnostics](../../smoke_diagnostics.go), [identity listing](../../listing.go), [requirement facade](../../requirements.go) |
 | `examples/go-library/prepare` | Demonstrates offline preparation and explicit configuration inspection with a prompt file, programmatic profile, and inline input. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
 | `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, programmatic profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
 | `examples/go-library/observe` | Demonstrates offline built-in HTTP retry and synchronous metadata observation across structured proposal and assessment runs using embedded sources, two profiles on one finite backend, and an application decision. It is not a public library package. | [Example program](../../examples/go-library/observe/main.go) |
-| `internal/backend` | Holds a defensive, immutable lookup and capacity-policy index for final backend definitions validated by the configuration adapter. | [Backend registry](../../internal/backend/registry.go) |
+| `internal/backend` | Holds a defensive, immutable lookup of final backend definitions, including diagnostic metadata, plus capacity policies validated by the configuration adapter. | [Backend registry](../../internal/backend/registry.go) |
 | `internal/catalog` | Selects released keyed catalog assets and PromptKit-owned profile definitions as ConfigKit underlays; effective validation belongs to the configuration loader. | [Catalog adapter](../../internal/catalog/catalog.go), [internal sources](sources.md#profiles-and-built-ins) |
-| `internal/configuration` | Loads keyed backend, profile, and prompt configuration through ConfigKit; records effective field presence, validates final values, captures indexed prompt definitions, referenced message content, compiled templates, input metadata snapshots, sorted identity-only profile and prompt inventories, backend and presence-aware profile snapshots, and an explicit read-only inspection view. | [Typed loader](../../internal/configuration/load.go), [identity listing](../../internal/configuration/listing.go), [internal sources](sources.md#typed-configuration-loading) |
+| `internal/configuration` | Loads keyed backend, profile, and prompt configuration through ConfigKit; records effective field presence, validates final values including optional diagnostic metadata, captures indexed prompt definitions, referenced message content, compiled templates, input metadata snapshots, sorted identity-only profile and prompt inventories, backend and presence-aware profile snapshots, and an explicit read-only inspection view. | [Typed loader](../../internal/configuration/load.go), [identity listing](../../internal/configuration/listing.go), [internal sources](sources.md#typed-configuration-loading) |
 | `internal/capacity` | Owns engine-local bounded execution admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
-| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, requirement inspection, profiles, execution targets, rendering, generation, and validation, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, and output contracts. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [requirement values](../../internal/domain/requirements.go), [endpoint invariant](../../internal/domain/endpoint.go) |
-| `internal/engine` | Assembles selected catalog and caller configuration into one snapshot, then constructs capacity, rendering, validation, generation, repair, and the use-case runner for each engine. It delegates identity listing directly to the captured configuration inventory rather than the runner. | [Engine assembly](../../internal/engine/engine.go), [internal sources](sources.md#profiles-and-built-ins), [internal runner](runner.md) |
-| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries, cancellable backoff, and attempt observations. | [Internal generation coordination](generation.md) |
+| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, requirement inspection, profiles, execution targets, rendering, generation, validation, and source-neutral smoke diagnostics, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, and output contracts. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [requirement values](../../internal/domain/requirements.go), [endpoint invariant](../../internal/domain/endpoint.go) |
+| `internal/engine` | Assembles selected catalog and caller configuration into one snapshot, then constructs shared capacity, rendering, validation, generation, repair, the ordinary runner, and the smoke tester for each engine. Built-in engines provide the diagnostic HTTP collaborator; injected-client engines do not. It delegates identity listing directly to the captured configuration inventory rather than the runner. | [Engine assembly](../../internal/engine/engine.go), [internal sources](sources.md#profiles-and-built-ins), [internal runner](runner.md) |
+| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries, cancellable backoff, ordinary attempt observations, and invocation-local evidence for diagnostics. | [Internal generation coordination](generation.md) |
 | `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
 | `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
 | `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types and rejecting cycles or excessive depth and work. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
@@ -30,9 +30,9 @@ contributor workflow and validation.
 | `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data, and conservatively enumerates dependencies from captured template syntax. | [Go-template renderer](../../internal/prompt/go_renderer.go), [requirement discovery](../../internal/prompt/requirements.go) |
 | `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
 | `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
-| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
-| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, and synchronous metadata event delivery for ordinary and claimed prepared runs and generation attempts. | [Internal execution observation](observation.md) |
-| `internal/usecase` | Selects captured prompts and profiles, enumerates requirements from captured prompt syntax, applies request overrides and completion checks, and coordinates ordinary and single-claim prepared execution across rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [requirement inspection](../../internal/usecase/requirements_inspection.go), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
+| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded response decoding, authentication, deadline handling, per-invocation HTTP evidence and built-in diagnostic probes, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
+| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, and synchronous metadata event delivery for ordinary and claimed prepared runs and generation attempts; smoke diagnostics keep their private evidence outside ordinary observer events. | [Internal execution observation](observation.md) |
+| `internal/usecase` | Selects captured prompts and profiles, enumerates requirements from captured prompt syntax, applies request overrides and completion checks, coordinates ordinary and single-claim prepared execution across rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair, and runs explicit smoke diagnostics with private cases, shared generation/capacity, and optional built-in HTTP probes. | [Internal runner](runner.md), [requirement inspection](../../internal/usecase/requirements_inspection.go), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
 
 The internal engine assembles these internal components without exposing their
 representations. Consumers depend only on the root facade.

docs/internal/runner.md

@@ -70,6 +70,35 @@ artifact, schema, rendering, admission, or model-client work. The root
 [`Engine.InspectProfile`](../../engine.go) GoDoc owns the public operation's
 exact contract.
 
+## Smoke-Test Diagnostics
+
+The separate `SmokeTester` in `internal/usecase` selects from the same captured
+configuration snapshot and backend/profile collaborators as the runner. Explicit
+profile selection takes precedence over prompt-default selection and a backend
+diagnostic default; HTTP-only selection does not resolve a default inference
+profile. Exact public selection rules belong to the root
+[`Engine.SmokeTest` GoDoc](../../smoke_diagnostics.go).
+
+Diagnostics do not call `Run`, `Prepare`, or the prompt preparation pipeline.
+They do not render application messages, load application artifacts, use
+application schemas, or inspect session templates. Private text and structured
+cases and the structured case's schema remain internal to the diagnostic
+operation; they are not entries in prompt/profile listings or inspection.
+`InspectConfiguration` remains a separate, explicit operator-oriented view of
+effective settings and source history. A diagnostic report may include selected
+effective settings and identifiers, so consumers still apply their disclosure
+policy.
+
+The engine assembles the smoke tester with the runner's generation coordinator
+and capacity manager. Inference holds the ordinary logical admission lease and
+uses the same active permits, retries, timeout, and request evidence path as
+generation. Reachability and authentication probes bypass generation capacity
+and do not require a model. An engine with an injected `LLMClient` can run
+inference diagnostics but has no built-in HTTP probe collaborator; HTTP-only
+diagnostics report that limitation instead of falling back to another client.
+The public GoDoc owns the caller-visible timeout, cancellation, report, and
+transport semantics.
+
 ## Shared Preparation Pipeline
 
 `Prepare` and `Run` share one private preparation pipeline split at the point
@@ -256,4 +285,8 @@ Changes to orchestration should continue to use the existing package
 interfaces, keep request state local to an invocation, and preserve the shared
 resolution and completion pipeline. Source, renderer, validator, or
 model-client contract changes belong first in their owning package and
-document.
+document. The
+[smoke selection tests](../../internal/usecase/smoke_selection_test.go),
+[diagnostic orchestration tests](../../internal/usecase/smoke_tester_test.go),
+and [private inference-case tests](../../internal/usecase/smoke_inference_test.go)
+own diagnostic selection, report assembly, and inference findings.

docs/internal/sources.md

@@ -118,8 +118,10 @@ change rendering behavior.
 Rakestrawhome `v3.20261013.0` assets as one ConfigKit underlay and PromptKit-owned
 profiles as the next. The selected catalog tags are local candidates and may not
 be publicly available. It discovers YAML inputs, including each catalog's
-opt-in diagnostic root, and supplies passive filesystem context without parsing
-or pre-validating catalog definitions. The configuration loader
+opt-in `smoke-tests` root in addition to its existing `catalog` root, and
+supplies passive filesystem context without parsing or pre-validating catalog
+definitions. Catalog metadata remains separate from existing `/v3` assets for
+strict-loader compatibility. The configuration loader
 validates only the final effective backend and profile values, including
 consumer overlays of maintained bases or backends. There is no catalog-only
 preflight, reserved source namespace, or replay of backend registration.