Document smoke-test diagnostics
This commit is contained in:
@@ -216,6 +216,69 @@ request-specific overrides. An empty diff is not proof of equivalent execution.
|
||||
Records can contain ordinary application data, so use the application's
|
||||
disclosure policy when presenting them.
|
||||
|
||||
## Run Explicit Smoke Diagnostics
|
||||
|
||||
[`Engine.SmokeTest`](../../smoke_diagnostics.go) runs one explicit diagnostic
|
||||
using a target resolved from the engine's captured configuration. It does not
|
||||
execute the selected application prompt or its inputs, session, artifacts, or
|
||||
output schema. Unlike the offline examples and default test suite, calling it
|
||||
can contact the selected provider and use configured credentials. It is a
|
||||
consumer operation, not an automatic readiness verdict or a release check.
|
||||
|
||||
For example, a caller can request the built-in text inference check and treat
|
||||
its optional fact as an observation rather than assuming that a nil Go error
|
||||
means success:
|
||||
|
||||
```go
|
||||
report, err := engine.SmokeTest(ctx, promptkit.SmokeTestRequest{
|
||||
Kind: promptkit.SmokeTestTextInference,
|
||||
ProfileID: profileID,
|
||||
APIKey: apiKey,
|
||||
})
|
||||
if err != nil {
|
||||
if report != nil {
|
||||
retainDisclosureSafeSummary(report)
|
||||
}
|
||||
return err
|
||||
}
|
||||
if report != nil {
|
||||
matched := report.Facts.ExpectedTextMatched
|
||||
switch {
|
||||
case matched == nil:
|
||||
// The comparison was not evaluated or is unknown/inapplicable.
|
||||
case *matched:
|
||||
// The private expected-text comparison matched.
|
||||
default:
|
||||
// It did not match; the operation may still have concluded normally.
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
A nil Go error means the diagnostic concluded, not that its finding was
|
||||
positive. Fact pointers are tri-state: nil means unknown, not evaluated, or
|
||||
inapplicable. Reports contain factual observations and effective settings, not
|
||||
generated text or provider bodies; identifiers and settings can still be
|
||||
sensitive. The
|
||||
[`SmokeTest` GoDoc](../../smoke_diagnostics.go) owns selection precedence,
|
||||
exact fields, timeout and retry behavior, supported transports, and disclosure
|
||||
details.
|
||||
|
||||
Caller cancellation can return both an error and a partial report. Preserve the
|
||||
context error identity and inspect any available observations before returning:
|
||||
|
||||
```go
|
||||
report, err := engine.SmokeTest(ctx, request)
|
||||
if report != nil {
|
||||
retainDisclosureSafeSummary(report)
|
||||
}
|
||||
if err != nil {
|
||||
if errors.Is(err, context.Canceled) || errors.Is(err, context.DeadlineExceeded) {
|
||||
// The report may describe work completed before interruption.
|
||||
}
|
||||
return err
|
||||
}
|
||||
```
|
||||
|
||||
## Inspect A Prompt Before Preparation
|
||||
|
||||
Use [`Engine.InspectPrompt`](../../engine.go) to check one configured prompt's
|
||||
|
||||
@@ -195,7 +195,8 @@ backends:
|
||||
|
||||
An omitted or null `smoke_tests` block disables these metadata defaults.
|
||||
`default_profile` is preserved as an exact, nonblank profile ID; its existence
|
||||
is checked only if a diagnostic selects that default. Omitted/null
|
||||
is checked only if an inference diagnostic selects that default. Omitted/null
|
||||
`default_profile` means there is no inference default. Omitted/null
|
||||
`reachability` or its endpoint uses the effective backend endpoint. An explicit
|
||||
reachability endpoint is trimmed and must be an absolute HTTP(S) URL with a
|
||||
host; unlike the backend base endpoint, its path and query are allowed. Null
|
||||
@@ -204,9 +205,30 @@ means fallback, while an explicit blank value is invalid.
|
||||
Omitted/null `authentication` means no authentication adapter is declared. A
|
||||
non-null object requires `kind`, which is trimmed and case-normalized; the
|
||||
recognized values are `none` and `openrouter_key`. Its omitted/null endpoint
|
||||
uses the adapter's backend-relative destination; an explicit endpoint follows
|
||||
the same full-URL rules as reachability. These values are configuration only:
|
||||
engine construction does not contact a backend.
|
||||
uses the adapter's backend-relative destination (`openrouter_key` appends
|
||||
`/key` to the effective backend base); an explicit endpoint follows the same
|
||||
full-URL rules as reachability. `none` declares authentication inapplicable.
|
||||
These values are configuration only: engine construction does not contact a
|
||||
backend.
|
||||
|
||||
These fields are resolved through the ordinary ConfigKit source layers before
|
||||
Promptkit validates the final effective values. A higher-priority mapping can
|
||||
override individual nested fields; an explicit null can clear the whole
|
||||
`smoke_tests` block or one of its nested objects. An endpoint set to null uses
|
||||
its documented fallback, while an explicit blank endpoint is invalid. The
|
||||
metadata does not change ordinary generation credential requirements or
|
||||
provider settings.
|
||||
|
||||
The selected v3 catalog modules ship diagnostic defaults in a separate,
|
||||
opt-in `smoke-tests` root rather than changing the existing `catalog` root.
|
||||
This Promptkit build selects [OpenRouter v3 `v3.20261013.0`](oss/attribution.md#promptkit-openrouter-catalog-v3)
|
||||
with default profile `promptkit-openrouter-standard` and adapter
|
||||
`openrouter_key`, and [Rakestrawhome v3 `v3.20261013.0`](oss/attribution.md#promptkit-rakestrawhome-catalog-v3)
|
||||
with default profile `rakestrawhome-gemma-4-31b` and adapter `none`. The
|
||||
attribution entries identify the selected module commits; these catalog
|
||||
candidates are local and may not be publicly available. A consumer selecting
|
||||
the extra root needs a compatible loader. These defaults do not promise
|
||||
provider availability, account access, or free test requests.
|
||||
|
||||
## Profile Definitions
|
||||
|
||||
|
||||
@@ -45,6 +45,45 @@ precedence over the backend default. Only the name reaches prepared metadata;
|
||||
the environment value is read just before the provider call and is never added
|
||||
to the JSON body.
|
||||
|
||||
## Explicit Smoke Diagnostics
|
||||
|
||||
Promptkit's explicit smoke diagnostics are separate consumer operations; they
|
||||
are not part of ordinary `Run`, the offline examples, or release validation.
|
||||
With the built-in client, inference diagnostics use the same configured client
|
||||
and `/chat/completions` request as generation. Their content cases are private
|
||||
to Promptkit and are not application prompt, schema, or provider protocol
|
||||
extensions. The public
|
||||
[`Engine.SmokeTest` GoDoc](../../smoke_diagnostics.go) owns request selection,
|
||||
report facts, and injected-client behavior.
|
||||
|
||||
HTTP-only diagnostics use the configured built-in HTTP client. Reachability is
|
||||
one unauthenticated `GET` to the selected diagnostic URL; an observed response
|
||||
counts as reachable even when its status is non-2xx. Authentication behavior
|
||||
is selected from backend metadata, not inferred from the host. The
|
||||
`openrouter_key` adapter makes one `GET` to `/key` relative to the effective
|
||||
backend base, unless configuration supplies a full diagnostic endpoint (see
|
||||
the [backend format reference](../formats.md#built-in-backends)). It may send
|
||||
the available bearer credential, but can dispatch without one. A 401 is
|
||||
recognized as rejection. A 200 is recognized as acceptance only when its JSON
|
||||
object has a `data` object containing string `label`, Boolean `is_free_tier`,
|
||||
and numeric `usage`; unrelated fields and account limits are ignored. This is
|
||||
only a minimal identifying subset of OpenRouter's documented
|
||||
[current-key response](https://github.com/OpenRouterTeam/docs/blob/main/api_reference/limits.mdx#checking-your-limits),
|
||||
not validation of the full account document. Other statuses and unrecognized
|
||||
200 responses do not establish acceptance.
|
||||
|
||||
Diagnostic HTTP evidence describes the last request observed at Promptkit's
|
||||
RoundTripper boundary and any response associated with it. It includes the
|
||||
selected destination without URL user information, response status, and
|
||||
credential presence when observed. Redirects can change the last observed
|
||||
destination or strip credentials. This evidence does not prove that bytes
|
||||
reached a peer or reveal dispatch hidden inside a custom transport. Diagnostic
|
||||
reports do not retain headers, credentials, provider
|
||||
bodies, or generated text. A consumer-injected `LLMClient` can support inference
|
||||
diagnostics, but does not establish HTTP status, destination, reachability, or
|
||||
upstream credential-supply facts; Promptkit does not fall back to a built-in
|
||||
transport for HTTP-only diagnostics.
|
||||
|
||||
## Request Body
|
||||
|
||||
The request body always contains `model` and `messages`. The execution
|
||||
|
||||
@@ -88,6 +88,17 @@ release, including canceled acquisition without dispatch. A retry wait holds
|
||||
the logical admission lease but no active permit. Observer callbacks must not
|
||||
block on work that requires the same run's admission slot.
|
||||
|
||||
## Smoke-Test Capacity
|
||||
|
||||
The engine's smoke tester shares this manager and generation coordinator with
|
||||
the ordinary runner. Inference diagnostics reserve the same logical admission
|
||||
slot and acquire the same backend-scoped active permits for actual model
|
||||
attempts, including retries. They therefore participate in the same configured
|
||||
backend limits as `Run`. Reachability and authentication HTTP probes do not
|
||||
enter generation admission or consume active permits. The public
|
||||
[`Engine.SmokeTest` GoDoc](../../smoke_diagnostics.go) owns the caller-visible
|
||||
operation and injected-client limits.
|
||||
|
||||
## Cancellation And Release
|
||||
|
||||
Admission checks the caller context before reserving a slot. A call canceled
|
||||
@@ -127,4 +138,7 @@ cover shared and independent backend pools during retry and repair. The
|
||||
own the public prepared-capacity boundary. The
|
||||
[root error-boundary tests](../../errors_internal_test.go) own preservation of
|
||||
the public generation category and context identity when generation is
|
||||
canceled.
|
||||
canceled. The assembled
|
||||
[smoke diagnostic contract tests](../../smoke_integration_contract_test.go)
|
||||
verify that inference diagnostics share the engine's Run admission and recover
|
||||
capacity after cancellation or failure.
|
||||
|
||||
@@ -41,9 +41,20 @@ original error chain. This keeps injected attempt classifications consistent
|
||||
with terminal observations without parsing diagnostic text or retrying injected
|
||||
clients.
|
||||
|
||||
The diagnostic path uses the same private attempt and retry implementation as
|
||||
ordinary generation. It retains invocation-local transport, response,
|
||||
generation, usage, capacity-wait, and retry-delay evidence before normal
|
||||
result-discard and cancellation checks. This evidence is returned to diagnostic
|
||||
orchestration; it is not added to public `ExecutionEvent` values or ordinary
|
||||
observer accounting. The [model-client document](llm.md) describes the
|
||||
transport-observation boundary, and the root
|
||||
[`SmokeTestReport` GoDoc](../../smoke_diagnostics.go) owns report semantics.
|
||||
|
||||
Focused coordinator tests use controlled transport and wait functions. They
|
||||
cover eligibility, exhaustion, cancellation, backoff, request freezing, permit
|
||||
release, and the injected-client path. The capacity manager tests own permit
|
||||
release, and the injected-client path. The
|
||||
[evidence tests](../../internal/generation/evidence_test.go) own diagnostic
|
||||
attempt retention and retry evidence. The capacity manager tests own permit
|
||||
acquisition and idempotent release. Public
|
||||
[integration tests](../../reliable_execution_contract_test.go) cover retry,
|
||||
repair, observation, frozen prepared execution, and shared-backend behavior
|
||||
|
||||
@@ -59,6 +59,33 @@ This client does not schedule retries; the [generation coordinator](generation.m
|
||||
does so around its prepared attempts. The client has no tool-call support,
|
||||
provider catalog, inbound HTTP behavior, or durable session store.
|
||||
|
||||
## Diagnostic HTTP Evidence
|
||||
|
||||
Built-in diagnostics wrap the selected HTTP client's `RoundTripper` for one
|
||||
invocation using a shallow client copy, preserving its timeout, redirect policy,
|
||||
cookie jar, and underlying transport without mutating the configured client.
|
||||
Evidence distinguishes a diagnostic dispatch invocation from an observed
|
||||
request and response. A newly observed request replaces the previous
|
||||
destination-specific response evidence, so redirects retain only the last
|
||||
observed hop. A response is recorded before envelope decoding or later context
|
||||
checks can discard a generation result.
|
||||
|
||||
The collector keeps only the last observed destination (without URL user
|
||||
information), status, credential-presence, response-availability, decoded
|
||||
generation, and presence-aware usage facts needed by the diagnostic. It does
|
||||
not retain headers, credentials, raw request or response bodies, or a redirect
|
||||
trace. These are observations at Promptkit's RoundTripper boundary, not proof
|
||||
that bytes reached a peer or that a custom transport did no hidden dispatch.
|
||||
Reachability and named authentication probes use the same configured built-in
|
||||
client but discard their response bodies after local recognition. The
|
||||
[integration contract](../integrations/openai-compatible-chat.md#explicit-smoke-diagnostics)
|
||||
owns their observable HTTP semantics.
|
||||
|
||||
An injected `LLMClient` does not pass through this HTTP boundary. Its returned
|
||||
response can establish response and inference facts, but not an HTTP
|
||||
destination, status, reachability, or upstream credential-supply fact. The
|
||||
facade does not silently install a built-in transport for those diagnostics.
|
||||
|
||||
## Prepared Generation
|
||||
|
||||
For [`RunPrepared`](../../engine.go), the runner supplies the model client with
|
||||
@@ -137,7 +164,10 @@ own configuration, client cloning, deterministic deadline precedence,
|
||||
authentication, request and response mapping, malformed data, error identity,
|
||||
cancellation, endpoint selection and composition, pre-transport rejection, and
|
||||
bounded single-document successful-response framing, closure, and
|
||||
response-body suppression. The focused
|
||||
response-body suppression. The [HTTP evidence tests](../../internal/llm/http_evidence_test.go)
|
||||
own invocation-local last-hop request/response facts, while the
|
||||
[prepared-generation evidence tests](../../internal/llm/prepared_generation_evidence_test.go)
|
||||
own retention across decoding and cancellation. The focused
|
||||
[provider HTTP error tests](../../internal/llm/provider_http_error_test.go)
|
||||
own envelope parsing, normalization, and bounded-reader cases; their
|
||||
[transport tests](../../internal/llm/provider_http_error_transport_test.go)
|
||||
|
||||
@@ -41,6 +41,11 @@ attempts above the first in a generation, and repair generations with at least
|
||||
one dispatch. A terminal event is an aggregate and must not be added to its
|
||||
attempt events.
|
||||
|
||||
Smoke diagnostics mask inherited execution-recorder state while invoking
|
||||
generation or HTTP collaborators. Their private attempt evidence is assembled
|
||||
into a `SmokeTestReport`; it does not create ordinary run events or enter an
|
||||
enclosing run's observation totals.
|
||||
|
||||
The [recorder tests](../../internal/observation/recorder_test.go) own usage
|
||||
presence, phase sums, event ordering, empty-run accounting, identity snapshots,
|
||||
and callback panic behavior. The [generation coordinator](generation.md) owns
|
||||
|
||||
@@ -11,17 +11,17 @@ contributor workflow and validation.
|
||||
|
||||
| Component | Implemented responsibility | References |
|
||||
| --- | --- | --- |
|
||||
| Root `promptkit` package | Provides the supported engine facade, ConfigKit source helpers, profile and backend mapping conveniences, injection options, public requests and results, prompt/profile/requirement inspection and identity listing, concrete-request validation, observation values, opaque prepared-execution handles, value conversion, redacted formatting, and typed capacity and generation error mapping. | [Package GoDoc](../../doc.go), [observations](../../observations.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine facade](../../engine.go), [identity listing](../../listing.go), [requirement facade](../../requirements.go) |
|
||||
| Root `promptkit` package | Provides the supported engine facade, ConfigKit source helpers, profile and backend mapping conveniences, injection options, public requests and results, prompt/profile/requirement inspection and identity listing, concrete-request validation, explicit smoke diagnostics, observation values, opaque prepared-execution handles, value conversion, redacted formatting, and typed capacity and generation error mapping. | [Package GoDoc](../../doc.go), [observations](../../observations.go), [prepared execution](../../prepared_execution.go), [backend API](../../backends.go), [engine facade](../../engine.go), [smoke diagnostics](../../smoke_diagnostics.go), [identity listing](../../listing.go), [requirement facade](../../requirements.go) |
|
||||
| `examples/go-library/prepare` | Demonstrates offline preparation and explicit configuration inspection with a prompt file, programmatic profile, and inline input. It is not a public library package. | [Example program](../../examples/go-library/prepare/main.go) |
|
||||
| `examples/go-library/run` | Demonstrates an offline downstream consumer using a prompt file, programmatic profile, inline input, an injected deterministic model client, and `Run`. It is not a public library package. | [Example program](../../examples/go-library/run/main.go) |
|
||||
| `examples/go-library/observe` | Demonstrates offline built-in HTTP retry and synchronous metadata observation across structured proposal and assessment runs using embedded sources, two profiles on one finite backend, and an application decision. It is not a public library package. | [Example program](../../examples/go-library/observe/main.go) |
|
||||
| `internal/backend` | Holds a defensive, immutable lookup and capacity-policy index for final backend definitions validated by the configuration adapter. | [Backend registry](../../internal/backend/registry.go) |
|
||||
| `internal/backend` | Holds a defensive, immutable lookup of final backend definitions, including diagnostic metadata, plus capacity policies validated by the configuration adapter. | [Backend registry](../../internal/backend/registry.go) |
|
||||
| `internal/catalog` | Selects released keyed catalog assets and PromptKit-owned profile definitions as ConfigKit underlays; effective validation belongs to the configuration loader. | [Catalog adapter](../../internal/catalog/catalog.go), [internal sources](sources.md#profiles-and-built-ins) |
|
||||
| `internal/configuration` | Loads keyed backend, profile, and prompt configuration through ConfigKit; records effective field presence, validates final values, captures indexed prompt definitions, referenced message content, compiled templates, input metadata snapshots, sorted identity-only profile and prompt inventories, backend and presence-aware profile snapshots, and an explicit read-only inspection view. | [Typed loader](../../internal/configuration/load.go), [identity listing](../../internal/configuration/listing.go), [internal sources](sources.md#typed-configuration-loading) |
|
||||
| `internal/configuration` | Loads keyed backend, profile, and prompt configuration through ConfigKit; records effective field presence, validates final values including optional diagnostic metadata, captures indexed prompt definitions, referenced message content, compiled templates, input metadata snapshots, sorted identity-only profile and prompt inventories, backend and presence-aware profile snapshots, and an explicit read-only inspection view. | [Typed loader](../../internal/configuration/load.go), [identity listing](../../internal/configuration/listing.go), [internal sources](sources.md#typed-configuration-loading) |
|
||||
| `internal/capacity` | Owns engine-local bounded execution admission and FIFO model-generation permits for limited backend IDs, including cancellation-safe waiter removal and client wrapping. | [Internal capacity management](capacity.md) |
|
||||
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, requirement inspection, profiles, execution targets, rendering, generation, and validation, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, and output contracts. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [requirement values](../../internal/domain/requirements.go), [endpoint invariant](../../internal/domain/endpoint.go) |
|
||||
| `internal/engine` | Assembles selected catalog and caller configuration into one snapshot, then constructs capacity, rendering, validation, generation, repair, and the use-case runner for each engine. It delegates identity listing directly to the captured configuration inventory rather than the runner. | [Engine assembly](../../internal/engine/engine.go), [internal sources](sources.md#profiles-and-built-ins), [internal runner](runner.md) |
|
||||
| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries, cancellable backoff, and attempt observations. | [Internal generation coordination](generation.md) |
|
||||
| `internal/domain` | Defines internal framework values for requests, artifacts, prompt definitions, requirement inspection, profiles, execution targets, rendering, generation, validation, and source-neutral smoke diagnostics, and owns source-neutral invariants for shared execution settings, OpenAI-compatible base endpoints, session identifiers, and output contracts. Source parsing, required fields, other source-specific normalization, defaulting, and boundary-specific error classification remain with their callers. | [Domain declarations](../../internal/domain/domain.go), [requirement values](../../internal/domain/requirements.go), [endpoint invariant](../../internal/domain/endpoint.go) |
|
||||
| `internal/engine` | Assembles selected catalog and caller configuration into one snapshot, then constructs shared capacity, rendering, validation, generation, repair, the ordinary runner, and the smoke tester for each engine. Built-in engines provide the diagnostic HTTP collaborator; injected-client engines do not. It delegates identity listing directly to the captured configuration inventory rather than the runner. | [Engine assembly](../../internal/engine/engine.go), [internal sources](sources.md#profiles-and-built-ins), [internal runner](runner.md) |
|
||||
| `internal/generation` | Coordinates frozen built-in HTTP attempts and one-shot injected calls through active backend permits, including bounded status retries, cancellable backoff, ordinary attempt observations, and invocation-local evidence for diagnostics. | [Internal generation coordination](generation.md) |
|
||||
| `internal/defaults` | Defines application-neutral framework constants and constructs the default execution target. It contains no CLI, server, or inbound HTTP limits. | [Framework defaults](../../internal/defaults/defaults.go) |
|
||||
| `internal/filecatalog` | Provides deterministic YAML discovery and path helpers for operating-system filesystems and `fs.FS` sources. | [File catalog](../../internal/filecatalog/catalog.go) |
|
||||
| `internal/jsonvalue` | Validates and deeply copies bounded JSON-compatible extra-parameter and prepared-schema trees while preserving supported concrete value types and rejecting cycles or excessive depth and work. | [JSON values](../../internal/jsonvalue/jsonvalue.go) |
|
||||
@@ -30,9 +30,9 @@ contributor workflow and validation.
|
||||
| `internal/prompt` | Renders prompt messages from Go templates with artifact, variable, session, and cache-control data, and conservatively enumerates dependencies from captured template syntax. | [Go-template renderer](../../internal/prompt/go_renderer.go), [requirement discovery](../../internal/prompt/requirements.go) |
|
||||
| `internal/artifact` | Resolves ordinary inline and unrestricted caller-selected file references into copied artifacts with metadata and hashes. | [Internal sources and validation](sources.md) |
|
||||
| `internal/validate` | Validates basic, JSON, and JSON Schema output using operating-system filesystem or `fs.FS` schema sources and creates operation-local validation plans with canonical contained schema resources. | [Framework formats](../formats.md#schemas), [internal sources and validation](sources.md) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded structured non-success response decoding, successful-response decoding, authentication, deadline handling, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||
| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, and synchronous metadata event delivery for ordinary and claimed prepared runs and generation attempts. | [Internal execution observation](observation.md) |
|
||||
| `internal/usecase` | Selects captured prompts and profiles, enumerates requirements from captured prompt syntax, applies request overrides and completion checks, and coordinates ordinary and single-claim prepared execution across rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair. | [Internal runner](runner.md), [requirement inspection](../../internal/usecase/requirements_inspection.go), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
| `internal/llm` | Defines the internal generation boundary and implements outbound OpenAI-compatible chat requests from resolved execution targets, including bounded response decoding, authentication, deadline handling, per-invocation HTTP evidence and built-in diagnostic probes, and ownership of the OpenAI-compatible reserved request-field policy. | [Internal model client](llm.md) |
|
||||
| `internal/observation` | Owns invocation-local run IDs, presence-aware usage and phase accounting, and synchronous metadata event delivery for ordinary and claimed prepared runs and generation attempts; smoke diagnostics keep their private evidence outside ordinary observer events. | [Internal execution observation](observation.md) |
|
||||
| `internal/usecase` | Selects captured prompts and profiles, enumerates requirements from captured prompt syntax, applies request overrides and completion checks, coordinates ordinary and single-claim prepared execution across rendering, artifact loading, operation-local validation plans, generation, capacity, and bounded repair, and runs explicit smoke diagnostics with private cases, shared generation/capacity, and optional built-in HTTP probes. | [Internal runner](runner.md), [requirement inspection](../../internal/usecase/requirements_inspection.go), [prepared-execution implementation](../../internal/usecase/prepared_execution.go) |
|
||||
|
||||
The internal engine assembles these internal components without exposing their
|
||||
representations. Consumers depend only on the root facade.
|
||||
|
||||
@@ -70,6 +70,35 @@ artifact, schema, rendering, admission, or model-client work. The root
|
||||
[`Engine.InspectProfile`](../../engine.go) GoDoc owns the public operation's
|
||||
exact contract.
|
||||
|
||||
## Smoke-Test Diagnostics
|
||||
|
||||
The separate `SmokeTester` in `internal/usecase` selects from the same captured
|
||||
configuration snapshot and backend/profile collaborators as the runner. Explicit
|
||||
profile selection takes precedence over prompt-default selection and a backend
|
||||
diagnostic default; HTTP-only selection does not resolve a default inference
|
||||
profile. Exact public selection rules belong to the root
|
||||
[`Engine.SmokeTest` GoDoc](../../smoke_diagnostics.go).
|
||||
|
||||
Diagnostics do not call `Run`, `Prepare`, or the prompt preparation pipeline.
|
||||
They do not render application messages, load application artifacts, use
|
||||
application schemas, or inspect session templates. Private text and structured
|
||||
cases and the structured case's schema remain internal to the diagnostic
|
||||
operation; they are not entries in prompt/profile listings or inspection.
|
||||
`InspectConfiguration` remains a separate, explicit operator-oriented view of
|
||||
effective settings and source history. A diagnostic report may include selected
|
||||
effective settings and identifiers, so consumers still apply their disclosure
|
||||
policy.
|
||||
|
||||
The engine assembles the smoke tester with the runner's generation coordinator
|
||||
and capacity manager. Inference holds the ordinary logical admission lease and
|
||||
uses the same active permits, retries, timeout, and request evidence path as
|
||||
generation. Reachability and authentication probes bypass generation capacity
|
||||
and do not require a model. An engine with an injected `LLMClient` can run
|
||||
inference diagnostics but has no built-in HTTP probe collaborator; HTTP-only
|
||||
diagnostics report that limitation instead of falling back to another client.
|
||||
The public GoDoc owns the caller-visible timeout, cancellation, report, and
|
||||
transport semantics.
|
||||
|
||||
## Shared Preparation Pipeline
|
||||
|
||||
`Prepare` and `Run` share one private preparation pipeline split at the point
|
||||
@@ -256,4 +285,8 @@ Changes to orchestration should continue to use the existing package
|
||||
interfaces, keep request state local to an invocation, and preserve the shared
|
||||
resolution and completion pipeline. Source, renderer, validator, or
|
||||
model-client contract changes belong first in their owning package and
|
||||
document.
|
||||
document. The
|
||||
[smoke selection tests](../../internal/usecase/smoke_selection_test.go),
|
||||
[diagnostic orchestration tests](../../internal/usecase/smoke_tester_test.go),
|
||||
and [private inference-case tests](../../internal/usecase/smoke_inference_test.go)
|
||||
own diagnostic selection, report assembly, and inference findings.
|
||||
|
||||
@@ -118,8 +118,10 @@ change rendering behavior.
|
||||
Rakestrawhome `v3.20261013.0` assets as one ConfigKit underlay and PromptKit-owned
|
||||
profiles as the next. The selected catalog tags are local candidates and may not
|
||||
be publicly available. It discovers YAML inputs, including each catalog's
|
||||
opt-in diagnostic root, and supplies passive filesystem context without parsing
|
||||
or pre-validating catalog definitions. The configuration loader
|
||||
opt-in `smoke-tests` root in addition to its existing `catalog` root, and
|
||||
supplies passive filesystem context without parsing or pre-validating catalog
|
||||
definitions. Catalog metadata remains separate from existing `/v3` assets for
|
||||
strict-loader compatibility. The configuration loader
|
||||
validates only the final effective backend and profile values, including
|
||||
consumer overlays of maintained bases or backends. There is no catalog-only
|
||||
preflight, reserved source namespace, or replay of backend registration.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Smoke Test Diagnostics Implementation Plan
|
||||
|
||||
Status: ready for implementation.
|
||||
Status: implementation in progress; integrated candidate validation remains.
|
||||
|
||||
Implement the accepted [feature roadmap](smoke-tests.md). This document owns the
|
||||
implementation sequence; the feature roadmap owns the requested API, factual
|
||||
@@ -711,6 +711,18 @@ Validate: local documentation paths/anchors, GoDoc against implemented behavior,
|
||||
and any changed example snippets through focused offline tests. No online link
|
||||
availability or human sign-off gate.
|
||||
|
||||
**Completed: 2026-10-13.** Updated the consumer guide, backend format reference,
|
||||
OpenAI-compatible integration contract, internal component inventory and
|
||||
diagnostic mechanics, and README navigation. The later release note should
|
||||
describe the explicit live `Engine.SmokeTest` API and its operational boundary;
|
||||
the selected catalog candidates `v3.20261013.0` remain local and unpublished
|
||||
(OpenRouter `2a69672b1bb220f1a730153341470f5ac67e15f4`, Rakestrawhome
|
||||
`d40a8aad50111c15d279469170f7b5032ac22529`). Publication-date re-pinning may be
|
||||
needed before a distributable PromptKit release. No release note was rewritten
|
||||
and no catalog or PromptKit publication was attempted. Validation passed:
|
||||
local Markdown paths/anchors, public GoDoc review, `go test . -run Smoke`,
|
||||
`go test ./internal/usecase -run Smoke`, and `git diff --check`.
|
||||
|
||||
## Stage 19: Validate The Integrated Candidate And Record Handoff
|
||||
|
||||
Scope: final verification, completion records, and unintended-file cleanup.
|
||||
|
||||
Reference in New Issue
Block a user