11 Commits

54 changed files with 2347 additions and 215 deletions

View File

@@ -96,7 +96,7 @@ period, prompt version, timezone, and status. Successful output has an absolute
"command": "generate", "command": "generate",
"reportId": "today", "reportId": "today",
"promptId": "weather.today_generated_text", "promptId": "weather.today_generated_text",
"promptVersion": "2.0.0", "promptVersion": "2.1.0",
"runId": "20260529T120000.000000000Z_today", "runId": "20260529T120000.000000000Z_today",
"status": "succeeded", "status": "succeeded",
"timezone": "America/Chicago", "timezone": "America/Chicago",
@@ -106,7 +106,10 @@ period, prompt version, timezone, and status. Successful output has an absolute
When available, the summary also includes the effective `profileId`, When available, the summary also includes the effective `profileId`,
`backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested `backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested
`llmDebugPath`, and compact Distributor `notification` result. It does not `repairAttempts`, `llmDebugPath`, and compact Distributor `notification`
result. `repairAttempts` is `0` when the initial output passed validation,
positive when PromptKit made corrective generation calls, and omitted when
validation did not complete. The summary does not
include historical or transient artifact paths such as metadata, prompt input, include historical or transient artifact paths such as metadata, prompt input,
raw generated text, render context, or notification receipts. raw generated text, render context, or notification receipts.
@@ -115,7 +118,8 @@ raw generated text, render context, or notification receipts.
A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`, A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`,
`total`, `succeeded`, `failed`, and a `reports` array. Each report item includes `total`, `succeeded`, `failed`, and a `reports` array. Each report item includes
its identity, status, effective profile and model details when available, its identity, status, effective profile and model details when available,
source warnings, validation status, and absolute `outputPath` after publication. source warnings, validation status, repair-attempt count when validation
completed, and absolute `outputPath` after publication.
The top-level summary may also contain a batch `notification` object and The top-level summary may also contain a batch `notification` object and
`error`. Batch status is `failed` if any report or the batch notification fails. `error`. Batch status is `failed` if any report or the batch notification fails.
The `total`, `succeeded`, and `failed` counters describe report items only, so The `total`, `succeeded`, and `failed` counters describe report items only, so
@@ -143,7 +147,8 @@ A comparison summary contains these fields in this order: `command`,
successful `results[].reportPath` are absolute. `results` preserves the successful `results[].reportPath` are absolute. `results` preserves the
supplied profile order and each item contains `position`, `profileId`, optional supplied profile order and each item contains `position`, `profileId`, optional
`backendId`, `modelName`, `status`, optional `validationStatus`, optional `backendId`, `modelName`, `status`, optional `validationStatus`, optional
`reportPath`, optional `llmDebugPath`, and optional safe `error`. `repairAttempts`, optional `reportPath`, optional `llmDebugPath`, and optional
safe `error`. The repair-attempt semantics match the generate summary.
The comparison status is `succeeded` only when every selected profile succeeds The comparison status is `succeeded` only when every selected profile succeeds
and the bundle is published. Individual profile failures still publish a and the bundle is published. Individual profile failures still publish a
@@ -157,7 +162,8 @@ published Promptkit category; destination failures use `destination_<kind>`;
and committed cleanup failures use `publication_cleanup` with a message that and committed cleanup failures use `publication_cleanup` with a message that
states whether a complete prior bundle, partial remnants, or no prior bundle states whether a complete prior bundle, partial remnants, or no prior bundle
remains, or that recovery state could not be inspected. It does not expose remains, or that recovery state could not be inspected. It does not expose
provider diagnostics, filesystem causes, or recovery paths. See the provider diagnostics, filesystem causes, or recovery paths. A provider HTTP
failure may include its numeric status in the safe message. See the
[comparison bundle contract](integrations/comparison-bundle.md) for durable [comparison bundle contract](integrations/comparison-bundle.md) for durable
artifact fields and failure invariants. artifact fields and failure invariants.

View File

@@ -203,16 +203,23 @@ for platform availability, security, and retention requirements.
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions. `profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
They are separate decisions. An explicit `profile` applies to every selected They are separate decisions. An explicit `profile` applies to every selected
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
Tomorrow select `weather-balanced` through their exact `2.0.0` prompt Tomorrow select `weather-balanced` through their exact `2.1.0` prompt
definitions. definitions.
Promptkit resolves a selected profile definition from a test or embedding Promptkit resolves a selected profile definition from a test or embedding
consumer's explicit in-memory profile, then the configured `profile_file` or consumer's explicit in-memory profile, then the configured `profile_file` or
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's `profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
built-in catalog. Sources provide complete definitions; fields are never built-in catalog. Weatherreporter's embedded `weather-*` definitions are small
merged. A matching malformed external profile fails rather than using the aliases of Promptkit's maintained base profiles, so Promptkit also resolves
embedded definition. The [Promptkit integration guide](integrations/promptkit.md) their inherited target and settings. A configured definition with the same ID
owns the catalog and precedence details. as either a selected profile or an inherited base takes precedence. A matching
malformed external profile fails rather than using the embedded definition. The
[Promptkit integration guide](integrations/promptkit.md) owns the catalog and
precedence details.
An endpoint-only profile may intentionally have no backend identity. Profiles
that require a direct API key are rejected before collection, while Promptkit
resolves optional environment credential sources during execution.
To replace the default Hourly definition with a local OpenAI-compatible To replace the default Hourly definition with a local OpenAI-compatible
endpoint, set `profile_file` to a copy of endpoint, set `profile_file` to a copy of

View File

@@ -9,7 +9,7 @@ and retention belong to the [operations guide](../operations.md).
## Version And Layout ## Version And Layout
The current and only supported manifest schema version is The current and only supported manifest schema version is
`weatherreporter.comparison.v1`. A bundle directory contains exactly these `weatherreporter.comparison.v2`. A bundle directory contains exactly these
regular, non-symlinked files: regular, non-symlinked files:
```text ```text
@@ -51,7 +51,7 @@ this order:
```text ```text
position, profileId, backendId, modelName, status, validationStatus, position, profileId, backendId, modelName, status, validationStatus,
reportPath, error repairAttempts, reportPath, error
``` ```
`startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not `startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not
@@ -66,12 +66,16 @@ contiguous from one, profile IDs are distinct and nonblank, and
`succeeded + failed == total`. `succeeded + failed == total`.
A successful result has `status: "succeeded"`, `validationStatus: "passed"`, A successful result has `status: "succeeded"`, `validationStatus: "passed"`,
and a non-negative `repairAttempts` count,
a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for
its position, total, and logical profile ID, and no `error`. A failed result has its position, total, and logical profile ID, and no `error`. A failed result has
`status: "failed"`, no `reportPath`, and an `error` object with nonblank `status: "failed"`, no `reportPath`, and an `error` object with nonblank
`category` and `message`. Its validation status is absent, `failed`, or `category` and `message`. Its validation status is absent, `failed`, or
`skipped`. Error messages are valid UTF-8 and no longer than 1,024 bytes. `skipped`; it may also be `passed` when a WeatherReporter step after PromptKit
`backendId` and `validationStatus` are omitted when unavailable. validation failed. Error messages are valid UTF-8 and no longer than 1,024 bytes.
`backendId` and `validationStatus` are omitted when unavailable. A failed
result with any completed validation status must retain its non-negative
`repairAttempts`; early operational failures omit both fields.
Every successful Markdown file is declared by exactly one successful result. Every successful Markdown file is declared by exactly one successful result.
The directory contains no extra entries. Consumers can therefore verify the The directory contains no extra entries. Consumers can therefore verify the
@@ -88,7 +92,9 @@ case-variant, or duplicate fields; multiple JSON values; extra entries;
symlinks; and future or otherwise unsupported versions. Treat a bundle that symlinks; and future or otherwise unsupported versions. Treat a bundle that
fails recognition as an ordinary directory, not as a compatible bundle. fails recognition as an ordinary directory, not as a compatible bundle.
When replacing a recognized bundle, cancellation observed before the new Only v2 is recognized as a replaceable bundle; v1 is unsupported and must be
moved or removed before a replacement at the same destination. When replacing
a recognized bundle, cancellation observed before the new
bundle is installed preserves the prior bundle rather than committing the bundle is installed preserves the prior bundle rather than committing the
replacement. replacement.

View File

@@ -1,10 +1,12 @@
# Promptkit Integration # Promptkit Integration
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.0.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`. Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.1.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
## Logical Profile Catalog ## Logical Profile Catalog
Prompt definitions select a stable Weatherreporter profile ID. The embedded definitions currently use Promptkit's `openrouter` backend: Prompt definitions select a stable Weatherreporter profile ID. Each embedded
definition contains only its ID and one Promptkit base-profile reference; the
effective execution settings resolve from Promptkit's maintained catalog:
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports | | Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
| --- | --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
@@ -12,22 +14,42 @@ Prompt definitions select a stable Weatherreporter profile ID. The embedded defi
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow | | `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None | | `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded profiles intentionally omit endpoints, credentials, temperature, `top_p`, and output-token limits. The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
profiles intentionally omit endpoints, credentials, and execution settings;
Promptkit owns inherited resolution and its provider-native defaults.
Promptkit's built-in `rakestrawhome-gemma-4-31b` is also available for ordinary
and comparison selection and reports the `rakestrawhome` backend without
Weatherreporter-specific configuration.
## Selection And Active Execution ## Selection And Active Execution
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank backend and model identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order: Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank model; an endpoint-only profile may intentionally have no backend identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
1. explicit in-memory profiles used by an embedding consumer or test; 1. explicit in-memory profiles used by an embedding consumer or test;
2. the configured `profile_file` or `profile_dir`; 2. the configured `profile_file` or `profile_dir`;
3. Weatherreporter's embedded fallback profiles; and 3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's built-in catalog. 4. Promptkit's built-in catalog.
A source falls through only when the selected ID is absent. Each source supplies a complete definition, so profile fields are not merged. A malformed matching operator definition is an error and does not fall back. A source falls through only when the selected ID is absent. Promptkit resolves a
derived profile's base with the same source precedence, so a configured base
can shadow a built-in base. A missing, cyclic, malformed, or incomplete
selected inheritance chain is an error and does not fall back.
Profiles that require a direct API key are unsupported; a profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. Active results retain the selected logical profile ID and resolved backend and model. Ordinary errors, summaries, logs, and outputs exclude endpoints, credentials, rendered messages, schemas, request bodies, response bodies, and complete parameter maps. Profiles that require a direct API key are unsupported. Optional environment
credential sources are Promptkit runtime concerns and are not checked by
Weatherreporter during profile inspection. Active results retain the selected
logical profile ID and resolved backend and model. Ordinary errors, summaries,
logs, and outputs exclude endpoints, credentials, rendered messages, schemas,
request bodies, response bodies, and complete parameter maps.
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions. Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions. Ordinary generation errors disclose only the safe Weatherreporter category and optional HTTP status; provider code, type, and message are written only to the explicit secure failure-debug artifact.
Each embedded prompt permits one Promptkit-owned corrective generation after an
eligible failed or explicitly empty result. This is not an application retry:
Weatherreporter performs no provider retry, profile fallback, or request-level
output-contract override. Promptkit reports cumulative usage and the actual
number of corrective calls; repair exhaustion remains a completed validation
failure.
When capture is enabled, its preparation artifact projects a provider endpoint When capture is enabled, its preparation artifact projects a provider endpoint
to its scheme and host and retains only reviewed execution settings. Provider to its scheme and host and retains only reviewed execution settings. Provider

View File

@@ -7,7 +7,7 @@ is owned by the [CLI reference](../cli.md) and [operations guide](../operations.
## Single-Report Flow ## Single-Report Flow
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. The resolved profile, backend, and model are carried in the active result. `GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. Profile inspection requires a model, permits an empty backend identity for endpoint-only profiles, and leaves inherited resolution and optional credential sources to Promptkit. The resolved profile, backend, model, and actual repair count (once a completed execution exists) are carried in the active result; the configured repair budget remains part of the exact prompt contract.
The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run. The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run.

View File

@@ -21,8 +21,9 @@ Before accepting generated JSON, the execution boundary reconciles the prepared
report definition, inspected prompt hash and selected profile identity, the one report definition, inspected prompt hash and selected profile identity, the one
preparation callback, and the completed Promptkit result. The callback and preparation callback, and the completed Promptkit result. The callback and
completion must agree on prompt, profile, backend, model, and rendered/input completion must agree on prompt, profile, backend, model, and rendered/input
hashes; the callback output and completed validation must name the prepared hashes; the callback output must also carry the prepared report's configured
report's JSON Schema. A mismatch produces no rendered Markdown and leaves repair budget, while the completed validation records the actual corrective
calls used within that budget. A mismatch produces no rendered Markdown and leaves
results with only the inspected safe identity. results with only the inspected safe identity.
Single-report generation executes one prepared profile and publishes its Single-report generation executes one prepared profile and publishes its

View File

@@ -2,9 +2,9 @@
`internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter. `internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter.
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, the embedded fallback catalog, and its built-in catalog; the adapter does not parse profile YAML, merge sources, or probe endpoints. The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, embedded profile aliases and their bases, and its built-in catalog; the adapter does not parse profile YAML, resolve inheritance, merge sources, inspect optional environment credentials, or probe endpoints.
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file. The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. An empty backend identity remains valid for an endpoint-only profile; a nonblank model is required. PromptKit's configured repair-call budget and the completed result's actual corrective-call count are retained, along with its cumulative provider usage and final candidate. Structured provider generation failures become project-owned redacted generation errors that retain only bounded details through explicit accessors. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications. The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications.

View File

@@ -122,7 +122,9 @@ The destination is preflighted before prompt inspection and collection, then
rechecked immediately before an atomic publish. A missing or empty directory rechecked immediately before an atomic publish. A missing or empty directory
is usable. A nonempty directory can be replaced only when `--replace` is given is usable. A nonempty directory can be replaced only when `--replace` is given
and it is recognized as a current Weatherreporter comparison bundle; ordinary and it is recognized as a current Weatherreporter comparison bundle; ordinary
directories, symlinks, and unsafe destinations are rejected. Cancellation and directories, symlinks, and unsafe destinations are rejected. Existing v1
bundles are not recognized for replacement: move or remove them first.
Cancellation and
all failures before publication preserve an existing bundle, including a all failures before publication preserve an existing bundle, including a
cancellation observed while a replacement is being prepared. If guarded cancellation observed while a replacement is being prepared. If guarded
restoration cannot complete, the error names the retained sibling bundle for restoration cannot complete, the error names the retained sibling bundle for
@@ -176,6 +178,12 @@ Preparation captures retain only the provider endpoint origin and reviewed
execution settings. URL user information, paths, queries, fragments, and execution settings. URL user information, paths, queries, fragments, and
unrecognized provider parameters are omitted. unrecognized provider parameters are omitted.
Each run directory may contain `preparation.json` (v3), `execution.json` (v3),
and, for a provider generation failure, `failure.json` (v1). The failure
artifact retains the safe category, HTTP status, and provider code, type, and
message for trusted debugging only. Ordinary command output never includes
those provider details.
Capture writes are confined to the requested root and fail if an unsafe Capture writes are confined to the requested root and fail if an unsafe
filesystem component prevents secure artifact creation. filesystem component prevents secure artifact creation.

View File

@@ -50,10 +50,14 @@ directly.
- Prompts receive curated module packages, never unbounded raw weather payloads. - Prompts receive curated module packages, never unbounded raw weather payloads.
- Every execution validates the exact prompt version and output contract before - Every execution validates the exact prompt version and output contract before
collection. The selected profile is configured explicitly or declared by the collection. The selected profile is configured explicitly or declared by the
prompt; unsupported direct-key profiles and missing reported credentials fail prompt; profiles requiring unsupported direct API keys fail before collection.
before collection. A profile may have an empty backend identity when it supplies an endpoint;
PromptKit resolves inherited profiles and optional credential sources when it
executes them.
- Prompt and profile validation completes before weather collection. Raw output - Prompt and profile validation completes before weather collection. Raw output
is validated before template rendering. is validated before template rendering.
- PromptKit may make at most the prompt contract's one corrective generation;
exhaustion is a validation rejection, not an application-level retry.
- Comparison validates every explicit profile before collection, prepares one - Comparison validates every explicit profile before collection, prepares one
immutable report input, and delegates backend capacity to Promptkit rather immutable report input, and delegates backend capacity to Promptkit rather
than adding an application-wide execution limit. than adding an application-wide execution limit.
@@ -62,6 +66,8 @@ directly.
- Sensitive rendered prompts, schemas, input bodies, provider endpoints, and - Sensitive rendered prompts, schemas, input bodies, provider endpoints, and
credentials never enter normal summaries or logs. They are written only to credentials never enter normal summaries or logs. They are written only to
an explicit secure debug root when requested. an explicit secure debug root when requested.
- Provider-controlled diagnostics never enter ordinary outputs; they are
retained only in explicit secure failure-debug artifacts.
## Output, Notification, And Testing Invariants ## Output, Notification, And Testing Invariants

View File

@@ -0,0 +1,595 @@
# PromptKit v0.8.0 Upgrade Implementation Plan
Status: Complete.
Completion note: Stages 19 upgraded PromptKit, adopted inherited profiles and
current credential handling, added repair and provider-failure contracts,
enabled one corrective generation for v2.1.0 prompts, exposed repair
provenance in ordinary and comparison results, migrated comparison bundles to
v2, and added secure failure-debug capture.
## Purpose And Authority
This plan translates the accepted
[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered,
decision-complete implementation procedure. The feature roadmap owns purpose,
scope, policy, and the desired end state. This document owns implementation
order, concrete work allocation, stage boundaries, and verification until the
upgrade is complete.
The implementing agent must complete the stages in numerical order. Each stage
is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning.
Do not combine stages merely because adjacent work touches the same package.
## Locked Decisions
The following decisions are final for this implementation:
- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`;
- declare one corrective call in each embedded prompt through
`repair_attempts: 1` rather than adding WeatherReporter repair logic;
- keep the repair budget in the exact embedded prompt definition and add no
global, per-report, CLI, profile, or operator configuration override;
- advance all four exact prompt versions from `2.0.0` to `2.1.0`;
- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases
of the corresponding PromptKit built-ins through `base_profile`;
- retain the existing report-to-profile assignments and effective model
ladder;
- allow successfully inspected endpoint-only profiles to have an empty backend
ID while continuing to require a nonblank model;
- treat `APIKeyEnv` as an optional lookup source and reject only profiles that
report `APIKeyRequired`, because WeatherReporter supplies no direct request
credential;
- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without
WeatherReporter-specific backend configuration;
- expose provider HTTP status through a project-owned safe generation error,
while writing provider code, type, and message only to explicit secure debug
capture;
- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not
preserve v1 guarded-replacement support; and
- retain PromptKit dependency types inside the adapter and preserve all
stateless execution, atomic publication, comparison independence, and
disclosure invariants.
## Implementation Rules
For every stage:
- read `docs/development.md`, all files under `docs/policy/`, this plan, the
feature roadmap, and the task-specific documents named by the stage;
- inspect the current code and tests before editing; use the repository's code
knowledge graph first for code discovery and fall back to text search for
literals, assets, and documentation;
- implement only the stage's scope and preserve unrelated user changes;
- keep PromptKit/provider types, client construction, YAML parsing, repair
mechanics, and provider transport inside the existing adapter boundary;
- use deterministic, offline, credential-free tests and injected clients or
synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local
endpoint calls;
- add tests at the narrowest stable owner identified by the testing policy and
avoid copying PromptKit's internal test matrices;
- update the canonical documentation owners listed for that stage in the same
change as the implemented contract;
- run `gofmt` on changed Go files, the stage's focused tests,
`GOWORK=off go test -count=1 ./...`, and `git diff --check`; and
- leave the repository passing before proceeding to the next stage.
Stages affecting concurrent comparison, cancellation, or secure debug
filesystem work must also run the named focused packages with `-race`. Do not
weaken an existing assertion solely to accommodate the new dependency. When a
test encodes an intentionally changed contract, replace it with a behavioral
assertion for the accepted policy.
## Implementation Stages
### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline
Status: Complete.
Purpose: move to the tagged dependency and isolate compatibility changes before
adopting new WeatherReporter behavior.
Work:
1. Update `go.mod` to require
`gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with
`GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace`
directive and do not change WeatherReporter's Go version.
2. Resolve any compile failures using PromptKit's public root package only.
Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not
register the now-reserved `rakestrawhome` backend.
3. Reconcile adapter tests that directly exercise PromptKit's changed optional
credential behavior. A profile whose only credential metadata is
`api_key_env` must reach an injected client when the environment value is
absent; it must no longer expect PromptKit to return
`ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight
in this stage.
4. Verify that every current embedded prompt, content file, schema, and fallback
profile inspects under v0.8.0. Verify selected invalid local endpoints and
malformed selected profile definitions still map to project-owned
configuration or profile-load categories.
5. Review the v0.6.0 compatibility corrections against supported
WeatherReporter inputs: metadata-authoritative identities, exact contained
`content_file` paths, regular embedded files, structurally valid endpoints,
bounded JSON-compatible values, cancellation identity, and strict response
framing. Add consumer tests only for a WeatherReporter boundary not already
protected by PromptKit.
Do not enable profile inheritance or output repair yet. The expected result is
the current WeatherReporter feature set running against PromptKit v0.8.0.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
```
### Stage 2: Adopt Profile Inheritance And Current Credential Routing
Status: Complete.
Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional
credential semantics, and the Rakestrawhome built-in without changing the model
ladder.
Work:
1. Replace the three embedded profile bodies with these exact leaf/base
relationships and no duplicated execution settings:
| Leaf | Base |
| --- | --- |
| `weather-light` | `deepseek-4-flash` |
| `weather-balanced` | `gemini-flash-latest` |
| `weather-deep` | `claude-sonnet-latest` |
2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert
that all three leaf IDs remain selected identities and resolve to the same
backend, model, timeout, service tier, and reasoning settings exposed by the
current standalone definitions. Test relationships and effective behavior,
not copied private constants beyond the intentional model-ladder contract.
3. Preserve source precedence. Cover a standalone same-ID operator override, a
derived operator override, a configured source that shadows a base ID, and
selected missing-base, cyclic, malformed-base, and incomplete-target
failures. Do not implement inheritance or merging in WeatherReporter; all
resolution must remain PromptKit-owned.
4. Change application profile preflight to accept a successful inspection with
a nonblank model and an empty backend ID. Trust PromptKit inspection to have
resolved either a backend or endpoint; do not add the endpoint to
`promptexec.ProfileInspection` or ordinary provenance.
5. Remove application-level environment lookup and rejection for a nonblank
`APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from
prompt, batch, and comparison inspection requests. Continue rejecting
`CredentialRequired`/`APIKeyRequired` before weather collection with the
existing missing-credential category.
6. Add an end-to-end offline regression proving the maintained endpoint-only
`weather-light` example passes application inspection, retains an empty
backend ID, and does not expose its endpoint.
7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile
inspection through the existing adapter and reports the PromptKit
`rakestrawhome` backend ID. Do not make a provider call or add
Rakestrawhome-specific configuration.
Canonical documentation in this stage:
- update `docs/policy/architecture.md` so only direct-key-required profiles
fail credential preflight and backend identity is optional for endpoint-only
profiles;
- update `docs/config.md` to distinguish same-ID source replacement from
`base_profile` chain inheritance and to describe optional environment
credentials;
- update `docs/integrations/promptkit.md` for profile composition, parent
lookup precedence, endpoint-only identity, optional credentials, and
Rakestrawhome availability; and
- update `docs/internal/promptkit-adapter.md` and focused app internals for the
implemented inspection behavior.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
```
### Stage 3: Extend The Project-Owned Prompt Execution Contract
Status: Complete.
Purpose: establish dependency-neutral repair and structured-generation-error
values before the adapter or application relies on them.
Work:
1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured
additional-call budget from the exact prompt contract.
2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of
corrective calls actually started for the completed result. Update
`NewValidation` and every caller so construction is explicit; reject or
normalize no values here because PromptKit owns output-contract validity.
3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so
repair values are retained without sharing mutable state.
4. Add a project-owned immutable `promptexec.GenerationError` with unexported
status and provider-detail fields plus safe accessors:
- `StatusCode() int`
- `ProviderCode() string`
- `ProviderType() string`
- `ProviderMessage() string`
- `Category() ErrorCategory`, always returning `Generation`
- `Error()`, exposing only the WeatherReporter generation category/message
and optional HTTP status
- `GoString()`, returning the same safe representation
- `Unwrap()`, preserving a project-owned `*promptexec.Error`
5. Provide one constructor used by adapters. Defensively normalize valid UTF-8
and bound code/type to 256 Unicode code points and message to 4,096 Unicode
code points, even though PromptKit already bounds its accessors. Do not
expose fields through struct formatting, JSON tags, or exported mutable
fields. Preserve the dependency cause only behind the project-owned error so
`errors.Is`/`errors.As` identities remain available without entering error
text.
6. Add focused tests proving nil/zero safety, category and unwrap behavior,
status-only ordinary formatting, `%#v` redaction, provider-detail bounds,
and repair-value copying.
Do not import PromptKit from `internal/promptexec` and do not change CLI or
artifact schemas in this stage.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptexec
```
### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter
Status: Complete.
Purpose: make the adapter faithfully translate v0.8.0 preparation, execution,
validation, usage, and failure values into the Stage 3 contract.
Work:
1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and
prepared-execution details. Map
`promptkit.ValidationResult.RepairAttempts` in completed execution.
2. Preserve PromptKit's cumulative usage exactly as reported across the initial
call and every completed correction. Continue returning only the final raw
candidate and final validation result, subject to WeatherReporter's 64 KiB
generated-output bound.
3. In adapter error classification, retain cancellation, deadline, and capacity
precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for
`*promptkit.GenerationError` and construct the project-owned
`promptexec.GenerationError` with status, code, type, message, and hidden
cause. Initial and corrective generation failures use the same mapping.
4. Extend the injected adapter client used by tests so it can return an ordered
sequence of responses or errors and record each request safely.
5. Use a synthetic PromptKit prompt with JSON Schema validation and
`repair_attempts: 1` to cover:
- first-pass valid output with zero corrections;
- explicitly empty or invalid output followed by valid corrected output;
- one-attempt exhaustion returning a final failed validation result rather
than an operational error;
- a non-2xx-style `GenerationError` during correction;
- cumulative token usage and actual repair count; and
- the same prepared prompt/profile identity across the corrective flow.
6. Keep these tests at the adapter boundary. Do not assert PromptKit's private
corrective-message wording or reconstruct its internal repair algorithm.
Canonical documentation in this stage: update
`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do
not yet claim that embedded WeatherReporter prompts enable repair.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
```
### Stage 5: Carry Repair Provenance Through Application Workflows
Status: Complete.
Purpose: make application orchestration understand configured and actual repair
counts before changing the embedded prompt policy.
Work:
1. Add an expected generated-text repair budget to `report.Definition` and set
it explicitly to zero for all four current `2.0.0` definitions in this
stage. Include it in report-definition validation and retained contract
tests.
2. Extend exact prompt preflight so format, validation mode, schema path, and
repair budget must all match the resolved report definition. Extend
preparation and completion provenance checks to require the same repair
budget across inspection and the opaque prepared snapshot.
3. Add `RepairAttempts *int` to application outcomes where execution may fail
before validation exists. Set it to a fresh pointer immediately after a
non-nil completed execution is returned, before WeatherReporter's secondary
generated-text validation. A pointer is required so completed first-pass
zero is distinguishable from unavailable provenance.
4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch
conversion, comparison execution's internal outcome, and relevant test
fakes. Do not expose the new value in CLI or comparison JSON yet.
5. Preserve the actual count on PromptKit validation rejection and on later
WeatherReporter generated-text or render failures. Leave it unavailable on
preparation, capacity, cancellation, deadline, and generation errors that
return no completed PromptKit result.
6. Add focused tests for provenance mismatch, completed zero, completed
positive, validation rejection, later local validation failure, early
operational failure, batch copying, and independent concurrent profile
outcomes.
Canonical documentation in this stage: update the focused prepared-report and
app-orchestration internals to describe configured versus actual repair
provenance. Current public documents should continue to report the embedded
budget as zero until Stage 6.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/app
```
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
Status: Complete.
Purpose: switch the operational prompts to the accepted one-correction policy
and make ordinary generate/run/batch output report what occurred.
Work:
1. Add `repair_attempts: 1` to the output contract of all four embedded prompt
definitions and change each exact prompt version from `2.0.0` to `2.1.0`.
Do not change prompt text or generated-text schemas solely for this upgrade.
2. Change all four report registry definitions to exact prompt version `2.1.0`
and expected repair budget one. Update exact-version fixtures and assertions
throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests
that classify `repair_attempts` as a retired setting and replace them with
an exact one-attempt contract assertion.
3. Add `repairAttempts` to successful and failed generate and batch JSON result
shapes through the Stage 5 pointers. Emit integer zero for a completed
first-pass result, a positive integer for a completed repaired result, and
omit the field when no completed validation made it available.
4. Keep the existing `validationStatus` and failure categories authoritative.
A repaired valid result proceeds normally. Repair exhaustion remains
`validation_rejected`, publishes no report for that profile, and retains the
actual attempt count.
5. Add representative offline assembled tests proving first-pass success,
repaired success, exhaustion, explicit empty initial content, and batch
result propagation. Reuse the real PromptKit adapter with an injected
sequence client for at least one end-to-end repaired execution; use the
existing app fake at other boundaries where lower-level repair is already
covered.
6. Confirm no application loop, provider retry, profile fallback, or
request-level `OutputContract` override was introduced.
Canonical documentation in this stage:
- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and
failed-exhaustion invariants;
- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one
configured repair, actual-count semantics, cumulative usage, explicit-empty
handling, and the distinction from operational retries;
- update `docs/cli.md` for generate and batch `repairAttempts` fields;
- update `docs/internal/report-registry.md`, prepared-report internals, and app
orchestration internals for exact version and repair flow; and
- keep configuration documentation unchanged because no repair setting is
added.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
```
### Stage 7: Migrate Comparison Bundles To v2
Status: Complete.
Purpose: preserve repair activity in the profile-evaluation artifact and make
the strict durable schema change explicit.
Work:
1. Change `comparison.SchemaVersion` to
`weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a
v1 parser or guarded-replacement compatibility path.
2. Add `RepairAttempts *int` to each application comparison profile result,
CLI comparison profile summary, and durable `comparison.Result`. Propagate a
fresh copy from Stage 5's execution outcome.
3. Place `repairAttempts` immediately after `validationStatus` in the canonical
result-object JSON field order. Encode zero for completed first-pass
validation, a positive integer for completed correction, and omit it only
when no completed validation exists.
4. Tighten manifest invariants: every non-nil repair count is non-negative; a
successful result must have `validationStatus: "passed"` and a non-nil
repair count; a failed result with a completed validation status must also
have a non-nil count; and an early operational failure may omit both.
5. Update the strict token-level JSON recognizer to accept only the canonical
`repairAttempts` field at its correct object level, reject duplicate,
unknown, negative, fractional, string, overflow, and malformed values, and
continue rejecting v1 as an unsupported current bundle.
6. Update manifest construction, cloning, validation, exact serialization
tests, guarded replacement tests, malicious bundle tests, partial-success
tests, and CLI comparison summaries. Preserve flat layout, result ordering,
hashes, atomic publication, cancellation safety, and no Distributor calls.
7. Cover concurrent peers where one succeeds first-pass, one repairs, one
exhausts, and one fails operationally. The counts must remain attached to
the selected profile positions without races or cross-contamination.
Canonical documentation in this stage:
- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2,
including exact field order, presence rules, and the lack of v1 replacement
compatibility;
- update `docs/cli.md` for comparison `repairAttempts`;
- update `docs/operations.md` to tell operators to move or remove an existing
v1 bundle before replacing at the same destination; and
- update comparison execution/publication internals and architecture policy as
needed for the current-only version invariant.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app
```
### Stage 8: Add Secure Provider-Failure Debug Capture
Status: Complete.
Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the
existing explicit secure debug boundary while keeping ordinary errors safe.
Work:
1. Extend prompt preparation debug output with configured
`repairAttempts` and advance its schema identifier from
`weatherreporter.prompt_preparation_debug.v2` to
`weatherreporter.prompt_preparation_debug.v3`.
2. Extend execution validation debug output with actual `repairAttempts` and
advance its schema identifier from
`weatherreporter.prompt_execution_debug.v2` to
`weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage.
3. Add a dedicated `failure.json` artifact with schema identifier
`weatherreporter.prompt_failure_debug.v1`. Its canonical fields are:
- top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`;
- failure object: `category`, `statusCode`, `providerCode`, `providerType`,
`providerMessage`;
- omit absent provider fields and zero status; and
- never include the raw provider body, headers, endpoint, credentials,
request, schema, rendered prompt, or generated candidate.
4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative
secure run directory, `0700` directory and `0600` file modes, canonical JSON
encoding, and no-follow/atomic replacement behavior. Disabled writers must
perform no filesystem work.
5. When execution returns an error, use `errors.As` only against the
project-owned `*promptexec.GenerationError`. If explicit debug capture is
enabled, write `failure.json` using that profile's existing debug reference.
This applies equally to initial and corrective provider failures and keeps
comparison profile directories isolated.
6. If failure-debug writing also fails, retain the generation failure as the
primary categorized error and join the safe debug-write failure rather than
replacing or hiding the provider failure. Never place provider code, type,
or message in the joined error text.
7. Ordinary generate, batch, and comparison errors should gain only the safe
HTTP status already rendered by `promptexec.GenerationError.Error`; do not
add provider detail fields to CLI summaries, comparison manifests, logs, or
Distributor requests.
8. Add adversarial tests for formatter redaction, malicious provider strings,
JSON escaping, bounds, absent fields, file modes, symlink/path attacks,
write failure, cancellation identity, initial versus corrective failures,
and concurrent comparison captures.
Canonical documentation in this stage:
- update `docs/operations.md` with the three debug artifact versions,
`failure.json`, sensitivity, permissions, and retention;
- update `docs/integrations/promptkit.md` with ordinary status-only disclosure
and debug-only provider detail;
- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and
- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled
diagnostics from ordinary outputs.
Focused verification:
```sh
GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli
GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app
```
### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit
Status: Complete.
Purpose: verify the complete end state as one coherent WeatherReporter feature
and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile,
repair, or debug claims.
Work:
1. Re-read the feature roadmap, `docs/development.md`, every policy document,
and every canonical document changed by Stages 1-8. Reconcile them against
executable behavior and remove duplicated or stale definitions. Keep
unimplemented future ideas in `docs/roadmap/future.md`, not current-state
documents.
2. Search code, embedded assets, examples, tests, and documentation for stale
contractual literals and review every occurrence of:
- PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim;
- prompt version `2.0.0`;
- `weatherreporter.comparison.v1`;
- prompt debug schema v2 identifiers;
- claims that profile fields never inherit;
- claims that every `APIKeyEnv` must be populated;
- claims that backend ID is always required;
- claims that repair is disabled or `repair_attempts` is retired; and
- provider detail in ordinary output.
Historical release documents may retain accurate historical literals.
3. Verify canonical ownership:
- architecture owns invariants and boundaries;
- config owns operator profile and credential behavior, but no repair field;
- PromptKit integration owns logical prompt/profile/output contracts;
- CLI owns result fields;
- operations owns explicit debug handling and old comparison-bundle cleanup;
- comparison integration owns the complete v2 manifest; and
- internal documents own implementation flow without duplicating the public
references.
4. Verify maintained examples remain valid, secret-free, and tested. The local
`weather-light` example remains a standalone endpoint-only profile rather
than inheriting an OpenRouter backend it cannot clear.
5. Review the complete diff for architecture leakage. Production packages
outside `internal/adapters/promptkit` must not import PromptKit; no
application repair loop, provider client, raw provider diagnostic, profile
YAML parser, or durable application state may have appeared.
6. Review tests under the testing policy. Keep consumer contract and regression
coverage, remove accidental duplication of upstream implementation tests,
and ensure every default test is offline and repeatable.
7. Run the complete validation set:
```sh
gofmt -w <all changed Go files>
GOWORK=off go test -count=1 ./...
GOWORK=off go test -race -count=1 ./...
GOWORK=off go vet ./...
GOWORK=off go build ./...
GOWORK=off go mod tidy -diff
go run ./cmd/weatherreporter --help
go run ./cmd/weatherreporter compare --help
test -z "$(git ls-files go.work go.work.sum)"
test ! -e vendor
git diff --check
```
8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly
v0.8.0, and no live credential or provider call occurred during validation.
9. After every check passes, update this plan's status to Completed and add a
concise completion note listing the implemented stages. Do not delete either
roadmap until the maintainer has reviewed the implementation. Do not create
a release document or tag; release preparation remains a separate maintainer
action once a version is selected.
## Completion Standard
The implementation is complete only when all nine stages pass their focused
and repository-wide checks, all locked decisions are observable in code and
canonical documentation, and the feature roadmap's completion criteria are
satisfied. Passing compilation alone is insufficient. The final state must
demonstrate repaired success, repair exhaustion, comparison provenance,
endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome
inspection, safe ordinary provider failures, secure debug-only detail, and
unchanged publication and concurrency invariants.

View File

@@ -0,0 +1,488 @@
# PromptKit v0.8.0 Upgrade Roadmap
Status: Implemented.
## Purpose
WeatherReporter should upgrade its PromptKit dependency from `v0.5.0` to
`v0.8.0` and deliberately adopt the useful consumer-facing capabilities added
in `v0.6.0`, `v0.7.0`, and `v0.8.0`. The upgrade should improve output-contract
reliability, profile composition, local and alternate endpoint support, and
provider-failure diagnosis without moving PromptKit responsibilities into
WeatherReporter or weakening the application's stateless and security
boundaries.
This roadmap defines the intended scope, policy, and end state. The
[implementation plan](implementation.md) owns the procedure for reaching that
state.
## User Intent
The upgrade is intended to:
- use PromptKit's bounded output repair to recover from occasional malformed
structured weather prose;
- keep WeatherReporter's domain profile IDs stable while inheriting maintained
PromptKit model definitions;
- make PromptKit's additional built-in backend and profile available for
explicit generation and profile comparisons;
- support unauthenticated or optionally authenticated OpenAI-compatible
endpoints without inventing a WeatherReporter transport layer;
- make provider HTTP failures more actionable under an explicit
WeatherReporter disclosure policy; and
- receive PromptKit's intervening correctness, safety, cancellation, resource,
and efficiency improvements as part of one tested dependency upgrade.
The model ladder and report assignments do not change as part of this work:
Hourly continues to select `weather-light`; Daily, Today, and Tomorrow continue
to select `weather-balanced`; and `weather-deep` remains available for explicit
selection. This upgrade does not promote the new Rakestrawhome profile into
that default ladder.
## Current State
WeatherReporter currently depends on
`gitea.maximumdirect.net/eric/promptkit` at `v0.5.0`. The PromptKit adapter
supplies embedded prompts, JSON Schemas, and
application-fallback profiles, plus an optional configured profile source and
the conventional local backend.
The four generated-text prompts are exact version `2.0.0` JSON Schema prompts.
They omit `repair_attempts`, so execution is single-pass. The project-owned
`promptexec.OutputContract` and validation result also omit repair budgets and
actual repair counts.
The three embedded WeatherReporter profiles duplicate the effective fields of
these PromptKit built-ins:
| WeatherReporter profile | PromptKit built-in with the same target |
| --- | --- |
| `weather-light` | `deepseek-4-flash` |
| `weather-balanced` | `gemini-flash-latest` |
| `weather-deep` | `claude-sonnet-latest` |
WeatherReporter preflights any nonblank `api_key_env` as a required credential,
even though PromptKit v0.7.0 distinguishes an optional environment source from
an explicit `APIKeyRequired` target. Provider generation failures are reduced
to WeatherReporter's safe `generation` category; the PromptKit dependency error
is retained as a hidden cause, but its structured HTTP status and provider
diagnostics are not mapped into project-owned values.
The maintained `weather-light` local override is an endpoint-only profile, and
the configuration contract says endpoint-only profiles are supported. PromptKit
inspection correctly reports no backend ID for that form, but WeatherReporter
application preflight currently requires both a nonblank backend and model.
That mismatch prevents the documented example from reaching generation and
should be corrected as part of adopting the current PromptKit target contract.
## Upstream Release Assessment
### PromptKit v0.6.0
`v0.6.0` adds no public declarations, but it is a material compatibility and
safety release. It centralizes execution-setting, output-contract, endpoint,
and JSON-compatible-value validation; makes YAML metadata authoritative for
prompt and profile identity; hardens `content_file` containment and regular-file
requirements; validates OpenAI-compatible endpoints structurally; bounds JSON
trees and successful provider bodies; requires exactly one JSON value in
provider responses; preserves cancellation and transport error identities; and
reuses schema and rendered-artifact work within an operation.
WeatherReporter should receive these improvements directly from the dependency
and audit its own supported assets and configuration paths against the stricter
contracts. It should not duplicate PromptKit's internal validators or tests.
The existing embedded prompt paths, inline data-package input, generated-output
limit, and adapter boundary remain conceptually correct.
### PromptKit v0.7.0
`v0.7.0` adds four potentially useful consumer features:
- linear, cycle-safe profile inheritance through `base_profile` and
`Profile.BaseProfileID`;
- the built-in `rakestrawhome` backend and
`rakestrawhome-gemma-4-31b` profile;
- optional API-key environment sources, with `APIKeyRequired` reserved for an
explicit local credential requirement; and
- bounded structured `GenerationError` details for non-2xx responses from the
built-in OpenAI-compatible client.
WeatherReporter has no manual `rakestrawhome` registration and uses keyed
PromptKit profile literals, so the two source-compatibility hazards called out
by the release do not require migration shims. The profile, credential, and
error features do require deliberate application-policy choices described
below.
### PromptKit v0.8.0
`v0.8.0` activates the existing output-contract repair budget. A positive
`repair_attempts` value authorizes up to that many corrective model calls after
eligible `basic`, `json`, or `json_schema` validation failures. The supported
budget is zero through three. Repairs preserve the original rendered
conversation, effective target, session, structured-output contract, and
backend capacity policy. The final result reports cumulative token usage and
the number of corrective calls actually made.
Repair exhaustion is a completed generation with failed validation, not an
operational error. WeatherReporter's existing policy should continue to reject
that result and publish no report for that profile. Explicit empty provider
content now reaches output validation; for WeatherReporter's JSON Schema
prompts it is therefore eligible for repair rather than being misclassified as
a malformed provider envelope.
## Desired End State
WeatherReporter builds and tests against PromptKit `v0.8.0` with no workspace,
vendor, or module replacement dependency. Its public behavior remains
stateless, its PromptKit dependency types remain confined to the adapter, and
its ordinary summaries and logs remain safe.
The completed integration:
- benefits from the v0.6.0 safety and efficiency corrections;
- composes WeatherReporter domain profiles from PromptKit's maintained built-in
profiles while preserving WeatherReporter-owned leaf IDs and operator
override precedence;
- accepts a successfully inspected endpoint-only profile with a nonblank model
even though it has no logical backend ID;
- permits explicit use of PromptKit's Rakestrawhome profile without custom
backend wiring;
- applies an accepted bounded-repair policy to every operational structured
prompt;
- validates repair configuration during preflight and records actual repair
activity in project-owned result values;
- retains PromptKit's cumulative usage accounting in explicit debug output;
- distinguishes safe provider HTTP status from potentially sensitive provider
diagnostics; and
- documents the changed profile, credential, repair, comparison, debug, and
failure contracts in their canonical owners.
## Dependency And Compatibility Policy
The module requirement should move directly from `v0.5.0` to `v0.8.0`, followed
by a clean module tidy. WeatherReporter already requires Go 1.26 while PromptKit
`v0.8.0` requires Go 1.25.5, so no Go version change is needed for this upgrade.
Consumer validation must cover the paths called out by PromptKit v0.6.0:
- every embedded prompt, content file, schema, and fallback profile inspects
through PromptKit `v0.8.0`;
- configured single-file and directory profile sources retain their lazy,
metadata-authoritative identity and precedence behavior;
- malformed selected profiles and invalid local endpoints retain actionable
WeatherReporter categories;
- the inline YAML data package and prepared-execution path remain within the
new JSON and response bounds; and
- cancellation, deadline, and backend-capacity identities still cross the
adapter correctly.
PromptKit owns its 16 MiB successful transport-response bound and JSON framing.
WeatherReporter retains its stricter 64 KiB generated-text acceptance bound.
The consumer suite should protect that relationship without reproducing
PromptKit's lower-level transport matrix.
## Domain Profile Composition
The embedded profiles should become application-owned aliases:
```yaml
id: weather-light
base_profile: deepseek-4-flash
```
```yaml
id: weather-balanced
base_profile: gemini-flash-latest
```
```yaml
id: weather-deep
base_profile: claude-sonnet-latest
```
The effective backend, model, timeout, service tier, and reasoning settings
must initially remain identical to the current WeatherReporter definitions.
The selected leaf remains the durable logical profile identity even though its
effective target is inherited.
Profile source precedence remains:
1. explicit in-memory profiles used by tests or embedding consumers;
2. the configured `profile_file` or `profile_dir` source;
3. WeatherReporter's embedded fallback catalog; and
4. PromptKit's built-in catalog.
Sources still do not merge definitions of the same ID. Once a selected
definition names `base_profile`, however, each parent ID is resolved through
that same precedence order and the resulting linear chain is merged from root
to leaf according to PromptKit's inheritance contract. Documentation must make
that distinction explicit. A malformed leaf, missing or malformed base, cycle,
overlong chain, or incomplete resolved target fails profile inspection before
weather collection.
An operator may continue to replace `weather-light`, `weather-balanced`, or
`weather-deep` with a standalone definition. An operator may also define a
derived replacement. The maintained local endpoint example should remain
standalone because PromptKit profile inheritance has no clearing syntax: using
an OpenRouter base would retain its backend identity and capacity policy even
when the child replaces the endpoint.
WeatherReporter should treat the adapter's successful profile inspection as
authoritative that PromptKit resolved a usable route. A nonblank model remains
required, but backend ID is optional for an endpoint-only profile and should be
omitted from safe provenance where unavailable. WeatherReporter still must not
surface the endpoint outside explicit debug capture. This aligns application
preflight with PromptKit and with the existing CLI, comparison, and debug value
shapes, all of which already permit an absent backend identity.
## Rakestrawhome Availability
The reserved `rakestrawhome` backend and built-in
`rakestrawhome-gemma-4-31b` profile should be supported automatically through
ordinary PromptKit selection. Operators may choose that profile with the
existing global profile setting or as one entry in `compare`, and PromptKit's
backend capacity policy remains authoritative.
WeatherReporter should not register, wrap, or duplicate the backend or profile,
and should not add a Rakestrawhome-specific configuration field. Its canonical
PromptKit integration documentation should link to PromptKit for the current
built-in catalog and credential contract rather than copying volatile endpoint
or capacity values. Offline inspection coverage should prove that the built-in
profile crosses the WeatherReporter adapter with the expected logical backend
identity.
## Bounded Structured-Output Repair
The accepted repair budget belongs to the exact PromptKit output contract, not
to a WeatherReporter retry loop. PromptKit alone should construct corrective
messages, perform additional calls, enforce the budget, aggregate usage, and
coordinate backend capacity. WeatherReporter must not retry provider failures,
switch profiles, or layer another repair mechanism around `RunPrepared`.
All four embedded prompt definitions should declare `repair_attempts: 1`.
Because this changes prompt
execution behavior, latency, cost, hash, and provenance, each definition and
its report-registry binding should advance from exact version `2.0.0` to
`2.1.0`. Prompt text and generated-text schemas do not need to change solely
for this feature.
The embedded prompt definition is the per-report pipeline policy owner. This
upgrade should not add a global or per-report operator configuration field for
repair attempts and should not construct a request-level replacement output
contract. A future pipeline may select another budget only through a deliberate
prompt-definition and exact-version change.
The project-owned PromptKit boundary should retain:
- the configured repair budget in prompt inspection and preparation output
contracts;
- the number of corrective calls actually made in completed validation;
- cumulative PromptKit token usage across initial and corrective calls; and
- the final candidate and final validation result only, consistent with the
PromptKit contract.
Prompt inspection and preparation provenance must require the repair budget to
match the exact expected prompt definition just as they currently require the
format, validation mode, and schema path to match. A zero-attempt successful
result is normal when the first candidate passes. A repair-exhausted result
continues through WeatherReporter's ordinary `validation_rejected` failure
path, and an operational or generation failure during correction remains that
profile's ordinary operational failure.
For concurrent comparison, every profile should use the same prompt repair
budget. A corrective call remains part of that profile's one prepared
execution and uses PromptKit's existing backend capacity pool. One profile's
repair or failure must not cancel independent peers.
## Repair Observability And Comparison Contract
The actual repair count is safe operational provenance and should be visible
where WeatherReporter already reports completed validation. Generation, batch,
and comparison action summaries should expose it without exposing candidates,
schemas, or diagnostics. Explicit execution debug output should add it to the
validation object alongside PromptKit's already mapped cumulative usage.
Profile comparison needs this value in `comparison.json`: a successful result
that required correction is materially different from a first-pass success
when evaluating model reliability, latency, and cost. The manifest should
therefore advance to `weatherreporter.comparison.v2` and add a non-negative
`repairAttempts` field to each result. The field is zero when no corrective
call began, including ordinary first-pass success. A failure carries the count
when PromptKit returned a completed validation result; it is omitted only when
execution failed before a completed validation result made the value known.
The v2 manifest should remain flat, strict, deterministic, and atomically
published. WeatherReporter does not need to preserve v1 replacement
compatibility: comparison bundles are operator-owned development outputs, and
the current integration contract intentionally recognizes only its current
schema. The release notes and comparison documentation must call out the
version change so an operator can remove or relocate an older bundle before
using guarded replacement at the same destination.
## Credential Semantics
PromptKit v0.7.0 treats `APIKeyEnv` as an optional lookup source. If the
environment variable is absent or blank and no direct credential is supplied,
the built-in client omits `Authorization` and lets the endpoint respond.
`APIKeyRequired` is the distinct signal that a usable credential must be
provided locally.
WeatherReporter cannot supply PromptKit's request-scoped direct API-key value,
so a profile reporting `APIKeyRequired` remains unsupported and must fail
before weather collection. WeatherReporter should not require a nonblank value
for an optional `APIKeyEnv` during application preflight. The built-in client
should omit `Authorization` when that source is unavailable and let the
endpoint return any authentication failure through the ordinary structured
generation-error path.
## Structured Generation Failures
PromptKit v0.7.0's `GenerationError` can report a provider HTTP status plus
bounded provider code, type, and message. WeatherReporter should consume that
type only inside the PromptKit adapter and map any adopted fields into a
project-owned immutable error. PromptKit dependency types must not become app
or CLI contracts.
HTTP status is safe enough for ordinary diagnostics. Provider code, type, and
message remain untrusted and may contain request or schema fragments. They
must never enter ordinary errors, action summaries, comparison manifests,
logs, generated reports, or Distributor payloads. The full structured
diagnostic belongs only in an explicitly requested secure `--llm-debug-dir`
`failure.json` artifact. That artifact may contain PromptKit's normalized
bounded fields but never the raw provider body, headers, endpoint, credentials,
or reconstructed request.
Initial-call and corrective-call non-2xx responses should follow the same
mapping. Cancellation and deadline categories continue to take precedence over
provider classification where PromptKit preserves those identities.
## Testing Policy
The default suite must remain deterministic, offline, and credential-free.
Use injected PromptKit clients and synthetic embedded or temporary assets for
consumer behavior; do not call OpenRouter, Rakestrawhome, or a local endpoint.
Risk-based coverage should include:
- all embedded prompts and inherited domain profiles inspecting successfully
under PromptKit `v0.8.0`;
- unchanged effective targets and report-to-profile assignments after the
alias refactor;
- external standalone and derived profile precedence, plus selected missing,
cyclic, and malformed-base failures at the WeatherReporter boundary;
- end-to-end preflight and prepared execution through the maintained
endpoint-only `weather-light` override without exposing its endpoint;
- offline inspection of `rakestrawhome-gemma-4-31b`;
- a first-pass valid result with zero repairs;
- an invalid structured result repaired successfully within one corrective
call;
- one-attempt exhaustion returning failed validation and no published report;
- a corrective generation failure retaining its safe category and provider
status policy;
- cumulative usage and actual repair-count mapping;
- comparison peers remaining independent when one profile repairs, exhausts,
or fails;
- v2 comparison manifest validation and guarded replacement; and
- absent or blank optional `APIKeyEnv` values reaching the provider without an
`Authorization` header, while `APIKeyRequired` profiles fail preflight.
Do not reproduce PromptKit's internal matrices for path traversal, JSON tree
bounds, response framing, inheritance depth, repair prompt construction, or
provider-detail normalization. WeatherReporter tests should protect only its
adapter mappings, application policy, provenance, publication, and public
contracts. Run ordinary and race-enabled repository tests because the repaired
execution path participates in concurrent comparisons.
## Documentation And Release Impact
Implementation must update each canonical owner whose contract changes:
- `docs/policy/architecture.md` for prompt-execution, credential, diagnostic,
and comparison invariants;
- `docs/config.md` and the maintained local profile example for profile-source,
inheritance, and credential semantics;
- `docs/integrations/promptkit.md` for exact prompt versions, repair policy,
profile composition, Rakestrawhome availability, and safe errors;
- `docs/integrations/comparison-bundle.md` for the v2 manifest and repair count;
- `docs/cli.md` for repair-count fields in action summaries;
- `docs/operations.md` for changed failure behavior and any explicit provider
diagnostic capture;
- focused internal PromptKit adapter, app orchestration, prompt-debug, and
comparison documentation; and
- release notes for the dependency jump, prompt version change, possible
additional model call, credential behavior, profile inheritance, diagnostic
behavior, and comparison schema change.
Current-state documentation must not describe this behavior until the
implementation lands. PromptKit remains the canonical owner of its complete
built-in catalogs, YAML merge rules, transport limits, corrective-message
construction, and public Go API.
## Scope
The completed feature includes:
- the direct module upgrade and tidy dependency graph;
- a v0.6.0 compatibility audit of WeatherReporter's supported PromptKit paths;
- inherited WeatherReporter domain profile definitions with unchanged
effective targets;
- correction of application preflight so PromptKit endpoint-only profiles work
as documented while retaining a required model identity;
- ordinary access to the Rakestrawhome built-in profile;
- PromptKit's optional-credential policy, while direct-key-required profiles
remain unsupported;
- `repair_attempts` on all operational prompts and exact prompt-version bumps;
- project-owned repair budget, actual-attempt, usage, and provenance mappings;
- repair observability in action summaries, explicit debug output, and a v2
comparison manifest;
- safe provider HTTP status in ordinary errors and bounded provider detail only
in explicit secure debug capture;
- focused offline and race-enabled regression coverage; and
- canonical current-state and release documentation updated with the code.
## Non-Goals
This upgrade does not include:
- application-implemented repair prompts or provider transport;
- retries for HTTP, network, timeout, capacity, or other operational failures;
- automatic profile escalation, fallback, ranking, or resampling;
- changing the weather profile ladder, default report assignments, or concrete
model targets beyond inheriting their maintained PromptKit definitions;
- making Rakestrawhome a default or adding provider-specific configuration;
- live-provider tests or a permanent benchmark framework;
- exposing raw provider responses or sensitive diagnostics routinely;
- a general prompt-source or pipeline plugin system; or
- compatibility shims for PromptKit versions older than `v0.8.0`.
## Completion Criteria
The roadmap is complete when:
- `go.mod` and `go.sum` resolve PromptKit `v0.8.0` without a replacement,
workspace, or vendor tree;
- all PromptKit v0.6.0 compatibility points relevant to WeatherReporter have
been checked and valid supported inputs retain project-owned error identity;
- the three WeatherReporter profiles inherit the intended PromptKit built-ins,
retain their logical IDs, and inspect to the intended effective targets;
- external standalone and inherited overrides obey documented precedence and
failure behavior;
- the maintained endpoint-only local override passes application preflight,
retains an empty backend ID, and keeps its endpoint out of ordinary values;
- `rakestrawhome-gemma-4-31b` is selectable through ordinary generation and
comparison paths without WeatherReporter backend registration;
- every operational prompt has one bounded repair attempt at exact
version `2.1.0`;
- inspection, preparation, execution, debug, and comparison values accurately
preserve configured and actual repair counts;
- first-pass success, repaired success, repair exhaustion, repair generation
failure, and explicit empty content follow the documented outcomes;
- the v2 comparison bundle distinguishes first-pass and repaired results;
- credential preflight accepts absent optional environment credentials while
rejecting direct-key-required profiles, and provider-error disclosure does
not leak sensitive values;
- the default test suite is offline and deterministic, ordinary and race
validation pass, and no redundant upstream implementation suite is copied;
and
- every implemented contract is documented by its canonical current-state
owner and disclosed in the eventual release notes.

2
go.mod
View File

@@ -6,7 +6,7 @@ require gopkg.in/yaml.v3 v3.0.1
require ( require (
gitea.maximumdirect.net/eric/distributor v0.5.0 gitea.maximumdirect.net/eric/distributor v0.5.0
gitea.maximumdirect.net/eric/promptkit v0.5.0 gitea.maximumdirect.net/eric/promptkit v0.8.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
golang.org/x/sys v0.45.0 golang.org/x/sys v0.45.0
) )

4
go.sum
View File

@@ -1,7 +1,7 @@
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk= gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8= gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc= gitea.maximumdirect.net/eric/promptkit v0.8.0 h1:NGd9hDLu0UMxKbvittMrqM5Ua94eFb+kOE7UIir8l08=
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ= gitea.maximumdirect.net/eric/promptkit v0.8.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4= github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo= github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=

View File

@@ -153,6 +153,7 @@ func outputContract(value promptkit.OutputContract) promptexec.OutputContract {
Format: string(value.Format), Format: string(value.Format),
ValidationMode: string(value.ValidationMode), ValidationMode: string(value.ValidationMode),
SchemaPath: value.SchemaPath, SchemaPath: value.SchemaPath,
RepairAttempts: value.RepairAttempts,
} }
} }
@@ -193,6 +194,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
promptexec.ValidationStatus(value.Validation.Status), promptexec.ValidationStatus(value.Validation.Status),
string(value.Validation.Mode), string(value.Validation.Mode),
value.Validation.SchemaPath, value.Validation.SchemaPath,
value.Validation.RepairAttempts,
value.Validation.Errors, value.Validation.Errors,
) )
rawOutput := []byte(nil) rawOutput := []byte(nil)
@@ -203,6 +205,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
promptexec.ValidationFailed, promptexec.ValidationFailed,
string(value.Validation.Mode), string(value.Validation.Mode),
value.Validation.SchemaPath, value.Validation.SchemaPath,
value.Validation.RepairAttempts,
[]string{"generated output exceeds the configured size limit"}, []string{"generated output exceeds the configured size limit"},
) )
} }
@@ -299,6 +302,16 @@ func classifyError(err error) error {
if errors.As(err, &capacityError) { if errors.As(err, &capacityError) {
return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err) return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err)
} }
var generationError *promptkit.GenerationError
if errors.As(err, &generationError) {
return promptexec.NewGenerationError(
generationError.StatusCode(),
generationError.ProviderCode(),
generationError.ProviderType(),
generationError.ProviderMessage(),
err,
)
}
switch { switch {
case errors.Is(err, promptkit.ErrInvalidConfig): case errors.Is(err, promptkit.ErrInvalidConfig):
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err) return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err)

View File

@@ -4,11 +4,15 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"net/http"
"net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"strings" "strings"
"sync" "sync"
"testing" "testing"
"testing/fstest"
"time" "time"
promptkit "gitea.maximumdirect.net/eric/promptkit" promptkit "gitea.maximumdirect.net/eric/promptkit"
@@ -20,12 +24,19 @@ type fakeClient struct {
mu sync.Mutex mu sync.Mutex
response *promptkit.GenerateResponse response *promptkit.GenerateResponse
err error err error
outcomes []generationOutcome
next int
calls int calls int
requests []promptkit.GenerateRequest requests []promptkit.GenerateRequest
block bool block bool
started chan struct{} started chan struct{}
} }
type generationOutcome struct {
response *promptkit.GenerateResponse
err error
}
type recordingReader struct { type recordingReader struct {
ref promptkit.ArtifactRef ref promptkit.ArtifactRef
} }
@@ -49,6 +60,11 @@ func (client *fakeClient) Generate(ctx context.Context, request promptkit.Genera
started := client.started started := client.started
response := client.response response := client.response
err := client.err err := client.err
if client.next < len(client.outcomes) {
outcome := client.outcomes[client.next]
client.next++
response, err = outcome.response, outcome.err
}
client.mu.Unlock() client.mu.Unlock()
if started != nil { if started != nil {
started <- struct{}{} started <- struct{}{}
@@ -102,13 +118,19 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
return client.requests[0] return client.requests[0]
} }
func (client *fakeClient) allRequests() []promptkit.GenerateRequest {
client.mu.Lock()
defer client.mu.Unlock()
return append([]promptkit.GenerateRequest(nil), client.requests...)
}
func TestInspectPromptAndProfile(t *testing.T) { func TestInspectPromptAndProfile(t *testing.T) {
adapter := newTestAdapter(t, &fakeClient{}) adapter := newTestAdapter(t, &fakeClient{})
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.0.0") inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.1.0")
if err != nil { if err != nil {
t.Fatalf("InspectPrompt() error = %v", err) t.Fatalf("InspectPrompt() error = %v", err)
} }
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != "weather-balanced" { if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != "weather-balanced" || inspection.Output.RepairAttempts != 1 {
t.Fatalf("inspection = %#v", inspection) t.Fatalf("inspection = %#v", inspection)
} }
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" { if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
@@ -175,15 +197,38 @@ model: file-light
} }
assertProfile(t, fileAdapter, "weather-light", "", "file-light") assertProfile(t, fileAdapter, "weather-light", "", "file-light")
directory := testProfileDirectory(t, `id: weather-light directory := testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
backend: local backend: local
model: directory-light model: directory-light
`) `})
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"}) directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil { if err != nil {
t.Fatalf("New(profile directory) error = %v", err) t.Fatalf("New(profile directory) error = %v", err)
} }
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light") assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
derived := writeProfileFile(t, `id: weather-light
base_profile: gemini-flash-latest
`)
derivedAdapter, err := New(Config{ProfileFile: derived})
if err != nil {
t.Fatalf("New(derived profile) error = %v", err)
}
assertProfile(t, derivedAdapter, "weather-light", "openrouter", "~google/gemini-flash-latest")
}
func TestConfiguredBaseProfileOverridesEmbeddedProfileTarget(t *testing.T) {
directory := testProfileDirectory(t, map[string]string{
"deepseek.yml": `id: deepseek-4-flash
backend: local
model: shadowed-deepseek
`,
})
adapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil {
t.Fatalf("New() error = %v", err)
}
assertProfile(t, adapter, "weather-light", promptkit.BackendLocal, "shadowed-deepseek")
} }
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) { func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
@@ -194,19 +239,69 @@ func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T)
assertProfile(t, adapter, "weather-light", "", "weather-local") assertProfile(t, adapter, "weather-light", "", "weather-local")
} }
func TestMaintainedWeatherLightLocalProfileExampleExecutesThroughProductionClient(t *testing.T) {
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
for _, test := range []struct {
name string
credentialSource string
}{
{name: "without credential source"},
{name: "with blank optional credential source", credentialSource: "\napi_key_env: WEATHERREPORTER_TEST_MISSING_KEY\n"},
} {
t.Run(test.name, func(t *testing.T) {
var authorization string
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
authorization = request.Header.Get("Authorization")
writer.Header().Set("Content-Type", "application/json")
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":12,"completion_tokens":8,"total_tokens":20}}`, validResponse().Content)
}))
defer server.Close()
example, err := os.ReadFile(filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml"))
if err != nil {
t.Fatal(err)
}
profile := strings.Replace(string(example), "http://127.0.0.1:11434/v1", server.URL+"/v1", 1) + test.credentialSource
adapter, err := New(Config{ProfileFile: writeProfileFile(t, profile)})
if err != nil {
t.Fatalf("New() error = %v", err)
}
request := testExecuteRequest()
request.ProfileID = "weather-light"
var preparation promptexec.Preparation
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
preparation = value
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if authorization != "" {
t.Fatalf("Authorization header = %q, want absent", authorization)
}
if preparation.ProfileID != "weather-light" || preparation.BackendID != "" || preparation.ModelName != "weather-local" || preparation.Output.RepairAttempts != 1 {
t.Fatalf("preparation = %#v", preparation)
}
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" || result.Validation.Status != promptexec.ValidationPassed || result.Validation.RepairAttempts != 0 {
t.Fatalf("execution = %#v", result)
}
})
}
}
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) { func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: other-profile
backend: openrouter backend: openrouter
model: other-model model: other-model
`)}) `})})
if err != nil { if err != nil {
t.Fatalf("New(absent profile) error = %v", err) t.Fatalf("New(absent profile) error = %v", err)
} }
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash") assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
backend: openrouter backend: openrouter
`)}) `})})
if err != nil { if err != nil {
t.Fatalf("New(malformed profile) error = %v", err) t.Fatalf("New(malformed profile) error = %v", err)
} }
@@ -215,6 +310,78 @@ backend: openrouter
} }
} }
func TestProfileResolutionReturnsConfiguredInheritanceFailures(t *testing.T) {
tests := []struct {
name string
profile string
profiles map[string]string
}{
{
name: "missing base",
profile: "missing-base",
profiles: map[string]string{"missing.yml": `id: missing-base
base_profile: unavailable
`},
},
{
name: "cyclic bases",
profile: "first",
profiles: map[string]string{
"first.yml": `id: first
base_profile: second
`,
"second.yml": `id: second
base_profile: first
`,
},
},
{
name: "malformed base",
profile: "child",
profiles: map[string]string{
"child.yml": `id: child
base_profile: malformed
`,
"malformed.yml": `id: malformed
base_profile: [not-a-profile]
`,
},
},
{
name: "incomplete target",
profile: "incomplete",
profiles: map[string]string{"incomplete.yml": `id: incomplete
backend: openrouter
`},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
adapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, test.profiles)})
if err != nil {
t.Fatalf("New() error = %v", err)
}
if _, err := adapter.InspectProfile(context.Background(), test.profile); err == nil {
t.Fatal("InspectProfile() error = nil, want configured inheritance error")
}
})
}
}
func TestRakestrawhomeBuiltInProfileInspectsOffline(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
profile, err := adapter.InspectProfile(context.Background(), "rakestrawhome-gemma-4-31b")
if err != nil {
t.Fatalf("InspectProfile() error = %v", err)
}
if profile.ProfileID != "rakestrawhome-gemma-4-31b" || profile.BackendID != "rakestrawhome" || profile.ModelName == "" {
t.Fatalf("profile = %#v", profile)
}
}
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) { func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
adapter, err := New(Config{}) adapter, err := New(Config{})
if err != nil { if err != nil {
@@ -287,7 +454,7 @@ func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
} }
request := promptexec.ExecuteRequest{ request := promptexec.ExecuteRequest{
PromptID: "weather.hourly_generated_text", PromptID: "weather.hourly_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
ProfileID: "weather-light", ProfileID: "weather-light",
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"), DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
} }
@@ -400,6 +567,114 @@ func TestExecuteReturnsCompletedValidationRejection(t *testing.T) {
} }
} }
func TestExecuteMapsCorrectiveGenerationResults(t *testing.T) {
valid := `{"summary":"valid"}`
invalid := `{"summary":42}`
tests := []struct {
name string
outcomes []generationOutcome
wantStatus promptexec.ValidationStatus
wantRepairs int
wantCalls int
wantRaw string
wantUsage promptexec.TokenUsage
}{
{
name: "first pass valid",
outcomes: []generationOutcome{{response: generationResponse(valid, 2, 3, 5)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 0,
wantCalls: 1,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5},
},
{
name: "empty output repaired",
outcomes: []generationOutcome{{response: generationResponse("", 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
{
name: "invalid output repaired",
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
{
name: "repair budget exhausted",
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(invalid, 7, 11, 18)}},
wantStatus: promptexec.ValidationFailed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: invalid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
client := &fakeClient{outcomes: test.outcomes}
adapter := newRepairAdapter(t, client, "https://repair.example/v1")
var preparation promptexec.Preparation
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
preparation = value
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if preparation.Output.RepairAttempts != 1 || result == nil || result.Validation.Status != test.wantStatus || result.Validation.RepairAttempts != test.wantRepairs || string(result.RawOutput) != test.wantRaw || result.Usage != test.wantUsage {
t.Fatalf("preparation/result = %#v/%#v", preparation, result)
}
requests := client.allRequests()
if len(requests) != test.wantCalls {
t.Fatalf("provider requests = %d, want %d", len(requests), test.wantCalls)
}
if test.wantCalls == 2 && !reflect.DeepEqual(requests[0].Target, requests[1].Target) {
t.Fatalf("corrective target = %#v, want same prepared identity as %#v", requests[1].Target, requests[0].Target)
}
if result.ProfileID != preparation.ProfileID || result.BackendID != preparation.BackendID || result.ModelName != preparation.ModelName || result.PromptID != preparation.PromptID || result.PromptVersion != preparation.PromptVersion || result.PromptHash != preparation.PromptHash {
t.Fatalf("prepared/result identity = %#v/%#v", preparation, result)
}
})
}
}
func TestExecuteMapsCorrectiveGenerationError(t *testing.T) {
const providerBody = `{"error":{"code":"repair-code","type":"repair-type","message":"repair-message"}}`
calls := 0
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) {
calls++
if calls == 1 {
writer.Header().Set("Content-Type", "application/json")
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":2,"completion_tokens":3,"total_tokens":5}}`, `{"summary":42}`)
return
}
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusUnprocessableEntity)
_, _ = writer.Write([]byte(providerBody))
}))
defer server.Close()
adapter := newRepairAdapter(t, nil, server.URL)
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), nil)
if result != nil || err == nil || calls != 2 {
t.Fatalf("result/error/calls = %#v/%v/%d", result, err, calls)
}
var generationError *promptexec.GenerationError
if !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusUnprocessableEntity || generationError.ProviderCode() != "repair-code" || generationError.ProviderType() != "repair-type" || generationError.ProviderMessage() != "repair-message" {
t.Fatalf("generation error = %#v", err)
}
if strings.Contains(err.Error(), "repair-message") || !errors.Is(err, promptkit.ErrLLMGenerate) {
t.Fatalf("generation error = %v", err)
}
}
func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) { func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) {
client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}} client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}}
adapter := newTestAdapter(t, client) adapter := newTestAdapter(t, client)
@@ -507,12 +782,12 @@ func TestNewValidatesConfiguration(t *testing.T) {
} }
} }
func TestLocalBackendAndMissingCredentialBehavior(t *testing.T) { func TestLocalBackendAndOptionalCredentialSourceBehavior(t *testing.T) {
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "") t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
profiles := testProfileDirectory(t, `id: local-profile profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: local-profile
backend: local backend: local
model: local-model model: local-model
`) `})
adapter, err := newAdapterForTest(Config{ adapter, err := newAdapterForTest(Config{
ProfileDirectory: profiles, ProfileDirectory: profiles,
LocalEndpoint: "https://local.example/v1", LocalEndpoint: "https://local.example/v1",
@@ -529,11 +804,11 @@ model: local-model
t.Fatalf("capacity classification = %v", got) t.Fatalf("capacity classification = %v", got)
} }
credentialProfiles := testProfileDirectory(t, `id: credential-profile credentialProfiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: credential-profile
endpoint: https://profile.example/v1 endpoint: https://profile.example/v1
model: test-model model: test-model
api_key_env: WEATHERREPORTER_TEST_MISSING_KEY api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
`) `})
client := &fakeClient{response: validResponse()} client := &fakeClient{response: validResponse()}
credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client) credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client)
if err != nil { if err != nil {
@@ -546,8 +821,8 @@ api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
request := testExecuteRequest() request := testExecuteRequest()
request.ProfileID = "credential-profile" request.ProfileID = "credential-profile"
result, err := credentialAdapter.Execute(context.Background(), request, nil) result, err := credentialAdapter.Execute(context.Background(), request, nil)
if result != nil || promptexec.CategoryOf(err) != promptexec.MissingCredential || client.callCount() != 0 { if err != nil || result == nil || client.callCount() != 1 {
t.Fatalf("credential result/category/calls = %#v/%q/%d", result, promptexec.CategoryOf(err), client.callCount()) t.Fatalf("credential result/error/calls = %#v/%v/%d", result, err, client.callCount())
} }
} }
@@ -568,14 +843,14 @@ func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, mo
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter { func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
t.Helper() t.Helper()
profiles := testProfileDirectory(t, `id: test-profile profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: test-profile
endpoint: https://profile.example/v1 endpoint: https://profile.example/v1
model: test-model model: test-model
temperature: 0.2 temperature: 0.2
max_tokens: 300 max_tokens: 300
top_p: 1 top_p: 1
timeout_seconds: 30 timeout_seconds: 30
`) `})
options = append(options, promptkit.WithLLMClient(client)) options = append(options, promptkit.WithLLMClient(client))
adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...) adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...)
if err != nil { if err != nil {
@@ -584,13 +859,15 @@ timeout_seconds: 30
return adapter return adapter
} }
func testProfileDirectory(t *testing.T, profile string) string { func testProfileDirectory(t *testing.T, profiles map[string]string) string {
t.Helper() t.Helper()
profiles := t.TempDir() directory := t.TempDir()
if err := os.WriteFile(filepath.Join(profiles, "profile.yml"), []byte(profile), 0o600); err != nil { for name, profile := range profiles {
t.Fatalf("write profile: %v", err) if err := os.WriteFile(filepath.Join(directory, name), []byte(profile), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
} }
return profiles return directory
} }
func writeProfileFile(t *testing.T, profile string) string { func writeProfileFile(t *testing.T, profile string) string {
@@ -605,7 +882,7 @@ func writeProfileFile(t *testing.T, profile string) string {
func testExecuteRequest() promptexec.ExecuteRequest { func testExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{ return promptexec.ExecuteRequest{
PromptID: "weather.daily_generated_text", PromptID: "weather.daily_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
ProfileID: "test-profile", ProfileID: "test-profile",
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"), DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
} }
@@ -618,6 +895,61 @@ func validResponse() *promptkit.GenerateResponse {
} }
} }
func generationResponse(content string, promptTokens int, completionTokens int, totalTokens int) *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{
Content: content,
Usage: promptkit.TokenUsage{
PromptTokens: promptTokens,
CompletionTokens: completionTokens,
TotalTokens: totalTokens,
},
}
}
func newRepairAdapter(t *testing.T, client promptkit.LLMClient, endpoint string) *Adapter {
t.Helper()
profiles := testProfileDirectory(t, map[string]string{"profile.yml": "id: repair-profile\nendpoint: " + endpoint + "\nmodel: repair-model\n"})
options := []promptkit.Option{
promptkit.WithPromptFS(fstest.MapFS{
"repair.yml": &fstest.MapFile{Data: []byte(`id: weather.repair
version: "1.0.0"
default_profile: repair-profile
inputs:
- name: data_package
required: true
content_type: application/yaml
messages:
- role: user
content: "{{input \"data_package\"}}"
output:
format: json
validation_mode: json_schema
schema_path: repair.schema.json
repair_attempts: 1
`)}}, "."),
promptkit.WithSchemaFS(fstest.MapFS{
"repair.schema.json": &fstest.MapFile{Data: []byte(`{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"],"additionalProperties":false}`)},
}, "."),
}
if client != nil {
options = append(options, promptkit.WithLLMClient(client))
}
adapter, err := newAdapter(Config{ProfileDirectory: profiles}, options...)
if err != nil {
t.Fatalf("newAdapter() error = %v", err)
}
return adapter
}
func repairExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{
PromptID: "weather.repair",
PromptVersion: "1.0.0",
ProfileID: "repair-profile",
DataPackage: []byte("report: repair\n"),
}
}
func hourlyValidResponse() *promptkit.GenerateResponse { func hourlyValidResponse() *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{ return &promptkit.GenerateResponse{
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`, Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,

View File

@@ -89,6 +89,7 @@ type ReportResult struct {
ModelName string ModelName string
SourceWarnings []weatherdata.SourceWarning SourceWarnings []weatherdata.SourceWarning
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string LLMDebugPath string
OutputPath string OutputPath string
Notification *NotificationResult Notification *NotificationResult
@@ -139,6 +140,7 @@ type BatchReportResult struct {
ModelName string `json:"modelName,omitempty"` ModelName string `json:"modelName,omitempty"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"` SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"` ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
LLMDebugPath string `json:"llmDebugPath,omitempty"` LLMDebugPath string `json:"llmDebugPath,omitempty"`
OutputPath string `json:"outputPath,omitempty"` OutputPath string `json:"outputPath,omitempty"`
} }
@@ -457,6 +459,9 @@ func copyBatchReportDetails(item *BatchReportResult, result *ReportResult) {
item.Timezone = result.Timezone item.Timezone = result.Timezone
item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...) item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
item.ValidationStatus = result.ValidationStatus item.ValidationStatus = result.ValidationStatus
if result.RepairAttempts != nil {
item.RepairAttempts = repairAttemptsPointer(*result.RepairAttempts)
}
} }
func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) { func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) {

View File

@@ -3,7 +3,6 @@ package app
import ( import (
"context" "context"
"fmt" "fmt"
"os"
"path/filepath" "path/filepath"
"time" "time"
@@ -60,6 +59,7 @@ type ComparisonProfileResult struct {
ModelName string ModelName string
Status string Status string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
ReportPath string ReportPath string
LLMDebugPath string LLMDebugPath string
Error *comparison.SafeError Error *comparison.SafeError
@@ -121,7 +121,7 @@ func compareDetailed(ctx context.Context, req ComparisonRequest, publish compari
} }
defer func() { _ = debugWriter.Close() }() defer func() { _ = debugWriter.Close() }()
inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{ inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor, LookupEnv: os.LookupEnv, Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor,
}) })
result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash
if err != nil { if err != nil {
@@ -198,6 +198,9 @@ func copyComparisonOutcomes(result *ComparisonResult, outcomes []comparisonProfi
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName, Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error, Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error,
} }
if outcome.RepairAttempts != nil {
profile.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
if published && outcome.Status == comparison.StatusSucceeded { if published && outcome.Status == comparison.StatusSucceeded {
profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath) profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath)
} }
@@ -227,6 +230,9 @@ func comparisonBundle(result *ComparisonResult, dataPackage []byte, outcomes []c
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName, Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error, Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error,
} }
if outcome.RepairAttempts != nil {
manifestResult.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
if outcome.Status == comparison.StatusSucceeded { if outcome.Status == comparison.StatusSucceeded {
manifestResult.ReportPath = outcome.ReportPath manifestResult.ReportPath = outcome.ReportPath
bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown}) bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown})

View File

@@ -31,6 +31,7 @@ type comparisonProfileOutcome struct {
ModelName string ModelName string
Status string Status string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
ReportPath string ReportPath string
Markdown []byte Markdown []byte
LLMDebugPath string LLMDebugPath string
@@ -108,6 +109,9 @@ func executeComparisonProfile(ctx context.Context, req comparisonExecutionReques
}) })
outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName
outcome.ValidationStatus = execution.ValidationStatus outcome.ValidationStatus = execution.ValidationStatus
if execution.RepairAttempts != nil {
outcome.RepairAttempts = repairAttemptsPointer(*execution.RepairAttempts)
}
outcome.LLMDebugPath = execution.LLMDebugPath outcome.LLMDebugPath = execution.LLMDebugPath
if err != nil { if err != nil {
outcome.canceled = cancellationError(err) outcome.canceled = cancellationError(err)
@@ -169,9 +173,14 @@ func comparisonExecutionMessage(err error) string {
if errors.Is(err, context.DeadlineExceeded) { if errors.Is(err, context.DeadlineExceeded) {
return "profile execution deadline exceeded" return "profile execution deadline exceeded"
} }
operation := "profile execution"
var execution *profileExecutionError var execution *profileExecutionError
if errors.As(err, &execution) { if errors.As(err, &execution) {
return comparison.TruncateErrorMessage(execution.operation + " failed") operation = execution.operation
} }
return "profile execution failed" var generation *promptexec.GenerationError
if errors.As(err, &generation) && generation.StatusCode() > 0 {
return comparison.TruncateErrorMessage(fmt.Sprintf("%s failed (HTTP %d)", operation, generation.StatusCode()))
}
return comparison.TruncateErrorMessage(operation + " failed")
} }

View File

@@ -4,9 +4,11 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"net/http"
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
"strings"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -72,6 +74,66 @@ func TestExecuteComparisonProfilesContinuesAfterProfileFailure(t *testing.T) {
} }
} }
func TestExecuteComparisonProfilesPreservesIndependentRepairOutcomes(t *testing.T) {
prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(4)
executor := newBarrierExecutor(profiles)
executor.setValidation(profiles[0].ProfileID, promptexec.ValidationPassed, 0)
executor.setValidation(profiles[1].ProfileID, promptexec.ValidationPassed, 1)
executor.setValidation(profiles[2].ProfileID, promptexec.ValidationFailed, 1)
executor.setError(profiles[3].ProfileID, errors.New("provider failure"))
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", Executor: executor,
}, executor)
waitForProfileStarts(t, executor, profiles, results)
executor.releaseAll()
result := <-results
wantStatuses := []string{comparison.StatusSucceeded, comparison.StatusSucceeded, comparison.StatusFailed, comparison.StatusFailed}
wantValidations := []promptexec.ValidationStatus{promptexec.ValidationPassed, promptexec.ValidationPassed, promptexec.ValidationFailed, ""}
wantRepairs := []*int{intPointer(0), intPointer(1), intPointer(1), nil}
for index, outcome := range result.Outcomes {
if outcome.Status != wantStatuses[index] || outcome.ValidationStatus != wantValidations[index] || !reflect.DeepEqual(outcome.RepairAttempts, wantRepairs[index]) {
t.Fatalf("outcome[%d] = %#v, want status/validation/repairs %q/%q/%#v", index, outcome, wantStatuses[index], wantValidations[index], wantRepairs[index])
}
}
}
func TestExecuteComparisonProfilesCapturesConcurrentProviderFailures(t *testing.T) {
prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(2)
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
if err != nil {
t.Fatalf("NewPromptDebugWriter() error = %v", err)
}
markers := []string{"first-provider-private-marker", "second-provider-private-marker"}
statuses := []int{http.StatusTooManyRequests, http.StatusServiceUnavailable}
executor := newBarrierExecutor(profiles)
for index, profile := range profiles {
executor.setError(profile.ProfileID, promptexec.NewGenerationError(statuses[index], "provider_code", "provider_type", markers[index], nil))
}
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", DebugWriter: debugWriter, Executor: executor,
}, executor)
waitForProfileStarts(t, executor, profiles, results)
executor.releaseAll()
result := <-results
for index, outcome := range result.Outcomes {
if outcome.Status != comparison.StatusFailed || outcome.Error == nil || outcome.Error.Category != string(promptexec.Generation) || outcome.Error.Message != fmt.Sprintf("execute prompt failed (HTTP %d)", statuses[index]) || strings.Contains(outcome.Error.Message, markers[index]) || outcome.LLMDebugPath == "" {
t.Fatalf("outcome[%d] = %#v", index, outcome)
}
failure, readErr := os.ReadFile(filepath.Join(outcome.LLMDebugPath, "failure.json"))
if readErr != nil {
t.Fatal(readErr)
}
if !strings.Contains(string(failure), markers[index]) || strings.Contains(string(failure), markers[1-index]) {
t.Fatalf("failure[%d] = %s", index, failure)
}
}
}
func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) { func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) {
prepared, prompt := preparedDailyProfile(t) prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(4) profiles := comparisonProfiles(4)
@@ -139,6 +201,8 @@ type barrierExecutor struct {
releases map[string]chan struct{} releases map[string]chan struct{}
requests map[string]promptexec.ExecuteRequest requests map[string]promptexec.ExecuteRequest
errors map[string]error errors map[string]error
validations map[string]promptexec.ValidationStatus
repairAttempts map[string]int
profiles map[string]ComparisonProfileInspection profiles map[string]ComparisonProfileInspection
inFlight int inFlight int
maximum int maximum int
@@ -153,7 +217,7 @@ func newBarrierExecutor(profiles []ComparisonProfileInspection) *barrierExecutor
} }
return &barrierExecutor{ return &barrierExecutor{
started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases, started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases,
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, profiles: identities, requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, validations: map[string]promptexec.ValidationStatus{}, repairAttempts: map[string]int{}, profiles: identities,
} }
} }
@@ -170,7 +234,8 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
e.mu.Lock() e.mu.Lock()
profile := e.profiles[req.ProfileID] profile := e.profiles[req.ProfileID]
e.mu.Unlock() e.mu.Unlock()
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil { definition := generationDefinitionForPrompt(req.PromptID)
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
e.callbackFailures <- err e.callbackFailures <- err
return nil, err return nil, err
} }
@@ -194,15 +259,20 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
e.mu.Lock() e.mu.Lock()
e.inFlight-- e.inFlight--
err := e.errors[req.ProfileID] err := e.errors[req.ProfileID]
validationStatus := e.validations[req.ProfileID]
repairAttempts := e.repairAttempts[req.ProfileID]
e.mu.Unlock() e.mu.Unlock()
if err != nil { if err != nil {
return nil, err return nil, err
} }
if validationStatus == "" {
validationStatus = promptexec.ValidationPassed
}
return &promptexec.Execution{ return &promptexec.Execution{
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash,
ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName,
StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(), StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(),
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil), Validation: promptexec.NewValidation(validationStatus, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil),
}, nil }, nil
} }
@@ -219,6 +289,13 @@ func (e *barrierExecutor) setError(profileID string, err error) {
e.errors[profileID] = err e.errors[profileID] = err
} }
func (e *barrierExecutor) setValidation(profileID string, status promptexec.ValidationStatus, repairAttempts int) {
e.mu.Lock()
defer e.mu.Unlock()
e.validations[profileID] = status
e.repairAttempts[profileID] = repairAttempts
}
func (e *barrierExecutor) release(profileID string) { func (e *barrierExecutor) release(profileID string) {
close(e.releases[profileID]) close(e.releases[profileID])
} }
@@ -317,4 +394,8 @@ func bytesEqual(left, right []byte) bool {
return reflect.DeepEqual(left, right) return reflect.DeepEqual(left, right)
} }
func intPointer(value int) *int {
return &value
}
var _ promptexec.Executor = (*barrierExecutor)(nil) var _ promptexec.Executor = (*barrierExecutor)(nil)

View File

@@ -87,6 +87,40 @@ func TestCompareDetailedPublishesPartialBundleAndReturnsAggregateError(t *testin
} }
} }
func TestCompareDetailedPublishesPostValidationProfileFailure(t *testing.T) {
bundle := generationBundle(t)
executor := &generationExecutor{complete: func(execution *promptexec.Execution) {
if execution.ProfileID == "weather-deep" {
execution.RawOutput = []byte(`{"summary":42}`)
}
}}
result, err := CompareDetailed(context.Background(), ComparisonRequest{
Config: comparisonConfig(), Report: ReportDaily, ProfileIDs: []string{"weather-light", "weather-deep"},
WorkingDir: t.TempDir(), Date: generationTime("2026-05-29T12:00:00-05:00"),
Clock: timeutil.FixedClock{Time: generationTime("2026-05-29T08:30:00-05:00")},
Collector: &generationCollector{bundle: &bundle}, Executor: executor,
})
if err == nil || result == nil || result.Succeeded != 1 || result.Failed != 1 || result.ManifestPath == "" {
t.Fatalf("CompareDetailed() result/error = %#v/%v", result, err)
}
failure := result.Results[1]
if failure.Status != comparison.StatusFailed || failure.ValidationStatus != promptexec.ValidationPassed || failure.RepairAttempts == nil || *failure.RepairAttempts != 0 {
t.Fatalf("post-validation failure = %#v", failure)
}
data, readErr := os.ReadFile(result.ManifestPath)
if readErr != nil {
t.Fatal(readErr)
}
var manifest comparison.Manifest
if decodeErr := json.Unmarshal(data, &manifest); decodeErr != nil {
t.Fatal(decodeErr)
}
manifestFailure := manifest.Results[1]
if manifestFailure.ValidationStatus != "passed" || manifestFailure.RepairAttempts == nil || *manifestFailure.RepairAttempts != 0 {
t.Fatalf("published post-validation failure = %#v", manifestFailure)
}
}
func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) { func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) {
for _, test := range []struct { for _, test := range []struct {
name string name string

View File

@@ -4,6 +4,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"net/http"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -68,6 +69,7 @@ type generationExecutor struct {
beforeExecute func(promptexec.ExecuteRequest) beforeExecute func(promptexec.ExecuteRequest)
cancelBeforeReturn context.CancelFunc cancelBeforeReturn context.CancelFunc
validation promptexec.ValidationStatus validation promptexec.ValidationStatus
repairAttempts int
validations map[string]promptexec.ValidationStatus validations map[string]promptexec.ValidationStatus
rawOutput []byte rawOutput []byte
waitForCancellation map[string]bool waitForCancellation map[string]bool
@@ -88,7 +90,7 @@ func (e *generationExecutor) InspectPrompt(_ context.Context, id, version string
return promptexec.PromptInspection{}, e.inspectErr return promptexec.PromptInspection{}, e.inspectErr
} }
definition := generationDefinitionForPrompt(id) definition := generationDefinitionForPrompt(id)
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}}, nil return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}}, nil
} }
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) { func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
generationExecutorMu.Lock() generationExecutorMu.Lock()
@@ -112,7 +114,8 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
calls = 1 calls = 1
} }
for range calls { for range calls {
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp} definition := generationDefinitionForPrompt(req.PromptID)
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}
if prepare != nil { if prepare != nil {
prepare(&preparation) prepare(&preparation)
} }
@@ -128,6 +131,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
profileErr := e.executeErrors[req.ProfileID] profileErr := e.executeErrors[req.ProfileID]
executeErr := e.executeErr executeErr := e.executeErr
status := e.validation status := e.validation
repairAttempts := e.repairAttempts
if profileStatus, ok := e.validations[req.ProfileID]; ok { if profileStatus, ok := e.validations[req.ProfileID]; ok {
status = profileStatus status = profileStatus
} }
@@ -162,7 +166,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
if cancelBeforeReturn != nil { if cancelBeforeReturn != nil {
cancelBeforeReturn() cancelBeforeReturn()
} }
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil)} execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil)}
if complete != nil { if complete != nil {
complete(execution) complete(execution)
} }
@@ -546,6 +550,66 @@ func TestGenerateDetailedWritesRequestedPromptDebugArtifacts(t *testing.T) {
} }
} }
func TestGenerateDetailedCapturesProviderFailureOnlyInDebugArtifacts(t *testing.T) {
bundle := generationBundle(t)
debugRoot := t.TempDir()
const marker = "provider-private-generation-marker"
result, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: generationConfig(), Report: ReportDaily,
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle},
Executor: &generationExecutor{executeErr: promptexec.NewGenerationError(http.StatusTooManyRequests, "rate_limit", "provider_error", marker, nil)},
})
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
if err == nil || result == nil || result.LLMDebugPath == "" || promptexec.CategoryOf(err) != promptexec.Generation || !strings.Contains(err.Error(), "HTTP 429") || strings.Contains(err.Error(), marker) || result.OutputPath != "" {
t.Fatalf("GenerateDetailed() result/error = %#v/%v", result, err)
}
for _, name := range []string{"preparation.json", "failure.json"} {
if _, statErr := os.Stat(filepath.Join(result.LLMDebugPath, name)); statErr != nil {
t.Fatalf("debug artifact %q: %v", name, statErr)
}
}
data, readErr := os.ReadFile(filepath.Join(result.LLMDebugPath, "failure.json"))
if readErr != nil || !strings.Contains(string(data), marker) {
t.Fatalf("failure artifact = %q, error = %v", data, readErr)
}
}
func TestGenerateDetailedPreservesProviderFailureWhenFailureDebugWriteFails(t *testing.T) {
bundle := generationBundle(t)
debugRoot := t.TempDir()
const marker = "provider-private-write-failure-marker"
var setupErr error
executor := &generationExecutor{
executeErr: promptexec.NewGenerationError(http.StatusServiceUnavailable, "unavailable", "provider_error", marker, nil),
beforeExecute: func(promptexec.ExecuteRequest) {
setupErr = filepath.Walk(debugRoot, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.Name() == "preparation.json" {
return os.Mkdir(filepath.Join(filepath.Dir(path), "failure.json"), 0o700)
}
return nil
})
},
}
result, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: generationConfig(), Report: ReportDaily,
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle}, Executor: executor,
})
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
var generationError *promptexec.GenerationError
if setupErr != nil || err == nil || result == nil || result.OutputPath != "" || promptexec.CategoryOf(err) != promptexec.Generation || !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusServiceUnavailable || !strings.Contains(err.Error(), "HTTP 503") || strings.Contains(err.Error(), marker) {
t.Fatalf("GenerateDetailed() setup/result/error = %v/%#v/%v", setupErr, result, err)
}
}
func generationConfig() config.Config { func generationConfig() config.Config {
cfg := config.Defaults() cfg := config.Defaults()
cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home" cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home"

View File

@@ -2,6 +2,7 @@ package app
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"reflect" "reflect"
@@ -24,6 +25,7 @@ type profileExecutionOutcome struct {
BackendID string BackendID string
ModelName string ModelName string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string LLMDebugPath string
} }
@@ -94,11 +96,24 @@ func executePreparedProfile(ctx context.Context, req profileExecutionRequest) (p
if callbackFailed { if callbackFailed {
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true} return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true}
} }
if req.DebugWriter != nil && req.DebugWriter.Enabled() && req.DebugRef != nil {
var generationError *promptexec.GenerationError
if errors.As(err, &generationError) {
path, debugErr := req.DebugWriter.WriteFailure(*req.DebugRef, generationError)
if path != "" {
outcome.LLMDebugPath = path
}
if debugErr != nil {
err = errors.Join(err, promptDebugWriteError(debugErr))
}
}
}
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)} return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)}
} }
if execution == nil { if execution == nil {
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)} return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)}
} }
outcome.RepairAttempts = repairAttemptsPointer(execution.Validation.RepairAttempts)
if preparationCount != 1 { if preparationCount != 1 {
return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()} return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()}
} }
@@ -157,7 +172,7 @@ func validatePreparedExecutionRequest(req profileExecutionRequest) error {
if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) { if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) {
return promptProvenanceError() return promptProvenanceError()
} }
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.BackendID == "" || req.Profile.ModelName == "" { if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.ModelName == "" {
return promptProvenanceError() return promptProvenanceError()
} }
return nil return nil
@@ -178,12 +193,19 @@ func validateExecutionProvenance(req profileExecutionRequest, preparation prompt
if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash || if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash ||
execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) || execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) ||
execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName || execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName ||
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" { execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" ||
execution.Validation.RepairAttempts < 0 || execution.Validation.RepairAttempts > preparation.Output.RepairAttempts ||
preparation.Output.RepairAttempts != definition.GeneratedTextRepairAttempts {
return promptProvenanceError() return promptProvenanceError()
} }
return nil return nil
} }
func repairAttemptsPointer(value int) *int {
copy := value
return &copy
}
func promptProvenanceError() error { func promptProvenanceError() error {
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil) return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil)
} }

View File

@@ -26,7 +26,7 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("executePreparedProfile() error = %v", err) t.Fatalf("executePreparedProfile() error = %v", err)
} }
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 { if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 0 || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls) t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls)
} }
if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) { if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) {
@@ -34,6 +34,20 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
} }
} }
func TestExecutePreparedProfileRetainsCompletedRepairAttemptsOnLaterFailure(t *testing.T) {
prepared, inspection := preparedDailyProfile(t)
prepared.resolved.Definition.GeneratedTextRepairAttempts = 1
executor := &generationExecutor{repairAttempts: 1, rawOutput: []byte(`{"summary":42}`), prepare: func(value *promptexec.Preparation) { value.Output.RepairAttempts = 1 }}
outcome, _, err := executePreparedProfile(context.Background(), profileExecutionRequest{
Prepared: prepared, Prompt: inspection,
Profile: promptexec.ProfileInspection{ProfileID: inspection.ProfileID, BackendID: inspection.BackendID, ModelName: inspection.ModelName},
Executor: executor,
})
if err == nil || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 1 {
t.Fatalf("outcome/error = %#v/%v", outcome, err)
}
}
func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) { func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) {
prepared, inspection := preparedDailyProfile(t) prepared, inspection := preparedDailyProfile(t)
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir()) debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())

View File

@@ -48,6 +48,9 @@ func generatePromptReport(ctx context.Context, req promptReportRequest) (*Report
}) })
result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName
result.ValidationStatus = outcome.ValidationStatus result.ValidationStatus = outcome.ValidationStatus
if outcome.RepairAttempts != nil {
result.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
result.LLMDebugPath = outcome.LLMDebugPath result.LLMDebugPath = outcome.LLMDebugPath
if err != nil { if err != nil {
return result, generatedProfileExecutionError(req.Resolved, result.RunID, err) return result, generatedProfileExecutionError(req.Resolved, result.RunID, err)

View File

@@ -2,7 +2,6 @@ package app
import ( import (
"context" "context"
"os"
"strings" "strings"
"gitea.maximumdirect.net/eric/weatherreporter/internal/comparison" "gitea.maximumdirect.net/eric/weatherreporter/internal/comparison"
@@ -18,7 +17,6 @@ type PromptInspectionRequest struct {
Resolved report.Resolved Resolved report.Resolved
Executor promptexec.Executor Executor promptexec.Executor
Promptkit config.PromptkitConfig Promptkit config.PromptkitConfig
LookupEnv func(string) (string, bool)
} }
// PromptInspectionResult contains only safe identity and provenance from a // PromptInspectionResult contains only safe identity and provenance from a
@@ -39,7 +37,6 @@ type PromptExecutionsInspectionRequest struct {
Resolved []report.Resolved Resolved []report.Resolved
Executor promptexec.Executor Executor promptexec.Executor
Promptkit config.PromptkitConfig Promptkit config.PromptkitConfig
LookupEnv func(string) (string, bool)
} }
// ComparisonInspectionRequest contains the explicit profile selection for one // ComparisonInspectionRequest contains the explicit profile selection for one
@@ -48,7 +45,6 @@ type ComparisonInspectionRequest struct {
Resolved report.Resolved Resolved report.Resolved
ProfileIDs []string ProfileIDs []string
Executor promptexec.Executor Executor promptexec.Executor
LookupEnv func(string) (string, bool)
} }
// ComparisonInspectionResult contains the safe, shared prompt identity and // ComparisonInspectionResult contains the safe, shared prompt identity and
@@ -76,7 +72,6 @@ func InspectPromptExecution(ctx context.Context, req PromptInspectionRequest) (P
Resolved: []report.Resolved{req.Resolved}, Resolved: []report.Resolved{req.Resolved},
Executor: req.Executor, Executor: req.Executor,
Promptkit: req.Promptkit, Promptkit: req.Promptkit,
LookupEnv: req.LookupEnv,
}) })
if err != nil { if err != nil {
return PromptInspectionResult{}, err return PromptInspectionResult{}, err
@@ -111,7 +106,7 @@ func InspectPromptExecutions(ctx context.Context, req PromptExecutionsInspection
} }
profile, ok := profiles[profileID] profile, ok := profiles[profileID]
if !ok { if !ok {
profile, err = inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv) profile, err = inspectPromptProfile(ctx, req.Executor, profileID)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -154,7 +149,7 @@ func InspectComparisonExecution(ctx context.Context, req ComparisonInspectionReq
handler: handler, handler: handler,
} }
for _, profileID := range req.ProfileIDs { for _, profileID := range req.ProfileIDs {
profile, err := inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv) profile, err := inspectPromptProfile(ctx, req.Executor, profileID)
if err != nil { if err != nil {
return result, comparisonInspectionError("comparison profile inspection failed", err) return result, comparisonInspectionError("comparison profile inspection failed", err)
} }
@@ -190,7 +185,7 @@ func inspectPromptContract(ctx context.Context, executor promptexec.Executor, de
return inspection, nil return inspection, nil
} }
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string, lookupEnv func(string) (string, bool)) (promptexec.ProfileInspection, error) { func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string) (promptexec.ProfileInspection, error) {
profile, err := executor.InspectProfile(ctx, profileID) profile, err := executor.InspectProfile(ctx, profileID)
if err != nil { if err != nil {
return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err) return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err)
@@ -201,16 +196,7 @@ func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, pro
if profile.CredentialRequired { if profile.CredentialRequired {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil) return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil)
} }
if strings.TrimSpace(profile.APIKeyEnv) != "" { if strings.TrimSpace(profile.ModelName) == "" {
if lookupEnv == nil {
lookupEnv = os.LookupEnv
}
value, present := lookupEnv(profile.APIKeyEnv)
if !present || strings.TrimSpace(value) == "" {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile credential is unavailable", nil)
}
}
if strings.TrimSpace(profile.BackendID) == "" || strings.TrimSpace(profile.ModelName) == "" {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil) return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil)
} }
return profile, nil return profile, nil
@@ -221,7 +207,7 @@ func validPromptInput(inputs []promptexec.InputDefinition) bool {
} }
func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool { func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool {
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" && output.RepairAttempts == definition.GeneratedTextRepairAttempts
} }
func promptInspectionError(operation string, err error) error { func promptInspectionError(operation string, err error) error {

View File

@@ -50,7 +50,6 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
name string name string
prompt promptexec.PromptInspection prompt promptexec.PromptInspection
profile promptexec.ProfileInspection profile promptexec.ProfileInspection
lookupEnv func(string) (string, bool)
wantCategory promptexec.ErrorCategory wantCategory promptexec.ErrorCategory
}{ }{
{ {
@@ -81,9 +80,9 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
wantCategory: promptexec.InvalidConfiguration, wantCategory: promptexec.InvalidConfiguration,
}, },
{ {
name: "missing profile backend", name: "missing profile model",
prompt: basePrompt, prompt: basePrompt,
profile: promptexec.ProfileInspection{ProfileID: "default-profile", ModelName: "model"}, profile: promptexec.ProfileInspection{ProfileID: "default-profile", BackendID: "backend"},
wantCategory: promptexec.InvalidConfiguration, wantCategory: promptexec.InvalidConfiguration,
}, },
{ {
@@ -92,18 +91,11 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true}, profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true},
wantCategory: promptexec.MissingCredential, wantCategory: promptexec.MissingCredential,
}, },
{
name: "missing environment credential",
prompt: basePrompt,
profile: promptexec.ProfileInspection{ProfileID: "default-profile", APIKeyEnv: "PROMPT_API_KEY"},
lookupEnv: func(string) (string, bool) { return "", false },
wantCategory: promptexec.MissingCredential,
},
} }
for _, test := range tests { for _, test := range tests {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}} executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}}
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor, LookupEnv: test.lookupEnv}) _, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor})
if err == nil || promptexec.CategoryOf(err) != test.wantCategory { if err == nil || promptexec.CategoryOf(err) != test.wantCategory {
t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory) t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory)
} }
@@ -265,13 +257,12 @@ func TestInspectComparisonExecutionStopsAtFirstProfileFailure(t *testing.T) {
prompt: validPromptInspection(resolved.Definition), prompt: validPromptInspection(resolved.Definition),
profiles: map[string]promptexec.ProfileInspection{ profiles: map[string]promptexec.ProfileInspection{
"weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"}, "weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"},
"missing-key": {ProfileID: "missing-key", APIKeyEnv: "PROMPT_API_KEY"}, "missing-key": {ProfileID: "missing-key", CredentialRequired: true},
"weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"}, "weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"},
}, },
} }
result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{ result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{
Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor, Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor,
LookupEnv: func(string) (string, bool) { return "", false },
}) })
if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential { if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential {
t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err)) t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err))
@@ -359,7 +350,7 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
return promptexec.PromptInspection{ return promptexec.PromptInspection{
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile", PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts},
} }
} }

View File

@@ -2,8 +2,10 @@ package app_test
import ( import (
"context" "context"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"time" "time"
@@ -14,14 +16,12 @@ import (
) )
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) { func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
lookupEnv := func(string) (string, bool) { return "test-key", true }
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) { inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
t.Helper() t.Helper()
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{ result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, id), Resolved: resolvedPromptProfile(t, id),
Executor: adapter, Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: profile}, Promptkit: config.PromptkitConfig{Profile: profile},
LookupEnv: lookupEnv,
}) })
if err != nil { if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err) t.Fatalf("InspectPromptExecution() error = %v", err)
@@ -50,6 +50,55 @@ model: local-weather
inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather") inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather")
} }
func TestPromptInspectionAcceptsMaintainedEndpointOnlyProfile(t *testing.T) {
adapter, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: filepath.Join("..", "..", "examples", "weather-light-local-profile.yml")})
if err != nil {
t.Fatalf("New() error = %v", err)
}
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly), Executor: adapter,
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" {
t.Fatalf("inspection = %#v", result)
}
if strings.Contains(fmt.Sprintf("%#v", result), "127.0.0.1") {
t.Fatalf("inspection leaks endpoint: %#v", result)
}
}
func TestPromptInspectionSupportsRakestrawhomeProfileOffline(t *testing.T) {
adapter, err := promptkitadapter.New(promptkitadapter.Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
prompt, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly),
Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: "rakestrawhome-gemma-4-31b"},
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if prompt.ProfileID != "rakestrawhome-gemma-4-31b" || prompt.BackendID != "rakestrawhome" || prompt.ModelName == "" {
t.Fatalf("prompt inspection = %#v", prompt)
}
comparison, err := app.InspectComparisonExecution(context.Background(), app.ComparisonInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly),
ProfileIDs: []string{"rakestrawhome-gemma-4-31b", "weather-deep"},
Executor: adapter,
})
if err != nil {
t.Fatalf("InspectComparisonExecution() error = %v", err)
}
if len(comparison.Profiles) != 2 || comparison.Profiles[0].ProfileID != "rakestrawhome-gemma-4-31b" || comparison.Profiles[0].BackendID != "rakestrawhome" || comparison.Profiles[0].ModelName == "" {
t.Fatalf("comparison inspection = %#v", comparison)
}
}
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved { func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
t.Helper() t.Helper()
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC) now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)

View File

@@ -355,7 +355,7 @@ func comparisonResult(outputDirectory string, results []app.ComparisonProfileRes
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
result := &app.ComparisonResult{ result := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report", ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64), PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago", StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory, ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"), ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),

View File

@@ -132,7 +132,7 @@ func actionConfigPath(t *testing.T) string {
func generatedReportResult() *app.ReportResult { func generatedReportResult() *app.ReportResult {
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
return &app.ReportResult{ return &app.ReportResult{
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0",
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago", RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed, ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,

View File

@@ -43,6 +43,7 @@ type generateSummary struct {
ModelName string `json:"modelName,omitempty"` ModelName string `json:"modelName,omitempty"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"` SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
ValidationStatus string `json:"validationStatus,omitempty"` ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
Notification *generateNotificationSummary `json:"notification,omitempty"` Notification *generateNotificationSummary `json:"notification,omitempty"`
Error string `json:"error,omitempty"` Error string `json:"error,omitempty"`
} }
@@ -106,6 +107,7 @@ type comparisonProfileSummary struct {
ModelName string `json:"modelName"` ModelName string `json:"modelName"`
Status string `json:"status"` Status string `json:"status"`
ValidationStatus string `json:"validationStatus,omitempty"` ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
ReportPath string `json:"reportPath,omitempty"` ReportPath string `json:"reportPath,omitempty"`
LLMDebugPath string `json:"llmDebugPath,omitempty"` LLMDebugPath string `json:"llmDebugPath,omitempty"`
Error *comparison.SafeError `json:"error,omitempty"` Error *comparison.SafeError `json:"error,omitempty"`
@@ -129,6 +131,10 @@ func newGenerateSummary(result *app.ReportResult, err error) generateSummary {
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...) summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
summary.ValidationStatus = string(result.ValidationStatus) summary.ValidationStatus = string(result.ValidationStatus)
if result.RepairAttempts != nil {
value := *result.RepairAttempts
summary.RepairAttempts = &value
}
summary.OutputPath = result.OutputPath summary.OutputPath = result.OutputPath
summary.LLMDebugPath = result.LLMDebugPath summary.LLMDebugPath = result.LLMDebugPath
summary.Notification = newGenerateNotificationSummary(result.Notification) summary.Notification = newGenerateNotificationSummary(result.Notification)
@@ -162,6 +168,11 @@ func newGenerateNotificationSummary(result *app.NotificationResult) *generateNot
return summary return summary
} }
func repairAttemptsCopy(value int) *int {
copy := value
return &copy
}
func newBatchSummary(result *app.BatchResult, err error) batchSummary { func newBatchSummary(result *app.BatchResult, err error) batchSummary {
summary := batchSummary{Command: commandRun} summary := batchSummary{Command: commandRun}
if result == nil { if result == nil {
@@ -203,6 +214,9 @@ func newComparisonSummary(result *app.ComparisonResult, err error) comparisonSum
Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath, Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath,
LLMDebugPath: profile.LLMDebugPath, Error: profile.Error, LLMDebugPath: profile.LLMDebugPath, Error: profile.Error,
}) })
if profile.RepairAttempts != nil {
summary.Results[len(summary.Results)-1].RepairAttempts = repairAttemptsCopy(*profile.RepairAttempts)
}
} }
summary.Status = comparisonSummaryStatus(result, err) summary.Status = comparisonSummaryStatus(result, err)
if err != nil { if err != nil {

View File

@@ -19,7 +19,7 @@ import (
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) { func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil) summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 { if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
t.Fatalf("summary = %#v", summary) t.Fatalf("summary = %#v", summary)
} }
@@ -39,7 +39,7 @@ func TestComparisonSummaryUsesLockedOrderAndSafeFields(t *testing.T) {
profileFailure := comparison.NewSafeError("generation", "execute prompt failed") profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
result := &app.ComparisonResult{ result := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report", ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64), PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago", StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily", ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
@@ -113,7 +113,7 @@ func TestComparisonSummaryClassifiesCompleteAndAllFailedResults(t *testing.T) {
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
failure := comparison.NewSafeError("generation", "execute prompt failed") failure := comparison.NewSafeError("generation", "execute prompt failed")
complete := &app.ComparisonResult{ complete := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64), ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)}, StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml", OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
Total: 2, Succeeded: 2, Total: 2, Succeeded: 2,

View File

@@ -15,7 +15,7 @@ import (
const ( const (
// SchemaVersion identifies the supported comparison manifest schema. // SchemaVersion identifies the supported comparison manifest schema.
SchemaVersion = "weatherreporter.comparison.v1" SchemaVersion = "weatherreporter.comparison.v2"
// ManifestFilename is the canonical name of a comparison manifest. // ManifestFilename is the canonical name of a comparison manifest.
ManifestFilename = "comparison.json" ManifestFilename = "comparison.json"
@@ -71,6 +71,7 @@ type Result struct {
ModelName string `json:"modelName"` ModelName string `json:"modelName"`
Status string `json:"status"` Status string `json:"status"`
ValidationStatus string `json:"validationStatus,omitempty"` ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
ReportPath string `json:"reportPath,omitempty"` ReportPath string `json:"reportPath,omitempty"`
Error *SafeError `json:"error,omitempty"` Error *SafeError `json:"error,omitempty"`
} }
@@ -276,11 +277,14 @@ func (manifest Manifest) Validate() error {
if strings.TrimSpace(result.ModelName) == "" { if strings.TrimSpace(result.ModelName) == "" {
return fmt.Errorf("result %d has a blank model name", result.Position) return fmt.Errorf("result %d has a blank model name", result.Position)
} }
if result.RepairAttempts != nil && *result.RepairAttempts < 0 {
return fmt.Errorf("result %d has negative repair attempts", result.Position)
}
switch result.Status { switch result.Status {
case StatusSucceeded: case StatusSucceeded:
succeeded++ succeeded++
if result.ValidationStatus != "passed" { if result.ValidationStatus != "passed" || result.RepairAttempts == nil {
return fmt.Errorf("successful result %d did not pass validation", result.Position) return fmt.Errorf("successful result %d did not pass validation", result.Position)
} }
expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID) expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID)
@@ -302,6 +306,9 @@ func (manifest Manifest) Validate() error {
if err := result.Error.validate(); err != nil { if err := result.Error.validate(); err != nil {
return fmt.Errorf("failed result %d: %w", result.Position, err) return fmt.Errorf("failed result %d: %w", result.Position, err)
} }
if result.ValidationStatus != "" && result.RepairAttempts == nil {
return fmt.Errorf("failed result %d has validation without repair provenance", result.Position)
}
default: default:
return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status) return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status)
} }
@@ -324,7 +331,7 @@ func (safeError SafeError) validate() error {
} }
func isValidationStatus(status string) bool { func isValidationStatus(status string) bool {
return status == "" || status == "failed" || status == "skipped" return status == "" || status == "passed" || status == "failed" || status == "skipped"
} }
func isSHA256(value string) bool { func isSHA256(value string) bool {

View File

@@ -118,7 +118,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
t.Fatalf("EncodeManifest() error = %v", err) t.Fatalf("EncodeManifest() error = %v", err)
} }
want := "{\n" + want := "{\n" +
" \"schemaVersion\": \"weatherreporter.comparison.v1\",\n" + " \"schemaVersion\": \"weatherreporter.comparison.v2\",\n" +
" \"comparisonId\": \"comparison_daily-2026-08-24\",\n" + " \"comparisonId\": \"comparison_daily-2026-08-24\",\n" +
" \"startedAt\": \"2026-08-24T12:00:00Z\",\n" + " \"startedAt\": \"2026-08-24T12:00:00Z\",\n" +
" \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" + " \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" +
@@ -146,6 +146,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
" \"modelName\": \"gpt-5-mini\",\n" + " \"modelName\": \"gpt-5-mini\",\n" +
" \"status\": \"succeeded\",\n" + " \"status\": \"succeeded\",\n" +
" \"validationStatus\": \"passed\",\n" + " \"validationStatus\": \"passed\",\n" +
" \"repairAttempts\": 0,\n" +
" \"reportPath\": \"01-weather-light.md\"\n" + " \"reportPath\": \"01-weather-light.md\"\n" +
" },\n" + " },\n" +
" {\n" + " {\n" +
@@ -201,6 +202,9 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
{name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }}, {name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }},
{name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }}, {name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }},
{name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }}, {name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }},
{name: "failed passed result without repair provenance", mutate: func(manifest *Manifest) {
manifest.Results[1].ValidationStatus = "passed"
}},
{name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }}, {name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }},
{name: "duplicate report path", mutate: func(manifest *Manifest) { {name: "duplicate report path", mutate: func(manifest *Manifest) {
manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath} manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath}
@@ -222,6 +226,17 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
} }
} }
func TestManifestValidateAcceptsPostValidationFailure(t *testing.T) {
t.Parallel()
manifest := validManifest()
manifest.Results[1].ValidationStatus = "passed"
manifest.Results[1].RepairAttempts = intPtr(0)
if err := manifest.Validate(); err != nil {
t.Fatalf("Manifest.Validate() rejected post-validation failure: %v", err)
}
}
func TestLogicalBundleValidate(t *testing.T) { func TestLogicalBundleValidate(t *testing.T) {
t.Parallel() t.Parallel()
@@ -309,6 +324,7 @@ func validManifest() Manifest {
ModelName: "gpt-5-mini", ModelName: "gpt-5-mini",
Status: StatusSucceeded, Status: StatusSucceeded,
ValidationStatus: "passed", ValidationStatus: "passed",
RepairAttempts: intPtr(0),
ReportPath: "01-weather-light.md", ReportPath: "01-weather-light.md",
}, },
{ {
@@ -321,3 +337,8 @@ func validManifest() Manifest {
}, },
} }
} }
func intPtr(value int) *int {
copy := value
return &copy
}

View File

@@ -388,6 +388,7 @@ var resultFields = map[string]jsonValueValidator{
"modelName": nil, "modelName": nil,
"status": nil, "status": nil,
"validationStatus": nil, "validationStatus": nil,
"repairAttempts": validateRepairAttemptsJSON,
"reportPath": nil, "reportPath": nil,
"error": validateSafeErrorJSON, "error": validateSafeErrorJSON,
} }
@@ -430,7 +431,7 @@ func validateResultsJSON(decoder *json.Decoder) error {
return fmt.Errorf("results must be an array") return fmt.Errorf("results must be an array")
} }
for decoder.More() { for decoder.More() {
if err := validateJSONObject(decoder, resultFields); err != nil { if err := validateOrderedJSONObject(decoder, resultFields, []string{"position", "profileId", "backendId", "modelName", "status", "validationStatus", "repairAttempts", "reportPath", "error"}); err != nil {
return err return err
} }
} }
@@ -444,6 +445,65 @@ func validateResultsJSON(decoder *json.Decoder) error {
return nil return nil
} }
func validateOrderedJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator, order []string) error {
token, err := decoder.Token()
if err != nil {
return err
}
if delimiter, ok := token.(json.Delim); !ok || delimiter != '{' {
return fmt.Errorf("manifest value must be an object")
}
seen := make(map[string]struct{}, len(fields))
last := -1
for decoder.More() {
token, err := decoder.Token()
if err != nil {
return err
}
name, ok := token.(string)
if !ok {
return fmt.Errorf("manifest field name is invalid")
}
validator, known := fields[name]
if !known {
return fmt.Errorf("manifest field %q is not canonical", name)
}
if _, duplicate := seen[name]; duplicate {
return fmt.Errorf("manifest field %q is duplicated", name)
}
position := -1
for index, candidate := range order {
if candidate == name {
position = index
break
}
}
if position <= last {
return fmt.Errorf("manifest field %q is out of canonical order", name)
}
seen[name] = struct{}{}
last = position
if validator == nil {
var value json.RawMessage
if err := decoder.Decode(&value); err != nil {
return err
}
continue
}
if err := validator(decoder); err != nil {
return err
}
}
token, err = decoder.Token()
if err != nil {
return err
}
if delimiter, ok := token.(json.Delim); !ok || delimiter != '}' {
return fmt.Errorf("manifest object has an invalid terminator")
}
return nil
}
func validateSafeErrorJSON(decoder *json.Decoder) error { func validateSafeErrorJSON(decoder *json.Decoder) error {
token, err := decoder.Token() token, err := decoder.Token()
if err != nil { if err != nil {
@@ -458,6 +518,24 @@ func validateSafeErrorJSON(decoder *json.Decoder) error {
return validateJSONObjectBody(decoder, safeErrorFields) return validateJSONObjectBody(decoder, safeErrorFields)
} }
func validateRepairAttemptsJSON(decoder *json.Decoder) error {
var raw json.RawMessage
if err := decoder.Decode(&raw); err != nil {
return err
}
if bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
return fmt.Errorf("repairAttempts must be an integer")
}
var value int
if err := json.Unmarshal(raw, &value); err != nil {
return err
}
if value < 0 {
return fmt.Errorf("repairAttempts must not be negative")
}
return nil
}
func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error { func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error {
token, err := decoder.Token() token, err := decoder.Token()
if err != nil { if err != nil {

View File

@@ -1,6 +1,7 @@
package comparison package comparison
import ( import (
"bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
@@ -146,6 +147,23 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
t.Helper() t.Helper()
appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`) appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`)
}}, }},
{name: "null repair attempts", mutate: func(t *testing.T, directory string) {
t.Helper()
path := filepath.Join(directory, ManifestFilename)
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
before := []byte(" \"status\": \"failed\",\n \"error\": {")
after := []byte(" \"status\": \"failed\",\n \"repairAttempts\": null,\n \"error\": {")
data = bytes.Replace(data, before, after, 1)
if bytes.Equal(data, readFile(t, path)) {
t.Fatal("failed to add null repairAttempts")
}
if err := os.WriteFile(path, data, 0o600); err != nil {
t.Fatal(err)
}
}},
{name: "traversal report path", mutate: func(t *testing.T, directory string) { {name: "traversal report path", mutate: func(t *testing.T, directory string) {
t.Helper() t.Helper()
path := filepath.Join(directory, ManifestFilename) path := filepath.Join(directory, ManifestFilename)
@@ -202,6 +220,30 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
} }
} }
func TestValidateRepairAttemptsJSONRejectsNonIntegerValues(t *testing.T) {
t.Parallel()
for _, value := range []string{"null", "-1", "1.5", `"1"`, "9223372036854775808", "{"} {
value := value
t.Run(value, func(t *testing.T) {
t.Parallel()
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err == nil {
t.Fatalf("validateRepairAttemptsJSON(%q) error = nil", value)
}
})
}
}
func TestValidateRepairAttemptsJSONAcceptsNonnegativeIntegers(t *testing.T) {
t.Parallel()
for _, value := range []string{"0", "1"} {
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err != nil {
t.Fatalf("validateRepairAttemptsJSON(%q) error = %v", value, err)
}
}
}
func appendManifestField(t *testing.T, directory, field string) { func appendManifestField(t *testing.T, directory, field string) {
t.Helper() t.Helper()
path := filepath.Join(directory, ManifestFilename) path := filepath.Join(directory, ManifestFilename)
@@ -1077,7 +1119,7 @@ func testBundleWithReports(t *testing.T, count int) LogicalBundle {
} }
manifest.Results[i] = Result{ manifest.Results[i] = Result{
Position: position, ProfileID: profileID, ModelName: "gpt-5-mini", Position: position, ProfileID: profileID, ModelName: "gpt-5-mini",
Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: path, Status: StatusSucceeded, ValidationStatus: "passed", RepairAttempts: intPtr(0), ReportPath: path,
} }
reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")} reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")}
} }

View File

@@ -1,6 +1,2 @@
id: weather-balanced id: weather-balanced
backend: openrouter base_profile: gemini-flash-latest
model: "~google/gemini-flash-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -1,6 +1,2 @@
id: weather-deep id: weather-deep
backend: openrouter base_profile: claude-sonnet-latest
model: "~anthropic/claude-sonnet-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -1,5 +1,2 @@
id: weather-light id: weather-light
backend: openrouter base_profile: deepseek-4-flash
model: deepseek/deepseek-v4-flash
timeout_seconds: 180
service_tier: flex

View File

@@ -1,5 +1,5 @@
id: weather.daily_generated_text id: weather.daily_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-balanced default_profile: weather-balanced
description: Daily weather report analysis prompt. description: Daily weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: daily.generated_text.schema.json schema_path: daily.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.hourly_generated_text id: weather.hourly_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-light default_profile: weather-light
description: Hourly weather report analysis prompt. description: Hourly weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: hourly.generated_text.schema.json schema_path: hourly.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.today_generated_text id: weather.today_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-balanced default_profile: weather-balanced
description: Today's weather report analysis prompt. description: Today's weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: today.generated_text.schema.json schema_path: today.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.tomorrow_generated_text id: weather.tomorrow_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-balanced default_profile: weather-balanced
description: Tomorrow's weather report analysis prompt. description: Tomorrow's weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: tomorrow.generated_text.schema.json schema_path: tomorrow.generated_text.schema.json
repair_attempts: 1

View File

@@ -77,8 +77,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if err := yaml.Unmarshal(data, &definition); err != nil { if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("decode prompt definition: %v", err) t.Fatalf("decode prompt definition: %v", err)
} }
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile { if definition.ID != tc.id || definition.Version != "2.1.0" || definition.DefaultProfile != tc.profile {
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile) t.Fatalf("definition = %#v, want %s version 2.1.0 and profile %s", definition, tc.id, tc.profile)
} }
sharedInstruction := false sharedInstruction := false
for _, message := range definition.Messages { for _, message := range definition.Messages {
@@ -92,8 +92,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" { if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs) t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
} }
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts != nil { if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts == nil || *definition.Output.RepairAttempts != 1 {
t.Fatalf("output = %#v, want JSON schema output without repair attempts", definition.Output) t.Fatalf("output = %#v, want JSON schema output with one repair attempt", definition.Output)
} }
if _, err := promptassets.Schema(tc.schemaID); err != nil { if _, err := promptassets.Schema(tc.schemaID); err != nil {
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err) t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
@@ -286,11 +286,11 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"}, {"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
} { } {
t.Run(want.id, func(t *testing.T) { t.Run(want.id, func(t *testing.T) {
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.0.0") inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.1.0")
if err != nil { if err != nil {
t.Fatalf("InspectPrompt() error = %v", err) t.Fatalf("InspectPrompt() error = %v", err)
} }
if inspection.PromptID != want.id || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != want.profile { if inspection.PromptID != want.id || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != want.profile || inspection.OutputContract.RepairAttempts != 1 {
t.Fatalf("inspection = %#v", inspection) t.Fatalf("inspection = %#v", inspection)
} }
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID) profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
@@ -361,6 +361,29 @@ func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
} }
} }
func TestEmbeddedProfilesAreMinimalBaseAliases(t *testing.T) {
wantBases := map[string]string{
"weather-balanced.yml": "gemini-flash-latest",
"weather-deep.yml": "claude-sonnet-latest",
"weather-light.yml": "deepseek-4-flash",
}
for path, wantBase := range wantBases {
t.Run(path, func(t *testing.T) {
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
if err != nil {
t.Fatalf("read profile: %v", err)
}
var definition map[string]string
if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("unmarshal profile: %v", err)
}
if definition["id"] != strings.TrimSuffix(path, ".yml") || definition["base_profile"] != wantBase || len(definition) != 2 {
t.Fatalf("profile definition = %#v, want only its id and base profile %q", definition, wantBase)
}
})
}
}
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) { func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"} forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error { if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
@@ -391,7 +414,7 @@ func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
if err != nil { if err != nil {
return err return err
} }
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "repair_attempts:", "weather.daily_report"} { for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "weather.daily_report"} {
if strings.Contains(string(data), unwanted) { if strings.Contains(string(data), unwanted) {
t.Fatalf("%s contains retired runtime setting %q", path, unwanted) t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
} }

View File

@@ -21,8 +21,9 @@ import (
var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform") var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform")
const ( const (
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v2" promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v3"
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v2" promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v3"
promptFailureDebugSchemaVersion = "weatherreporter.prompt_failure_debug.v1"
debugDirectoryMode = 0o700 debugDirectoryMode = 0o700
debugFileMode = 0o600 debugFileMode = 0o600
) )
@@ -53,6 +54,7 @@ type PromptDebugOutput struct {
Format string `json:"format"` Format string `json:"format"`
ValidationMode string `json:"validationMode"` ValidationMode string `json:"validationMode"`
SchemaPath string `json:"schemaPath"` SchemaPath string `json:"schemaPath"`
RepairAttempts int `json:"repairAttempts"`
} }
// PromptDebugPreparation is the explicit, content-safe mapping of preparation // PromptDebugPreparation is the explicit, content-safe mapping of preparation
@@ -97,10 +99,28 @@ type PromptDebugUsage struct {
// PromptDebugValidation is the completed validation detail retained in the // PromptDebugValidation is the completed validation detail retained in the
// explicitly enabled debug store. // explicitly enabled debug store.
type PromptDebugValidation struct { type PromptDebugValidation struct {
Status string `json:"status"` Status string `json:"status"`
Mode string `json:"mode"` Mode string `json:"mode"`
SchemaPath string `json:"schemaPath"` SchemaPath string `json:"schemaPath"`
Diagnostics []string `json:"diagnostics,omitempty"` RepairAttempts int `json:"repairAttempts"`
Diagnostics []string `json:"diagnostics,omitempty"`
}
// PromptFailureDebugArtifact records provider detail only in explicit debug storage.
type PromptFailureDebugArtifact struct {
SchemaVersion string `json:"schemaVersion"`
ReportID report.ID `json:"reportId"`
ValidDate string `json:"validDate"`
RunID string `json:"runId"`
Failure PromptFailure `json:"failure"`
}
type PromptFailure struct {
Category string `json:"category"`
StatusCode int `json:"statusCode,omitempty"`
ProviderCode string `json:"providerCode,omitempty"`
ProviderType string `json:"providerType,omitempty"`
ProviderMessage string `json:"providerMessage,omitempty"`
} }
// PromptDebugExecution is the explicit mapping of execution provenance. // PromptDebugExecution is the explicit mapping of execution provenance.
@@ -235,6 +255,24 @@ func (w *PromptDebugWriter) WriteExecution(ref PromptDebugRef, execution prompte
return directory, nil return directory, nil
} }
// WriteFailure stores the project-owned structured provider failure.
func (w *PromptDebugWriter) WriteFailure(ref PromptDebugRef, failure *promptexec.GenerationError) (string, error) {
if !w.Enabled() || failure == nil {
return "", nil
}
directory, secureDirectory, err := w.runDirectory(ref)
if err != nil {
return "", err
}
defer secureDirectory.Close()
artifact := PromptFailureDebugArtifact{SchemaVersion: promptFailureDebugSchemaVersion, ReportID: ref.ReportID, ValidDate: ref.ValidDate, RunID: ref.RunID,
Failure: PromptFailure{Category: string(failure.Category()), StatusCode: failure.StatusCode(), ProviderCode: failure.ProviderCode(), ProviderType: failure.ProviderType(), ProviderMessage: failure.ProviderMessage()}}
if err := secureDirectory.writeJSON("failure.json", artifact); err != nil {
return "", err
}
return directory, nil
}
func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) { func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) {
if err := validatePromptDebugRef(ref); err != nil { if err := validatePromptDebugRef(ref); err != nil {
return "", nil, err return "", nil, err
@@ -283,7 +321,7 @@ func promptDebugPreparation(value promptexec.Preparation) PromptDebugPreparation
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash, PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash,
RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes), RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes),
ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName, ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName,
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath}, Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath, RepairAttempts: value.Output.RepairAttempts},
StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration, StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration,
} }
} }
@@ -300,7 +338,7 @@ func promptDebugExecution(value promptexec.Execution) PromptDebugExecution {
} }
func promptDebugValidation(value promptexec.Validation) PromptDebugValidation { func promptDebugValidation(value promptexec.Validation) PromptDebugValidation {
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, Diagnostics: append([]string(nil), value.Diagnostics...)} return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, RepairAttempts: value.RepairAttempts, Diagnostics: append([]string(nil), value.Diagnostics...)}
} }
func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage { func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage {

View File

@@ -44,13 +44,13 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir) t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir)
} }
preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json")) preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json"))
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v2", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`} { for _, want := range []string{"weatherreporter.prompt_preparation_debug.v3", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`, `"repairAttempts": 0`} {
if !strings.Contains(string(preparationData), want) { if !strings.Contains(string(preparationData), want) {
t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData) t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData)
} }
} }
executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json")) executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json"))
for _, want := range []string{"weatherreporter.prompt_execution_debug.v2", "Generated forecast prose.", "validation details", `"status": "passed"`} { for _, want := range []string{"weatherreporter.prompt_execution_debug.v3", "Generated forecast prose.", "validation details", `"status": "passed"`, `"repairAttempts": 0`} {
if !strings.Contains(string(executionData), want) { if !strings.Contains(string(executionData), want) {
t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData) t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData)
} }
@@ -66,6 +66,25 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode) assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode)
} }
func TestPromptDebugWriterWritesProviderFailureOnlyToDebugStore(t *testing.T) {
writer, err := NewPromptDebugWriter(filepath.Join(t.TempDir(), "operator-debug"))
if err != nil {
t.Fatalf("NewPromptDebugWriter() error = %v", err)
}
marker := "provider-private-marker"
directory, err := writer.WriteFailure(promptDebugRef(), promptexec.NewGenerationError(429, "rate_limit", "provider_error", marker, nil))
if err != nil {
t.Fatalf("WriteFailure() error = %v", err)
}
data := readPromptDebugFile(t, filepath.Join(directory, "failure.json"))
for _, want := range []string{"weatherreporter.prompt_failure_debug.v1", `"category": "generation"`, `"statusCode": 429`, marker} {
if !strings.Contains(string(data), want) {
t.Fatalf("failure artifact missing %q: %s", want, data)
}
}
assertPromptDebugMode(t, filepath.Join(directory, "failure.json"), debugFileMode)
}
func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) { func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) {
const marker = "private-debug-marker" const marker = "private-debug-marker"
tests := []struct { tests := []struct {
@@ -349,7 +368,7 @@ func promptDebugExecutionFixture() promptexec.Execution {
InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model", InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model",
GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15}, GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15},
StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second, StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second,
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", []string{"validation details"}), Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", 0, []string{"validation details"}),
RawOutput: []byte("Generated forecast prose."), RawOutput: []byte("Generated forecast prose."),
Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}}, Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}},
} }

View File

@@ -78,3 +78,15 @@ func boundText(value string, limit int) string {
} }
return value return value
} }
func boundCodePoints(value string, limit int) string {
if limit <= 0 || value == "" {
return ""
}
value = strings.ToValidUTF8(value, "<22>")
runes := []rune(value)
if len(runes) <= limit {
return value
}
return string(runes[:limit])
}

View File

@@ -51,6 +51,7 @@ type OutputContract struct {
Format string Format string
ValidationMode string ValidationMode string
SchemaPath string SchemaPath string
RepairAttempts int
} }
// ProfileInspection describes the safe, selected execution identity for one profile. // ProfileInspection describes the safe, selected execution identity for one profile.
@@ -141,19 +142,21 @@ type TokenUsage struct {
// Validation records a completed output validation check. // Validation records a completed output validation check.
type Validation struct { type Validation struct {
Status ValidationStatus Status ValidationStatus
Mode string Mode string
SchemaPath string SchemaPath string
Diagnostics []string RepairAttempts int
Diagnostics []string
} }
// NewValidation returns a completed validation value with bounded diagnostics. // NewValidation returns a completed validation value with bounded diagnostics.
func NewValidation(status ValidationStatus, mode string, schemaPath string, diagnostics []string) Validation { func NewValidation(status ValidationStatus, mode string, schemaPath string, repairAttempts int, diagnostics []string) Validation {
return Validation{ return Validation{
Status: status, Status: status,
Mode: mode, Mode: mode,
SchemaPath: schemaPath, SchemaPath: schemaPath,
Diagnostics: boundDiagnostics(diagnostics), RepairAttempts: repairAttempts,
Diagnostics: boundDiagnostics(diagnostics),
} }
} }
@@ -235,6 +238,86 @@ func (e *Error) Category() ErrorCategory {
return e.category return e.category
} }
// GenerationError retains provider failure details for programmatic handling
// without exposing them through routine formatting or serialization.
type GenerationError struct {
statusCode int
providerCode string
providerType string
providerMessage string
err *Error
}
// NewGenerationError returns a classified generation failure with bounded
// provider details. The dependency cause remains reachable through err only.
func NewGenerationError(statusCode int, providerCode string, providerType string, providerMessage string, cause error) *GenerationError {
return &GenerationError{
statusCode: statusCode,
providerCode: boundCodePoints(providerCode, 256),
providerType: boundCodePoints(providerType, 256),
providerMessage: boundCodePoints(providerMessage, 4096),
err: NewError(Generation, "provider generation failed", cause),
}
}
// StatusCode returns the provider HTTP status when one was available.
func (e *GenerationError) StatusCode() int {
if e == nil {
return 0
}
return e.statusCode
}
// ProviderCode returns the bounded provider error code.
func (e *GenerationError) ProviderCode() string {
if e == nil {
return ""
}
return e.providerCode
}
// ProviderType returns the bounded provider error type.
func (e *GenerationError) ProviderType() string {
if e == nil {
return ""
}
return e.providerType
}
// ProviderMessage returns the bounded provider error message.
func (e *GenerationError) ProviderMessage() string {
if e == nil {
return ""
}
return e.providerMessage
}
// Category returns Generation for every generation failure.
func (e *GenerationError) Category() ErrorCategory { return Generation }
// Error intentionally excludes provider details from ordinary error text.
func (e *GenerationError) Error() string {
if e == nil {
return ""
}
message := NewError(Generation, "provider generation failed", nil).Error()
if e.statusCode == 0 {
return message
}
return fmt.Sprintf("%s (HTTP %d)", message, e.statusCode)
}
// GoString keeps %#v formatting as safe as ordinary error formatting.
func (e *GenerationError) GoString() string { return e.Error() }
// Unwrap preserves the project-owned classified error and its hidden cause.
func (e *GenerationError) Unwrap() error {
if e == nil {
return nil
}
return e.err
}
// CapacityError adds the safe backend identity to a capacity failure. // CapacityError adds the safe backend identity to a capacity failure.
type CapacityError struct { type CapacityError struct {
BackendID string BackendID string

View File

@@ -3,6 +3,7 @@ package promptexec
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"strings" "strings"
"testing" "testing"
"unicode/utf8" "unicode/utf8"
@@ -172,8 +173,8 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
if bounded[0] != "a<>b" { if bounded[0] != "a<>b" {
t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0]) t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0])
} }
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", values) validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", 2, values)
if len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" { if validation.RepairAttempts != 2 || len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
t.Fatalf("validation = %#v, want bounded diagnostics", validation) t.Fatalf("validation = %#v, want bounded diagnostics", validation)
} }
@@ -187,8 +188,53 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
} }
} }
func TestGenerationErrorKeepsProviderDetailsOutOfRoutineFormatting(t *testing.T) {
cause := errors.New("provider response must stay hidden")
error := NewGenerationError(
429,
string([]byte{'c', 0xff})+strings.Repeat("界", 300),
strings.Repeat("type", 100),
string([]byte{'m', 0xff})+strings.Repeat("界", 4_200),
cause,
)
if error.StatusCode() != 429 || error.Category() != Generation || CategoryOf(error) != Generation {
t.Fatalf("generation error identity = %#v", error)
}
if utf8.RuneCountInString(error.ProviderCode()) != 256 || utf8.RuneCountInString(error.ProviderType()) != 256 || utf8.RuneCountInString(error.ProviderMessage()) != 4096 {
t.Fatalf("provider detail bounds = %d/%d/%d", utf8.RuneCountInString(error.ProviderCode()), utf8.RuneCountInString(error.ProviderType()), utf8.RuneCountInString(error.ProviderMessage()))
}
if !utf8.ValidString(error.ProviderCode()) || !utf8.ValidString(error.ProviderType()) || !utf8.ValidString(error.ProviderMessage()) {
t.Fatalf("provider details are not valid UTF-8: %#v", error)
}
if !errors.Is(error, cause) {
t.Fatal("errors.Is() = false, want preserved dependency cause")
}
var owned *Error
if !errors.As(error, &owned) || owned.Category() != Generation {
t.Fatalf("errors.As(*Error) = %#v, want project-owned generation error", owned)
}
want := "generation: provider generation failed (HTTP 429)"
if error.Error() != want || fmt.Sprintf("%#v", error) != want {
t.Fatalf("ordinary formatting = %q / %#v, want %q", error.Error(), error, want)
}
for _, private := range []string{cause.Error(), error.ProviderCode(), error.ProviderType(), error.ProviderMessage()} {
if strings.Contains(error.Error(), private) || strings.Contains(fmt.Sprintf("%#v", error), private) {
t.Fatalf("generation error leaks provider detail %q", private)
}
}
statusOnly := NewGenerationError(503, "", "", "", nil)
if statusOnly.Error() != "generation: provider generation failed (HTTP 503)" {
t.Fatalf("status-only error = %q", statusOnly)
}
var nilError *GenerationError
if nilError.StatusCode() != 0 || nilError.ProviderCode() != "" || nilError.ProviderType() != "" || nilError.ProviderMessage() != "" || nilError.Category() != Generation || nilError.Error() != "" || nilError.GoString() != "" || nilError.Unwrap() != nil {
t.Fatalf("nil generation error = %#v", nilError)
}
}
func TestContractCopiesMutableValues(t *testing.T) { func TestContractCopiesMutableValues(t *testing.T) {
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}} preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}, Output: OutputContract{RepairAttempts: 3}}
preparationDebug := &PreparationDebug{ preparationDebug := &PreparationDebug{
RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}}, RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}},
StructuredSchema: []byte("schema body"), StructuredSchema: []byte("schema body"),
@@ -196,7 +242,7 @@ func TestContractCopiesMutableValues(t *testing.T) {
} }
execution := Execution{ execution := Execution{
InputHashes: map[string]string{"data_package": "input-hash"}, InputHashes: map[string]string{"data_package": "input-hash"},
Validation: Validation{Diagnostics: []string{"validation detail"}}, Validation: Validation{RepairAttempts: 2, Diagnostics: []string{"validation detail"}},
RawOutput: []byte("generated output"), RawOutput: []byte("generated output"),
Debug: &ExecutionDebug{ Debug: &ExecutionDebug{
RawOutput: []byte("provider output"), RawOutput: []byte("provider output"),
@@ -217,13 +263,13 @@ func TestContractCopiesMutableValues(t *testing.T) {
execution.Debug.RawOutput[0] = 'x' execution.Debug.RawOutput[0] = 'x'
execution.Debug.ValidationDiagnostics[0] = "changed" execution.Debug.ValidationDiagnostics[0] = "changed"
if preparationCopy.InputHashes["data_package"] != "input-hash" { if preparationCopy.InputHashes["data_package"] != "input-hash" || preparationCopy.Output.RepairAttempts != 3 {
t.Fatalf("preparation copy = %#v", preparationCopy) t.Fatalf("preparation copy = %#v", preparationCopy)
} }
if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` { if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` {
t.Fatalf("preparation debug copy = %#v", debugCopy) t.Fatalf("preparation debug copy = %#v", debugCopy)
} }
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" { if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.RepairAttempts != 2 || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
t.Fatalf("execution copy = %#v", executionCopy) t.Fatalf("execution copy = %#v", executionCopy)
} }
} }

View File

@@ -9,14 +9,15 @@ import (
func dailyDefinition() Definition { func dailyDefinition() Definition {
return Definition{ return Definition{
ID: Daily, ID: Daily,
Name: "Daily Report", Name: "Daily Report",
PromptID: "weather.daily_generated_text", PromptID: "weather.daily_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "daily", TemplateID: "daily",
GeneratedTextSchemaID: "daily", GeneratedTextSchemaID: "daily",
ArtifactGroup: "daily", GeneratedTextRepairAttempts: 1,
OutputName: "daily.md", ArtifactGroup: "daily",
OutputName: "daily.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",

View File

@@ -27,20 +27,21 @@ const (
) )
type Definition struct { type Definition struct {
ID ID ID ID
Name string Name string
PromptID string PromptID string
PromptVersion string PromptVersion string
TemplateID string TemplateID string
GeneratedTextSchemaID string GeneratedTextSchemaID string
ArtifactGroup string GeneratedTextRepairAttempts int
OutputName string ArtifactGroup string
DistributorPathTemplates []string OutputName string
Modules []module.ConfigItem DistributorPathTemplates []string
Morning bool Modules []module.ConfigItem
Evening bool Morning bool
resolve func(ResolveRequest) (timeutil.Period, error) Evening bool
runIDDisambiguator func(Resolved) string resolve func(ResolveRequest) (timeutil.Period, error)
runIDDisambiguator func(Resolved) string
} }
func (r Resolved) OutputName() (string, error) { func (r Resolved) OutputName() (string, error) {

View File

@@ -11,14 +11,15 @@ const hourlyReportHours = 6
func hourlyDefinition() Definition { func hourlyDefinition() Definition {
return Definition{ return Definition{
ID: Hourly, ID: Hourly,
Name: "Hourly Report", Name: "Hourly Report",
PromptID: "weather.hourly_generated_text", PromptID: "weather.hourly_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "hourly", TemplateID: "hourly",
GeneratedTextSchemaID: "hourly", GeneratedTextSchemaID: "hourly",
ArtifactGroup: "hourly", GeneratedTextRepairAttempts: 1,
OutputName: "hourly.md", ArtifactGroup: "hourly",
OutputName: "hourly.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"hourly/index.md", "hourly/index.md",
}, },

View File

@@ -82,8 +82,8 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
} }
for _, definition := range definitions { for _, definition := range definitions {
if definition.PromptVersion != "2.0.0" { if definition.PromptVersion != "2.1.0" {
t.Fatalf("%s PromptVersion = %q, want 2.0.0", definition.ID, definition.PromptVersion) t.Fatalf("%s PromptVersion = %q, want 2.1.0", definition.ID, definition.PromptVersion)
} }
if definition.PromptID == "" { if definition.PromptID == "" {
t.Fatalf("%s PromptID is empty", definition.ID) t.Fatalf("%s PromptID is empty", definition.ID)
@@ -91,6 +91,9 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" { if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID) t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
} }
if definition.GeneratedTextRepairAttempts != 1 {
t.Fatalf("%s repair attempts = %d, want 1", definition.ID, definition.GeneratedTextRepairAttempts)
}
} }
if _, err := registry.Lookup(ID("three_day")); err == nil { if _, err := registry.Lookup(ID("three_day")); err == nil {

View File

@@ -7,14 +7,15 @@ import (
func todayDefinition() Definition { func todayDefinition() Definition {
return Definition{ return Definition{
ID: Today, ID: Today,
Name: "Today Report", Name: "Today Report",
PromptID: "weather.today_generated_text", PromptID: "weather.today_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "today", TemplateID: "today",
GeneratedTextSchemaID: "today", GeneratedTextSchemaID: "today",
ArtifactGroup: "today", GeneratedTextRepairAttempts: 1,
OutputName: "today.md", ArtifactGroup: "today",
OutputName: "today.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",

View File

@@ -7,14 +7,15 @@ import (
func tomorrowDefinition() Definition { func tomorrowDefinition() Definition {
return Definition{ return Definition{
ID: Tomorrow, ID: Tomorrow,
Name: "Tomorrow Report", Name: "Tomorrow Report",
PromptID: "weather.tomorrow_generated_text", PromptID: "weather.tomorrow_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "tomorrow", TemplateID: "tomorrow",
GeneratedTextSchemaID: "tomorrow", GeneratedTextSchemaID: "tomorrow",
ArtifactGroup: "tomorrow", GeneratedTextRepairAttempts: 1,
OutputName: "tomorrow.md", ArtifactGroup: "tomorrow",
OutputName: "tomorrow.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",