14 Commits

53 changed files with 1341 additions and 217 deletions

View File

@@ -96,7 +96,7 @@ period, prompt version, timezone, and status. Successful output has an absolute
"command": "generate", "command": "generate",
"reportId": "today", "reportId": "today",
"promptId": "weather.today_generated_text", "promptId": "weather.today_generated_text",
"promptVersion": "2.0.0", "promptVersion": "2.1.0",
"runId": "20260529T120000.000000000Z_today", "runId": "20260529T120000.000000000Z_today",
"status": "succeeded", "status": "succeeded",
"timezone": "America/Chicago", "timezone": "America/Chicago",
@@ -106,7 +106,10 @@ period, prompt version, timezone, and status. Successful output has an absolute
When available, the summary also includes the effective `profileId`, When available, the summary also includes the effective `profileId`,
`backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested `backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested
`llmDebugPath`, and compact Distributor `notification` result. It does not `repairAttempts`, `llmDebugPath`, and compact Distributor `notification`
result. `repairAttempts` is `0` when the initial output passed validation,
positive when PromptKit made corrective generation calls, and omitted when
validation did not complete. The summary does not
include historical or transient artifact paths such as metadata, prompt input, include historical or transient artifact paths such as metadata, prompt input,
raw generated text, render context, or notification receipts. raw generated text, render context, or notification receipts.
@@ -115,7 +118,8 @@ raw generated text, render context, or notification receipts.
A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`, A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`,
`total`, `succeeded`, `failed`, and a `reports` array. Each report item includes `total`, `succeeded`, `failed`, and a `reports` array. Each report item includes
its identity, status, effective profile and model details when available, its identity, status, effective profile and model details when available,
source warnings, validation status, and absolute `outputPath` after publication. source warnings, validation status, repair-attempt count when validation
completed, and absolute `outputPath` after publication.
The top-level summary may also contain a batch `notification` object and The top-level summary may also contain a batch `notification` object and
`error`. Batch status is `failed` if any report or the batch notification fails. `error`. Batch status is `failed` if any report or the batch notification fails.
The `total`, `succeeded`, and `failed` counters describe report items only, so The `total`, `succeeded`, and `failed` counters describe report items only, so
@@ -143,7 +147,8 @@ A comparison summary contains these fields in this order: `command`,
successful `results[].reportPath` are absolute. `results` preserves the successful `results[].reportPath` are absolute. `results` preserves the
supplied profile order and each item contains `position`, `profileId`, optional supplied profile order and each item contains `position`, `profileId`, optional
`backendId`, `modelName`, `status`, optional `validationStatus`, optional `backendId`, `modelName`, `status`, optional `validationStatus`, optional
`reportPath`, optional `llmDebugPath`, and optional safe `error`. `repairAttempts`, optional `reportPath`, optional `llmDebugPath`, and optional
safe `error`. The repair-attempt semantics match the generate summary.
The comparison status is `succeeded` only when every selected profile succeeds The comparison status is `succeeded` only when every selected profile succeeds
and the bundle is published. Individual profile failures still publish a and the bundle is published. Individual profile failures still publish a
@@ -157,7 +162,8 @@ published Promptkit category; destination failures use `destination_<kind>`;
and committed cleanup failures use `publication_cleanup` with a message that and committed cleanup failures use `publication_cleanup` with a message that
states whether a complete prior bundle, partial remnants, or no prior bundle states whether a complete prior bundle, partial remnants, or no prior bundle
remains, or that recovery state could not be inspected. It does not expose remains, or that recovery state could not be inspected. It does not expose
provider diagnostics, filesystem causes, or recovery paths. See the provider diagnostics, filesystem causes, or recovery paths. A provider HTTP
failure may include its numeric status in the safe message. See the
[comparison bundle contract](integrations/comparison-bundle.md) for durable [comparison bundle contract](integrations/comparison-bundle.md) for durable
artifact fields and failure invariants. artifact fields and failure invariants.

View File

@@ -203,16 +203,23 @@ for platform availability, security, and retention requirements.
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions. `profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
They are separate decisions. An explicit `profile` applies to every selected They are separate decisions. An explicit `profile` applies to every selected
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
Tomorrow select `weather-balanced` through their exact `2.0.0` prompt Tomorrow select `weather-balanced` through their exact `2.1.0` prompt
definitions. definitions.
Promptkit resolves a selected profile definition from a test or embedding Promptkit resolves a selected profile definition from a test or embedding
consumer's explicit in-memory profile, then the configured `profile_file` or consumer's explicit in-memory profile, then the configured `profile_file` or
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's `profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
built-in catalog. Sources provide complete definitions; fields are never maintained catalog. Weatherreporter's embedded `weather-*` definitions are small
merged. A matching malformed external profile fails rather than using the aliases of Promptkit's maintained base profiles, so Promptkit also resolves
embedded definition. The [Promptkit integration guide](integrations/promptkit.md) their inherited target and settings. A configured definition with the same ID
owns the catalog and precedence details. as either a selected profile or an inherited base takes precedence. A matching
malformed external profile fails rather than using the embedded definition. The
[Promptkit integration guide](integrations/promptkit.md) owns the catalog and
precedence details.
An endpoint-only profile may intentionally have no backend identity. Profiles
that require a direct API key are rejected before collection, while Promptkit
resolves optional environment credential sources during execution.
To replace the default Hourly definition with a local OpenAI-compatible To replace the default Hourly definition with a local OpenAI-compatible
endpoint, set `profile_file` to a copy of endpoint, set `profile_file` to a copy of

View File

@@ -9,7 +9,7 @@ and retention belong to the [operations guide](../operations.md).
## Version And Layout ## Version And Layout
The current and only supported manifest schema version is The current and only supported manifest schema version is
`weatherreporter.comparison.v1`. A bundle directory contains exactly these `weatherreporter.comparison.v2`. A bundle directory contains exactly these
regular, non-symlinked files: regular, non-symlinked files:
```text ```text
@@ -51,7 +51,7 @@ this order:
```text ```text
position, profileId, backendId, modelName, status, validationStatus, position, profileId, backendId, modelName, status, validationStatus,
reportPath, error repairAttempts, reportPath, error
``` ```
`startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not `startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not
@@ -66,12 +66,16 @@ contiguous from one, profile IDs are distinct and nonblank, and
`succeeded + failed == total`. `succeeded + failed == total`.
A successful result has `status: "succeeded"`, `validationStatus: "passed"`, A successful result has `status: "succeeded"`, `validationStatus: "passed"`,
and a non-negative `repairAttempts` count,
a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for
its position, total, and logical profile ID, and no `error`. A failed result has its position, total, and logical profile ID, and no `error`. A failed result has
`status: "failed"`, no `reportPath`, and an `error` object with nonblank `status: "failed"`, no `reportPath`, and an `error` object with nonblank
`category` and `message`. Its validation status is absent, `failed`, or `category` and `message`. Its validation status is absent, `failed`, or
`skipped`. Error messages are valid UTF-8 and no longer than 1,024 bytes. `skipped`; it may also be `passed` when a WeatherReporter step after PromptKit
`backendId` and `validationStatus` are omitted when unavailable. validation failed. Error messages are valid UTF-8 and no longer than 1,024 bytes.
`backendId` and `validationStatus` are omitted when unavailable. A failed
result with any completed validation status must retain its non-negative
`repairAttempts`; early operational failures omit both fields.
Every successful Markdown file is declared by exactly one successful result. Every successful Markdown file is declared by exactly one successful result.
The directory contains no extra entries. Consumers can therefore verify the The directory contains no extra entries. Consumers can therefore verify the
@@ -88,7 +92,9 @@ case-variant, or duplicate fields; multiple JSON values; extra entries;
symlinks; and future or otherwise unsupported versions. Treat a bundle that symlinks; and future or otherwise unsupported versions. Treat a bundle that
fails recognition as an ordinary directory, not as a compatible bundle. fails recognition as an ordinary directory, not as a compatible bundle.
When replacing a recognized bundle, cancellation observed before the new Only v2 is recognized as a replaceable bundle; v1 is unsupported and must be
moved or removed before a replacement at the same destination. When replacing
a recognized bundle, cancellation observed before the new
bundle is installed preserves the prior bundle rather than committing the bundle is installed preserves the prior bundle rather than committing the
replacement. replacement.

View File

@@ -1,10 +1,12 @@
# Promptkit Integration # Promptkit Integration
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.0.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`. Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.1.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
## Logical Profile Catalog ## Logical Profile Catalog
Prompt definitions select a stable Weatherreporter profile ID. The embedded definitions currently use Promptkit's `openrouter` backend: Prompt definitions select a stable Weatherreporter profile ID. Each embedded
definition contains only its ID and one Promptkit base-profile reference; the
effective execution settings resolve from Promptkit's maintained catalog:
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports | | Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
| --- | --- | --- | --- | --- | --- | | --- | --- | --- | --- | --- | --- |
@@ -12,22 +14,45 @@ Prompt definitions select a stable Weatherreporter profile ID. The embedded defi
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow | | `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None | | `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded profiles intentionally omit endpoints, credentials, temperature, `top_p`, and output-token limits. The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
profiles intentionally omit endpoints, credentials, and execution settings;
Promptkit owns inherited resolution and its provider-native defaults. Promptkit
assembles its maintained catalog from independently versioned OpenRouter and
Rakestrawhome modules selected by the Promptkit release. Weatherreporter does
not import or register those catalog modules directly. The maintained
`rakestrawhome-gemma-4-31b` profile is also available for ordinary and
comparison selection and reports the `rakestrawhome` backend without
Weatherreporter-specific configuration.
## Selection And Active Execution ## Selection And Active Execution
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank backend and model identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order: Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank model; an endpoint-only profile may intentionally have no backend identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
1. explicit in-memory profiles used by an embedding consumer or test; 1. explicit in-memory profiles used by an embedding consumer or test;
2. the configured `profile_file` or `profile_dir`; 2. the configured `profile_file` or `profile_dir`;
3. Weatherreporter's embedded fallback profiles; and 3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's built-in catalog. 4. Promptkit's maintained catalog.
A source falls through only when the selected ID is absent. Each source supplies a complete definition, so profile fields are not merged. A malformed matching operator definition is an error and does not fall back. A source falls through only when the selected ID is absent. Promptkit resolves a
derived profile's base with the same source precedence, so a configured base
can shadow a built-in base. A missing, cyclic, malformed, or incomplete
selected inheritance chain is an error and does not fall back.
Profiles that require a direct API key are unsupported; a profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. Active results retain the selected logical profile ID and resolved backend and model. Ordinary errors, summaries, logs, and outputs exclude endpoints, credentials, rendered messages, schemas, request bodies, response bodies, and complete parameter maps. Profiles that require a direct API key are unsupported. Optional environment
credential sources are Promptkit runtime concerns and are not checked by
Weatherreporter during profile inspection. Active results retain the selected
logical profile ID and resolved backend and model. Ordinary errors, summaries,
logs, and outputs exclude endpoints, credentials, rendered messages, schemas,
request bodies, response bodies, and complete parameter maps.
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions. Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions. Ordinary generation errors disclose only the safe Weatherreporter category and optional HTTP status; provider code, type, and message are written only to the explicit secure failure-debug artifact.
Each embedded prompt permits one Promptkit-owned corrective generation after an
eligible failed or explicitly empty result. This is not an application retry:
Weatherreporter performs no provider retry, profile fallback, or request-level
output-contract override. Promptkit reports cumulative usage and the actual
number of corrective calls; repair exhaustion remains a completed validation
failure.
When capture is enabled, its preparation artifact projects a provider endpoint When capture is enabled, its preparation artifact projects a provider endpoint
to its scheme and host and retains only reviewed execution settings. Provider to its scheme and host and retains only reviewed execution settings. Provider

View File

@@ -7,7 +7,7 @@ is owned by the [CLI reference](../cli.md) and [operations guide](../operations.
## Single-Report Flow ## Single-Report Flow
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. The resolved profile, backend, and model are carried in the active result. `GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. Profile inspection requires a model, permits an empty backend identity for endpoint-only profiles, and leaves inherited resolution and optional credential sources to Promptkit. The resolved profile, backend, model, and actual repair count (once a completed execution exists) are carried in the active result; the configured repair budget remains part of the exact prompt contract.
The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run. The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run.

View File

@@ -21,8 +21,9 @@ Before accepting generated JSON, the execution boundary reconciles the prepared
report definition, inspected prompt hash and selected profile identity, the one report definition, inspected prompt hash and selected profile identity, the one
preparation callback, and the completed Promptkit result. The callback and preparation callback, and the completed Promptkit result. The callback and
completion must agree on prompt, profile, backend, model, and rendered/input completion must agree on prompt, profile, backend, model, and rendered/input
hashes; the callback output and completed validation must name the prepared hashes; the callback output must also carry the prepared report's configured
report's JSON Schema. A mismatch produces no rendered Markdown and leaves repair budget, while the completed validation records the actual corrective
calls used within that budget. A mismatch produces no rendered Markdown and leaves
results with only the inspected safe identity. results with only the inspected safe identity.
Single-report generation executes one prepared profile and publishes its Single-report generation executes one prepared profile and publishes its

View File

@@ -2,9 +2,9 @@
`internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter. `internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter.
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, the embedded fallback catalog, and its built-in catalog; the adapter does not parse profile YAML, merge sources, or probe endpoints. The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, embedded profile aliases and their bases, and its maintained catalog. Promptkit privately assembles that catalog from the independently versioned OpenRouter and Rakestrawhome catalog modules selected by its release; Weatherreporter neither imports nor registers them. The adapter does not parse profile YAML, resolve inheritance, merge sources, inspect optional environment credentials, or probe endpoints.
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file. The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. An empty backend identity remains valid for an endpoint-only profile; a nonblank model is required. PromptKit's configured repair-call budget and the completed result's actual corrective-call count are retained, along with its cumulative provider usage and final candidate. Structured provider generation failures become project-owned redacted generation errors that retain only bounded details through explicit accessors. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications. The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications.

View File

@@ -122,7 +122,9 @@ The destination is preflighted before prompt inspection and collection, then
rechecked immediately before an atomic publish. A missing or empty directory rechecked immediately before an atomic publish. A missing or empty directory
is usable. A nonempty directory can be replaced only when `--replace` is given is usable. A nonempty directory can be replaced only when `--replace` is given
and it is recognized as a current Weatherreporter comparison bundle; ordinary and it is recognized as a current Weatherreporter comparison bundle; ordinary
directories, symlinks, and unsafe destinations are rejected. Cancellation and directories, symlinks, and unsafe destinations are rejected. Existing v1
bundles are not recognized for replacement: move or remove them first.
Cancellation and
all failures before publication preserve an existing bundle, including a all failures before publication preserve an existing bundle, including a
cancellation observed while a replacement is being prepared. If guarded cancellation observed while a replacement is being prepared. If guarded
restoration cannot complete, the error names the retained sibling bundle for restoration cannot complete, the error names the retained sibling bundle for
@@ -176,6 +178,12 @@ Preparation captures retain only the provider endpoint origin and reviewed
execution settings. URL user information, paths, queries, fragments, and execution settings. URL user information, paths, queries, fragments, and
unrecognized provider parameters are omitted. unrecognized provider parameters are omitted.
Each run directory may contain `preparation.json` (v3), `execution.json` (v3),
and, for a provider generation failure, `failure.json` (v1). The failure
artifact retains the safe category, HTTP status, and provider code, type, and
message for trusted debugging only. Ordinary command output never includes
those provider details.
Capture writes are confined to the requested root and fail if an unsafe Capture writes are confined to the requested root and fail if an unsafe
filesystem component prevents secure artifact creation. filesystem component prevents secure artifact creation.

View File

@@ -50,10 +50,14 @@ directly.
- Prompts receive curated module packages, never unbounded raw weather payloads. - Prompts receive curated module packages, never unbounded raw weather payloads.
- Every execution validates the exact prompt version and output contract before - Every execution validates the exact prompt version and output contract before
collection. The selected profile is configured explicitly or declared by the collection. The selected profile is configured explicitly or declared by the
prompt; unsupported direct-key profiles and missing reported credentials fail prompt; profiles requiring unsupported direct API keys fail before collection.
before collection. A profile may have an empty backend identity when it supplies an endpoint;
PromptKit resolves inherited profiles and optional credential sources when it
executes them.
- Prompt and profile validation completes before weather collection. Raw output - Prompt and profile validation completes before weather collection. Raw output
is validated before template rendering. is validated before template rendering.
- PromptKit may make at most the prompt contract's one corrective generation;
exhaustion is a validation rejection, not an application-level retry.
- Comparison validates every explicit profile before collection, prepares one - Comparison validates every explicit profile before collection, prepares one
immutable report input, and delegates backend capacity to Promptkit rather immutable report input, and delegates backend capacity to Promptkit rather
than adding an application-wide execution limit. than adding an application-wide execution limit.
@@ -62,6 +66,8 @@ directly.
- Sensitive rendered prompts, schemas, input bodies, provider endpoints, and - Sensitive rendered prompts, schemas, input bodies, provider endpoints, and
credentials never enter normal summaries or logs. They are written only to credentials never enter normal summaries or logs. They are written only to
an explicit secure debug root when requested. an explicit secure debug root when requested.
- Provider-controlled diagnostics never enter ordinary outputs; they are
retained only in explicit secure failure-debug artifacts.
## Output, Notification, And Testing Invariants ## Output, Notification, And Testing Invariants

64
docs/releases/v0.13.0.md Normal file
View File

@@ -0,0 +1,64 @@
# Weatherreporter v0.13.0
This release upgrades Weatherreporter's generated-text integration through
PromptKit v0.9.0 and adds bounded structured-output repair, inherited domain
profiles, safer provider diagnostics, and repair provenance.
## Summary
Weatherreporter now permits one PromptKit-owned corrective generation when a
model returns invalid or explicitly empty structured output. Successful and
failed results report the actual correction count when validation completed,
and explicit prompt-debug capture can retain bounded provider failure details
without exposing them in ordinary output.
The stable `weather-light`, `weather-balanced`, and `weather-deep` profiles now
inherit their execution targets from PromptKit's maintained catalog. PromptKit
v0.9.0 assembles that catalog from independently versioned OpenRouter and
Rakestrawhome modules while preserving Weatherreporter's existing effective
profile targets and configured-profile override behavior.
## Compatibility
The documented CLI, configuration fields, report IDs, report filenames,
Markdown outputs, and Distributor integration remain compatible with v0.12.0.
The four generated-text prompts advance to version `2.1.0`; an eligible failed
validation may therefore make one additional provider call, increasing latency
and usage for that execution.
Comparison bundles advance from schema v1 to v2 so each completed validation
can record `repairAttempts`. Weatherreporter intentionally recognizes only the
current bundle version for guarded replacement. An existing v1 comparison
bundle cannot be replaced with `--replace` by this release.
## Upgrade
No special action is required for ordinary generation, batch execution,
configuration, or notification. Operators who want to reuse the destination
of a v1 comparison bundle must first remove or relocate that old bundle, or
select a new `--out-dir`.
Operators should account for the possibility of one additional model call
when output repair is needed. Existing configured standalone, inherited, and
endpoint-only profiles retain their documented precedence.
## Changes
- Upgraded PromptKit from v0.5.0 through v0.9.0 and adopted its maintained
profile inheritance, optional credential-source behavior, structured repair,
generation-error contract, and externally maintained provider catalogs.
- Converted the three Weatherreporter domain profiles into minimal aliases of
maintained PromptKit profiles without changing their effective model ladder.
- Added one corrective-generation allowance to all four embedded prompts and
exposed actual repair counts in generate, batch, and comparison summaries.
- Migrated the [comparison bundle contract](../integrations/comparison-bundle.md)
to v2 with strict repair-provenance validation and safe partial publication
after independent profile failures.
- Added safe provider HTTP status to ordinary errors while confining provider
code, type, and message to explicit secure `failure.json` debug artifacts.
- Strengthened offline coverage for local endpoint-only profiles, optional
credentials, concurrent comparison repair outcomes, and failure-debug
isolation.
- Updated the [PromptKit integration guide](../integrations/promptkit.md),
[configuration reference](../config.md), [CLI reference](../cli.md), and
[operations guide](../operations.md) for the implemented contracts.

8
go.mod
View File

@@ -6,9 +6,13 @@ require gopkg.in/yaml.v3 v3.0.1
require ( require (
gitea.maximumdirect.net/eric/distributor v0.5.0 gitea.maximumdirect.net/eric/distributor v0.5.0
gitea.maximumdirect.net/eric/promptkit v0.5.0 gitea.maximumdirect.net/eric/promptkit v0.9.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2 github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
golang.org/x/sys v0.45.0 golang.org/x/sys v0.45.0
) )
require golang.org/x/text v0.14.0 // indirect require (
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 // indirect
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 // indirect
golang.org/x/text v0.14.0 // indirect
)

8
go.sum
View File

@@ -1,7 +1,11 @@
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk= gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8= gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc= gitea.maximumdirect.net/eric/promptkit v0.9.0 h1:IpvDRC8L6xRxQ9hpuyKOmMc5b6MeLTKYyx+h1YAjy08=
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ= gitea.maximumdirect.net/eric/promptkit v0.9.0/go.mod h1:oMJ/WUJImUtwJ5e+6MAGECPYAErAkOaKel0G+3T/b4E=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 h1:lc062euk2qseO//D762i3JaFyulDNML3eQQX7DkYTho=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0/go.mod h1:AIa7kAu2mfrRQgcspe4L+DW51WqgnALQT60lqkEywJI=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 h1:j9YY7wsTVjzke2kHH4YAzpU0oUpM+x+nXwl1IeS+2eg=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0/go.mod h1:4RNS+LILDg4JbS4Ts9Lwy1C92wauXJIbeQaalps4Koo=
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4= github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo= github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I= github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=

View File

@@ -153,6 +153,7 @@ func outputContract(value promptkit.OutputContract) promptexec.OutputContract {
Format: string(value.Format), Format: string(value.Format),
ValidationMode: string(value.ValidationMode), ValidationMode: string(value.ValidationMode),
SchemaPath: value.SchemaPath, SchemaPath: value.SchemaPath,
RepairAttempts: value.RepairAttempts,
} }
} }
@@ -193,6 +194,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
promptexec.ValidationStatus(value.Validation.Status), promptexec.ValidationStatus(value.Validation.Status),
string(value.Validation.Mode), string(value.Validation.Mode),
value.Validation.SchemaPath, value.Validation.SchemaPath,
value.Validation.RepairAttempts,
value.Validation.Errors, value.Validation.Errors,
) )
rawOutput := []byte(nil) rawOutput := []byte(nil)
@@ -203,6 +205,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
promptexec.ValidationFailed, promptexec.ValidationFailed,
string(value.Validation.Mode), string(value.Validation.Mode),
value.Validation.SchemaPath, value.Validation.SchemaPath,
value.Validation.RepairAttempts,
[]string{"generated output exceeds the configured size limit"}, []string{"generated output exceeds the configured size limit"},
) )
} }
@@ -299,6 +302,16 @@ func classifyError(err error) error {
if errors.As(err, &capacityError) { if errors.As(err, &capacityError) {
return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err) return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err)
} }
var generationError *promptkit.GenerationError
if errors.As(err, &generationError) {
return promptexec.NewGenerationError(
generationError.StatusCode(),
generationError.ProviderCode(),
generationError.ProviderType(),
generationError.ProviderMessage(),
err,
)
}
switch { switch {
case errors.Is(err, promptkit.ErrInvalidConfig): case errors.Is(err, promptkit.ErrInvalidConfig):
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err) return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err)

View File

@@ -4,11 +4,15 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"net/http"
"net/http/httptest"
"os" "os"
"path/filepath" "path/filepath"
"reflect"
"strings" "strings"
"sync" "sync"
"testing" "testing"
"testing/fstest"
"time" "time"
promptkit "gitea.maximumdirect.net/eric/promptkit" promptkit "gitea.maximumdirect.net/eric/promptkit"
@@ -20,12 +24,19 @@ type fakeClient struct {
mu sync.Mutex mu sync.Mutex
response *promptkit.GenerateResponse response *promptkit.GenerateResponse
err error err error
outcomes []generationOutcome
next int
calls int calls int
requests []promptkit.GenerateRequest requests []promptkit.GenerateRequest
block bool block bool
started chan struct{} started chan struct{}
} }
type generationOutcome struct {
response *promptkit.GenerateResponse
err error
}
type recordingReader struct { type recordingReader struct {
ref promptkit.ArtifactRef ref promptkit.ArtifactRef
} }
@@ -49,6 +60,11 @@ func (client *fakeClient) Generate(ctx context.Context, request promptkit.Genera
started := client.started started := client.started
response := client.response response := client.response
err := client.err err := client.err
if client.next < len(client.outcomes) {
outcome := client.outcomes[client.next]
client.next++
response, err = outcome.response, outcome.err
}
client.mu.Unlock() client.mu.Unlock()
if started != nil { if started != nil {
started <- struct{}{} started <- struct{}{}
@@ -102,13 +118,19 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
return client.requests[0] return client.requests[0]
} }
func (client *fakeClient) allRequests() []promptkit.GenerateRequest {
client.mu.Lock()
defer client.mu.Unlock()
return append([]promptkit.GenerateRequest(nil), client.requests...)
}
func TestInspectPromptAndProfile(t *testing.T) { func TestInspectPromptAndProfile(t *testing.T) {
adapter := newTestAdapter(t, &fakeClient{}) adapter := newTestAdapter(t, &fakeClient{})
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.0.0") inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.1.0")
if err != nil { if err != nil {
t.Fatalf("InspectPrompt() error = %v", err) t.Fatalf("InspectPrompt() error = %v", err)
} }
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != "weather-balanced" { if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != "weather-balanced" || inspection.Output.RepairAttempts != 1 {
t.Fatalf("inspection = %#v", inspection) t.Fatalf("inspection = %#v", inspection)
} }
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" { if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
@@ -175,15 +197,38 @@ model: file-light
} }
assertProfile(t, fileAdapter, "weather-light", "", "file-light") assertProfile(t, fileAdapter, "weather-light", "", "file-light")
directory := testProfileDirectory(t, `id: weather-light directory := testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
backend: local backend: local
model: directory-light model: directory-light
`) `})
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"}) directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil { if err != nil {
t.Fatalf("New(profile directory) error = %v", err) t.Fatalf("New(profile directory) error = %v", err)
} }
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light") assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
derived := writeProfileFile(t, `id: weather-light
base_profile: gemini-flash-latest
`)
derivedAdapter, err := New(Config{ProfileFile: derived})
if err != nil {
t.Fatalf("New(derived profile) error = %v", err)
}
assertProfile(t, derivedAdapter, "weather-light", "openrouter", "~google/gemini-flash-latest")
}
func TestConfiguredBaseProfileOverridesEmbeddedProfileTarget(t *testing.T) {
directory := testProfileDirectory(t, map[string]string{
"deepseek.yml": `id: deepseek-4-flash
backend: local
model: shadowed-deepseek
`,
})
adapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil {
t.Fatalf("New() error = %v", err)
}
assertProfile(t, adapter, "weather-light", promptkit.BackendLocal, "shadowed-deepseek")
} }
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) { func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
@@ -194,19 +239,69 @@ func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T)
assertProfile(t, adapter, "weather-light", "", "weather-local") assertProfile(t, adapter, "weather-light", "", "weather-local")
} }
func TestMaintainedWeatherLightLocalProfileExampleExecutesThroughProductionClient(t *testing.T) {
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
for _, test := range []struct {
name string
credentialSource string
}{
{name: "without credential source"},
{name: "with blank optional credential source", credentialSource: "\napi_key_env: WEATHERREPORTER_TEST_MISSING_KEY\n"},
} {
t.Run(test.name, func(t *testing.T) {
var authorization string
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
authorization = request.Header.Get("Authorization")
writer.Header().Set("Content-Type", "application/json")
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":12,"completion_tokens":8,"total_tokens":20}}`, validResponse().Content)
}))
defer server.Close()
example, err := os.ReadFile(filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml"))
if err != nil {
t.Fatal(err)
}
profile := strings.Replace(string(example), "http://127.0.0.1:11434/v1", server.URL+"/v1", 1) + test.credentialSource
adapter, err := New(Config{ProfileFile: writeProfileFile(t, profile)})
if err != nil {
t.Fatalf("New() error = %v", err)
}
request := testExecuteRequest()
request.ProfileID = "weather-light"
var preparation promptexec.Preparation
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
preparation = value
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if authorization != "" {
t.Fatalf("Authorization header = %q, want absent", authorization)
}
if preparation.ProfileID != "weather-light" || preparation.BackendID != "" || preparation.ModelName != "weather-local" || preparation.Output.RepairAttempts != 1 {
t.Fatalf("preparation = %#v", preparation)
}
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" || result.Validation.Status != promptexec.ValidationPassed || result.Validation.RepairAttempts != 0 {
t.Fatalf("execution = %#v", result)
}
})
}
}
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) { func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: other-profile
backend: openrouter backend: openrouter
model: other-model model: other-model
`)}) `})})
if err != nil { if err != nil {
t.Fatalf("New(absent profile) error = %v", err) t.Fatalf("New(absent profile) error = %v", err)
} }
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash") assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
backend: openrouter backend: openrouter
`)}) `})})
if err != nil { if err != nil {
t.Fatalf("New(malformed profile) error = %v", err) t.Fatalf("New(malformed profile) error = %v", err)
} }
@@ -215,6 +310,78 @@ backend: openrouter
} }
} }
func TestProfileResolutionReturnsConfiguredInheritanceFailures(t *testing.T) {
tests := []struct {
name string
profile string
profiles map[string]string
}{
{
name: "missing base",
profile: "missing-base",
profiles: map[string]string{"missing.yml": `id: missing-base
base_profile: unavailable
`},
},
{
name: "cyclic bases",
profile: "first",
profiles: map[string]string{
"first.yml": `id: first
base_profile: second
`,
"second.yml": `id: second
base_profile: first
`,
},
},
{
name: "malformed base",
profile: "child",
profiles: map[string]string{
"child.yml": `id: child
base_profile: malformed
`,
"malformed.yml": `id: malformed
base_profile: [not-a-profile]
`,
},
},
{
name: "incomplete target",
profile: "incomplete",
profiles: map[string]string{"incomplete.yml": `id: incomplete
backend: openrouter
`},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
adapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, test.profiles)})
if err != nil {
t.Fatalf("New() error = %v", err)
}
if _, err := adapter.InspectProfile(context.Background(), test.profile); err == nil {
t.Fatal("InspectProfile() error = nil, want configured inheritance error")
}
})
}
}
func TestRakestrawhomeBuiltInProfileInspectsOffline(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
profile, err := adapter.InspectProfile(context.Background(), "rakestrawhome-gemma-4-31b")
if err != nil {
t.Fatalf("InspectProfile() error = %v", err)
}
if profile.ProfileID != "rakestrawhome-gemma-4-31b" || profile.BackendID != "rakestrawhome" || profile.ModelName == "" {
t.Fatalf("profile = %#v", profile)
}
}
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) { func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
adapter, err := New(Config{}) adapter, err := New(Config{})
if err != nil { if err != nil {
@@ -287,7 +454,7 @@ func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
} }
request := promptexec.ExecuteRequest{ request := promptexec.ExecuteRequest{
PromptID: "weather.hourly_generated_text", PromptID: "weather.hourly_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
ProfileID: "weather-light", ProfileID: "weather-light",
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"), DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
} }
@@ -400,6 +567,114 @@ func TestExecuteReturnsCompletedValidationRejection(t *testing.T) {
} }
} }
func TestExecuteMapsCorrectiveGenerationResults(t *testing.T) {
valid := `{"summary":"valid"}`
invalid := `{"summary":42}`
tests := []struct {
name string
outcomes []generationOutcome
wantStatus promptexec.ValidationStatus
wantRepairs int
wantCalls int
wantRaw string
wantUsage promptexec.TokenUsage
}{
{
name: "first pass valid",
outcomes: []generationOutcome{{response: generationResponse(valid, 2, 3, 5)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 0,
wantCalls: 1,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5},
},
{
name: "empty output repaired",
outcomes: []generationOutcome{{response: generationResponse("", 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
{
name: "invalid output repaired",
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
{
name: "repair budget exhausted",
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(invalid, 7, 11, 18)}},
wantStatus: promptexec.ValidationFailed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: invalid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
client := &fakeClient{outcomes: test.outcomes}
adapter := newRepairAdapter(t, client, "https://repair.example/v1")
var preparation promptexec.Preparation
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
preparation = value
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if preparation.Output.RepairAttempts != 1 || result == nil || result.Validation.Status != test.wantStatus || result.Validation.RepairAttempts != test.wantRepairs || string(result.RawOutput) != test.wantRaw || result.Usage != test.wantUsage {
t.Fatalf("preparation/result = %#v/%#v", preparation, result)
}
requests := client.allRequests()
if len(requests) != test.wantCalls {
t.Fatalf("provider requests = %d, want %d", len(requests), test.wantCalls)
}
if test.wantCalls == 2 && !reflect.DeepEqual(requests[0].Target, requests[1].Target) {
t.Fatalf("corrective target = %#v, want same prepared identity as %#v", requests[1].Target, requests[0].Target)
}
if result.ProfileID != preparation.ProfileID || result.BackendID != preparation.BackendID || result.ModelName != preparation.ModelName || result.PromptID != preparation.PromptID || result.PromptVersion != preparation.PromptVersion || result.PromptHash != preparation.PromptHash {
t.Fatalf("prepared/result identity = %#v/%#v", preparation, result)
}
})
}
}
func TestExecuteMapsCorrectiveGenerationError(t *testing.T) {
const providerBody = `{"error":{"code":"repair-code","type":"repair-type","message":"repair-message"}}`
calls := 0
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) {
calls++
if calls == 1 {
writer.Header().Set("Content-Type", "application/json")
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":2,"completion_tokens":3,"total_tokens":5}}`, `{"summary":42}`)
return
}
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusUnprocessableEntity)
_, _ = writer.Write([]byte(providerBody))
}))
defer server.Close()
adapter := newRepairAdapter(t, nil, server.URL)
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), nil)
if result != nil || err == nil || calls != 2 {
t.Fatalf("result/error/calls = %#v/%v/%d", result, err, calls)
}
var generationError *promptexec.GenerationError
if !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusUnprocessableEntity || generationError.ProviderCode() != "repair-code" || generationError.ProviderType() != "repair-type" || generationError.ProviderMessage() != "repair-message" {
t.Fatalf("generation error = %#v", err)
}
if strings.Contains(err.Error(), "repair-message") || !errors.Is(err, promptkit.ErrLLMGenerate) {
t.Fatalf("generation error = %v", err)
}
}
func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) { func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) {
client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}} client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}}
adapter := newTestAdapter(t, client) adapter := newTestAdapter(t, client)
@@ -507,12 +782,12 @@ func TestNewValidatesConfiguration(t *testing.T) {
} }
} }
func TestLocalBackendAndMissingCredentialBehavior(t *testing.T) { func TestLocalBackendAndOptionalCredentialSourceBehavior(t *testing.T) {
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "") t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
profiles := testProfileDirectory(t, `id: local-profile profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: local-profile
backend: local backend: local
model: local-model model: local-model
`) `})
adapter, err := newAdapterForTest(Config{ adapter, err := newAdapterForTest(Config{
ProfileDirectory: profiles, ProfileDirectory: profiles,
LocalEndpoint: "https://local.example/v1", LocalEndpoint: "https://local.example/v1",
@@ -529,11 +804,11 @@ model: local-model
t.Fatalf("capacity classification = %v", got) t.Fatalf("capacity classification = %v", got)
} }
credentialProfiles := testProfileDirectory(t, `id: credential-profile credentialProfiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: credential-profile
endpoint: https://profile.example/v1 endpoint: https://profile.example/v1
model: test-model model: test-model
api_key_env: WEATHERREPORTER_TEST_MISSING_KEY api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
`) `})
client := &fakeClient{response: validResponse()} client := &fakeClient{response: validResponse()}
credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client) credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client)
if err != nil { if err != nil {
@@ -546,8 +821,8 @@ api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
request := testExecuteRequest() request := testExecuteRequest()
request.ProfileID = "credential-profile" request.ProfileID = "credential-profile"
result, err := credentialAdapter.Execute(context.Background(), request, nil) result, err := credentialAdapter.Execute(context.Background(), request, nil)
if result != nil || promptexec.CategoryOf(err) != promptexec.MissingCredential || client.callCount() != 0 { if err != nil || result == nil || client.callCount() != 1 {
t.Fatalf("credential result/category/calls = %#v/%q/%d", result, promptexec.CategoryOf(err), client.callCount()) t.Fatalf("credential result/error/calls = %#v/%v/%d", result, err, client.callCount())
} }
} }
@@ -568,14 +843,14 @@ func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, mo
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter { func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
t.Helper() t.Helper()
profiles := testProfileDirectory(t, `id: test-profile profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: test-profile
endpoint: https://profile.example/v1 endpoint: https://profile.example/v1
model: test-model model: test-model
temperature: 0.2 temperature: 0.2
max_tokens: 300 max_tokens: 300
top_p: 1 top_p: 1
timeout_seconds: 30 timeout_seconds: 30
`) `})
options = append(options, promptkit.WithLLMClient(client)) options = append(options, promptkit.WithLLMClient(client))
adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...) adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...)
if err != nil { if err != nil {
@@ -584,13 +859,15 @@ timeout_seconds: 30
return adapter return adapter
} }
func testProfileDirectory(t *testing.T, profile string) string { func testProfileDirectory(t *testing.T, profiles map[string]string) string {
t.Helper() t.Helper()
profiles := t.TempDir() directory := t.TempDir()
if err := os.WriteFile(filepath.Join(profiles, "profile.yml"), []byte(profile), 0o600); err != nil { for name, profile := range profiles {
t.Fatalf("write profile: %v", err) if err := os.WriteFile(filepath.Join(directory, name), []byte(profile), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
} }
return profiles return directory
} }
func writeProfileFile(t *testing.T, profile string) string { func writeProfileFile(t *testing.T, profile string) string {
@@ -605,7 +882,7 @@ func writeProfileFile(t *testing.T, profile string) string {
func testExecuteRequest() promptexec.ExecuteRequest { func testExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{ return promptexec.ExecuteRequest{
PromptID: "weather.daily_generated_text", PromptID: "weather.daily_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
ProfileID: "test-profile", ProfileID: "test-profile",
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"), DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
} }
@@ -618,6 +895,61 @@ func validResponse() *promptkit.GenerateResponse {
} }
} }
func generationResponse(content string, promptTokens int, completionTokens int, totalTokens int) *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{
Content: content,
Usage: promptkit.TokenUsage{
PromptTokens: promptTokens,
CompletionTokens: completionTokens,
TotalTokens: totalTokens,
},
}
}
func newRepairAdapter(t *testing.T, client promptkit.LLMClient, endpoint string) *Adapter {
t.Helper()
profiles := testProfileDirectory(t, map[string]string{"profile.yml": "id: repair-profile\nendpoint: " + endpoint + "\nmodel: repair-model\n"})
options := []promptkit.Option{
promptkit.WithPromptFS(fstest.MapFS{
"repair.yml": &fstest.MapFile{Data: []byte(`id: weather.repair
version: "1.0.0"
default_profile: repair-profile
inputs:
- name: data_package
required: true
content_type: application/yaml
messages:
- role: user
content: "{{input \"data_package\"}}"
output:
format: json
validation_mode: json_schema
schema_path: repair.schema.json
repair_attempts: 1
`)}}, "."),
promptkit.WithSchemaFS(fstest.MapFS{
"repair.schema.json": &fstest.MapFile{Data: []byte(`{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"],"additionalProperties":false}`)},
}, "."),
}
if client != nil {
options = append(options, promptkit.WithLLMClient(client))
}
adapter, err := newAdapter(Config{ProfileDirectory: profiles}, options...)
if err != nil {
t.Fatalf("newAdapter() error = %v", err)
}
return adapter
}
func repairExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{
PromptID: "weather.repair",
PromptVersion: "1.0.0",
ProfileID: "repair-profile",
DataPackage: []byte("report: repair\n"),
}
}
func hourlyValidResponse() *promptkit.GenerateResponse { func hourlyValidResponse() *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{ return &promptkit.GenerateResponse{
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`, Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,

View File

@@ -89,6 +89,7 @@ type ReportResult struct {
ModelName string ModelName string
SourceWarnings []weatherdata.SourceWarning SourceWarnings []weatherdata.SourceWarning
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string LLMDebugPath string
OutputPath string OutputPath string
Notification *NotificationResult Notification *NotificationResult
@@ -139,6 +140,7 @@ type BatchReportResult struct {
ModelName string `json:"modelName,omitempty"` ModelName string `json:"modelName,omitempty"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"` SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"` ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
LLMDebugPath string `json:"llmDebugPath,omitempty"` LLMDebugPath string `json:"llmDebugPath,omitempty"`
OutputPath string `json:"outputPath,omitempty"` OutputPath string `json:"outputPath,omitempty"`
} }
@@ -457,6 +459,9 @@ func copyBatchReportDetails(item *BatchReportResult, result *ReportResult) {
item.Timezone = result.Timezone item.Timezone = result.Timezone
item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...) item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
item.ValidationStatus = result.ValidationStatus item.ValidationStatus = result.ValidationStatus
if result.RepairAttempts != nil {
item.RepairAttempts = repairAttemptsPointer(*result.RepairAttempts)
}
} }
func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) { func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) {

View File

@@ -3,7 +3,6 @@ package app
import ( import (
"context" "context"
"fmt" "fmt"
"os"
"path/filepath" "path/filepath"
"time" "time"
@@ -60,6 +59,7 @@ type ComparisonProfileResult struct {
ModelName string ModelName string
Status string Status string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
ReportPath string ReportPath string
LLMDebugPath string LLMDebugPath string
Error *comparison.SafeError Error *comparison.SafeError
@@ -121,7 +121,7 @@ func compareDetailed(ctx context.Context, req ComparisonRequest, publish compari
} }
defer func() { _ = debugWriter.Close() }() defer func() { _ = debugWriter.Close() }()
inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{ inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor, LookupEnv: os.LookupEnv, Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor,
}) })
result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash
if err != nil { if err != nil {
@@ -198,6 +198,9 @@ func copyComparisonOutcomes(result *ComparisonResult, outcomes []comparisonProfi
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName, Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error, Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error,
} }
if outcome.RepairAttempts != nil {
profile.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
if published && outcome.Status == comparison.StatusSucceeded { if published && outcome.Status == comparison.StatusSucceeded {
profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath) profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath)
} }
@@ -227,6 +230,9 @@ func comparisonBundle(result *ComparisonResult, dataPackage []byte, outcomes []c
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName, Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error, Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error,
} }
if outcome.RepairAttempts != nil {
manifestResult.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
if outcome.Status == comparison.StatusSucceeded { if outcome.Status == comparison.StatusSucceeded {
manifestResult.ReportPath = outcome.ReportPath manifestResult.ReportPath = outcome.ReportPath
bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown}) bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown})

View File

@@ -31,6 +31,7 @@ type comparisonProfileOutcome struct {
ModelName string ModelName string
Status string Status string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
ReportPath string ReportPath string
Markdown []byte Markdown []byte
LLMDebugPath string LLMDebugPath string
@@ -108,6 +109,9 @@ func executeComparisonProfile(ctx context.Context, req comparisonExecutionReques
}) })
outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName
outcome.ValidationStatus = execution.ValidationStatus outcome.ValidationStatus = execution.ValidationStatus
if execution.RepairAttempts != nil {
outcome.RepairAttempts = repairAttemptsPointer(*execution.RepairAttempts)
}
outcome.LLMDebugPath = execution.LLMDebugPath outcome.LLMDebugPath = execution.LLMDebugPath
if err != nil { if err != nil {
outcome.canceled = cancellationError(err) outcome.canceled = cancellationError(err)
@@ -169,9 +173,14 @@ func comparisonExecutionMessage(err error) string {
if errors.Is(err, context.DeadlineExceeded) { if errors.Is(err, context.DeadlineExceeded) {
return "profile execution deadline exceeded" return "profile execution deadline exceeded"
} }
operation := "profile execution"
var execution *profileExecutionError var execution *profileExecutionError
if errors.As(err, &execution) { if errors.As(err, &execution) {
return comparison.TruncateErrorMessage(execution.operation + " failed") operation = execution.operation
} }
return "profile execution failed" var generation *promptexec.GenerationError
if errors.As(err, &generation) && generation.StatusCode() > 0 {
return comparison.TruncateErrorMessage(fmt.Sprintf("%s failed (HTTP %d)", operation, generation.StatusCode()))
}
return comparison.TruncateErrorMessage(operation + " failed")
} }

View File

@@ -4,9 +4,11 @@ import (
"context" "context"
"errors" "errors"
"fmt" "fmt"
"net/http"
"os" "os"
"path/filepath" "path/filepath"
"reflect" "reflect"
"strings"
"sync" "sync"
"testing" "testing"
"time" "time"
@@ -72,6 +74,66 @@ func TestExecuteComparisonProfilesContinuesAfterProfileFailure(t *testing.T) {
} }
} }
func TestExecuteComparisonProfilesPreservesIndependentRepairOutcomes(t *testing.T) {
prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(4)
executor := newBarrierExecutor(profiles)
executor.setValidation(profiles[0].ProfileID, promptexec.ValidationPassed, 0)
executor.setValidation(profiles[1].ProfileID, promptexec.ValidationPassed, 1)
executor.setValidation(profiles[2].ProfileID, promptexec.ValidationFailed, 1)
executor.setError(profiles[3].ProfileID, errors.New("provider failure"))
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", Executor: executor,
}, executor)
waitForProfileStarts(t, executor, profiles, results)
executor.releaseAll()
result := <-results
wantStatuses := []string{comparison.StatusSucceeded, comparison.StatusSucceeded, comparison.StatusFailed, comparison.StatusFailed}
wantValidations := []promptexec.ValidationStatus{promptexec.ValidationPassed, promptexec.ValidationPassed, promptexec.ValidationFailed, ""}
wantRepairs := []*int{intPointer(0), intPointer(1), intPointer(1), nil}
for index, outcome := range result.Outcomes {
if outcome.Status != wantStatuses[index] || outcome.ValidationStatus != wantValidations[index] || !reflect.DeepEqual(outcome.RepairAttempts, wantRepairs[index]) {
t.Fatalf("outcome[%d] = %#v, want status/validation/repairs %q/%q/%#v", index, outcome, wantStatuses[index], wantValidations[index], wantRepairs[index])
}
}
}
func TestExecuteComparisonProfilesCapturesConcurrentProviderFailures(t *testing.T) {
prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(2)
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
if err != nil {
t.Fatalf("NewPromptDebugWriter() error = %v", err)
}
markers := []string{"first-provider-private-marker", "second-provider-private-marker"}
statuses := []int{http.StatusTooManyRequests, http.StatusServiceUnavailable}
executor := newBarrierExecutor(profiles)
for index, profile := range profiles {
executor.setError(profile.ProfileID, promptexec.NewGenerationError(statuses[index], "provider_code", "provider_type", markers[index], nil))
}
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", DebugWriter: debugWriter, Executor: executor,
}, executor)
waitForProfileStarts(t, executor, profiles, results)
executor.releaseAll()
result := <-results
for index, outcome := range result.Outcomes {
if outcome.Status != comparison.StatusFailed || outcome.Error == nil || outcome.Error.Category != string(promptexec.Generation) || outcome.Error.Message != fmt.Sprintf("execute prompt failed (HTTP %d)", statuses[index]) || strings.Contains(outcome.Error.Message, markers[index]) || outcome.LLMDebugPath == "" {
t.Fatalf("outcome[%d] = %#v", index, outcome)
}
failure, readErr := os.ReadFile(filepath.Join(outcome.LLMDebugPath, "failure.json"))
if readErr != nil {
t.Fatal(readErr)
}
if !strings.Contains(string(failure), markers[index]) || strings.Contains(string(failure), markers[1-index]) {
t.Fatalf("failure[%d] = %s", index, failure)
}
}
}
func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) { func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) {
prepared, prompt := preparedDailyProfile(t) prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(4) profiles := comparisonProfiles(4)
@@ -139,6 +201,8 @@ type barrierExecutor struct {
releases map[string]chan struct{} releases map[string]chan struct{}
requests map[string]promptexec.ExecuteRequest requests map[string]promptexec.ExecuteRequest
errors map[string]error errors map[string]error
validations map[string]promptexec.ValidationStatus
repairAttempts map[string]int
profiles map[string]ComparisonProfileInspection profiles map[string]ComparisonProfileInspection
inFlight int inFlight int
maximum int maximum int
@@ -153,7 +217,7 @@ func newBarrierExecutor(profiles []ComparisonProfileInspection) *barrierExecutor
} }
return &barrierExecutor{ return &barrierExecutor{
started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases, started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases,
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, profiles: identities, requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, validations: map[string]promptexec.ValidationStatus{}, repairAttempts: map[string]int{}, profiles: identities,
} }
} }
@@ -170,7 +234,8 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
e.mu.Lock() e.mu.Lock()
profile := e.profiles[req.ProfileID] profile := e.profiles[req.ProfileID]
e.mu.Unlock() e.mu.Unlock()
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil { definition := generationDefinitionForPrompt(req.PromptID)
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
e.callbackFailures <- err e.callbackFailures <- err
return nil, err return nil, err
} }
@@ -194,15 +259,20 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
e.mu.Lock() e.mu.Lock()
e.inFlight-- e.inFlight--
err := e.errors[req.ProfileID] err := e.errors[req.ProfileID]
validationStatus := e.validations[req.ProfileID]
repairAttempts := e.repairAttempts[req.ProfileID]
e.mu.Unlock() e.mu.Unlock()
if err != nil { if err != nil {
return nil, err return nil, err
} }
if validationStatus == "" {
validationStatus = promptexec.ValidationPassed
}
return &promptexec.Execution{ return &promptexec.Execution{
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash,
ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName,
StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(), StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(),
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil), Validation: promptexec.NewValidation(validationStatus, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil),
}, nil }, nil
} }
@@ -219,6 +289,13 @@ func (e *barrierExecutor) setError(profileID string, err error) {
e.errors[profileID] = err e.errors[profileID] = err
} }
func (e *barrierExecutor) setValidation(profileID string, status promptexec.ValidationStatus, repairAttempts int) {
e.mu.Lock()
defer e.mu.Unlock()
e.validations[profileID] = status
e.repairAttempts[profileID] = repairAttempts
}
func (e *barrierExecutor) release(profileID string) { func (e *barrierExecutor) release(profileID string) {
close(e.releases[profileID]) close(e.releases[profileID])
} }
@@ -317,4 +394,8 @@ func bytesEqual(left, right []byte) bool {
return reflect.DeepEqual(left, right) return reflect.DeepEqual(left, right)
} }
func intPointer(value int) *int {
return &value
}
var _ promptexec.Executor = (*barrierExecutor)(nil) var _ promptexec.Executor = (*barrierExecutor)(nil)

View File

@@ -87,6 +87,40 @@ func TestCompareDetailedPublishesPartialBundleAndReturnsAggregateError(t *testin
} }
} }
func TestCompareDetailedPublishesPostValidationProfileFailure(t *testing.T) {
bundle := generationBundle(t)
executor := &generationExecutor{complete: func(execution *promptexec.Execution) {
if execution.ProfileID == "weather-deep" {
execution.RawOutput = []byte(`{"summary":42}`)
}
}}
result, err := CompareDetailed(context.Background(), ComparisonRequest{
Config: comparisonConfig(), Report: ReportDaily, ProfileIDs: []string{"weather-light", "weather-deep"},
WorkingDir: t.TempDir(), Date: generationTime("2026-05-29T12:00:00-05:00"),
Clock: timeutil.FixedClock{Time: generationTime("2026-05-29T08:30:00-05:00")},
Collector: &generationCollector{bundle: &bundle}, Executor: executor,
})
if err == nil || result == nil || result.Succeeded != 1 || result.Failed != 1 || result.ManifestPath == "" {
t.Fatalf("CompareDetailed() result/error = %#v/%v", result, err)
}
failure := result.Results[1]
if failure.Status != comparison.StatusFailed || failure.ValidationStatus != promptexec.ValidationPassed || failure.RepairAttempts == nil || *failure.RepairAttempts != 0 {
t.Fatalf("post-validation failure = %#v", failure)
}
data, readErr := os.ReadFile(result.ManifestPath)
if readErr != nil {
t.Fatal(readErr)
}
var manifest comparison.Manifest
if decodeErr := json.Unmarshal(data, &manifest); decodeErr != nil {
t.Fatal(decodeErr)
}
manifestFailure := manifest.Results[1]
if manifestFailure.ValidationStatus != "passed" || manifestFailure.RepairAttempts == nil || *manifestFailure.RepairAttempts != 0 {
t.Fatalf("published post-validation failure = %#v", manifestFailure)
}
}
func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) { func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) {
for _, test := range []struct { for _, test := range []struct {
name string name string

View File

@@ -4,6 +4,7 @@ import (
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
"net/http"
"os" "os"
"path/filepath" "path/filepath"
"strings" "strings"
@@ -68,6 +69,7 @@ type generationExecutor struct {
beforeExecute func(promptexec.ExecuteRequest) beforeExecute func(promptexec.ExecuteRequest)
cancelBeforeReturn context.CancelFunc cancelBeforeReturn context.CancelFunc
validation promptexec.ValidationStatus validation promptexec.ValidationStatus
repairAttempts int
validations map[string]promptexec.ValidationStatus validations map[string]promptexec.ValidationStatus
rawOutput []byte rawOutput []byte
waitForCancellation map[string]bool waitForCancellation map[string]bool
@@ -88,7 +90,7 @@ func (e *generationExecutor) InspectPrompt(_ context.Context, id, version string
return promptexec.PromptInspection{}, e.inspectErr return promptexec.PromptInspection{}, e.inspectErr
} }
definition := generationDefinitionForPrompt(id) definition := generationDefinitionForPrompt(id)
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}}, nil return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}}, nil
} }
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) { func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
generationExecutorMu.Lock() generationExecutorMu.Lock()
@@ -112,7 +114,8 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
calls = 1 calls = 1
} }
for range calls { for range calls {
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp} definition := generationDefinitionForPrompt(req.PromptID)
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}
if prepare != nil { if prepare != nil {
prepare(&preparation) prepare(&preparation)
} }
@@ -128,6 +131,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
profileErr := e.executeErrors[req.ProfileID] profileErr := e.executeErrors[req.ProfileID]
executeErr := e.executeErr executeErr := e.executeErr
status := e.validation status := e.validation
repairAttempts := e.repairAttempts
if profileStatus, ok := e.validations[req.ProfileID]; ok { if profileStatus, ok := e.validations[req.ProfileID]; ok {
status = profileStatus status = profileStatus
} }
@@ -162,7 +166,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
if cancelBeforeReturn != nil { if cancelBeforeReturn != nil {
cancelBeforeReturn() cancelBeforeReturn()
} }
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil)} execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil)}
if complete != nil { if complete != nil {
complete(execution) complete(execution)
} }
@@ -546,6 +550,66 @@ func TestGenerateDetailedWritesRequestedPromptDebugArtifacts(t *testing.T) {
} }
} }
func TestGenerateDetailedCapturesProviderFailureOnlyInDebugArtifacts(t *testing.T) {
bundle := generationBundle(t)
debugRoot := t.TempDir()
const marker = "provider-private-generation-marker"
result, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: generationConfig(), Report: ReportDaily,
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle},
Executor: &generationExecutor{executeErr: promptexec.NewGenerationError(http.StatusTooManyRequests, "rate_limit", "provider_error", marker, nil)},
})
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
if err == nil || result == nil || result.LLMDebugPath == "" || promptexec.CategoryOf(err) != promptexec.Generation || !strings.Contains(err.Error(), "HTTP 429") || strings.Contains(err.Error(), marker) || result.OutputPath != "" {
t.Fatalf("GenerateDetailed() result/error = %#v/%v", result, err)
}
for _, name := range []string{"preparation.json", "failure.json"} {
if _, statErr := os.Stat(filepath.Join(result.LLMDebugPath, name)); statErr != nil {
t.Fatalf("debug artifact %q: %v", name, statErr)
}
}
data, readErr := os.ReadFile(filepath.Join(result.LLMDebugPath, "failure.json"))
if readErr != nil || !strings.Contains(string(data), marker) {
t.Fatalf("failure artifact = %q, error = %v", data, readErr)
}
}
func TestGenerateDetailedPreservesProviderFailureWhenFailureDebugWriteFails(t *testing.T) {
bundle := generationBundle(t)
debugRoot := t.TempDir()
const marker = "provider-private-write-failure-marker"
var setupErr error
executor := &generationExecutor{
executeErr: promptexec.NewGenerationError(http.StatusServiceUnavailable, "unavailable", "provider_error", marker, nil),
beforeExecute: func(promptexec.ExecuteRequest) {
setupErr = filepath.Walk(debugRoot, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.Name() == "preparation.json" {
return os.Mkdir(filepath.Join(filepath.Dir(path), "failure.json"), 0o700)
}
return nil
})
},
}
result, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: generationConfig(), Report: ReportDaily,
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle}, Executor: executor,
})
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
var generationError *promptexec.GenerationError
if setupErr != nil || err == nil || result == nil || result.OutputPath != "" || promptexec.CategoryOf(err) != promptexec.Generation || !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusServiceUnavailable || !strings.Contains(err.Error(), "HTTP 503") || strings.Contains(err.Error(), marker) {
t.Fatalf("GenerateDetailed() setup/result/error = %v/%#v/%v", setupErr, result, err)
}
}
func generationConfig() config.Config { func generationConfig() config.Config {
cfg := config.Defaults() cfg := config.Defaults()
cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home" cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home"

View File

@@ -2,6 +2,7 @@ package app
import ( import (
"context" "context"
"errors"
"fmt" "fmt"
"reflect" "reflect"
@@ -24,6 +25,7 @@ type profileExecutionOutcome struct {
BackendID string BackendID string
ModelName string ModelName string
ValidationStatus promptexec.ValidationStatus ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string LLMDebugPath string
} }
@@ -94,11 +96,24 @@ func executePreparedProfile(ctx context.Context, req profileExecutionRequest) (p
if callbackFailed { if callbackFailed {
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true} return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true}
} }
if req.DebugWriter != nil && req.DebugWriter.Enabled() && req.DebugRef != nil {
var generationError *promptexec.GenerationError
if errors.As(err, &generationError) {
path, debugErr := req.DebugWriter.WriteFailure(*req.DebugRef, generationError)
if path != "" {
outcome.LLMDebugPath = path
}
if debugErr != nil {
err = errors.Join(err, promptDebugWriteError(debugErr))
}
}
}
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)} return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)}
} }
if execution == nil { if execution == nil {
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)} return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)}
} }
outcome.RepairAttempts = repairAttemptsPointer(execution.Validation.RepairAttempts)
if preparationCount != 1 { if preparationCount != 1 {
return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()} return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()}
} }
@@ -157,7 +172,7 @@ func validatePreparedExecutionRequest(req profileExecutionRequest) error {
if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) { if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) {
return promptProvenanceError() return promptProvenanceError()
} }
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.BackendID == "" || req.Profile.ModelName == "" { if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.ModelName == "" {
return promptProvenanceError() return promptProvenanceError()
} }
return nil return nil
@@ -178,12 +193,19 @@ func validateExecutionProvenance(req profileExecutionRequest, preparation prompt
if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash || if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash ||
execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) || execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) ||
execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName || execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName ||
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" { execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" ||
execution.Validation.RepairAttempts < 0 || execution.Validation.RepairAttempts > preparation.Output.RepairAttempts ||
preparation.Output.RepairAttempts != definition.GeneratedTextRepairAttempts {
return promptProvenanceError() return promptProvenanceError()
} }
return nil return nil
} }
func repairAttemptsPointer(value int) *int {
copy := value
return &copy
}
func promptProvenanceError() error { func promptProvenanceError() error {
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil) return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil)
} }

View File

@@ -26,7 +26,7 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
if err != nil { if err != nil {
t.Fatalf("executePreparedProfile() error = %v", err) t.Fatalf("executePreparedProfile() error = %v", err)
} }
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 { if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 0 || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls) t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls)
} }
if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) { if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) {
@@ -34,6 +34,20 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
} }
} }
func TestExecutePreparedProfileRetainsCompletedRepairAttemptsOnLaterFailure(t *testing.T) {
prepared, inspection := preparedDailyProfile(t)
prepared.resolved.Definition.GeneratedTextRepairAttempts = 1
executor := &generationExecutor{repairAttempts: 1, rawOutput: []byte(`{"summary":42}`), prepare: func(value *promptexec.Preparation) { value.Output.RepairAttempts = 1 }}
outcome, _, err := executePreparedProfile(context.Background(), profileExecutionRequest{
Prepared: prepared, Prompt: inspection,
Profile: promptexec.ProfileInspection{ProfileID: inspection.ProfileID, BackendID: inspection.BackendID, ModelName: inspection.ModelName},
Executor: executor,
})
if err == nil || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 1 {
t.Fatalf("outcome/error = %#v/%v", outcome, err)
}
}
func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) { func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) {
prepared, inspection := preparedDailyProfile(t) prepared, inspection := preparedDailyProfile(t)
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir()) debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())

View File

@@ -48,6 +48,9 @@ func generatePromptReport(ctx context.Context, req promptReportRequest) (*Report
}) })
result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName
result.ValidationStatus = outcome.ValidationStatus result.ValidationStatus = outcome.ValidationStatus
if outcome.RepairAttempts != nil {
result.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
result.LLMDebugPath = outcome.LLMDebugPath result.LLMDebugPath = outcome.LLMDebugPath
if err != nil { if err != nil {
return result, generatedProfileExecutionError(req.Resolved, result.RunID, err) return result, generatedProfileExecutionError(req.Resolved, result.RunID, err)

View File

@@ -2,7 +2,6 @@ package app
import ( import (
"context" "context"
"os"
"strings" "strings"
"gitea.maximumdirect.net/eric/weatherreporter/internal/comparison" "gitea.maximumdirect.net/eric/weatherreporter/internal/comparison"
@@ -18,7 +17,6 @@ type PromptInspectionRequest struct {
Resolved report.Resolved Resolved report.Resolved
Executor promptexec.Executor Executor promptexec.Executor
Promptkit config.PromptkitConfig Promptkit config.PromptkitConfig
LookupEnv func(string) (string, bool)
} }
// PromptInspectionResult contains only safe identity and provenance from a // PromptInspectionResult contains only safe identity and provenance from a
@@ -39,7 +37,6 @@ type PromptExecutionsInspectionRequest struct {
Resolved []report.Resolved Resolved []report.Resolved
Executor promptexec.Executor Executor promptexec.Executor
Promptkit config.PromptkitConfig Promptkit config.PromptkitConfig
LookupEnv func(string) (string, bool)
} }
// ComparisonInspectionRequest contains the explicit profile selection for one // ComparisonInspectionRequest contains the explicit profile selection for one
@@ -48,7 +45,6 @@ type ComparisonInspectionRequest struct {
Resolved report.Resolved Resolved report.Resolved
ProfileIDs []string ProfileIDs []string
Executor promptexec.Executor Executor promptexec.Executor
LookupEnv func(string) (string, bool)
} }
// ComparisonInspectionResult contains the safe, shared prompt identity and // ComparisonInspectionResult contains the safe, shared prompt identity and
@@ -76,7 +72,6 @@ func InspectPromptExecution(ctx context.Context, req PromptInspectionRequest) (P
Resolved: []report.Resolved{req.Resolved}, Resolved: []report.Resolved{req.Resolved},
Executor: req.Executor, Executor: req.Executor,
Promptkit: req.Promptkit, Promptkit: req.Promptkit,
LookupEnv: req.LookupEnv,
}) })
if err != nil { if err != nil {
return PromptInspectionResult{}, err return PromptInspectionResult{}, err
@@ -111,7 +106,7 @@ func InspectPromptExecutions(ctx context.Context, req PromptExecutionsInspection
} }
profile, ok := profiles[profileID] profile, ok := profiles[profileID]
if !ok { if !ok {
profile, err = inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv) profile, err = inspectPromptProfile(ctx, req.Executor, profileID)
if err != nil { if err != nil {
return nil, err return nil, err
} }
@@ -154,7 +149,7 @@ func InspectComparisonExecution(ctx context.Context, req ComparisonInspectionReq
handler: handler, handler: handler,
} }
for _, profileID := range req.ProfileIDs { for _, profileID := range req.ProfileIDs {
profile, err := inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv) profile, err := inspectPromptProfile(ctx, req.Executor, profileID)
if err != nil { if err != nil {
return result, comparisonInspectionError("comparison profile inspection failed", err) return result, comparisonInspectionError("comparison profile inspection failed", err)
} }
@@ -190,7 +185,7 @@ func inspectPromptContract(ctx context.Context, executor promptexec.Executor, de
return inspection, nil return inspection, nil
} }
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string, lookupEnv func(string) (string, bool)) (promptexec.ProfileInspection, error) { func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string) (promptexec.ProfileInspection, error) {
profile, err := executor.InspectProfile(ctx, profileID) profile, err := executor.InspectProfile(ctx, profileID)
if err != nil { if err != nil {
return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err) return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err)
@@ -201,16 +196,7 @@ func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, pro
if profile.CredentialRequired { if profile.CredentialRequired {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil) return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil)
} }
if strings.TrimSpace(profile.APIKeyEnv) != "" { if strings.TrimSpace(profile.ModelName) == "" {
if lookupEnv == nil {
lookupEnv = os.LookupEnv
}
value, present := lookupEnv(profile.APIKeyEnv)
if !present || strings.TrimSpace(value) == "" {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile credential is unavailable", nil)
}
}
if strings.TrimSpace(profile.BackendID) == "" || strings.TrimSpace(profile.ModelName) == "" {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil) return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil)
} }
return profile, nil return profile, nil
@@ -221,7 +207,7 @@ func validPromptInput(inputs []promptexec.InputDefinition) bool {
} }
func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool { func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool {
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" && output.RepairAttempts == definition.GeneratedTextRepairAttempts
} }
func promptInspectionError(operation string, err error) error { func promptInspectionError(operation string, err error) error {

View File

@@ -50,7 +50,6 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
name string name string
prompt promptexec.PromptInspection prompt promptexec.PromptInspection
profile promptexec.ProfileInspection profile promptexec.ProfileInspection
lookupEnv func(string) (string, bool)
wantCategory promptexec.ErrorCategory wantCategory promptexec.ErrorCategory
}{ }{
{ {
@@ -81,9 +80,9 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
wantCategory: promptexec.InvalidConfiguration, wantCategory: promptexec.InvalidConfiguration,
}, },
{ {
name: "missing profile backend", name: "missing profile model",
prompt: basePrompt, prompt: basePrompt,
profile: promptexec.ProfileInspection{ProfileID: "default-profile", ModelName: "model"}, profile: promptexec.ProfileInspection{ProfileID: "default-profile", BackendID: "backend"},
wantCategory: promptexec.InvalidConfiguration, wantCategory: promptexec.InvalidConfiguration,
}, },
{ {
@@ -92,18 +91,11 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true}, profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true},
wantCategory: promptexec.MissingCredential, wantCategory: promptexec.MissingCredential,
}, },
{
name: "missing environment credential",
prompt: basePrompt,
profile: promptexec.ProfileInspection{ProfileID: "default-profile", APIKeyEnv: "PROMPT_API_KEY"},
lookupEnv: func(string) (string, bool) { return "", false },
wantCategory: promptexec.MissingCredential,
},
} }
for _, test := range tests { for _, test := range tests {
t.Run(test.name, func(t *testing.T) { t.Run(test.name, func(t *testing.T) {
executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}} executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}}
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor, LookupEnv: test.lookupEnv}) _, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor})
if err == nil || promptexec.CategoryOf(err) != test.wantCategory { if err == nil || promptexec.CategoryOf(err) != test.wantCategory {
t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory) t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory)
} }
@@ -265,13 +257,12 @@ func TestInspectComparisonExecutionStopsAtFirstProfileFailure(t *testing.T) {
prompt: validPromptInspection(resolved.Definition), prompt: validPromptInspection(resolved.Definition),
profiles: map[string]promptexec.ProfileInspection{ profiles: map[string]promptexec.ProfileInspection{
"weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"}, "weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"},
"missing-key": {ProfileID: "missing-key", APIKeyEnv: "PROMPT_API_KEY"}, "missing-key": {ProfileID: "missing-key", CredentialRequired: true},
"weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"}, "weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"},
}, },
} }
result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{ result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{
Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor, Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor,
LookupEnv: func(string) (string, bool) { return "", false },
}) })
if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential { if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential {
t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err)) t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err))
@@ -359,7 +350,7 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
return promptexec.PromptInspection{ return promptexec.PromptInspection{
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile", PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts},
} }
} }

View File

@@ -2,8 +2,10 @@ package app_test
import ( import (
"context" "context"
"fmt"
"os" "os"
"path/filepath" "path/filepath"
"strings"
"testing" "testing"
"time" "time"
@@ -14,14 +16,12 @@ import (
) )
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) { func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
lookupEnv := func(string) (string, bool) { return "test-key", true }
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) { inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
t.Helper() t.Helper()
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{ result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, id), Resolved: resolvedPromptProfile(t, id),
Executor: adapter, Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: profile}, Promptkit: config.PromptkitConfig{Profile: profile},
LookupEnv: lookupEnv,
}) })
if err != nil { if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err) t.Fatalf("InspectPromptExecution() error = %v", err)
@@ -50,6 +50,55 @@ model: local-weather
inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather") inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather")
} }
func TestPromptInspectionAcceptsMaintainedEndpointOnlyProfile(t *testing.T) {
adapter, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: filepath.Join("..", "..", "examples", "weather-light-local-profile.yml")})
if err != nil {
t.Fatalf("New() error = %v", err)
}
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly), Executor: adapter,
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" {
t.Fatalf("inspection = %#v", result)
}
if strings.Contains(fmt.Sprintf("%#v", result), "127.0.0.1") {
t.Fatalf("inspection leaks endpoint: %#v", result)
}
}
func TestPromptInspectionSupportsRakestrawhomeProfileOffline(t *testing.T) {
adapter, err := promptkitadapter.New(promptkitadapter.Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
prompt, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly),
Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: "rakestrawhome-gemma-4-31b"},
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if prompt.ProfileID != "rakestrawhome-gemma-4-31b" || prompt.BackendID != "rakestrawhome" || prompt.ModelName == "" {
t.Fatalf("prompt inspection = %#v", prompt)
}
comparison, err := app.InspectComparisonExecution(context.Background(), app.ComparisonInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly),
ProfileIDs: []string{"rakestrawhome-gemma-4-31b", "weather-deep"},
Executor: adapter,
})
if err != nil {
t.Fatalf("InspectComparisonExecution() error = %v", err)
}
if len(comparison.Profiles) != 2 || comparison.Profiles[0].ProfileID != "rakestrawhome-gemma-4-31b" || comparison.Profiles[0].BackendID != "rakestrawhome" || comparison.Profiles[0].ModelName == "" {
t.Fatalf("comparison inspection = %#v", comparison)
}
}
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved { func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
t.Helper() t.Helper()
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC) now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)

View File

@@ -355,7 +355,7 @@ func comparisonResult(outputDirectory string, results []app.ComparisonProfileRes
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
result := &app.ComparisonResult{ result := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report", ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64), PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago", StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory, ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"), ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),

View File

@@ -132,7 +132,7 @@ func actionConfigPath(t *testing.T) string {
func generatedReportResult() *app.ReportResult { func generatedReportResult() *app.ReportResult {
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
return &app.ReportResult{ return &app.ReportResult{
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0",
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago", RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed, ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,

View File

@@ -43,6 +43,7 @@ type generateSummary struct {
ModelName string `json:"modelName,omitempty"` ModelName string `json:"modelName,omitempty"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"` SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
ValidationStatus string `json:"validationStatus,omitempty"` ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
Notification *generateNotificationSummary `json:"notification,omitempty"` Notification *generateNotificationSummary `json:"notification,omitempty"`
Error string `json:"error,omitempty"` Error string `json:"error,omitempty"`
} }
@@ -106,6 +107,7 @@ type comparisonProfileSummary struct {
ModelName string `json:"modelName"` ModelName string `json:"modelName"`
Status string `json:"status"` Status string `json:"status"`
ValidationStatus string `json:"validationStatus,omitempty"` ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
ReportPath string `json:"reportPath,omitempty"` ReportPath string `json:"reportPath,omitempty"`
LLMDebugPath string `json:"llmDebugPath,omitempty"` LLMDebugPath string `json:"llmDebugPath,omitempty"`
Error *comparison.SafeError `json:"error,omitempty"` Error *comparison.SafeError `json:"error,omitempty"`
@@ -129,6 +131,10 @@ func newGenerateSummary(result *app.ReportResult, err error) generateSummary {
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...) summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
summary.ValidationStatus = string(result.ValidationStatus) summary.ValidationStatus = string(result.ValidationStatus)
if result.RepairAttempts != nil {
value := *result.RepairAttempts
summary.RepairAttempts = &value
}
summary.OutputPath = result.OutputPath summary.OutputPath = result.OutputPath
summary.LLMDebugPath = result.LLMDebugPath summary.LLMDebugPath = result.LLMDebugPath
summary.Notification = newGenerateNotificationSummary(result.Notification) summary.Notification = newGenerateNotificationSummary(result.Notification)
@@ -162,6 +168,11 @@ func newGenerateNotificationSummary(result *app.NotificationResult) *generateNot
return summary return summary
} }
func repairAttemptsCopy(value int) *int {
copy := value
return &copy
}
func newBatchSummary(result *app.BatchResult, err error) batchSummary { func newBatchSummary(result *app.BatchResult, err error) batchSummary {
summary := batchSummary{Command: commandRun} summary := batchSummary{Command: commandRun}
if result == nil { if result == nil {
@@ -203,6 +214,9 @@ func newComparisonSummary(result *app.ComparisonResult, err error) comparisonSum
Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath, Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath,
LLMDebugPath: profile.LLMDebugPath, Error: profile.Error, LLMDebugPath: profile.LLMDebugPath, Error: profile.Error,
}) })
if profile.RepairAttempts != nil {
summary.Results[len(summary.Results)-1].RepairAttempts = repairAttemptsCopy(*profile.RepairAttempts)
}
} }
summary.Status = comparisonSummaryStatus(result, err) summary.Status = comparisonSummaryStatus(result, err)
if err != nil { if err != nil {

View File

@@ -19,7 +19,7 @@ import (
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) { func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil) summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 { if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
t.Fatalf("summary = %#v", summary) t.Fatalf("summary = %#v", summary)
} }
@@ -39,7 +39,7 @@ func TestComparisonSummaryUsesLockedOrderAndSafeFields(t *testing.T) {
profileFailure := comparison.NewSafeError("generation", "execute prompt failed") profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
result := &app.ComparisonResult{ result := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report", ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64), PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago", StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily", ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
@@ -113,7 +113,7 @@ func TestComparisonSummaryClassifiesCompleteAndAllFailedResults(t *testing.T) {
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC) started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
failure := comparison.NewSafeError("generation", "execute prompt failed") failure := comparison.NewSafeError("generation", "execute prompt failed")
complete := &app.ComparisonResult{ complete := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64), ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)}, StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml", OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
Total: 2, Succeeded: 2, Total: 2, Succeeded: 2,

View File

@@ -15,7 +15,7 @@ import (
const ( const (
// SchemaVersion identifies the supported comparison manifest schema. // SchemaVersion identifies the supported comparison manifest schema.
SchemaVersion = "weatherreporter.comparison.v1" SchemaVersion = "weatherreporter.comparison.v2"
// ManifestFilename is the canonical name of a comparison manifest. // ManifestFilename is the canonical name of a comparison manifest.
ManifestFilename = "comparison.json" ManifestFilename = "comparison.json"
@@ -71,6 +71,7 @@ type Result struct {
ModelName string `json:"modelName"` ModelName string `json:"modelName"`
Status string `json:"status"` Status string `json:"status"`
ValidationStatus string `json:"validationStatus,omitempty"` ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
ReportPath string `json:"reportPath,omitempty"` ReportPath string `json:"reportPath,omitempty"`
Error *SafeError `json:"error,omitempty"` Error *SafeError `json:"error,omitempty"`
} }
@@ -276,11 +277,14 @@ func (manifest Manifest) Validate() error {
if strings.TrimSpace(result.ModelName) == "" { if strings.TrimSpace(result.ModelName) == "" {
return fmt.Errorf("result %d has a blank model name", result.Position) return fmt.Errorf("result %d has a blank model name", result.Position)
} }
if result.RepairAttempts != nil && *result.RepairAttempts < 0 {
return fmt.Errorf("result %d has negative repair attempts", result.Position)
}
switch result.Status { switch result.Status {
case StatusSucceeded: case StatusSucceeded:
succeeded++ succeeded++
if result.ValidationStatus != "passed" { if result.ValidationStatus != "passed" || result.RepairAttempts == nil {
return fmt.Errorf("successful result %d did not pass validation", result.Position) return fmt.Errorf("successful result %d did not pass validation", result.Position)
} }
expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID) expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID)
@@ -302,6 +306,9 @@ func (manifest Manifest) Validate() error {
if err := result.Error.validate(); err != nil { if err := result.Error.validate(); err != nil {
return fmt.Errorf("failed result %d: %w", result.Position, err) return fmt.Errorf("failed result %d: %w", result.Position, err)
} }
if result.ValidationStatus != "" && result.RepairAttempts == nil {
return fmt.Errorf("failed result %d has validation without repair provenance", result.Position)
}
default: default:
return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status) return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status)
} }
@@ -324,7 +331,7 @@ func (safeError SafeError) validate() error {
} }
func isValidationStatus(status string) bool { func isValidationStatus(status string) bool {
return status == "" || status == "failed" || status == "skipped" return status == "" || status == "passed" || status == "failed" || status == "skipped"
} }
func isSHA256(value string) bool { func isSHA256(value string) bool {

View File

@@ -118,7 +118,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
t.Fatalf("EncodeManifest() error = %v", err) t.Fatalf("EncodeManifest() error = %v", err)
} }
want := "{\n" + want := "{\n" +
" \"schemaVersion\": \"weatherreporter.comparison.v1\",\n" + " \"schemaVersion\": \"weatherreporter.comparison.v2\",\n" +
" \"comparisonId\": \"comparison_daily-2026-08-24\",\n" + " \"comparisonId\": \"comparison_daily-2026-08-24\",\n" +
" \"startedAt\": \"2026-08-24T12:00:00Z\",\n" + " \"startedAt\": \"2026-08-24T12:00:00Z\",\n" +
" \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" + " \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" +
@@ -146,6 +146,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
" \"modelName\": \"gpt-5-mini\",\n" + " \"modelName\": \"gpt-5-mini\",\n" +
" \"status\": \"succeeded\",\n" + " \"status\": \"succeeded\",\n" +
" \"validationStatus\": \"passed\",\n" + " \"validationStatus\": \"passed\",\n" +
" \"repairAttempts\": 0,\n" +
" \"reportPath\": \"01-weather-light.md\"\n" + " \"reportPath\": \"01-weather-light.md\"\n" +
" },\n" + " },\n" +
" {\n" + " {\n" +
@@ -201,6 +202,9 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
{name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }}, {name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }},
{name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }}, {name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }},
{name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }}, {name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }},
{name: "failed passed result without repair provenance", mutate: func(manifest *Manifest) {
manifest.Results[1].ValidationStatus = "passed"
}},
{name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }}, {name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }},
{name: "duplicate report path", mutate: func(manifest *Manifest) { {name: "duplicate report path", mutate: func(manifest *Manifest) {
manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath} manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath}
@@ -222,6 +226,17 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
} }
} }
func TestManifestValidateAcceptsPostValidationFailure(t *testing.T) {
t.Parallel()
manifest := validManifest()
manifest.Results[1].ValidationStatus = "passed"
manifest.Results[1].RepairAttempts = intPtr(0)
if err := manifest.Validate(); err != nil {
t.Fatalf("Manifest.Validate() rejected post-validation failure: %v", err)
}
}
func TestLogicalBundleValidate(t *testing.T) { func TestLogicalBundleValidate(t *testing.T) {
t.Parallel() t.Parallel()
@@ -309,6 +324,7 @@ func validManifest() Manifest {
ModelName: "gpt-5-mini", ModelName: "gpt-5-mini",
Status: StatusSucceeded, Status: StatusSucceeded,
ValidationStatus: "passed", ValidationStatus: "passed",
RepairAttempts: intPtr(0),
ReportPath: "01-weather-light.md", ReportPath: "01-weather-light.md",
}, },
{ {
@@ -321,3 +337,8 @@ func validManifest() Manifest {
}, },
} }
} }
func intPtr(value int) *int {
copy := value
return &copy
}

View File

@@ -388,6 +388,7 @@ var resultFields = map[string]jsonValueValidator{
"modelName": nil, "modelName": nil,
"status": nil, "status": nil,
"validationStatus": nil, "validationStatus": nil,
"repairAttempts": validateRepairAttemptsJSON,
"reportPath": nil, "reportPath": nil,
"error": validateSafeErrorJSON, "error": validateSafeErrorJSON,
} }
@@ -430,7 +431,7 @@ func validateResultsJSON(decoder *json.Decoder) error {
return fmt.Errorf("results must be an array") return fmt.Errorf("results must be an array")
} }
for decoder.More() { for decoder.More() {
if err := validateJSONObject(decoder, resultFields); err != nil { if err := validateOrderedJSONObject(decoder, resultFields, []string{"position", "profileId", "backendId", "modelName", "status", "validationStatus", "repairAttempts", "reportPath", "error"}); err != nil {
return err return err
} }
} }
@@ -444,6 +445,65 @@ func validateResultsJSON(decoder *json.Decoder) error {
return nil return nil
} }
func validateOrderedJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator, order []string) error {
token, err := decoder.Token()
if err != nil {
return err
}
if delimiter, ok := token.(json.Delim); !ok || delimiter != '{' {
return fmt.Errorf("manifest value must be an object")
}
seen := make(map[string]struct{}, len(fields))
last := -1
for decoder.More() {
token, err := decoder.Token()
if err != nil {
return err
}
name, ok := token.(string)
if !ok {
return fmt.Errorf("manifest field name is invalid")
}
validator, known := fields[name]
if !known {
return fmt.Errorf("manifest field %q is not canonical", name)
}
if _, duplicate := seen[name]; duplicate {
return fmt.Errorf("manifest field %q is duplicated", name)
}
position := -1
for index, candidate := range order {
if candidate == name {
position = index
break
}
}
if position <= last {
return fmt.Errorf("manifest field %q is out of canonical order", name)
}
seen[name] = struct{}{}
last = position
if validator == nil {
var value json.RawMessage
if err := decoder.Decode(&value); err != nil {
return err
}
continue
}
if err := validator(decoder); err != nil {
return err
}
}
token, err = decoder.Token()
if err != nil {
return err
}
if delimiter, ok := token.(json.Delim); !ok || delimiter != '}' {
return fmt.Errorf("manifest object has an invalid terminator")
}
return nil
}
func validateSafeErrorJSON(decoder *json.Decoder) error { func validateSafeErrorJSON(decoder *json.Decoder) error {
token, err := decoder.Token() token, err := decoder.Token()
if err != nil { if err != nil {
@@ -458,6 +518,24 @@ func validateSafeErrorJSON(decoder *json.Decoder) error {
return validateJSONObjectBody(decoder, safeErrorFields) return validateJSONObjectBody(decoder, safeErrorFields)
} }
func validateRepairAttemptsJSON(decoder *json.Decoder) error {
var raw json.RawMessage
if err := decoder.Decode(&raw); err != nil {
return err
}
if bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
return fmt.Errorf("repairAttempts must be an integer")
}
var value int
if err := json.Unmarshal(raw, &value); err != nil {
return err
}
if value < 0 {
return fmt.Errorf("repairAttempts must not be negative")
}
return nil
}
func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error { func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error {
token, err := decoder.Token() token, err := decoder.Token()
if err != nil { if err != nil {

View File

@@ -1,6 +1,7 @@
package comparison package comparison
import ( import (
"bytes"
"context" "context"
"encoding/json" "encoding/json"
"errors" "errors"
@@ -146,6 +147,23 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
t.Helper() t.Helper()
appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`) appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`)
}}, }},
{name: "null repair attempts", mutate: func(t *testing.T, directory string) {
t.Helper()
path := filepath.Join(directory, ManifestFilename)
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
before := []byte(" \"status\": \"failed\",\n \"error\": {")
after := []byte(" \"status\": \"failed\",\n \"repairAttempts\": null,\n \"error\": {")
data = bytes.Replace(data, before, after, 1)
if bytes.Equal(data, readFile(t, path)) {
t.Fatal("failed to add null repairAttempts")
}
if err := os.WriteFile(path, data, 0o600); err != nil {
t.Fatal(err)
}
}},
{name: "traversal report path", mutate: func(t *testing.T, directory string) { {name: "traversal report path", mutate: func(t *testing.T, directory string) {
t.Helper() t.Helper()
path := filepath.Join(directory, ManifestFilename) path := filepath.Join(directory, ManifestFilename)
@@ -202,6 +220,30 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
} }
} }
func TestValidateRepairAttemptsJSONRejectsNonIntegerValues(t *testing.T) {
t.Parallel()
for _, value := range []string{"null", "-1", "1.5", `"1"`, "9223372036854775808", "{"} {
value := value
t.Run(value, func(t *testing.T) {
t.Parallel()
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err == nil {
t.Fatalf("validateRepairAttemptsJSON(%q) error = nil", value)
}
})
}
}
func TestValidateRepairAttemptsJSONAcceptsNonnegativeIntegers(t *testing.T) {
t.Parallel()
for _, value := range []string{"0", "1"} {
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err != nil {
t.Fatalf("validateRepairAttemptsJSON(%q) error = %v", value, err)
}
}
}
func appendManifestField(t *testing.T, directory, field string) { func appendManifestField(t *testing.T, directory, field string) {
t.Helper() t.Helper()
path := filepath.Join(directory, ManifestFilename) path := filepath.Join(directory, ManifestFilename)
@@ -1077,7 +1119,7 @@ func testBundleWithReports(t *testing.T, count int) LogicalBundle {
} }
manifest.Results[i] = Result{ manifest.Results[i] = Result{
Position: position, ProfileID: profileID, ModelName: "gpt-5-mini", Position: position, ProfileID: profileID, ModelName: "gpt-5-mini",
Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: path, Status: StatusSucceeded, ValidationStatus: "passed", RepairAttempts: intPtr(0), ReportPath: path,
} }
reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")} reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")}
} }

View File

@@ -1,6 +1,2 @@
id: weather-balanced id: weather-balanced
backend: openrouter base_profile: gemini-flash-latest
model: "~google/gemini-flash-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -1,6 +1,2 @@
id: weather-deep id: weather-deep
backend: openrouter base_profile: claude-sonnet-latest
model: "~anthropic/claude-sonnet-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -1,5 +1,2 @@
id: weather-light id: weather-light
backend: openrouter base_profile: deepseek-4-flash
model: deepseek/deepseek-v4-flash
timeout_seconds: 180
service_tier: flex

View File

@@ -1,5 +1,5 @@
id: weather.daily_generated_text id: weather.daily_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-balanced default_profile: weather-balanced
description: Daily weather report analysis prompt. description: Daily weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: daily.generated_text.schema.json schema_path: daily.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.hourly_generated_text id: weather.hourly_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-light default_profile: weather-light
description: Hourly weather report analysis prompt. description: Hourly weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: hourly.generated_text.schema.json schema_path: hourly.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.today_generated_text id: weather.today_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-balanced default_profile: weather-balanced
description: Today's weather report analysis prompt. description: Today's weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: today.generated_text.schema.json schema_path: today.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.tomorrow_generated_text id: weather.tomorrow_generated_text
version: "2.0.0" version: "2.1.0"
default_profile: weather-balanced default_profile: weather-balanced
description: Tomorrow's weather report analysis prompt. description: Tomorrow's weather report analysis prompt.
inputs: inputs:
@@ -21,3 +21,4 @@ output:
format: json format: json
validation_mode: json_schema validation_mode: json_schema
schema_path: tomorrow.generated_text.schema.json schema_path: tomorrow.generated_text.schema.json
repair_attempts: 1

View File

@@ -77,8 +77,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if err := yaml.Unmarshal(data, &definition); err != nil { if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("decode prompt definition: %v", err) t.Fatalf("decode prompt definition: %v", err)
} }
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile { if definition.ID != tc.id || definition.Version != "2.1.0" || definition.DefaultProfile != tc.profile {
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile) t.Fatalf("definition = %#v, want %s version 2.1.0 and profile %s", definition, tc.id, tc.profile)
} }
sharedInstruction := false sharedInstruction := false
for _, message := range definition.Messages { for _, message := range definition.Messages {
@@ -92,8 +92,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" { if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs) t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
} }
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts != nil { if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts == nil || *definition.Output.RepairAttempts != 1 {
t.Fatalf("output = %#v, want JSON schema output without repair attempts", definition.Output) t.Fatalf("output = %#v, want JSON schema output with one repair attempt", definition.Output)
} }
if _, err := promptassets.Schema(tc.schemaID); err != nil { if _, err := promptassets.Schema(tc.schemaID); err != nil {
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err) t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
@@ -286,11 +286,11 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"}, {"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
} { } {
t.Run(want.id, func(t *testing.T) { t.Run(want.id, func(t *testing.T) {
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.0.0") inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.1.0")
if err != nil { if err != nil {
t.Fatalf("InspectPrompt() error = %v", err) t.Fatalf("InspectPrompt() error = %v", err)
} }
if inspection.PromptID != want.id || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != want.profile { if inspection.PromptID != want.id || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != want.profile || inspection.OutputContract.RepairAttempts != 1 {
t.Fatalf("inspection = %#v", inspection) t.Fatalf("inspection = %#v", inspection)
} }
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID) profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
@@ -361,6 +361,29 @@ func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
} }
} }
func TestEmbeddedProfilesAreMinimalBaseAliases(t *testing.T) {
wantBases := map[string]string{
"weather-balanced.yml": "gemini-flash-latest",
"weather-deep.yml": "claude-sonnet-latest",
"weather-light.yml": "deepseek-4-flash",
}
for path, wantBase := range wantBases {
t.Run(path, func(t *testing.T) {
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
if err != nil {
t.Fatalf("read profile: %v", err)
}
var definition map[string]string
if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("unmarshal profile: %v", err)
}
if definition["id"] != strings.TrimSuffix(path, ".yml") || definition["base_profile"] != wantBase || len(definition) != 2 {
t.Fatalf("profile definition = %#v, want only its id and base profile %q", definition, wantBase)
}
})
}
}
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) { func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"} forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error { if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
@@ -391,7 +414,7 @@ func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
if err != nil { if err != nil {
return err return err
} }
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "repair_attempts:", "weather.daily_report"} { for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "weather.daily_report"} {
if strings.Contains(string(data), unwanted) { if strings.Contains(string(data), unwanted) {
t.Fatalf("%s contains retired runtime setting %q", path, unwanted) t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
} }

View File

@@ -21,8 +21,9 @@ import (
var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform") var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform")
const ( const (
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v2" promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v3"
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v2" promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v3"
promptFailureDebugSchemaVersion = "weatherreporter.prompt_failure_debug.v1"
debugDirectoryMode = 0o700 debugDirectoryMode = 0o700
debugFileMode = 0o600 debugFileMode = 0o600
) )
@@ -53,6 +54,7 @@ type PromptDebugOutput struct {
Format string `json:"format"` Format string `json:"format"`
ValidationMode string `json:"validationMode"` ValidationMode string `json:"validationMode"`
SchemaPath string `json:"schemaPath"` SchemaPath string `json:"schemaPath"`
RepairAttempts int `json:"repairAttempts"`
} }
// PromptDebugPreparation is the explicit, content-safe mapping of preparation // PromptDebugPreparation is the explicit, content-safe mapping of preparation
@@ -97,10 +99,28 @@ type PromptDebugUsage struct {
// PromptDebugValidation is the completed validation detail retained in the // PromptDebugValidation is the completed validation detail retained in the
// explicitly enabled debug store. // explicitly enabled debug store.
type PromptDebugValidation struct { type PromptDebugValidation struct {
Status string `json:"status"` Status string `json:"status"`
Mode string `json:"mode"` Mode string `json:"mode"`
SchemaPath string `json:"schemaPath"` SchemaPath string `json:"schemaPath"`
Diagnostics []string `json:"diagnostics,omitempty"` RepairAttempts int `json:"repairAttempts"`
Diagnostics []string `json:"diagnostics,omitempty"`
}
// PromptFailureDebugArtifact records provider detail only in explicit debug storage.
type PromptFailureDebugArtifact struct {
SchemaVersion string `json:"schemaVersion"`
ReportID report.ID `json:"reportId"`
ValidDate string `json:"validDate"`
RunID string `json:"runId"`
Failure PromptFailure `json:"failure"`
}
type PromptFailure struct {
Category string `json:"category"`
StatusCode int `json:"statusCode,omitempty"`
ProviderCode string `json:"providerCode,omitempty"`
ProviderType string `json:"providerType,omitempty"`
ProviderMessage string `json:"providerMessage,omitempty"`
} }
// PromptDebugExecution is the explicit mapping of execution provenance. // PromptDebugExecution is the explicit mapping of execution provenance.
@@ -235,6 +255,24 @@ func (w *PromptDebugWriter) WriteExecution(ref PromptDebugRef, execution prompte
return directory, nil return directory, nil
} }
// WriteFailure stores the project-owned structured provider failure.
func (w *PromptDebugWriter) WriteFailure(ref PromptDebugRef, failure *promptexec.GenerationError) (string, error) {
if !w.Enabled() || failure == nil {
return "", nil
}
directory, secureDirectory, err := w.runDirectory(ref)
if err != nil {
return "", err
}
defer secureDirectory.Close()
artifact := PromptFailureDebugArtifact{SchemaVersion: promptFailureDebugSchemaVersion, ReportID: ref.ReportID, ValidDate: ref.ValidDate, RunID: ref.RunID,
Failure: PromptFailure{Category: string(failure.Category()), StatusCode: failure.StatusCode(), ProviderCode: failure.ProviderCode(), ProviderType: failure.ProviderType(), ProviderMessage: failure.ProviderMessage()}}
if err := secureDirectory.writeJSON("failure.json", artifact); err != nil {
return "", err
}
return directory, nil
}
func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) { func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) {
if err := validatePromptDebugRef(ref); err != nil { if err := validatePromptDebugRef(ref); err != nil {
return "", nil, err return "", nil, err
@@ -283,7 +321,7 @@ func promptDebugPreparation(value promptexec.Preparation) PromptDebugPreparation
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash, PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash,
RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes), RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes),
ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName, ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName,
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath}, Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath, RepairAttempts: value.Output.RepairAttempts},
StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration, StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration,
} }
} }
@@ -300,7 +338,7 @@ func promptDebugExecution(value promptexec.Execution) PromptDebugExecution {
} }
func promptDebugValidation(value promptexec.Validation) PromptDebugValidation { func promptDebugValidation(value promptexec.Validation) PromptDebugValidation {
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, Diagnostics: append([]string(nil), value.Diagnostics...)} return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, RepairAttempts: value.RepairAttempts, Diagnostics: append([]string(nil), value.Diagnostics...)}
} }
func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage { func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage {

View File

@@ -44,13 +44,13 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir) t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir)
} }
preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json")) preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json"))
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v2", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`} { for _, want := range []string{"weatherreporter.prompt_preparation_debug.v3", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`, `"repairAttempts": 0`} {
if !strings.Contains(string(preparationData), want) { if !strings.Contains(string(preparationData), want) {
t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData) t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData)
} }
} }
executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json")) executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json"))
for _, want := range []string{"weatherreporter.prompt_execution_debug.v2", "Generated forecast prose.", "validation details", `"status": "passed"`} { for _, want := range []string{"weatherreporter.prompt_execution_debug.v3", "Generated forecast prose.", "validation details", `"status": "passed"`, `"repairAttempts": 0`} {
if !strings.Contains(string(executionData), want) { if !strings.Contains(string(executionData), want) {
t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData) t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData)
} }
@@ -66,6 +66,25 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode) assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode)
} }
func TestPromptDebugWriterWritesProviderFailureOnlyToDebugStore(t *testing.T) {
writer, err := NewPromptDebugWriter(filepath.Join(t.TempDir(), "operator-debug"))
if err != nil {
t.Fatalf("NewPromptDebugWriter() error = %v", err)
}
marker := "provider-private-marker"
directory, err := writer.WriteFailure(promptDebugRef(), promptexec.NewGenerationError(429, "rate_limit", "provider_error", marker, nil))
if err != nil {
t.Fatalf("WriteFailure() error = %v", err)
}
data := readPromptDebugFile(t, filepath.Join(directory, "failure.json"))
for _, want := range []string{"weatherreporter.prompt_failure_debug.v1", `"category": "generation"`, `"statusCode": 429`, marker} {
if !strings.Contains(string(data), want) {
t.Fatalf("failure artifact missing %q: %s", want, data)
}
}
assertPromptDebugMode(t, filepath.Join(directory, "failure.json"), debugFileMode)
}
func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) { func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) {
const marker = "private-debug-marker" const marker = "private-debug-marker"
tests := []struct { tests := []struct {
@@ -349,7 +368,7 @@ func promptDebugExecutionFixture() promptexec.Execution {
InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model", InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model",
GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15}, GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15},
StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second, StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second,
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", []string{"validation details"}), Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", 0, []string{"validation details"}),
RawOutput: []byte("Generated forecast prose."), RawOutput: []byte("Generated forecast prose."),
Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}}, Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}},
} }

View File

@@ -78,3 +78,15 @@ func boundText(value string, limit int) string {
} }
return value return value
} }
func boundCodePoints(value string, limit int) string {
if limit <= 0 || value == "" {
return ""
}
value = strings.ToValidUTF8(value, "<22>")
runes := []rune(value)
if len(runes) <= limit {
return value
}
return string(runes[:limit])
}

View File

@@ -51,6 +51,7 @@ type OutputContract struct {
Format string Format string
ValidationMode string ValidationMode string
SchemaPath string SchemaPath string
RepairAttempts int
} }
// ProfileInspection describes the safe, selected execution identity for one profile. // ProfileInspection describes the safe, selected execution identity for one profile.
@@ -141,19 +142,21 @@ type TokenUsage struct {
// Validation records a completed output validation check. // Validation records a completed output validation check.
type Validation struct { type Validation struct {
Status ValidationStatus Status ValidationStatus
Mode string Mode string
SchemaPath string SchemaPath string
Diagnostics []string RepairAttempts int
Diagnostics []string
} }
// NewValidation returns a completed validation value with bounded diagnostics. // NewValidation returns a completed validation value with bounded diagnostics.
func NewValidation(status ValidationStatus, mode string, schemaPath string, diagnostics []string) Validation { func NewValidation(status ValidationStatus, mode string, schemaPath string, repairAttempts int, diagnostics []string) Validation {
return Validation{ return Validation{
Status: status, Status: status,
Mode: mode, Mode: mode,
SchemaPath: schemaPath, SchemaPath: schemaPath,
Diagnostics: boundDiagnostics(diagnostics), RepairAttempts: repairAttempts,
Diagnostics: boundDiagnostics(diagnostics),
} }
} }
@@ -235,6 +238,86 @@ func (e *Error) Category() ErrorCategory {
return e.category return e.category
} }
// GenerationError retains provider failure details for programmatic handling
// without exposing them through routine formatting or serialization.
type GenerationError struct {
statusCode int
providerCode string
providerType string
providerMessage string
err *Error
}
// NewGenerationError returns a classified generation failure with bounded
// provider details. The dependency cause remains reachable through err only.
func NewGenerationError(statusCode int, providerCode string, providerType string, providerMessage string, cause error) *GenerationError {
return &GenerationError{
statusCode: statusCode,
providerCode: boundCodePoints(providerCode, 256),
providerType: boundCodePoints(providerType, 256),
providerMessage: boundCodePoints(providerMessage, 4096),
err: NewError(Generation, "provider generation failed", cause),
}
}
// StatusCode returns the provider HTTP status when one was available.
func (e *GenerationError) StatusCode() int {
if e == nil {
return 0
}
return e.statusCode
}
// ProviderCode returns the bounded provider error code.
func (e *GenerationError) ProviderCode() string {
if e == nil {
return ""
}
return e.providerCode
}
// ProviderType returns the bounded provider error type.
func (e *GenerationError) ProviderType() string {
if e == nil {
return ""
}
return e.providerType
}
// ProviderMessage returns the bounded provider error message.
func (e *GenerationError) ProviderMessage() string {
if e == nil {
return ""
}
return e.providerMessage
}
// Category returns Generation for every generation failure.
func (e *GenerationError) Category() ErrorCategory { return Generation }
// Error intentionally excludes provider details from ordinary error text.
func (e *GenerationError) Error() string {
if e == nil {
return ""
}
message := NewError(Generation, "provider generation failed", nil).Error()
if e.statusCode == 0 {
return message
}
return fmt.Sprintf("%s (HTTP %d)", message, e.statusCode)
}
// GoString keeps %#v formatting as safe as ordinary error formatting.
func (e *GenerationError) GoString() string { return e.Error() }
// Unwrap preserves the project-owned classified error and its hidden cause.
func (e *GenerationError) Unwrap() error {
if e == nil {
return nil
}
return e.err
}
// CapacityError adds the safe backend identity to a capacity failure. // CapacityError adds the safe backend identity to a capacity failure.
type CapacityError struct { type CapacityError struct {
BackendID string BackendID string

View File

@@ -3,6 +3,7 @@ package promptexec
import ( import (
"context" "context"
"errors" "errors"
"fmt"
"strings" "strings"
"testing" "testing"
"unicode/utf8" "unicode/utf8"
@@ -172,8 +173,8 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
if bounded[0] != "a<>b" { if bounded[0] != "a<>b" {
t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0]) t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0])
} }
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", values) validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", 2, values)
if len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" { if validation.RepairAttempts != 2 || len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
t.Fatalf("validation = %#v, want bounded diagnostics", validation) t.Fatalf("validation = %#v, want bounded diagnostics", validation)
} }
@@ -187,8 +188,53 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
} }
} }
func TestGenerationErrorKeepsProviderDetailsOutOfRoutineFormatting(t *testing.T) {
cause := errors.New("provider response must stay hidden")
error := NewGenerationError(
429,
string([]byte{'c', 0xff})+strings.Repeat("界", 300),
strings.Repeat("type", 100),
string([]byte{'m', 0xff})+strings.Repeat("界", 4_200),
cause,
)
if error.StatusCode() != 429 || error.Category() != Generation || CategoryOf(error) != Generation {
t.Fatalf("generation error identity = %#v", error)
}
if utf8.RuneCountInString(error.ProviderCode()) != 256 || utf8.RuneCountInString(error.ProviderType()) != 256 || utf8.RuneCountInString(error.ProviderMessage()) != 4096 {
t.Fatalf("provider detail bounds = %d/%d/%d", utf8.RuneCountInString(error.ProviderCode()), utf8.RuneCountInString(error.ProviderType()), utf8.RuneCountInString(error.ProviderMessage()))
}
if !utf8.ValidString(error.ProviderCode()) || !utf8.ValidString(error.ProviderType()) || !utf8.ValidString(error.ProviderMessage()) {
t.Fatalf("provider details are not valid UTF-8: %#v", error)
}
if !errors.Is(error, cause) {
t.Fatal("errors.Is() = false, want preserved dependency cause")
}
var owned *Error
if !errors.As(error, &owned) || owned.Category() != Generation {
t.Fatalf("errors.As(*Error) = %#v, want project-owned generation error", owned)
}
want := "generation: provider generation failed (HTTP 429)"
if error.Error() != want || fmt.Sprintf("%#v", error) != want {
t.Fatalf("ordinary formatting = %q / %#v, want %q", error.Error(), error, want)
}
for _, private := range []string{cause.Error(), error.ProviderCode(), error.ProviderType(), error.ProviderMessage()} {
if strings.Contains(error.Error(), private) || strings.Contains(fmt.Sprintf("%#v", error), private) {
t.Fatalf("generation error leaks provider detail %q", private)
}
}
statusOnly := NewGenerationError(503, "", "", "", nil)
if statusOnly.Error() != "generation: provider generation failed (HTTP 503)" {
t.Fatalf("status-only error = %q", statusOnly)
}
var nilError *GenerationError
if nilError.StatusCode() != 0 || nilError.ProviderCode() != "" || nilError.ProviderType() != "" || nilError.ProviderMessage() != "" || nilError.Category() != Generation || nilError.Error() != "" || nilError.GoString() != "" || nilError.Unwrap() != nil {
t.Fatalf("nil generation error = %#v", nilError)
}
}
func TestContractCopiesMutableValues(t *testing.T) { func TestContractCopiesMutableValues(t *testing.T) {
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}} preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}, Output: OutputContract{RepairAttempts: 3}}
preparationDebug := &PreparationDebug{ preparationDebug := &PreparationDebug{
RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}}, RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}},
StructuredSchema: []byte("schema body"), StructuredSchema: []byte("schema body"),
@@ -196,7 +242,7 @@ func TestContractCopiesMutableValues(t *testing.T) {
} }
execution := Execution{ execution := Execution{
InputHashes: map[string]string{"data_package": "input-hash"}, InputHashes: map[string]string{"data_package": "input-hash"},
Validation: Validation{Diagnostics: []string{"validation detail"}}, Validation: Validation{RepairAttempts: 2, Diagnostics: []string{"validation detail"}},
RawOutput: []byte("generated output"), RawOutput: []byte("generated output"),
Debug: &ExecutionDebug{ Debug: &ExecutionDebug{
RawOutput: []byte("provider output"), RawOutput: []byte("provider output"),
@@ -217,13 +263,13 @@ func TestContractCopiesMutableValues(t *testing.T) {
execution.Debug.RawOutput[0] = 'x' execution.Debug.RawOutput[0] = 'x'
execution.Debug.ValidationDiagnostics[0] = "changed" execution.Debug.ValidationDiagnostics[0] = "changed"
if preparationCopy.InputHashes["data_package"] != "input-hash" { if preparationCopy.InputHashes["data_package"] != "input-hash" || preparationCopy.Output.RepairAttempts != 3 {
t.Fatalf("preparation copy = %#v", preparationCopy) t.Fatalf("preparation copy = %#v", preparationCopy)
} }
if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` { if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` {
t.Fatalf("preparation debug copy = %#v", debugCopy) t.Fatalf("preparation debug copy = %#v", debugCopy)
} }
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" { if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.RepairAttempts != 2 || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
t.Fatalf("execution copy = %#v", executionCopy) t.Fatalf("execution copy = %#v", executionCopy)
} }
} }

View File

@@ -9,14 +9,15 @@ import (
func dailyDefinition() Definition { func dailyDefinition() Definition {
return Definition{ return Definition{
ID: Daily, ID: Daily,
Name: "Daily Report", Name: "Daily Report",
PromptID: "weather.daily_generated_text", PromptID: "weather.daily_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "daily", TemplateID: "daily",
GeneratedTextSchemaID: "daily", GeneratedTextSchemaID: "daily",
ArtifactGroup: "daily", GeneratedTextRepairAttempts: 1,
OutputName: "daily.md", ArtifactGroup: "daily",
OutputName: "daily.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",

View File

@@ -27,20 +27,21 @@ const (
) )
type Definition struct { type Definition struct {
ID ID ID ID
Name string Name string
PromptID string PromptID string
PromptVersion string PromptVersion string
TemplateID string TemplateID string
GeneratedTextSchemaID string GeneratedTextSchemaID string
ArtifactGroup string GeneratedTextRepairAttempts int
OutputName string ArtifactGroup string
DistributorPathTemplates []string OutputName string
Modules []module.ConfigItem DistributorPathTemplates []string
Morning bool Modules []module.ConfigItem
Evening bool Morning bool
resolve func(ResolveRequest) (timeutil.Period, error) Evening bool
runIDDisambiguator func(Resolved) string resolve func(ResolveRequest) (timeutil.Period, error)
runIDDisambiguator func(Resolved) string
} }
func (r Resolved) OutputName() (string, error) { func (r Resolved) OutputName() (string, error) {

View File

@@ -11,14 +11,15 @@ const hourlyReportHours = 6
func hourlyDefinition() Definition { func hourlyDefinition() Definition {
return Definition{ return Definition{
ID: Hourly, ID: Hourly,
Name: "Hourly Report", Name: "Hourly Report",
PromptID: "weather.hourly_generated_text", PromptID: "weather.hourly_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "hourly", TemplateID: "hourly",
GeneratedTextSchemaID: "hourly", GeneratedTextSchemaID: "hourly",
ArtifactGroup: "hourly", GeneratedTextRepairAttempts: 1,
OutputName: "hourly.md", ArtifactGroup: "hourly",
OutputName: "hourly.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"hourly/index.md", "hourly/index.md",
}, },

View File

@@ -82,8 +82,8 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
} }
for _, definition := range definitions { for _, definition := range definitions {
if definition.PromptVersion != "2.0.0" { if definition.PromptVersion != "2.1.0" {
t.Fatalf("%s PromptVersion = %q, want 2.0.0", definition.ID, definition.PromptVersion) t.Fatalf("%s PromptVersion = %q, want 2.1.0", definition.ID, definition.PromptVersion)
} }
if definition.PromptID == "" { if definition.PromptID == "" {
t.Fatalf("%s PromptID is empty", definition.ID) t.Fatalf("%s PromptID is empty", definition.ID)
@@ -91,6 +91,9 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" { if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID) t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
} }
if definition.GeneratedTextRepairAttempts != 1 {
t.Fatalf("%s repair attempts = %d, want 1", definition.ID, definition.GeneratedTextRepairAttempts)
}
} }
if _, err := registry.Lookup(ID("three_day")); err == nil { if _, err := registry.Lookup(ID("three_day")); err == nil {

View File

@@ -7,14 +7,15 @@ import (
func todayDefinition() Definition { func todayDefinition() Definition {
return Definition{ return Definition{
ID: Today, ID: Today,
Name: "Today Report", Name: "Today Report",
PromptID: "weather.today_generated_text", PromptID: "weather.today_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "today", TemplateID: "today",
GeneratedTextSchemaID: "today", GeneratedTextSchemaID: "today",
ArtifactGroup: "today", GeneratedTextRepairAttempts: 1,
OutputName: "today.md", ArtifactGroup: "today",
OutputName: "today.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",

View File

@@ -7,14 +7,15 @@ import (
func tomorrowDefinition() Definition { func tomorrowDefinition() Definition {
return Definition{ return Definition{
ID: Tomorrow, ID: Tomorrow,
Name: "Tomorrow Report", Name: "Tomorrow Report",
PromptID: "weather.tomorrow_generated_text", PromptID: "weather.tomorrow_generated_text",
PromptVersion: "2.0.0", PromptVersion: "2.1.0",
TemplateID: "tomorrow", TemplateID: "tomorrow",
GeneratedTextSchemaID: "tomorrow", GeneratedTextSchemaID: "tomorrow",
ArtifactGroup: "tomorrow", GeneratedTextRepairAttempts: 1,
OutputName: "tomorrow.md", ArtifactGroup: "tomorrow",
OutputName: "tomorrow.md",
DistributorPathTemplates: []string{ DistributorPathTemplates: []string{
"daily/{valid_start_date}/{run_id}.md", "daily/{valid_start_date}/{run_id}.md",
"daily/{valid_start_date}/index.md", "daily/{valid_start_date}/index.md",