Compare commits
11 Commits
53aa0b0a55
...
0bafbcb21f
| Author | SHA1 | Date | |
|---|---|---|---|
| 0bafbcb21f | |||
| 7005688b80 | |||
| 4cd5f505df | |||
| b3b23fb381 | |||
| 0c9cd6d5fb | |||
| ce79ea92c5 | |||
| 20107b0dfd | |||
| 1b38f66240 | |||
| b92f83e49b | |||
| 24a8579cee | |||
| 515cdada04 |
16
docs/cli.md
16
docs/cli.md
@@ -96,7 +96,7 @@ period, prompt version, timezone, and status. Successful output has an absolute
|
|||||||
"command": "generate",
|
"command": "generate",
|
||||||
"reportId": "today",
|
"reportId": "today",
|
||||||
"promptId": "weather.today_generated_text",
|
"promptId": "weather.today_generated_text",
|
||||||
"promptVersion": "2.0.0",
|
"promptVersion": "2.1.0",
|
||||||
"runId": "20260529T120000.000000000Z_today",
|
"runId": "20260529T120000.000000000Z_today",
|
||||||
"status": "succeeded",
|
"status": "succeeded",
|
||||||
"timezone": "America/Chicago",
|
"timezone": "America/Chicago",
|
||||||
@@ -106,7 +106,10 @@ period, prompt version, timezone, and status. Successful output has an absolute
|
|||||||
|
|
||||||
When available, the summary also includes the effective `profileId`,
|
When available, the summary also includes the effective `profileId`,
|
||||||
`backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested
|
`backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested
|
||||||
`llmDebugPath`, and compact Distributor `notification` result. It does not
|
`repairAttempts`, `llmDebugPath`, and compact Distributor `notification`
|
||||||
|
result. `repairAttempts` is `0` when the initial output passed validation,
|
||||||
|
positive when PromptKit made corrective generation calls, and omitted when
|
||||||
|
validation did not complete. The summary does not
|
||||||
include historical or transient artifact paths such as metadata, prompt input,
|
include historical or transient artifact paths such as metadata, prompt input,
|
||||||
raw generated text, render context, or notification receipts.
|
raw generated text, render context, or notification receipts.
|
||||||
|
|
||||||
@@ -115,7 +118,8 @@ raw generated text, render context, or notification receipts.
|
|||||||
A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`,
|
A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`,
|
||||||
`total`, `succeeded`, `failed`, and a `reports` array. Each report item includes
|
`total`, `succeeded`, `failed`, and a `reports` array. Each report item includes
|
||||||
its identity, status, effective profile and model details when available,
|
its identity, status, effective profile and model details when available,
|
||||||
source warnings, validation status, and absolute `outputPath` after publication.
|
source warnings, validation status, repair-attempt count when validation
|
||||||
|
completed, and absolute `outputPath` after publication.
|
||||||
The top-level summary may also contain a batch `notification` object and
|
The top-level summary may also contain a batch `notification` object and
|
||||||
`error`. Batch status is `failed` if any report or the batch notification fails.
|
`error`. Batch status is `failed` if any report or the batch notification fails.
|
||||||
The `total`, `succeeded`, and `failed` counters describe report items only, so
|
The `total`, `succeeded`, and `failed` counters describe report items only, so
|
||||||
@@ -143,7 +147,8 @@ A comparison summary contains these fields in this order: `command`,
|
|||||||
successful `results[].reportPath` are absolute. `results` preserves the
|
successful `results[].reportPath` are absolute. `results` preserves the
|
||||||
supplied profile order and each item contains `position`, `profileId`, optional
|
supplied profile order and each item contains `position`, `profileId`, optional
|
||||||
`backendId`, `modelName`, `status`, optional `validationStatus`, optional
|
`backendId`, `modelName`, `status`, optional `validationStatus`, optional
|
||||||
`reportPath`, optional `llmDebugPath`, and optional safe `error`.
|
`repairAttempts`, optional `reportPath`, optional `llmDebugPath`, and optional
|
||||||
|
safe `error`. The repair-attempt semantics match the generate summary.
|
||||||
|
|
||||||
The comparison status is `succeeded` only when every selected profile succeeds
|
The comparison status is `succeeded` only when every selected profile succeeds
|
||||||
and the bundle is published. Individual profile failures still publish a
|
and the bundle is published. Individual profile failures still publish a
|
||||||
@@ -157,7 +162,8 @@ published Promptkit category; destination failures use `destination_<kind>`;
|
|||||||
and committed cleanup failures use `publication_cleanup` with a message that
|
and committed cleanup failures use `publication_cleanup` with a message that
|
||||||
states whether a complete prior bundle, partial remnants, or no prior bundle
|
states whether a complete prior bundle, partial remnants, or no prior bundle
|
||||||
remains, or that recovery state could not be inspected. It does not expose
|
remains, or that recovery state could not be inspected. It does not expose
|
||||||
provider diagnostics, filesystem causes, or recovery paths. See the
|
provider diagnostics, filesystem causes, or recovery paths. A provider HTTP
|
||||||
|
failure may include its numeric status in the safe message. See the
|
||||||
[comparison bundle contract](integrations/comparison-bundle.md) for durable
|
[comparison bundle contract](integrations/comparison-bundle.md) for durable
|
||||||
artifact fields and failure invariants.
|
artifact fields and failure invariants.
|
||||||
|
|
||||||
|
|||||||
@@ -203,16 +203,23 @@ for platform availability, security, and retention requirements.
|
|||||||
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
|
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
|
||||||
They are separate decisions. An explicit `profile` applies to every selected
|
They are separate decisions. An explicit `profile` applies to every selected
|
||||||
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
|
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
|
||||||
Tomorrow select `weather-balanced` through their exact `2.0.0` prompt
|
Tomorrow select `weather-balanced` through their exact `2.1.0` prompt
|
||||||
definitions.
|
definitions.
|
||||||
|
|
||||||
Promptkit resolves a selected profile definition from a test or embedding
|
Promptkit resolves a selected profile definition from a test or embedding
|
||||||
consumer's explicit in-memory profile, then the configured `profile_file` or
|
consumer's explicit in-memory profile, then the configured `profile_file` or
|
||||||
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
|
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
|
||||||
built-in catalog. Sources provide complete definitions; fields are never
|
built-in catalog. Weatherreporter's embedded `weather-*` definitions are small
|
||||||
merged. A matching malformed external profile fails rather than using the
|
aliases of Promptkit's maintained base profiles, so Promptkit also resolves
|
||||||
embedded definition. The [Promptkit integration guide](integrations/promptkit.md)
|
their inherited target and settings. A configured definition with the same ID
|
||||||
owns the catalog and precedence details.
|
as either a selected profile or an inherited base takes precedence. A matching
|
||||||
|
malformed external profile fails rather than using the embedded definition. The
|
||||||
|
[Promptkit integration guide](integrations/promptkit.md) owns the catalog and
|
||||||
|
precedence details.
|
||||||
|
|
||||||
|
An endpoint-only profile may intentionally have no backend identity. Profiles
|
||||||
|
that require a direct API key are rejected before collection, while Promptkit
|
||||||
|
resolves optional environment credential sources during execution.
|
||||||
|
|
||||||
To replace the default Hourly definition with a local OpenAI-compatible
|
To replace the default Hourly definition with a local OpenAI-compatible
|
||||||
endpoint, set `profile_file` to a copy of
|
endpoint, set `profile_file` to a copy of
|
||||||
|
|||||||
@@ -9,7 +9,7 @@ and retention belong to the [operations guide](../operations.md).
|
|||||||
## Version And Layout
|
## Version And Layout
|
||||||
|
|
||||||
The current and only supported manifest schema version is
|
The current and only supported manifest schema version is
|
||||||
`weatherreporter.comparison.v1`. A bundle directory contains exactly these
|
`weatherreporter.comparison.v2`. A bundle directory contains exactly these
|
||||||
regular, non-symlinked files:
|
regular, non-symlinked files:
|
||||||
|
|
||||||
```text
|
```text
|
||||||
@@ -51,7 +51,7 @@ this order:
|
|||||||
|
|
||||||
```text
|
```text
|
||||||
position, profileId, backendId, modelName, status, validationStatus,
|
position, profileId, backendId, modelName, status, validationStatus,
|
||||||
reportPath, error
|
repairAttempts, reportPath, error
|
||||||
```
|
```
|
||||||
|
|
||||||
`startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not
|
`startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not
|
||||||
@@ -66,12 +66,16 @@ contiguous from one, profile IDs are distinct and nonblank, and
|
|||||||
`succeeded + failed == total`.
|
`succeeded + failed == total`.
|
||||||
|
|
||||||
A successful result has `status: "succeeded"`, `validationStatus: "passed"`,
|
A successful result has `status: "succeeded"`, `validationStatus: "passed"`,
|
||||||
|
and a non-negative `repairAttempts` count,
|
||||||
a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for
|
a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for
|
||||||
its position, total, and logical profile ID, and no `error`. A failed result has
|
its position, total, and logical profile ID, and no `error`. A failed result has
|
||||||
`status: "failed"`, no `reportPath`, and an `error` object with nonblank
|
`status: "failed"`, no `reportPath`, and an `error` object with nonblank
|
||||||
`category` and `message`. Its validation status is absent, `failed`, or
|
`category` and `message`. Its validation status is absent, `failed`, or
|
||||||
`skipped`. Error messages are valid UTF-8 and no longer than 1,024 bytes.
|
`skipped`; it may also be `passed` when a WeatherReporter step after PromptKit
|
||||||
`backendId` and `validationStatus` are omitted when unavailable.
|
validation failed. Error messages are valid UTF-8 and no longer than 1,024 bytes.
|
||||||
|
`backendId` and `validationStatus` are omitted when unavailable. A failed
|
||||||
|
result with any completed validation status must retain its non-negative
|
||||||
|
`repairAttempts`; early operational failures omit both fields.
|
||||||
|
|
||||||
Every successful Markdown file is declared by exactly one successful result.
|
Every successful Markdown file is declared by exactly one successful result.
|
||||||
The directory contains no extra entries. Consumers can therefore verify the
|
The directory contains no extra entries. Consumers can therefore verify the
|
||||||
@@ -88,7 +92,9 @@ case-variant, or duplicate fields; multiple JSON values; extra entries;
|
|||||||
symlinks; and future or otherwise unsupported versions. Treat a bundle that
|
symlinks; and future or otherwise unsupported versions. Treat a bundle that
|
||||||
fails recognition as an ordinary directory, not as a compatible bundle.
|
fails recognition as an ordinary directory, not as a compatible bundle.
|
||||||
|
|
||||||
When replacing a recognized bundle, cancellation observed before the new
|
Only v2 is recognized as a replaceable bundle; v1 is unsupported and must be
|
||||||
|
moved or removed before a replacement at the same destination. When replacing
|
||||||
|
a recognized bundle, cancellation observed before the new
|
||||||
bundle is installed preserves the prior bundle rather than committing the
|
bundle is installed preserves the prior bundle rather than committing the
|
||||||
replacement.
|
replacement.
|
||||||
|
|
||||||
|
|||||||
@@ -1,10 +1,12 @@
|
|||||||
# Promptkit Integration
|
# Promptkit Integration
|
||||||
|
|
||||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.0.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.1.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
||||||
|
|
||||||
## Logical Profile Catalog
|
## Logical Profile Catalog
|
||||||
|
|
||||||
Prompt definitions select a stable Weatherreporter profile ID. The embedded definitions currently use Promptkit's `openrouter` backend:
|
Prompt definitions select a stable Weatherreporter profile ID. Each embedded
|
||||||
|
definition contains only its ID and one Promptkit base-profile reference; the
|
||||||
|
effective execution settings resolve from Promptkit's maintained catalog:
|
||||||
|
|
||||||
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
|
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
|
||||||
| --- | --- | --- | --- | --- | --- |
|
| --- | --- | --- | --- | --- | --- |
|
||||||
@@ -12,22 +14,42 @@ Prompt definitions select a stable Weatherreporter profile ID. The embedded defi
|
|||||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
|
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
|
||||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
|
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
|
||||||
|
|
||||||
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded profiles intentionally omit endpoints, credentials, temperature, `top_p`, and output-token limits.
|
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
|
||||||
|
profiles intentionally omit endpoints, credentials, and execution settings;
|
||||||
|
Promptkit owns inherited resolution and its provider-native defaults.
|
||||||
|
Promptkit's built-in `rakestrawhome-gemma-4-31b` is also available for ordinary
|
||||||
|
and comparison selection and reports the `rakestrawhome` backend without
|
||||||
|
Weatherreporter-specific configuration.
|
||||||
|
|
||||||
## Selection And Active Execution
|
## Selection And Active Execution
|
||||||
|
|
||||||
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank backend and model identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
|
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank model; an endpoint-only profile may intentionally have no backend identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
|
||||||
|
|
||||||
1. explicit in-memory profiles used by an embedding consumer or test;
|
1. explicit in-memory profiles used by an embedding consumer or test;
|
||||||
2. the configured `profile_file` or `profile_dir`;
|
2. the configured `profile_file` or `profile_dir`;
|
||||||
3. Weatherreporter's embedded fallback profiles; and
|
3. Weatherreporter's embedded fallback profiles; and
|
||||||
4. Promptkit's built-in catalog.
|
4. Promptkit's built-in catalog.
|
||||||
|
|
||||||
A source falls through only when the selected ID is absent. Each source supplies a complete definition, so profile fields are not merged. A malformed matching operator definition is an error and does not fall back.
|
A source falls through only when the selected ID is absent. Promptkit resolves a
|
||||||
|
derived profile's base with the same source precedence, so a configured base
|
||||||
|
can shadow a built-in base. A missing, cyclic, malformed, or incomplete
|
||||||
|
selected inheritance chain is an error and does not fall back.
|
||||||
|
|
||||||
Profiles that require a direct API key are unsupported; a profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. Active results retain the selected logical profile ID and resolved backend and model. Ordinary errors, summaries, logs, and outputs exclude endpoints, credentials, rendered messages, schemas, request bodies, response bodies, and complete parameter maps.
|
Profiles that require a direct API key are unsupported. Optional environment
|
||||||
|
credential sources are Promptkit runtime concerns and are not checked by
|
||||||
|
Weatherreporter during profile inspection. Active results retain the selected
|
||||||
|
logical profile ID and resolved backend and model. Ordinary errors, summaries,
|
||||||
|
logs, and outputs exclude endpoints, credentials, rendered messages, schemas,
|
||||||
|
request bodies, response bodies, and complete parameter maps.
|
||||||
|
|
||||||
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
|
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions. Ordinary generation errors disclose only the safe Weatherreporter category and optional HTTP status; provider code, type, and message are written only to the explicit secure failure-debug artifact.
|
||||||
|
|
||||||
|
Each embedded prompt permits one Promptkit-owned corrective generation after an
|
||||||
|
eligible failed or explicitly empty result. This is not an application retry:
|
||||||
|
Weatherreporter performs no provider retry, profile fallback, or request-level
|
||||||
|
output-contract override. Promptkit reports cumulative usage and the actual
|
||||||
|
number of corrective calls; repair exhaustion remains a completed validation
|
||||||
|
failure.
|
||||||
|
|
||||||
When capture is enabled, its preparation artifact projects a provider endpoint
|
When capture is enabled, its preparation artifact projects a provider endpoint
|
||||||
to its scheme and host and retains only reviewed execution settings. Provider
|
to its scheme and host and retains only reviewed execution settings. Provider
|
||||||
|
|||||||
@@ -7,7 +7,7 @@ is owned by the [CLI reference](../cli.md) and [operations guide](../operations.
|
|||||||
|
|
||||||
## Single-Report Flow
|
## Single-Report Flow
|
||||||
|
|
||||||
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. The resolved profile, backend, and model are carried in the active result.
|
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. Profile inspection requires a model, permits an empty backend identity for endpoint-only profiles, and leaves inherited resolution and optional credential sources to Promptkit. The resolved profile, backend, model, and actual repair count (once a completed execution exists) are carried in the active result; the configured repair budget remains part of the exact prompt contract.
|
||||||
|
|
||||||
The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run.
|
The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run.
|
||||||
|
|
||||||
|
|||||||
@@ -21,8 +21,9 @@ Before accepting generated JSON, the execution boundary reconciles the prepared
|
|||||||
report definition, inspected prompt hash and selected profile identity, the one
|
report definition, inspected prompt hash and selected profile identity, the one
|
||||||
preparation callback, and the completed Promptkit result. The callback and
|
preparation callback, and the completed Promptkit result. The callback and
|
||||||
completion must agree on prompt, profile, backend, model, and rendered/input
|
completion must agree on prompt, profile, backend, model, and rendered/input
|
||||||
hashes; the callback output and completed validation must name the prepared
|
hashes; the callback output must also carry the prepared report's configured
|
||||||
report's JSON Schema. A mismatch produces no rendered Markdown and leaves
|
repair budget, while the completed validation records the actual corrective
|
||||||
|
calls used within that budget. A mismatch produces no rendered Markdown and leaves
|
||||||
results with only the inspected safe identity.
|
results with only the inspected safe identity.
|
||||||
|
|
||||||
Single-report generation executes one prepared profile and publishes its
|
Single-report generation executes one prepared profile and publishes its
|
||||||
|
|||||||
@@ -2,9 +2,9 @@
|
|||||||
|
|
||||||
`internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter.
|
`internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter.
|
||||||
|
|
||||||
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, the embedded fallback catalog, and its built-in catalog; the adapter does not parse profile YAML, merge sources, or probe endpoints.
|
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, embedded profile aliases and their bases, and its built-in catalog; the adapter does not parse profile YAML, resolve inheritance, merge sources, inspect optional environment credentials, or probe endpoints.
|
||||||
|
|
||||||
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
|
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. An empty backend identity remains valid for an endpoint-only profile; a nonblank model is required. PromptKit's configured repair-call budget and the completed result's actual corrective-call count are retained, along with its cumulative provider usage and final candidate. Structured provider generation failures become project-owned redacted generation errors that retain only bounded details through explicit accessors. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
|
||||||
|
|
||||||
The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications.
|
The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications.
|
||||||
|
|
||||||
|
|||||||
@@ -122,7 +122,9 @@ The destination is preflighted before prompt inspection and collection, then
|
|||||||
rechecked immediately before an atomic publish. A missing or empty directory
|
rechecked immediately before an atomic publish. A missing or empty directory
|
||||||
is usable. A nonempty directory can be replaced only when `--replace` is given
|
is usable. A nonempty directory can be replaced only when `--replace` is given
|
||||||
and it is recognized as a current Weatherreporter comparison bundle; ordinary
|
and it is recognized as a current Weatherreporter comparison bundle; ordinary
|
||||||
directories, symlinks, and unsafe destinations are rejected. Cancellation and
|
directories, symlinks, and unsafe destinations are rejected. Existing v1
|
||||||
|
bundles are not recognized for replacement: move or remove them first.
|
||||||
|
Cancellation and
|
||||||
all failures before publication preserve an existing bundle, including a
|
all failures before publication preserve an existing bundle, including a
|
||||||
cancellation observed while a replacement is being prepared. If guarded
|
cancellation observed while a replacement is being prepared. If guarded
|
||||||
restoration cannot complete, the error names the retained sibling bundle for
|
restoration cannot complete, the error names the retained sibling bundle for
|
||||||
@@ -176,6 +178,12 @@ Preparation captures retain only the provider endpoint origin and reviewed
|
|||||||
execution settings. URL user information, paths, queries, fragments, and
|
execution settings. URL user information, paths, queries, fragments, and
|
||||||
unrecognized provider parameters are omitted.
|
unrecognized provider parameters are omitted.
|
||||||
|
|
||||||
|
Each run directory may contain `preparation.json` (v3), `execution.json` (v3),
|
||||||
|
and, for a provider generation failure, `failure.json` (v1). The failure
|
||||||
|
artifact retains the safe category, HTTP status, and provider code, type, and
|
||||||
|
message for trusted debugging only. Ordinary command output never includes
|
||||||
|
those provider details.
|
||||||
|
|
||||||
Capture writes are confined to the requested root and fail if an unsafe
|
Capture writes are confined to the requested root and fail if an unsafe
|
||||||
filesystem component prevents secure artifact creation.
|
filesystem component prevents secure artifact creation.
|
||||||
|
|
||||||
|
|||||||
@@ -50,10 +50,14 @@ directly.
|
|||||||
- Prompts receive curated module packages, never unbounded raw weather payloads.
|
- Prompts receive curated module packages, never unbounded raw weather payloads.
|
||||||
- Every execution validates the exact prompt version and output contract before
|
- Every execution validates the exact prompt version and output contract before
|
||||||
collection. The selected profile is configured explicitly or declared by the
|
collection. The selected profile is configured explicitly or declared by the
|
||||||
prompt; unsupported direct-key profiles and missing reported credentials fail
|
prompt; profiles requiring unsupported direct API keys fail before collection.
|
||||||
before collection.
|
A profile may have an empty backend identity when it supplies an endpoint;
|
||||||
|
PromptKit resolves inherited profiles and optional credential sources when it
|
||||||
|
executes them.
|
||||||
- Prompt and profile validation completes before weather collection. Raw output
|
- Prompt and profile validation completes before weather collection. Raw output
|
||||||
is validated before template rendering.
|
is validated before template rendering.
|
||||||
|
- PromptKit may make at most the prompt contract's one corrective generation;
|
||||||
|
exhaustion is a validation rejection, not an application-level retry.
|
||||||
- Comparison validates every explicit profile before collection, prepares one
|
- Comparison validates every explicit profile before collection, prepares one
|
||||||
immutable report input, and delegates backend capacity to Promptkit rather
|
immutable report input, and delegates backend capacity to Promptkit rather
|
||||||
than adding an application-wide execution limit.
|
than adding an application-wide execution limit.
|
||||||
@@ -62,6 +66,8 @@ directly.
|
|||||||
- Sensitive rendered prompts, schemas, input bodies, provider endpoints, and
|
- Sensitive rendered prompts, schemas, input bodies, provider endpoints, and
|
||||||
credentials never enter normal summaries or logs. They are written only to
|
credentials never enter normal summaries or logs. They are written only to
|
||||||
an explicit secure debug root when requested.
|
an explicit secure debug root when requested.
|
||||||
|
- Provider-controlled diagnostics never enter ordinary outputs; they are
|
||||||
|
retained only in explicit secure failure-debug artifacts.
|
||||||
|
|
||||||
## Output, Notification, And Testing Invariants
|
## Output, Notification, And Testing Invariants
|
||||||
|
|
||||||
|
|||||||
595
docs/roadmap/implementation.md
Normal file
595
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,595 @@
|
|||||||
|
# PromptKit v0.8.0 Upgrade Implementation Plan
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Completion note: Stages 1–9 upgraded PromptKit, adopted inherited profiles and
|
||||||
|
current credential handling, added repair and provider-failure contracts,
|
||||||
|
enabled one corrective generation for v2.1.0 prompts, exposed repair
|
||||||
|
provenance in ordinary and comparison results, migrated comparison bundles to
|
||||||
|
v2, and added secure failure-debug capture.
|
||||||
|
|
||||||
|
## Purpose And Authority
|
||||||
|
|
||||||
|
This plan translates the accepted
|
||||||
|
[PromptKit v0.8.0 upgrade roadmap](promptkit-v0.8.0.md) into an ordered,
|
||||||
|
decision-complete implementation procedure. The feature roadmap owns purpose,
|
||||||
|
scope, policy, and the desired end state. This document owns implementation
|
||||||
|
order, concrete work allocation, stage boundaries, and verification until the
|
||||||
|
upgrade is complete.
|
||||||
|
|
||||||
|
The implementing agent must complete the stages in numerical order. Each stage
|
||||||
|
is sized for one focused prompt handled by `gpt-5.6-terra` with high reasoning.
|
||||||
|
Do not combine stages merely because adjacent work touches the same package.
|
||||||
|
|
||||||
|
## Locked Decisions
|
||||||
|
|
||||||
|
The following decisions are final for this implementation:
|
||||||
|
|
||||||
|
- upgrade directly from PromptKit `v0.5.0` to `v0.8.0`;
|
||||||
|
- declare one corrective call in each embedded prompt through
|
||||||
|
`repair_attempts: 1` rather than adding WeatherReporter repair logic;
|
||||||
|
- keep the repair budget in the exact embedded prompt definition and add no
|
||||||
|
global, per-report, CLI, profile, or operator configuration override;
|
||||||
|
- advance all four exact prompt versions from `2.0.0` to `2.1.0`;
|
||||||
|
- make `weather-light`, `weather-balanced`, and `weather-deep` minimal aliases
|
||||||
|
of the corresponding PromptKit built-ins through `base_profile`;
|
||||||
|
- retain the existing report-to-profile assignments and effective model
|
||||||
|
ladder;
|
||||||
|
- allow successfully inspected endpoint-only profiles to have an empty backend
|
||||||
|
ID while continuing to require a nonblank model;
|
||||||
|
- treat `APIKeyEnv` as an optional lookup source and reject only profiles that
|
||||||
|
report `APIKeyRequired`, because WeatherReporter supplies no direct request
|
||||||
|
credential;
|
||||||
|
- support PromptKit's built-in `rakestrawhome-gemma-4-31b` profile without
|
||||||
|
WeatherReporter-specific backend configuration;
|
||||||
|
- expose provider HTTP status through a project-owned safe generation error,
|
||||||
|
while writing provider code, type, and message only to explicit secure debug
|
||||||
|
capture;
|
||||||
|
- emit only `weatherreporter.comparison.v2`, with repair provenance, and do not
|
||||||
|
preserve v1 guarded-replacement support; and
|
||||||
|
- retain PromptKit dependency types inside the adapter and preserve all
|
||||||
|
stateless execution, atomic publication, comparison independence, and
|
||||||
|
disclosure invariants.
|
||||||
|
|
||||||
|
## Implementation Rules
|
||||||
|
|
||||||
|
For every stage:
|
||||||
|
|
||||||
|
- read `docs/development.md`, all files under `docs/policy/`, this plan, the
|
||||||
|
feature roadmap, and the task-specific documents named by the stage;
|
||||||
|
- inspect the current code and tests before editing; use the repository's code
|
||||||
|
knowledge graph first for code discovery and fall back to text search for
|
||||||
|
literals, assets, and documentation;
|
||||||
|
- implement only the stage's scope and preserve unrelated user changes;
|
||||||
|
- keep PromptKit/provider types, client construction, YAML parsing, repair
|
||||||
|
mechanics, and provider transport inside the existing adapter boundary;
|
||||||
|
- use deterministic, offline, credential-free tests and injected clients or
|
||||||
|
synthetic fixtures rather than live OpenRouter, Rakestrawhome, or local
|
||||||
|
endpoint calls;
|
||||||
|
- add tests at the narrowest stable owner identified by the testing policy and
|
||||||
|
avoid copying PromptKit's internal test matrices;
|
||||||
|
- update the canonical documentation owners listed for that stage in the same
|
||||||
|
change as the implemented contract;
|
||||||
|
- run `gofmt` on changed Go files, the stage's focused tests,
|
||||||
|
`GOWORK=off go test -count=1 ./...`, and `git diff --check`; and
|
||||||
|
- leave the repository passing before proceeding to the next stage.
|
||||||
|
|
||||||
|
Stages affecting concurrent comparison, cancellation, or secure debug
|
||||||
|
filesystem work must also run the named focused packages with `-race`. Do not
|
||||||
|
weaken an existing assertion solely to accommodate the new dependency. When a
|
||||||
|
test encodes an intentionally changed contract, replace it with a behavioral
|
||||||
|
assertion for the accepted policy.
|
||||||
|
|
||||||
|
## Implementation Stages
|
||||||
|
|
||||||
|
### Stage 1: Upgrade The Dependency And Establish A v0.8.0 Baseline
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: move to the tagged dependency and isolate compatibility changes before
|
||||||
|
adopting new WeatherReporter behavior.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Update `go.mod` to require
|
||||||
|
`gitea.maximumdirect.net/eric/promptkit v0.8.0` and refresh `go.sum` with
|
||||||
|
`GOWORK=off go mod tidy`. Do not add `go.work`, `vendor`, or a `replace`
|
||||||
|
directive and do not change WeatherReporter's Go version.
|
||||||
|
2. Resolve any compile failures using PromptKit's public root package only.
|
||||||
|
Keep all `Profile` and `OpenAICompatibleProfileConfig` literals keyed. Do not
|
||||||
|
register the now-reserved `rakestrawhome` backend.
|
||||||
|
3. Reconcile adapter tests that directly exercise PromptKit's changed optional
|
||||||
|
credential behavior. A profile whose only credential metadata is
|
||||||
|
`api_key_env` must reach an injected client when the environment value is
|
||||||
|
absent; it must no longer expect PromptKit to return
|
||||||
|
`ErrAPIKeyEnvMissing`. Do not change WeatherReporter's application preflight
|
||||||
|
in this stage.
|
||||||
|
4. Verify that every current embedded prompt, content file, schema, and fallback
|
||||||
|
profile inspects under v0.8.0. Verify selected invalid local endpoints and
|
||||||
|
malformed selected profile definitions still map to project-owned
|
||||||
|
configuration or profile-load categories.
|
||||||
|
5. Review the v0.6.0 compatibility corrections against supported
|
||||||
|
WeatherReporter inputs: metadata-authoritative identities, exact contained
|
||||||
|
`content_file` paths, regular embedded files, structurally valid endpoints,
|
||||||
|
bounded JSON-compatible values, cancellation identity, and strict response
|
||||||
|
framing. Add consumer tests only for a WeatherReporter boundary not already
|
||||||
|
protected by PromptKit.
|
||||||
|
|
||||||
|
Do not enable profile inheritance or output repair yet. The expected result is
|
||||||
|
the current WeatherReporter feature set running against PromptKit v0.8.0.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptassets
|
||||||
|
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 2: Adopt Profile Inheritance And Current Credential Routing
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: adopt v0.7.0 profile composition, endpoint-only routing, optional
|
||||||
|
credential semantics, and the Rakestrawhome built-in without changing the model
|
||||||
|
ladder.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Replace the three embedded profile bodies with these exact leaf/base
|
||||||
|
relationships and no duplicated execution settings:
|
||||||
|
|
||||||
|
| Leaf | Base |
|
||||||
|
| --- | --- |
|
||||||
|
| `weather-light` | `deepseek-4-flash` |
|
||||||
|
| `weather-balanced` | `gemini-flash-latest` |
|
||||||
|
| `weather-deep` | `claude-sonnet-latest` |
|
||||||
|
|
||||||
|
2. Update prompt-asset fixtures and tests to understand `base_profile`. Assert
|
||||||
|
that all three leaf IDs remain selected identities and resolve to the same
|
||||||
|
backend, model, timeout, service tier, and reasoning settings exposed by the
|
||||||
|
current standalone definitions. Test relationships and effective behavior,
|
||||||
|
not copied private constants beyond the intentional model-ladder contract.
|
||||||
|
3. Preserve source precedence. Cover a standalone same-ID operator override, a
|
||||||
|
derived operator override, a configured source that shadows a base ID, and
|
||||||
|
selected missing-base, cyclic, malformed-base, and incomplete-target
|
||||||
|
failures. Do not implement inheritance or merging in WeatherReporter; all
|
||||||
|
resolution must remain PromptKit-owned.
|
||||||
|
4. Change application profile preflight to accept a successful inspection with
|
||||||
|
a nonblank model and an empty backend ID. Trust PromptKit inspection to have
|
||||||
|
resolved either a backend or endpoint; do not add the endpoint to
|
||||||
|
`promptexec.ProfileInspection` or ordinary provenance.
|
||||||
|
5. Remove application-level environment lookup and rejection for a nonblank
|
||||||
|
`APIKeyEnv`. Remove the now-unused `LookupEnv` fields and plumbing from
|
||||||
|
prompt, batch, and comparison inspection requests. Continue rejecting
|
||||||
|
`CredentialRequired`/`APIKeyRequired` before weather collection with the
|
||||||
|
existing missing-credential category.
|
||||||
|
6. Add an end-to-end offline regression proving the maintained endpoint-only
|
||||||
|
`weather-light` example passes application inspection, retains an empty
|
||||||
|
backend ID, and does not expose its endpoint.
|
||||||
|
7. Prove `rakestrawhome-gemma-4-31b` can pass ordinary and comparison profile
|
||||||
|
inspection through the existing adapter and reports the PromptKit
|
||||||
|
`rakestrawhome` backend ID. Do not make a provider call or add
|
||||||
|
Rakestrawhome-specific configuration.
|
||||||
|
|
||||||
|
Canonical documentation in this stage:
|
||||||
|
|
||||||
|
- update `docs/policy/architecture.md` so only direct-key-required profiles
|
||||||
|
fail credential preflight and backend identity is optional for endpoint-only
|
||||||
|
profiles;
|
||||||
|
- update `docs/config.md` to distinguish same-ID source replacement from
|
||||||
|
`base_profile` chain inheritance and to describe optional environment
|
||||||
|
credentials;
|
||||||
|
- update `docs/integrations/promptkit.md` for profile composition, parent
|
||||||
|
lookup precedence, endpoint-only identity, optional credentials, and
|
||||||
|
Rakestrawhome availability; and
|
||||||
|
- update `docs/internal/promptkit-adapter.md` and focused app internals for the
|
||||||
|
implemented inspection behavior.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/promptassets ./internal/adapters/promptkit ./internal/app
|
||||||
|
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 3: Extend The Project-Owned Prompt Execution Contract
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: establish dependency-neutral repair and structured-generation-error
|
||||||
|
values before the adapter or application relies on them.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Add `RepairAttempts int` to `promptexec.OutputContract`. It is the configured
|
||||||
|
additional-call budget from the exact prompt contract.
|
||||||
|
2. Add `RepairAttempts int` to `promptexec.Validation`. It is the number of
|
||||||
|
corrective calls actually started for the completed result. Update
|
||||||
|
`NewValidation` and every caller so construction is explicit; reject or
|
||||||
|
normalize no values here because PromptKit owns output-contract validity.
|
||||||
|
3. Update all copy helpers, equality/provenance helpers, fixtures, and tests so
|
||||||
|
repair values are retained without sharing mutable state.
|
||||||
|
4. Add a project-owned immutable `promptexec.GenerationError` with unexported
|
||||||
|
status and provider-detail fields plus safe accessors:
|
||||||
|
|
||||||
|
- `StatusCode() int`
|
||||||
|
- `ProviderCode() string`
|
||||||
|
- `ProviderType() string`
|
||||||
|
- `ProviderMessage() string`
|
||||||
|
- `Category() ErrorCategory`, always returning `Generation`
|
||||||
|
- `Error()`, exposing only the WeatherReporter generation category/message
|
||||||
|
and optional HTTP status
|
||||||
|
- `GoString()`, returning the same safe representation
|
||||||
|
- `Unwrap()`, preserving a project-owned `*promptexec.Error`
|
||||||
|
|
||||||
|
5. Provide one constructor used by adapters. Defensively normalize valid UTF-8
|
||||||
|
and bound code/type to 256 Unicode code points and message to 4,096 Unicode
|
||||||
|
code points, even though PromptKit already bounds its accessors. Do not
|
||||||
|
expose fields through struct formatting, JSON tags, or exported mutable
|
||||||
|
fields. Preserve the dependency cause only behind the project-owned error so
|
||||||
|
`errors.Is`/`errors.As` identities remain available without entering error
|
||||||
|
text.
|
||||||
|
6. Add focused tests proving nil/zero safety, category and unwrap behavior,
|
||||||
|
status-only ordinary formatting, `%#v` redaction, provider-detail bounds,
|
||||||
|
and repair-value copying.
|
||||||
|
|
||||||
|
Do not import PromptKit from `internal/promptexec` and do not change CLI or
|
||||||
|
artifact schemas in this stage.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/promptexec
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 4: Map PromptKit v0.8.0 Repair And Generation Errors In The Adapter
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: make the adapter faithfully translate v0.8.0 preparation, execution,
|
||||||
|
validation, usage, and failure values into the Stage 3 contract.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Map `promptkit.OutputContract.RepairAttempts` in prompt inspection and
|
||||||
|
prepared-execution details. Map
|
||||||
|
`promptkit.ValidationResult.RepairAttempts` in completed execution.
|
||||||
|
2. Preserve PromptKit's cumulative usage exactly as reported across the initial
|
||||||
|
call and every completed correction. Continue returning only the final raw
|
||||||
|
candidate and final validation result, subject to WeatherReporter's 64 KiB
|
||||||
|
generated-output bound.
|
||||||
|
3. In adapter error classification, retain cancellation, deadline, and capacity
|
||||||
|
precedence. Before the generic `ErrLLMGenerate` branch, use `errors.As` for
|
||||||
|
`*promptkit.GenerationError` and construct the project-owned
|
||||||
|
`promptexec.GenerationError` with status, code, type, message, and hidden
|
||||||
|
cause. Initial and corrective generation failures use the same mapping.
|
||||||
|
4. Extend the injected adapter client used by tests so it can return an ordered
|
||||||
|
sequence of responses or errors and record each request safely.
|
||||||
|
5. Use a synthetic PromptKit prompt with JSON Schema validation and
|
||||||
|
`repair_attempts: 1` to cover:
|
||||||
|
|
||||||
|
- first-pass valid output with zero corrections;
|
||||||
|
- explicitly empty or invalid output followed by valid corrected output;
|
||||||
|
- one-attempt exhaustion returning a final failed validation result rather
|
||||||
|
than an operational error;
|
||||||
|
- a non-2xx-style `GenerationError` during correction;
|
||||||
|
- cumulative token usage and actual repair count; and
|
||||||
|
- the same prepared prompt/profile identity across the corrective flow.
|
||||||
|
|
||||||
|
6. Keep these tests at the adapter boundary. Do not assert PromptKit's private
|
||||||
|
corrective-message wording or reconstruct its internal repair algorithm.
|
||||||
|
|
||||||
|
Canonical documentation in this stage: update
|
||||||
|
`docs/internal/promptkit-adapter.md` for the repair/result/error mappings. Do
|
||||||
|
not yet claim that embedded WeatherReporter prompts enable repair.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/adapters/promptkit ./internal/promptexec
|
||||||
|
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 5: Carry Repair Provenance Through Application Workflows
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: make application orchestration understand configured and actual repair
|
||||||
|
counts before changing the embedded prompt policy.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Add an expected generated-text repair budget to `report.Definition` and set
|
||||||
|
it explicitly to zero for all four current `2.0.0` definitions in this
|
||||||
|
stage. Include it in report-definition validation and retained contract
|
||||||
|
tests.
|
||||||
|
2. Extend exact prompt preflight so format, validation mode, schema path, and
|
||||||
|
repair budget must all match the resolved report definition. Extend
|
||||||
|
preparation and completion provenance checks to require the same repair
|
||||||
|
budget across inspection and the opaque prepared snapshot.
|
||||||
|
3. Add `RepairAttempts *int` to application outcomes where execution may fail
|
||||||
|
before validation exists. Set it to a fresh pointer immediately after a
|
||||||
|
non-nil completed execution is returned, before WeatherReporter's secondary
|
||||||
|
generated-text validation. A pointer is required so completed first-pass
|
||||||
|
zero is distinguishable from unavailable provenance.
|
||||||
|
4. Carry independent copies through `ReportResult`, `BatchReportResult`, batch
|
||||||
|
conversion, comparison execution's internal outcome, and relevant test
|
||||||
|
fakes. Do not expose the new value in CLI or comparison JSON yet.
|
||||||
|
5. Preserve the actual count on PromptKit validation rejection and on later
|
||||||
|
WeatherReporter generated-text or render failures. Leave it unavailable on
|
||||||
|
preparation, capacity, cancellation, deadline, and generation errors that
|
||||||
|
return no completed PromptKit result.
|
||||||
|
6. Add focused tests for provenance mismatch, completed zero, completed
|
||||||
|
positive, validation rejection, later local validation failure, early
|
||||||
|
operational failure, batch copying, and independent concurrent profile
|
||||||
|
outcomes.
|
||||||
|
|
||||||
|
Canonical documentation in this stage: update the focused prepared-report and
|
||||||
|
app-orchestration internals to describe configured versus actual repair
|
||||||
|
provenance. Current public documents should continue to report the embedded
|
||||||
|
budget as zero until Stage 6.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/report ./internal/app ./internal/cli
|
||||||
|
GOWORK=off go test -race -count=1 ./internal/app
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 6: Activate One Repair And Expose Ordinary Result Provenance
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: switch the operational prompts to the accepted one-correction policy
|
||||||
|
and make ordinary generate/run/batch output report what occurred.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Add `repair_attempts: 1` to the output contract of all four embedded prompt
|
||||||
|
definitions and change each exact prompt version from `2.0.0` to `2.1.0`.
|
||||||
|
Do not change prompt text or generated-text schemas solely for this upgrade.
|
||||||
|
2. Change all four report registry definitions to exact prompt version `2.1.0`
|
||||||
|
and expected repair budget one. Update exact-version fixtures and assertions
|
||||||
|
throughout adapter, app, CLI, report, and prompt-asset tests. Remove tests
|
||||||
|
that classify `repair_attempts` as a retired setting and replace them with
|
||||||
|
an exact one-attempt contract assertion.
|
||||||
|
3. Add `repairAttempts` to successful and failed generate and batch JSON result
|
||||||
|
shapes through the Stage 5 pointers. Emit integer zero for a completed
|
||||||
|
first-pass result, a positive integer for a completed repaired result, and
|
||||||
|
omit the field when no completed validation made it available.
|
||||||
|
4. Keep the existing `validationStatus` and failure categories authoritative.
|
||||||
|
A repaired valid result proceeds normally. Repair exhaustion remains
|
||||||
|
`validation_rejected`, publishes no report for that profile, and retains the
|
||||||
|
actual attempt count.
|
||||||
|
5. Add representative offline assembled tests proving first-pass success,
|
||||||
|
repaired success, exhaustion, explicit empty initial content, and batch
|
||||||
|
result propagation. Reuse the real PromptKit adapter with an injected
|
||||||
|
sequence client for at least one end-to-end repaired execution; use the
|
||||||
|
existing app fake at other boundaries where lower-level repair is already
|
||||||
|
covered.
|
||||||
|
6. Confirm no application loop, provider retry, profile fallback, or
|
||||||
|
request-level `OutputContract` override was introduced.
|
||||||
|
|
||||||
|
Canonical documentation in this stage:
|
||||||
|
|
||||||
|
- update `docs/policy/architecture.md` with PromptKit-owned bounded repair and
|
||||||
|
failed-exhaustion invariants;
|
||||||
|
- update `docs/integrations/promptkit.md` with exact prompt version `2.1.0`, one
|
||||||
|
configured repair, actual-count semantics, cumulative usage, explicit-empty
|
||||||
|
handling, and the distinction from operational retries;
|
||||||
|
- update `docs/cli.md` for generate and batch `repairAttempts` fields;
|
||||||
|
- update `docs/internal/report-registry.md`, prepared-report internals, and app
|
||||||
|
orchestration internals for exact version and repair flow; and
|
||||||
|
- keep configuration documentation unchanged because no repair setting is
|
||||||
|
added.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/promptassets ./internal/report ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||||
|
GOWORK=off go test -race -count=1 ./internal/adapters/promptkit ./internal/app
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 7: Migrate Comparison Bundles To v2
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: preserve repair activity in the profile-evaluation artifact and make
|
||||||
|
the strict durable schema change explicit.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Change `comparison.SchemaVersion` to
|
||||||
|
`weatherreporter.comparison.v2`. Emit and recognize v2 only; do not retain a
|
||||||
|
v1 parser or guarded-replacement compatibility path.
|
||||||
|
2. Add `RepairAttempts *int` to each application comparison profile result,
|
||||||
|
CLI comparison profile summary, and durable `comparison.Result`. Propagate a
|
||||||
|
fresh copy from Stage 5's execution outcome.
|
||||||
|
3. Place `repairAttempts` immediately after `validationStatus` in the canonical
|
||||||
|
result-object JSON field order. Encode zero for completed first-pass
|
||||||
|
validation, a positive integer for completed correction, and omit it only
|
||||||
|
when no completed validation exists.
|
||||||
|
4. Tighten manifest invariants: every non-nil repair count is non-negative; a
|
||||||
|
successful result must have `validationStatus: "passed"` and a non-nil
|
||||||
|
repair count; a failed result with a completed validation status must also
|
||||||
|
have a non-nil count; and an early operational failure may omit both.
|
||||||
|
5. Update the strict token-level JSON recognizer to accept only the canonical
|
||||||
|
`repairAttempts` field at its correct object level, reject duplicate,
|
||||||
|
unknown, negative, fractional, string, overflow, and malformed values, and
|
||||||
|
continue rejecting v1 as an unsupported current bundle.
|
||||||
|
6. Update manifest construction, cloning, validation, exact serialization
|
||||||
|
tests, guarded replacement tests, malicious bundle tests, partial-success
|
||||||
|
tests, and CLI comparison summaries. Preserve flat layout, result ordering,
|
||||||
|
hashes, atomic publication, cancellation safety, and no Distributor calls.
|
||||||
|
7. Cover concurrent peers where one succeeds first-pass, one repairs, one
|
||||||
|
exhausts, and one fails operationally. The counts must remain attached to
|
||||||
|
the selected profile positions without races or cross-contamination.
|
||||||
|
|
||||||
|
Canonical documentation in this stage:
|
||||||
|
|
||||||
|
- replace the v1 contract in `docs/integrations/comparison-bundle.md` with v2,
|
||||||
|
including exact field order, presence rules, and the lack of v1 replacement
|
||||||
|
compatibility;
|
||||||
|
- update `docs/cli.md` for comparison `repairAttempts`;
|
||||||
|
- update `docs/operations.md` to tell operators to move or remove an existing
|
||||||
|
v1 bundle before replacing at the same destination; and
|
||||||
|
- update comparison execution/publication internals and architecture policy as
|
||||||
|
needed for the current-only version invariant.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/comparison ./internal/app ./internal/cli
|
||||||
|
GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 8: Add Secure Provider-Failure Debug Capture
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: expose useful PromptKit v0.7.0 provider diagnostics only through the
|
||||||
|
existing explicit secure debug boundary while keeping ordinary errors safe.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Extend prompt preparation debug output with configured
|
||||||
|
`repairAttempts` and advance its schema identifier from
|
||||||
|
`weatherreporter.prompt_preparation_debug.v2` to
|
||||||
|
`weatherreporter.prompt_preparation_debug.v3`.
|
||||||
|
2. Extend execution validation debug output with actual `repairAttempts` and
|
||||||
|
advance its schema identifier from
|
||||||
|
`weatherreporter.prompt_execution_debug.v2` to
|
||||||
|
`weatherreporter.prompt_execution_debug.v3`. Retain cumulative token usage.
|
||||||
|
3. Add a dedicated `failure.json` artifact with schema identifier
|
||||||
|
`weatherreporter.prompt_failure_debug.v1`. Its canonical fields are:
|
||||||
|
|
||||||
|
- top level: `schemaVersion`, `reportId`, `validDate`, `runId`, `failure`;
|
||||||
|
- failure object: `category`, `statusCode`, `providerCode`, `providerType`,
|
||||||
|
`providerMessage`;
|
||||||
|
- omit absent provider fields and zero status; and
|
||||||
|
- never include the raw provider body, headers, endpoint, credentials,
|
||||||
|
request, schema, rendered prompt, or generated candidate.
|
||||||
|
|
||||||
|
4. Add `PromptDebugWriter.WriteFailure` using the existing handle-relative
|
||||||
|
secure run directory, `0700` directory and `0600` file modes, canonical JSON
|
||||||
|
encoding, and no-follow/atomic replacement behavior. Disabled writers must
|
||||||
|
perform no filesystem work.
|
||||||
|
5. When execution returns an error, use `errors.As` only against the
|
||||||
|
project-owned `*promptexec.GenerationError`. If explicit debug capture is
|
||||||
|
enabled, write `failure.json` using that profile's existing debug reference.
|
||||||
|
This applies equally to initial and corrective provider failures and keeps
|
||||||
|
comparison profile directories isolated.
|
||||||
|
6. If failure-debug writing also fails, retain the generation failure as the
|
||||||
|
primary categorized error and join the safe debug-write failure rather than
|
||||||
|
replacing or hiding the provider failure. Never place provider code, type,
|
||||||
|
or message in the joined error text.
|
||||||
|
7. Ordinary generate, batch, and comparison errors should gain only the safe
|
||||||
|
HTTP status already rendered by `promptexec.GenerationError.Error`; do not
|
||||||
|
add provider detail fields to CLI summaries, comparison manifests, logs, or
|
||||||
|
Distributor requests.
|
||||||
|
8. Add adversarial tests for formatter redaction, malicious provider strings,
|
||||||
|
JSON escaping, bounds, absent fields, file modes, symlink/path attacks,
|
||||||
|
write failure, cancellation identity, initial versus corrective failures,
|
||||||
|
and concurrent comparison captures.
|
||||||
|
|
||||||
|
Canonical documentation in this stage:
|
||||||
|
|
||||||
|
- update `docs/operations.md` with the three debug artifact versions,
|
||||||
|
`failure.json`, sensitivity, permissions, and retention;
|
||||||
|
- update `docs/integrations/promptkit.md` with ordinary status-only disclosure
|
||||||
|
and debug-only provider detail;
|
||||||
|
- update prompt-debug, PromptKit-adapter, and app-orchestration internals; and
|
||||||
|
- ensure `docs/policy/architecture.md` explicitly prohibits provider-controlled
|
||||||
|
diagnostics from ordinary outputs.
|
||||||
|
|
||||||
|
Focused verification:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
GOWORK=off go test -count=1 ./internal/promptexec ./internal/promptdebug ./internal/adapters/promptkit ./internal/app ./internal/cli
|
||||||
|
GOWORK=off go test -race -count=1 ./internal/promptdebug ./internal/adapters/promptkit ./internal/app
|
||||||
|
```
|
||||||
|
|
||||||
|
### Stage 9: Reconcile Documentation And Perform The Final Upgrade Audit
|
||||||
|
|
||||||
|
Status: Complete.
|
||||||
|
|
||||||
|
Purpose: verify the complete end state as one coherent WeatherReporter feature
|
||||||
|
and leave no stale v0.5.0, prompt v2.0.0, comparison v1, credential, profile,
|
||||||
|
repair, or debug claims.
|
||||||
|
|
||||||
|
Work:
|
||||||
|
|
||||||
|
1. Re-read the feature roadmap, `docs/development.md`, every policy document,
|
||||||
|
and every canonical document changed by Stages 1-8. Reconcile them against
|
||||||
|
executable behavior and remove duplicated or stale definitions. Keep
|
||||||
|
unimplemented future ideas in `docs/roadmap/future.md`, not current-state
|
||||||
|
documents.
|
||||||
|
2. Search code, embedded assets, examples, tests, and documentation for stale
|
||||||
|
contractual literals and review every occurrence of:
|
||||||
|
|
||||||
|
- PromptKit `v0.5.0`, `v0.6.0`, and `v0.7.0` as an active dependency claim;
|
||||||
|
- prompt version `2.0.0`;
|
||||||
|
- `weatherreporter.comparison.v1`;
|
||||||
|
- prompt debug schema v2 identifiers;
|
||||||
|
- claims that profile fields never inherit;
|
||||||
|
- claims that every `APIKeyEnv` must be populated;
|
||||||
|
- claims that backend ID is always required;
|
||||||
|
- claims that repair is disabled or `repair_attempts` is retired; and
|
||||||
|
- provider detail in ordinary output.
|
||||||
|
|
||||||
|
Historical release documents may retain accurate historical literals.
|
||||||
|
3. Verify canonical ownership:
|
||||||
|
|
||||||
|
- architecture owns invariants and boundaries;
|
||||||
|
- config owns operator profile and credential behavior, but no repair field;
|
||||||
|
- PromptKit integration owns logical prompt/profile/output contracts;
|
||||||
|
- CLI owns result fields;
|
||||||
|
- operations owns explicit debug handling and old comparison-bundle cleanup;
|
||||||
|
- comparison integration owns the complete v2 manifest; and
|
||||||
|
- internal documents own implementation flow without duplicating the public
|
||||||
|
references.
|
||||||
|
|
||||||
|
4. Verify maintained examples remain valid, secret-free, and tested. The local
|
||||||
|
`weather-light` example remains a standalone endpoint-only profile rather
|
||||||
|
than inheriting an OpenRouter backend it cannot clear.
|
||||||
|
5. Review the complete diff for architecture leakage. Production packages
|
||||||
|
outside `internal/adapters/promptkit` must not import PromptKit; no
|
||||||
|
application repair loop, provider client, raw provider diagnostic, profile
|
||||||
|
YAML parser, or durable application state may have appeared.
|
||||||
|
6. Review tests under the testing policy. Keep consumer contract and regression
|
||||||
|
coverage, remove accidental duplication of upstream implementation tests,
|
||||||
|
and ensure every default test is offline and repeatable.
|
||||||
|
7. Run the complete validation set:
|
||||||
|
|
||||||
|
```sh
|
||||||
|
gofmt -w <all changed Go files>
|
||||||
|
GOWORK=off go test -count=1 ./...
|
||||||
|
GOWORK=off go test -race -count=1 ./...
|
||||||
|
GOWORK=off go vet ./...
|
||||||
|
GOWORK=off go build ./...
|
||||||
|
GOWORK=off go mod tidy -diff
|
||||||
|
go run ./cmd/weatherreporter --help
|
||||||
|
go run ./cmd/weatherreporter compare --help
|
||||||
|
test -z "$(git ls-files go.work go.work.sum)"
|
||||||
|
test ! -e vendor
|
||||||
|
git diff --check
|
||||||
|
```
|
||||||
|
|
||||||
|
8. Confirm `go.mod` has no `replace`, the resolved PromptKit module is exactly
|
||||||
|
v0.8.0, and no live credential or provider call occurred during validation.
|
||||||
|
9. After every check passes, update this plan's status to Completed and add a
|
||||||
|
concise completion note listing the implemented stages. Do not delete either
|
||||||
|
roadmap until the maintainer has reviewed the implementation. Do not create
|
||||||
|
a release document or tag; release preparation remains a separate maintainer
|
||||||
|
action once a version is selected.
|
||||||
|
|
||||||
|
## Completion Standard
|
||||||
|
|
||||||
|
The implementation is complete only when all nine stages pass their focused
|
||||||
|
and repository-wide checks, all locked decisions are observable in code and
|
||||||
|
canonical documentation, and the feature roadmap's completion criteria are
|
||||||
|
satisfied. Passing compilation alone is insufficient. The final state must
|
||||||
|
demonstrate repaired success, repair exhaustion, comparison provenance,
|
||||||
|
endpoint-only routing, optional credentials, inherited profiles, Rakestrawhome
|
||||||
|
inspection, safe ordinary provider failures, secure debug-only detail, and
|
||||||
|
unchanged publication and concurrency invariants.
|
||||||
488
docs/roadmap/promptkit-v0.8.0.md
Normal file
488
docs/roadmap/promptkit-v0.8.0.md
Normal file
@@ -0,0 +1,488 @@
|
|||||||
|
# PromptKit v0.8.0 Upgrade Roadmap
|
||||||
|
|
||||||
|
Status: Implemented.
|
||||||
|
|
||||||
|
## Purpose
|
||||||
|
|
||||||
|
WeatherReporter should upgrade its PromptKit dependency from `v0.5.0` to
|
||||||
|
`v0.8.0` and deliberately adopt the useful consumer-facing capabilities added
|
||||||
|
in `v0.6.0`, `v0.7.0`, and `v0.8.0`. The upgrade should improve output-contract
|
||||||
|
reliability, profile composition, local and alternate endpoint support, and
|
||||||
|
provider-failure diagnosis without moving PromptKit responsibilities into
|
||||||
|
WeatherReporter or weakening the application's stateless and security
|
||||||
|
boundaries.
|
||||||
|
|
||||||
|
This roadmap defines the intended scope, policy, and end state. The
|
||||||
|
[implementation plan](implementation.md) owns the procedure for reaching that
|
||||||
|
state.
|
||||||
|
|
||||||
|
## User Intent
|
||||||
|
|
||||||
|
The upgrade is intended to:
|
||||||
|
|
||||||
|
- use PromptKit's bounded output repair to recover from occasional malformed
|
||||||
|
structured weather prose;
|
||||||
|
- keep WeatherReporter's domain profile IDs stable while inheriting maintained
|
||||||
|
PromptKit model definitions;
|
||||||
|
- make PromptKit's additional built-in backend and profile available for
|
||||||
|
explicit generation and profile comparisons;
|
||||||
|
- support unauthenticated or optionally authenticated OpenAI-compatible
|
||||||
|
endpoints without inventing a WeatherReporter transport layer;
|
||||||
|
- make provider HTTP failures more actionable under an explicit
|
||||||
|
WeatherReporter disclosure policy; and
|
||||||
|
- receive PromptKit's intervening correctness, safety, cancellation, resource,
|
||||||
|
and efficiency improvements as part of one tested dependency upgrade.
|
||||||
|
|
||||||
|
The model ladder and report assignments do not change as part of this work:
|
||||||
|
Hourly continues to select `weather-light`; Daily, Today, and Tomorrow continue
|
||||||
|
to select `weather-balanced`; and `weather-deep` remains available for explicit
|
||||||
|
selection. This upgrade does not promote the new Rakestrawhome profile into
|
||||||
|
that default ladder.
|
||||||
|
|
||||||
|
## Current State
|
||||||
|
|
||||||
|
WeatherReporter currently depends on
|
||||||
|
`gitea.maximumdirect.net/eric/promptkit` at `v0.5.0`. The PromptKit adapter
|
||||||
|
supplies embedded prompts, JSON Schemas, and
|
||||||
|
application-fallback profiles, plus an optional configured profile source and
|
||||||
|
the conventional local backend.
|
||||||
|
|
||||||
|
The four generated-text prompts are exact version `2.0.0` JSON Schema prompts.
|
||||||
|
They omit `repair_attempts`, so execution is single-pass. The project-owned
|
||||||
|
`promptexec.OutputContract` and validation result also omit repair budgets and
|
||||||
|
actual repair counts.
|
||||||
|
|
||||||
|
The three embedded WeatherReporter profiles duplicate the effective fields of
|
||||||
|
these PromptKit built-ins:
|
||||||
|
|
||||||
|
| WeatherReporter profile | PromptKit built-in with the same target |
|
||||||
|
| --- | --- |
|
||||||
|
| `weather-light` | `deepseek-4-flash` |
|
||||||
|
| `weather-balanced` | `gemini-flash-latest` |
|
||||||
|
| `weather-deep` | `claude-sonnet-latest` |
|
||||||
|
|
||||||
|
WeatherReporter preflights any nonblank `api_key_env` as a required credential,
|
||||||
|
even though PromptKit v0.7.0 distinguishes an optional environment source from
|
||||||
|
an explicit `APIKeyRequired` target. Provider generation failures are reduced
|
||||||
|
to WeatherReporter's safe `generation` category; the PromptKit dependency error
|
||||||
|
is retained as a hidden cause, but its structured HTTP status and provider
|
||||||
|
diagnostics are not mapped into project-owned values.
|
||||||
|
|
||||||
|
The maintained `weather-light` local override is an endpoint-only profile, and
|
||||||
|
the configuration contract says endpoint-only profiles are supported. PromptKit
|
||||||
|
inspection correctly reports no backend ID for that form, but WeatherReporter
|
||||||
|
application preflight currently requires both a nonblank backend and model.
|
||||||
|
That mismatch prevents the documented example from reaching generation and
|
||||||
|
should be corrected as part of adopting the current PromptKit target contract.
|
||||||
|
|
||||||
|
## Upstream Release Assessment
|
||||||
|
|
||||||
|
### PromptKit v0.6.0
|
||||||
|
|
||||||
|
`v0.6.0` adds no public declarations, but it is a material compatibility and
|
||||||
|
safety release. It centralizes execution-setting, output-contract, endpoint,
|
||||||
|
and JSON-compatible-value validation; makes YAML metadata authoritative for
|
||||||
|
prompt and profile identity; hardens `content_file` containment and regular-file
|
||||||
|
requirements; validates OpenAI-compatible endpoints structurally; bounds JSON
|
||||||
|
trees and successful provider bodies; requires exactly one JSON value in
|
||||||
|
provider responses; preserves cancellation and transport error identities; and
|
||||||
|
reuses schema and rendered-artifact work within an operation.
|
||||||
|
|
||||||
|
WeatherReporter should receive these improvements directly from the dependency
|
||||||
|
and audit its own supported assets and configuration paths against the stricter
|
||||||
|
contracts. It should not duplicate PromptKit's internal validators or tests.
|
||||||
|
The existing embedded prompt paths, inline data-package input, generated-output
|
||||||
|
limit, and adapter boundary remain conceptually correct.
|
||||||
|
|
||||||
|
### PromptKit v0.7.0
|
||||||
|
|
||||||
|
`v0.7.0` adds four potentially useful consumer features:
|
||||||
|
|
||||||
|
- linear, cycle-safe profile inheritance through `base_profile` and
|
||||||
|
`Profile.BaseProfileID`;
|
||||||
|
- the built-in `rakestrawhome` backend and
|
||||||
|
`rakestrawhome-gemma-4-31b` profile;
|
||||||
|
- optional API-key environment sources, with `APIKeyRequired` reserved for an
|
||||||
|
explicit local credential requirement; and
|
||||||
|
- bounded structured `GenerationError` details for non-2xx responses from the
|
||||||
|
built-in OpenAI-compatible client.
|
||||||
|
|
||||||
|
WeatherReporter has no manual `rakestrawhome` registration and uses keyed
|
||||||
|
PromptKit profile literals, so the two source-compatibility hazards called out
|
||||||
|
by the release do not require migration shims. The profile, credential, and
|
||||||
|
error features do require deliberate application-policy choices described
|
||||||
|
below.
|
||||||
|
|
||||||
|
### PromptKit v0.8.0
|
||||||
|
|
||||||
|
`v0.8.0` activates the existing output-contract repair budget. A positive
|
||||||
|
`repair_attempts` value authorizes up to that many corrective model calls after
|
||||||
|
eligible `basic`, `json`, or `json_schema` validation failures. The supported
|
||||||
|
budget is zero through three. Repairs preserve the original rendered
|
||||||
|
conversation, effective target, session, structured-output contract, and
|
||||||
|
backend capacity policy. The final result reports cumulative token usage and
|
||||||
|
the number of corrective calls actually made.
|
||||||
|
|
||||||
|
Repair exhaustion is a completed generation with failed validation, not an
|
||||||
|
operational error. WeatherReporter's existing policy should continue to reject
|
||||||
|
that result and publish no report for that profile. Explicit empty provider
|
||||||
|
content now reaches output validation; for WeatherReporter's JSON Schema
|
||||||
|
prompts it is therefore eligible for repair rather than being misclassified as
|
||||||
|
a malformed provider envelope.
|
||||||
|
|
||||||
|
## Desired End State
|
||||||
|
|
||||||
|
WeatherReporter builds and tests against PromptKit `v0.8.0` with no workspace,
|
||||||
|
vendor, or module replacement dependency. Its public behavior remains
|
||||||
|
stateless, its PromptKit dependency types remain confined to the adapter, and
|
||||||
|
its ordinary summaries and logs remain safe.
|
||||||
|
|
||||||
|
The completed integration:
|
||||||
|
|
||||||
|
- benefits from the v0.6.0 safety and efficiency corrections;
|
||||||
|
- composes WeatherReporter domain profiles from PromptKit's maintained built-in
|
||||||
|
profiles while preserving WeatherReporter-owned leaf IDs and operator
|
||||||
|
override precedence;
|
||||||
|
- accepts a successfully inspected endpoint-only profile with a nonblank model
|
||||||
|
even though it has no logical backend ID;
|
||||||
|
- permits explicit use of PromptKit's Rakestrawhome profile without custom
|
||||||
|
backend wiring;
|
||||||
|
- applies an accepted bounded-repair policy to every operational structured
|
||||||
|
prompt;
|
||||||
|
- validates repair configuration during preflight and records actual repair
|
||||||
|
activity in project-owned result values;
|
||||||
|
- retains PromptKit's cumulative usage accounting in explicit debug output;
|
||||||
|
- distinguishes safe provider HTTP status from potentially sensitive provider
|
||||||
|
diagnostics; and
|
||||||
|
- documents the changed profile, credential, repair, comparison, debug, and
|
||||||
|
failure contracts in their canonical owners.
|
||||||
|
|
||||||
|
## Dependency And Compatibility Policy
|
||||||
|
|
||||||
|
The module requirement should move directly from `v0.5.0` to `v0.8.0`, followed
|
||||||
|
by a clean module tidy. WeatherReporter already requires Go 1.26 while PromptKit
|
||||||
|
`v0.8.0` requires Go 1.25.5, so no Go version change is needed for this upgrade.
|
||||||
|
|
||||||
|
Consumer validation must cover the paths called out by PromptKit v0.6.0:
|
||||||
|
|
||||||
|
- every embedded prompt, content file, schema, and fallback profile inspects
|
||||||
|
through PromptKit `v0.8.0`;
|
||||||
|
- configured single-file and directory profile sources retain their lazy,
|
||||||
|
metadata-authoritative identity and precedence behavior;
|
||||||
|
- malformed selected profiles and invalid local endpoints retain actionable
|
||||||
|
WeatherReporter categories;
|
||||||
|
- the inline YAML data package and prepared-execution path remain within the
|
||||||
|
new JSON and response bounds; and
|
||||||
|
- cancellation, deadline, and backend-capacity identities still cross the
|
||||||
|
adapter correctly.
|
||||||
|
|
||||||
|
PromptKit owns its 16 MiB successful transport-response bound and JSON framing.
|
||||||
|
WeatherReporter retains its stricter 64 KiB generated-text acceptance bound.
|
||||||
|
The consumer suite should protect that relationship without reproducing
|
||||||
|
PromptKit's lower-level transport matrix.
|
||||||
|
|
||||||
|
## Domain Profile Composition
|
||||||
|
|
||||||
|
The embedded profiles should become application-owned aliases:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
id: weather-light
|
||||||
|
base_profile: deepseek-4-flash
|
||||||
|
```
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
id: weather-balanced
|
||||||
|
base_profile: gemini-flash-latest
|
||||||
|
```
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
id: weather-deep
|
||||||
|
base_profile: claude-sonnet-latest
|
||||||
|
```
|
||||||
|
|
||||||
|
The effective backend, model, timeout, service tier, and reasoning settings
|
||||||
|
must initially remain identical to the current WeatherReporter definitions.
|
||||||
|
The selected leaf remains the durable logical profile identity even though its
|
||||||
|
effective target is inherited.
|
||||||
|
|
||||||
|
Profile source precedence remains:
|
||||||
|
|
||||||
|
1. explicit in-memory profiles used by tests or embedding consumers;
|
||||||
|
2. the configured `profile_file` or `profile_dir` source;
|
||||||
|
3. WeatherReporter's embedded fallback catalog; and
|
||||||
|
4. PromptKit's built-in catalog.
|
||||||
|
|
||||||
|
Sources still do not merge definitions of the same ID. Once a selected
|
||||||
|
definition names `base_profile`, however, each parent ID is resolved through
|
||||||
|
that same precedence order and the resulting linear chain is merged from root
|
||||||
|
to leaf according to PromptKit's inheritance contract. Documentation must make
|
||||||
|
that distinction explicit. A malformed leaf, missing or malformed base, cycle,
|
||||||
|
overlong chain, or incomplete resolved target fails profile inspection before
|
||||||
|
weather collection.
|
||||||
|
|
||||||
|
An operator may continue to replace `weather-light`, `weather-balanced`, or
|
||||||
|
`weather-deep` with a standalone definition. An operator may also define a
|
||||||
|
derived replacement. The maintained local endpoint example should remain
|
||||||
|
standalone because PromptKit profile inheritance has no clearing syntax: using
|
||||||
|
an OpenRouter base would retain its backend identity and capacity policy even
|
||||||
|
when the child replaces the endpoint.
|
||||||
|
|
||||||
|
WeatherReporter should treat the adapter's successful profile inspection as
|
||||||
|
authoritative that PromptKit resolved a usable route. A nonblank model remains
|
||||||
|
required, but backend ID is optional for an endpoint-only profile and should be
|
||||||
|
omitted from safe provenance where unavailable. WeatherReporter still must not
|
||||||
|
surface the endpoint outside explicit debug capture. This aligns application
|
||||||
|
preflight with PromptKit and with the existing CLI, comparison, and debug value
|
||||||
|
shapes, all of which already permit an absent backend identity.
|
||||||
|
|
||||||
|
## Rakestrawhome Availability
|
||||||
|
|
||||||
|
The reserved `rakestrawhome` backend and built-in
|
||||||
|
`rakestrawhome-gemma-4-31b` profile should be supported automatically through
|
||||||
|
ordinary PromptKit selection. Operators may choose that profile with the
|
||||||
|
existing global profile setting or as one entry in `compare`, and PromptKit's
|
||||||
|
backend capacity policy remains authoritative.
|
||||||
|
|
||||||
|
WeatherReporter should not register, wrap, or duplicate the backend or profile,
|
||||||
|
and should not add a Rakestrawhome-specific configuration field. Its canonical
|
||||||
|
PromptKit integration documentation should link to PromptKit for the current
|
||||||
|
built-in catalog and credential contract rather than copying volatile endpoint
|
||||||
|
or capacity values. Offline inspection coverage should prove that the built-in
|
||||||
|
profile crosses the WeatherReporter adapter with the expected logical backend
|
||||||
|
identity.
|
||||||
|
|
||||||
|
## Bounded Structured-Output Repair
|
||||||
|
|
||||||
|
The accepted repair budget belongs to the exact PromptKit output contract, not
|
||||||
|
to a WeatherReporter retry loop. PromptKit alone should construct corrective
|
||||||
|
messages, perform additional calls, enforce the budget, aggregate usage, and
|
||||||
|
coordinate backend capacity. WeatherReporter must not retry provider failures,
|
||||||
|
switch profiles, or layer another repair mechanism around `RunPrepared`.
|
||||||
|
|
||||||
|
All four embedded prompt definitions should declare `repair_attempts: 1`.
|
||||||
|
Because this changes prompt
|
||||||
|
execution behavior, latency, cost, hash, and provenance, each definition and
|
||||||
|
its report-registry binding should advance from exact version `2.0.0` to
|
||||||
|
`2.1.0`. Prompt text and generated-text schemas do not need to change solely
|
||||||
|
for this feature.
|
||||||
|
|
||||||
|
The embedded prompt definition is the per-report pipeline policy owner. This
|
||||||
|
upgrade should not add a global or per-report operator configuration field for
|
||||||
|
repair attempts and should not construct a request-level replacement output
|
||||||
|
contract. A future pipeline may select another budget only through a deliberate
|
||||||
|
prompt-definition and exact-version change.
|
||||||
|
|
||||||
|
The project-owned PromptKit boundary should retain:
|
||||||
|
|
||||||
|
- the configured repair budget in prompt inspection and preparation output
|
||||||
|
contracts;
|
||||||
|
- the number of corrective calls actually made in completed validation;
|
||||||
|
- cumulative PromptKit token usage across initial and corrective calls; and
|
||||||
|
- the final candidate and final validation result only, consistent with the
|
||||||
|
PromptKit contract.
|
||||||
|
|
||||||
|
Prompt inspection and preparation provenance must require the repair budget to
|
||||||
|
match the exact expected prompt definition just as they currently require the
|
||||||
|
format, validation mode, and schema path to match. A zero-attempt successful
|
||||||
|
result is normal when the first candidate passes. A repair-exhausted result
|
||||||
|
continues through WeatherReporter's ordinary `validation_rejected` failure
|
||||||
|
path, and an operational or generation failure during correction remains that
|
||||||
|
profile's ordinary operational failure.
|
||||||
|
|
||||||
|
For concurrent comparison, every profile should use the same prompt repair
|
||||||
|
budget. A corrective call remains part of that profile's one prepared
|
||||||
|
execution and uses PromptKit's existing backend capacity pool. One profile's
|
||||||
|
repair or failure must not cancel independent peers.
|
||||||
|
|
||||||
|
## Repair Observability And Comparison Contract
|
||||||
|
|
||||||
|
The actual repair count is safe operational provenance and should be visible
|
||||||
|
where WeatherReporter already reports completed validation. Generation, batch,
|
||||||
|
and comparison action summaries should expose it without exposing candidates,
|
||||||
|
schemas, or diagnostics. Explicit execution debug output should add it to the
|
||||||
|
validation object alongside PromptKit's already mapped cumulative usage.
|
||||||
|
|
||||||
|
Profile comparison needs this value in `comparison.json`: a successful result
|
||||||
|
that required correction is materially different from a first-pass success
|
||||||
|
when evaluating model reliability, latency, and cost. The manifest should
|
||||||
|
therefore advance to `weatherreporter.comparison.v2` and add a non-negative
|
||||||
|
`repairAttempts` field to each result. The field is zero when no corrective
|
||||||
|
call began, including ordinary first-pass success. A failure carries the count
|
||||||
|
when PromptKit returned a completed validation result; it is omitted only when
|
||||||
|
execution failed before a completed validation result made the value known.
|
||||||
|
|
||||||
|
The v2 manifest should remain flat, strict, deterministic, and atomically
|
||||||
|
published. WeatherReporter does not need to preserve v1 replacement
|
||||||
|
compatibility: comparison bundles are operator-owned development outputs, and
|
||||||
|
the current integration contract intentionally recognizes only its current
|
||||||
|
schema. The release notes and comparison documentation must call out the
|
||||||
|
version change so an operator can remove or relocate an older bundle before
|
||||||
|
using guarded replacement at the same destination.
|
||||||
|
|
||||||
|
## Credential Semantics
|
||||||
|
|
||||||
|
PromptKit v0.7.0 treats `APIKeyEnv` as an optional lookup source. If the
|
||||||
|
environment variable is absent or blank and no direct credential is supplied,
|
||||||
|
the built-in client omits `Authorization` and lets the endpoint respond.
|
||||||
|
`APIKeyRequired` is the distinct signal that a usable credential must be
|
||||||
|
provided locally.
|
||||||
|
|
||||||
|
WeatherReporter cannot supply PromptKit's request-scoped direct API-key value,
|
||||||
|
so a profile reporting `APIKeyRequired` remains unsupported and must fail
|
||||||
|
before weather collection. WeatherReporter should not require a nonblank value
|
||||||
|
for an optional `APIKeyEnv` during application preflight. The built-in client
|
||||||
|
should omit `Authorization` when that source is unavailable and let the
|
||||||
|
endpoint return any authentication failure through the ordinary structured
|
||||||
|
generation-error path.
|
||||||
|
|
||||||
|
## Structured Generation Failures
|
||||||
|
|
||||||
|
PromptKit v0.7.0's `GenerationError` can report a provider HTTP status plus
|
||||||
|
bounded provider code, type, and message. WeatherReporter should consume that
|
||||||
|
type only inside the PromptKit adapter and map any adopted fields into a
|
||||||
|
project-owned immutable error. PromptKit dependency types must not become app
|
||||||
|
or CLI contracts.
|
||||||
|
|
||||||
|
HTTP status is safe enough for ordinary diagnostics. Provider code, type, and
|
||||||
|
message remain untrusted and may contain request or schema fragments. They
|
||||||
|
must never enter ordinary errors, action summaries, comparison manifests,
|
||||||
|
logs, generated reports, or Distributor payloads. The full structured
|
||||||
|
diagnostic belongs only in an explicitly requested secure `--llm-debug-dir`
|
||||||
|
`failure.json` artifact. That artifact may contain PromptKit's normalized
|
||||||
|
bounded fields but never the raw provider body, headers, endpoint, credentials,
|
||||||
|
or reconstructed request.
|
||||||
|
|
||||||
|
Initial-call and corrective-call non-2xx responses should follow the same
|
||||||
|
mapping. Cancellation and deadline categories continue to take precedence over
|
||||||
|
provider classification where PromptKit preserves those identities.
|
||||||
|
|
||||||
|
## Testing Policy
|
||||||
|
|
||||||
|
The default suite must remain deterministic, offline, and credential-free.
|
||||||
|
Use injected PromptKit clients and synthetic embedded or temporary assets for
|
||||||
|
consumer behavior; do not call OpenRouter, Rakestrawhome, or a local endpoint.
|
||||||
|
|
||||||
|
Risk-based coverage should include:
|
||||||
|
|
||||||
|
- all embedded prompts and inherited domain profiles inspecting successfully
|
||||||
|
under PromptKit `v0.8.0`;
|
||||||
|
- unchanged effective targets and report-to-profile assignments after the
|
||||||
|
alias refactor;
|
||||||
|
- external standalone and derived profile precedence, plus selected missing,
|
||||||
|
cyclic, and malformed-base failures at the WeatherReporter boundary;
|
||||||
|
- end-to-end preflight and prepared execution through the maintained
|
||||||
|
endpoint-only `weather-light` override without exposing its endpoint;
|
||||||
|
- offline inspection of `rakestrawhome-gemma-4-31b`;
|
||||||
|
- a first-pass valid result with zero repairs;
|
||||||
|
- an invalid structured result repaired successfully within one corrective
|
||||||
|
call;
|
||||||
|
- one-attempt exhaustion returning failed validation and no published report;
|
||||||
|
- a corrective generation failure retaining its safe category and provider
|
||||||
|
status policy;
|
||||||
|
- cumulative usage and actual repair-count mapping;
|
||||||
|
- comparison peers remaining independent when one profile repairs, exhausts,
|
||||||
|
or fails;
|
||||||
|
- v2 comparison manifest validation and guarded replacement; and
|
||||||
|
- absent or blank optional `APIKeyEnv` values reaching the provider without an
|
||||||
|
`Authorization` header, while `APIKeyRequired` profiles fail preflight.
|
||||||
|
|
||||||
|
Do not reproduce PromptKit's internal matrices for path traversal, JSON tree
|
||||||
|
bounds, response framing, inheritance depth, repair prompt construction, or
|
||||||
|
provider-detail normalization. WeatherReporter tests should protect only its
|
||||||
|
adapter mappings, application policy, provenance, publication, and public
|
||||||
|
contracts. Run ordinary and race-enabled repository tests because the repaired
|
||||||
|
execution path participates in concurrent comparisons.
|
||||||
|
|
||||||
|
## Documentation And Release Impact
|
||||||
|
|
||||||
|
Implementation must update each canonical owner whose contract changes:
|
||||||
|
|
||||||
|
- `docs/policy/architecture.md` for prompt-execution, credential, diagnostic,
|
||||||
|
and comparison invariants;
|
||||||
|
- `docs/config.md` and the maintained local profile example for profile-source,
|
||||||
|
inheritance, and credential semantics;
|
||||||
|
- `docs/integrations/promptkit.md` for exact prompt versions, repair policy,
|
||||||
|
profile composition, Rakestrawhome availability, and safe errors;
|
||||||
|
- `docs/integrations/comparison-bundle.md` for the v2 manifest and repair count;
|
||||||
|
- `docs/cli.md` for repair-count fields in action summaries;
|
||||||
|
- `docs/operations.md` for changed failure behavior and any explicit provider
|
||||||
|
diagnostic capture;
|
||||||
|
- focused internal PromptKit adapter, app orchestration, prompt-debug, and
|
||||||
|
comparison documentation; and
|
||||||
|
- release notes for the dependency jump, prompt version change, possible
|
||||||
|
additional model call, credential behavior, profile inheritance, diagnostic
|
||||||
|
behavior, and comparison schema change.
|
||||||
|
|
||||||
|
Current-state documentation must not describe this behavior until the
|
||||||
|
implementation lands. PromptKit remains the canonical owner of its complete
|
||||||
|
built-in catalogs, YAML merge rules, transport limits, corrective-message
|
||||||
|
construction, and public Go API.
|
||||||
|
|
||||||
|
## Scope
|
||||||
|
|
||||||
|
The completed feature includes:
|
||||||
|
|
||||||
|
- the direct module upgrade and tidy dependency graph;
|
||||||
|
- a v0.6.0 compatibility audit of WeatherReporter's supported PromptKit paths;
|
||||||
|
- inherited WeatherReporter domain profile definitions with unchanged
|
||||||
|
effective targets;
|
||||||
|
- correction of application preflight so PromptKit endpoint-only profiles work
|
||||||
|
as documented while retaining a required model identity;
|
||||||
|
- ordinary access to the Rakestrawhome built-in profile;
|
||||||
|
- PromptKit's optional-credential policy, while direct-key-required profiles
|
||||||
|
remain unsupported;
|
||||||
|
- `repair_attempts` on all operational prompts and exact prompt-version bumps;
|
||||||
|
- project-owned repair budget, actual-attempt, usage, and provenance mappings;
|
||||||
|
- repair observability in action summaries, explicit debug output, and a v2
|
||||||
|
comparison manifest;
|
||||||
|
- safe provider HTTP status in ordinary errors and bounded provider detail only
|
||||||
|
in explicit secure debug capture;
|
||||||
|
- focused offline and race-enabled regression coverage; and
|
||||||
|
- canonical current-state and release documentation updated with the code.
|
||||||
|
|
||||||
|
## Non-Goals
|
||||||
|
|
||||||
|
This upgrade does not include:
|
||||||
|
|
||||||
|
- application-implemented repair prompts or provider transport;
|
||||||
|
- retries for HTTP, network, timeout, capacity, or other operational failures;
|
||||||
|
- automatic profile escalation, fallback, ranking, or resampling;
|
||||||
|
- changing the weather profile ladder, default report assignments, or concrete
|
||||||
|
model targets beyond inheriting their maintained PromptKit definitions;
|
||||||
|
- making Rakestrawhome a default or adding provider-specific configuration;
|
||||||
|
- live-provider tests or a permanent benchmark framework;
|
||||||
|
- exposing raw provider responses or sensitive diagnostics routinely;
|
||||||
|
- a general prompt-source or pipeline plugin system; or
|
||||||
|
- compatibility shims for PromptKit versions older than `v0.8.0`.
|
||||||
|
|
||||||
|
## Completion Criteria
|
||||||
|
|
||||||
|
The roadmap is complete when:
|
||||||
|
|
||||||
|
- `go.mod` and `go.sum` resolve PromptKit `v0.8.0` without a replacement,
|
||||||
|
workspace, or vendor tree;
|
||||||
|
- all PromptKit v0.6.0 compatibility points relevant to WeatherReporter have
|
||||||
|
been checked and valid supported inputs retain project-owned error identity;
|
||||||
|
- the three WeatherReporter profiles inherit the intended PromptKit built-ins,
|
||||||
|
retain their logical IDs, and inspect to the intended effective targets;
|
||||||
|
- external standalone and inherited overrides obey documented precedence and
|
||||||
|
failure behavior;
|
||||||
|
- the maintained endpoint-only local override passes application preflight,
|
||||||
|
retains an empty backend ID, and keeps its endpoint out of ordinary values;
|
||||||
|
- `rakestrawhome-gemma-4-31b` is selectable through ordinary generation and
|
||||||
|
comparison paths without WeatherReporter backend registration;
|
||||||
|
- every operational prompt has one bounded repair attempt at exact
|
||||||
|
version `2.1.0`;
|
||||||
|
- inspection, preparation, execution, debug, and comparison values accurately
|
||||||
|
preserve configured and actual repair counts;
|
||||||
|
- first-pass success, repaired success, repair exhaustion, repair generation
|
||||||
|
failure, and explicit empty content follow the documented outcomes;
|
||||||
|
- the v2 comparison bundle distinguishes first-pass and repaired results;
|
||||||
|
- credential preflight accepts absent optional environment credentials while
|
||||||
|
rejecting direct-key-required profiles, and provider-error disclosure does
|
||||||
|
not leak sensitive values;
|
||||||
|
- the default test suite is offline and deterministic, ordinary and race
|
||||||
|
validation pass, and no redundant upstream implementation suite is copied;
|
||||||
|
and
|
||||||
|
- every implemented contract is documented by its canonical current-state
|
||||||
|
owner and disclosed in the eventual release notes.
|
||||||
2
go.mod
2
go.mod
@@ -6,7 +6,7 @@ require gopkg.in/yaml.v3 v3.0.1
|
|||||||
|
|
||||||
require (
|
require (
|
||||||
gitea.maximumdirect.net/eric/distributor v0.5.0
|
gitea.maximumdirect.net/eric/distributor v0.5.0
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.5.0
|
gitea.maximumdirect.net/eric/promptkit v0.8.0
|
||||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||||
golang.org/x/sys v0.45.0
|
golang.org/x/sys v0.45.0
|
||||||
)
|
)
|
||||||
|
|||||||
4
go.sum
4
go.sum
@@ -1,7 +1,7 @@
|
|||||||
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
|
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
|
||||||
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
|
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
|
gitea.maximumdirect.net/eric/promptkit v0.8.0 h1:NGd9hDLu0UMxKbvittMrqM5Ua94eFb+kOE7UIir8l08=
|
||||||
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
gitea.maximumdirect.net/eric/promptkit v0.8.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||||
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
|
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
|
||||||
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
|
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
|
||||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
|
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
|
||||||
|
|||||||
@@ -153,6 +153,7 @@ func outputContract(value promptkit.OutputContract) promptexec.OutputContract {
|
|||||||
Format: string(value.Format),
|
Format: string(value.Format),
|
||||||
ValidationMode: string(value.ValidationMode),
|
ValidationMode: string(value.ValidationMode),
|
||||||
SchemaPath: value.SchemaPath,
|
SchemaPath: value.SchemaPath,
|
||||||
|
RepairAttempts: value.RepairAttempts,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -193,6 +194,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
|
|||||||
promptexec.ValidationStatus(value.Validation.Status),
|
promptexec.ValidationStatus(value.Validation.Status),
|
||||||
string(value.Validation.Mode),
|
string(value.Validation.Mode),
|
||||||
value.Validation.SchemaPath,
|
value.Validation.SchemaPath,
|
||||||
|
value.Validation.RepairAttempts,
|
||||||
value.Validation.Errors,
|
value.Validation.Errors,
|
||||||
)
|
)
|
||||||
rawOutput := []byte(nil)
|
rawOutput := []byte(nil)
|
||||||
@@ -203,6 +205,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
|
|||||||
promptexec.ValidationFailed,
|
promptexec.ValidationFailed,
|
||||||
string(value.Validation.Mode),
|
string(value.Validation.Mode),
|
||||||
value.Validation.SchemaPath,
|
value.Validation.SchemaPath,
|
||||||
|
value.Validation.RepairAttempts,
|
||||||
[]string{"generated output exceeds the configured size limit"},
|
[]string{"generated output exceeds the configured size limit"},
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
@@ -299,6 +302,16 @@ func classifyError(err error) error {
|
|||||||
if errors.As(err, &capacityError) {
|
if errors.As(err, &capacityError) {
|
||||||
return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err)
|
return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err)
|
||||||
}
|
}
|
||||||
|
var generationError *promptkit.GenerationError
|
||||||
|
if errors.As(err, &generationError) {
|
||||||
|
return promptexec.NewGenerationError(
|
||||||
|
generationError.StatusCode(),
|
||||||
|
generationError.ProviderCode(),
|
||||||
|
generationError.ProviderType(),
|
||||||
|
generationError.ProviderMessage(),
|
||||||
|
err,
|
||||||
|
)
|
||||||
|
}
|
||||||
switch {
|
switch {
|
||||||
case errors.Is(err, promptkit.ErrInvalidConfig):
|
case errors.Is(err, promptkit.ErrInvalidConfig):
|
||||||
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err)
|
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err)
|
||||||
|
|||||||
@@ -4,11 +4,15 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"reflect"
|
||||||
"strings"
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
|
"testing/fstest"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
promptkit "gitea.maximumdirect.net/eric/promptkit"
|
promptkit "gitea.maximumdirect.net/eric/promptkit"
|
||||||
@@ -20,12 +24,19 @@ type fakeClient struct {
|
|||||||
mu sync.Mutex
|
mu sync.Mutex
|
||||||
response *promptkit.GenerateResponse
|
response *promptkit.GenerateResponse
|
||||||
err error
|
err error
|
||||||
|
outcomes []generationOutcome
|
||||||
|
next int
|
||||||
calls int
|
calls int
|
||||||
requests []promptkit.GenerateRequest
|
requests []promptkit.GenerateRequest
|
||||||
block bool
|
block bool
|
||||||
started chan struct{}
|
started chan struct{}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
type generationOutcome struct {
|
||||||
|
response *promptkit.GenerateResponse
|
||||||
|
err error
|
||||||
|
}
|
||||||
|
|
||||||
type recordingReader struct {
|
type recordingReader struct {
|
||||||
ref promptkit.ArtifactRef
|
ref promptkit.ArtifactRef
|
||||||
}
|
}
|
||||||
@@ -49,6 +60,11 @@ func (client *fakeClient) Generate(ctx context.Context, request promptkit.Genera
|
|||||||
started := client.started
|
started := client.started
|
||||||
response := client.response
|
response := client.response
|
||||||
err := client.err
|
err := client.err
|
||||||
|
if client.next < len(client.outcomes) {
|
||||||
|
outcome := client.outcomes[client.next]
|
||||||
|
client.next++
|
||||||
|
response, err = outcome.response, outcome.err
|
||||||
|
}
|
||||||
client.mu.Unlock()
|
client.mu.Unlock()
|
||||||
if started != nil {
|
if started != nil {
|
||||||
started <- struct{}{}
|
started <- struct{}{}
|
||||||
@@ -102,13 +118,19 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
|
|||||||
return client.requests[0]
|
return client.requests[0]
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (client *fakeClient) allRequests() []promptkit.GenerateRequest {
|
||||||
|
client.mu.Lock()
|
||||||
|
defer client.mu.Unlock()
|
||||||
|
return append([]promptkit.GenerateRequest(nil), client.requests...)
|
||||||
|
}
|
||||||
|
|
||||||
func TestInspectPromptAndProfile(t *testing.T) {
|
func TestInspectPromptAndProfile(t *testing.T) {
|
||||||
adapter := newTestAdapter(t, &fakeClient{})
|
adapter := newTestAdapter(t, &fakeClient{})
|
||||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.0.0")
|
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.1.0")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("InspectPrompt() error = %v", err)
|
t.Fatalf("InspectPrompt() error = %v", err)
|
||||||
}
|
}
|
||||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != "weather-balanced" {
|
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != "weather-balanced" || inspection.Output.RepairAttempts != 1 {
|
||||||
t.Fatalf("inspection = %#v", inspection)
|
t.Fatalf("inspection = %#v", inspection)
|
||||||
}
|
}
|
||||||
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
|
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
|
||||||
@@ -175,15 +197,38 @@ model: file-light
|
|||||||
}
|
}
|
||||||
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
|
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
|
||||||
|
|
||||||
directory := testProfileDirectory(t, `id: weather-light
|
directory := testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
|
||||||
backend: local
|
backend: local
|
||||||
model: directory-light
|
model: directory-light
|
||||||
`)
|
`})
|
||||||
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
|
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("New(profile directory) error = %v", err)
|
t.Fatalf("New(profile directory) error = %v", err)
|
||||||
}
|
}
|
||||||
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
|
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
|
||||||
|
|
||||||
|
derived := writeProfileFile(t, `id: weather-light
|
||||||
|
base_profile: gemini-flash-latest
|
||||||
|
`)
|
||||||
|
derivedAdapter, err := New(Config{ProfileFile: derived})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New(derived profile) error = %v", err)
|
||||||
|
}
|
||||||
|
assertProfile(t, derivedAdapter, "weather-light", "openrouter", "~google/gemini-flash-latest")
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestConfiguredBaseProfileOverridesEmbeddedProfileTarget(t *testing.T) {
|
||||||
|
directory := testProfileDirectory(t, map[string]string{
|
||||||
|
"deepseek.yml": `id: deepseek-4-flash
|
||||||
|
backend: local
|
||||||
|
model: shadowed-deepseek
|
||||||
|
`,
|
||||||
|
})
|
||||||
|
adapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v", err)
|
||||||
|
}
|
||||||
|
assertProfile(t, adapter, "weather-light", promptkit.BackendLocal, "shadowed-deepseek")
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
|
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
|
||||||
@@ -194,19 +239,69 @@ func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T)
|
|||||||
assertProfile(t, adapter, "weather-light", "", "weather-local")
|
assertProfile(t, adapter, "weather-light", "", "weather-local")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestMaintainedWeatherLightLocalProfileExampleExecutesThroughProductionClient(t *testing.T) {
|
||||||
|
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
|
||||||
|
for _, test := range []struct {
|
||||||
|
name string
|
||||||
|
credentialSource string
|
||||||
|
}{
|
||||||
|
{name: "without credential source"},
|
||||||
|
{name: "with blank optional credential source", credentialSource: "\napi_key_env: WEATHERREPORTER_TEST_MISSING_KEY\n"},
|
||||||
|
} {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
var authorization string
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
|
||||||
|
authorization = request.Header.Get("Authorization")
|
||||||
|
writer.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":12,"completion_tokens":8,"total_tokens":20}}`, validResponse().Content)
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
example, err := os.ReadFile(filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
profile := strings.Replace(string(example), "http://127.0.0.1:11434/v1", server.URL+"/v1", 1) + test.credentialSource
|
||||||
|
adapter, err := New(Config{ProfileFile: writeProfileFile(t, profile)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v", err)
|
||||||
|
}
|
||||||
|
request := testExecuteRequest()
|
||||||
|
request.ProfileID = "weather-light"
|
||||||
|
var preparation promptexec.Preparation
|
||||||
|
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
|
||||||
|
preparation = value
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Execute() error = %v", err)
|
||||||
|
}
|
||||||
|
if authorization != "" {
|
||||||
|
t.Fatalf("Authorization header = %q, want absent", authorization)
|
||||||
|
}
|
||||||
|
if preparation.ProfileID != "weather-light" || preparation.BackendID != "" || preparation.ModelName != "weather-local" || preparation.Output.RepairAttempts != 1 {
|
||||||
|
t.Fatalf("preparation = %#v", preparation)
|
||||||
|
}
|
||||||
|
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" || result.Validation.Status != promptexec.ValidationPassed || result.Validation.RepairAttempts != 0 {
|
||||||
|
t.Fatalf("execution = %#v", result)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
|
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
|
||||||
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile
|
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: other-profile
|
||||||
backend: openrouter
|
backend: openrouter
|
||||||
model: other-model
|
model: other-model
|
||||||
`)})
|
`})})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("New(absent profile) error = %v", err)
|
t.Fatalf("New(absent profile) error = %v", err)
|
||||||
}
|
}
|
||||||
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||||
|
|
||||||
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light
|
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
|
||||||
backend: openrouter
|
backend: openrouter
|
||||||
`)})
|
`})})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("New(malformed profile) error = %v", err)
|
t.Fatalf("New(malformed profile) error = %v", err)
|
||||||
}
|
}
|
||||||
@@ -215,6 +310,78 @@ backend: openrouter
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestProfileResolutionReturnsConfiguredInheritanceFailures(t *testing.T) {
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
profile string
|
||||||
|
profiles map[string]string
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "missing base",
|
||||||
|
profile: "missing-base",
|
||||||
|
profiles: map[string]string{"missing.yml": `id: missing-base
|
||||||
|
base_profile: unavailable
|
||||||
|
`},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "cyclic bases",
|
||||||
|
profile: "first",
|
||||||
|
profiles: map[string]string{
|
||||||
|
"first.yml": `id: first
|
||||||
|
base_profile: second
|
||||||
|
`,
|
||||||
|
"second.yml": `id: second
|
||||||
|
base_profile: first
|
||||||
|
`,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "malformed base",
|
||||||
|
profile: "child",
|
||||||
|
profiles: map[string]string{
|
||||||
|
"child.yml": `id: child
|
||||||
|
base_profile: malformed
|
||||||
|
`,
|
||||||
|
"malformed.yml": `id: malformed
|
||||||
|
base_profile: [not-a-profile]
|
||||||
|
`,
|
||||||
|
},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "incomplete target",
|
||||||
|
profile: "incomplete",
|
||||||
|
profiles: map[string]string{"incomplete.yml": `id: incomplete
|
||||||
|
backend: openrouter
|
||||||
|
`},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
adapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, test.profiles)})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v", err)
|
||||||
|
}
|
||||||
|
if _, err := adapter.InspectProfile(context.Background(), test.profile); err == nil {
|
||||||
|
t.Fatal("InspectProfile() error = nil, want configured inheritance error")
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestRakestrawhomeBuiltInProfileInspectsOffline(t *testing.T) {
|
||||||
|
adapter, err := New(Config{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v", err)
|
||||||
|
}
|
||||||
|
profile, err := adapter.InspectProfile(context.Background(), "rakestrawhome-gemma-4-31b")
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectProfile() error = %v", err)
|
||||||
|
}
|
||||||
|
if profile.ProfileID != "rakestrawhome-gemma-4-31b" || profile.BackendID != "rakestrawhome" || profile.ModelName == "" {
|
||||||
|
t.Fatalf("profile = %#v", profile)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
|
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
|
||||||
adapter, err := New(Config{})
|
adapter, err := New(Config{})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -287,7 +454,7 @@ func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
|
|||||||
}
|
}
|
||||||
request := promptexec.ExecuteRequest{
|
request := promptexec.ExecuteRequest{
|
||||||
PromptID: "weather.hourly_generated_text",
|
PromptID: "weather.hourly_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
ProfileID: "weather-light",
|
ProfileID: "weather-light",
|
||||||
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
|
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
|
||||||
}
|
}
|
||||||
@@ -400,6 +567,114 @@ func TestExecuteReturnsCompletedValidationRejection(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteMapsCorrectiveGenerationResults(t *testing.T) {
|
||||||
|
valid := `{"summary":"valid"}`
|
||||||
|
invalid := `{"summary":42}`
|
||||||
|
tests := []struct {
|
||||||
|
name string
|
||||||
|
outcomes []generationOutcome
|
||||||
|
wantStatus promptexec.ValidationStatus
|
||||||
|
wantRepairs int
|
||||||
|
wantCalls int
|
||||||
|
wantRaw string
|
||||||
|
wantUsage promptexec.TokenUsage
|
||||||
|
}{
|
||||||
|
{
|
||||||
|
name: "first pass valid",
|
||||||
|
outcomes: []generationOutcome{{response: generationResponse(valid, 2, 3, 5)}},
|
||||||
|
wantStatus: promptexec.ValidationPassed,
|
||||||
|
wantRepairs: 0,
|
||||||
|
wantCalls: 1,
|
||||||
|
wantRaw: valid,
|
||||||
|
wantUsage: promptexec.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "empty output repaired",
|
||||||
|
outcomes: []generationOutcome{{response: generationResponse("", 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
|
||||||
|
wantStatus: promptexec.ValidationPassed,
|
||||||
|
wantRepairs: 1,
|
||||||
|
wantCalls: 2,
|
||||||
|
wantRaw: valid,
|
||||||
|
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "invalid output repaired",
|
||||||
|
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
|
||||||
|
wantStatus: promptexec.ValidationPassed,
|
||||||
|
wantRepairs: 1,
|
||||||
|
wantCalls: 2,
|
||||||
|
wantRaw: valid,
|
||||||
|
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
|
||||||
|
},
|
||||||
|
{
|
||||||
|
name: "repair budget exhausted",
|
||||||
|
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(invalid, 7, 11, 18)}},
|
||||||
|
wantStatus: promptexec.ValidationFailed,
|
||||||
|
wantRepairs: 1,
|
||||||
|
wantCalls: 2,
|
||||||
|
wantRaw: invalid,
|
||||||
|
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
|
||||||
|
},
|
||||||
|
}
|
||||||
|
for _, test := range tests {
|
||||||
|
t.Run(test.name, func(t *testing.T) {
|
||||||
|
client := &fakeClient{outcomes: test.outcomes}
|
||||||
|
adapter := newRepairAdapter(t, client, "https://repair.example/v1")
|
||||||
|
var preparation promptexec.Preparation
|
||||||
|
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
|
||||||
|
preparation = value
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("Execute() error = %v", err)
|
||||||
|
}
|
||||||
|
if preparation.Output.RepairAttempts != 1 || result == nil || result.Validation.Status != test.wantStatus || result.Validation.RepairAttempts != test.wantRepairs || string(result.RawOutput) != test.wantRaw || result.Usage != test.wantUsage {
|
||||||
|
t.Fatalf("preparation/result = %#v/%#v", preparation, result)
|
||||||
|
}
|
||||||
|
requests := client.allRequests()
|
||||||
|
if len(requests) != test.wantCalls {
|
||||||
|
t.Fatalf("provider requests = %d, want %d", len(requests), test.wantCalls)
|
||||||
|
}
|
||||||
|
if test.wantCalls == 2 && !reflect.DeepEqual(requests[0].Target, requests[1].Target) {
|
||||||
|
t.Fatalf("corrective target = %#v, want same prepared identity as %#v", requests[1].Target, requests[0].Target)
|
||||||
|
}
|
||||||
|
if result.ProfileID != preparation.ProfileID || result.BackendID != preparation.BackendID || result.ModelName != preparation.ModelName || result.PromptID != preparation.PromptID || result.PromptVersion != preparation.PromptVersion || result.PromptHash != preparation.PromptHash {
|
||||||
|
t.Fatalf("prepared/result identity = %#v/%#v", preparation, result)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteMapsCorrectiveGenerationError(t *testing.T) {
|
||||||
|
const providerBody = `{"error":{"code":"repair-code","type":"repair-type","message":"repair-message"}}`
|
||||||
|
calls := 0
|
||||||
|
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) {
|
||||||
|
calls++
|
||||||
|
if calls == 1 {
|
||||||
|
writer.Header().Set("Content-Type", "application/json")
|
||||||
|
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":2,"completion_tokens":3,"total_tokens":5}}`, `{"summary":42}`)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
writer.Header().Set("Content-Type", "application/json")
|
||||||
|
writer.WriteHeader(http.StatusUnprocessableEntity)
|
||||||
|
_, _ = writer.Write([]byte(providerBody))
|
||||||
|
}))
|
||||||
|
defer server.Close()
|
||||||
|
|
||||||
|
adapter := newRepairAdapter(t, nil, server.URL)
|
||||||
|
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), nil)
|
||||||
|
if result != nil || err == nil || calls != 2 {
|
||||||
|
t.Fatalf("result/error/calls = %#v/%v/%d", result, err, calls)
|
||||||
|
}
|
||||||
|
var generationError *promptexec.GenerationError
|
||||||
|
if !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusUnprocessableEntity || generationError.ProviderCode() != "repair-code" || generationError.ProviderType() != "repair-type" || generationError.ProviderMessage() != "repair-message" {
|
||||||
|
t.Fatalf("generation error = %#v", err)
|
||||||
|
}
|
||||||
|
if strings.Contains(err.Error(), "repair-message") || !errors.Is(err, promptkit.ErrLLMGenerate) {
|
||||||
|
t.Fatalf("generation error = %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) {
|
func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) {
|
||||||
client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}}
|
client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}}
|
||||||
adapter := newTestAdapter(t, client)
|
adapter := newTestAdapter(t, client)
|
||||||
@@ -507,12 +782,12 @@ func TestNewValidatesConfiguration(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func TestLocalBackendAndMissingCredentialBehavior(t *testing.T) {
|
func TestLocalBackendAndOptionalCredentialSourceBehavior(t *testing.T) {
|
||||||
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
|
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
|
||||||
profiles := testProfileDirectory(t, `id: local-profile
|
profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: local-profile
|
||||||
backend: local
|
backend: local
|
||||||
model: local-model
|
model: local-model
|
||||||
`)
|
`})
|
||||||
adapter, err := newAdapterForTest(Config{
|
adapter, err := newAdapterForTest(Config{
|
||||||
ProfileDirectory: profiles,
|
ProfileDirectory: profiles,
|
||||||
LocalEndpoint: "https://local.example/v1",
|
LocalEndpoint: "https://local.example/v1",
|
||||||
@@ -529,11 +804,11 @@ model: local-model
|
|||||||
t.Fatalf("capacity classification = %v", got)
|
t.Fatalf("capacity classification = %v", got)
|
||||||
}
|
}
|
||||||
|
|
||||||
credentialProfiles := testProfileDirectory(t, `id: credential-profile
|
credentialProfiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: credential-profile
|
||||||
endpoint: https://profile.example/v1
|
endpoint: https://profile.example/v1
|
||||||
model: test-model
|
model: test-model
|
||||||
api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
|
api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
|
||||||
`)
|
`})
|
||||||
client := &fakeClient{response: validResponse()}
|
client := &fakeClient{response: validResponse()}
|
||||||
credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client)
|
credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -546,8 +821,8 @@ api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
|
|||||||
request := testExecuteRequest()
|
request := testExecuteRequest()
|
||||||
request.ProfileID = "credential-profile"
|
request.ProfileID = "credential-profile"
|
||||||
result, err := credentialAdapter.Execute(context.Background(), request, nil)
|
result, err := credentialAdapter.Execute(context.Background(), request, nil)
|
||||||
if result != nil || promptexec.CategoryOf(err) != promptexec.MissingCredential || client.callCount() != 0 {
|
if err != nil || result == nil || client.callCount() != 1 {
|
||||||
t.Fatalf("credential result/category/calls = %#v/%q/%d", result, promptexec.CategoryOf(err), client.callCount())
|
t.Fatalf("credential result/error/calls = %#v/%v/%d", result, err, client.callCount())
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -568,14 +843,14 @@ func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, mo
|
|||||||
|
|
||||||
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
|
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
profiles := testProfileDirectory(t, `id: test-profile
|
profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: test-profile
|
||||||
endpoint: https://profile.example/v1
|
endpoint: https://profile.example/v1
|
||||||
model: test-model
|
model: test-model
|
||||||
temperature: 0.2
|
temperature: 0.2
|
||||||
max_tokens: 300
|
max_tokens: 300
|
||||||
top_p: 1
|
top_p: 1
|
||||||
timeout_seconds: 30
|
timeout_seconds: 30
|
||||||
`)
|
`})
|
||||||
options = append(options, promptkit.WithLLMClient(client))
|
options = append(options, promptkit.WithLLMClient(client))
|
||||||
adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...)
|
adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -584,13 +859,15 @@ timeout_seconds: 30
|
|||||||
return adapter
|
return adapter
|
||||||
}
|
}
|
||||||
|
|
||||||
func testProfileDirectory(t *testing.T, profile string) string {
|
func testProfileDirectory(t *testing.T, profiles map[string]string) string {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
profiles := t.TempDir()
|
directory := t.TempDir()
|
||||||
if err := os.WriteFile(filepath.Join(profiles, "profile.yml"), []byte(profile), 0o600); err != nil {
|
for name, profile := range profiles {
|
||||||
t.Fatalf("write profile: %v", err)
|
if err := os.WriteFile(filepath.Join(directory, name), []byte(profile), 0o600); err != nil {
|
||||||
|
t.Fatalf("write profile: %v", err)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
return profiles
|
return directory
|
||||||
}
|
}
|
||||||
|
|
||||||
func writeProfileFile(t *testing.T, profile string) string {
|
func writeProfileFile(t *testing.T, profile string) string {
|
||||||
@@ -605,7 +882,7 @@ func writeProfileFile(t *testing.T, profile string) string {
|
|||||||
func testExecuteRequest() promptexec.ExecuteRequest {
|
func testExecuteRequest() promptexec.ExecuteRequest {
|
||||||
return promptexec.ExecuteRequest{
|
return promptexec.ExecuteRequest{
|
||||||
PromptID: "weather.daily_generated_text",
|
PromptID: "weather.daily_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
ProfileID: "test-profile",
|
ProfileID: "test-profile",
|
||||||
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
|
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
|
||||||
}
|
}
|
||||||
@@ -618,6 +895,61 @@ func validResponse() *promptkit.GenerateResponse {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func generationResponse(content string, promptTokens int, completionTokens int, totalTokens int) *promptkit.GenerateResponse {
|
||||||
|
return &promptkit.GenerateResponse{
|
||||||
|
Content: content,
|
||||||
|
Usage: promptkit.TokenUsage{
|
||||||
|
PromptTokens: promptTokens,
|
||||||
|
CompletionTokens: completionTokens,
|
||||||
|
TotalTokens: totalTokens,
|
||||||
|
},
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func newRepairAdapter(t *testing.T, client promptkit.LLMClient, endpoint string) *Adapter {
|
||||||
|
t.Helper()
|
||||||
|
profiles := testProfileDirectory(t, map[string]string{"profile.yml": "id: repair-profile\nendpoint: " + endpoint + "\nmodel: repair-model\n"})
|
||||||
|
options := []promptkit.Option{
|
||||||
|
promptkit.WithPromptFS(fstest.MapFS{
|
||||||
|
"repair.yml": &fstest.MapFile{Data: []byte(`id: weather.repair
|
||||||
|
version: "1.0.0"
|
||||||
|
default_profile: repair-profile
|
||||||
|
inputs:
|
||||||
|
- name: data_package
|
||||||
|
required: true
|
||||||
|
content_type: application/yaml
|
||||||
|
messages:
|
||||||
|
- role: user
|
||||||
|
content: "{{input \"data_package\"}}"
|
||||||
|
output:
|
||||||
|
format: json
|
||||||
|
validation_mode: json_schema
|
||||||
|
schema_path: repair.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
`)}}, "."),
|
||||||
|
promptkit.WithSchemaFS(fstest.MapFS{
|
||||||
|
"repair.schema.json": &fstest.MapFile{Data: []byte(`{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"],"additionalProperties":false}`)},
|
||||||
|
}, "."),
|
||||||
|
}
|
||||||
|
if client != nil {
|
||||||
|
options = append(options, promptkit.WithLLMClient(client))
|
||||||
|
}
|
||||||
|
adapter, err := newAdapter(Config{ProfileDirectory: profiles}, options...)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("newAdapter() error = %v", err)
|
||||||
|
}
|
||||||
|
return adapter
|
||||||
|
}
|
||||||
|
|
||||||
|
func repairExecuteRequest() promptexec.ExecuteRequest {
|
||||||
|
return promptexec.ExecuteRequest{
|
||||||
|
PromptID: "weather.repair",
|
||||||
|
PromptVersion: "1.0.0",
|
||||||
|
ProfileID: "repair-profile",
|
||||||
|
DataPackage: []byte("report: repair\n"),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func hourlyValidResponse() *promptkit.GenerateResponse {
|
func hourlyValidResponse() *promptkit.GenerateResponse {
|
||||||
return &promptkit.GenerateResponse{
|
return &promptkit.GenerateResponse{
|
||||||
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,
|
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,
|
||||||
|
|||||||
@@ -89,6 +89,7 @@ type ReportResult struct {
|
|||||||
ModelName string
|
ModelName string
|
||||||
SourceWarnings []weatherdata.SourceWarning
|
SourceWarnings []weatherdata.SourceWarning
|
||||||
ValidationStatus promptexec.ValidationStatus
|
ValidationStatus promptexec.ValidationStatus
|
||||||
|
RepairAttempts *int
|
||||||
LLMDebugPath string
|
LLMDebugPath string
|
||||||
OutputPath string
|
OutputPath string
|
||||||
Notification *NotificationResult
|
Notification *NotificationResult
|
||||||
@@ -139,6 +140,7 @@ type BatchReportResult struct {
|
|||||||
ModelName string `json:"modelName,omitempty"`
|
ModelName string `json:"modelName,omitempty"`
|
||||||
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
||||||
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
|
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
|
||||||
|
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||||
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
||||||
OutputPath string `json:"outputPath,omitempty"`
|
OutputPath string `json:"outputPath,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -457,6 +459,9 @@ func copyBatchReportDetails(item *BatchReportResult, result *ReportResult) {
|
|||||||
item.Timezone = result.Timezone
|
item.Timezone = result.Timezone
|
||||||
item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
||||||
item.ValidationStatus = result.ValidationStatus
|
item.ValidationStatus = result.ValidationStatus
|
||||||
|
if result.RepairAttempts != nil {
|
||||||
|
item.RepairAttempts = repairAttemptsPointer(*result.RepairAttempts)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) {
|
func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) {
|
||||||
|
|||||||
@@ -3,7 +3,6 @@ package app
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"fmt"
|
"fmt"
|
||||||
"os"
|
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -60,6 +59,7 @@ type ComparisonProfileResult struct {
|
|||||||
ModelName string
|
ModelName string
|
||||||
Status string
|
Status string
|
||||||
ValidationStatus promptexec.ValidationStatus
|
ValidationStatus promptexec.ValidationStatus
|
||||||
|
RepairAttempts *int
|
||||||
ReportPath string
|
ReportPath string
|
||||||
LLMDebugPath string
|
LLMDebugPath string
|
||||||
Error *comparison.SafeError
|
Error *comparison.SafeError
|
||||||
@@ -121,7 +121,7 @@ func compareDetailed(ctx context.Context, req ComparisonRequest, publish compari
|
|||||||
}
|
}
|
||||||
defer func() { _ = debugWriter.Close() }()
|
defer func() { _ = debugWriter.Close() }()
|
||||||
inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{
|
inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{
|
||||||
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor, LookupEnv: os.LookupEnv,
|
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor,
|
||||||
})
|
})
|
||||||
result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash
|
result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -198,6 +198,9 @@ func copyComparisonOutcomes(result *ComparisonResult, outcomes []comparisonProfi
|
|||||||
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
|
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
|
||||||
Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error,
|
Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error,
|
||||||
}
|
}
|
||||||
|
if outcome.RepairAttempts != nil {
|
||||||
|
profile.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
|
||||||
|
}
|
||||||
if published && outcome.Status == comparison.StatusSucceeded {
|
if published && outcome.Status == comparison.StatusSucceeded {
|
||||||
profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath)
|
profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath)
|
||||||
}
|
}
|
||||||
@@ -227,6 +230,9 @@ func comparisonBundle(result *ComparisonResult, dataPackage []byte, outcomes []c
|
|||||||
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
|
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
|
||||||
Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error,
|
Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error,
|
||||||
}
|
}
|
||||||
|
if outcome.RepairAttempts != nil {
|
||||||
|
manifestResult.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
|
||||||
|
}
|
||||||
if outcome.Status == comparison.StatusSucceeded {
|
if outcome.Status == comparison.StatusSucceeded {
|
||||||
manifestResult.ReportPath = outcome.ReportPath
|
manifestResult.ReportPath = outcome.ReportPath
|
||||||
bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown})
|
bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown})
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ type comparisonProfileOutcome struct {
|
|||||||
ModelName string
|
ModelName string
|
||||||
Status string
|
Status string
|
||||||
ValidationStatus promptexec.ValidationStatus
|
ValidationStatus promptexec.ValidationStatus
|
||||||
|
RepairAttempts *int
|
||||||
ReportPath string
|
ReportPath string
|
||||||
Markdown []byte
|
Markdown []byte
|
||||||
LLMDebugPath string
|
LLMDebugPath string
|
||||||
@@ -108,6 +109,9 @@ func executeComparisonProfile(ctx context.Context, req comparisonExecutionReques
|
|||||||
})
|
})
|
||||||
outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName
|
outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName
|
||||||
outcome.ValidationStatus = execution.ValidationStatus
|
outcome.ValidationStatus = execution.ValidationStatus
|
||||||
|
if execution.RepairAttempts != nil {
|
||||||
|
outcome.RepairAttempts = repairAttemptsPointer(*execution.RepairAttempts)
|
||||||
|
}
|
||||||
outcome.LLMDebugPath = execution.LLMDebugPath
|
outcome.LLMDebugPath = execution.LLMDebugPath
|
||||||
if err != nil {
|
if err != nil {
|
||||||
outcome.canceled = cancellationError(err)
|
outcome.canceled = cancellationError(err)
|
||||||
@@ -169,9 +173,14 @@ func comparisonExecutionMessage(err error) string {
|
|||||||
if errors.Is(err, context.DeadlineExceeded) {
|
if errors.Is(err, context.DeadlineExceeded) {
|
||||||
return "profile execution deadline exceeded"
|
return "profile execution deadline exceeded"
|
||||||
}
|
}
|
||||||
|
operation := "profile execution"
|
||||||
var execution *profileExecutionError
|
var execution *profileExecutionError
|
||||||
if errors.As(err, &execution) {
|
if errors.As(err, &execution) {
|
||||||
return comparison.TruncateErrorMessage(execution.operation + " failed")
|
operation = execution.operation
|
||||||
}
|
}
|
||||||
return "profile execution failed"
|
var generation *promptexec.GenerationError
|
||||||
|
if errors.As(err, &generation) && generation.StatusCode() > 0 {
|
||||||
|
return comparison.TruncateErrorMessage(fmt.Sprintf("%s failed (HTTP %d)", operation, generation.StatusCode()))
|
||||||
|
}
|
||||||
|
return comparison.TruncateErrorMessage(operation + " failed")
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -4,9 +4,11 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"reflect"
|
"reflect"
|
||||||
|
"strings"
|
||||||
"sync"
|
"sync"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
@@ -72,6 +74,66 @@ func TestExecuteComparisonProfilesContinuesAfterProfileFailure(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecuteComparisonProfilesPreservesIndependentRepairOutcomes(t *testing.T) {
|
||||||
|
prepared, prompt := preparedDailyProfile(t)
|
||||||
|
profiles := comparisonProfiles(4)
|
||||||
|
executor := newBarrierExecutor(profiles)
|
||||||
|
executor.setValidation(profiles[0].ProfileID, promptexec.ValidationPassed, 0)
|
||||||
|
executor.setValidation(profiles[1].ProfileID, promptexec.ValidationPassed, 1)
|
||||||
|
executor.setValidation(profiles[2].ProfileID, promptexec.ValidationFailed, 1)
|
||||||
|
executor.setError(profiles[3].ProfileID, errors.New("provider failure"))
|
||||||
|
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
|
||||||
|
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", Executor: executor,
|
||||||
|
}, executor)
|
||||||
|
waitForProfileStarts(t, executor, profiles, results)
|
||||||
|
executor.releaseAll()
|
||||||
|
result := <-results
|
||||||
|
wantStatuses := []string{comparison.StatusSucceeded, comparison.StatusSucceeded, comparison.StatusFailed, comparison.StatusFailed}
|
||||||
|
wantValidations := []promptexec.ValidationStatus{promptexec.ValidationPassed, promptexec.ValidationPassed, promptexec.ValidationFailed, ""}
|
||||||
|
wantRepairs := []*int{intPointer(0), intPointer(1), intPointer(1), nil}
|
||||||
|
for index, outcome := range result.Outcomes {
|
||||||
|
if outcome.Status != wantStatuses[index] || outcome.ValidationStatus != wantValidations[index] || !reflect.DeepEqual(outcome.RepairAttempts, wantRepairs[index]) {
|
||||||
|
t.Fatalf("outcome[%d] = %#v, want status/validation/repairs %q/%q/%#v", index, outcome, wantStatuses[index], wantValidations[index], wantRepairs[index])
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestExecuteComparisonProfilesCapturesConcurrentProviderFailures(t *testing.T) {
|
||||||
|
prepared, prompt := preparedDailyProfile(t)
|
||||||
|
profiles := comparisonProfiles(2)
|
||||||
|
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
|
||||||
|
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
|
||||||
|
t.Skipf("secure prompt debug capture is unavailable: %v", err)
|
||||||
|
}
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptDebugWriter() error = %v", err)
|
||||||
|
}
|
||||||
|
markers := []string{"first-provider-private-marker", "second-provider-private-marker"}
|
||||||
|
statuses := []int{http.StatusTooManyRequests, http.StatusServiceUnavailable}
|
||||||
|
executor := newBarrierExecutor(profiles)
|
||||||
|
for index, profile := range profiles {
|
||||||
|
executor.setError(profile.ProfileID, promptexec.NewGenerationError(statuses[index], "provider_code", "provider_type", markers[index], nil))
|
||||||
|
}
|
||||||
|
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
|
||||||
|
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", DebugWriter: debugWriter, Executor: executor,
|
||||||
|
}, executor)
|
||||||
|
waitForProfileStarts(t, executor, profiles, results)
|
||||||
|
executor.releaseAll()
|
||||||
|
result := <-results
|
||||||
|
for index, outcome := range result.Outcomes {
|
||||||
|
if outcome.Status != comparison.StatusFailed || outcome.Error == nil || outcome.Error.Category != string(promptexec.Generation) || outcome.Error.Message != fmt.Sprintf("execute prompt failed (HTTP %d)", statuses[index]) || strings.Contains(outcome.Error.Message, markers[index]) || outcome.LLMDebugPath == "" {
|
||||||
|
t.Fatalf("outcome[%d] = %#v", index, outcome)
|
||||||
|
}
|
||||||
|
failure, readErr := os.ReadFile(filepath.Join(outcome.LLMDebugPath, "failure.json"))
|
||||||
|
if readErr != nil {
|
||||||
|
t.Fatal(readErr)
|
||||||
|
}
|
||||||
|
if !strings.Contains(string(failure), markers[index]) || strings.Contains(string(failure), markers[1-index]) {
|
||||||
|
t.Fatalf("failure[%d] = %s", index, failure)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) {
|
func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) {
|
||||||
prepared, prompt := preparedDailyProfile(t)
|
prepared, prompt := preparedDailyProfile(t)
|
||||||
profiles := comparisonProfiles(4)
|
profiles := comparisonProfiles(4)
|
||||||
@@ -139,6 +201,8 @@ type barrierExecutor struct {
|
|||||||
releases map[string]chan struct{}
|
releases map[string]chan struct{}
|
||||||
requests map[string]promptexec.ExecuteRequest
|
requests map[string]promptexec.ExecuteRequest
|
||||||
errors map[string]error
|
errors map[string]error
|
||||||
|
validations map[string]promptexec.ValidationStatus
|
||||||
|
repairAttempts map[string]int
|
||||||
profiles map[string]ComparisonProfileInspection
|
profiles map[string]ComparisonProfileInspection
|
||||||
inFlight int
|
inFlight int
|
||||||
maximum int
|
maximum int
|
||||||
@@ -153,7 +217,7 @@ func newBarrierExecutor(profiles []ComparisonProfileInspection) *barrierExecutor
|
|||||||
}
|
}
|
||||||
return &barrierExecutor{
|
return &barrierExecutor{
|
||||||
started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases,
|
started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases,
|
||||||
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, profiles: identities,
|
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, validations: map[string]promptexec.ValidationStatus{}, repairAttempts: map[string]int{}, profiles: identities,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -170,7 +234,8 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
|
|||||||
e.mu.Lock()
|
e.mu.Lock()
|
||||||
profile := e.profiles[req.ProfileID]
|
profile := e.profiles[req.ProfileID]
|
||||||
e.mu.Unlock()
|
e.mu.Unlock()
|
||||||
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
|
definition := generationDefinitionForPrompt(req.PromptID)
|
||||||
|
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
|
||||||
e.callbackFailures <- err
|
e.callbackFailures <- err
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -194,15 +259,20 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
|
|||||||
e.mu.Lock()
|
e.mu.Lock()
|
||||||
e.inFlight--
|
e.inFlight--
|
||||||
err := e.errors[req.ProfileID]
|
err := e.errors[req.ProfileID]
|
||||||
|
validationStatus := e.validations[req.ProfileID]
|
||||||
|
repairAttempts := e.repairAttempts[req.ProfileID]
|
||||||
e.mu.Unlock()
|
e.mu.Unlock()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
if validationStatus == "" {
|
||||||
|
validationStatus = promptexec.ValidationPassed
|
||||||
|
}
|
||||||
return &promptexec.Execution{
|
return &promptexec.Execution{
|
||||||
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash,
|
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash,
|
||||||
ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName,
|
ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName,
|
||||||
StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(),
|
StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(),
|
||||||
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil),
|
Validation: promptexec.NewValidation(validationStatus, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil),
|
||||||
}, nil
|
}, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -219,6 +289,13 @@ func (e *barrierExecutor) setError(profileID string, err error) {
|
|||||||
e.errors[profileID] = err
|
e.errors[profileID] = err
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (e *barrierExecutor) setValidation(profileID string, status promptexec.ValidationStatus, repairAttempts int) {
|
||||||
|
e.mu.Lock()
|
||||||
|
defer e.mu.Unlock()
|
||||||
|
e.validations[profileID] = status
|
||||||
|
e.repairAttempts[profileID] = repairAttempts
|
||||||
|
}
|
||||||
|
|
||||||
func (e *barrierExecutor) release(profileID string) {
|
func (e *barrierExecutor) release(profileID string) {
|
||||||
close(e.releases[profileID])
|
close(e.releases[profileID])
|
||||||
}
|
}
|
||||||
@@ -317,4 +394,8 @@ func bytesEqual(left, right []byte) bool {
|
|||||||
return reflect.DeepEqual(left, right)
|
return reflect.DeepEqual(left, right)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func intPointer(value int) *int {
|
||||||
|
return &value
|
||||||
|
}
|
||||||
|
|
||||||
var _ promptexec.Executor = (*barrierExecutor)(nil)
|
var _ promptexec.Executor = (*barrierExecutor)(nil)
|
||||||
|
|||||||
@@ -87,6 +87,40 @@ func TestCompareDetailedPublishesPartialBundleAndReturnsAggregateError(t *testin
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestCompareDetailedPublishesPostValidationProfileFailure(t *testing.T) {
|
||||||
|
bundle := generationBundle(t)
|
||||||
|
executor := &generationExecutor{complete: func(execution *promptexec.Execution) {
|
||||||
|
if execution.ProfileID == "weather-deep" {
|
||||||
|
execution.RawOutput = []byte(`{"summary":42}`)
|
||||||
|
}
|
||||||
|
}}
|
||||||
|
result, err := CompareDetailed(context.Background(), ComparisonRequest{
|
||||||
|
Config: comparisonConfig(), Report: ReportDaily, ProfileIDs: []string{"weather-light", "weather-deep"},
|
||||||
|
WorkingDir: t.TempDir(), Date: generationTime("2026-05-29T12:00:00-05:00"),
|
||||||
|
Clock: timeutil.FixedClock{Time: generationTime("2026-05-29T08:30:00-05:00")},
|
||||||
|
Collector: &generationCollector{bundle: &bundle}, Executor: executor,
|
||||||
|
})
|
||||||
|
if err == nil || result == nil || result.Succeeded != 1 || result.Failed != 1 || result.ManifestPath == "" {
|
||||||
|
t.Fatalf("CompareDetailed() result/error = %#v/%v", result, err)
|
||||||
|
}
|
||||||
|
failure := result.Results[1]
|
||||||
|
if failure.Status != comparison.StatusFailed || failure.ValidationStatus != promptexec.ValidationPassed || failure.RepairAttempts == nil || *failure.RepairAttempts != 0 {
|
||||||
|
t.Fatalf("post-validation failure = %#v", failure)
|
||||||
|
}
|
||||||
|
data, readErr := os.ReadFile(result.ManifestPath)
|
||||||
|
if readErr != nil {
|
||||||
|
t.Fatal(readErr)
|
||||||
|
}
|
||||||
|
var manifest comparison.Manifest
|
||||||
|
if decodeErr := json.Unmarshal(data, &manifest); decodeErr != nil {
|
||||||
|
t.Fatal(decodeErr)
|
||||||
|
}
|
||||||
|
manifestFailure := manifest.Results[1]
|
||||||
|
if manifestFailure.ValidationStatus != "passed" || manifestFailure.RepairAttempts == nil || *manifestFailure.RepairAttempts != 0 {
|
||||||
|
t.Fatalf("published post-validation failure = %#v", manifestFailure)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) {
|
func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) {
|
||||||
for _, test := range []struct {
|
for _, test := range []struct {
|
||||||
name string
|
name string
|
||||||
|
|||||||
@@ -4,6 +4,7 @@ import (
|
|||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
|
"net/http"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
"strings"
|
"strings"
|
||||||
@@ -68,6 +69,7 @@ type generationExecutor struct {
|
|||||||
beforeExecute func(promptexec.ExecuteRequest)
|
beforeExecute func(promptexec.ExecuteRequest)
|
||||||
cancelBeforeReturn context.CancelFunc
|
cancelBeforeReturn context.CancelFunc
|
||||||
validation promptexec.ValidationStatus
|
validation promptexec.ValidationStatus
|
||||||
|
repairAttempts int
|
||||||
validations map[string]promptexec.ValidationStatus
|
validations map[string]promptexec.ValidationStatus
|
||||||
rawOutput []byte
|
rawOutput []byte
|
||||||
waitForCancellation map[string]bool
|
waitForCancellation map[string]bool
|
||||||
@@ -88,7 +90,7 @@ func (e *generationExecutor) InspectPrompt(_ context.Context, id, version string
|
|||||||
return promptexec.PromptInspection{}, e.inspectErr
|
return promptexec.PromptInspection{}, e.inspectErr
|
||||||
}
|
}
|
||||||
definition := generationDefinitionForPrompt(id)
|
definition := generationDefinitionForPrompt(id)
|
||||||
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}}, nil
|
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}}, nil
|
||||||
}
|
}
|
||||||
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
||||||
generationExecutorMu.Lock()
|
generationExecutorMu.Lock()
|
||||||
@@ -112,7 +114,8 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
|
|||||||
calls = 1
|
calls = 1
|
||||||
}
|
}
|
||||||
for range calls {
|
for range calls {
|
||||||
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}
|
definition := generationDefinitionForPrompt(req.PromptID)
|
||||||
|
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}
|
||||||
if prepare != nil {
|
if prepare != nil {
|
||||||
prepare(&preparation)
|
prepare(&preparation)
|
||||||
}
|
}
|
||||||
@@ -128,6 +131,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
|
|||||||
profileErr := e.executeErrors[req.ProfileID]
|
profileErr := e.executeErrors[req.ProfileID]
|
||||||
executeErr := e.executeErr
|
executeErr := e.executeErr
|
||||||
status := e.validation
|
status := e.validation
|
||||||
|
repairAttempts := e.repairAttempts
|
||||||
if profileStatus, ok := e.validations[req.ProfileID]; ok {
|
if profileStatus, ok := e.validations[req.ProfileID]; ok {
|
||||||
status = profileStatus
|
status = profileStatus
|
||||||
}
|
}
|
||||||
@@ -162,7 +166,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
|
|||||||
if cancelBeforeReturn != nil {
|
if cancelBeforeReturn != nil {
|
||||||
cancelBeforeReturn()
|
cancelBeforeReturn()
|
||||||
}
|
}
|
||||||
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil)}
|
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil)}
|
||||||
if complete != nil {
|
if complete != nil {
|
||||||
complete(execution)
|
complete(execution)
|
||||||
}
|
}
|
||||||
@@ -546,6 +550,66 @@ func TestGenerateDetailedWritesRequestedPromptDebugArtifacts(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestGenerateDetailedCapturesProviderFailureOnlyInDebugArtifacts(t *testing.T) {
|
||||||
|
bundle := generationBundle(t)
|
||||||
|
debugRoot := t.TempDir()
|
||||||
|
const marker = "provider-private-generation-marker"
|
||||||
|
result, err := GenerateDetailed(context.Background(), GenerateRequest{
|
||||||
|
Config: generationConfig(), Report: ReportDaily,
|
||||||
|
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
|
||||||
|
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle},
|
||||||
|
Executor: &generationExecutor{executeErr: promptexec.NewGenerationError(http.StatusTooManyRequests, "rate_limit", "provider_error", marker, nil)},
|
||||||
|
})
|
||||||
|
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
|
||||||
|
t.Skipf("secure prompt debug capture is unavailable: %v", err)
|
||||||
|
}
|
||||||
|
if err == nil || result == nil || result.LLMDebugPath == "" || promptexec.CategoryOf(err) != promptexec.Generation || !strings.Contains(err.Error(), "HTTP 429") || strings.Contains(err.Error(), marker) || result.OutputPath != "" {
|
||||||
|
t.Fatalf("GenerateDetailed() result/error = %#v/%v", result, err)
|
||||||
|
}
|
||||||
|
for _, name := range []string{"preparation.json", "failure.json"} {
|
||||||
|
if _, statErr := os.Stat(filepath.Join(result.LLMDebugPath, name)); statErr != nil {
|
||||||
|
t.Fatalf("debug artifact %q: %v", name, statErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
data, readErr := os.ReadFile(filepath.Join(result.LLMDebugPath, "failure.json"))
|
||||||
|
if readErr != nil || !strings.Contains(string(data), marker) {
|
||||||
|
t.Fatalf("failure artifact = %q, error = %v", data, readErr)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestGenerateDetailedPreservesProviderFailureWhenFailureDebugWriteFails(t *testing.T) {
|
||||||
|
bundle := generationBundle(t)
|
||||||
|
debugRoot := t.TempDir()
|
||||||
|
const marker = "provider-private-write-failure-marker"
|
||||||
|
var setupErr error
|
||||||
|
executor := &generationExecutor{
|
||||||
|
executeErr: promptexec.NewGenerationError(http.StatusServiceUnavailable, "unavailable", "provider_error", marker, nil),
|
||||||
|
beforeExecute: func(promptexec.ExecuteRequest) {
|
||||||
|
setupErr = filepath.Walk(debugRoot, func(path string, info os.FileInfo, err error) error {
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if info.Name() == "preparation.json" {
|
||||||
|
return os.Mkdir(filepath.Join(filepath.Dir(path), "failure.json"), 0o700)
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
})
|
||||||
|
},
|
||||||
|
}
|
||||||
|
result, err := GenerateDetailed(context.Background(), GenerateRequest{
|
||||||
|
Config: generationConfig(), Report: ReportDaily,
|
||||||
|
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
|
||||||
|
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle}, Executor: executor,
|
||||||
|
})
|
||||||
|
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
|
||||||
|
t.Skipf("secure prompt debug capture is unavailable: %v", err)
|
||||||
|
}
|
||||||
|
var generationError *promptexec.GenerationError
|
||||||
|
if setupErr != nil || err == nil || result == nil || result.OutputPath != "" || promptexec.CategoryOf(err) != promptexec.Generation || !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusServiceUnavailable || !strings.Contains(err.Error(), "HTTP 503") || strings.Contains(err.Error(), marker) {
|
||||||
|
t.Fatalf("GenerateDetailed() setup/result/error = %v/%#v/%v", setupErr, result, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func generationConfig() config.Config {
|
func generationConfig() config.Config {
|
||||||
cfg := config.Defaults()
|
cfg := config.Defaults()
|
||||||
cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home"
|
cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home"
|
||||||
|
|||||||
@@ -2,6 +2,7 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"errors"
|
||||||
"fmt"
|
"fmt"
|
||||||
"reflect"
|
"reflect"
|
||||||
|
|
||||||
@@ -24,6 +25,7 @@ type profileExecutionOutcome struct {
|
|||||||
BackendID string
|
BackendID string
|
||||||
ModelName string
|
ModelName string
|
||||||
ValidationStatus promptexec.ValidationStatus
|
ValidationStatus promptexec.ValidationStatus
|
||||||
|
RepairAttempts *int
|
||||||
LLMDebugPath string
|
LLMDebugPath string
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -94,11 +96,24 @@ func executePreparedProfile(ctx context.Context, req profileExecutionRequest) (p
|
|||||||
if callbackFailed {
|
if callbackFailed {
|
||||||
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true}
|
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true}
|
||||||
}
|
}
|
||||||
|
if req.DebugWriter != nil && req.DebugWriter.Enabled() && req.DebugRef != nil {
|
||||||
|
var generationError *promptexec.GenerationError
|
||||||
|
if errors.As(err, &generationError) {
|
||||||
|
path, debugErr := req.DebugWriter.WriteFailure(*req.DebugRef, generationError)
|
||||||
|
if path != "" {
|
||||||
|
outcome.LLMDebugPath = path
|
||||||
|
}
|
||||||
|
if debugErr != nil {
|
||||||
|
err = errors.Join(err, promptDebugWriteError(debugErr))
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)}
|
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)}
|
||||||
}
|
}
|
||||||
if execution == nil {
|
if execution == nil {
|
||||||
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)}
|
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)}
|
||||||
}
|
}
|
||||||
|
outcome.RepairAttempts = repairAttemptsPointer(execution.Validation.RepairAttempts)
|
||||||
if preparationCount != 1 {
|
if preparationCount != 1 {
|
||||||
return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()}
|
return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()}
|
||||||
}
|
}
|
||||||
@@ -157,7 +172,7 @@ func validatePreparedExecutionRequest(req profileExecutionRequest) error {
|
|||||||
if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) {
|
if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) {
|
||||||
return promptProvenanceError()
|
return promptProvenanceError()
|
||||||
}
|
}
|
||||||
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.BackendID == "" || req.Profile.ModelName == "" {
|
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.ModelName == "" {
|
||||||
return promptProvenanceError()
|
return promptProvenanceError()
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
@@ -178,12 +193,19 @@ func validateExecutionProvenance(req profileExecutionRequest, preparation prompt
|
|||||||
if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash ||
|
if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash ||
|
||||||
execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) ||
|
execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) ||
|
||||||
execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName ||
|
execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName ||
|
||||||
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" {
|
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" ||
|
||||||
|
execution.Validation.RepairAttempts < 0 || execution.Validation.RepairAttempts > preparation.Output.RepairAttempts ||
|
||||||
|
preparation.Output.RepairAttempts != definition.GeneratedTextRepairAttempts {
|
||||||
return promptProvenanceError()
|
return promptProvenanceError()
|
||||||
}
|
}
|
||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func repairAttemptsPointer(value int) *int {
|
||||||
|
copy := value
|
||||||
|
return ©
|
||||||
|
}
|
||||||
|
|
||||||
func promptProvenanceError() error {
|
func promptProvenanceError() error {
|
||||||
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil)
|
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -26,7 +26,7 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("executePreparedProfile() error = %v", err)
|
t.Fatalf("executePreparedProfile() error = %v", err)
|
||||||
}
|
}
|
||||||
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
|
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 0 || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
|
||||||
t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls)
|
t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls)
|
||||||
}
|
}
|
||||||
if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) {
|
if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) {
|
||||||
@@ -34,6 +34,20 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestExecutePreparedProfileRetainsCompletedRepairAttemptsOnLaterFailure(t *testing.T) {
|
||||||
|
prepared, inspection := preparedDailyProfile(t)
|
||||||
|
prepared.resolved.Definition.GeneratedTextRepairAttempts = 1
|
||||||
|
executor := &generationExecutor{repairAttempts: 1, rawOutput: []byte(`{"summary":42}`), prepare: func(value *promptexec.Preparation) { value.Output.RepairAttempts = 1 }}
|
||||||
|
outcome, _, err := executePreparedProfile(context.Background(), profileExecutionRequest{
|
||||||
|
Prepared: prepared, Prompt: inspection,
|
||||||
|
Profile: promptexec.ProfileInspection{ProfileID: inspection.ProfileID, BackendID: inspection.BackendID, ModelName: inspection.ModelName},
|
||||||
|
Executor: executor,
|
||||||
|
})
|
||||||
|
if err == nil || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 1 {
|
||||||
|
t.Fatalf("outcome/error = %#v/%v", outcome, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) {
|
func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) {
|
||||||
prepared, inspection := preparedDailyProfile(t)
|
prepared, inspection := preparedDailyProfile(t)
|
||||||
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
|
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
|
||||||
|
|||||||
@@ -48,6 +48,9 @@ func generatePromptReport(ctx context.Context, req promptReportRequest) (*Report
|
|||||||
})
|
})
|
||||||
result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName
|
result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName
|
||||||
result.ValidationStatus = outcome.ValidationStatus
|
result.ValidationStatus = outcome.ValidationStatus
|
||||||
|
if outcome.RepairAttempts != nil {
|
||||||
|
result.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
|
||||||
|
}
|
||||||
result.LLMDebugPath = outcome.LLMDebugPath
|
result.LLMDebugPath = outcome.LLMDebugPath
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return result, generatedProfileExecutionError(req.Resolved, result.RunID, err)
|
return result, generatedProfileExecutionError(req.Resolved, result.RunID, err)
|
||||||
|
|||||||
@@ -2,7 +2,6 @@ package app
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"os"
|
|
||||||
"strings"
|
"strings"
|
||||||
|
|
||||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/comparison"
|
"gitea.maximumdirect.net/eric/weatherreporter/internal/comparison"
|
||||||
@@ -18,7 +17,6 @@ type PromptInspectionRequest struct {
|
|||||||
Resolved report.Resolved
|
Resolved report.Resolved
|
||||||
Executor promptexec.Executor
|
Executor promptexec.Executor
|
||||||
Promptkit config.PromptkitConfig
|
Promptkit config.PromptkitConfig
|
||||||
LookupEnv func(string) (string, bool)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// PromptInspectionResult contains only safe identity and provenance from a
|
// PromptInspectionResult contains only safe identity and provenance from a
|
||||||
@@ -39,7 +37,6 @@ type PromptExecutionsInspectionRequest struct {
|
|||||||
Resolved []report.Resolved
|
Resolved []report.Resolved
|
||||||
Executor promptexec.Executor
|
Executor promptexec.Executor
|
||||||
Promptkit config.PromptkitConfig
|
Promptkit config.PromptkitConfig
|
||||||
LookupEnv func(string) (string, bool)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ComparisonInspectionRequest contains the explicit profile selection for one
|
// ComparisonInspectionRequest contains the explicit profile selection for one
|
||||||
@@ -48,7 +45,6 @@ type ComparisonInspectionRequest struct {
|
|||||||
Resolved report.Resolved
|
Resolved report.Resolved
|
||||||
ProfileIDs []string
|
ProfileIDs []string
|
||||||
Executor promptexec.Executor
|
Executor promptexec.Executor
|
||||||
LookupEnv func(string) (string, bool)
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// ComparisonInspectionResult contains the safe, shared prompt identity and
|
// ComparisonInspectionResult contains the safe, shared prompt identity and
|
||||||
@@ -76,7 +72,6 @@ func InspectPromptExecution(ctx context.Context, req PromptInspectionRequest) (P
|
|||||||
Resolved: []report.Resolved{req.Resolved},
|
Resolved: []report.Resolved{req.Resolved},
|
||||||
Executor: req.Executor,
|
Executor: req.Executor,
|
||||||
Promptkit: req.Promptkit,
|
Promptkit: req.Promptkit,
|
||||||
LookupEnv: req.LookupEnv,
|
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return PromptInspectionResult{}, err
|
return PromptInspectionResult{}, err
|
||||||
@@ -111,7 +106,7 @@ func InspectPromptExecutions(ctx context.Context, req PromptExecutionsInspection
|
|||||||
}
|
}
|
||||||
profile, ok := profiles[profileID]
|
profile, ok := profiles[profileID]
|
||||||
if !ok {
|
if !ok {
|
||||||
profile, err = inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv)
|
profile, err = inspectPromptProfile(ctx, req.Executor, profileID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
@@ -154,7 +149,7 @@ func InspectComparisonExecution(ctx context.Context, req ComparisonInspectionReq
|
|||||||
handler: handler,
|
handler: handler,
|
||||||
}
|
}
|
||||||
for _, profileID := range req.ProfileIDs {
|
for _, profileID := range req.ProfileIDs {
|
||||||
profile, err := inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv)
|
profile, err := inspectPromptProfile(ctx, req.Executor, profileID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return result, comparisonInspectionError("comparison profile inspection failed", err)
|
return result, comparisonInspectionError("comparison profile inspection failed", err)
|
||||||
}
|
}
|
||||||
@@ -190,7 +185,7 @@ func inspectPromptContract(ctx context.Context, executor promptexec.Executor, de
|
|||||||
return inspection, nil
|
return inspection, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string, lookupEnv func(string) (string, bool)) (promptexec.ProfileInspection, error) {
|
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string) (promptexec.ProfileInspection, error) {
|
||||||
profile, err := executor.InspectProfile(ctx, profileID)
|
profile, err := executor.InspectProfile(ctx, profileID)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err)
|
return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err)
|
||||||
@@ -201,16 +196,7 @@ func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, pro
|
|||||||
if profile.CredentialRequired {
|
if profile.CredentialRequired {
|
||||||
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil)
|
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil)
|
||||||
}
|
}
|
||||||
if strings.TrimSpace(profile.APIKeyEnv) != "" {
|
if strings.TrimSpace(profile.ModelName) == "" {
|
||||||
if lookupEnv == nil {
|
|
||||||
lookupEnv = os.LookupEnv
|
|
||||||
}
|
|
||||||
value, present := lookupEnv(profile.APIKeyEnv)
|
|
||||||
if !present || strings.TrimSpace(value) == "" {
|
|
||||||
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile credential is unavailable", nil)
|
|
||||||
}
|
|
||||||
}
|
|
||||||
if strings.TrimSpace(profile.BackendID) == "" || strings.TrimSpace(profile.ModelName) == "" {
|
|
||||||
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil)
|
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil)
|
||||||
}
|
}
|
||||||
return profile, nil
|
return profile, nil
|
||||||
@@ -221,7 +207,7 @@ func validPromptInput(inputs []promptexec.InputDefinition) bool {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool {
|
func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool {
|
||||||
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json"
|
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" && output.RepairAttempts == definition.GeneratedTextRepairAttempts
|
||||||
}
|
}
|
||||||
|
|
||||||
func promptInspectionError(operation string, err error) error {
|
func promptInspectionError(operation string, err error) error {
|
||||||
|
|||||||
@@ -50,7 +50,6 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
|
|||||||
name string
|
name string
|
||||||
prompt promptexec.PromptInspection
|
prompt promptexec.PromptInspection
|
||||||
profile promptexec.ProfileInspection
|
profile promptexec.ProfileInspection
|
||||||
lookupEnv func(string) (string, bool)
|
|
||||||
wantCategory promptexec.ErrorCategory
|
wantCategory promptexec.ErrorCategory
|
||||||
}{
|
}{
|
||||||
{
|
{
|
||||||
@@ -81,9 +80,9 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
|
|||||||
wantCategory: promptexec.InvalidConfiguration,
|
wantCategory: promptexec.InvalidConfiguration,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
name: "missing profile backend",
|
name: "missing profile model",
|
||||||
prompt: basePrompt,
|
prompt: basePrompt,
|
||||||
profile: promptexec.ProfileInspection{ProfileID: "default-profile", ModelName: "model"},
|
profile: promptexec.ProfileInspection{ProfileID: "default-profile", BackendID: "backend"},
|
||||||
wantCategory: promptexec.InvalidConfiguration,
|
wantCategory: promptexec.InvalidConfiguration,
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -92,18 +91,11 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
|
|||||||
profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true},
|
profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true},
|
||||||
wantCategory: promptexec.MissingCredential,
|
wantCategory: promptexec.MissingCredential,
|
||||||
},
|
},
|
||||||
{
|
|
||||||
name: "missing environment credential",
|
|
||||||
prompt: basePrompt,
|
|
||||||
profile: promptexec.ProfileInspection{ProfileID: "default-profile", APIKeyEnv: "PROMPT_API_KEY"},
|
|
||||||
lookupEnv: func(string) (string, bool) { return "", false },
|
|
||||||
wantCategory: promptexec.MissingCredential,
|
|
||||||
},
|
|
||||||
}
|
}
|
||||||
for _, test := range tests {
|
for _, test := range tests {
|
||||||
t.Run(test.name, func(t *testing.T) {
|
t.Run(test.name, func(t *testing.T) {
|
||||||
executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}}
|
executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}}
|
||||||
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor, LookupEnv: test.lookupEnv})
|
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor})
|
||||||
if err == nil || promptexec.CategoryOf(err) != test.wantCategory {
|
if err == nil || promptexec.CategoryOf(err) != test.wantCategory {
|
||||||
t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory)
|
t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory)
|
||||||
}
|
}
|
||||||
@@ -265,13 +257,12 @@ func TestInspectComparisonExecutionStopsAtFirstProfileFailure(t *testing.T) {
|
|||||||
prompt: validPromptInspection(resolved.Definition),
|
prompt: validPromptInspection(resolved.Definition),
|
||||||
profiles: map[string]promptexec.ProfileInspection{
|
profiles: map[string]promptexec.ProfileInspection{
|
||||||
"weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"},
|
"weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"},
|
||||||
"missing-key": {ProfileID: "missing-key", APIKeyEnv: "PROMPT_API_KEY"},
|
"missing-key": {ProfileID: "missing-key", CredentialRequired: true},
|
||||||
"weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"},
|
"weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"},
|
||||||
},
|
},
|
||||||
}
|
}
|
||||||
result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{
|
result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{
|
||||||
Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor,
|
Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor,
|
||||||
LookupEnv: func(string) (string, bool) { return "", false },
|
|
||||||
})
|
})
|
||||||
if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential {
|
if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential {
|
||||||
t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err))
|
t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err))
|
||||||
@@ -359,7 +350,7 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
|
|||||||
return promptexec.PromptInspection{
|
return promptexec.PromptInspection{
|
||||||
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
|
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
|
||||||
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
|
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
|
||||||
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
|
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|||||||
@@ -2,8 +2,10 @@ package app_test
|
|||||||
|
|
||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
|
"fmt"
|
||||||
"os"
|
"os"
|
||||||
"path/filepath"
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"time"
|
"time"
|
||||||
|
|
||||||
@@ -14,14 +16,12 @@ import (
|
|||||||
)
|
)
|
||||||
|
|
||||||
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
|
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
|
||||||
lookupEnv := func(string) (string, bool) { return "test-key", true }
|
|
||||||
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
|
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||||
Resolved: resolvedPromptProfile(t, id),
|
Resolved: resolvedPromptProfile(t, id),
|
||||||
Executor: adapter,
|
Executor: adapter,
|
||||||
Promptkit: config.PromptkitConfig{Profile: profile},
|
Promptkit: config.PromptkitConfig{Profile: profile},
|
||||||
LookupEnv: lookupEnv,
|
|
||||||
})
|
})
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("InspectPromptExecution() error = %v", err)
|
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||||
@@ -50,6 +50,55 @@ model: local-weather
|
|||||||
inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather")
|
inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather")
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPromptInspectionAcceptsMaintainedEndpointOnlyProfile(t *testing.T) {
|
||||||
|
adapter, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: filepath.Join("..", "..", "examples", "weather-light-local-profile.yml")})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v", err)
|
||||||
|
}
|
||||||
|
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||||
|
Resolved: resolvedPromptProfile(t, report.Hourly), Executor: adapter,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||||
|
}
|
||||||
|
if result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" {
|
||||||
|
t.Fatalf("inspection = %#v", result)
|
||||||
|
}
|
||||||
|
if strings.Contains(fmt.Sprintf("%#v", result), "127.0.0.1") {
|
||||||
|
t.Fatalf("inspection leaks endpoint: %#v", result)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestPromptInspectionSupportsRakestrawhomeProfileOffline(t *testing.T) {
|
||||||
|
adapter, err := promptkitadapter.New(promptkitadapter.Config{})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("New() error = %v", err)
|
||||||
|
}
|
||||||
|
prompt, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||||
|
Resolved: resolvedPromptProfile(t, report.Hourly),
|
||||||
|
Executor: adapter,
|
||||||
|
Promptkit: config.PromptkitConfig{Profile: "rakestrawhome-gemma-4-31b"},
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||||
|
}
|
||||||
|
if prompt.ProfileID != "rakestrawhome-gemma-4-31b" || prompt.BackendID != "rakestrawhome" || prompt.ModelName == "" {
|
||||||
|
t.Fatalf("prompt inspection = %#v", prompt)
|
||||||
|
}
|
||||||
|
|
||||||
|
comparison, err := app.InspectComparisonExecution(context.Background(), app.ComparisonInspectionRequest{
|
||||||
|
Resolved: resolvedPromptProfile(t, report.Hourly),
|
||||||
|
ProfileIDs: []string{"rakestrawhome-gemma-4-31b", "weather-deep"},
|
||||||
|
Executor: adapter,
|
||||||
|
})
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("InspectComparisonExecution() error = %v", err)
|
||||||
|
}
|
||||||
|
if len(comparison.Profiles) != 2 || comparison.Profiles[0].ProfileID != "rakestrawhome-gemma-4-31b" || comparison.Profiles[0].BackendID != "rakestrawhome" || comparison.Profiles[0].ModelName == "" {
|
||||||
|
t.Fatalf("comparison inspection = %#v", comparison)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
|
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)
|
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)
|
||||||
|
|||||||
@@ -355,7 +355,7 @@ func comparisonResult(outputDirectory string, results []app.ComparisonProfileRes
|
|||||||
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
result := &app.ComparisonResult{
|
result := &app.ComparisonResult{
|
||||||
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
|
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
|
||||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||||
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
||||||
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
|
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
|
||||||
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),
|
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),
|
||||||
|
|||||||
@@ -132,7 +132,7 @@ func actionConfigPath(t *testing.T) string {
|
|||||||
func generatedReportResult() *app.ReportResult {
|
func generatedReportResult() *app.ReportResult {
|
||||||
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
return &app.ReportResult{
|
return &app.ReportResult{
|
||||||
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0",
|
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0",
|
||||||
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
|
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
|
||||||
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
|
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
|
||||||
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,
|
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,
|
||||||
|
|||||||
@@ -43,6 +43,7 @@ type generateSummary struct {
|
|||||||
ModelName string `json:"modelName,omitempty"`
|
ModelName string `json:"modelName,omitempty"`
|
||||||
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
||||||
ValidationStatus string `json:"validationStatus,omitempty"`
|
ValidationStatus string `json:"validationStatus,omitempty"`
|
||||||
|
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||||
Notification *generateNotificationSummary `json:"notification,omitempty"`
|
Notification *generateNotificationSummary `json:"notification,omitempty"`
|
||||||
Error string `json:"error,omitempty"`
|
Error string `json:"error,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -106,6 +107,7 @@ type comparisonProfileSummary struct {
|
|||||||
ModelName string `json:"modelName"`
|
ModelName string `json:"modelName"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
ValidationStatus string `json:"validationStatus,omitempty"`
|
ValidationStatus string `json:"validationStatus,omitempty"`
|
||||||
|
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||||
ReportPath string `json:"reportPath,omitempty"`
|
ReportPath string `json:"reportPath,omitempty"`
|
||||||
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
||||||
Error *comparison.SafeError `json:"error,omitempty"`
|
Error *comparison.SafeError `json:"error,omitempty"`
|
||||||
@@ -129,6 +131,10 @@ func newGenerateSummary(result *app.ReportResult, err error) generateSummary {
|
|||||||
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
|
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
|
||||||
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
||||||
summary.ValidationStatus = string(result.ValidationStatus)
|
summary.ValidationStatus = string(result.ValidationStatus)
|
||||||
|
if result.RepairAttempts != nil {
|
||||||
|
value := *result.RepairAttempts
|
||||||
|
summary.RepairAttempts = &value
|
||||||
|
}
|
||||||
summary.OutputPath = result.OutputPath
|
summary.OutputPath = result.OutputPath
|
||||||
summary.LLMDebugPath = result.LLMDebugPath
|
summary.LLMDebugPath = result.LLMDebugPath
|
||||||
summary.Notification = newGenerateNotificationSummary(result.Notification)
|
summary.Notification = newGenerateNotificationSummary(result.Notification)
|
||||||
@@ -162,6 +168,11 @@ func newGenerateNotificationSummary(result *app.NotificationResult) *generateNot
|
|||||||
return summary
|
return summary
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func repairAttemptsCopy(value int) *int {
|
||||||
|
copy := value
|
||||||
|
return ©
|
||||||
|
}
|
||||||
|
|
||||||
func newBatchSummary(result *app.BatchResult, err error) batchSummary {
|
func newBatchSummary(result *app.BatchResult, err error) batchSummary {
|
||||||
summary := batchSummary{Command: commandRun}
|
summary := batchSummary{Command: commandRun}
|
||||||
if result == nil {
|
if result == nil {
|
||||||
@@ -203,6 +214,9 @@ func newComparisonSummary(result *app.ComparisonResult, err error) comparisonSum
|
|||||||
Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath,
|
Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath,
|
||||||
LLMDebugPath: profile.LLMDebugPath, Error: profile.Error,
|
LLMDebugPath: profile.LLMDebugPath, Error: profile.Error,
|
||||||
})
|
})
|
||||||
|
if profile.RepairAttempts != nil {
|
||||||
|
summary.Results[len(summary.Results)-1].RepairAttempts = repairAttemptsCopy(*profile.RepairAttempts)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
summary.Status = comparisonSummaryStatus(result, err)
|
summary.Status = comparisonSummaryStatus(result, err)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -19,7 +19,7 @@ import (
|
|||||||
|
|
||||||
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
|
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
|
||||||
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
|
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
|
||||||
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
|
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
|
||||||
t.Fatalf("summary = %#v", summary)
|
t.Fatalf("summary = %#v", summary)
|
||||||
}
|
}
|
||||||
@@ -39,7 +39,7 @@ func TestComparisonSummaryUsesLockedOrderAndSafeFields(t *testing.T) {
|
|||||||
profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
|
profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
|
||||||
result := &app.ComparisonResult{
|
result := &app.ComparisonResult{
|
||||||
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
|
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
|
||||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||||
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
||||||
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
|
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
|
||||||
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
||||||
@@ -113,7 +113,7 @@ func TestComparisonSummaryClassifiesCompleteAndAllFailedResults(t *testing.T) {
|
|||||||
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||||
failure := comparison.NewSafeError("generation", "execute prompt failed")
|
failure := comparison.NewSafeError("generation", "execute prompt failed")
|
||||||
complete := &app.ComparisonResult{
|
complete := &app.ComparisonResult{
|
||||||
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||||
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
|
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
|
||||||
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
||||||
Total: 2, Succeeded: 2,
|
Total: 2, Succeeded: 2,
|
||||||
|
|||||||
@@ -15,7 +15,7 @@ import (
|
|||||||
|
|
||||||
const (
|
const (
|
||||||
// SchemaVersion identifies the supported comparison manifest schema.
|
// SchemaVersion identifies the supported comparison manifest schema.
|
||||||
SchemaVersion = "weatherreporter.comparison.v1"
|
SchemaVersion = "weatherreporter.comparison.v2"
|
||||||
|
|
||||||
// ManifestFilename is the canonical name of a comparison manifest.
|
// ManifestFilename is the canonical name of a comparison manifest.
|
||||||
ManifestFilename = "comparison.json"
|
ManifestFilename = "comparison.json"
|
||||||
@@ -71,6 +71,7 @@ type Result struct {
|
|||||||
ModelName string `json:"modelName"`
|
ModelName string `json:"modelName"`
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
ValidationStatus string `json:"validationStatus,omitempty"`
|
ValidationStatus string `json:"validationStatus,omitempty"`
|
||||||
|
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||||
ReportPath string `json:"reportPath,omitempty"`
|
ReportPath string `json:"reportPath,omitempty"`
|
||||||
Error *SafeError `json:"error,omitempty"`
|
Error *SafeError `json:"error,omitempty"`
|
||||||
}
|
}
|
||||||
@@ -276,11 +277,14 @@ func (manifest Manifest) Validate() error {
|
|||||||
if strings.TrimSpace(result.ModelName) == "" {
|
if strings.TrimSpace(result.ModelName) == "" {
|
||||||
return fmt.Errorf("result %d has a blank model name", result.Position)
|
return fmt.Errorf("result %d has a blank model name", result.Position)
|
||||||
}
|
}
|
||||||
|
if result.RepairAttempts != nil && *result.RepairAttempts < 0 {
|
||||||
|
return fmt.Errorf("result %d has negative repair attempts", result.Position)
|
||||||
|
}
|
||||||
|
|
||||||
switch result.Status {
|
switch result.Status {
|
||||||
case StatusSucceeded:
|
case StatusSucceeded:
|
||||||
succeeded++
|
succeeded++
|
||||||
if result.ValidationStatus != "passed" {
|
if result.ValidationStatus != "passed" || result.RepairAttempts == nil {
|
||||||
return fmt.Errorf("successful result %d did not pass validation", result.Position)
|
return fmt.Errorf("successful result %d did not pass validation", result.Position)
|
||||||
}
|
}
|
||||||
expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID)
|
expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID)
|
||||||
@@ -302,6 +306,9 @@ func (manifest Manifest) Validate() error {
|
|||||||
if err := result.Error.validate(); err != nil {
|
if err := result.Error.validate(); err != nil {
|
||||||
return fmt.Errorf("failed result %d: %w", result.Position, err)
|
return fmt.Errorf("failed result %d: %w", result.Position, err)
|
||||||
}
|
}
|
||||||
|
if result.ValidationStatus != "" && result.RepairAttempts == nil {
|
||||||
|
return fmt.Errorf("failed result %d has validation without repair provenance", result.Position)
|
||||||
|
}
|
||||||
default:
|
default:
|
||||||
return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status)
|
return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status)
|
||||||
}
|
}
|
||||||
@@ -324,7 +331,7 @@ func (safeError SafeError) validate() error {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func isValidationStatus(status string) bool {
|
func isValidationStatus(status string) bool {
|
||||||
return status == "" || status == "failed" || status == "skipped"
|
return status == "" || status == "passed" || status == "failed" || status == "skipped"
|
||||||
}
|
}
|
||||||
|
|
||||||
func isSHA256(value string) bool {
|
func isSHA256(value string) bool {
|
||||||
|
|||||||
@@ -118,7 +118,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
|
|||||||
t.Fatalf("EncodeManifest() error = %v", err)
|
t.Fatalf("EncodeManifest() error = %v", err)
|
||||||
}
|
}
|
||||||
want := "{\n" +
|
want := "{\n" +
|
||||||
" \"schemaVersion\": \"weatherreporter.comparison.v1\",\n" +
|
" \"schemaVersion\": \"weatherreporter.comparison.v2\",\n" +
|
||||||
" \"comparisonId\": \"comparison_daily-2026-08-24\",\n" +
|
" \"comparisonId\": \"comparison_daily-2026-08-24\",\n" +
|
||||||
" \"startedAt\": \"2026-08-24T12:00:00Z\",\n" +
|
" \"startedAt\": \"2026-08-24T12:00:00Z\",\n" +
|
||||||
" \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" +
|
" \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" +
|
||||||
@@ -146,6 +146,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
|
|||||||
" \"modelName\": \"gpt-5-mini\",\n" +
|
" \"modelName\": \"gpt-5-mini\",\n" +
|
||||||
" \"status\": \"succeeded\",\n" +
|
" \"status\": \"succeeded\",\n" +
|
||||||
" \"validationStatus\": \"passed\",\n" +
|
" \"validationStatus\": \"passed\",\n" +
|
||||||
|
" \"repairAttempts\": 0,\n" +
|
||||||
" \"reportPath\": \"01-weather-light.md\"\n" +
|
" \"reportPath\": \"01-weather-light.md\"\n" +
|
||||||
" },\n" +
|
" },\n" +
|
||||||
" {\n" +
|
" {\n" +
|
||||||
@@ -201,6 +202,9 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
|
|||||||
{name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }},
|
{name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }},
|
||||||
{name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }},
|
{name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }},
|
||||||
{name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }},
|
{name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }},
|
||||||
|
{name: "failed passed result without repair provenance", mutate: func(manifest *Manifest) {
|
||||||
|
manifest.Results[1].ValidationStatus = "passed"
|
||||||
|
}},
|
||||||
{name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }},
|
{name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }},
|
||||||
{name: "duplicate report path", mutate: func(manifest *Manifest) {
|
{name: "duplicate report path", mutate: func(manifest *Manifest) {
|
||||||
manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath}
|
manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath}
|
||||||
@@ -222,6 +226,17 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestManifestValidateAcceptsPostValidationFailure(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
manifest := validManifest()
|
||||||
|
manifest.Results[1].ValidationStatus = "passed"
|
||||||
|
manifest.Results[1].RepairAttempts = intPtr(0)
|
||||||
|
if err := manifest.Validate(); err != nil {
|
||||||
|
t.Fatalf("Manifest.Validate() rejected post-validation failure: %v", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestLogicalBundleValidate(t *testing.T) {
|
func TestLogicalBundleValidate(t *testing.T) {
|
||||||
t.Parallel()
|
t.Parallel()
|
||||||
|
|
||||||
@@ -309,6 +324,7 @@ func validManifest() Manifest {
|
|||||||
ModelName: "gpt-5-mini",
|
ModelName: "gpt-5-mini",
|
||||||
Status: StatusSucceeded,
|
Status: StatusSucceeded,
|
||||||
ValidationStatus: "passed",
|
ValidationStatus: "passed",
|
||||||
|
RepairAttempts: intPtr(0),
|
||||||
ReportPath: "01-weather-light.md",
|
ReportPath: "01-weather-light.md",
|
||||||
},
|
},
|
||||||
{
|
{
|
||||||
@@ -321,3 +337,8 @@ func validManifest() Manifest {
|
|||||||
},
|
},
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func intPtr(value int) *int {
|
||||||
|
copy := value
|
||||||
|
return ©
|
||||||
|
}
|
||||||
|
|||||||
@@ -388,6 +388,7 @@ var resultFields = map[string]jsonValueValidator{
|
|||||||
"modelName": nil,
|
"modelName": nil,
|
||||||
"status": nil,
|
"status": nil,
|
||||||
"validationStatus": nil,
|
"validationStatus": nil,
|
||||||
|
"repairAttempts": validateRepairAttemptsJSON,
|
||||||
"reportPath": nil,
|
"reportPath": nil,
|
||||||
"error": validateSafeErrorJSON,
|
"error": validateSafeErrorJSON,
|
||||||
}
|
}
|
||||||
@@ -430,7 +431,7 @@ func validateResultsJSON(decoder *json.Decoder) error {
|
|||||||
return fmt.Errorf("results must be an array")
|
return fmt.Errorf("results must be an array")
|
||||||
}
|
}
|
||||||
for decoder.More() {
|
for decoder.More() {
|
||||||
if err := validateJSONObject(decoder, resultFields); err != nil {
|
if err := validateOrderedJSONObject(decoder, resultFields, []string{"position", "profileId", "backendId", "modelName", "status", "validationStatus", "repairAttempts", "reportPath", "error"}); err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -444,6 +445,65 @@ func validateResultsJSON(decoder *json.Decoder) error {
|
|||||||
return nil
|
return nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateOrderedJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator, order []string) error {
|
||||||
|
token, err := decoder.Token()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if delimiter, ok := token.(json.Delim); !ok || delimiter != '{' {
|
||||||
|
return fmt.Errorf("manifest value must be an object")
|
||||||
|
}
|
||||||
|
seen := make(map[string]struct{}, len(fields))
|
||||||
|
last := -1
|
||||||
|
for decoder.More() {
|
||||||
|
token, err := decoder.Token()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
name, ok := token.(string)
|
||||||
|
if !ok {
|
||||||
|
return fmt.Errorf("manifest field name is invalid")
|
||||||
|
}
|
||||||
|
validator, known := fields[name]
|
||||||
|
if !known {
|
||||||
|
return fmt.Errorf("manifest field %q is not canonical", name)
|
||||||
|
}
|
||||||
|
if _, duplicate := seen[name]; duplicate {
|
||||||
|
return fmt.Errorf("manifest field %q is duplicated", name)
|
||||||
|
}
|
||||||
|
position := -1
|
||||||
|
for index, candidate := range order {
|
||||||
|
if candidate == name {
|
||||||
|
position = index
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if position <= last {
|
||||||
|
return fmt.Errorf("manifest field %q is out of canonical order", name)
|
||||||
|
}
|
||||||
|
seen[name] = struct{}{}
|
||||||
|
last = position
|
||||||
|
if validator == nil {
|
||||||
|
var value json.RawMessage
|
||||||
|
if err := decoder.Decode(&value); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if err := validator(decoder); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
}
|
||||||
|
token, err = decoder.Token()
|
||||||
|
if err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if delimiter, ok := token.(json.Delim); !ok || delimiter != '}' {
|
||||||
|
return fmt.Errorf("manifest object has an invalid terminator")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateSafeErrorJSON(decoder *json.Decoder) error {
|
func validateSafeErrorJSON(decoder *json.Decoder) error {
|
||||||
token, err := decoder.Token()
|
token, err := decoder.Token()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
@@ -458,6 +518,24 @@ func validateSafeErrorJSON(decoder *json.Decoder) error {
|
|||||||
return validateJSONObjectBody(decoder, safeErrorFields)
|
return validateJSONObjectBody(decoder, safeErrorFields)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func validateRepairAttemptsJSON(decoder *json.Decoder) error {
|
||||||
|
var raw json.RawMessage
|
||||||
|
if err := decoder.Decode(&raw); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
|
||||||
|
return fmt.Errorf("repairAttempts must be an integer")
|
||||||
|
}
|
||||||
|
var value int
|
||||||
|
if err := json.Unmarshal(raw, &value); err != nil {
|
||||||
|
return err
|
||||||
|
}
|
||||||
|
if value < 0 {
|
||||||
|
return fmt.Errorf("repairAttempts must not be negative")
|
||||||
|
}
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error {
|
func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error {
|
||||||
token, err := decoder.Token()
|
token, err := decoder.Token()
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
|||||||
@@ -1,6 +1,7 @@
|
|||||||
package comparison
|
package comparison
|
||||||
|
|
||||||
import (
|
import (
|
||||||
|
"bytes"
|
||||||
"context"
|
"context"
|
||||||
"encoding/json"
|
"encoding/json"
|
||||||
"errors"
|
"errors"
|
||||||
@@ -146,6 +147,23 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
|
|||||||
t.Helper()
|
t.Helper()
|
||||||
appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`)
|
appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`)
|
||||||
}},
|
}},
|
||||||
|
{name: "null repair attempts", mutate: func(t *testing.T, directory string) {
|
||||||
|
t.Helper()
|
||||||
|
path := filepath.Join(directory, ManifestFilename)
|
||||||
|
data, err := os.ReadFile(path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
before := []byte(" \"status\": \"failed\",\n \"error\": {")
|
||||||
|
after := []byte(" \"status\": \"failed\",\n \"repairAttempts\": null,\n \"error\": {")
|
||||||
|
data = bytes.Replace(data, before, after, 1)
|
||||||
|
if bytes.Equal(data, readFile(t, path)) {
|
||||||
|
t.Fatal("failed to add null repairAttempts")
|
||||||
|
}
|
||||||
|
if err := os.WriteFile(path, data, 0o600); err != nil {
|
||||||
|
t.Fatal(err)
|
||||||
|
}
|
||||||
|
}},
|
||||||
{name: "traversal report path", mutate: func(t *testing.T, directory string) {
|
{name: "traversal report path", mutate: func(t *testing.T, directory string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
path := filepath.Join(directory, ManifestFilename)
|
path := filepath.Join(directory, ManifestFilename)
|
||||||
@@ -202,6 +220,30 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestValidateRepairAttemptsJSONRejectsNonIntegerValues(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, value := range []string{"null", "-1", "1.5", `"1"`, "9223372036854775808", "{"} {
|
||||||
|
value := value
|
||||||
|
t.Run(value, func(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err == nil {
|
||||||
|
t.Fatalf("validateRepairAttemptsJSON(%q) error = nil", value)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestValidateRepairAttemptsJSONAcceptsNonnegativeIntegers(t *testing.T) {
|
||||||
|
t.Parallel()
|
||||||
|
|
||||||
|
for _, value := range []string{"0", "1"} {
|
||||||
|
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err != nil {
|
||||||
|
t.Fatalf("validateRepairAttemptsJSON(%q) error = %v", value, err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func appendManifestField(t *testing.T, directory, field string) {
|
func appendManifestField(t *testing.T, directory, field string) {
|
||||||
t.Helper()
|
t.Helper()
|
||||||
path := filepath.Join(directory, ManifestFilename)
|
path := filepath.Join(directory, ManifestFilename)
|
||||||
@@ -1077,7 +1119,7 @@ func testBundleWithReports(t *testing.T, count int) LogicalBundle {
|
|||||||
}
|
}
|
||||||
manifest.Results[i] = Result{
|
manifest.Results[i] = Result{
|
||||||
Position: position, ProfileID: profileID, ModelName: "gpt-5-mini",
|
Position: position, ProfileID: profileID, ModelName: "gpt-5-mini",
|
||||||
Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: path,
|
Status: StatusSucceeded, ValidationStatus: "passed", RepairAttempts: intPtr(0), ReportPath: path,
|
||||||
}
|
}
|
||||||
reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")}
|
reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -1,6 +1,2 @@
|
|||||||
id: weather-balanced
|
id: weather-balanced
|
||||||
backend: openrouter
|
base_profile: gemini-flash-latest
|
||||||
model: "~google/gemini-flash-latest"
|
|
||||||
reasoning_effort: high
|
|
||||||
timeout_seconds: 240
|
|
||||||
service_tier: flex
|
|
||||||
|
|||||||
@@ -1,6 +1,2 @@
|
|||||||
id: weather-deep
|
id: weather-deep
|
||||||
backend: openrouter
|
base_profile: claude-sonnet-latest
|
||||||
model: "~anthropic/claude-sonnet-latest"
|
|
||||||
reasoning_effort: high
|
|
||||||
timeout_seconds: 240
|
|
||||||
service_tier: flex
|
|
||||||
|
|||||||
@@ -1,5 +1,2 @@
|
|||||||
id: weather-light
|
id: weather-light
|
||||||
backend: openrouter
|
base_profile: deepseek-4-flash
|
||||||
model: deepseek/deepseek-v4-flash
|
|
||||||
timeout_seconds: 180
|
|
||||||
service_tier: flex
|
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.daily_generated_text
|
id: weather.daily_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-balanced
|
default_profile: weather-balanced
|
||||||
description: Daily weather report analysis prompt.
|
description: Daily weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: daily.generated_text.schema.json
|
schema_path: daily.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.hourly_generated_text
|
id: weather.hourly_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-light
|
default_profile: weather-light
|
||||||
description: Hourly weather report analysis prompt.
|
description: Hourly weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: hourly.generated_text.schema.json
|
schema_path: hourly.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.today_generated_text
|
id: weather.today_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-balanced
|
default_profile: weather-balanced
|
||||||
description: Today's weather report analysis prompt.
|
description: Today's weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: today.generated_text.schema.json
|
schema_path: today.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -1,5 +1,5 @@
|
|||||||
id: weather.tomorrow_generated_text
|
id: weather.tomorrow_generated_text
|
||||||
version: "2.0.0"
|
version: "2.1.0"
|
||||||
default_profile: weather-balanced
|
default_profile: weather-balanced
|
||||||
description: Tomorrow's weather report analysis prompt.
|
description: Tomorrow's weather report analysis prompt.
|
||||||
inputs:
|
inputs:
|
||||||
@@ -21,3 +21,4 @@ output:
|
|||||||
format: json
|
format: json
|
||||||
validation_mode: json_schema
|
validation_mode: json_schema
|
||||||
schema_path: tomorrow.generated_text.schema.json
|
schema_path: tomorrow.generated_text.schema.json
|
||||||
|
repair_attempts: 1
|
||||||
|
|||||||
@@ -77,8 +77,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
|||||||
if err := yaml.Unmarshal(data, &definition); err != nil {
|
if err := yaml.Unmarshal(data, &definition); err != nil {
|
||||||
t.Fatalf("decode prompt definition: %v", err)
|
t.Fatalf("decode prompt definition: %v", err)
|
||||||
}
|
}
|
||||||
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile {
|
if definition.ID != tc.id || definition.Version != "2.1.0" || definition.DefaultProfile != tc.profile {
|
||||||
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile)
|
t.Fatalf("definition = %#v, want %s version 2.1.0 and profile %s", definition, tc.id, tc.profile)
|
||||||
}
|
}
|
||||||
sharedInstruction := false
|
sharedInstruction := false
|
||||||
for _, message := range definition.Messages {
|
for _, message := range definition.Messages {
|
||||||
@@ -92,8 +92,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
|||||||
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
||||||
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
||||||
}
|
}
|
||||||
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts != nil {
|
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts == nil || *definition.Output.RepairAttempts != 1 {
|
||||||
t.Fatalf("output = %#v, want JSON schema output without repair attempts", definition.Output)
|
t.Fatalf("output = %#v, want JSON schema output with one repair attempt", definition.Output)
|
||||||
}
|
}
|
||||||
if _, err := promptassets.Schema(tc.schemaID); err != nil {
|
if _, err := promptassets.Schema(tc.schemaID); err != nil {
|
||||||
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
|
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
|
||||||
@@ -286,11 +286,11 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
|
|||||||
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
|
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
|
||||||
} {
|
} {
|
||||||
t.Run(want.id, func(t *testing.T) {
|
t.Run(want.id, func(t *testing.T) {
|
||||||
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.0.0")
|
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.1.0")
|
||||||
if err != nil {
|
if err != nil {
|
||||||
t.Fatalf("InspectPrompt() error = %v", err)
|
t.Fatalf("InspectPrompt() error = %v", err)
|
||||||
}
|
}
|
||||||
if inspection.PromptID != want.id || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != want.profile {
|
if inspection.PromptID != want.id || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != want.profile || inspection.OutputContract.RepairAttempts != 1 {
|
||||||
t.Fatalf("inspection = %#v", inspection)
|
t.Fatalf("inspection = %#v", inspection)
|
||||||
}
|
}
|
||||||
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
|
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
|
||||||
@@ -361,6 +361,29 @@ func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestEmbeddedProfilesAreMinimalBaseAliases(t *testing.T) {
|
||||||
|
wantBases := map[string]string{
|
||||||
|
"weather-balanced.yml": "gemini-flash-latest",
|
||||||
|
"weather-deep.yml": "claude-sonnet-latest",
|
||||||
|
"weather-light.yml": "deepseek-4-flash",
|
||||||
|
}
|
||||||
|
for path, wantBase := range wantBases {
|
||||||
|
t.Run(path, func(t *testing.T) {
|
||||||
|
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("read profile: %v", err)
|
||||||
|
}
|
||||||
|
var definition map[string]string
|
||||||
|
if err := yaml.Unmarshal(data, &definition); err != nil {
|
||||||
|
t.Fatalf("unmarshal profile: %v", err)
|
||||||
|
}
|
||||||
|
if definition["id"] != strings.TrimSuffix(path, ".yml") || definition["base_profile"] != wantBase || len(definition) != 2 {
|
||||||
|
t.Fatalf("profile definition = %#v, want only its id and base profile %q", definition, wantBase)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
|
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
|
||||||
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
|
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
|
||||||
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||||
@@ -391,7 +414,7 @@ func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
|
|||||||
if err != nil {
|
if err != nil {
|
||||||
return err
|
return err
|
||||||
}
|
}
|
||||||
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "repair_attempts:", "weather.daily_report"} {
|
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "weather.daily_report"} {
|
||||||
if strings.Contains(string(data), unwanted) {
|
if strings.Contains(string(data), unwanted) {
|
||||||
t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
|
t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -21,8 +21,9 @@ import (
|
|||||||
var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform")
|
var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform")
|
||||||
|
|
||||||
const (
|
const (
|
||||||
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v2"
|
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v3"
|
||||||
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v2"
|
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v3"
|
||||||
|
promptFailureDebugSchemaVersion = "weatherreporter.prompt_failure_debug.v1"
|
||||||
debugDirectoryMode = 0o700
|
debugDirectoryMode = 0o700
|
||||||
debugFileMode = 0o600
|
debugFileMode = 0o600
|
||||||
)
|
)
|
||||||
@@ -53,6 +54,7 @@ type PromptDebugOutput struct {
|
|||||||
Format string `json:"format"`
|
Format string `json:"format"`
|
||||||
ValidationMode string `json:"validationMode"`
|
ValidationMode string `json:"validationMode"`
|
||||||
SchemaPath string `json:"schemaPath"`
|
SchemaPath string `json:"schemaPath"`
|
||||||
|
RepairAttempts int `json:"repairAttempts"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// PromptDebugPreparation is the explicit, content-safe mapping of preparation
|
// PromptDebugPreparation is the explicit, content-safe mapping of preparation
|
||||||
@@ -97,10 +99,28 @@ type PromptDebugUsage struct {
|
|||||||
// PromptDebugValidation is the completed validation detail retained in the
|
// PromptDebugValidation is the completed validation detail retained in the
|
||||||
// explicitly enabled debug store.
|
// explicitly enabled debug store.
|
||||||
type PromptDebugValidation struct {
|
type PromptDebugValidation struct {
|
||||||
Status string `json:"status"`
|
Status string `json:"status"`
|
||||||
Mode string `json:"mode"`
|
Mode string `json:"mode"`
|
||||||
SchemaPath string `json:"schemaPath"`
|
SchemaPath string `json:"schemaPath"`
|
||||||
Diagnostics []string `json:"diagnostics,omitempty"`
|
RepairAttempts int `json:"repairAttempts"`
|
||||||
|
Diagnostics []string `json:"diagnostics,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// PromptFailureDebugArtifact records provider detail only in explicit debug storage.
|
||||||
|
type PromptFailureDebugArtifact struct {
|
||||||
|
SchemaVersion string `json:"schemaVersion"`
|
||||||
|
ReportID report.ID `json:"reportId"`
|
||||||
|
ValidDate string `json:"validDate"`
|
||||||
|
RunID string `json:"runId"`
|
||||||
|
Failure PromptFailure `json:"failure"`
|
||||||
|
}
|
||||||
|
|
||||||
|
type PromptFailure struct {
|
||||||
|
Category string `json:"category"`
|
||||||
|
StatusCode int `json:"statusCode,omitempty"`
|
||||||
|
ProviderCode string `json:"providerCode,omitempty"`
|
||||||
|
ProviderType string `json:"providerType,omitempty"`
|
||||||
|
ProviderMessage string `json:"providerMessage,omitempty"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// PromptDebugExecution is the explicit mapping of execution provenance.
|
// PromptDebugExecution is the explicit mapping of execution provenance.
|
||||||
@@ -235,6 +255,24 @@ func (w *PromptDebugWriter) WriteExecution(ref PromptDebugRef, execution prompte
|
|||||||
return directory, nil
|
return directory, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// WriteFailure stores the project-owned structured provider failure.
|
||||||
|
func (w *PromptDebugWriter) WriteFailure(ref PromptDebugRef, failure *promptexec.GenerationError) (string, error) {
|
||||||
|
if !w.Enabled() || failure == nil {
|
||||||
|
return "", nil
|
||||||
|
}
|
||||||
|
directory, secureDirectory, err := w.runDirectory(ref)
|
||||||
|
if err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
defer secureDirectory.Close()
|
||||||
|
artifact := PromptFailureDebugArtifact{SchemaVersion: promptFailureDebugSchemaVersion, ReportID: ref.ReportID, ValidDate: ref.ValidDate, RunID: ref.RunID,
|
||||||
|
Failure: PromptFailure{Category: string(failure.Category()), StatusCode: failure.StatusCode(), ProviderCode: failure.ProviderCode(), ProviderType: failure.ProviderType(), ProviderMessage: failure.ProviderMessage()}}
|
||||||
|
if err := secureDirectory.writeJSON("failure.json", artifact); err != nil {
|
||||||
|
return "", err
|
||||||
|
}
|
||||||
|
return directory, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) {
|
func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) {
|
||||||
if err := validatePromptDebugRef(ref); err != nil {
|
if err := validatePromptDebugRef(ref); err != nil {
|
||||||
return "", nil, err
|
return "", nil, err
|
||||||
@@ -283,7 +321,7 @@ func promptDebugPreparation(value promptexec.Preparation) PromptDebugPreparation
|
|||||||
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash,
|
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash,
|
||||||
RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes),
|
RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes),
|
||||||
ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName,
|
ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName,
|
||||||
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath},
|
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath, RepairAttempts: value.Output.RepairAttempts},
|
||||||
StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration,
|
StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration,
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -300,7 +338,7 @@ func promptDebugExecution(value promptexec.Execution) PromptDebugExecution {
|
|||||||
}
|
}
|
||||||
|
|
||||||
func promptDebugValidation(value promptexec.Validation) PromptDebugValidation {
|
func promptDebugValidation(value promptexec.Validation) PromptDebugValidation {
|
||||||
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, Diagnostics: append([]string(nil), value.Diagnostics...)}
|
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, RepairAttempts: value.RepairAttempts, Diagnostics: append([]string(nil), value.Diagnostics...)}
|
||||||
}
|
}
|
||||||
|
|
||||||
func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage {
|
func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage {
|
||||||
|
|||||||
@@ -44,13 +44,13 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
|
|||||||
t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir)
|
t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir)
|
||||||
}
|
}
|
||||||
preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json"))
|
preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json"))
|
||||||
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v2", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`} {
|
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v3", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`, `"repairAttempts": 0`} {
|
||||||
if !strings.Contains(string(preparationData), want) {
|
if !strings.Contains(string(preparationData), want) {
|
||||||
t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData)
|
t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json"))
|
executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json"))
|
||||||
for _, want := range []string{"weatherreporter.prompt_execution_debug.v2", "Generated forecast prose.", "validation details", `"status": "passed"`} {
|
for _, want := range []string{"weatherreporter.prompt_execution_debug.v3", "Generated forecast prose.", "validation details", `"status": "passed"`, `"repairAttempts": 0`} {
|
||||||
if !strings.Contains(string(executionData), want) {
|
if !strings.Contains(string(executionData), want) {
|
||||||
t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData)
|
t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData)
|
||||||
}
|
}
|
||||||
@@ -66,6 +66,25 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
|
|||||||
assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode)
|
assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode)
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestPromptDebugWriterWritesProviderFailureOnlyToDebugStore(t *testing.T) {
|
||||||
|
writer, err := NewPromptDebugWriter(filepath.Join(t.TempDir(), "operator-debug"))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("NewPromptDebugWriter() error = %v", err)
|
||||||
|
}
|
||||||
|
marker := "provider-private-marker"
|
||||||
|
directory, err := writer.WriteFailure(promptDebugRef(), promptexec.NewGenerationError(429, "rate_limit", "provider_error", marker, nil))
|
||||||
|
if err != nil {
|
||||||
|
t.Fatalf("WriteFailure() error = %v", err)
|
||||||
|
}
|
||||||
|
data := readPromptDebugFile(t, filepath.Join(directory, "failure.json"))
|
||||||
|
for _, want := range []string{"weatherreporter.prompt_failure_debug.v1", `"category": "generation"`, `"statusCode": 429`, marker} {
|
||||||
|
if !strings.Contains(string(data), want) {
|
||||||
|
t.Fatalf("failure artifact missing %q: %s", want, data)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
assertPromptDebugMode(t, filepath.Join(directory, "failure.json"), debugFileMode)
|
||||||
|
}
|
||||||
|
|
||||||
func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) {
|
func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) {
|
||||||
const marker = "private-debug-marker"
|
const marker = "private-debug-marker"
|
||||||
tests := []struct {
|
tests := []struct {
|
||||||
@@ -349,7 +368,7 @@ func promptDebugExecutionFixture() promptexec.Execution {
|
|||||||
InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model",
|
InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model",
|
||||||
GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15},
|
GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15},
|
||||||
StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second,
|
StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second,
|
||||||
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", []string{"validation details"}),
|
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", 0, []string{"validation details"}),
|
||||||
RawOutput: []byte("Generated forecast prose."),
|
RawOutput: []byte("Generated forecast prose."),
|
||||||
Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}},
|
Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}},
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -78,3 +78,15 @@ func boundText(value string, limit int) string {
|
|||||||
}
|
}
|
||||||
return value
|
return value
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func boundCodePoints(value string, limit int) string {
|
||||||
|
if limit <= 0 || value == "" {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
value = strings.ToValidUTF8(value, "<22>")
|
||||||
|
runes := []rune(value)
|
||||||
|
if len(runes) <= limit {
|
||||||
|
return value
|
||||||
|
}
|
||||||
|
return string(runes[:limit])
|
||||||
|
}
|
||||||
|
|||||||
@@ -51,6 +51,7 @@ type OutputContract struct {
|
|||||||
Format string
|
Format string
|
||||||
ValidationMode string
|
ValidationMode string
|
||||||
SchemaPath string
|
SchemaPath string
|
||||||
|
RepairAttempts int
|
||||||
}
|
}
|
||||||
|
|
||||||
// ProfileInspection describes the safe, selected execution identity for one profile.
|
// ProfileInspection describes the safe, selected execution identity for one profile.
|
||||||
@@ -141,19 +142,21 @@ type TokenUsage struct {
|
|||||||
|
|
||||||
// Validation records a completed output validation check.
|
// Validation records a completed output validation check.
|
||||||
type Validation struct {
|
type Validation struct {
|
||||||
Status ValidationStatus
|
Status ValidationStatus
|
||||||
Mode string
|
Mode string
|
||||||
SchemaPath string
|
SchemaPath string
|
||||||
Diagnostics []string
|
RepairAttempts int
|
||||||
|
Diagnostics []string
|
||||||
}
|
}
|
||||||
|
|
||||||
// NewValidation returns a completed validation value with bounded diagnostics.
|
// NewValidation returns a completed validation value with bounded diagnostics.
|
||||||
func NewValidation(status ValidationStatus, mode string, schemaPath string, diagnostics []string) Validation {
|
func NewValidation(status ValidationStatus, mode string, schemaPath string, repairAttempts int, diagnostics []string) Validation {
|
||||||
return Validation{
|
return Validation{
|
||||||
Status: status,
|
Status: status,
|
||||||
Mode: mode,
|
Mode: mode,
|
||||||
SchemaPath: schemaPath,
|
SchemaPath: schemaPath,
|
||||||
Diagnostics: boundDiagnostics(diagnostics),
|
RepairAttempts: repairAttempts,
|
||||||
|
Diagnostics: boundDiagnostics(diagnostics),
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -235,6 +238,86 @@ func (e *Error) Category() ErrorCategory {
|
|||||||
return e.category
|
return e.category
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// GenerationError retains provider failure details for programmatic handling
|
||||||
|
// without exposing them through routine formatting or serialization.
|
||||||
|
type GenerationError struct {
|
||||||
|
statusCode int
|
||||||
|
providerCode string
|
||||||
|
providerType string
|
||||||
|
providerMessage string
|
||||||
|
err *Error
|
||||||
|
}
|
||||||
|
|
||||||
|
// NewGenerationError returns a classified generation failure with bounded
|
||||||
|
// provider details. The dependency cause remains reachable through err only.
|
||||||
|
func NewGenerationError(statusCode int, providerCode string, providerType string, providerMessage string, cause error) *GenerationError {
|
||||||
|
return &GenerationError{
|
||||||
|
statusCode: statusCode,
|
||||||
|
providerCode: boundCodePoints(providerCode, 256),
|
||||||
|
providerType: boundCodePoints(providerType, 256),
|
||||||
|
providerMessage: boundCodePoints(providerMessage, 4096),
|
||||||
|
err: NewError(Generation, "provider generation failed", cause),
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// StatusCode returns the provider HTTP status when one was available.
|
||||||
|
func (e *GenerationError) StatusCode() int {
|
||||||
|
if e == nil {
|
||||||
|
return 0
|
||||||
|
}
|
||||||
|
return e.statusCode
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProviderCode returns the bounded provider error code.
|
||||||
|
func (e *GenerationError) ProviderCode() string {
|
||||||
|
if e == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return e.providerCode
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProviderType returns the bounded provider error type.
|
||||||
|
func (e *GenerationError) ProviderType() string {
|
||||||
|
if e == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return e.providerType
|
||||||
|
}
|
||||||
|
|
||||||
|
// ProviderMessage returns the bounded provider error message.
|
||||||
|
func (e *GenerationError) ProviderMessage() string {
|
||||||
|
if e == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
return e.providerMessage
|
||||||
|
}
|
||||||
|
|
||||||
|
// Category returns Generation for every generation failure.
|
||||||
|
func (e *GenerationError) Category() ErrorCategory { return Generation }
|
||||||
|
|
||||||
|
// Error intentionally excludes provider details from ordinary error text.
|
||||||
|
func (e *GenerationError) Error() string {
|
||||||
|
if e == nil {
|
||||||
|
return ""
|
||||||
|
}
|
||||||
|
message := NewError(Generation, "provider generation failed", nil).Error()
|
||||||
|
if e.statusCode == 0 {
|
||||||
|
return message
|
||||||
|
}
|
||||||
|
return fmt.Sprintf("%s (HTTP %d)", message, e.statusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
// GoString keeps %#v formatting as safe as ordinary error formatting.
|
||||||
|
func (e *GenerationError) GoString() string { return e.Error() }
|
||||||
|
|
||||||
|
// Unwrap preserves the project-owned classified error and its hidden cause.
|
||||||
|
func (e *GenerationError) Unwrap() error {
|
||||||
|
if e == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
return e.err
|
||||||
|
}
|
||||||
|
|
||||||
// CapacityError adds the safe backend identity to a capacity failure.
|
// CapacityError adds the safe backend identity to a capacity failure.
|
||||||
type CapacityError struct {
|
type CapacityError struct {
|
||||||
BackendID string
|
BackendID string
|
||||||
|
|||||||
@@ -3,6 +3,7 @@ package promptexec
|
|||||||
import (
|
import (
|
||||||
"context"
|
"context"
|
||||||
"errors"
|
"errors"
|
||||||
|
"fmt"
|
||||||
"strings"
|
"strings"
|
||||||
"testing"
|
"testing"
|
||||||
"unicode/utf8"
|
"unicode/utf8"
|
||||||
@@ -172,8 +173,8 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
|
|||||||
if bounded[0] != "a<>b" {
|
if bounded[0] != "a<>b" {
|
||||||
t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0])
|
t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0])
|
||||||
}
|
}
|
||||||
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", values)
|
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", 2, values)
|
||||||
if len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
|
if validation.RepairAttempts != 2 || len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
|
||||||
t.Fatalf("validation = %#v, want bounded diagnostics", validation)
|
t.Fatalf("validation = %#v, want bounded diagnostics", validation)
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -187,8 +188,53 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func TestGenerationErrorKeepsProviderDetailsOutOfRoutineFormatting(t *testing.T) {
|
||||||
|
cause := errors.New("provider response must stay hidden")
|
||||||
|
error := NewGenerationError(
|
||||||
|
429,
|
||||||
|
string([]byte{'c', 0xff})+strings.Repeat("界", 300),
|
||||||
|
strings.Repeat("type", 100),
|
||||||
|
string([]byte{'m', 0xff})+strings.Repeat("界", 4_200),
|
||||||
|
cause,
|
||||||
|
)
|
||||||
|
if error.StatusCode() != 429 || error.Category() != Generation || CategoryOf(error) != Generation {
|
||||||
|
t.Fatalf("generation error identity = %#v", error)
|
||||||
|
}
|
||||||
|
if utf8.RuneCountInString(error.ProviderCode()) != 256 || utf8.RuneCountInString(error.ProviderType()) != 256 || utf8.RuneCountInString(error.ProviderMessage()) != 4096 {
|
||||||
|
t.Fatalf("provider detail bounds = %d/%d/%d", utf8.RuneCountInString(error.ProviderCode()), utf8.RuneCountInString(error.ProviderType()), utf8.RuneCountInString(error.ProviderMessage()))
|
||||||
|
}
|
||||||
|
if !utf8.ValidString(error.ProviderCode()) || !utf8.ValidString(error.ProviderType()) || !utf8.ValidString(error.ProviderMessage()) {
|
||||||
|
t.Fatalf("provider details are not valid UTF-8: %#v", error)
|
||||||
|
}
|
||||||
|
if !errors.Is(error, cause) {
|
||||||
|
t.Fatal("errors.Is() = false, want preserved dependency cause")
|
||||||
|
}
|
||||||
|
var owned *Error
|
||||||
|
if !errors.As(error, &owned) || owned.Category() != Generation {
|
||||||
|
t.Fatalf("errors.As(*Error) = %#v, want project-owned generation error", owned)
|
||||||
|
}
|
||||||
|
want := "generation: provider generation failed (HTTP 429)"
|
||||||
|
if error.Error() != want || fmt.Sprintf("%#v", error) != want {
|
||||||
|
t.Fatalf("ordinary formatting = %q / %#v, want %q", error.Error(), error, want)
|
||||||
|
}
|
||||||
|
for _, private := range []string{cause.Error(), error.ProviderCode(), error.ProviderType(), error.ProviderMessage()} {
|
||||||
|
if strings.Contains(error.Error(), private) || strings.Contains(fmt.Sprintf("%#v", error), private) {
|
||||||
|
t.Fatalf("generation error leaks provider detail %q", private)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
statusOnly := NewGenerationError(503, "", "", "", nil)
|
||||||
|
if statusOnly.Error() != "generation: provider generation failed (HTTP 503)" {
|
||||||
|
t.Fatalf("status-only error = %q", statusOnly)
|
||||||
|
}
|
||||||
|
var nilError *GenerationError
|
||||||
|
if nilError.StatusCode() != 0 || nilError.ProviderCode() != "" || nilError.ProviderType() != "" || nilError.ProviderMessage() != "" || nilError.Category() != Generation || nilError.Error() != "" || nilError.GoString() != "" || nilError.Unwrap() != nil {
|
||||||
|
t.Fatalf("nil generation error = %#v", nilError)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
func TestContractCopiesMutableValues(t *testing.T) {
|
func TestContractCopiesMutableValues(t *testing.T) {
|
||||||
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}}
|
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}, Output: OutputContract{RepairAttempts: 3}}
|
||||||
preparationDebug := &PreparationDebug{
|
preparationDebug := &PreparationDebug{
|
||||||
RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}},
|
RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}},
|
||||||
StructuredSchema: []byte("schema body"),
|
StructuredSchema: []byte("schema body"),
|
||||||
@@ -196,7 +242,7 @@ func TestContractCopiesMutableValues(t *testing.T) {
|
|||||||
}
|
}
|
||||||
execution := Execution{
|
execution := Execution{
|
||||||
InputHashes: map[string]string{"data_package": "input-hash"},
|
InputHashes: map[string]string{"data_package": "input-hash"},
|
||||||
Validation: Validation{Diagnostics: []string{"validation detail"}},
|
Validation: Validation{RepairAttempts: 2, Diagnostics: []string{"validation detail"}},
|
||||||
RawOutput: []byte("generated output"),
|
RawOutput: []byte("generated output"),
|
||||||
Debug: &ExecutionDebug{
|
Debug: &ExecutionDebug{
|
||||||
RawOutput: []byte("provider output"),
|
RawOutput: []byte("provider output"),
|
||||||
@@ -217,13 +263,13 @@ func TestContractCopiesMutableValues(t *testing.T) {
|
|||||||
execution.Debug.RawOutput[0] = 'x'
|
execution.Debug.RawOutput[0] = 'x'
|
||||||
execution.Debug.ValidationDiagnostics[0] = "changed"
|
execution.Debug.ValidationDiagnostics[0] = "changed"
|
||||||
|
|
||||||
if preparationCopy.InputHashes["data_package"] != "input-hash" {
|
if preparationCopy.InputHashes["data_package"] != "input-hash" || preparationCopy.Output.RepairAttempts != 3 {
|
||||||
t.Fatalf("preparation copy = %#v", preparationCopy)
|
t.Fatalf("preparation copy = %#v", preparationCopy)
|
||||||
}
|
}
|
||||||
if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` {
|
if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` {
|
||||||
t.Fatalf("preparation debug copy = %#v", debugCopy)
|
t.Fatalf("preparation debug copy = %#v", debugCopy)
|
||||||
}
|
}
|
||||||
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
|
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.RepairAttempts != 2 || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
|
||||||
t.Fatalf("execution copy = %#v", executionCopy)
|
t.Fatalf("execution copy = %#v", executionCopy)
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -9,14 +9,15 @@ import (
|
|||||||
|
|
||||||
func dailyDefinition() Definition {
|
func dailyDefinition() Definition {
|
||||||
return Definition{
|
return Definition{
|
||||||
ID: Daily,
|
ID: Daily,
|
||||||
Name: "Daily Report",
|
Name: "Daily Report",
|
||||||
PromptID: "weather.daily_generated_text",
|
PromptID: "weather.daily_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "daily",
|
TemplateID: "daily",
|
||||||
GeneratedTextSchemaID: "daily",
|
GeneratedTextSchemaID: "daily",
|
||||||
ArtifactGroup: "daily",
|
GeneratedTextRepairAttempts: 1,
|
||||||
OutputName: "daily.md",
|
ArtifactGroup: "daily",
|
||||||
|
OutputName: "daily.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
"daily/{valid_start_date}/{run_id}.md",
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
"daily/{valid_start_date}/index.md",
|
"daily/{valid_start_date}/index.md",
|
||||||
|
|||||||
@@ -27,20 +27,21 @@ const (
|
|||||||
)
|
)
|
||||||
|
|
||||||
type Definition struct {
|
type Definition struct {
|
||||||
ID ID
|
ID ID
|
||||||
Name string
|
Name string
|
||||||
PromptID string
|
PromptID string
|
||||||
PromptVersion string
|
PromptVersion string
|
||||||
TemplateID string
|
TemplateID string
|
||||||
GeneratedTextSchemaID string
|
GeneratedTextSchemaID string
|
||||||
ArtifactGroup string
|
GeneratedTextRepairAttempts int
|
||||||
OutputName string
|
ArtifactGroup string
|
||||||
DistributorPathTemplates []string
|
OutputName string
|
||||||
Modules []module.ConfigItem
|
DistributorPathTemplates []string
|
||||||
Morning bool
|
Modules []module.ConfigItem
|
||||||
Evening bool
|
Morning bool
|
||||||
resolve func(ResolveRequest) (timeutil.Period, error)
|
Evening bool
|
||||||
runIDDisambiguator func(Resolved) string
|
resolve func(ResolveRequest) (timeutil.Period, error)
|
||||||
|
runIDDisambiguator func(Resolved) string
|
||||||
}
|
}
|
||||||
|
|
||||||
func (r Resolved) OutputName() (string, error) {
|
func (r Resolved) OutputName() (string, error) {
|
||||||
|
|||||||
@@ -11,14 +11,15 @@ const hourlyReportHours = 6
|
|||||||
|
|
||||||
func hourlyDefinition() Definition {
|
func hourlyDefinition() Definition {
|
||||||
return Definition{
|
return Definition{
|
||||||
ID: Hourly,
|
ID: Hourly,
|
||||||
Name: "Hourly Report",
|
Name: "Hourly Report",
|
||||||
PromptID: "weather.hourly_generated_text",
|
PromptID: "weather.hourly_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "hourly",
|
TemplateID: "hourly",
|
||||||
GeneratedTextSchemaID: "hourly",
|
GeneratedTextSchemaID: "hourly",
|
||||||
ArtifactGroup: "hourly",
|
GeneratedTextRepairAttempts: 1,
|
||||||
OutputName: "hourly.md",
|
ArtifactGroup: "hourly",
|
||||||
|
OutputName: "hourly.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
"hourly/index.md",
|
"hourly/index.md",
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -82,8 +82,8 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
for _, definition := range definitions {
|
for _, definition := range definitions {
|
||||||
if definition.PromptVersion != "2.0.0" {
|
if definition.PromptVersion != "2.1.0" {
|
||||||
t.Fatalf("%s PromptVersion = %q, want 2.0.0", definition.ID, definition.PromptVersion)
|
t.Fatalf("%s PromptVersion = %q, want 2.1.0", definition.ID, definition.PromptVersion)
|
||||||
}
|
}
|
||||||
if definition.PromptID == "" {
|
if definition.PromptID == "" {
|
||||||
t.Fatalf("%s PromptID is empty", definition.ID)
|
t.Fatalf("%s PromptID is empty", definition.ID)
|
||||||
@@ -91,6 +91,9 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
|||||||
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
|
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
|
||||||
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
|
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
|
||||||
}
|
}
|
||||||
|
if definition.GeneratedTextRepairAttempts != 1 {
|
||||||
|
t.Fatalf("%s repair attempts = %d, want 1", definition.ID, definition.GeneratedTextRepairAttempts)
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
if _, err := registry.Lookup(ID("three_day")); err == nil {
|
if _, err := registry.Lookup(ID("three_day")); err == nil {
|
||||||
|
|||||||
@@ -7,14 +7,15 @@ import (
|
|||||||
|
|
||||||
func todayDefinition() Definition {
|
func todayDefinition() Definition {
|
||||||
return Definition{
|
return Definition{
|
||||||
ID: Today,
|
ID: Today,
|
||||||
Name: "Today Report",
|
Name: "Today Report",
|
||||||
PromptID: "weather.today_generated_text",
|
PromptID: "weather.today_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "today",
|
TemplateID: "today",
|
||||||
GeneratedTextSchemaID: "today",
|
GeneratedTextSchemaID: "today",
|
||||||
ArtifactGroup: "today",
|
GeneratedTextRepairAttempts: 1,
|
||||||
OutputName: "today.md",
|
ArtifactGroup: "today",
|
||||||
|
OutputName: "today.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
"daily/{valid_start_date}/{run_id}.md",
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
"daily/{valid_start_date}/index.md",
|
"daily/{valid_start_date}/index.md",
|
||||||
|
|||||||
@@ -7,14 +7,15 @@ import (
|
|||||||
|
|
||||||
func tomorrowDefinition() Definition {
|
func tomorrowDefinition() Definition {
|
||||||
return Definition{
|
return Definition{
|
||||||
ID: Tomorrow,
|
ID: Tomorrow,
|
||||||
Name: "Tomorrow Report",
|
Name: "Tomorrow Report",
|
||||||
PromptID: "weather.tomorrow_generated_text",
|
PromptID: "weather.tomorrow_generated_text",
|
||||||
PromptVersion: "2.0.0",
|
PromptVersion: "2.1.0",
|
||||||
TemplateID: "tomorrow",
|
TemplateID: "tomorrow",
|
||||||
GeneratedTextSchemaID: "tomorrow",
|
GeneratedTextSchemaID: "tomorrow",
|
||||||
ArtifactGroup: "tomorrow",
|
GeneratedTextRepairAttempts: 1,
|
||||||
OutputName: "tomorrow.md",
|
ArtifactGroup: "tomorrow",
|
||||||
|
OutputName: "tomorrow.md",
|
||||||
DistributorPathTemplates: []string{
|
DistributorPathTemplates: []string{
|
||||||
"daily/{valid_start_date}/{run_id}.md",
|
"daily/{valid_start_date}/{run_id}.md",
|
||||||
"daily/{valid_start_date}/index.md",
|
"daily/{valid_start_date}/index.md",
|
||||||
|
|||||||
Reference in New Issue
Block a user