Compare commits
14 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 5d306a2024 | |||
| cb572e4b93 | |||
| 668cf74879 | |||
| 0bafbcb21f | |||
| 7005688b80 | |||
| 4cd5f505df | |||
| b3b23fb381 | |||
| 0c9cd6d5fb | |||
| ce79ea92c5 | |||
| 20107b0dfd | |||
| 1b38f66240 | |||
| b92f83e49b | |||
| 24a8579cee | |||
| 515cdada04 |
16
docs/cli.md
16
docs/cli.md
@@ -96,7 +96,7 @@ period, prompt version, timezone, and status. Successful output has an absolute
|
||||
"command": "generate",
|
||||
"reportId": "today",
|
||||
"promptId": "weather.today_generated_text",
|
||||
"promptVersion": "2.0.0",
|
||||
"promptVersion": "2.1.0",
|
||||
"runId": "20260529T120000.000000000Z_today",
|
||||
"status": "succeeded",
|
||||
"timezone": "America/Chicago",
|
||||
@@ -106,7 +106,10 @@ period, prompt version, timezone, and status. Successful output has an absolute
|
||||
|
||||
When available, the summary also includes the effective `profileId`,
|
||||
`backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested
|
||||
`llmDebugPath`, and compact Distributor `notification` result. It does not
|
||||
`repairAttempts`, `llmDebugPath`, and compact Distributor `notification`
|
||||
result. `repairAttempts` is `0` when the initial output passed validation,
|
||||
positive when PromptKit made corrective generation calls, and omitted when
|
||||
validation did not complete. The summary does not
|
||||
include historical or transient artifact paths such as metadata, prompt input,
|
||||
raw generated text, render context, or notification receipts.
|
||||
|
||||
@@ -115,7 +118,8 @@ raw generated text, render context, or notification receipts.
|
||||
A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`,
|
||||
`total`, `succeeded`, `failed`, and a `reports` array. Each report item includes
|
||||
its identity, status, effective profile and model details when available,
|
||||
source warnings, validation status, and absolute `outputPath` after publication.
|
||||
source warnings, validation status, repair-attempt count when validation
|
||||
completed, and absolute `outputPath` after publication.
|
||||
The top-level summary may also contain a batch `notification` object and
|
||||
`error`. Batch status is `failed` if any report or the batch notification fails.
|
||||
The `total`, `succeeded`, and `failed` counters describe report items only, so
|
||||
@@ -143,7 +147,8 @@ A comparison summary contains these fields in this order: `command`,
|
||||
successful `results[].reportPath` are absolute. `results` preserves the
|
||||
supplied profile order and each item contains `position`, `profileId`, optional
|
||||
`backendId`, `modelName`, `status`, optional `validationStatus`, optional
|
||||
`reportPath`, optional `llmDebugPath`, and optional safe `error`.
|
||||
`repairAttempts`, optional `reportPath`, optional `llmDebugPath`, and optional
|
||||
safe `error`. The repair-attempt semantics match the generate summary.
|
||||
|
||||
The comparison status is `succeeded` only when every selected profile succeeds
|
||||
and the bundle is published. Individual profile failures still publish a
|
||||
@@ -157,7 +162,8 @@ published Promptkit category; destination failures use `destination_<kind>`;
|
||||
and committed cleanup failures use `publication_cleanup` with a message that
|
||||
states whether a complete prior bundle, partial remnants, or no prior bundle
|
||||
remains, or that recovery state could not be inspected. It does not expose
|
||||
provider diagnostics, filesystem causes, or recovery paths. See the
|
||||
provider diagnostics, filesystem causes, or recovery paths. A provider HTTP
|
||||
failure may include its numeric status in the safe message. See the
|
||||
[comparison bundle contract](integrations/comparison-bundle.md) for durable
|
||||
artifact fields and failure invariants.
|
||||
|
||||
|
||||
@@ -203,16 +203,23 @@ for platform availability, security, and retention requirements.
|
||||
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
|
||||
They are separate decisions. An explicit `profile` applies to every selected
|
||||
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
|
||||
Tomorrow select `weather-balanced` through their exact `2.0.0` prompt
|
||||
Tomorrow select `weather-balanced` through their exact `2.1.0` prompt
|
||||
definitions.
|
||||
|
||||
Promptkit resolves a selected profile definition from a test or embedding
|
||||
consumer's explicit in-memory profile, then the configured `profile_file` or
|
||||
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
|
||||
built-in catalog. Sources provide complete definitions; fields are never
|
||||
merged. A matching malformed external profile fails rather than using the
|
||||
embedded definition. The [Promptkit integration guide](integrations/promptkit.md)
|
||||
owns the catalog and precedence details.
|
||||
maintained catalog. Weatherreporter's embedded `weather-*` definitions are small
|
||||
aliases of Promptkit's maintained base profiles, so Promptkit also resolves
|
||||
their inherited target and settings. A configured definition with the same ID
|
||||
as either a selected profile or an inherited base takes precedence. A matching
|
||||
malformed external profile fails rather than using the embedded definition. The
|
||||
[Promptkit integration guide](integrations/promptkit.md) owns the catalog and
|
||||
precedence details.
|
||||
|
||||
An endpoint-only profile may intentionally have no backend identity. Profiles
|
||||
that require a direct API key are rejected before collection, while Promptkit
|
||||
resolves optional environment credential sources during execution.
|
||||
|
||||
To replace the default Hourly definition with a local OpenAI-compatible
|
||||
endpoint, set `profile_file` to a copy of
|
||||
|
||||
@@ -9,7 +9,7 @@ and retention belong to the [operations guide](../operations.md).
|
||||
## Version And Layout
|
||||
|
||||
The current and only supported manifest schema version is
|
||||
`weatherreporter.comparison.v1`. A bundle directory contains exactly these
|
||||
`weatherreporter.comparison.v2`. A bundle directory contains exactly these
|
||||
regular, non-symlinked files:
|
||||
|
||||
```text
|
||||
@@ -51,7 +51,7 @@ this order:
|
||||
|
||||
```text
|
||||
position, profileId, backendId, modelName, status, validationStatus,
|
||||
reportPath, error
|
||||
repairAttempts, reportPath, error
|
||||
```
|
||||
|
||||
`startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not
|
||||
@@ -66,12 +66,16 @@ contiguous from one, profile IDs are distinct and nonblank, and
|
||||
`succeeded + failed == total`.
|
||||
|
||||
A successful result has `status: "succeeded"`, `validationStatus: "passed"`,
|
||||
and a non-negative `repairAttempts` count,
|
||||
a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for
|
||||
its position, total, and logical profile ID, and no `error`. A failed result has
|
||||
`status: "failed"`, no `reportPath`, and an `error` object with nonblank
|
||||
`category` and `message`. Its validation status is absent, `failed`, or
|
||||
`skipped`. Error messages are valid UTF-8 and no longer than 1,024 bytes.
|
||||
`backendId` and `validationStatus` are omitted when unavailable.
|
||||
`skipped`; it may also be `passed` when a WeatherReporter step after PromptKit
|
||||
validation failed. Error messages are valid UTF-8 and no longer than 1,024 bytes.
|
||||
`backendId` and `validationStatus` are omitted when unavailable. A failed
|
||||
result with any completed validation status must retain its non-negative
|
||||
`repairAttempts`; early operational failures omit both fields.
|
||||
|
||||
Every successful Markdown file is declared by exactly one successful result.
|
||||
The directory contains no extra entries. Consumers can therefore verify the
|
||||
@@ -88,7 +92,9 @@ case-variant, or duplicate fields; multiple JSON values; extra entries;
|
||||
symlinks; and future or otherwise unsupported versions. Treat a bundle that
|
||||
fails recognition as an ordinary directory, not as a compatible bundle.
|
||||
|
||||
When replacing a recognized bundle, cancellation observed before the new
|
||||
Only v2 is recognized as a replaceable bundle; v1 is unsupported and must be
|
||||
moved or removed before a replacement at the same destination. When replacing
|
||||
a recognized bundle, cancellation observed before the new
|
||||
bundle is installed preserves the prior bundle rather than committing the
|
||||
replacement.
|
||||
|
||||
|
||||
@@ -1,10 +1,12 @@
|
||||
# Promptkit Integration
|
||||
|
||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.0.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.1.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
||||
|
||||
## Logical Profile Catalog
|
||||
|
||||
Prompt definitions select a stable Weatherreporter profile ID. The embedded definitions currently use Promptkit's `openrouter` backend:
|
||||
Prompt definitions select a stable Weatherreporter profile ID. Each embedded
|
||||
definition contains only its ID and one Promptkit base-profile reference; the
|
||||
effective execution settings resolve from Promptkit's maintained catalog:
|
||||
|
||||
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
@@ -12,22 +14,45 @@ Prompt definitions select a stable Weatherreporter profile ID. The embedded defi
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
|
||||
|
||||
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded profiles intentionally omit endpoints, credentials, temperature, `top_p`, and output-token limits.
|
||||
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
|
||||
profiles intentionally omit endpoints, credentials, and execution settings;
|
||||
Promptkit owns inherited resolution and its provider-native defaults. Promptkit
|
||||
assembles its maintained catalog from independently versioned OpenRouter and
|
||||
Rakestrawhome modules selected by the Promptkit release. Weatherreporter does
|
||||
not import or register those catalog modules directly. The maintained
|
||||
`rakestrawhome-gemma-4-31b` profile is also available for ordinary and
|
||||
comparison selection and reports the `rakestrawhome` backend without
|
||||
Weatherreporter-specific configuration.
|
||||
|
||||
## Selection And Active Execution
|
||||
|
||||
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank backend and model identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
|
||||
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank model; an endpoint-only profile may intentionally have no backend identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
|
||||
|
||||
1. explicit in-memory profiles used by an embedding consumer or test;
|
||||
2. the configured `profile_file` or `profile_dir`;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's built-in catalog.
|
||||
4. Promptkit's maintained catalog.
|
||||
|
||||
A source falls through only when the selected ID is absent. Each source supplies a complete definition, so profile fields are not merged. A malformed matching operator definition is an error and does not fall back.
|
||||
A source falls through only when the selected ID is absent. Promptkit resolves a
|
||||
derived profile's base with the same source precedence, so a configured base
|
||||
can shadow a built-in base. A missing, cyclic, malformed, or incomplete
|
||||
selected inheritance chain is an error and does not fall back.
|
||||
|
||||
Profiles that require a direct API key are unsupported; a profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. Active results retain the selected logical profile ID and resolved backend and model. Ordinary errors, summaries, logs, and outputs exclude endpoints, credentials, rendered messages, schemas, request bodies, response bodies, and complete parameter maps.
|
||||
Profiles that require a direct API key are unsupported. Optional environment
|
||||
credential sources are Promptkit runtime concerns and are not checked by
|
||||
Weatherreporter during profile inspection. Active results retain the selected
|
||||
logical profile ID and resolved backend and model. Ordinary errors, summaries,
|
||||
logs, and outputs exclude endpoints, credentials, rendered messages, schemas,
|
||||
request bodies, response bodies, and complete parameter maps.
|
||||
|
||||
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
|
||||
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions. Ordinary generation errors disclose only the safe Weatherreporter category and optional HTTP status; provider code, type, and message are written only to the explicit secure failure-debug artifact.
|
||||
|
||||
Each embedded prompt permits one Promptkit-owned corrective generation after an
|
||||
eligible failed or explicitly empty result. This is not an application retry:
|
||||
Weatherreporter performs no provider retry, profile fallback, or request-level
|
||||
output-contract override. Promptkit reports cumulative usage and the actual
|
||||
number of corrective calls; repair exhaustion remains a completed validation
|
||||
failure.
|
||||
|
||||
When capture is enabled, its preparation artifact projects a provider endpoint
|
||||
to its scheme and host and retains only reviewed execution settings. Provider
|
||||
|
||||
@@ -7,7 +7,7 @@ is owned by the [CLI reference](../cli.md) and [operations guide](../operations.
|
||||
|
||||
## Single-Report Flow
|
||||
|
||||
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. The resolved profile, backend, and model are carried in the active result.
|
||||
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. Profile inspection requires a model, permits an empty backend identity for endpoint-only profiles, and leaves inherited resolution and optional credential sources to Promptkit. The resolved profile, backend, model, and actual repair count (once a completed execution exists) are carried in the active result; the configured repair budget remains part of the exact prompt contract.
|
||||
|
||||
The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run.
|
||||
|
||||
|
||||
@@ -21,8 +21,9 @@ Before accepting generated JSON, the execution boundary reconciles the prepared
|
||||
report definition, inspected prompt hash and selected profile identity, the one
|
||||
preparation callback, and the completed Promptkit result. The callback and
|
||||
completion must agree on prompt, profile, backend, model, and rendered/input
|
||||
hashes; the callback output and completed validation must name the prepared
|
||||
report's JSON Schema. A mismatch produces no rendered Markdown and leaves
|
||||
hashes; the callback output must also carry the prepared report's configured
|
||||
repair budget, while the completed validation records the actual corrective
|
||||
calls used within that budget. A mismatch produces no rendered Markdown and leaves
|
||||
results with only the inspected safe identity.
|
||||
|
||||
Single-report generation executes one prepared profile and publishes its
|
||||
|
||||
@@ -2,9 +2,9 @@
|
||||
|
||||
`internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter.
|
||||
|
||||
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, the embedded fallback catalog, and its built-in catalog; the adapter does not parse profile YAML, merge sources, or probe endpoints.
|
||||
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, embedded profile aliases and their bases, and its maintained catalog. Promptkit privately assembles that catalog from the independently versioned OpenRouter and Rakestrawhome catalog modules selected by its release; Weatherreporter neither imports nor registers them. The adapter does not parse profile YAML, resolve inheritance, merge sources, inspect optional environment credentials, or probe endpoints.
|
||||
|
||||
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
|
||||
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. An empty backend identity remains valid for an endpoint-only profile; a nonblank model is required. PromptKit's configured repair-call budget and the completed result's actual corrective-call count are retained, along with its cumulative provider usage and final candidate. Structured provider generation failures become project-owned redacted generation errors that retain only bounded details through explicit accessors. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
|
||||
|
||||
The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications.
|
||||
|
||||
|
||||
@@ -122,7 +122,9 @@ The destination is preflighted before prompt inspection and collection, then
|
||||
rechecked immediately before an atomic publish. A missing or empty directory
|
||||
is usable. A nonempty directory can be replaced only when `--replace` is given
|
||||
and it is recognized as a current Weatherreporter comparison bundle; ordinary
|
||||
directories, symlinks, and unsafe destinations are rejected. Cancellation and
|
||||
directories, symlinks, and unsafe destinations are rejected. Existing v1
|
||||
bundles are not recognized for replacement: move or remove them first.
|
||||
Cancellation and
|
||||
all failures before publication preserve an existing bundle, including a
|
||||
cancellation observed while a replacement is being prepared. If guarded
|
||||
restoration cannot complete, the error names the retained sibling bundle for
|
||||
@@ -176,6 +178,12 @@ Preparation captures retain only the provider endpoint origin and reviewed
|
||||
execution settings. URL user information, paths, queries, fragments, and
|
||||
unrecognized provider parameters are omitted.
|
||||
|
||||
Each run directory may contain `preparation.json` (v3), `execution.json` (v3),
|
||||
and, for a provider generation failure, `failure.json` (v1). The failure
|
||||
artifact retains the safe category, HTTP status, and provider code, type, and
|
||||
message for trusted debugging only. Ordinary command output never includes
|
||||
those provider details.
|
||||
|
||||
Capture writes are confined to the requested root and fail if an unsafe
|
||||
filesystem component prevents secure artifact creation.
|
||||
|
||||
|
||||
@@ -50,10 +50,14 @@ directly.
|
||||
- Prompts receive curated module packages, never unbounded raw weather payloads.
|
||||
- Every execution validates the exact prompt version and output contract before
|
||||
collection. The selected profile is configured explicitly or declared by the
|
||||
prompt; unsupported direct-key profiles and missing reported credentials fail
|
||||
before collection.
|
||||
prompt; profiles requiring unsupported direct API keys fail before collection.
|
||||
A profile may have an empty backend identity when it supplies an endpoint;
|
||||
PromptKit resolves inherited profiles and optional credential sources when it
|
||||
executes them.
|
||||
- Prompt and profile validation completes before weather collection. Raw output
|
||||
is validated before template rendering.
|
||||
- PromptKit may make at most the prompt contract's one corrective generation;
|
||||
exhaustion is a validation rejection, not an application-level retry.
|
||||
- Comparison validates every explicit profile before collection, prepares one
|
||||
immutable report input, and delegates backend capacity to Promptkit rather
|
||||
than adding an application-wide execution limit.
|
||||
@@ -62,6 +66,8 @@ directly.
|
||||
- Sensitive rendered prompts, schemas, input bodies, provider endpoints, and
|
||||
credentials never enter normal summaries or logs. They are written only to
|
||||
an explicit secure debug root when requested.
|
||||
- Provider-controlled diagnostics never enter ordinary outputs; they are
|
||||
retained only in explicit secure failure-debug artifacts.
|
||||
|
||||
## Output, Notification, And Testing Invariants
|
||||
|
||||
|
||||
64
docs/releases/v0.13.0.md
Normal file
64
docs/releases/v0.13.0.md
Normal file
@@ -0,0 +1,64 @@
|
||||
# Weatherreporter v0.13.0
|
||||
|
||||
This release upgrades Weatherreporter's generated-text integration through
|
||||
PromptKit v0.9.0 and adds bounded structured-output repair, inherited domain
|
||||
profiles, safer provider diagnostics, and repair provenance.
|
||||
|
||||
## Summary
|
||||
|
||||
Weatherreporter now permits one PromptKit-owned corrective generation when a
|
||||
model returns invalid or explicitly empty structured output. Successful and
|
||||
failed results report the actual correction count when validation completed,
|
||||
and explicit prompt-debug capture can retain bounded provider failure details
|
||||
without exposing them in ordinary output.
|
||||
|
||||
The stable `weather-light`, `weather-balanced`, and `weather-deep` profiles now
|
||||
inherit their execution targets from PromptKit's maintained catalog. PromptKit
|
||||
v0.9.0 assembles that catalog from independently versioned OpenRouter and
|
||||
Rakestrawhome modules while preserving Weatherreporter's existing effective
|
||||
profile targets and configured-profile override behavior.
|
||||
|
||||
## Compatibility
|
||||
|
||||
The documented CLI, configuration fields, report IDs, report filenames,
|
||||
Markdown outputs, and Distributor integration remain compatible with v0.12.0.
|
||||
The four generated-text prompts advance to version `2.1.0`; an eligible failed
|
||||
validation may therefore make one additional provider call, increasing latency
|
||||
and usage for that execution.
|
||||
|
||||
Comparison bundles advance from schema v1 to v2 so each completed validation
|
||||
can record `repairAttempts`. Weatherreporter intentionally recognizes only the
|
||||
current bundle version for guarded replacement. An existing v1 comparison
|
||||
bundle cannot be replaced with `--replace` by this release.
|
||||
|
||||
## Upgrade
|
||||
|
||||
No special action is required for ordinary generation, batch execution,
|
||||
configuration, or notification. Operators who want to reuse the destination
|
||||
of a v1 comparison bundle must first remove or relocate that old bundle, or
|
||||
select a new `--out-dir`.
|
||||
|
||||
Operators should account for the possibility of one additional model call
|
||||
when output repair is needed. Existing configured standalone, inherited, and
|
||||
endpoint-only profiles retain their documented precedence.
|
||||
|
||||
## Changes
|
||||
|
||||
- Upgraded PromptKit from v0.5.0 through v0.9.0 and adopted its maintained
|
||||
profile inheritance, optional credential-source behavior, structured repair,
|
||||
generation-error contract, and externally maintained provider catalogs.
|
||||
- Converted the three Weatherreporter domain profiles into minimal aliases of
|
||||
maintained PromptKit profiles without changing their effective model ladder.
|
||||
- Added one corrective-generation allowance to all four embedded prompts and
|
||||
exposed actual repair counts in generate, batch, and comparison summaries.
|
||||
- Migrated the [comparison bundle contract](../integrations/comparison-bundle.md)
|
||||
to v2 with strict repair-provenance validation and safe partial publication
|
||||
after independent profile failures.
|
||||
- Added safe provider HTTP status to ordinary errors while confining provider
|
||||
code, type, and message to explicit secure `failure.json` debug artifacts.
|
||||
- Strengthened offline coverage for local endpoint-only profiles, optional
|
||||
credentials, concurrent comparison repair outcomes, and failure-debug
|
||||
isolation.
|
||||
- Updated the [PromptKit integration guide](../integrations/promptkit.md),
|
||||
[configuration reference](../config.md), [CLI reference](../cli.md), and
|
||||
[operations guide](../operations.md) for the implemented contracts.
|
||||
8
go.mod
8
go.mod
@@ -6,9 +6,13 @@ require gopkg.in/yaml.v3 v3.0.1
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/distributor v0.5.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.9.0
|
||||
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
|
||||
golang.org/x/sys v0.45.0
|
||||
)
|
||||
|
||||
require golang.org/x/text v0.14.0 // indirect
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 // indirect
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 // indirect
|
||||
golang.org/x/text v0.14.0 // indirect
|
||||
)
|
||||
|
||||
8
go.sum
8
go.sum
@@ -1,7 +1,11 @@
|
||||
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
|
||||
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.9.0 h1:IpvDRC8L6xRxQ9hpuyKOmMc5b6MeLTKYyx+h1YAjy08=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.9.0/go.mod h1:oMJ/WUJImUtwJ5e+6MAGECPYAErAkOaKel0G+3T/b4E=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 h1:lc062euk2qseO//D762i3JaFyulDNML3eQQX7DkYTho=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0/go.mod h1:AIa7kAu2mfrRQgcspe4L+DW51WqgnALQT60lqkEywJI=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 h1:j9YY7wsTVjzke2kHH4YAzpU0oUpM+x+nXwl1IeS+2eg=
|
||||
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0/go.mod h1:4RNS+LILDg4JbS4Ts9Lwy1C92wauXJIbeQaalps4Koo=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
|
||||
|
||||
@@ -153,6 +153,7 @@ func outputContract(value promptkit.OutputContract) promptexec.OutputContract {
|
||||
Format: string(value.Format),
|
||||
ValidationMode: string(value.ValidationMode),
|
||||
SchemaPath: value.SchemaPath,
|
||||
RepairAttempts: value.RepairAttempts,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -193,6 +194,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
|
||||
promptexec.ValidationStatus(value.Validation.Status),
|
||||
string(value.Validation.Mode),
|
||||
value.Validation.SchemaPath,
|
||||
value.Validation.RepairAttempts,
|
||||
value.Validation.Errors,
|
||||
)
|
||||
rawOutput := []byte(nil)
|
||||
@@ -203,6 +205,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
|
||||
promptexec.ValidationFailed,
|
||||
string(value.Validation.Mode),
|
||||
value.Validation.SchemaPath,
|
||||
value.Validation.RepairAttempts,
|
||||
[]string{"generated output exceeds the configured size limit"},
|
||||
)
|
||||
}
|
||||
@@ -299,6 +302,16 @@ func classifyError(err error) error {
|
||||
if errors.As(err, &capacityError) {
|
||||
return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err)
|
||||
}
|
||||
var generationError *promptkit.GenerationError
|
||||
if errors.As(err, &generationError) {
|
||||
return promptexec.NewGenerationError(
|
||||
generationError.StatusCode(),
|
||||
generationError.ProviderCode(),
|
||||
generationError.ProviderType(),
|
||||
generationError.ProviderMessage(),
|
||||
err,
|
||||
)
|
||||
}
|
||||
switch {
|
||||
case errors.Is(err, promptkit.ErrInvalidConfig):
|
||||
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err)
|
||||
|
||||
@@ -4,11 +4,15 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"net/http/httptest"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"testing/fstest"
|
||||
"time"
|
||||
|
||||
promptkit "gitea.maximumdirect.net/eric/promptkit"
|
||||
@@ -20,12 +24,19 @@ type fakeClient struct {
|
||||
mu sync.Mutex
|
||||
response *promptkit.GenerateResponse
|
||||
err error
|
||||
outcomes []generationOutcome
|
||||
next int
|
||||
calls int
|
||||
requests []promptkit.GenerateRequest
|
||||
block bool
|
||||
started chan struct{}
|
||||
}
|
||||
|
||||
type generationOutcome struct {
|
||||
response *promptkit.GenerateResponse
|
||||
err error
|
||||
}
|
||||
|
||||
type recordingReader struct {
|
||||
ref promptkit.ArtifactRef
|
||||
}
|
||||
@@ -49,6 +60,11 @@ func (client *fakeClient) Generate(ctx context.Context, request promptkit.Genera
|
||||
started := client.started
|
||||
response := client.response
|
||||
err := client.err
|
||||
if client.next < len(client.outcomes) {
|
||||
outcome := client.outcomes[client.next]
|
||||
client.next++
|
||||
response, err = outcome.response, outcome.err
|
||||
}
|
||||
client.mu.Unlock()
|
||||
if started != nil {
|
||||
started <- struct{}{}
|
||||
@@ -102,13 +118,19 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
|
||||
return client.requests[0]
|
||||
}
|
||||
|
||||
func (client *fakeClient) allRequests() []promptkit.GenerateRequest {
|
||||
client.mu.Lock()
|
||||
defer client.mu.Unlock()
|
||||
return append([]promptkit.GenerateRequest(nil), client.requests...)
|
||||
}
|
||||
|
||||
func TestInspectPromptAndProfile(t *testing.T) {
|
||||
adapter := newTestAdapter(t, &fakeClient{})
|
||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.0.0")
|
||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.1.0")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPrompt() error = %v", err)
|
||||
}
|
||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != "weather-balanced" {
|
||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != "weather-balanced" || inspection.Output.RepairAttempts != 1 {
|
||||
t.Fatalf("inspection = %#v", inspection)
|
||||
}
|
||||
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
|
||||
@@ -175,15 +197,38 @@ model: file-light
|
||||
}
|
||||
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
|
||||
|
||||
directory := testProfileDirectory(t, `id: weather-light
|
||||
directory := testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
|
||||
backend: local
|
||||
model: directory-light
|
||||
`)
|
||||
`})
|
||||
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
|
||||
if err != nil {
|
||||
t.Fatalf("New(profile directory) error = %v", err)
|
||||
}
|
||||
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
|
||||
|
||||
derived := writeProfileFile(t, `id: weather-light
|
||||
base_profile: gemini-flash-latest
|
||||
`)
|
||||
derivedAdapter, err := New(Config{ProfileFile: derived})
|
||||
if err != nil {
|
||||
t.Fatalf("New(derived profile) error = %v", err)
|
||||
}
|
||||
assertProfile(t, derivedAdapter, "weather-light", "openrouter", "~google/gemini-flash-latest")
|
||||
}
|
||||
|
||||
func TestConfiguredBaseProfileOverridesEmbeddedProfileTarget(t *testing.T) {
|
||||
directory := testProfileDirectory(t, map[string]string{
|
||||
"deepseek.yml": `id: deepseek-4-flash
|
||||
backend: local
|
||||
model: shadowed-deepseek
|
||||
`,
|
||||
})
|
||||
adapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
assertProfile(t, adapter, "weather-light", promptkit.BackendLocal, "shadowed-deepseek")
|
||||
}
|
||||
|
||||
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
|
||||
@@ -194,19 +239,69 @@ func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T)
|
||||
assertProfile(t, adapter, "weather-light", "", "weather-local")
|
||||
}
|
||||
|
||||
func TestMaintainedWeatherLightLocalProfileExampleExecutesThroughProductionClient(t *testing.T) {
|
||||
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
credentialSource string
|
||||
}{
|
||||
{name: "without credential source"},
|
||||
{name: "with blank optional credential source", credentialSource: "\napi_key_env: WEATHERREPORTER_TEST_MISSING_KEY\n"},
|
||||
} {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
var authorization string
|
||||
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
|
||||
authorization = request.Header.Get("Authorization")
|
||||
writer.Header().Set("Content-Type", "application/json")
|
||||
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":12,"completion_tokens":8,"total_tokens":20}}`, validResponse().Content)
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
example, err := os.ReadFile(filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml"))
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
profile := strings.Replace(string(example), "http://127.0.0.1:11434/v1", server.URL+"/v1", 1) + test.credentialSource
|
||||
adapter, err := New(Config{ProfileFile: writeProfileFile(t, profile)})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
request := testExecuteRequest()
|
||||
request.ProfileID = "weather-light"
|
||||
var preparation promptexec.Preparation
|
||||
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
|
||||
preparation = value
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
if authorization != "" {
|
||||
t.Fatalf("Authorization header = %q, want absent", authorization)
|
||||
}
|
||||
if preparation.ProfileID != "weather-light" || preparation.BackendID != "" || preparation.ModelName != "weather-local" || preparation.Output.RepairAttempts != 1 {
|
||||
t.Fatalf("preparation = %#v", preparation)
|
||||
}
|
||||
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" || result.Validation.Status != promptexec.ValidationPassed || result.Validation.RepairAttempts != 0 {
|
||||
t.Fatalf("execution = %#v", result)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
|
||||
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile
|
||||
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: other-profile
|
||||
backend: openrouter
|
||||
model: other-model
|
||||
`)})
|
||||
`})})
|
||||
if err != nil {
|
||||
t.Fatalf("New(absent profile) error = %v", err)
|
||||
}
|
||||
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||
|
||||
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light
|
||||
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
|
||||
backend: openrouter
|
||||
`)})
|
||||
`})})
|
||||
if err != nil {
|
||||
t.Fatalf("New(malformed profile) error = %v", err)
|
||||
}
|
||||
@@ -215,6 +310,78 @@ backend: openrouter
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileResolutionReturnsConfiguredInheritanceFailures(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
profile string
|
||||
profiles map[string]string
|
||||
}{
|
||||
{
|
||||
name: "missing base",
|
||||
profile: "missing-base",
|
||||
profiles: map[string]string{"missing.yml": `id: missing-base
|
||||
base_profile: unavailable
|
||||
`},
|
||||
},
|
||||
{
|
||||
name: "cyclic bases",
|
||||
profile: "first",
|
||||
profiles: map[string]string{
|
||||
"first.yml": `id: first
|
||||
base_profile: second
|
||||
`,
|
||||
"second.yml": `id: second
|
||||
base_profile: first
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "malformed base",
|
||||
profile: "child",
|
||||
profiles: map[string]string{
|
||||
"child.yml": `id: child
|
||||
base_profile: malformed
|
||||
`,
|
||||
"malformed.yml": `id: malformed
|
||||
base_profile: [not-a-profile]
|
||||
`,
|
||||
},
|
||||
},
|
||||
{
|
||||
name: "incomplete target",
|
||||
profile: "incomplete",
|
||||
profiles: map[string]string{"incomplete.yml": `id: incomplete
|
||||
backend: openrouter
|
||||
`},
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
adapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, test.profiles)})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
if _, err := adapter.InspectProfile(context.Background(), test.profile); err == nil {
|
||||
t.Fatal("InspectProfile() error = nil, want configured inheritance error")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRakestrawhomeBuiltInProfileInspectsOffline(t *testing.T) {
|
||||
adapter, err := New(Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
profile, err := adapter.InspectProfile(context.Background(), "rakestrawhome-gemma-4-31b")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile() error = %v", err)
|
||||
}
|
||||
if profile.ProfileID != "rakestrawhome-gemma-4-31b" || profile.BackendID != "rakestrawhome" || profile.ModelName == "" {
|
||||
t.Fatalf("profile = %#v", profile)
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
|
||||
adapter, err := New(Config{})
|
||||
if err != nil {
|
||||
@@ -287,7 +454,7 @@ func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
|
||||
}
|
||||
request := promptexec.ExecuteRequest{
|
||||
PromptID: "weather.hourly_generated_text",
|
||||
PromptVersion: "2.0.0",
|
||||
PromptVersion: "2.1.0",
|
||||
ProfileID: "weather-light",
|
||||
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
|
||||
}
|
||||
@@ -400,6 +567,114 @@ func TestExecuteReturnsCompletedValidationRejection(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteMapsCorrectiveGenerationResults(t *testing.T) {
|
||||
valid := `{"summary":"valid"}`
|
||||
invalid := `{"summary":42}`
|
||||
tests := []struct {
|
||||
name string
|
||||
outcomes []generationOutcome
|
||||
wantStatus promptexec.ValidationStatus
|
||||
wantRepairs int
|
||||
wantCalls int
|
||||
wantRaw string
|
||||
wantUsage promptexec.TokenUsage
|
||||
}{
|
||||
{
|
||||
name: "first pass valid",
|
||||
outcomes: []generationOutcome{{response: generationResponse(valid, 2, 3, 5)}},
|
||||
wantStatus: promptexec.ValidationPassed,
|
||||
wantRepairs: 0,
|
||||
wantCalls: 1,
|
||||
wantRaw: valid,
|
||||
wantUsage: promptexec.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5},
|
||||
},
|
||||
{
|
||||
name: "empty output repaired",
|
||||
outcomes: []generationOutcome{{response: generationResponse("", 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
|
||||
wantStatus: promptexec.ValidationPassed,
|
||||
wantRepairs: 1,
|
||||
wantCalls: 2,
|
||||
wantRaw: valid,
|
||||
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
|
||||
},
|
||||
{
|
||||
name: "invalid output repaired",
|
||||
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
|
||||
wantStatus: promptexec.ValidationPassed,
|
||||
wantRepairs: 1,
|
||||
wantCalls: 2,
|
||||
wantRaw: valid,
|
||||
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
|
||||
},
|
||||
{
|
||||
name: "repair budget exhausted",
|
||||
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(invalid, 7, 11, 18)}},
|
||||
wantStatus: promptexec.ValidationFailed,
|
||||
wantRepairs: 1,
|
||||
wantCalls: 2,
|
||||
wantRaw: invalid,
|
||||
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
client := &fakeClient{outcomes: test.outcomes}
|
||||
adapter := newRepairAdapter(t, client, "https://repair.example/v1")
|
||||
var preparation promptexec.Preparation
|
||||
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
|
||||
preparation = value
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
if preparation.Output.RepairAttempts != 1 || result == nil || result.Validation.Status != test.wantStatus || result.Validation.RepairAttempts != test.wantRepairs || string(result.RawOutput) != test.wantRaw || result.Usage != test.wantUsage {
|
||||
t.Fatalf("preparation/result = %#v/%#v", preparation, result)
|
||||
}
|
||||
requests := client.allRequests()
|
||||
if len(requests) != test.wantCalls {
|
||||
t.Fatalf("provider requests = %d, want %d", len(requests), test.wantCalls)
|
||||
}
|
||||
if test.wantCalls == 2 && !reflect.DeepEqual(requests[0].Target, requests[1].Target) {
|
||||
t.Fatalf("corrective target = %#v, want same prepared identity as %#v", requests[1].Target, requests[0].Target)
|
||||
}
|
||||
if result.ProfileID != preparation.ProfileID || result.BackendID != preparation.BackendID || result.ModelName != preparation.ModelName || result.PromptID != preparation.PromptID || result.PromptVersion != preparation.PromptVersion || result.PromptHash != preparation.PromptHash {
|
||||
t.Fatalf("prepared/result identity = %#v/%#v", preparation, result)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteMapsCorrectiveGenerationError(t *testing.T) {
|
||||
const providerBody = `{"error":{"code":"repair-code","type":"repair-type","message":"repair-message"}}`
|
||||
calls := 0
|
||||
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) {
|
||||
calls++
|
||||
if calls == 1 {
|
||||
writer.Header().Set("Content-Type", "application/json")
|
||||
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":2,"completion_tokens":3,"total_tokens":5}}`, `{"summary":42}`)
|
||||
return
|
||||
}
|
||||
writer.Header().Set("Content-Type", "application/json")
|
||||
writer.WriteHeader(http.StatusUnprocessableEntity)
|
||||
_, _ = writer.Write([]byte(providerBody))
|
||||
}))
|
||||
defer server.Close()
|
||||
|
||||
adapter := newRepairAdapter(t, nil, server.URL)
|
||||
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), nil)
|
||||
if result != nil || err == nil || calls != 2 {
|
||||
t.Fatalf("result/error/calls = %#v/%v/%d", result, err, calls)
|
||||
}
|
||||
var generationError *promptexec.GenerationError
|
||||
if !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusUnprocessableEntity || generationError.ProviderCode() != "repair-code" || generationError.ProviderType() != "repair-type" || generationError.ProviderMessage() != "repair-message" {
|
||||
t.Fatalf("generation error = %#v", err)
|
||||
}
|
||||
if strings.Contains(err.Error(), "repair-message") || !errors.Is(err, promptkit.ErrLLMGenerate) {
|
||||
t.Fatalf("generation error = %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) {
|
||||
client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}}
|
||||
adapter := newTestAdapter(t, client)
|
||||
@@ -507,12 +782,12 @@ func TestNewValidatesConfiguration(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestLocalBackendAndMissingCredentialBehavior(t *testing.T) {
|
||||
func TestLocalBackendAndOptionalCredentialSourceBehavior(t *testing.T) {
|
||||
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
|
||||
profiles := testProfileDirectory(t, `id: local-profile
|
||||
profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: local-profile
|
||||
backend: local
|
||||
model: local-model
|
||||
`)
|
||||
`})
|
||||
adapter, err := newAdapterForTest(Config{
|
||||
ProfileDirectory: profiles,
|
||||
LocalEndpoint: "https://local.example/v1",
|
||||
@@ -529,11 +804,11 @@ model: local-model
|
||||
t.Fatalf("capacity classification = %v", got)
|
||||
}
|
||||
|
||||
credentialProfiles := testProfileDirectory(t, `id: credential-profile
|
||||
credentialProfiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: credential-profile
|
||||
endpoint: https://profile.example/v1
|
||||
model: test-model
|
||||
api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
|
||||
`)
|
||||
`})
|
||||
client := &fakeClient{response: validResponse()}
|
||||
credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client)
|
||||
if err != nil {
|
||||
@@ -546,8 +821,8 @@ api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
|
||||
request := testExecuteRequest()
|
||||
request.ProfileID = "credential-profile"
|
||||
result, err := credentialAdapter.Execute(context.Background(), request, nil)
|
||||
if result != nil || promptexec.CategoryOf(err) != promptexec.MissingCredential || client.callCount() != 0 {
|
||||
t.Fatalf("credential result/category/calls = %#v/%q/%d", result, promptexec.CategoryOf(err), client.callCount())
|
||||
if err != nil || result == nil || client.callCount() != 1 {
|
||||
t.Fatalf("credential result/error/calls = %#v/%v/%d", result, err, client.callCount())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -568,14 +843,14 @@ func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, mo
|
||||
|
||||
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
|
||||
t.Helper()
|
||||
profiles := testProfileDirectory(t, `id: test-profile
|
||||
profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: test-profile
|
||||
endpoint: https://profile.example/v1
|
||||
model: test-model
|
||||
temperature: 0.2
|
||||
max_tokens: 300
|
||||
top_p: 1
|
||||
timeout_seconds: 30
|
||||
`)
|
||||
`})
|
||||
options = append(options, promptkit.WithLLMClient(client))
|
||||
adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...)
|
||||
if err != nil {
|
||||
@@ -584,13 +859,15 @@ timeout_seconds: 30
|
||||
return adapter
|
||||
}
|
||||
|
||||
func testProfileDirectory(t *testing.T, profile string) string {
|
||||
func testProfileDirectory(t *testing.T, profiles map[string]string) string {
|
||||
t.Helper()
|
||||
profiles := t.TempDir()
|
||||
if err := os.WriteFile(filepath.Join(profiles, "profile.yml"), []byte(profile), 0o600); err != nil {
|
||||
t.Fatalf("write profile: %v", err)
|
||||
directory := t.TempDir()
|
||||
for name, profile := range profiles {
|
||||
if err := os.WriteFile(filepath.Join(directory, name), []byte(profile), 0o600); err != nil {
|
||||
t.Fatalf("write profile: %v", err)
|
||||
}
|
||||
}
|
||||
return profiles
|
||||
return directory
|
||||
}
|
||||
|
||||
func writeProfileFile(t *testing.T, profile string) string {
|
||||
@@ -605,7 +882,7 @@ func writeProfileFile(t *testing.T, profile string) string {
|
||||
func testExecuteRequest() promptexec.ExecuteRequest {
|
||||
return promptexec.ExecuteRequest{
|
||||
PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "2.0.0",
|
||||
PromptVersion: "2.1.0",
|
||||
ProfileID: "test-profile",
|
||||
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
|
||||
}
|
||||
@@ -618,6 +895,61 @@ func validResponse() *promptkit.GenerateResponse {
|
||||
}
|
||||
}
|
||||
|
||||
func generationResponse(content string, promptTokens int, completionTokens int, totalTokens int) *promptkit.GenerateResponse {
|
||||
return &promptkit.GenerateResponse{
|
||||
Content: content,
|
||||
Usage: promptkit.TokenUsage{
|
||||
PromptTokens: promptTokens,
|
||||
CompletionTokens: completionTokens,
|
||||
TotalTokens: totalTokens,
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func newRepairAdapter(t *testing.T, client promptkit.LLMClient, endpoint string) *Adapter {
|
||||
t.Helper()
|
||||
profiles := testProfileDirectory(t, map[string]string{"profile.yml": "id: repair-profile\nendpoint: " + endpoint + "\nmodel: repair-model\n"})
|
||||
options := []promptkit.Option{
|
||||
promptkit.WithPromptFS(fstest.MapFS{
|
||||
"repair.yml": &fstest.MapFile{Data: []byte(`id: weather.repair
|
||||
version: "1.0.0"
|
||||
default_profile: repair-profile
|
||||
inputs:
|
||||
- name: data_package
|
||||
required: true
|
||||
content_type: application/yaml
|
||||
messages:
|
||||
- role: user
|
||||
content: "{{input \"data_package\"}}"
|
||||
output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: repair.schema.json
|
||||
repair_attempts: 1
|
||||
`)}}, "."),
|
||||
promptkit.WithSchemaFS(fstest.MapFS{
|
||||
"repair.schema.json": &fstest.MapFile{Data: []byte(`{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"],"additionalProperties":false}`)},
|
||||
}, "."),
|
||||
}
|
||||
if client != nil {
|
||||
options = append(options, promptkit.WithLLMClient(client))
|
||||
}
|
||||
adapter, err := newAdapter(Config{ProfileDirectory: profiles}, options...)
|
||||
if err != nil {
|
||||
t.Fatalf("newAdapter() error = %v", err)
|
||||
}
|
||||
return adapter
|
||||
}
|
||||
|
||||
func repairExecuteRequest() promptexec.ExecuteRequest {
|
||||
return promptexec.ExecuteRequest{
|
||||
PromptID: "weather.repair",
|
||||
PromptVersion: "1.0.0",
|
||||
ProfileID: "repair-profile",
|
||||
DataPackage: []byte("report: repair\n"),
|
||||
}
|
||||
}
|
||||
|
||||
func hourlyValidResponse() *promptkit.GenerateResponse {
|
||||
return &promptkit.GenerateResponse{
|
||||
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,
|
||||
|
||||
@@ -89,6 +89,7 @@ type ReportResult struct {
|
||||
ModelName string
|
||||
SourceWarnings []weatherdata.SourceWarning
|
||||
ValidationStatus promptexec.ValidationStatus
|
||||
RepairAttempts *int
|
||||
LLMDebugPath string
|
||||
OutputPath string
|
||||
Notification *NotificationResult
|
||||
@@ -139,6 +140,7 @@ type BatchReportResult struct {
|
||||
ModelName string `json:"modelName,omitempty"`
|
||||
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
||||
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
|
||||
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
||||
OutputPath string `json:"outputPath,omitempty"`
|
||||
}
|
||||
@@ -457,6 +459,9 @@ func copyBatchReportDetails(item *BatchReportResult, result *ReportResult) {
|
||||
item.Timezone = result.Timezone
|
||||
item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
||||
item.ValidationStatus = result.ValidationStatus
|
||||
if result.RepairAttempts != nil {
|
||||
item.RepairAttempts = repairAttemptsPointer(*result.RepairAttempts)
|
||||
}
|
||||
}
|
||||
|
||||
func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) {
|
||||
|
||||
@@ -3,7 +3,6 @@ package app
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"time"
|
||||
|
||||
@@ -60,6 +59,7 @@ type ComparisonProfileResult struct {
|
||||
ModelName string
|
||||
Status string
|
||||
ValidationStatus promptexec.ValidationStatus
|
||||
RepairAttempts *int
|
||||
ReportPath string
|
||||
LLMDebugPath string
|
||||
Error *comparison.SafeError
|
||||
@@ -121,7 +121,7 @@ func compareDetailed(ctx context.Context, req ComparisonRequest, publish compari
|
||||
}
|
||||
defer func() { _ = debugWriter.Close() }()
|
||||
inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{
|
||||
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor, LookupEnv: os.LookupEnv,
|
||||
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor,
|
||||
})
|
||||
result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash
|
||||
if err != nil {
|
||||
@@ -198,6 +198,9 @@ func copyComparisonOutcomes(result *ComparisonResult, outcomes []comparisonProfi
|
||||
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
|
||||
Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error,
|
||||
}
|
||||
if outcome.RepairAttempts != nil {
|
||||
profile.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
|
||||
}
|
||||
if published && outcome.Status == comparison.StatusSucceeded {
|
||||
profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath)
|
||||
}
|
||||
@@ -227,6 +230,9 @@ func comparisonBundle(result *ComparisonResult, dataPackage []byte, outcomes []c
|
||||
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
|
||||
Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error,
|
||||
}
|
||||
if outcome.RepairAttempts != nil {
|
||||
manifestResult.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
|
||||
}
|
||||
if outcome.Status == comparison.StatusSucceeded {
|
||||
manifestResult.ReportPath = outcome.ReportPath
|
||||
bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown})
|
||||
|
||||
@@ -31,6 +31,7 @@ type comparisonProfileOutcome struct {
|
||||
ModelName string
|
||||
Status string
|
||||
ValidationStatus promptexec.ValidationStatus
|
||||
RepairAttempts *int
|
||||
ReportPath string
|
||||
Markdown []byte
|
||||
LLMDebugPath string
|
||||
@@ -108,6 +109,9 @@ func executeComparisonProfile(ctx context.Context, req comparisonExecutionReques
|
||||
})
|
||||
outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName
|
||||
outcome.ValidationStatus = execution.ValidationStatus
|
||||
if execution.RepairAttempts != nil {
|
||||
outcome.RepairAttempts = repairAttemptsPointer(*execution.RepairAttempts)
|
||||
}
|
||||
outcome.LLMDebugPath = execution.LLMDebugPath
|
||||
if err != nil {
|
||||
outcome.canceled = cancellationError(err)
|
||||
@@ -169,9 +173,14 @@ func comparisonExecutionMessage(err error) string {
|
||||
if errors.Is(err, context.DeadlineExceeded) {
|
||||
return "profile execution deadline exceeded"
|
||||
}
|
||||
operation := "profile execution"
|
||||
var execution *profileExecutionError
|
||||
if errors.As(err, &execution) {
|
||||
return comparison.TruncateErrorMessage(execution.operation + " failed")
|
||||
operation = execution.operation
|
||||
}
|
||||
return "profile execution failed"
|
||||
var generation *promptexec.GenerationError
|
||||
if errors.As(err, &generation) && generation.StatusCode() > 0 {
|
||||
return comparison.TruncateErrorMessage(fmt.Sprintf("%s failed (HTTP %d)", operation, generation.StatusCode()))
|
||||
}
|
||||
return comparison.TruncateErrorMessage(operation + " failed")
|
||||
}
|
||||
|
||||
@@ -4,9 +4,11 @@ import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"reflect"
|
||||
"strings"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
@@ -72,6 +74,66 @@ func TestExecuteComparisonProfilesContinuesAfterProfileFailure(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteComparisonProfilesPreservesIndependentRepairOutcomes(t *testing.T) {
|
||||
prepared, prompt := preparedDailyProfile(t)
|
||||
profiles := comparisonProfiles(4)
|
||||
executor := newBarrierExecutor(profiles)
|
||||
executor.setValidation(profiles[0].ProfileID, promptexec.ValidationPassed, 0)
|
||||
executor.setValidation(profiles[1].ProfileID, promptexec.ValidationPassed, 1)
|
||||
executor.setValidation(profiles[2].ProfileID, promptexec.ValidationFailed, 1)
|
||||
executor.setError(profiles[3].ProfileID, errors.New("provider failure"))
|
||||
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
|
||||
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", Executor: executor,
|
||||
}, executor)
|
||||
waitForProfileStarts(t, executor, profiles, results)
|
||||
executor.releaseAll()
|
||||
result := <-results
|
||||
wantStatuses := []string{comparison.StatusSucceeded, comparison.StatusSucceeded, comparison.StatusFailed, comparison.StatusFailed}
|
||||
wantValidations := []promptexec.ValidationStatus{promptexec.ValidationPassed, promptexec.ValidationPassed, promptexec.ValidationFailed, ""}
|
||||
wantRepairs := []*int{intPointer(0), intPointer(1), intPointer(1), nil}
|
||||
for index, outcome := range result.Outcomes {
|
||||
if outcome.Status != wantStatuses[index] || outcome.ValidationStatus != wantValidations[index] || !reflect.DeepEqual(outcome.RepairAttempts, wantRepairs[index]) {
|
||||
t.Fatalf("outcome[%d] = %#v, want status/validation/repairs %q/%q/%#v", index, outcome, wantStatuses[index], wantValidations[index], wantRepairs[index])
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteComparisonProfilesCapturesConcurrentProviderFailures(t *testing.T) {
|
||||
prepared, prompt := preparedDailyProfile(t)
|
||||
profiles := comparisonProfiles(2)
|
||||
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
|
||||
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
|
||||
t.Skipf("secure prompt debug capture is unavailable: %v", err)
|
||||
}
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptDebugWriter() error = %v", err)
|
||||
}
|
||||
markers := []string{"first-provider-private-marker", "second-provider-private-marker"}
|
||||
statuses := []int{http.StatusTooManyRequests, http.StatusServiceUnavailable}
|
||||
executor := newBarrierExecutor(profiles)
|
||||
for index, profile := range profiles {
|
||||
executor.setError(profile.ProfileID, promptexec.NewGenerationError(statuses[index], "provider_code", "provider_type", markers[index], nil))
|
||||
}
|
||||
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
|
||||
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", DebugWriter: debugWriter, Executor: executor,
|
||||
}, executor)
|
||||
waitForProfileStarts(t, executor, profiles, results)
|
||||
executor.releaseAll()
|
||||
result := <-results
|
||||
for index, outcome := range result.Outcomes {
|
||||
if outcome.Status != comparison.StatusFailed || outcome.Error == nil || outcome.Error.Category != string(promptexec.Generation) || outcome.Error.Message != fmt.Sprintf("execute prompt failed (HTTP %d)", statuses[index]) || strings.Contains(outcome.Error.Message, markers[index]) || outcome.LLMDebugPath == "" {
|
||||
t.Fatalf("outcome[%d] = %#v", index, outcome)
|
||||
}
|
||||
failure, readErr := os.ReadFile(filepath.Join(outcome.LLMDebugPath, "failure.json"))
|
||||
if readErr != nil {
|
||||
t.Fatal(readErr)
|
||||
}
|
||||
if !strings.Contains(string(failure), markers[index]) || strings.Contains(string(failure), markers[1-index]) {
|
||||
t.Fatalf("failure[%d] = %s", index, failure)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) {
|
||||
prepared, prompt := preparedDailyProfile(t)
|
||||
profiles := comparisonProfiles(4)
|
||||
@@ -139,6 +201,8 @@ type barrierExecutor struct {
|
||||
releases map[string]chan struct{}
|
||||
requests map[string]promptexec.ExecuteRequest
|
||||
errors map[string]error
|
||||
validations map[string]promptexec.ValidationStatus
|
||||
repairAttempts map[string]int
|
||||
profiles map[string]ComparisonProfileInspection
|
||||
inFlight int
|
||||
maximum int
|
||||
@@ -153,7 +217,7 @@ func newBarrierExecutor(profiles []ComparisonProfileInspection) *barrierExecutor
|
||||
}
|
||||
return &barrierExecutor{
|
||||
started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases,
|
||||
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, profiles: identities,
|
||||
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, validations: map[string]promptexec.ValidationStatus{}, repairAttempts: map[string]int{}, profiles: identities,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -170,7 +234,8 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
|
||||
e.mu.Lock()
|
||||
profile := e.profiles[req.ProfileID]
|
||||
e.mu.Unlock()
|
||||
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
|
||||
definition := generationDefinitionForPrompt(req.PromptID)
|
||||
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
|
||||
e.callbackFailures <- err
|
||||
return nil, err
|
||||
}
|
||||
@@ -194,15 +259,20 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
|
||||
e.mu.Lock()
|
||||
e.inFlight--
|
||||
err := e.errors[req.ProfileID]
|
||||
validationStatus := e.validations[req.ProfileID]
|
||||
repairAttempts := e.repairAttempts[req.ProfileID]
|
||||
e.mu.Unlock()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
if validationStatus == "" {
|
||||
validationStatus = promptexec.ValidationPassed
|
||||
}
|
||||
return &promptexec.Execution{
|
||||
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash,
|
||||
ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName,
|
||||
StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(),
|
||||
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil),
|
||||
Validation: promptexec.NewValidation(validationStatus, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -219,6 +289,13 @@ func (e *barrierExecutor) setError(profileID string, err error) {
|
||||
e.errors[profileID] = err
|
||||
}
|
||||
|
||||
func (e *barrierExecutor) setValidation(profileID string, status promptexec.ValidationStatus, repairAttempts int) {
|
||||
e.mu.Lock()
|
||||
defer e.mu.Unlock()
|
||||
e.validations[profileID] = status
|
||||
e.repairAttempts[profileID] = repairAttempts
|
||||
}
|
||||
|
||||
func (e *barrierExecutor) release(profileID string) {
|
||||
close(e.releases[profileID])
|
||||
}
|
||||
@@ -317,4 +394,8 @@ func bytesEqual(left, right []byte) bool {
|
||||
return reflect.DeepEqual(left, right)
|
||||
}
|
||||
|
||||
func intPointer(value int) *int {
|
||||
return &value
|
||||
}
|
||||
|
||||
var _ promptexec.Executor = (*barrierExecutor)(nil)
|
||||
|
||||
@@ -87,6 +87,40 @@ func TestCompareDetailedPublishesPartialBundleAndReturnsAggregateError(t *testin
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompareDetailedPublishesPostValidationProfileFailure(t *testing.T) {
|
||||
bundle := generationBundle(t)
|
||||
executor := &generationExecutor{complete: func(execution *promptexec.Execution) {
|
||||
if execution.ProfileID == "weather-deep" {
|
||||
execution.RawOutput = []byte(`{"summary":42}`)
|
||||
}
|
||||
}}
|
||||
result, err := CompareDetailed(context.Background(), ComparisonRequest{
|
||||
Config: comparisonConfig(), Report: ReportDaily, ProfileIDs: []string{"weather-light", "weather-deep"},
|
||||
WorkingDir: t.TempDir(), Date: generationTime("2026-05-29T12:00:00-05:00"),
|
||||
Clock: timeutil.FixedClock{Time: generationTime("2026-05-29T08:30:00-05:00")},
|
||||
Collector: &generationCollector{bundle: &bundle}, Executor: executor,
|
||||
})
|
||||
if err == nil || result == nil || result.Succeeded != 1 || result.Failed != 1 || result.ManifestPath == "" {
|
||||
t.Fatalf("CompareDetailed() result/error = %#v/%v", result, err)
|
||||
}
|
||||
failure := result.Results[1]
|
||||
if failure.Status != comparison.StatusFailed || failure.ValidationStatus != promptexec.ValidationPassed || failure.RepairAttempts == nil || *failure.RepairAttempts != 0 {
|
||||
t.Fatalf("post-validation failure = %#v", failure)
|
||||
}
|
||||
data, readErr := os.ReadFile(result.ManifestPath)
|
||||
if readErr != nil {
|
||||
t.Fatal(readErr)
|
||||
}
|
||||
var manifest comparison.Manifest
|
||||
if decodeErr := json.Unmarshal(data, &manifest); decodeErr != nil {
|
||||
t.Fatal(decodeErr)
|
||||
}
|
||||
manifestFailure := manifest.Results[1]
|
||||
if manifestFailure.ValidationStatus != "passed" || manifestFailure.RepairAttempts == nil || *manifestFailure.RepairAttempts != 0 {
|
||||
t.Fatalf("published post-validation failure = %#v", manifestFailure)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) {
|
||||
for _, test := range []struct {
|
||||
name string
|
||||
|
||||
@@ -4,6 +4,7 @@ import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"net/http"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
@@ -68,6 +69,7 @@ type generationExecutor struct {
|
||||
beforeExecute func(promptexec.ExecuteRequest)
|
||||
cancelBeforeReturn context.CancelFunc
|
||||
validation promptexec.ValidationStatus
|
||||
repairAttempts int
|
||||
validations map[string]promptexec.ValidationStatus
|
||||
rawOutput []byte
|
||||
waitForCancellation map[string]bool
|
||||
@@ -88,7 +90,7 @@ func (e *generationExecutor) InspectPrompt(_ context.Context, id, version string
|
||||
return promptexec.PromptInspection{}, e.inspectErr
|
||||
}
|
||||
definition := generationDefinitionForPrompt(id)
|
||||
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}}, nil
|
||||
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}}, nil
|
||||
}
|
||||
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
||||
generationExecutorMu.Lock()
|
||||
@@ -112,7 +114,8 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
|
||||
calls = 1
|
||||
}
|
||||
for range calls {
|
||||
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}
|
||||
definition := generationDefinitionForPrompt(req.PromptID)
|
||||
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}
|
||||
if prepare != nil {
|
||||
prepare(&preparation)
|
||||
}
|
||||
@@ -128,6 +131,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
|
||||
profileErr := e.executeErrors[req.ProfileID]
|
||||
executeErr := e.executeErr
|
||||
status := e.validation
|
||||
repairAttempts := e.repairAttempts
|
||||
if profileStatus, ok := e.validations[req.ProfileID]; ok {
|
||||
status = profileStatus
|
||||
}
|
||||
@@ -162,7 +166,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
|
||||
if cancelBeforeReturn != nil {
|
||||
cancelBeforeReturn()
|
||||
}
|
||||
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil)}
|
||||
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil)}
|
||||
if complete != nil {
|
||||
complete(execution)
|
||||
}
|
||||
@@ -546,6 +550,66 @@ func TestGenerateDetailedWritesRequestedPromptDebugArtifacts(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateDetailedCapturesProviderFailureOnlyInDebugArtifacts(t *testing.T) {
|
||||
bundle := generationBundle(t)
|
||||
debugRoot := t.TempDir()
|
||||
const marker = "provider-private-generation-marker"
|
||||
result, err := GenerateDetailed(context.Background(), GenerateRequest{
|
||||
Config: generationConfig(), Report: ReportDaily,
|
||||
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
|
||||
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle},
|
||||
Executor: &generationExecutor{executeErr: promptexec.NewGenerationError(http.StatusTooManyRequests, "rate_limit", "provider_error", marker, nil)},
|
||||
})
|
||||
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
|
||||
t.Skipf("secure prompt debug capture is unavailable: %v", err)
|
||||
}
|
||||
if err == nil || result == nil || result.LLMDebugPath == "" || promptexec.CategoryOf(err) != promptexec.Generation || !strings.Contains(err.Error(), "HTTP 429") || strings.Contains(err.Error(), marker) || result.OutputPath != "" {
|
||||
t.Fatalf("GenerateDetailed() result/error = %#v/%v", result, err)
|
||||
}
|
||||
for _, name := range []string{"preparation.json", "failure.json"} {
|
||||
if _, statErr := os.Stat(filepath.Join(result.LLMDebugPath, name)); statErr != nil {
|
||||
t.Fatalf("debug artifact %q: %v", name, statErr)
|
||||
}
|
||||
}
|
||||
data, readErr := os.ReadFile(filepath.Join(result.LLMDebugPath, "failure.json"))
|
||||
if readErr != nil || !strings.Contains(string(data), marker) {
|
||||
t.Fatalf("failure artifact = %q, error = %v", data, readErr)
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateDetailedPreservesProviderFailureWhenFailureDebugWriteFails(t *testing.T) {
|
||||
bundle := generationBundle(t)
|
||||
debugRoot := t.TempDir()
|
||||
const marker = "provider-private-write-failure-marker"
|
||||
var setupErr error
|
||||
executor := &generationExecutor{
|
||||
executeErr: promptexec.NewGenerationError(http.StatusServiceUnavailable, "unavailable", "provider_error", marker, nil),
|
||||
beforeExecute: func(promptexec.ExecuteRequest) {
|
||||
setupErr = filepath.Walk(debugRoot, func(path string, info os.FileInfo, err error) error {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if info.Name() == "preparation.json" {
|
||||
return os.Mkdir(filepath.Join(filepath.Dir(path), "failure.json"), 0o700)
|
||||
}
|
||||
return nil
|
||||
})
|
||||
},
|
||||
}
|
||||
result, err := GenerateDetailed(context.Background(), GenerateRequest{
|
||||
Config: generationConfig(), Report: ReportDaily,
|
||||
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
|
||||
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle}, Executor: executor,
|
||||
})
|
||||
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
|
||||
t.Skipf("secure prompt debug capture is unavailable: %v", err)
|
||||
}
|
||||
var generationError *promptexec.GenerationError
|
||||
if setupErr != nil || err == nil || result == nil || result.OutputPath != "" || promptexec.CategoryOf(err) != promptexec.Generation || !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusServiceUnavailable || !strings.Contains(err.Error(), "HTTP 503") || strings.Contains(err.Error(), marker) {
|
||||
t.Fatalf("GenerateDetailed() setup/result/error = %v/%#v/%v", setupErr, result, err)
|
||||
}
|
||||
}
|
||||
|
||||
func generationConfig() config.Config {
|
||||
cfg := config.Defaults()
|
||||
cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home"
|
||||
|
||||
@@ -2,6 +2,7 @@ package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"reflect"
|
||||
|
||||
@@ -24,6 +25,7 @@ type profileExecutionOutcome struct {
|
||||
BackendID string
|
||||
ModelName string
|
||||
ValidationStatus promptexec.ValidationStatus
|
||||
RepairAttempts *int
|
||||
LLMDebugPath string
|
||||
}
|
||||
|
||||
@@ -94,11 +96,24 @@ func executePreparedProfile(ctx context.Context, req profileExecutionRequest) (p
|
||||
if callbackFailed {
|
||||
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true}
|
||||
}
|
||||
if req.DebugWriter != nil && req.DebugWriter.Enabled() && req.DebugRef != nil {
|
||||
var generationError *promptexec.GenerationError
|
||||
if errors.As(err, &generationError) {
|
||||
path, debugErr := req.DebugWriter.WriteFailure(*req.DebugRef, generationError)
|
||||
if path != "" {
|
||||
outcome.LLMDebugPath = path
|
||||
}
|
||||
if debugErr != nil {
|
||||
err = errors.Join(err, promptDebugWriteError(debugErr))
|
||||
}
|
||||
}
|
||||
}
|
||||
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)}
|
||||
}
|
||||
if execution == nil {
|
||||
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)}
|
||||
}
|
||||
outcome.RepairAttempts = repairAttemptsPointer(execution.Validation.RepairAttempts)
|
||||
if preparationCount != 1 {
|
||||
return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()}
|
||||
}
|
||||
@@ -157,7 +172,7 @@ func validatePreparedExecutionRequest(req profileExecutionRequest) error {
|
||||
if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) {
|
||||
return promptProvenanceError()
|
||||
}
|
||||
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.BackendID == "" || req.Profile.ModelName == "" {
|
||||
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.ModelName == "" {
|
||||
return promptProvenanceError()
|
||||
}
|
||||
return nil
|
||||
@@ -178,12 +193,19 @@ func validateExecutionProvenance(req profileExecutionRequest, preparation prompt
|
||||
if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash ||
|
||||
execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) ||
|
||||
execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName ||
|
||||
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" {
|
||||
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" ||
|
||||
execution.Validation.RepairAttempts < 0 || execution.Validation.RepairAttempts > preparation.Output.RepairAttempts ||
|
||||
preparation.Output.RepairAttempts != definition.GeneratedTextRepairAttempts {
|
||||
return promptProvenanceError()
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func repairAttemptsPointer(value int) *int {
|
||||
copy := value
|
||||
return ©
|
||||
}
|
||||
|
||||
func promptProvenanceError() error {
|
||||
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil)
|
||||
}
|
||||
|
||||
@@ -26,7 +26,7 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("executePreparedProfile() error = %v", err)
|
||||
}
|
||||
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
|
||||
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 0 || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
|
||||
t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls)
|
||||
}
|
||||
if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) {
|
||||
@@ -34,6 +34,20 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutePreparedProfileRetainsCompletedRepairAttemptsOnLaterFailure(t *testing.T) {
|
||||
prepared, inspection := preparedDailyProfile(t)
|
||||
prepared.resolved.Definition.GeneratedTextRepairAttempts = 1
|
||||
executor := &generationExecutor{repairAttempts: 1, rawOutput: []byte(`{"summary":42}`), prepare: func(value *promptexec.Preparation) { value.Output.RepairAttempts = 1 }}
|
||||
outcome, _, err := executePreparedProfile(context.Background(), profileExecutionRequest{
|
||||
Prepared: prepared, Prompt: inspection,
|
||||
Profile: promptexec.ProfileInspection{ProfileID: inspection.ProfileID, BackendID: inspection.BackendID, ModelName: inspection.ModelName},
|
||||
Executor: executor,
|
||||
})
|
||||
if err == nil || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 1 {
|
||||
t.Fatalf("outcome/error = %#v/%v", outcome, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) {
|
||||
prepared, inspection := preparedDailyProfile(t)
|
||||
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
|
||||
|
||||
@@ -48,6 +48,9 @@ func generatePromptReport(ctx context.Context, req promptReportRequest) (*Report
|
||||
})
|
||||
result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName
|
||||
result.ValidationStatus = outcome.ValidationStatus
|
||||
if outcome.RepairAttempts != nil {
|
||||
result.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
|
||||
}
|
||||
result.LLMDebugPath = outcome.LLMDebugPath
|
||||
if err != nil {
|
||||
return result, generatedProfileExecutionError(req.Resolved, result.RunID, err)
|
||||
|
||||
@@ -2,7 +2,6 @@ package app
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/comparison"
|
||||
@@ -18,7 +17,6 @@ type PromptInspectionRequest struct {
|
||||
Resolved report.Resolved
|
||||
Executor promptexec.Executor
|
||||
Promptkit config.PromptkitConfig
|
||||
LookupEnv func(string) (string, bool)
|
||||
}
|
||||
|
||||
// PromptInspectionResult contains only safe identity and provenance from a
|
||||
@@ -39,7 +37,6 @@ type PromptExecutionsInspectionRequest struct {
|
||||
Resolved []report.Resolved
|
||||
Executor promptexec.Executor
|
||||
Promptkit config.PromptkitConfig
|
||||
LookupEnv func(string) (string, bool)
|
||||
}
|
||||
|
||||
// ComparisonInspectionRequest contains the explicit profile selection for one
|
||||
@@ -48,7 +45,6 @@ type ComparisonInspectionRequest struct {
|
||||
Resolved report.Resolved
|
||||
ProfileIDs []string
|
||||
Executor promptexec.Executor
|
||||
LookupEnv func(string) (string, bool)
|
||||
}
|
||||
|
||||
// ComparisonInspectionResult contains the safe, shared prompt identity and
|
||||
@@ -76,7 +72,6 @@ func InspectPromptExecution(ctx context.Context, req PromptInspectionRequest) (P
|
||||
Resolved: []report.Resolved{req.Resolved},
|
||||
Executor: req.Executor,
|
||||
Promptkit: req.Promptkit,
|
||||
LookupEnv: req.LookupEnv,
|
||||
})
|
||||
if err != nil {
|
||||
return PromptInspectionResult{}, err
|
||||
@@ -111,7 +106,7 @@ func InspectPromptExecutions(ctx context.Context, req PromptExecutionsInspection
|
||||
}
|
||||
profile, ok := profiles[profileID]
|
||||
if !ok {
|
||||
profile, err = inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv)
|
||||
profile, err = inspectPromptProfile(ctx, req.Executor, profileID)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -154,7 +149,7 @@ func InspectComparisonExecution(ctx context.Context, req ComparisonInspectionReq
|
||||
handler: handler,
|
||||
}
|
||||
for _, profileID := range req.ProfileIDs {
|
||||
profile, err := inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv)
|
||||
profile, err := inspectPromptProfile(ctx, req.Executor, profileID)
|
||||
if err != nil {
|
||||
return result, comparisonInspectionError("comparison profile inspection failed", err)
|
||||
}
|
||||
@@ -190,7 +185,7 @@ func inspectPromptContract(ctx context.Context, executor promptexec.Executor, de
|
||||
return inspection, nil
|
||||
}
|
||||
|
||||
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string, lookupEnv func(string) (string, bool)) (promptexec.ProfileInspection, error) {
|
||||
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string) (promptexec.ProfileInspection, error) {
|
||||
profile, err := executor.InspectProfile(ctx, profileID)
|
||||
if err != nil {
|
||||
return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err)
|
||||
@@ -201,16 +196,7 @@ func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, pro
|
||||
if profile.CredentialRequired {
|
||||
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil)
|
||||
}
|
||||
if strings.TrimSpace(profile.APIKeyEnv) != "" {
|
||||
if lookupEnv == nil {
|
||||
lookupEnv = os.LookupEnv
|
||||
}
|
||||
value, present := lookupEnv(profile.APIKeyEnv)
|
||||
if !present || strings.TrimSpace(value) == "" {
|
||||
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile credential is unavailable", nil)
|
||||
}
|
||||
}
|
||||
if strings.TrimSpace(profile.BackendID) == "" || strings.TrimSpace(profile.ModelName) == "" {
|
||||
if strings.TrimSpace(profile.ModelName) == "" {
|
||||
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil)
|
||||
}
|
||||
return profile, nil
|
||||
@@ -221,7 +207,7 @@ func validPromptInput(inputs []promptexec.InputDefinition) bool {
|
||||
}
|
||||
|
||||
func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool {
|
||||
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json"
|
||||
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" && output.RepairAttempts == definition.GeneratedTextRepairAttempts
|
||||
}
|
||||
|
||||
func promptInspectionError(operation string, err error) error {
|
||||
|
||||
@@ -50,7 +50,6 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
|
||||
name string
|
||||
prompt promptexec.PromptInspection
|
||||
profile promptexec.ProfileInspection
|
||||
lookupEnv func(string) (string, bool)
|
||||
wantCategory promptexec.ErrorCategory
|
||||
}{
|
||||
{
|
||||
@@ -81,9 +80,9 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
|
||||
wantCategory: promptexec.InvalidConfiguration,
|
||||
},
|
||||
{
|
||||
name: "missing profile backend",
|
||||
name: "missing profile model",
|
||||
prompt: basePrompt,
|
||||
profile: promptexec.ProfileInspection{ProfileID: "default-profile", ModelName: "model"},
|
||||
profile: promptexec.ProfileInspection{ProfileID: "default-profile", BackendID: "backend"},
|
||||
wantCategory: promptexec.InvalidConfiguration,
|
||||
},
|
||||
{
|
||||
@@ -92,18 +91,11 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
|
||||
profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true},
|
||||
wantCategory: promptexec.MissingCredential,
|
||||
},
|
||||
{
|
||||
name: "missing environment credential",
|
||||
prompt: basePrompt,
|
||||
profile: promptexec.ProfileInspection{ProfileID: "default-profile", APIKeyEnv: "PROMPT_API_KEY"},
|
||||
lookupEnv: func(string) (string, bool) { return "", false },
|
||||
wantCategory: promptexec.MissingCredential,
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}}
|
||||
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor, LookupEnv: test.lookupEnv})
|
||||
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor})
|
||||
if err == nil || promptexec.CategoryOf(err) != test.wantCategory {
|
||||
t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory)
|
||||
}
|
||||
@@ -265,13 +257,12 @@ func TestInspectComparisonExecutionStopsAtFirstProfileFailure(t *testing.T) {
|
||||
prompt: validPromptInspection(resolved.Definition),
|
||||
profiles: map[string]promptexec.ProfileInspection{
|
||||
"weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"},
|
||||
"missing-key": {ProfileID: "missing-key", APIKeyEnv: "PROMPT_API_KEY"},
|
||||
"missing-key": {ProfileID: "missing-key", CredentialRequired: true},
|
||||
"weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"},
|
||||
},
|
||||
}
|
||||
result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{
|
||||
Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor,
|
||||
LookupEnv: func(string) (string, bool) { return "", false },
|
||||
})
|
||||
if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential {
|
||||
t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err))
|
||||
@@ -359,7 +350,7 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
|
||||
return promptexec.PromptInspection{
|
||||
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
|
||||
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
|
||||
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
|
||||
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts},
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -2,8 +2,10 @@ package app_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -14,14 +16,12 @@ import (
|
||||
)
|
||||
|
||||
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
|
||||
lookupEnv := func(string) (string, bool) { return "test-key", true }
|
||||
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
|
||||
t.Helper()
|
||||
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||
Resolved: resolvedPromptProfile(t, id),
|
||||
Executor: adapter,
|
||||
Promptkit: config.PromptkitConfig{Profile: profile},
|
||||
LookupEnv: lookupEnv,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||
@@ -50,6 +50,55 @@ model: local-weather
|
||||
inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather")
|
||||
}
|
||||
|
||||
func TestPromptInspectionAcceptsMaintainedEndpointOnlyProfile(t *testing.T) {
|
||||
adapter, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: filepath.Join("..", "..", "examples", "weather-light-local-profile.yml")})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||
Resolved: resolvedPromptProfile(t, report.Hourly), Executor: adapter,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||
}
|
||||
if result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" {
|
||||
t.Fatalf("inspection = %#v", result)
|
||||
}
|
||||
if strings.Contains(fmt.Sprintf("%#v", result), "127.0.0.1") {
|
||||
t.Fatalf("inspection leaks endpoint: %#v", result)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptInspectionSupportsRakestrawhomeProfileOffline(t *testing.T) {
|
||||
adapter, err := promptkitadapter.New(promptkitadapter.Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
prompt, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||
Resolved: resolvedPromptProfile(t, report.Hourly),
|
||||
Executor: adapter,
|
||||
Promptkit: config.PromptkitConfig{Profile: "rakestrawhome-gemma-4-31b"},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||
}
|
||||
if prompt.ProfileID != "rakestrawhome-gemma-4-31b" || prompt.BackendID != "rakestrawhome" || prompt.ModelName == "" {
|
||||
t.Fatalf("prompt inspection = %#v", prompt)
|
||||
}
|
||||
|
||||
comparison, err := app.InspectComparisonExecution(context.Background(), app.ComparisonInspectionRequest{
|
||||
Resolved: resolvedPromptProfile(t, report.Hourly),
|
||||
ProfileIDs: []string{"rakestrawhome-gemma-4-31b", "weather-deep"},
|
||||
Executor: adapter,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InspectComparisonExecution() error = %v", err)
|
||||
}
|
||||
if len(comparison.Profiles) != 2 || comparison.Profiles[0].ProfileID != "rakestrawhome-gemma-4-31b" || comparison.Profiles[0].BackendID != "rakestrawhome" || comparison.Profiles[0].ModelName == "" {
|
||||
t.Fatalf("comparison inspection = %#v", comparison)
|
||||
}
|
||||
}
|
||||
|
||||
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
|
||||
t.Helper()
|
||||
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)
|
||||
|
||||
@@ -355,7 +355,7 @@ func comparisonResult(outputDirectory string, results []app.ComparisonProfileRes
|
||||
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||
result := &app.ComparisonResult{
|
||||
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
||||
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
|
||||
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),
|
||||
|
||||
@@ -132,7 +132,7 @@ func actionConfigPath(t *testing.T) string {
|
||||
func generatedReportResult() *app.ReportResult {
|
||||
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||
return &app.ReportResult{
|
||||
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0",
|
||||
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0",
|
||||
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
|
||||
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
|
||||
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,
|
||||
|
||||
@@ -43,6 +43,7 @@ type generateSummary struct {
|
||||
ModelName string `json:"modelName,omitempty"`
|
||||
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
|
||||
ValidationStatus string `json:"validationStatus,omitempty"`
|
||||
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||
Notification *generateNotificationSummary `json:"notification,omitempty"`
|
||||
Error string `json:"error,omitempty"`
|
||||
}
|
||||
@@ -106,6 +107,7 @@ type comparisonProfileSummary struct {
|
||||
ModelName string `json:"modelName"`
|
||||
Status string `json:"status"`
|
||||
ValidationStatus string `json:"validationStatus,omitempty"`
|
||||
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||
ReportPath string `json:"reportPath,omitempty"`
|
||||
LLMDebugPath string `json:"llmDebugPath,omitempty"`
|
||||
Error *comparison.SafeError `json:"error,omitempty"`
|
||||
@@ -129,6 +131,10 @@ func newGenerateSummary(result *app.ReportResult, err error) generateSummary {
|
||||
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
|
||||
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
|
||||
summary.ValidationStatus = string(result.ValidationStatus)
|
||||
if result.RepairAttempts != nil {
|
||||
value := *result.RepairAttempts
|
||||
summary.RepairAttempts = &value
|
||||
}
|
||||
summary.OutputPath = result.OutputPath
|
||||
summary.LLMDebugPath = result.LLMDebugPath
|
||||
summary.Notification = newGenerateNotificationSummary(result.Notification)
|
||||
@@ -162,6 +168,11 @@ func newGenerateNotificationSummary(result *app.NotificationResult) *generateNot
|
||||
return summary
|
||||
}
|
||||
|
||||
func repairAttemptsCopy(value int) *int {
|
||||
copy := value
|
||||
return ©
|
||||
}
|
||||
|
||||
func newBatchSummary(result *app.BatchResult, err error) batchSummary {
|
||||
summary := batchSummary{Command: commandRun}
|
||||
if result == nil {
|
||||
@@ -203,6 +214,9 @@ func newComparisonSummary(result *app.ComparisonResult, err error) comparisonSum
|
||||
Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath,
|
||||
LLMDebugPath: profile.LLMDebugPath, Error: profile.Error,
|
||||
})
|
||||
if profile.RepairAttempts != nil {
|
||||
summary.Results[len(summary.Results)-1].RepairAttempts = repairAttemptsCopy(*profile.RepairAttempts)
|
||||
}
|
||||
}
|
||||
summary.Status = comparisonSummaryStatus(result, err)
|
||||
if err != nil {
|
||||
|
||||
@@ -19,7 +19,7 @@ import (
|
||||
|
||||
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
|
||||
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
|
||||
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
|
||||
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
|
||||
t.Fatalf("summary = %#v", summary)
|
||||
}
|
||||
@@ -39,7 +39,7 @@ func TestComparisonSummaryUsesLockedOrderAndSafeFields(t *testing.T) {
|
||||
profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
|
||||
result := &app.ComparisonResult{
|
||||
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
|
||||
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
|
||||
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
||||
@@ -113,7 +113,7 @@ func TestComparisonSummaryClassifiesCompleteAndAllFailedResults(t *testing.T) {
|
||||
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
|
||||
failure := comparison.NewSafeError("generation", "execute prompt failed")
|
||||
complete := &app.ComparisonResult{
|
||||
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
|
||||
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
|
||||
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
|
||||
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
|
||||
Total: 2, Succeeded: 2,
|
||||
|
||||
@@ -15,7 +15,7 @@ import (
|
||||
|
||||
const (
|
||||
// SchemaVersion identifies the supported comparison manifest schema.
|
||||
SchemaVersion = "weatherreporter.comparison.v1"
|
||||
SchemaVersion = "weatherreporter.comparison.v2"
|
||||
|
||||
// ManifestFilename is the canonical name of a comparison manifest.
|
||||
ManifestFilename = "comparison.json"
|
||||
@@ -71,6 +71,7 @@ type Result struct {
|
||||
ModelName string `json:"modelName"`
|
||||
Status string `json:"status"`
|
||||
ValidationStatus string `json:"validationStatus,omitempty"`
|
||||
RepairAttempts *int `json:"repairAttempts,omitempty"`
|
||||
ReportPath string `json:"reportPath,omitempty"`
|
||||
Error *SafeError `json:"error,omitempty"`
|
||||
}
|
||||
@@ -276,11 +277,14 @@ func (manifest Manifest) Validate() error {
|
||||
if strings.TrimSpace(result.ModelName) == "" {
|
||||
return fmt.Errorf("result %d has a blank model name", result.Position)
|
||||
}
|
||||
if result.RepairAttempts != nil && *result.RepairAttempts < 0 {
|
||||
return fmt.Errorf("result %d has negative repair attempts", result.Position)
|
||||
}
|
||||
|
||||
switch result.Status {
|
||||
case StatusSucceeded:
|
||||
succeeded++
|
||||
if result.ValidationStatus != "passed" {
|
||||
if result.ValidationStatus != "passed" || result.RepairAttempts == nil {
|
||||
return fmt.Errorf("successful result %d did not pass validation", result.Position)
|
||||
}
|
||||
expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID)
|
||||
@@ -302,6 +306,9 @@ func (manifest Manifest) Validate() error {
|
||||
if err := result.Error.validate(); err != nil {
|
||||
return fmt.Errorf("failed result %d: %w", result.Position, err)
|
||||
}
|
||||
if result.ValidationStatus != "" && result.RepairAttempts == nil {
|
||||
return fmt.Errorf("failed result %d has validation without repair provenance", result.Position)
|
||||
}
|
||||
default:
|
||||
return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status)
|
||||
}
|
||||
@@ -324,7 +331,7 @@ func (safeError SafeError) validate() error {
|
||||
}
|
||||
|
||||
func isValidationStatus(status string) bool {
|
||||
return status == "" || status == "failed" || status == "skipped"
|
||||
return status == "" || status == "passed" || status == "failed" || status == "skipped"
|
||||
}
|
||||
|
||||
func isSHA256(value string) bool {
|
||||
|
||||
@@ -118,7 +118,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
|
||||
t.Fatalf("EncodeManifest() error = %v", err)
|
||||
}
|
||||
want := "{\n" +
|
||||
" \"schemaVersion\": \"weatherreporter.comparison.v1\",\n" +
|
||||
" \"schemaVersion\": \"weatherreporter.comparison.v2\",\n" +
|
||||
" \"comparisonId\": \"comparison_daily-2026-08-24\",\n" +
|
||||
" \"startedAt\": \"2026-08-24T12:00:00Z\",\n" +
|
||||
" \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" +
|
||||
@@ -146,6 +146,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
|
||||
" \"modelName\": \"gpt-5-mini\",\n" +
|
||||
" \"status\": \"succeeded\",\n" +
|
||||
" \"validationStatus\": \"passed\",\n" +
|
||||
" \"repairAttempts\": 0,\n" +
|
||||
" \"reportPath\": \"01-weather-light.md\"\n" +
|
||||
" },\n" +
|
||||
" {\n" +
|
||||
@@ -201,6 +202,9 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
|
||||
{name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }},
|
||||
{name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }},
|
||||
{name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }},
|
||||
{name: "failed passed result without repair provenance", mutate: func(manifest *Manifest) {
|
||||
manifest.Results[1].ValidationStatus = "passed"
|
||||
}},
|
||||
{name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }},
|
||||
{name: "duplicate report path", mutate: func(manifest *Manifest) {
|
||||
manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath}
|
||||
@@ -222,6 +226,17 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestManifestValidateAcceptsPostValidationFailure(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
manifest := validManifest()
|
||||
manifest.Results[1].ValidationStatus = "passed"
|
||||
manifest.Results[1].RepairAttempts = intPtr(0)
|
||||
if err := manifest.Validate(); err != nil {
|
||||
t.Fatalf("Manifest.Validate() rejected post-validation failure: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLogicalBundleValidate(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
@@ -309,6 +324,7 @@ func validManifest() Manifest {
|
||||
ModelName: "gpt-5-mini",
|
||||
Status: StatusSucceeded,
|
||||
ValidationStatus: "passed",
|
||||
RepairAttempts: intPtr(0),
|
||||
ReportPath: "01-weather-light.md",
|
||||
},
|
||||
{
|
||||
@@ -321,3 +337,8 @@ func validManifest() Manifest {
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func intPtr(value int) *int {
|
||||
copy := value
|
||||
return ©
|
||||
}
|
||||
|
||||
@@ -388,6 +388,7 @@ var resultFields = map[string]jsonValueValidator{
|
||||
"modelName": nil,
|
||||
"status": nil,
|
||||
"validationStatus": nil,
|
||||
"repairAttempts": validateRepairAttemptsJSON,
|
||||
"reportPath": nil,
|
||||
"error": validateSafeErrorJSON,
|
||||
}
|
||||
@@ -430,7 +431,7 @@ func validateResultsJSON(decoder *json.Decoder) error {
|
||||
return fmt.Errorf("results must be an array")
|
||||
}
|
||||
for decoder.More() {
|
||||
if err := validateJSONObject(decoder, resultFields); err != nil {
|
||||
if err := validateOrderedJSONObject(decoder, resultFields, []string{"position", "profileId", "backendId", "modelName", "status", "validationStatus", "repairAttempts", "reportPath", "error"}); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
@@ -444,6 +445,65 @@ func validateResultsJSON(decoder *json.Decoder) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateOrderedJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator, order []string) error {
|
||||
token, err := decoder.Token()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if delimiter, ok := token.(json.Delim); !ok || delimiter != '{' {
|
||||
return fmt.Errorf("manifest value must be an object")
|
||||
}
|
||||
seen := make(map[string]struct{}, len(fields))
|
||||
last := -1
|
||||
for decoder.More() {
|
||||
token, err := decoder.Token()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
name, ok := token.(string)
|
||||
if !ok {
|
||||
return fmt.Errorf("manifest field name is invalid")
|
||||
}
|
||||
validator, known := fields[name]
|
||||
if !known {
|
||||
return fmt.Errorf("manifest field %q is not canonical", name)
|
||||
}
|
||||
if _, duplicate := seen[name]; duplicate {
|
||||
return fmt.Errorf("manifest field %q is duplicated", name)
|
||||
}
|
||||
position := -1
|
||||
for index, candidate := range order {
|
||||
if candidate == name {
|
||||
position = index
|
||||
break
|
||||
}
|
||||
}
|
||||
if position <= last {
|
||||
return fmt.Errorf("manifest field %q is out of canonical order", name)
|
||||
}
|
||||
seen[name] = struct{}{}
|
||||
last = position
|
||||
if validator == nil {
|
||||
var value json.RawMessage
|
||||
if err := decoder.Decode(&value); err != nil {
|
||||
return err
|
||||
}
|
||||
continue
|
||||
}
|
||||
if err := validator(decoder); err != nil {
|
||||
return err
|
||||
}
|
||||
}
|
||||
token, err = decoder.Token()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if delimiter, ok := token.(json.Delim); !ok || delimiter != '}' {
|
||||
return fmt.Errorf("manifest object has an invalid terminator")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateSafeErrorJSON(decoder *json.Decoder) error {
|
||||
token, err := decoder.Token()
|
||||
if err != nil {
|
||||
@@ -458,6 +518,24 @@ func validateSafeErrorJSON(decoder *json.Decoder) error {
|
||||
return validateJSONObjectBody(decoder, safeErrorFields)
|
||||
}
|
||||
|
||||
func validateRepairAttemptsJSON(decoder *json.Decoder) error {
|
||||
var raw json.RawMessage
|
||||
if err := decoder.Decode(&raw); err != nil {
|
||||
return err
|
||||
}
|
||||
if bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
|
||||
return fmt.Errorf("repairAttempts must be an integer")
|
||||
}
|
||||
var value int
|
||||
if err := json.Unmarshal(raw, &value); err != nil {
|
||||
return err
|
||||
}
|
||||
if value < 0 {
|
||||
return fmt.Errorf("repairAttempts must not be negative")
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error {
|
||||
token, err := decoder.Token()
|
||||
if err != nil {
|
||||
|
||||
@@ -1,6 +1,7 @@
|
||||
package comparison
|
||||
|
||||
import (
|
||||
"bytes"
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
@@ -146,6 +147,23 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
|
||||
t.Helper()
|
||||
appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`)
|
||||
}},
|
||||
{name: "null repair attempts", mutate: func(t *testing.T, directory string) {
|
||||
t.Helper()
|
||||
path := filepath.Join(directory, ManifestFilename)
|
||||
data, err := os.ReadFile(path)
|
||||
if err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
before := []byte(" \"status\": \"failed\",\n \"error\": {")
|
||||
after := []byte(" \"status\": \"failed\",\n \"repairAttempts\": null,\n \"error\": {")
|
||||
data = bytes.Replace(data, before, after, 1)
|
||||
if bytes.Equal(data, readFile(t, path)) {
|
||||
t.Fatal("failed to add null repairAttempts")
|
||||
}
|
||||
if err := os.WriteFile(path, data, 0o600); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}},
|
||||
{name: "traversal report path", mutate: func(t *testing.T, directory string) {
|
||||
t.Helper()
|
||||
path := filepath.Join(directory, ManifestFilename)
|
||||
@@ -202,6 +220,30 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRepairAttemptsJSONRejectsNonIntegerValues(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, value := range []string{"null", "-1", "1.5", `"1"`, "9223372036854775808", "{"} {
|
||||
value := value
|
||||
t.Run(value, func(t *testing.T) {
|
||||
t.Parallel()
|
||||
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err == nil {
|
||||
t.Fatalf("validateRepairAttemptsJSON(%q) error = nil", value)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateRepairAttemptsJSONAcceptsNonnegativeIntegers(t *testing.T) {
|
||||
t.Parallel()
|
||||
|
||||
for _, value := range []string{"0", "1"} {
|
||||
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err != nil {
|
||||
t.Fatalf("validateRepairAttemptsJSON(%q) error = %v", value, err)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func appendManifestField(t *testing.T, directory, field string) {
|
||||
t.Helper()
|
||||
path := filepath.Join(directory, ManifestFilename)
|
||||
@@ -1077,7 +1119,7 @@ func testBundleWithReports(t *testing.T, count int) LogicalBundle {
|
||||
}
|
||||
manifest.Results[i] = Result{
|
||||
Position: position, ProfileID: profileID, ModelName: "gpt-5-mini",
|
||||
Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: path,
|
||||
Status: StatusSucceeded, ValidationStatus: "passed", RepairAttempts: intPtr(0), ReportPath: path,
|
||||
}
|
||||
reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")}
|
||||
}
|
||||
|
||||
@@ -1,6 +1,2 @@
|
||||
id: weather-balanced
|
||||
backend: openrouter
|
||||
model: "~google/gemini-flash-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
base_profile: gemini-flash-latest
|
||||
|
||||
@@ -1,6 +1,2 @@
|
||||
id: weather-deep
|
||||
backend: openrouter
|
||||
model: "~anthropic/claude-sonnet-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
base_profile: claude-sonnet-latest
|
||||
|
||||
@@ -1,5 +1,2 @@
|
||||
id: weather-light
|
||||
backend: openrouter
|
||||
model: deepseek/deepseek-v4-flash
|
||||
timeout_seconds: 180
|
||||
service_tier: flex
|
||||
base_profile: deepseek-4-flash
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
id: weather.daily_generated_text
|
||||
version: "2.0.0"
|
||||
version: "2.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Daily weather report analysis prompt.
|
||||
inputs:
|
||||
@@ -21,3 +21,4 @@ output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: daily.generated_text.schema.json
|
||||
repair_attempts: 1
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
id: weather.hourly_generated_text
|
||||
version: "2.0.0"
|
||||
version: "2.1.0"
|
||||
default_profile: weather-light
|
||||
description: Hourly weather report analysis prompt.
|
||||
inputs:
|
||||
@@ -21,3 +21,4 @@ output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: hourly.generated_text.schema.json
|
||||
repair_attempts: 1
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
id: weather.today_generated_text
|
||||
version: "2.0.0"
|
||||
version: "2.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Today's weather report analysis prompt.
|
||||
inputs:
|
||||
@@ -21,3 +21,4 @@ output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: today.generated_text.schema.json
|
||||
repair_attempts: 1
|
||||
|
||||
@@ -1,5 +1,5 @@
|
||||
id: weather.tomorrow_generated_text
|
||||
version: "2.0.0"
|
||||
version: "2.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Tomorrow's weather report analysis prompt.
|
||||
inputs:
|
||||
@@ -21,3 +21,4 @@ output:
|
||||
format: json
|
||||
validation_mode: json_schema
|
||||
schema_path: tomorrow.generated_text.schema.json
|
||||
repair_attempts: 1
|
||||
|
||||
@@ -77,8 +77,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
if err := yaml.Unmarshal(data, &definition); err != nil {
|
||||
t.Fatalf("decode prompt definition: %v", err)
|
||||
}
|
||||
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile {
|
||||
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile)
|
||||
if definition.ID != tc.id || definition.Version != "2.1.0" || definition.DefaultProfile != tc.profile {
|
||||
t.Fatalf("definition = %#v, want %s version 2.1.0 and profile %s", definition, tc.id, tc.profile)
|
||||
}
|
||||
sharedInstruction := false
|
||||
for _, message := range definition.Messages {
|
||||
@@ -92,8 +92,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
||||
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
||||
}
|
||||
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts != nil {
|
||||
t.Fatalf("output = %#v, want JSON schema output without repair attempts", definition.Output)
|
||||
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts == nil || *definition.Output.RepairAttempts != 1 {
|
||||
t.Fatalf("output = %#v, want JSON schema output with one repair attempt", definition.Output)
|
||||
}
|
||||
if _, err := promptassets.Schema(tc.schemaID); err != nil {
|
||||
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
|
||||
@@ -286,11 +286,11 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
|
||||
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
|
||||
} {
|
||||
t.Run(want.id, func(t *testing.T) {
|
||||
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.0.0")
|
||||
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.1.0")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPrompt() error = %v", err)
|
||||
}
|
||||
if inspection.PromptID != want.id || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != want.profile {
|
||||
if inspection.PromptID != want.id || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != want.profile || inspection.OutputContract.RepairAttempts != 1 {
|
||||
t.Fatalf("inspection = %#v", inspection)
|
||||
}
|
||||
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
|
||||
@@ -361,6 +361,29 @@ func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesAreMinimalBaseAliases(t *testing.T) {
|
||||
wantBases := map[string]string{
|
||||
"weather-balanced.yml": "gemini-flash-latest",
|
||||
"weather-deep.yml": "claude-sonnet-latest",
|
||||
"weather-light.yml": "deepseek-4-flash",
|
||||
}
|
||||
for path, wantBase := range wantBases {
|
||||
t.Run(path, func(t *testing.T) {
|
||||
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
|
||||
if err != nil {
|
||||
t.Fatalf("read profile: %v", err)
|
||||
}
|
||||
var definition map[string]string
|
||||
if err := yaml.Unmarshal(data, &definition); err != nil {
|
||||
t.Fatalf("unmarshal profile: %v", err)
|
||||
}
|
||||
if definition["id"] != strings.TrimSuffix(path, ".yml") || definition["base_profile"] != wantBase || len(definition) != 2 {
|
||||
t.Fatalf("profile definition = %#v, want only its id and base profile %q", definition, wantBase)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
|
||||
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
|
||||
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||
@@ -391,7 +414,7 @@ func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "repair_attempts:", "weather.daily_report"} {
|
||||
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "weather.daily_report"} {
|
||||
if strings.Contains(string(data), unwanted) {
|
||||
t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
|
||||
}
|
||||
|
||||
@@ -21,8 +21,9 @@ import (
|
||||
var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform")
|
||||
|
||||
const (
|
||||
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v2"
|
||||
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v2"
|
||||
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v3"
|
||||
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v3"
|
||||
promptFailureDebugSchemaVersion = "weatherreporter.prompt_failure_debug.v1"
|
||||
debugDirectoryMode = 0o700
|
||||
debugFileMode = 0o600
|
||||
)
|
||||
@@ -53,6 +54,7 @@ type PromptDebugOutput struct {
|
||||
Format string `json:"format"`
|
||||
ValidationMode string `json:"validationMode"`
|
||||
SchemaPath string `json:"schemaPath"`
|
||||
RepairAttempts int `json:"repairAttempts"`
|
||||
}
|
||||
|
||||
// PromptDebugPreparation is the explicit, content-safe mapping of preparation
|
||||
@@ -97,10 +99,28 @@ type PromptDebugUsage struct {
|
||||
// PromptDebugValidation is the completed validation detail retained in the
|
||||
// explicitly enabled debug store.
|
||||
type PromptDebugValidation struct {
|
||||
Status string `json:"status"`
|
||||
Mode string `json:"mode"`
|
||||
SchemaPath string `json:"schemaPath"`
|
||||
Diagnostics []string `json:"diagnostics,omitempty"`
|
||||
Status string `json:"status"`
|
||||
Mode string `json:"mode"`
|
||||
SchemaPath string `json:"schemaPath"`
|
||||
RepairAttempts int `json:"repairAttempts"`
|
||||
Diagnostics []string `json:"diagnostics,omitempty"`
|
||||
}
|
||||
|
||||
// PromptFailureDebugArtifact records provider detail only in explicit debug storage.
|
||||
type PromptFailureDebugArtifact struct {
|
||||
SchemaVersion string `json:"schemaVersion"`
|
||||
ReportID report.ID `json:"reportId"`
|
||||
ValidDate string `json:"validDate"`
|
||||
RunID string `json:"runId"`
|
||||
Failure PromptFailure `json:"failure"`
|
||||
}
|
||||
|
||||
type PromptFailure struct {
|
||||
Category string `json:"category"`
|
||||
StatusCode int `json:"statusCode,omitempty"`
|
||||
ProviderCode string `json:"providerCode,omitempty"`
|
||||
ProviderType string `json:"providerType,omitempty"`
|
||||
ProviderMessage string `json:"providerMessage,omitempty"`
|
||||
}
|
||||
|
||||
// PromptDebugExecution is the explicit mapping of execution provenance.
|
||||
@@ -235,6 +255,24 @@ func (w *PromptDebugWriter) WriteExecution(ref PromptDebugRef, execution prompte
|
||||
return directory, nil
|
||||
}
|
||||
|
||||
// WriteFailure stores the project-owned structured provider failure.
|
||||
func (w *PromptDebugWriter) WriteFailure(ref PromptDebugRef, failure *promptexec.GenerationError) (string, error) {
|
||||
if !w.Enabled() || failure == nil {
|
||||
return "", nil
|
||||
}
|
||||
directory, secureDirectory, err := w.runDirectory(ref)
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
defer secureDirectory.Close()
|
||||
artifact := PromptFailureDebugArtifact{SchemaVersion: promptFailureDebugSchemaVersion, ReportID: ref.ReportID, ValidDate: ref.ValidDate, RunID: ref.RunID,
|
||||
Failure: PromptFailure{Category: string(failure.Category()), StatusCode: failure.StatusCode(), ProviderCode: failure.ProviderCode(), ProviderType: failure.ProviderType(), ProviderMessage: failure.ProviderMessage()}}
|
||||
if err := secureDirectory.writeJSON("failure.json", artifact); err != nil {
|
||||
return "", err
|
||||
}
|
||||
return directory, nil
|
||||
}
|
||||
|
||||
func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) {
|
||||
if err := validatePromptDebugRef(ref); err != nil {
|
||||
return "", nil, err
|
||||
@@ -283,7 +321,7 @@ func promptDebugPreparation(value promptexec.Preparation) PromptDebugPreparation
|
||||
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash,
|
||||
RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes),
|
||||
ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName,
|
||||
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath},
|
||||
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath, RepairAttempts: value.Output.RepairAttempts},
|
||||
StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration,
|
||||
}
|
||||
}
|
||||
@@ -300,7 +338,7 @@ func promptDebugExecution(value promptexec.Execution) PromptDebugExecution {
|
||||
}
|
||||
|
||||
func promptDebugValidation(value promptexec.Validation) PromptDebugValidation {
|
||||
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, Diagnostics: append([]string(nil), value.Diagnostics...)}
|
||||
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, RepairAttempts: value.RepairAttempts, Diagnostics: append([]string(nil), value.Diagnostics...)}
|
||||
}
|
||||
|
||||
func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage {
|
||||
|
||||
@@ -44,13 +44,13 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
|
||||
t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir)
|
||||
}
|
||||
preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json"))
|
||||
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v2", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`} {
|
||||
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v3", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`, `"repairAttempts": 0`} {
|
||||
if !strings.Contains(string(preparationData), want) {
|
||||
t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData)
|
||||
}
|
||||
}
|
||||
executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json"))
|
||||
for _, want := range []string{"weatherreporter.prompt_execution_debug.v2", "Generated forecast prose.", "validation details", `"status": "passed"`} {
|
||||
for _, want := range []string{"weatherreporter.prompt_execution_debug.v3", "Generated forecast prose.", "validation details", `"status": "passed"`, `"repairAttempts": 0`} {
|
||||
if !strings.Contains(string(executionData), want) {
|
||||
t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData)
|
||||
}
|
||||
@@ -66,6 +66,25 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
|
||||
assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode)
|
||||
}
|
||||
|
||||
func TestPromptDebugWriterWritesProviderFailureOnlyToDebugStore(t *testing.T) {
|
||||
writer, err := NewPromptDebugWriter(filepath.Join(t.TempDir(), "operator-debug"))
|
||||
if err != nil {
|
||||
t.Fatalf("NewPromptDebugWriter() error = %v", err)
|
||||
}
|
||||
marker := "provider-private-marker"
|
||||
directory, err := writer.WriteFailure(promptDebugRef(), promptexec.NewGenerationError(429, "rate_limit", "provider_error", marker, nil))
|
||||
if err != nil {
|
||||
t.Fatalf("WriteFailure() error = %v", err)
|
||||
}
|
||||
data := readPromptDebugFile(t, filepath.Join(directory, "failure.json"))
|
||||
for _, want := range []string{"weatherreporter.prompt_failure_debug.v1", `"category": "generation"`, `"statusCode": 429`, marker} {
|
||||
if !strings.Contains(string(data), want) {
|
||||
t.Fatalf("failure artifact missing %q: %s", want, data)
|
||||
}
|
||||
}
|
||||
assertPromptDebugMode(t, filepath.Join(directory, "failure.json"), debugFileMode)
|
||||
}
|
||||
|
||||
func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) {
|
||||
const marker = "private-debug-marker"
|
||||
tests := []struct {
|
||||
@@ -349,7 +368,7 @@ func promptDebugExecutionFixture() promptexec.Execution {
|
||||
InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model",
|
||||
GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15},
|
||||
StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second,
|
||||
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", []string{"validation details"}),
|
||||
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", 0, []string{"validation details"}),
|
||||
RawOutput: []byte("Generated forecast prose."),
|
||||
Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}},
|
||||
}
|
||||
|
||||
@@ -78,3 +78,15 @@ func boundText(value string, limit int) string {
|
||||
}
|
||||
return value
|
||||
}
|
||||
|
||||
func boundCodePoints(value string, limit int) string {
|
||||
if limit <= 0 || value == "" {
|
||||
return ""
|
||||
}
|
||||
value = strings.ToValidUTF8(value, "<22>")
|
||||
runes := []rune(value)
|
||||
if len(runes) <= limit {
|
||||
return value
|
||||
}
|
||||
return string(runes[:limit])
|
||||
}
|
||||
|
||||
@@ -51,6 +51,7 @@ type OutputContract struct {
|
||||
Format string
|
||||
ValidationMode string
|
||||
SchemaPath string
|
||||
RepairAttempts int
|
||||
}
|
||||
|
||||
// ProfileInspection describes the safe, selected execution identity for one profile.
|
||||
@@ -141,19 +142,21 @@ type TokenUsage struct {
|
||||
|
||||
// Validation records a completed output validation check.
|
||||
type Validation struct {
|
||||
Status ValidationStatus
|
||||
Mode string
|
||||
SchemaPath string
|
||||
Diagnostics []string
|
||||
Status ValidationStatus
|
||||
Mode string
|
||||
SchemaPath string
|
||||
RepairAttempts int
|
||||
Diagnostics []string
|
||||
}
|
||||
|
||||
// NewValidation returns a completed validation value with bounded diagnostics.
|
||||
func NewValidation(status ValidationStatus, mode string, schemaPath string, diagnostics []string) Validation {
|
||||
func NewValidation(status ValidationStatus, mode string, schemaPath string, repairAttempts int, diagnostics []string) Validation {
|
||||
return Validation{
|
||||
Status: status,
|
||||
Mode: mode,
|
||||
SchemaPath: schemaPath,
|
||||
Diagnostics: boundDiagnostics(diagnostics),
|
||||
Status: status,
|
||||
Mode: mode,
|
||||
SchemaPath: schemaPath,
|
||||
RepairAttempts: repairAttempts,
|
||||
Diagnostics: boundDiagnostics(diagnostics),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -235,6 +238,86 @@ func (e *Error) Category() ErrorCategory {
|
||||
return e.category
|
||||
}
|
||||
|
||||
// GenerationError retains provider failure details for programmatic handling
|
||||
// without exposing them through routine formatting or serialization.
|
||||
type GenerationError struct {
|
||||
statusCode int
|
||||
providerCode string
|
||||
providerType string
|
||||
providerMessage string
|
||||
err *Error
|
||||
}
|
||||
|
||||
// NewGenerationError returns a classified generation failure with bounded
|
||||
// provider details. The dependency cause remains reachable through err only.
|
||||
func NewGenerationError(statusCode int, providerCode string, providerType string, providerMessage string, cause error) *GenerationError {
|
||||
return &GenerationError{
|
||||
statusCode: statusCode,
|
||||
providerCode: boundCodePoints(providerCode, 256),
|
||||
providerType: boundCodePoints(providerType, 256),
|
||||
providerMessage: boundCodePoints(providerMessage, 4096),
|
||||
err: NewError(Generation, "provider generation failed", cause),
|
||||
}
|
||||
}
|
||||
|
||||
// StatusCode returns the provider HTTP status when one was available.
|
||||
func (e *GenerationError) StatusCode() int {
|
||||
if e == nil {
|
||||
return 0
|
||||
}
|
||||
return e.statusCode
|
||||
}
|
||||
|
||||
// ProviderCode returns the bounded provider error code.
|
||||
func (e *GenerationError) ProviderCode() string {
|
||||
if e == nil {
|
||||
return ""
|
||||
}
|
||||
return e.providerCode
|
||||
}
|
||||
|
||||
// ProviderType returns the bounded provider error type.
|
||||
func (e *GenerationError) ProviderType() string {
|
||||
if e == nil {
|
||||
return ""
|
||||
}
|
||||
return e.providerType
|
||||
}
|
||||
|
||||
// ProviderMessage returns the bounded provider error message.
|
||||
func (e *GenerationError) ProviderMessage() string {
|
||||
if e == nil {
|
||||
return ""
|
||||
}
|
||||
return e.providerMessage
|
||||
}
|
||||
|
||||
// Category returns Generation for every generation failure.
|
||||
func (e *GenerationError) Category() ErrorCategory { return Generation }
|
||||
|
||||
// Error intentionally excludes provider details from ordinary error text.
|
||||
func (e *GenerationError) Error() string {
|
||||
if e == nil {
|
||||
return ""
|
||||
}
|
||||
message := NewError(Generation, "provider generation failed", nil).Error()
|
||||
if e.statusCode == 0 {
|
||||
return message
|
||||
}
|
||||
return fmt.Sprintf("%s (HTTP %d)", message, e.statusCode)
|
||||
}
|
||||
|
||||
// GoString keeps %#v formatting as safe as ordinary error formatting.
|
||||
func (e *GenerationError) GoString() string { return e.Error() }
|
||||
|
||||
// Unwrap preserves the project-owned classified error and its hidden cause.
|
||||
func (e *GenerationError) Unwrap() error {
|
||||
if e == nil {
|
||||
return nil
|
||||
}
|
||||
return e.err
|
||||
}
|
||||
|
||||
// CapacityError adds the safe backend identity to a capacity failure.
|
||||
type CapacityError struct {
|
||||
BackendID string
|
||||
|
||||
@@ -3,6 +3,7 @@ package promptexec
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"fmt"
|
||||
"strings"
|
||||
"testing"
|
||||
"unicode/utf8"
|
||||
@@ -172,8 +173,8 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
|
||||
if bounded[0] != "a<>b" {
|
||||
t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0])
|
||||
}
|
||||
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", values)
|
||||
if len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
|
||||
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", 2, values)
|
||||
if validation.RepairAttempts != 2 || len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
|
||||
t.Fatalf("validation = %#v, want bounded diagnostics", validation)
|
||||
}
|
||||
|
||||
@@ -187,8 +188,53 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerationErrorKeepsProviderDetailsOutOfRoutineFormatting(t *testing.T) {
|
||||
cause := errors.New("provider response must stay hidden")
|
||||
error := NewGenerationError(
|
||||
429,
|
||||
string([]byte{'c', 0xff})+strings.Repeat("界", 300),
|
||||
strings.Repeat("type", 100),
|
||||
string([]byte{'m', 0xff})+strings.Repeat("界", 4_200),
|
||||
cause,
|
||||
)
|
||||
if error.StatusCode() != 429 || error.Category() != Generation || CategoryOf(error) != Generation {
|
||||
t.Fatalf("generation error identity = %#v", error)
|
||||
}
|
||||
if utf8.RuneCountInString(error.ProviderCode()) != 256 || utf8.RuneCountInString(error.ProviderType()) != 256 || utf8.RuneCountInString(error.ProviderMessage()) != 4096 {
|
||||
t.Fatalf("provider detail bounds = %d/%d/%d", utf8.RuneCountInString(error.ProviderCode()), utf8.RuneCountInString(error.ProviderType()), utf8.RuneCountInString(error.ProviderMessage()))
|
||||
}
|
||||
if !utf8.ValidString(error.ProviderCode()) || !utf8.ValidString(error.ProviderType()) || !utf8.ValidString(error.ProviderMessage()) {
|
||||
t.Fatalf("provider details are not valid UTF-8: %#v", error)
|
||||
}
|
||||
if !errors.Is(error, cause) {
|
||||
t.Fatal("errors.Is() = false, want preserved dependency cause")
|
||||
}
|
||||
var owned *Error
|
||||
if !errors.As(error, &owned) || owned.Category() != Generation {
|
||||
t.Fatalf("errors.As(*Error) = %#v, want project-owned generation error", owned)
|
||||
}
|
||||
want := "generation: provider generation failed (HTTP 429)"
|
||||
if error.Error() != want || fmt.Sprintf("%#v", error) != want {
|
||||
t.Fatalf("ordinary formatting = %q / %#v, want %q", error.Error(), error, want)
|
||||
}
|
||||
for _, private := range []string{cause.Error(), error.ProviderCode(), error.ProviderType(), error.ProviderMessage()} {
|
||||
if strings.Contains(error.Error(), private) || strings.Contains(fmt.Sprintf("%#v", error), private) {
|
||||
t.Fatalf("generation error leaks provider detail %q", private)
|
||||
}
|
||||
}
|
||||
|
||||
statusOnly := NewGenerationError(503, "", "", "", nil)
|
||||
if statusOnly.Error() != "generation: provider generation failed (HTTP 503)" {
|
||||
t.Fatalf("status-only error = %q", statusOnly)
|
||||
}
|
||||
var nilError *GenerationError
|
||||
if nilError.StatusCode() != 0 || nilError.ProviderCode() != "" || nilError.ProviderType() != "" || nilError.ProviderMessage() != "" || nilError.Category() != Generation || nilError.Error() != "" || nilError.GoString() != "" || nilError.Unwrap() != nil {
|
||||
t.Fatalf("nil generation error = %#v", nilError)
|
||||
}
|
||||
}
|
||||
|
||||
func TestContractCopiesMutableValues(t *testing.T) {
|
||||
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}}
|
||||
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}, Output: OutputContract{RepairAttempts: 3}}
|
||||
preparationDebug := &PreparationDebug{
|
||||
RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}},
|
||||
StructuredSchema: []byte("schema body"),
|
||||
@@ -196,7 +242,7 @@ func TestContractCopiesMutableValues(t *testing.T) {
|
||||
}
|
||||
execution := Execution{
|
||||
InputHashes: map[string]string{"data_package": "input-hash"},
|
||||
Validation: Validation{Diagnostics: []string{"validation detail"}},
|
||||
Validation: Validation{RepairAttempts: 2, Diagnostics: []string{"validation detail"}},
|
||||
RawOutput: []byte("generated output"),
|
||||
Debug: &ExecutionDebug{
|
||||
RawOutput: []byte("provider output"),
|
||||
@@ -217,13 +263,13 @@ func TestContractCopiesMutableValues(t *testing.T) {
|
||||
execution.Debug.RawOutput[0] = 'x'
|
||||
execution.Debug.ValidationDiagnostics[0] = "changed"
|
||||
|
||||
if preparationCopy.InputHashes["data_package"] != "input-hash" {
|
||||
if preparationCopy.InputHashes["data_package"] != "input-hash" || preparationCopy.Output.RepairAttempts != 3 {
|
||||
t.Fatalf("preparation copy = %#v", preparationCopy)
|
||||
}
|
||||
if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` {
|
||||
t.Fatalf("preparation debug copy = %#v", debugCopy)
|
||||
}
|
||||
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
|
||||
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.RepairAttempts != 2 || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
|
||||
t.Fatalf("execution copy = %#v", executionCopy)
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,14 +9,15 @@ import (
|
||||
|
||||
func dailyDefinition() Definition {
|
||||
return Definition{
|
||||
ID: Daily,
|
||||
Name: "Daily Report",
|
||||
PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "2.0.0",
|
||||
TemplateID: "daily",
|
||||
GeneratedTextSchemaID: "daily",
|
||||
ArtifactGroup: "daily",
|
||||
OutputName: "daily.md",
|
||||
ID: Daily,
|
||||
Name: "Daily Report",
|
||||
PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "2.1.0",
|
||||
TemplateID: "daily",
|
||||
GeneratedTextSchemaID: "daily",
|
||||
GeneratedTextRepairAttempts: 1,
|
||||
ArtifactGroup: "daily",
|
||||
OutputName: "daily.md",
|
||||
DistributorPathTemplates: []string{
|
||||
"daily/{valid_start_date}/{run_id}.md",
|
||||
"daily/{valid_start_date}/index.md",
|
||||
|
||||
@@ -27,20 +27,21 @@ const (
|
||||
)
|
||||
|
||||
type Definition struct {
|
||||
ID ID
|
||||
Name string
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
TemplateID string
|
||||
GeneratedTextSchemaID string
|
||||
ArtifactGroup string
|
||||
OutputName string
|
||||
DistributorPathTemplates []string
|
||||
Modules []module.ConfigItem
|
||||
Morning bool
|
||||
Evening bool
|
||||
resolve func(ResolveRequest) (timeutil.Period, error)
|
||||
runIDDisambiguator func(Resolved) string
|
||||
ID ID
|
||||
Name string
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
TemplateID string
|
||||
GeneratedTextSchemaID string
|
||||
GeneratedTextRepairAttempts int
|
||||
ArtifactGroup string
|
||||
OutputName string
|
||||
DistributorPathTemplates []string
|
||||
Modules []module.ConfigItem
|
||||
Morning bool
|
||||
Evening bool
|
||||
resolve func(ResolveRequest) (timeutil.Period, error)
|
||||
runIDDisambiguator func(Resolved) string
|
||||
}
|
||||
|
||||
func (r Resolved) OutputName() (string, error) {
|
||||
|
||||
@@ -11,14 +11,15 @@ const hourlyReportHours = 6
|
||||
|
||||
func hourlyDefinition() Definition {
|
||||
return Definition{
|
||||
ID: Hourly,
|
||||
Name: "Hourly Report",
|
||||
PromptID: "weather.hourly_generated_text",
|
||||
PromptVersion: "2.0.0",
|
||||
TemplateID: "hourly",
|
||||
GeneratedTextSchemaID: "hourly",
|
||||
ArtifactGroup: "hourly",
|
||||
OutputName: "hourly.md",
|
||||
ID: Hourly,
|
||||
Name: "Hourly Report",
|
||||
PromptID: "weather.hourly_generated_text",
|
||||
PromptVersion: "2.1.0",
|
||||
TemplateID: "hourly",
|
||||
GeneratedTextSchemaID: "hourly",
|
||||
GeneratedTextRepairAttempts: 1,
|
||||
ArtifactGroup: "hourly",
|
||||
OutputName: "hourly.md",
|
||||
DistributorPathTemplates: []string{
|
||||
"hourly/index.md",
|
||||
},
|
||||
|
||||
@@ -82,8 +82,8 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, definition := range definitions {
|
||||
if definition.PromptVersion != "2.0.0" {
|
||||
t.Fatalf("%s PromptVersion = %q, want 2.0.0", definition.ID, definition.PromptVersion)
|
||||
if definition.PromptVersion != "2.1.0" {
|
||||
t.Fatalf("%s PromptVersion = %q, want 2.1.0", definition.ID, definition.PromptVersion)
|
||||
}
|
||||
if definition.PromptID == "" {
|
||||
t.Fatalf("%s PromptID is empty", definition.ID)
|
||||
@@ -91,6 +91,9 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
||||
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
|
||||
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
|
||||
}
|
||||
if definition.GeneratedTextRepairAttempts != 1 {
|
||||
t.Fatalf("%s repair attempts = %d, want 1", definition.ID, definition.GeneratedTextRepairAttempts)
|
||||
}
|
||||
}
|
||||
|
||||
if _, err := registry.Lookup(ID("three_day")); err == nil {
|
||||
|
||||
@@ -7,14 +7,15 @@ import (
|
||||
|
||||
func todayDefinition() Definition {
|
||||
return Definition{
|
||||
ID: Today,
|
||||
Name: "Today Report",
|
||||
PromptID: "weather.today_generated_text",
|
||||
PromptVersion: "2.0.0",
|
||||
TemplateID: "today",
|
||||
GeneratedTextSchemaID: "today",
|
||||
ArtifactGroup: "today",
|
||||
OutputName: "today.md",
|
||||
ID: Today,
|
||||
Name: "Today Report",
|
||||
PromptID: "weather.today_generated_text",
|
||||
PromptVersion: "2.1.0",
|
||||
TemplateID: "today",
|
||||
GeneratedTextSchemaID: "today",
|
||||
GeneratedTextRepairAttempts: 1,
|
||||
ArtifactGroup: "today",
|
||||
OutputName: "today.md",
|
||||
DistributorPathTemplates: []string{
|
||||
"daily/{valid_start_date}/{run_id}.md",
|
||||
"daily/{valid_start_date}/index.md",
|
||||
|
||||
@@ -7,14 +7,15 @@ import (
|
||||
|
||||
func tomorrowDefinition() Definition {
|
||||
return Definition{
|
||||
ID: Tomorrow,
|
||||
Name: "Tomorrow Report",
|
||||
PromptID: "weather.tomorrow_generated_text",
|
||||
PromptVersion: "2.0.0",
|
||||
TemplateID: "tomorrow",
|
||||
GeneratedTextSchemaID: "tomorrow",
|
||||
ArtifactGroup: "tomorrow",
|
||||
OutputName: "tomorrow.md",
|
||||
ID: Tomorrow,
|
||||
Name: "Tomorrow Report",
|
||||
PromptID: "weather.tomorrow_generated_text",
|
||||
PromptVersion: "2.1.0",
|
||||
TemplateID: "tomorrow",
|
||||
GeneratedTextSchemaID: "tomorrow",
|
||||
GeneratedTextRepairAttempts: 1,
|
||||
ArtifactGroup: "tomorrow",
|
||||
OutputName: "tomorrow.md",
|
||||
DistributorPathTemplates: []string{
|
||||
"daily/{valid_start_date}/{run_id}.md",
|
||||
"daily/{valid_start_date}/index.md",
|
||||
|
||||
Reference in New Issue
Block a user