14 Commits

53 changed files with 1341 additions and 217 deletions

View File

@@ -96,7 +96,7 @@ period, prompt version, timezone, and status. Successful output has an absolute
"command": "generate",
"reportId": "today",
"promptId": "weather.today_generated_text",
"promptVersion": "2.0.0",
"promptVersion": "2.1.0",
"runId": "20260529T120000.000000000Z_today",
"status": "succeeded",
"timezone": "America/Chicago",
@@ -106,7 +106,10 @@ period, prompt version, timezone, and status. Successful output has an absolute
When available, the summary also includes the effective `profileId`,
`backendId`, `modelName`, `sourceWarnings`, `validationStatus`, requested
`llmDebugPath`, and compact Distributor `notification` result. It does not
`repairAttempts`, `llmDebugPath`, and compact Distributor `notification`
result. `repairAttempts` is `0` when the initial output passed validation,
positive when PromptKit made corrective generation calls, and omitted when
validation did not complete. The summary does not
include historical or transient artifact paths such as metadata, prompt input,
raw generated text, render context, or notification receipts.
@@ -115,7 +118,8 @@ raw generated text, render context, or notification receipts.
A run summary contains `command`, `batch`, `status`, `startedAt`, `finishedAt`,
`total`, `succeeded`, `failed`, and a `reports` array. Each report item includes
its identity, status, effective profile and model details when available,
source warnings, validation status, and absolute `outputPath` after publication.
source warnings, validation status, repair-attempt count when validation
completed, and absolute `outputPath` after publication.
The top-level summary may also contain a batch `notification` object and
`error`. Batch status is `failed` if any report or the batch notification fails.
The `total`, `succeeded`, and `failed` counters describe report items only, so
@@ -143,7 +147,8 @@ A comparison summary contains these fields in this order: `command`,
successful `results[].reportPath` are absolute. `results` preserves the
supplied profile order and each item contains `position`, `profileId`, optional
`backendId`, `modelName`, `status`, optional `validationStatus`, optional
`reportPath`, optional `llmDebugPath`, and optional safe `error`.
`repairAttempts`, optional `reportPath`, optional `llmDebugPath`, and optional
safe `error`. The repair-attempt semantics match the generate summary.
The comparison status is `succeeded` only when every selected profile succeeds
and the bundle is published. Individual profile failures still publish a
@@ -157,7 +162,8 @@ published Promptkit category; destination failures use `destination_<kind>`;
and committed cleanup failures use `publication_cleanup` with a message that
states whether a complete prior bundle, partial remnants, or no prior bundle
remains, or that recovery state could not be inspected. It does not expose
provider diagnostics, filesystem causes, or recovery paths. See the
provider diagnostics, filesystem causes, or recovery paths. A provider HTTP
failure may include its numeric status in the safe message. See the
[comparison bundle contract](integrations/comparison-bundle.md) for durable
artifact fields and failure invariants.

View File

@@ -203,16 +203,23 @@ for platform availability, security, and retention requirements.
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
They are separate decisions. An explicit `profile` applies to every selected
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
Tomorrow select `weather-balanced` through their exact `2.0.0` prompt
Tomorrow select `weather-balanced` through their exact `2.1.0` prompt
definitions.
Promptkit resolves a selected profile definition from a test or embedding
consumer's explicit in-memory profile, then the configured `profile_file` or
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
built-in catalog. Sources provide complete definitions; fields are never
merged. A matching malformed external profile fails rather than using the
embedded definition. The [Promptkit integration guide](integrations/promptkit.md)
owns the catalog and precedence details.
maintained catalog. Weatherreporter's embedded `weather-*` definitions are small
aliases of Promptkit's maintained base profiles, so Promptkit also resolves
their inherited target and settings. A configured definition with the same ID
as either a selected profile or an inherited base takes precedence. A matching
malformed external profile fails rather than using the embedded definition. The
[Promptkit integration guide](integrations/promptkit.md) owns the catalog and
precedence details.
An endpoint-only profile may intentionally have no backend identity. Profiles
that require a direct API key are rejected before collection, while Promptkit
resolves optional environment credential sources during execution.
To replace the default Hourly definition with a local OpenAI-compatible
endpoint, set `profile_file` to a copy of

View File

@@ -9,7 +9,7 @@ and retention belong to the [operations guide](../operations.md).
## Version And Layout
The current and only supported manifest schema version is
`weatherreporter.comparison.v1`. A bundle directory contains exactly these
`weatherreporter.comparison.v2`. A bundle directory contains exactly these
regular, non-symlinked files:
```text
@@ -51,7 +51,7 @@ this order:
```text
position, profileId, backendId, modelName, status, validationStatus,
reportPath, error
repairAttempts, reportPath, error
```
`startedAt` and `finishedAt` are nonzero UTC timestamps, and the latter is not
@@ -66,12 +66,16 @@ contiguous from one, profile IDs are distinct and nonblank, and
`succeeded + failed == total`.
A successful result has `status: "succeeded"`, `validationStatus: "passed"`,
and a non-negative `repairAttempts` count,
a `reportPath` exactly equal to the canonical `NN-profile-slug.md` filename for
its position, total, and logical profile ID, and no `error`. A failed result has
`status: "failed"`, no `reportPath`, and an `error` object with nonblank
`category` and `message`. Its validation status is absent, `failed`, or
`skipped`. Error messages are valid UTF-8 and no longer than 1,024 bytes.
`backendId` and `validationStatus` are omitted when unavailable.
`skipped`; it may also be `passed` when a WeatherReporter step after PromptKit
validation failed. Error messages are valid UTF-8 and no longer than 1,024 bytes.
`backendId` and `validationStatus` are omitted when unavailable. A failed
result with any completed validation status must retain its non-negative
`repairAttempts`; early operational failures omit both fields.
Every successful Markdown file is declared by exactly one successful result.
The directory contains no extra entries. Consumers can therefore verify the
@@ -88,7 +92,9 @@ case-variant, or duplicate fields; multiple JSON values; extra entries;
symlinks; and future or otherwise unsupported versions. Treat a bundle that
fails recognition as an ordinary directory, not as a compatible bundle.
When replacing a recognized bundle, cancellation observed before the new
Only v2 is recognized as a replaceable bundle; v1 is unsupported and must be
moved or removed before a replacement at the same destination. When replacing
a recognized bundle, cancellation observed before the new
bundle is installed preserves the prior bundle rather than committing the
replacement.

View File

@@ -1,10 +1,12 @@
# Promptkit Integration
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.0.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are `weather.daily_generated_text`, `weather.today_generated_text`, `weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version `2.1.0`. Their prompt assets, generated-text JSON Schemas, and Weatherreporter profile catalog are embedded by `internal/promptassets`.
## Logical Profile Catalog
Prompt definitions select a stable Weatherreporter profile ID. The embedded definitions currently use Promptkit's `openrouter` backend:
Prompt definitions select a stable Weatherreporter profile ID. Each embedded
definition contains only its ID and one Promptkit base-profile reference; the
effective execution settings resolve from Promptkit's maintained catalog:
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
| --- | --- | --- | --- | --- | --- |
@@ -12,22 +14,45 @@ Prompt definitions select a stable Weatherreporter profile ID. The embedded defi
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded profiles intentionally omit endpoints, credentials, temperature, `top_p`, and output-token limits.
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
profiles intentionally omit endpoints, credentials, and execution settings;
Promptkit owns inherited resolution and its provider-native defaults. Promptkit
assembles its maintained catalog from independently versioned OpenRouter and
Rakestrawhome modules selected by the Promptkit release. Weatherreporter does
not import or register those catalog modules directly. The maintained
`rakestrawhome-gemma-4-31b` profile is also available for ordinary and
comparison selection and reports the `rakestrawhome` backend without
Weatherreporter-specific configuration.
## Selection And Active Execution
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank backend and model identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
Before weather collection, Weatherreporter validates the report's exact generated-text report/schema/template catalog binding, prompt version and hash, output contract, and selected profile. Active profiles must resolve a nonblank model; an endpoint-only profile may intentionally have no backend identity. A nonblank `promptkit.profile` selects one profile ID for every report in the command; otherwise the prompt's declared default selects it. Promptkit resolves the selected definition in this order:
1. explicit in-memory profiles used by an embedding consumer or test;
2. the configured `profile_file` or `profile_dir`;
3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's built-in catalog.
4. Promptkit's maintained catalog.
A source falls through only when the selected ID is absent. Each source supplies a complete definition, so profile fields are not merged. A malformed matching operator definition is an error and does not fall back.
A source falls through only when the selected ID is absent. Promptkit resolves a
derived profile's base with the same source precedence, so a configured base
can shadow a built-in base. A missing, cyclic, malformed, or incomplete
selected inheritance chain is an error and does not fall back.
Profiles that require a direct API key are unsupported; a profile that reports `APIKeyEnv` requires a nonblank value in that environment variable. Active results retain the selected logical profile ID and resolved backend and model. Ordinary errors, summaries, logs, and outputs exclude endpoints, credentials, rendered messages, schemas, request bodies, response bodies, and complete parameter maps.
Profiles that require a direct API key are unsupported. Optional environment
credential sources are Promptkit runtime concerns and are not checked by
Weatherreporter during profile inspection. Active results retain the selected
logical profile ID and resolved backend and model. Ordinary errors, summaries,
logs, and outputs exclude endpoints, credentials, rendered messages, schemas,
request bodies, response bodies, and complete parameter maps.
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
Promptkit receives the YAML data package as an inline input and returns structured JSON that Weatherreporter validates before rendering its own Markdown template. Before accepting that JSON, Weatherreporter requires exactly one preparation callback and reconciles its prompt/profile/backend/model and rendered/input hashes with the inspected identity and completed result. The callback output contract and completed validation must use the report's expected JSON Schema mode and path. The package contains only reviewed prompt-facing warning summaries, never source transport or provenance details. Safe active provenance remains in memory. Content-rich diagnostics are opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions. Ordinary generation errors disclose only the safe Weatherreporter category and optional HTTP status; provider code, type, and message are written only to the explicit secure failure-debug artifact.
Each embedded prompt permits one Promptkit-owned corrective generation after an
eligible failed or explicitly empty result. This is not an application retry:
Weatherreporter performs no provider retry, profile fallback, or request-level
output-contract override. Promptkit reports cumulative usage and the actual
number of corrective calls; repair exhaustion remains a completed validation
failure.
When capture is enabled, its preparation artifact projects a provider endpoint
to its scheme and host and retains only reviewed execution settings. Provider

View File

@@ -7,7 +7,7 @@ is owned by the [CLI reference](../cli.md) and [operations guide](../operations.
## Single-Report Flow
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. The resolved profile, backend, and model are carried in the active result.
`GenerateDetailed` resolves the requested report and output destination before initializing an optional explicit debug writer. An explicit output file wins; otherwise the configured output directory is used, falling back to the captured working directory. Output preflight validates the final filename, permits only an absent or regular final destination, and validates the bounded same-directory temporary form without creating a missing parent. It then validates the report's generated-text catalog binding, exact Promptkit prompt, and selected profile before collecting weather data. Profile inspection requires a model, permits an empty backend identity for endpoint-only profiles, and leaves inherited resolution and optional credential sources to Promptkit. The resolved profile, backend, model, and actual repair count (once a completed execution exists) are carried in the active result; the configured repair budget remains part of the exact prompt contract.
The workflow builds facts, a module snapshot, briefing metadata, and the YAML prompt package in memory. It executes Promptkit only against the inspected prompt and profile, reconciles the preparation callback and completed result with that identity and the prepared report schema, validates the returned generated text, builds a render context, and renders Markdown. `fileutil` writes the completed Markdown through a same-directory temporary file, rechecks the final destination and context after close and immediately before the atomic rename. Only after that write succeeds does single-report notification run.

View File

@@ -21,8 +21,9 @@ Before accepting generated JSON, the execution boundary reconciles the prepared
report definition, inspected prompt hash and selected profile identity, the one
preparation callback, and the completed Promptkit result. The callback and
completion must agree on prompt, profile, backend, model, and rendered/input
hashes; the callback output and completed validation must name the prepared
report's JSON Schema. A mismatch produces no rendered Markdown and leaves
hashes; the callback output must also carry the prepared report's configured
repair budget, while the completed validation records the actual corrective
calls used within that budget. A mismatch produces no rendered Markdown and leaves
results with only the inspected safe identity.
Single-report generation executes one prepared profile and publishes its

View File

@@ -2,9 +2,9 @@
`internal/adapters/promptkit` maps Weatherreporter's project-owned executor contract to Promptkit. The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor per action. Promptkit dependency types do not escape the adapter.
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, the embedded fallback catalog, and its built-in catalog; the adapter does not parse profile YAML, merge sources, or probe endpoints.
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback profile filesystems to each engine. Promptkit resolves configured operator profile sources, embedded profile aliases and their bases, and its maintained catalog. Promptkit privately assembles that catalog from the independently versioned OpenRouter and Rakestrawhome catalog modules selected by its release; Weatherreporter neither imports nor registers them. The adapter does not parse profile YAML, resolve inheritance, merge sources, inspect optional environment credentials, or probe endpoints.
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
The adapter exposes exact prompt and profile validation plus prepared execution. It maps safe prompt identity, logical profile, effective backend/model, preparation, execution, validation, and optional debug values into `promptexec`. An empty backend identity remains valid for an endpoint-only profile; a nonblank model is required. PromptKit's configured repair-call budget and the completed result's actual corrective-call count are retained, along with its cumulative provider usage and final candidate. Structured provider generation failures become project-owned redacted generation errors that retain only bounded details through explicit accessors. `Execute` passes the YAML package as an inline Promptkit input; it does not construct a filesystem URI or write a package file.
The application uses the preparation callback to record active safe provenance in memory and optionally writes content-rich diagnostics only through an explicit debug writer. The adapter returns raw output for application validation and rendering. It does not retain application state, render Markdown, choose report definitions, or send Distributor notifications.

View File

@@ -122,7 +122,9 @@ The destination is preflighted before prompt inspection and collection, then
rechecked immediately before an atomic publish. A missing or empty directory
is usable. A nonempty directory can be replaced only when `--replace` is given
and it is recognized as a current Weatherreporter comparison bundle; ordinary
directories, symlinks, and unsafe destinations are rejected. Cancellation and
directories, symlinks, and unsafe destinations are rejected. Existing v1
bundles are not recognized for replacement: move or remove them first.
Cancellation and
all failures before publication preserve an existing bundle, including a
cancellation observed while a replacement is being prepared. If guarded
restoration cannot complete, the error names the retained sibling bundle for
@@ -176,6 +178,12 @@ Preparation captures retain only the provider endpoint origin and reviewed
execution settings. URL user information, paths, queries, fragments, and
unrecognized provider parameters are omitted.
Each run directory may contain `preparation.json` (v3), `execution.json` (v3),
and, for a provider generation failure, `failure.json` (v1). The failure
artifact retains the safe category, HTTP status, and provider code, type, and
message for trusted debugging only. Ordinary command output never includes
those provider details.
Capture writes are confined to the requested root and fail if an unsafe
filesystem component prevents secure artifact creation.

View File

@@ -50,10 +50,14 @@ directly.
- Prompts receive curated module packages, never unbounded raw weather payloads.
- Every execution validates the exact prompt version and output contract before
collection. The selected profile is configured explicitly or declared by the
prompt; unsupported direct-key profiles and missing reported credentials fail
before collection.
prompt; profiles requiring unsupported direct API keys fail before collection.
A profile may have an empty backend identity when it supplies an endpoint;
PromptKit resolves inherited profiles and optional credential sources when it
executes them.
- Prompt and profile validation completes before weather collection. Raw output
is validated before template rendering.
- PromptKit may make at most the prompt contract's one corrective generation;
exhaustion is a validation rejection, not an application-level retry.
- Comparison validates every explicit profile before collection, prepares one
immutable report input, and delegates backend capacity to Promptkit rather
than adding an application-wide execution limit.
@@ -62,6 +66,8 @@ directly.
- Sensitive rendered prompts, schemas, input bodies, provider endpoints, and
credentials never enter normal summaries or logs. They are written only to
an explicit secure debug root when requested.
- Provider-controlled diagnostics never enter ordinary outputs; they are
retained only in explicit secure failure-debug artifacts.
## Output, Notification, And Testing Invariants

64
docs/releases/v0.13.0.md Normal file
View File

@@ -0,0 +1,64 @@
# Weatherreporter v0.13.0
This release upgrades Weatherreporter's generated-text integration through
PromptKit v0.9.0 and adds bounded structured-output repair, inherited domain
profiles, safer provider diagnostics, and repair provenance.
## Summary
Weatherreporter now permits one PromptKit-owned corrective generation when a
model returns invalid or explicitly empty structured output. Successful and
failed results report the actual correction count when validation completed,
and explicit prompt-debug capture can retain bounded provider failure details
without exposing them in ordinary output.
The stable `weather-light`, `weather-balanced`, and `weather-deep` profiles now
inherit their execution targets from PromptKit's maintained catalog. PromptKit
v0.9.0 assembles that catalog from independently versioned OpenRouter and
Rakestrawhome modules while preserving Weatherreporter's existing effective
profile targets and configured-profile override behavior.
## Compatibility
The documented CLI, configuration fields, report IDs, report filenames,
Markdown outputs, and Distributor integration remain compatible with v0.12.0.
The four generated-text prompts advance to version `2.1.0`; an eligible failed
validation may therefore make one additional provider call, increasing latency
and usage for that execution.
Comparison bundles advance from schema v1 to v2 so each completed validation
can record `repairAttempts`. Weatherreporter intentionally recognizes only the
current bundle version for guarded replacement. An existing v1 comparison
bundle cannot be replaced with `--replace` by this release.
## Upgrade
No special action is required for ordinary generation, batch execution,
configuration, or notification. Operators who want to reuse the destination
of a v1 comparison bundle must first remove or relocate that old bundle, or
select a new `--out-dir`.
Operators should account for the possibility of one additional model call
when output repair is needed. Existing configured standalone, inherited, and
endpoint-only profiles retain their documented precedence.
## Changes
- Upgraded PromptKit from v0.5.0 through v0.9.0 and adopted its maintained
profile inheritance, optional credential-source behavior, structured repair,
generation-error contract, and externally maintained provider catalogs.
- Converted the three Weatherreporter domain profiles into minimal aliases of
maintained PromptKit profiles without changing their effective model ladder.
- Added one corrective-generation allowance to all four embedded prompts and
exposed actual repair counts in generate, batch, and comparison summaries.
- Migrated the [comparison bundle contract](../integrations/comparison-bundle.md)
to v2 with strict repair-provenance validation and safe partial publication
after independent profile failures.
- Added safe provider HTTP status to ordinary errors while confining provider
code, type, and message to explicit secure `failure.json` debug artifacts.
- Strengthened offline coverage for local endpoint-only profiles, optional
credentials, concurrent comparison repair outcomes, and failure-debug
isolation.
- Updated the [PromptKit integration guide](../integrations/promptkit.md),
[configuration reference](../config.md), [CLI reference](../cli.md), and
[operations guide](../operations.md) for the implemented contracts.

8
go.mod
View File

@@ -6,9 +6,13 @@ require gopkg.in/yaml.v3 v3.0.1
require (
gitea.maximumdirect.net/eric/distributor v0.5.0
gitea.maximumdirect.net/eric/promptkit v0.5.0
gitea.maximumdirect.net/eric/promptkit v0.9.0
github.com/santhosh-tekuri/jsonschema/v6 v6.0.2
golang.org/x/sys v0.45.0
)
require golang.org/x/text v0.14.0 // indirect
require (
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 // indirect
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 // indirect
golang.org/x/text v0.14.0 // indirect
)

8
go.sum
View File

@@ -1,7 +1,11 @@
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
gitea.maximumdirect.net/eric/promptkit v0.9.0 h1:IpvDRC8L6xRxQ9hpuyKOmMc5b6MeLTKYyx+h1YAjy08=
gitea.maximumdirect.net/eric/promptkit v0.9.0/go.mod h1:oMJ/WUJImUtwJ5e+6MAGECPYAErAkOaKel0G+3T/b4E=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0 h1:lc062euk2qseO//D762i3JaFyulDNML3eQQX7DkYTho=
gitea.maximumdirect.net/eric/promptkit-backend-openrouter v1.0.0/go.mod h1:AIa7kAu2mfrRQgcspe4L+DW51WqgnALQT60lqkEywJI=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0 h1:j9YY7wsTVjzke2kHH4YAzpU0oUpM+x+nXwl1IeS+2eg=
gitea.maximumdirect.net/eric/promptkit-backend-rakestrawhome v1.0.0/go.mod h1:4RNS+LILDg4JbS4Ts9Lwy1C92wauXJIbeQaalps4Koo=
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=

View File

@@ -153,6 +153,7 @@ func outputContract(value promptkit.OutputContract) promptexec.OutputContract {
Format: string(value.Format),
ValidationMode: string(value.ValidationMode),
SchemaPath: value.SchemaPath,
RepairAttempts: value.RepairAttempts,
}
}
@@ -193,6 +194,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
promptexec.ValidationStatus(value.Validation.Status),
string(value.Validation.Mode),
value.Validation.SchemaPath,
value.Validation.RepairAttempts,
value.Validation.Errors,
)
rawOutput := []byte(nil)
@@ -203,6 +205,7 @@ func executionValue(value *promptkit.RunResult, captureDebug bool) *promptexec.E
promptexec.ValidationFailed,
string(value.Validation.Mode),
value.Validation.SchemaPath,
value.Validation.RepairAttempts,
[]string{"generated output exceeds the configured size limit"},
)
}
@@ -299,6 +302,16 @@ func classifyError(err error) error {
if errors.As(err, &capacityError) {
return promptexec.NewCapacityError(capacityError.BackendID, "prompt backend capacity is unavailable", err)
}
var generationError *promptkit.GenerationError
if errors.As(err, &generationError) {
return promptexec.NewGenerationError(
generationError.StatusCode(),
generationError.ProviderCode(),
generationError.ProviderType(),
generationError.ProviderMessage(),
err,
)
}
switch {
case errors.Is(err, promptkit.ErrInvalidConfig):
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt executor configuration is invalid", err)

View File

@@ -4,11 +4,15 @@ import (
"context"
"errors"
"fmt"
"net/http"
"net/http/httptest"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"testing"
"testing/fstest"
"time"
promptkit "gitea.maximumdirect.net/eric/promptkit"
@@ -20,12 +24,19 @@ type fakeClient struct {
mu sync.Mutex
response *promptkit.GenerateResponse
err error
outcomes []generationOutcome
next int
calls int
requests []promptkit.GenerateRequest
block bool
started chan struct{}
}
type generationOutcome struct {
response *promptkit.GenerateResponse
err error
}
type recordingReader struct {
ref promptkit.ArtifactRef
}
@@ -49,6 +60,11 @@ func (client *fakeClient) Generate(ctx context.Context, request promptkit.Genera
started := client.started
response := client.response
err := client.err
if client.next < len(client.outcomes) {
outcome := client.outcomes[client.next]
client.next++
response, err = outcome.response, outcome.err
}
client.mu.Unlock()
if started != nil {
started <- struct{}{}
@@ -102,13 +118,19 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
return client.requests[0]
}
func (client *fakeClient) allRequests() []promptkit.GenerateRequest {
client.mu.Lock()
defer client.mu.Unlock()
return append([]promptkit.GenerateRequest(nil), client.requests...)
}
func TestInspectPromptAndProfile(t *testing.T) {
adapter := newTestAdapter(t, &fakeClient{})
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.0.0")
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "2.1.0")
if err != nil {
t.Fatalf("InspectPrompt() error = %v", err)
}
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != "weather-balanced" {
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != "weather-balanced" || inspection.Output.RepairAttempts != 1 {
t.Fatalf("inspection = %#v", inspection)
}
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
@@ -175,15 +197,38 @@ model: file-light
}
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
directory := testProfileDirectory(t, `id: weather-light
directory := testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
backend: local
model: directory-light
`)
`})
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil {
t.Fatalf("New(profile directory) error = %v", err)
}
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
derived := writeProfileFile(t, `id: weather-light
base_profile: gemini-flash-latest
`)
derivedAdapter, err := New(Config{ProfileFile: derived})
if err != nil {
t.Fatalf("New(derived profile) error = %v", err)
}
assertProfile(t, derivedAdapter, "weather-light", "openrouter", "~google/gemini-flash-latest")
}
func TestConfiguredBaseProfileOverridesEmbeddedProfileTarget(t *testing.T) {
directory := testProfileDirectory(t, map[string]string{
"deepseek.yml": `id: deepseek-4-flash
backend: local
model: shadowed-deepseek
`,
})
adapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil {
t.Fatalf("New() error = %v", err)
}
assertProfile(t, adapter, "weather-light", promptkit.BackendLocal, "shadowed-deepseek")
}
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
@@ -194,19 +239,69 @@ func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T)
assertProfile(t, adapter, "weather-light", "", "weather-local")
}
func TestMaintainedWeatherLightLocalProfileExampleExecutesThroughProductionClient(t *testing.T) {
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
for _, test := range []struct {
name string
credentialSource string
}{
{name: "without credential source"},
{name: "with blank optional credential source", credentialSource: "\napi_key_env: WEATHERREPORTER_TEST_MISSING_KEY\n"},
} {
t.Run(test.name, func(t *testing.T) {
var authorization string
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, request *http.Request) {
authorization = request.Header.Get("Authorization")
writer.Header().Set("Content-Type", "application/json")
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":12,"completion_tokens":8,"total_tokens":20}}`, validResponse().Content)
}))
defer server.Close()
example, err := os.ReadFile(filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml"))
if err != nil {
t.Fatal(err)
}
profile := strings.Replace(string(example), "http://127.0.0.1:11434/v1", server.URL+"/v1", 1) + test.credentialSource
adapter, err := New(Config{ProfileFile: writeProfileFile(t, profile)})
if err != nil {
t.Fatalf("New() error = %v", err)
}
request := testExecuteRequest()
request.ProfileID = "weather-light"
var preparation promptexec.Preparation
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
preparation = value
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if authorization != "" {
t.Fatalf("Authorization header = %q, want absent", authorization)
}
if preparation.ProfileID != "weather-light" || preparation.BackendID != "" || preparation.ModelName != "weather-local" || preparation.Output.RepairAttempts != 1 {
t.Fatalf("preparation = %#v", preparation)
}
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" || result.Validation.Status != promptexec.ValidationPassed || result.Validation.RepairAttempts != 0 {
t.Fatalf("execution = %#v", result)
}
})
}
}
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: other-profile
backend: openrouter
model: other-model
`)})
`})})
if err != nil {
t.Fatalf("New(absent profile) error = %v", err)
}
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, map[string]string{"profile.yml": `id: weather-light
backend: openrouter
`)})
`})})
if err != nil {
t.Fatalf("New(malformed profile) error = %v", err)
}
@@ -215,6 +310,78 @@ backend: openrouter
}
}
func TestProfileResolutionReturnsConfiguredInheritanceFailures(t *testing.T) {
tests := []struct {
name string
profile string
profiles map[string]string
}{
{
name: "missing base",
profile: "missing-base",
profiles: map[string]string{"missing.yml": `id: missing-base
base_profile: unavailable
`},
},
{
name: "cyclic bases",
profile: "first",
profiles: map[string]string{
"first.yml": `id: first
base_profile: second
`,
"second.yml": `id: second
base_profile: first
`,
},
},
{
name: "malformed base",
profile: "child",
profiles: map[string]string{
"child.yml": `id: child
base_profile: malformed
`,
"malformed.yml": `id: malformed
base_profile: [not-a-profile]
`,
},
},
{
name: "incomplete target",
profile: "incomplete",
profiles: map[string]string{"incomplete.yml": `id: incomplete
backend: openrouter
`},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
adapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, test.profiles)})
if err != nil {
t.Fatalf("New() error = %v", err)
}
if _, err := adapter.InspectProfile(context.Background(), test.profile); err == nil {
t.Fatal("InspectProfile() error = nil, want configured inheritance error")
}
})
}
}
func TestRakestrawhomeBuiltInProfileInspectsOffline(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
profile, err := adapter.InspectProfile(context.Background(), "rakestrawhome-gemma-4-31b")
if err != nil {
t.Fatalf("InspectProfile() error = %v", err)
}
if profile.ProfileID != "rakestrawhome-gemma-4-31b" || profile.BackendID != "rakestrawhome" || profile.ModelName == "" {
t.Fatalf("profile = %#v", profile)
}
}
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
@@ -287,7 +454,7 @@ func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
}
request := promptexec.ExecuteRequest{
PromptID: "weather.hourly_generated_text",
PromptVersion: "2.0.0",
PromptVersion: "2.1.0",
ProfileID: "weather-light",
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
}
@@ -400,6 +567,114 @@ func TestExecuteReturnsCompletedValidationRejection(t *testing.T) {
}
}
func TestExecuteMapsCorrectiveGenerationResults(t *testing.T) {
valid := `{"summary":"valid"}`
invalid := `{"summary":42}`
tests := []struct {
name string
outcomes []generationOutcome
wantStatus promptexec.ValidationStatus
wantRepairs int
wantCalls int
wantRaw string
wantUsage promptexec.TokenUsage
}{
{
name: "first pass valid",
outcomes: []generationOutcome{{response: generationResponse(valid, 2, 3, 5)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 0,
wantCalls: 1,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 2, CompletionTokens: 3, TotalTokens: 5},
},
{
name: "empty output repaired",
outcomes: []generationOutcome{{response: generationResponse("", 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
{
name: "invalid output repaired",
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(valid, 7, 11, 18)}},
wantStatus: promptexec.ValidationPassed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: valid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
{
name: "repair budget exhausted",
outcomes: []generationOutcome{{response: generationResponse(invalid, 2, 3, 5)}, {response: generationResponse(invalid, 7, 11, 18)}},
wantStatus: promptexec.ValidationFailed,
wantRepairs: 1,
wantCalls: 2,
wantRaw: invalid,
wantUsage: promptexec.TokenUsage{PromptTokens: 9, CompletionTokens: 14, TotalTokens: 23},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
client := &fakeClient{outcomes: test.outcomes}
adapter := newRepairAdapter(t, client, "https://repair.example/v1")
var preparation promptexec.Preparation
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
preparation = value
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if preparation.Output.RepairAttempts != 1 || result == nil || result.Validation.Status != test.wantStatus || result.Validation.RepairAttempts != test.wantRepairs || string(result.RawOutput) != test.wantRaw || result.Usage != test.wantUsage {
t.Fatalf("preparation/result = %#v/%#v", preparation, result)
}
requests := client.allRequests()
if len(requests) != test.wantCalls {
t.Fatalf("provider requests = %d, want %d", len(requests), test.wantCalls)
}
if test.wantCalls == 2 && !reflect.DeepEqual(requests[0].Target, requests[1].Target) {
t.Fatalf("corrective target = %#v, want same prepared identity as %#v", requests[1].Target, requests[0].Target)
}
if result.ProfileID != preparation.ProfileID || result.BackendID != preparation.BackendID || result.ModelName != preparation.ModelName || result.PromptID != preparation.PromptID || result.PromptVersion != preparation.PromptVersion || result.PromptHash != preparation.PromptHash {
t.Fatalf("prepared/result identity = %#v/%#v", preparation, result)
}
})
}
}
func TestExecuteMapsCorrectiveGenerationError(t *testing.T) {
const providerBody = `{"error":{"code":"repair-code","type":"repair-type","message":"repair-message"}}`
calls := 0
server := httptest.NewServer(http.HandlerFunc(func(writer http.ResponseWriter, _ *http.Request) {
calls++
if calls == 1 {
writer.Header().Set("Content-Type", "application/json")
_, _ = fmt.Fprintf(writer, `{"choices":[{"message":{"content":%q}}],"usage":{"prompt_tokens":2,"completion_tokens":3,"total_tokens":5}}`, `{"summary":42}`)
return
}
writer.Header().Set("Content-Type", "application/json")
writer.WriteHeader(http.StatusUnprocessableEntity)
_, _ = writer.Write([]byte(providerBody))
}))
defer server.Close()
adapter := newRepairAdapter(t, nil, server.URL)
result, err := adapter.Execute(context.Background(), repairExecuteRequest(), nil)
if result != nil || err == nil || calls != 2 {
t.Fatalf("result/error/calls = %#v/%v/%d", result, err, calls)
}
var generationError *promptexec.GenerationError
if !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusUnprocessableEntity || generationError.ProviderCode() != "repair-code" || generationError.ProviderType() != "repair-type" || generationError.ProviderMessage() != "repair-message" {
t.Fatalf("generation error = %#v", err)
}
if strings.Contains(err.Error(), "repair-message") || !errors.Is(err, promptkit.ErrLLMGenerate) {
t.Fatalf("generation error = %v", err)
}
}
func TestExecuteDropsOversizedGeneratedOutput(t *testing.T) {
client := &fakeClient{response: &promptkit.GenerateResponse{Content: strings.Repeat("x", generatedtext.MaxGeneratedTextBytes+1)}}
adapter := newTestAdapter(t, client)
@@ -507,12 +782,12 @@ func TestNewValidatesConfiguration(t *testing.T) {
}
}
func TestLocalBackendAndMissingCredentialBehavior(t *testing.T) {
func TestLocalBackendAndOptionalCredentialSourceBehavior(t *testing.T) {
t.Setenv("WEATHERREPORTER_TEST_MISSING_KEY", "")
profiles := testProfileDirectory(t, `id: local-profile
profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: local-profile
backend: local
model: local-model
`)
`})
adapter, err := newAdapterForTest(Config{
ProfileDirectory: profiles,
LocalEndpoint: "https://local.example/v1",
@@ -529,11 +804,11 @@ model: local-model
t.Fatalf("capacity classification = %v", got)
}
credentialProfiles := testProfileDirectory(t, `id: credential-profile
credentialProfiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: credential-profile
endpoint: https://profile.example/v1
model: test-model
api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
`)
`})
client := &fakeClient{response: validResponse()}
credentialAdapter, err := newAdapterForTest(Config{ProfileDirectory: credentialProfiles}, client)
if err != nil {
@@ -546,8 +821,8 @@ api_key_env: WEATHERREPORTER_TEST_MISSING_KEY
request := testExecuteRequest()
request.ProfileID = "credential-profile"
result, err := credentialAdapter.Execute(context.Background(), request, nil)
if result != nil || promptexec.CategoryOf(err) != promptexec.MissingCredential || client.callCount() != 0 {
t.Fatalf("credential result/category/calls = %#v/%q/%d", result, promptexec.CategoryOf(err), client.callCount())
if err != nil || result == nil || client.callCount() != 1 {
t.Fatalf("credential result/error/calls = %#v/%v/%d", result, err, client.callCount())
}
}
@@ -568,14 +843,14 @@ func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, mo
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
t.Helper()
profiles := testProfileDirectory(t, `id: test-profile
profiles := testProfileDirectory(t, map[string]string{"profile.yml": `id: test-profile
endpoint: https://profile.example/v1
model: test-model
temperature: 0.2
max_tokens: 300
top_p: 1
timeout_seconds: 30
`)
`})
options = append(options, promptkit.WithLLMClient(client))
adapter, err := newAdapter(Config{ProfileDirectory: profiles, Timeout: time.Second}, options...)
if err != nil {
@@ -584,13 +859,15 @@ timeout_seconds: 30
return adapter
}
func testProfileDirectory(t *testing.T, profile string) string {
func testProfileDirectory(t *testing.T, profiles map[string]string) string {
t.Helper()
profiles := t.TempDir()
if err := os.WriteFile(filepath.Join(profiles, "profile.yml"), []byte(profile), 0o600); err != nil {
directory := t.TempDir()
for name, profile := range profiles {
if err := os.WriteFile(filepath.Join(directory, name), []byte(profile), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
return profiles
}
return directory
}
func writeProfileFile(t *testing.T, profile string) string {
@@ -605,7 +882,7 @@ func writeProfileFile(t *testing.T, profile string) string {
func testExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{
PromptID: "weather.daily_generated_text",
PromptVersion: "2.0.0",
PromptVersion: "2.1.0",
ProfileID: "test-profile",
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
}
@@ -618,6 +895,61 @@ func validResponse() *promptkit.GenerateResponse {
}
}
func generationResponse(content string, promptTokens int, completionTokens int, totalTokens int) *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{
Content: content,
Usage: promptkit.TokenUsage{
PromptTokens: promptTokens,
CompletionTokens: completionTokens,
TotalTokens: totalTokens,
},
}
}
func newRepairAdapter(t *testing.T, client promptkit.LLMClient, endpoint string) *Adapter {
t.Helper()
profiles := testProfileDirectory(t, map[string]string{"profile.yml": "id: repair-profile\nendpoint: " + endpoint + "\nmodel: repair-model\n"})
options := []promptkit.Option{
promptkit.WithPromptFS(fstest.MapFS{
"repair.yml": &fstest.MapFile{Data: []byte(`id: weather.repair
version: "1.0.0"
default_profile: repair-profile
inputs:
- name: data_package
required: true
content_type: application/yaml
messages:
- role: user
content: "{{input \"data_package\"}}"
output:
format: json
validation_mode: json_schema
schema_path: repair.schema.json
repair_attempts: 1
`)}}, "."),
promptkit.WithSchemaFS(fstest.MapFS{
"repair.schema.json": &fstest.MapFile{Data: []byte(`{"type":"object","properties":{"summary":{"type":"string"}},"required":["summary"],"additionalProperties":false}`)},
}, "."),
}
if client != nil {
options = append(options, promptkit.WithLLMClient(client))
}
adapter, err := newAdapter(Config{ProfileDirectory: profiles}, options...)
if err != nil {
t.Fatalf("newAdapter() error = %v", err)
}
return adapter
}
func repairExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{
PromptID: "weather.repair",
PromptVersion: "1.0.0",
ProfileID: "repair-profile",
DataPackage: []byte("report: repair\n"),
}
}
func hourlyValidResponse() *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,

View File

@@ -89,6 +89,7 @@ type ReportResult struct {
ModelName string
SourceWarnings []weatherdata.SourceWarning
ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string
OutputPath string
Notification *NotificationResult
@@ -139,6 +140,7 @@ type BatchReportResult struct {
ModelName string `json:"modelName,omitempty"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
ValidationStatus promptexec.ValidationStatus `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
LLMDebugPath string `json:"llmDebugPath,omitempty"`
OutputPath string `json:"outputPath,omitempty"`
}
@@ -457,6 +459,9 @@ func copyBatchReportDetails(item *BatchReportResult, result *ReportResult) {
item.Timezone = result.Timezone
item.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
item.ValidationStatus = result.ValidationStatus
if result.RepairAttempts != nil {
item.RepairAttempts = repairAttemptsPointer(*result.RepairAttempts)
}
}
func batchInspectionCandidates(req BatchRequest, now time.Time) ([]report.Resolved, error) {

View File

@@ -3,7 +3,6 @@ package app
import (
"context"
"fmt"
"os"
"path/filepath"
"time"
@@ -60,6 +59,7 @@ type ComparisonProfileResult struct {
ModelName string
Status string
ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
ReportPath string
LLMDebugPath string
Error *comparison.SafeError
@@ -121,7 +121,7 @@ func compareDetailed(ctx context.Context, req ComparisonRequest, publish compari
}
defer func() { _ = debugWriter.Close() }()
inspection, err := InspectComparisonExecution(ctx, ComparisonInspectionRequest{
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor, LookupEnv: os.LookupEnv,
Resolved: resolved, ProfileIDs: req.ProfileIDs, Executor: req.Executor,
})
result.PromptID, result.PromptVersion, result.PromptHash = inspection.PromptID, inspection.PromptVersion, inspection.PromptHash
if err != nil {
@@ -198,6 +198,9 @@ func copyComparisonOutcomes(result *ComparisonResult, outcomes []comparisonProfi
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
Status: outcome.Status, ValidationStatus: outcome.ValidationStatus, LLMDebugPath: outcome.LLMDebugPath, Error: outcome.Error,
}
if outcome.RepairAttempts != nil {
profile.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
if published && outcome.Status == comparison.StatusSucceeded {
profile.ReportPath = filepath.Join(result.OutputDirectory, outcome.ReportPath)
}
@@ -227,6 +230,9 @@ func comparisonBundle(result *ComparisonResult, dataPackage []byte, outcomes []c
Position: outcome.Position, ProfileID: outcome.ProfileID, BackendID: outcome.BackendID, ModelName: outcome.ModelName,
Status: outcome.Status, ValidationStatus: string(outcome.ValidationStatus), Error: outcome.Error,
}
if outcome.RepairAttempts != nil {
manifestResult.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
if outcome.Status == comparison.StatusSucceeded {
manifestResult.ReportPath = outcome.ReportPath
bundle.Reports = append(bundle.Reports, comparison.BundleReport{Position: outcome.Position, Path: outcome.ReportPath, Markdown: outcome.Markdown})

View File

@@ -31,6 +31,7 @@ type comparisonProfileOutcome struct {
ModelName string
Status string
ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
ReportPath string
Markdown []byte
LLMDebugPath string
@@ -108,6 +109,9 @@ func executeComparisonProfile(ctx context.Context, req comparisonExecutionReques
})
outcome.ProfileID, outcome.BackendID, outcome.ModelName = execution.ProfileID, execution.BackendID, execution.ModelName
outcome.ValidationStatus = execution.ValidationStatus
if execution.RepairAttempts != nil {
outcome.RepairAttempts = repairAttemptsPointer(*execution.RepairAttempts)
}
outcome.LLMDebugPath = execution.LLMDebugPath
if err != nil {
outcome.canceled = cancellationError(err)
@@ -169,9 +173,14 @@ func comparisonExecutionMessage(err error) string {
if errors.Is(err, context.DeadlineExceeded) {
return "profile execution deadline exceeded"
}
operation := "profile execution"
var execution *profileExecutionError
if errors.As(err, &execution) {
return comparison.TruncateErrorMessage(execution.operation + " failed")
operation = execution.operation
}
return "profile execution failed"
var generation *promptexec.GenerationError
if errors.As(err, &generation) && generation.StatusCode() > 0 {
return comparison.TruncateErrorMessage(fmt.Sprintf("%s failed (HTTP %d)", operation, generation.StatusCode()))
}
return comparison.TruncateErrorMessage(operation + " failed")
}

View File

@@ -4,9 +4,11 @@ import (
"context"
"errors"
"fmt"
"net/http"
"os"
"path/filepath"
"reflect"
"strings"
"sync"
"testing"
"time"
@@ -72,6 +74,66 @@ func TestExecuteComparisonProfilesContinuesAfterProfileFailure(t *testing.T) {
}
}
func TestExecuteComparisonProfilesPreservesIndependentRepairOutcomes(t *testing.T) {
prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(4)
executor := newBarrierExecutor(profiles)
executor.setValidation(profiles[0].ProfileID, promptexec.ValidationPassed, 0)
executor.setValidation(profiles[1].ProfileID, promptexec.ValidationPassed, 1)
executor.setValidation(profiles[2].ProfileID, promptexec.ValidationFailed, 1)
executor.setError(profiles[3].ProfileID, errors.New("provider failure"))
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", Executor: executor,
}, executor)
waitForProfileStarts(t, executor, profiles, results)
executor.releaseAll()
result := <-results
wantStatuses := []string{comparison.StatusSucceeded, comparison.StatusSucceeded, comparison.StatusFailed, comparison.StatusFailed}
wantValidations := []promptexec.ValidationStatus{promptexec.ValidationPassed, promptexec.ValidationPassed, promptexec.ValidationFailed, ""}
wantRepairs := []*int{intPointer(0), intPointer(1), intPointer(1), nil}
for index, outcome := range result.Outcomes {
if outcome.Status != wantStatuses[index] || outcome.ValidationStatus != wantValidations[index] || !reflect.DeepEqual(outcome.RepairAttempts, wantRepairs[index]) {
t.Fatalf("outcome[%d] = %#v, want status/validation/repairs %q/%q/%#v", index, outcome, wantStatuses[index], wantValidations[index], wantRepairs[index])
}
}
}
func TestExecuteComparisonProfilesCapturesConcurrentProviderFailures(t *testing.T) {
prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(2)
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
if err != nil {
t.Fatalf("NewPromptDebugWriter() error = %v", err)
}
markers := []string{"first-provider-private-marker", "second-provider-private-marker"}
statuses := []int{http.StatusTooManyRequests, http.StatusServiceUnavailable}
executor := newBarrierExecutor(profiles)
for index, profile := range profiles {
executor.setError(profile.ProfileID, promptexec.NewGenerationError(statuses[index], "provider_code", "provider_type", markers[index], nil))
}
results := startComparisonExecution(t, context.Background(), comparisonExecutionRequest{
Prepared: prepared, Inspection: comparisonInspection(prompt, profiles), ComparisonID: "comparison_daily", DebugWriter: debugWriter, Executor: executor,
}, executor)
waitForProfileStarts(t, executor, profiles, results)
executor.releaseAll()
result := <-results
for index, outcome := range result.Outcomes {
if outcome.Status != comparison.StatusFailed || outcome.Error == nil || outcome.Error.Category != string(promptexec.Generation) || outcome.Error.Message != fmt.Sprintf("execute prompt failed (HTTP %d)", statuses[index]) || strings.Contains(outcome.Error.Message, markers[index]) || outcome.LLMDebugPath == "" {
t.Fatalf("outcome[%d] = %#v", index, outcome)
}
failure, readErr := os.ReadFile(filepath.Join(outcome.LLMDebugPath, "failure.json"))
if readErr != nil {
t.Fatal(readErr)
}
if !strings.Contains(string(failure), markers[index]) || strings.Contains(string(failure), markers[1-index]) {
t.Fatalf("failure[%d] = %s", index, failure)
}
}
}
func TestExecuteComparisonProfilesPropagatesCancellationAndJoins(t *testing.T) {
prepared, prompt := preparedDailyProfile(t)
profiles := comparisonProfiles(4)
@@ -139,6 +201,8 @@ type barrierExecutor struct {
releases map[string]chan struct{}
requests map[string]promptexec.ExecuteRequest
errors map[string]error
validations map[string]promptexec.ValidationStatus
repairAttempts map[string]int
profiles map[string]ComparisonProfileInspection
inFlight int
maximum int
@@ -153,7 +217,7 @@ func newBarrierExecutor(profiles []ComparisonProfileInspection) *barrierExecutor
}
return &barrierExecutor{
started: make(chan string, len(profiles)), callbackFailures: make(chan error, len(profiles)), releases: releases,
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, profiles: identities,
requests: make(map[string]promptexec.ExecuteRequest, len(profiles)), errors: map[string]error{}, validations: map[string]promptexec.ValidationStatus{}, repairAttempts: map[string]int{}, profiles: identities,
}
}
@@ -170,7 +234,8 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
e.mu.Lock()
profile := e.profiles[req.ProfileID]
e.mu.Unlock()
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
definition := generationDefinitionForPrompt(req.PromptID)
if err := callback(promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}, nil); err != nil {
e.callbackFailures <- err
return nil, err
}
@@ -194,15 +259,20 @@ func (e *barrierExecutor) Execute(ctx context.Context, req promptexec.ExecuteReq
e.mu.Lock()
e.inFlight--
err := e.errors[req.ProfileID]
validationStatus := e.validations[req.ProfileID]
repairAttempts := e.repairAttempts[req.ProfileID]
e.mu.Unlock()
if err != nil {
return nil, err
}
if validationStatus == "" {
validationStatus = promptexec.ValidationPassed
}
return &promptexec.Execution{
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash,
ProfileID: req.ProfileID, BackendID: profile.BackendID, ModelName: profile.ModelName,
StartedAt: stamp, EndedAt: stamp, RawOutput: comparisonRawOutput(),
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil),
Validation: promptexec.NewValidation(validationStatus, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil),
}, nil
}
@@ -219,6 +289,13 @@ func (e *barrierExecutor) setError(profileID string, err error) {
e.errors[profileID] = err
}
func (e *barrierExecutor) setValidation(profileID string, status promptexec.ValidationStatus, repairAttempts int) {
e.mu.Lock()
defer e.mu.Unlock()
e.validations[profileID] = status
e.repairAttempts[profileID] = repairAttempts
}
func (e *barrierExecutor) release(profileID string) {
close(e.releases[profileID])
}
@@ -317,4 +394,8 @@ func bytesEqual(left, right []byte) bool {
return reflect.DeepEqual(left, right)
}
func intPointer(value int) *int {
return &value
}
var _ promptexec.Executor = (*barrierExecutor)(nil)

View File

@@ -87,6 +87,40 @@ func TestCompareDetailedPublishesPartialBundleAndReturnsAggregateError(t *testin
}
}
func TestCompareDetailedPublishesPostValidationProfileFailure(t *testing.T) {
bundle := generationBundle(t)
executor := &generationExecutor{complete: func(execution *promptexec.Execution) {
if execution.ProfileID == "weather-deep" {
execution.RawOutput = []byte(`{"summary":42}`)
}
}}
result, err := CompareDetailed(context.Background(), ComparisonRequest{
Config: comparisonConfig(), Report: ReportDaily, ProfileIDs: []string{"weather-light", "weather-deep"},
WorkingDir: t.TempDir(), Date: generationTime("2026-05-29T12:00:00-05:00"),
Clock: timeutil.FixedClock{Time: generationTime("2026-05-29T08:30:00-05:00")},
Collector: &generationCollector{bundle: &bundle}, Executor: executor,
})
if err == nil || result == nil || result.Succeeded != 1 || result.Failed != 1 || result.ManifestPath == "" {
t.Fatalf("CompareDetailed() result/error = %#v/%v", result, err)
}
failure := result.Results[1]
if failure.Status != comparison.StatusFailed || failure.ValidationStatus != promptexec.ValidationPassed || failure.RepairAttempts == nil || *failure.RepairAttempts != 0 {
t.Fatalf("post-validation failure = %#v", failure)
}
data, readErr := os.ReadFile(result.ManifestPath)
if readErr != nil {
t.Fatal(readErr)
}
var manifest comparison.Manifest
if decodeErr := json.Unmarshal(data, &manifest); decodeErr != nil {
t.Fatal(decodeErr)
}
manifestFailure := manifest.Results[1]
if manifestFailure.ValidationStatus != "passed" || manifestFailure.RepairAttempts == nil || *manifestFailure.RepairAttempts != 0 {
t.Fatalf("published post-validation failure = %#v", manifestFailure)
}
}
func TestCompareDetailedRetainsCommittedPathsWhenBackupCleanupFails(t *testing.T) {
for _, test := range []struct {
name string

View File

@@ -4,6 +4,7 @@ import (
"context"
"encoding/json"
"errors"
"net/http"
"os"
"path/filepath"
"strings"
@@ -68,6 +69,7 @@ type generationExecutor struct {
beforeExecute func(promptexec.ExecuteRequest)
cancelBeforeReturn context.CancelFunc
validation promptexec.ValidationStatus
repairAttempts int
validations map[string]promptexec.ValidationStatus
rawOutput []byte
waitForCancellation map[string]bool
@@ -88,7 +90,7 @@ func (e *generationExecutor) InspectPrompt(_ context.Context, id, version string
return promptexec.PromptInspection{}, e.inspectErr
}
definition := generationDefinitionForPrompt(id)
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"}}, nil
return promptexec.PromptInspection{PromptID: id, PromptVersion: version, PromptHash: generationPromptHash, DefaultProfileID: "fixture", Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}}, Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}}, nil
}
func (e *generationExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
generationExecutorMu.Lock()
@@ -112,7 +114,8 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
calls = 1
}
for range calls {
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID + ".generated_text.schema.json"}, StartedAt: stamp, EndedAt: stamp}
definition := generationDefinitionForPrompt(req.PromptID)
preparation := promptexec.Preparation{PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts}, StartedAt: stamp, EndedAt: stamp}
if prepare != nil {
prepare(&preparation)
}
@@ -128,6 +131,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
profileErr := e.executeErrors[req.ProfileID]
executeErr := e.executeErr
status := e.validation
repairAttempts := e.repairAttempts
if profileStatus, ok := e.validations[req.ProfileID]; ok {
status = profileStatus
}
@@ -162,7 +166,7 @@ func (e *generationExecutor) Execute(ctx context.Context, req promptexec.Execute
if cancelBeforeReturn != nil {
cancelBeforeReturn()
}
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", nil)}
execution := &promptexec.Execution{RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: generationPromptHash, RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model", StartedAt: stamp, EndedAt: stamp, RawOutput: rawOutput, Validation: promptexec.NewValidation(status, "json_schema", generationDefinitionForPrompt(req.PromptID).GeneratedTextSchemaID+".generated_text.schema.json", repairAttempts, nil)}
if complete != nil {
complete(execution)
}
@@ -546,6 +550,66 @@ func TestGenerateDetailedWritesRequestedPromptDebugArtifacts(t *testing.T) {
}
}
func TestGenerateDetailedCapturesProviderFailureOnlyInDebugArtifacts(t *testing.T) {
bundle := generationBundle(t)
debugRoot := t.TempDir()
const marker = "provider-private-generation-marker"
result, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: generationConfig(), Report: ReportDaily,
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle},
Executor: &generationExecutor{executeErr: promptexec.NewGenerationError(http.StatusTooManyRequests, "rate_limit", "provider_error", marker, nil)},
})
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
if err == nil || result == nil || result.LLMDebugPath == "" || promptexec.CategoryOf(err) != promptexec.Generation || !strings.Contains(err.Error(), "HTTP 429") || strings.Contains(err.Error(), marker) || result.OutputPath != "" {
t.Fatalf("GenerateDetailed() result/error = %#v/%v", result, err)
}
for _, name := range []string{"preparation.json", "failure.json"} {
if _, statErr := os.Stat(filepath.Join(result.LLMDebugPath, name)); statErr != nil {
t.Fatalf("debug artifact %q: %v", name, statErr)
}
}
data, readErr := os.ReadFile(filepath.Join(result.LLMDebugPath, "failure.json"))
if readErr != nil || !strings.Contains(string(data), marker) {
t.Fatalf("failure artifact = %q, error = %v", data, readErr)
}
}
func TestGenerateDetailedPreservesProviderFailureWhenFailureDebugWriteFails(t *testing.T) {
bundle := generationBundle(t)
debugRoot := t.TempDir()
const marker = "provider-private-write-failure-marker"
var setupErr error
executor := &generationExecutor{
executeErr: promptexec.NewGenerationError(http.StatusServiceUnavailable, "unavailable", "provider_error", marker, nil),
beforeExecute: func(promptexec.ExecuteRequest) {
setupErr = filepath.Walk(debugRoot, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
if info.Name() == "preparation.json" {
return os.Mkdir(filepath.Join(filepath.Dir(path), "failure.json"), 0o700)
}
return nil
})
},
}
result, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: generationConfig(), Report: ReportDaily,
Date: generationTime("2026-05-29T12:00:00-05:00"), Now: generationTime("2026-05-29T08:30:00-05:00"),
WorkingDir: t.TempDir(), LLMDebugDir: debugRoot, Collector: &generationCollector{bundle: &bundle}, Executor: executor,
})
if errors.Is(err, promptdebug.ErrSecureCaptureUnsupported) {
t.Skipf("secure prompt debug capture is unavailable: %v", err)
}
var generationError *promptexec.GenerationError
if setupErr != nil || err == nil || result == nil || result.OutputPath != "" || promptexec.CategoryOf(err) != promptexec.Generation || !errors.As(err, &generationError) || generationError.StatusCode() != http.StatusServiceUnavailable || !strings.Contains(err.Error(), "HTTP 503") || strings.Contains(err.Error(), marker) {
t.Fatalf("GenerateDetailed() setup/result/error = %v/%#v/%v", setupErr, result, err)
}
}
func generationConfig() config.Config {
cfg := config.Defaults()
cfg.WeatherAPI.Timezone, cfg.Location.ID = "America/Chicago", "home"

View File

@@ -2,6 +2,7 @@ package app
import (
"context"
"errors"
"fmt"
"reflect"
@@ -24,6 +25,7 @@ type profileExecutionOutcome struct {
BackendID string
ModelName string
ValidationStatus promptexec.ValidationStatus
RepairAttempts *int
LLMDebugPath string
}
@@ -94,11 +96,24 @@ func executePreparedProfile(ctx context.Context, req profileExecutionRequest) (p
if callbackFailed {
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: err, callbackFailure: true}
}
if req.DebugWriter != nil && req.DebugWriter.Enabled() && req.DebugRef != nil {
var generationError *promptexec.GenerationError
if errors.As(err, &generationError) {
path, debugErr := req.DebugWriter.WriteFailure(*req.DebugRef, generationError)
if path != "" {
outcome.LLMDebugPath = path
}
if debugErr != nil {
err = errors.Join(err, promptDebugWriteError(debugErr))
}
}
}
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: classifiedPromptError("prompt execution failed", err)}
}
if execution == nil {
return outcome, nil, &profileExecutionError{operation: "execute prompt", err: promptexec.NewError(promptexec.Generation, "prompt executor returned no execution", nil)}
}
outcome.RepairAttempts = repairAttemptsPointer(execution.Validation.RepairAttempts)
if preparationCount != 1 {
return outcome, nil, &profileExecutionError{operation: "validate prompt provenance", err: promptProvenanceError()}
}
@@ -157,7 +172,7 @@ func validatePreparedExecutionRequest(req profileExecutionRequest) error {
if req.Prompt.ProfileID != "" && (req.Prompt.ProfileID != req.Profile.ProfileID || req.Prompt.BackendID != req.Profile.BackendID || req.Prompt.ModelName != req.Profile.ModelName) {
return promptProvenanceError()
}
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.BackendID == "" || req.Profile.ModelName == "" {
if req.Prompt.PromptHash == "" || req.Profile.ProfileID == "" || req.Profile.ModelName == "" {
return promptProvenanceError()
}
return nil
@@ -178,12 +193,19 @@ func validateExecutionProvenance(req profileExecutionRequest, preparation prompt
if execution.PromptID != preparation.PromptID || execution.PromptVersion != preparation.PromptVersion || execution.PromptHash != preparation.PromptHash ||
execution.RenderedPromptHash != preparation.RenderedPromptHash || !reflect.DeepEqual(execution.InputHashes, preparation.InputHashes) ||
execution.ProfileID != preparation.ProfileID || execution.BackendID != preparation.BackendID || execution.ModelName != preparation.ModelName ||
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" {
execution.Validation.Mode != "json_schema" || execution.Validation.SchemaPath != definition.GeneratedTextSchemaID+".generated_text.schema.json" ||
execution.Validation.RepairAttempts < 0 || execution.Validation.RepairAttempts > preparation.Output.RepairAttempts ||
preparation.Output.RepairAttempts != definition.GeneratedTextRepairAttempts {
return promptProvenanceError()
}
return nil
}
func repairAttemptsPointer(value int) *int {
copy := value
return &copy
}
func promptProvenanceError() error {
return promptexec.NewError(promptexec.InvalidConfiguration, "prompt execution provenance is inconsistent", nil)
}

View File

@@ -26,7 +26,7 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
if err != nil {
t.Fatalf("executePreparedProfile() error = %v", err)
}
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
if len(rendered) == 0 || outcome.ValidationStatus != promptexec.ValidationPassed || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 0 || outcome.ProfileID != inspection.ProfileID || executor.executeCalls != 1 {
t.Fatalf("outcome/rendered/execution calls = %#v/%q/%d", outcome, rendered, executor.executeCalls)
}
if _, statErr := os.Stat(outputPath); !os.IsNotExist(statErr) {
@@ -34,6 +34,20 @@ func TestExecutePreparedProfileRendersWithoutPublishing(t *testing.T) {
}
}
func TestExecutePreparedProfileRetainsCompletedRepairAttemptsOnLaterFailure(t *testing.T) {
prepared, inspection := preparedDailyProfile(t)
prepared.resolved.Definition.GeneratedTextRepairAttempts = 1
executor := &generationExecutor{repairAttempts: 1, rawOutput: []byte(`{"summary":42}`), prepare: func(value *promptexec.Preparation) { value.Output.RepairAttempts = 1 }}
outcome, _, err := executePreparedProfile(context.Background(), profileExecutionRequest{
Prepared: prepared, Prompt: inspection,
Profile: promptexec.ProfileInspection{ProfileID: inspection.ProfileID, BackendID: inspection.BackendID, ModelName: inspection.ModelName},
Executor: executor,
})
if err == nil || outcome.RepairAttempts == nil || *outcome.RepairAttempts != 1 {
t.Fatalf("outcome/error = %#v/%v", outcome, err)
}
}
func TestExecutePreparedProfileKeepsDebugCallbackFailureLocal(t *testing.T) {
prepared, inspection := preparedDailyProfile(t)
debugWriter, err := promptdebug.NewPromptDebugWriter(t.TempDir())

View File

@@ -48,6 +48,9 @@ func generatePromptReport(ctx context.Context, req promptReportRequest) (*Report
})
result.ProfileID, result.BackendID, result.ModelName = outcome.ProfileID, outcome.BackendID, outcome.ModelName
result.ValidationStatus = outcome.ValidationStatus
if outcome.RepairAttempts != nil {
result.RepairAttempts = repairAttemptsPointer(*outcome.RepairAttempts)
}
result.LLMDebugPath = outcome.LLMDebugPath
if err != nil {
return result, generatedProfileExecutionError(req.Resolved, result.RunID, err)

View File

@@ -2,7 +2,6 @@ package app
import (
"context"
"os"
"strings"
"gitea.maximumdirect.net/eric/weatherreporter/internal/comparison"
@@ -18,7 +17,6 @@ type PromptInspectionRequest struct {
Resolved report.Resolved
Executor promptexec.Executor
Promptkit config.PromptkitConfig
LookupEnv func(string) (string, bool)
}
// PromptInspectionResult contains only safe identity and provenance from a
@@ -39,7 +37,6 @@ type PromptExecutionsInspectionRequest struct {
Resolved []report.Resolved
Executor promptexec.Executor
Promptkit config.PromptkitConfig
LookupEnv func(string) (string, bool)
}
// ComparisonInspectionRequest contains the explicit profile selection for one
@@ -48,7 +45,6 @@ type ComparisonInspectionRequest struct {
Resolved report.Resolved
ProfileIDs []string
Executor promptexec.Executor
LookupEnv func(string) (string, bool)
}
// ComparisonInspectionResult contains the safe, shared prompt identity and
@@ -76,7 +72,6 @@ func InspectPromptExecution(ctx context.Context, req PromptInspectionRequest) (P
Resolved: []report.Resolved{req.Resolved},
Executor: req.Executor,
Promptkit: req.Promptkit,
LookupEnv: req.LookupEnv,
})
if err != nil {
return PromptInspectionResult{}, err
@@ -111,7 +106,7 @@ func InspectPromptExecutions(ctx context.Context, req PromptExecutionsInspection
}
profile, ok := profiles[profileID]
if !ok {
profile, err = inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv)
profile, err = inspectPromptProfile(ctx, req.Executor, profileID)
if err != nil {
return nil, err
}
@@ -154,7 +149,7 @@ func InspectComparisonExecution(ctx context.Context, req ComparisonInspectionReq
handler: handler,
}
for _, profileID := range req.ProfileIDs {
profile, err := inspectPromptProfile(ctx, req.Executor, profileID, req.LookupEnv)
profile, err := inspectPromptProfile(ctx, req.Executor, profileID)
if err != nil {
return result, comparisonInspectionError("comparison profile inspection failed", err)
}
@@ -190,7 +185,7 @@ func inspectPromptContract(ctx context.Context, executor promptexec.Executor, de
return inspection, nil
}
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string, lookupEnv func(string) (string, bool)) (promptexec.ProfileInspection, error) {
func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, profileID string) (promptexec.ProfileInspection, error) {
profile, err := executor.InspectProfile(ctx, profileID)
if err != nil {
return promptexec.ProfileInspection{}, promptInspectionError("profile inspection failed", err)
@@ -201,16 +196,7 @@ func inspectPromptProfile(ctx context.Context, executor promptexec.Executor, pro
if profile.CredentialRequired {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile requires an unsupported direct API key", nil)
}
if strings.TrimSpace(profile.APIKeyEnv) != "" {
if lookupEnv == nil {
lookupEnv = os.LookupEnv
}
value, present := lookupEnv(profile.APIKeyEnv)
if !present || strings.TrimSpace(value) == "" {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.MissingCredential, "selected profile credential is unavailable", nil)
}
}
if strings.TrimSpace(profile.BackendID) == "" || strings.TrimSpace(profile.ModelName) == "" {
if strings.TrimSpace(profile.ModelName) == "" {
return promptexec.ProfileInspection{}, promptexec.NewError(promptexec.InvalidConfiguration, "profile inspection did not return a complete execution identity", nil)
}
return profile, nil
@@ -221,7 +207,7 @@ func validPromptInput(inputs []promptexec.InputDefinition) bool {
}
func validPromptOutput(definition report.Definition, output promptexec.OutputContract) bool {
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json"
return output.Format == "json" && output.ValidationMode == "json_schema" && output.SchemaPath == definition.GeneratedTextSchemaID+".generated_text.schema.json" && output.RepairAttempts == definition.GeneratedTextRepairAttempts
}
func promptInspectionError(operation string, err error) error {

View File

@@ -50,7 +50,6 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
name string
prompt promptexec.PromptInspection
profile promptexec.ProfileInspection
lookupEnv func(string) (string, bool)
wantCategory promptexec.ErrorCategory
}{
{
@@ -81,9 +80,9 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
wantCategory: promptexec.InvalidConfiguration,
},
{
name: "missing profile backend",
name: "missing profile model",
prompt: basePrompt,
profile: promptexec.ProfileInspection{ProfileID: "default-profile", ModelName: "model"},
profile: promptexec.ProfileInspection{ProfileID: "default-profile", BackendID: "backend"},
wantCategory: promptexec.InvalidConfiguration,
},
{
@@ -92,18 +91,11 @@ func TestInspectPromptExecutionRejectsInvalidContractsAndCredentials(t *testing.
profile: promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true},
wantCategory: promptexec.MissingCredential,
},
{
name: "missing environment credential",
prompt: basePrompt,
profile: promptexec.ProfileInspection{ProfileID: "default-profile", APIKeyEnv: "PROMPT_API_KEY"},
lookupEnv: func(string) (string, bool) { return "", false },
wantCategory: promptexec.MissingCredential,
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
executor := &inspectionExecutor{prompt: test.prompt, profiles: map[string]promptexec.ProfileInspection{"default-profile": test.profile}}
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor, LookupEnv: test.lookupEnv})
_, err := InspectPromptExecution(context.Background(), PromptInspectionRequest{Resolved: resolved, Executor: executor})
if err == nil || promptexec.CategoryOf(err) != test.wantCategory {
t.Fatalf("error/category = %v/%q, want %q", err, promptexec.CategoryOf(err), test.wantCategory)
}
@@ -265,13 +257,12 @@ func TestInspectComparisonExecutionStopsAtFirstProfileFailure(t *testing.T) {
prompt: validPromptInspection(resolved.Definition),
profiles: map[string]promptexec.ProfileInspection{
"weather-light": {ProfileID: "weather-light", BackendID: "local", ModelName: "light-model"},
"missing-key": {ProfileID: "missing-key", APIKeyEnv: "PROMPT_API_KEY"},
"missing-key": {ProfileID: "missing-key", CredentialRequired: true},
"weather-deep": {ProfileID: "weather-deep", BackendID: "cloud", ModelName: "deep-model"},
},
}
result, err := InspectComparisonExecution(context.Background(), ComparisonInspectionRequest{
Resolved: resolved, ProfileIDs: []string{"weather-light", "missing-key", "weather-deep"}, Executor: executor,
LookupEnv: func(string) (string, bool) { return "", false },
})
if err == nil || promptexec.CategoryOf(err) != promptexec.MissingCredential {
t.Fatalf("error/category = %v/%q, want missing credential", err, promptexec.CategoryOf(err))
@@ -359,7 +350,7 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
return promptexec.PromptInspection{
PromptID: definition.PromptID, PromptVersion: definition.PromptVersion, PromptHash: "prompt-hash", DefaultProfileID: "default-profile",
Inputs: []promptexec.InputDefinition{{Name: "data_package", Required: true, ContentType: "application/yaml"}},
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json", RepairAttempts: definition.GeneratedTextRepairAttempts},
}
}

View File

@@ -2,8 +2,10 @@ package app_test
import (
"context"
"fmt"
"os"
"path/filepath"
"strings"
"testing"
"time"
@@ -14,14 +16,12 @@ import (
)
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
lookupEnv := func(string) (string, bool) { return "test-key", true }
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
t.Helper()
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, id),
Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: profile},
LookupEnv: lookupEnv,
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
@@ -50,6 +50,55 @@ model: local-weather
inspect(t, override, report.Hourly, "", "weather-light", "openrouter", "local-weather")
}
func TestPromptInspectionAcceptsMaintainedEndpointOnlyProfile(t *testing.T) {
adapter, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: filepath.Join("..", "..", "examples", "weather-light-local-profile.yml")})
if err != nil {
t.Fatalf("New() error = %v", err)
}
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly), Executor: adapter,
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if result.ProfileID != "weather-light" || result.BackendID != "" || result.ModelName != "weather-local" {
t.Fatalf("inspection = %#v", result)
}
if strings.Contains(fmt.Sprintf("%#v", result), "127.0.0.1") {
t.Fatalf("inspection leaks endpoint: %#v", result)
}
}
func TestPromptInspectionSupportsRakestrawhomeProfileOffline(t *testing.T) {
adapter, err := promptkitadapter.New(promptkitadapter.Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
prompt, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly),
Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: "rakestrawhome-gemma-4-31b"},
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if prompt.ProfileID != "rakestrawhome-gemma-4-31b" || prompt.BackendID != "rakestrawhome" || prompt.ModelName == "" {
t.Fatalf("prompt inspection = %#v", prompt)
}
comparison, err := app.InspectComparisonExecution(context.Background(), app.ComparisonInspectionRequest{
Resolved: resolvedPromptProfile(t, report.Hourly),
ProfileIDs: []string{"rakestrawhome-gemma-4-31b", "weather-deep"},
Executor: adapter,
})
if err != nil {
t.Fatalf("InspectComparisonExecution() error = %v", err)
}
if len(comparison.Profiles) != 2 || comparison.Profiles[0].ProfileID != "rakestrawhome-gemma-4-31b" || comparison.Profiles[0].BackendID != "rakestrawhome" || comparison.Profiles[0].ModelName == "" {
t.Fatalf("comparison inspection = %#v", comparison)
}
}
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
t.Helper()
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)

View File

@@ -355,7 +355,7 @@ func comparisonResult(outputDirectory string, results []app.ComparisonProfileRes
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
result := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: "daily", ReportName: "Daily Report",
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: outputDirectory,
ManifestPath: filepath.Join(outputDirectory, "comparison.json"), DataPackagePath: filepath.Join(outputDirectory, "data-package.yml"),

View File

@@ -132,7 +132,7 @@ func actionConfigPath(t *testing.T) string {
func generatedReportResult() *app.ReportResult {
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
return &app.ReportResult{
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0",
ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0",
RunID: "daily-20260529", GeneratedAt: generatedAt, Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)},
ProfileID: "weather-light", BackendID: "local", ModelName: "weather-model", ValidationStatus: promptexec.ValidationPassed,

View File

@@ -43,6 +43,7 @@ type generateSummary struct {
ModelName string `json:"modelName,omitempty"`
SourceWarnings []weatherdata.SourceWarning `json:"sourceWarnings,omitempty"`
ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
Notification *generateNotificationSummary `json:"notification,omitempty"`
Error string `json:"error,omitempty"`
}
@@ -106,6 +107,7 @@ type comparisonProfileSummary struct {
ModelName string `json:"modelName"`
Status string `json:"status"`
ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
ReportPath string `json:"reportPath,omitempty"`
LLMDebugPath string `json:"llmDebugPath,omitempty"`
Error *comparison.SafeError `json:"error,omitempty"`
@@ -129,6 +131,10 @@ func newGenerateSummary(result *app.ReportResult, err error) generateSummary {
summary.ProfileID, summary.BackendID, summary.ModelName = result.ProfileID, result.BackendID, result.ModelName
summary.SourceWarnings = append([]weatherdata.SourceWarning(nil), result.SourceWarnings...)
summary.ValidationStatus = string(result.ValidationStatus)
if result.RepairAttempts != nil {
value := *result.RepairAttempts
summary.RepairAttempts = &value
}
summary.OutputPath = result.OutputPath
summary.LLMDebugPath = result.LLMDebugPath
summary.Notification = newGenerateNotificationSummary(result.Notification)
@@ -162,6 +168,11 @@ func newGenerateNotificationSummary(result *app.NotificationResult) *generateNot
return summary
}
func repairAttemptsCopy(value int) *int {
copy := value
return &copy
}
func newBatchSummary(result *app.BatchResult, err error) batchSummary {
summary := batchSummary{Command: commandRun}
if result == nil {
@@ -203,6 +214,9 @@ func newComparisonSummary(result *app.ComparisonResult, err error) comparisonSum
Status: profile.Status, ValidationStatus: string(profile.ValidationStatus), ReportPath: profile.ReportPath,
LLMDebugPath: profile.LLMDebugPath, Error: profile.Error,
})
if profile.RepairAttempts != nil {
summary.Results[len(summary.Results)-1].RepairAttempts = repairAttemptsCopy(*profile.RepairAttempts)
}
}
summary.Status = comparisonSummaryStatus(result, err)
if err != nil {

View File

@@ -19,7 +19,7 @@ import (
func TestGenerateSummaryUsesActiveResultFields(t *testing.T) {
generatedAt := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
summary := newGenerateSummary(&app.ReportResult{ReportID: report.Daily, ReportName: "Daily Report", PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", RunID: "run-123", GeneratedAt: generatedAt, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: generatedAt, End: generatedAt.Add(24 * time.Hour)}, ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "model", SourceWarnings: []weatherdata.SourceWarning{{Source: "alerts", Message: "source unavailable"}}, ValidationStatus: promptexec.ValidationPassed, OutputPath: "/reports/daily.md"}, nil)
if summary.OutputPath == "" || summary.ProfileID == "" || summary.ValidationStatus != string(promptexec.ValidationPassed) || len(summary.SourceWarnings) != 1 {
t.Fatalf("summary = %#v", summary)
}
@@ -39,7 +39,7 @@ func TestComparisonSummaryUsesLockedOrderAndSafeFields(t *testing.T) {
profileFailure := comparison.NewSafeError("generation", "execute prompt failed")
result := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: report.Daily, ReportName: "Daily Report",
PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started.Add(time.Minute), Timezone: "America/Chicago",
ValidPeriod: timeutil.Period{Start: started, End: started.Add(24 * time.Hour)}, OutputDirectory: "/reports/comparison-daily",
ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
@@ -113,7 +113,7 @@ func TestComparisonSummaryClassifiesCompleteAndAllFailedResults(t *testing.T) {
started := time.Date(2026, 5, 29, 13, 30, 0, 0, time.UTC)
failure := comparison.NewSafeError("generation", "execute prompt failed")
complete := &app.ComparisonResult{
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.0.0", PromptHash: strings.Repeat("a", 64),
ComparisonID: "comparison_run-123", ReportID: report.Daily, PromptID: "weather.daily_generated_text", PromptVersion: "2.1.0", PromptHash: strings.Repeat("a", 64),
StartedAt: started, FinishedAt: started, Timezone: "America/Chicago", ValidPeriod: timeutil.Period{Start: started, End: started.Add(time.Hour)},
OutputDirectory: "/reports/comparison-daily", ManifestPath: "/reports/comparison-daily/comparison.json", DataPackagePath: "/reports/comparison-daily/data-package.yml",
Total: 2, Succeeded: 2,

View File

@@ -15,7 +15,7 @@ import (
const (
// SchemaVersion identifies the supported comparison manifest schema.
SchemaVersion = "weatherreporter.comparison.v1"
SchemaVersion = "weatherreporter.comparison.v2"
// ManifestFilename is the canonical name of a comparison manifest.
ManifestFilename = "comparison.json"
@@ -71,6 +71,7 @@ type Result struct {
ModelName string `json:"modelName"`
Status string `json:"status"`
ValidationStatus string `json:"validationStatus,omitempty"`
RepairAttempts *int `json:"repairAttempts,omitempty"`
ReportPath string `json:"reportPath,omitempty"`
Error *SafeError `json:"error,omitempty"`
}
@@ -276,11 +277,14 @@ func (manifest Manifest) Validate() error {
if strings.TrimSpace(result.ModelName) == "" {
return fmt.Errorf("result %d has a blank model name", result.Position)
}
if result.RepairAttempts != nil && *result.RepairAttempts < 0 {
return fmt.Errorf("result %d has negative repair attempts", result.Position)
}
switch result.Status {
case StatusSucceeded:
succeeded++
if result.ValidationStatus != "passed" {
if result.ValidationStatus != "passed" || result.RepairAttempts == nil {
return fmt.Errorf("successful result %d did not pass validation", result.Position)
}
expectedPath, err := ReportFilename(result.Position, manifest.Total, result.ProfileID)
@@ -302,6 +306,9 @@ func (manifest Manifest) Validate() error {
if err := result.Error.validate(); err != nil {
return fmt.Errorf("failed result %d: %w", result.Position, err)
}
if result.ValidationStatus != "" && result.RepairAttempts == nil {
return fmt.Errorf("failed result %d has validation without repair provenance", result.Position)
}
default:
return fmt.Errorf("result %d has unsupported status %q", result.Position, result.Status)
}
@@ -324,7 +331,7 @@ func (safeError SafeError) validate() error {
}
func isValidationStatus(status string) bool {
return status == "" || status == "failed" || status == "skipped"
return status == "" || status == "passed" || status == "failed" || status == "skipped"
}
func isSHA256(value string) bool {

View File

@@ -118,7 +118,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
t.Fatalf("EncodeManifest() error = %v", err)
}
want := "{\n" +
" \"schemaVersion\": \"weatherreporter.comparison.v1\",\n" +
" \"schemaVersion\": \"weatherreporter.comparison.v2\",\n" +
" \"comparisonId\": \"comparison_daily-2026-08-24\",\n" +
" \"startedAt\": \"2026-08-24T12:00:00Z\",\n" +
" \"finishedAt\": \"2026-08-24T12:01:00Z\",\n" +
@@ -146,6 +146,7 @@ func TestManifestEncodingAndRoundTrip(t *testing.T) {
" \"modelName\": \"gpt-5-mini\",\n" +
" \"status\": \"succeeded\",\n" +
" \"validationStatus\": \"passed\",\n" +
" \"repairAttempts\": 0,\n" +
" \"reportPath\": \"01-weather-light.md\"\n" +
" },\n" +
" {\n" +
@@ -201,6 +202,9 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
{name: "successful result without passed validation", mutate: func(manifest *Manifest) { manifest.Results[0].ValidationStatus = "failed" }},
{name: "failed result with report", mutate: func(manifest *Manifest) { manifest.Results[1].ReportPath = "02-weather-deep.md" }},
{name: "failed result without error", mutate: func(manifest *Manifest) { manifest.Results[1].Error = nil }},
{name: "failed passed result without repair provenance", mutate: func(manifest *Manifest) {
manifest.Results[1].ValidationStatus = "passed"
}},
{name: "traversal report path", mutate: func(manifest *Manifest) { manifest.Results[0].ReportPath = "../report.md" }},
{name: "duplicate report path", mutate: func(manifest *Manifest) {
manifest.Results[1] = Result{Position: 2, ProfileID: "weather-deep", ModelName: "gpt-5", Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: manifest.Results[0].ReportPath}
@@ -222,6 +226,17 @@ func TestManifestValidateRejectsInvariants(t *testing.T) {
}
}
func TestManifestValidateAcceptsPostValidationFailure(t *testing.T) {
t.Parallel()
manifest := validManifest()
manifest.Results[1].ValidationStatus = "passed"
manifest.Results[1].RepairAttempts = intPtr(0)
if err := manifest.Validate(); err != nil {
t.Fatalf("Manifest.Validate() rejected post-validation failure: %v", err)
}
}
func TestLogicalBundleValidate(t *testing.T) {
t.Parallel()
@@ -309,6 +324,7 @@ func validManifest() Manifest {
ModelName: "gpt-5-mini",
Status: StatusSucceeded,
ValidationStatus: "passed",
RepairAttempts: intPtr(0),
ReportPath: "01-weather-light.md",
},
{
@@ -321,3 +337,8 @@ func validManifest() Manifest {
},
}
}
func intPtr(value int) *int {
copy := value
return &copy
}

View File

@@ -388,6 +388,7 @@ var resultFields = map[string]jsonValueValidator{
"modelName": nil,
"status": nil,
"validationStatus": nil,
"repairAttempts": validateRepairAttemptsJSON,
"reportPath": nil,
"error": validateSafeErrorJSON,
}
@@ -430,7 +431,7 @@ func validateResultsJSON(decoder *json.Decoder) error {
return fmt.Errorf("results must be an array")
}
for decoder.More() {
if err := validateJSONObject(decoder, resultFields); err != nil {
if err := validateOrderedJSONObject(decoder, resultFields, []string{"position", "profileId", "backendId", "modelName", "status", "validationStatus", "repairAttempts", "reportPath", "error"}); err != nil {
return err
}
}
@@ -444,6 +445,65 @@ func validateResultsJSON(decoder *json.Decoder) error {
return nil
}
func validateOrderedJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator, order []string) error {
token, err := decoder.Token()
if err != nil {
return err
}
if delimiter, ok := token.(json.Delim); !ok || delimiter != '{' {
return fmt.Errorf("manifest value must be an object")
}
seen := make(map[string]struct{}, len(fields))
last := -1
for decoder.More() {
token, err := decoder.Token()
if err != nil {
return err
}
name, ok := token.(string)
if !ok {
return fmt.Errorf("manifest field name is invalid")
}
validator, known := fields[name]
if !known {
return fmt.Errorf("manifest field %q is not canonical", name)
}
if _, duplicate := seen[name]; duplicate {
return fmt.Errorf("manifest field %q is duplicated", name)
}
position := -1
for index, candidate := range order {
if candidate == name {
position = index
break
}
}
if position <= last {
return fmt.Errorf("manifest field %q is out of canonical order", name)
}
seen[name] = struct{}{}
last = position
if validator == nil {
var value json.RawMessage
if err := decoder.Decode(&value); err != nil {
return err
}
continue
}
if err := validator(decoder); err != nil {
return err
}
}
token, err = decoder.Token()
if err != nil {
return err
}
if delimiter, ok := token.(json.Delim); !ok || delimiter != '}' {
return fmt.Errorf("manifest object has an invalid terminator")
}
return nil
}
func validateSafeErrorJSON(decoder *json.Decoder) error {
token, err := decoder.Token()
if err != nil {
@@ -458,6 +518,24 @@ func validateSafeErrorJSON(decoder *json.Decoder) error {
return validateJSONObjectBody(decoder, safeErrorFields)
}
func validateRepairAttemptsJSON(decoder *json.Decoder) error {
var raw json.RawMessage
if err := decoder.Decode(&raw); err != nil {
return err
}
if bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
return fmt.Errorf("repairAttempts must be an integer")
}
var value int
if err := json.Unmarshal(raw, &value); err != nil {
return err
}
if value < 0 {
return fmt.Errorf("repairAttempts must not be negative")
}
return nil
}
func validateJSONObject(decoder *json.Decoder, fields map[string]jsonValueValidator) error {
token, err := decoder.Token()
if err != nil {

View File

@@ -1,6 +1,7 @@
package comparison
import (
"bytes"
"context"
"encoding/json"
"errors"
@@ -146,6 +147,23 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
t.Helper()
appendManifestField(t, directory, `"SchemaVersion": "weatherreporter.comparison.v1"`)
}},
{name: "null repair attempts", mutate: func(t *testing.T, directory string) {
t.Helper()
path := filepath.Join(directory, ManifestFilename)
data, err := os.ReadFile(path)
if err != nil {
t.Fatal(err)
}
before := []byte(" \"status\": \"failed\",\n \"error\": {")
after := []byte(" \"status\": \"failed\",\n \"repairAttempts\": null,\n \"error\": {")
data = bytes.Replace(data, before, after, 1)
if bytes.Equal(data, readFile(t, path)) {
t.Fatal("failed to add null repairAttempts")
}
if err := os.WriteFile(path, data, 0o600); err != nil {
t.Fatal(err)
}
}},
{name: "traversal report path", mutate: func(t *testing.T, directory string) {
t.Helper()
path := filepath.Join(directory, ManifestFilename)
@@ -202,6 +220,30 @@ func TestRecognizeBundleRejectsUnsafeAndMalformedContents(t *testing.T) {
}
}
func TestValidateRepairAttemptsJSONRejectsNonIntegerValues(t *testing.T) {
t.Parallel()
for _, value := range []string{"null", "-1", "1.5", `"1"`, "9223372036854775808", "{"} {
value := value
t.Run(value, func(t *testing.T) {
t.Parallel()
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err == nil {
t.Fatalf("validateRepairAttemptsJSON(%q) error = nil", value)
}
})
}
}
func TestValidateRepairAttemptsJSONAcceptsNonnegativeIntegers(t *testing.T) {
t.Parallel()
for _, value := range []string{"0", "1"} {
if err := validateRepairAttemptsJSON(json.NewDecoder(strings.NewReader(value))); err != nil {
t.Fatalf("validateRepairAttemptsJSON(%q) error = %v", value, err)
}
}
}
func appendManifestField(t *testing.T, directory, field string) {
t.Helper()
path := filepath.Join(directory, ManifestFilename)
@@ -1077,7 +1119,7 @@ func testBundleWithReports(t *testing.T, count int) LogicalBundle {
}
manifest.Results[i] = Result{
Position: position, ProfileID: profileID, ModelName: "gpt-5-mini",
Status: StatusSucceeded, ValidationStatus: "passed", ReportPath: path,
Status: StatusSucceeded, ValidationStatus: "passed", RepairAttempts: intPtr(0), ReportPath: path,
}
reports[i] = BundleReport{Position: position, Path: path, Markdown: []byte("# Daily\n")}
}

View File

@@ -1,6 +1,2 @@
id: weather-balanced
backend: openrouter
model: "~google/gemini-flash-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex
base_profile: gemini-flash-latest

View File

@@ -1,6 +1,2 @@
id: weather-deep
backend: openrouter
model: "~anthropic/claude-sonnet-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex
base_profile: claude-sonnet-latest

View File

@@ -1,5 +1,2 @@
id: weather-light
backend: openrouter
model: deepseek/deepseek-v4-flash
timeout_seconds: 180
service_tier: flex
base_profile: deepseek-4-flash

View File

@@ -1,5 +1,5 @@
id: weather.daily_generated_text
version: "2.0.0"
version: "2.1.0"
default_profile: weather-balanced
description: Daily weather report analysis prompt.
inputs:
@@ -21,3 +21,4 @@ output:
format: json
validation_mode: json_schema
schema_path: daily.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.hourly_generated_text
version: "2.0.0"
version: "2.1.0"
default_profile: weather-light
description: Hourly weather report analysis prompt.
inputs:
@@ -21,3 +21,4 @@ output:
format: json
validation_mode: json_schema
schema_path: hourly.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.today_generated_text
version: "2.0.0"
version: "2.1.0"
default_profile: weather-balanced
description: Today's weather report analysis prompt.
inputs:
@@ -21,3 +21,4 @@ output:
format: json
validation_mode: json_schema
schema_path: today.generated_text.schema.json
repair_attempts: 1

View File

@@ -1,5 +1,5 @@
id: weather.tomorrow_generated_text
version: "2.0.0"
version: "2.1.0"
default_profile: weather-balanced
description: Tomorrow's weather report analysis prompt.
inputs:
@@ -21,3 +21,4 @@ output:
format: json
validation_mode: json_schema
schema_path: tomorrow.generated_text.schema.json
repair_attempts: 1

View File

@@ -77,8 +77,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("decode prompt definition: %v", err)
}
if definition.ID != tc.id || definition.Version != "2.0.0" || definition.DefaultProfile != tc.profile {
t.Fatalf("definition = %#v, want %s version 2.0.0 and profile %s", definition, tc.id, tc.profile)
if definition.ID != tc.id || definition.Version != "2.1.0" || definition.DefaultProfile != tc.profile {
t.Fatalf("definition = %#v, want %s version 2.1.0 and profile %s", definition, tc.id, tc.profile)
}
sharedInstruction := false
for _, message := range definition.Messages {
@@ -92,8 +92,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
}
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts != nil {
t.Fatalf("output = %#v, want JSON schema output without repair attempts", definition.Output)
if definition.Output.Format != "json" || definition.Output.ValidationMode != "json_schema" || definition.Output.SchemaPath != tc.schemaID+".generated_text.schema.json" || definition.Output.RepairAttempts == nil || *definition.Output.RepairAttempts != 1 {
t.Fatalf("output = %#v, want JSON schema output with one repair attempt", definition.Output)
}
if _, err := promptassets.Schema(tc.schemaID); err != nil {
t.Fatalf("Schema(%q) error = %v", tc.schemaID, err)
@@ -286,11 +286,11 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
} {
t.Run(want.id, func(t *testing.T) {
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.0.0")
inspection, err := engine.InspectPrompt(context.Background(), want.id, "2.1.0")
if err != nil {
t.Fatalf("InspectPrompt() error = %v", err)
}
if inspection.PromptID != want.id || inspection.PromptVersion != "2.0.0" || inspection.DefaultProfileID != want.profile {
if inspection.PromptID != want.id || inspection.PromptVersion != "2.1.0" || inspection.DefaultProfileID != want.profile || inspection.OutputContract.RepairAttempts != 1 {
t.Fatalf("inspection = %#v", inspection)
}
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
@@ -361,6 +361,29 @@ func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
}
}
func TestEmbeddedProfilesAreMinimalBaseAliases(t *testing.T) {
wantBases := map[string]string{
"weather-balanced.yml": "gemini-flash-latest",
"weather-deep.yml": "claude-sonnet-latest",
"weather-light.yml": "deepseek-4-flash",
}
for path, wantBase := range wantBases {
t.Run(path, func(t *testing.T) {
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
if err != nil {
t.Fatalf("read profile: %v", err)
}
var definition map[string]string
if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("unmarshal profile: %v", err)
}
if definition["id"] != strings.TrimSuffix(path, ".yml") || definition["base_profile"] != wantBase || len(definition) != 2 {
t.Fatalf("profile definition = %#v, want only its id and base profile %q", definition, wantBase)
}
})
}
}
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
@@ -391,7 +414,7 @@ func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
if err != nil {
return err
}
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "repair_attempts:", "weather.daily_report"} {
for _, unwanted := range []string{"local-heavy", "pipeline-weather/", "application/json", "weather.daily_report"} {
if strings.Contains(string(data), unwanted) {
t.Fatalf("%s contains retired runtime setting %q", path, unwanted)
}

View File

@@ -21,8 +21,9 @@ import (
var ErrSecureCaptureUnsupported = errors.New("secure prompt debug capture is unavailable on this platform")
const (
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v2"
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v2"
promptPreparationDebugSchemaVersion = "weatherreporter.prompt_preparation_debug.v3"
promptExecutionDebugSchemaVersion = "weatherreporter.prompt_execution_debug.v3"
promptFailureDebugSchemaVersion = "weatherreporter.prompt_failure_debug.v1"
debugDirectoryMode = 0o700
debugFileMode = 0o600
)
@@ -53,6 +54,7 @@ type PromptDebugOutput struct {
Format string `json:"format"`
ValidationMode string `json:"validationMode"`
SchemaPath string `json:"schemaPath"`
RepairAttempts int `json:"repairAttempts"`
}
// PromptDebugPreparation is the explicit, content-safe mapping of preparation
@@ -100,9 +102,27 @@ type PromptDebugValidation struct {
Status string `json:"status"`
Mode string `json:"mode"`
SchemaPath string `json:"schemaPath"`
RepairAttempts int `json:"repairAttempts"`
Diagnostics []string `json:"diagnostics,omitempty"`
}
// PromptFailureDebugArtifact records provider detail only in explicit debug storage.
type PromptFailureDebugArtifact struct {
SchemaVersion string `json:"schemaVersion"`
ReportID report.ID `json:"reportId"`
ValidDate string `json:"validDate"`
RunID string `json:"runId"`
Failure PromptFailure `json:"failure"`
}
type PromptFailure struct {
Category string `json:"category"`
StatusCode int `json:"statusCode,omitempty"`
ProviderCode string `json:"providerCode,omitempty"`
ProviderType string `json:"providerType,omitempty"`
ProviderMessage string `json:"providerMessage,omitempty"`
}
// PromptDebugExecution is the explicit mapping of execution provenance.
type PromptDebugExecution struct {
RunID string `json:"runId"`
@@ -235,6 +255,24 @@ func (w *PromptDebugWriter) WriteExecution(ref PromptDebugRef, execution prompte
return directory, nil
}
// WriteFailure stores the project-owned structured provider failure.
func (w *PromptDebugWriter) WriteFailure(ref PromptDebugRef, failure *promptexec.GenerationError) (string, error) {
if !w.Enabled() || failure == nil {
return "", nil
}
directory, secureDirectory, err := w.runDirectory(ref)
if err != nil {
return "", err
}
defer secureDirectory.Close()
artifact := PromptFailureDebugArtifact{SchemaVersion: promptFailureDebugSchemaVersion, ReportID: ref.ReportID, ValidDate: ref.ValidDate, RunID: ref.RunID,
Failure: PromptFailure{Category: string(failure.Category()), StatusCode: failure.StatusCode(), ProviderCode: failure.ProviderCode(), ProviderType: failure.ProviderType(), ProviderMessage: failure.ProviderMessage()}}
if err := secureDirectory.writeJSON("failure.json", artifact); err != nil {
return "", err
}
return directory, nil
}
func (w *PromptDebugWriter) runDirectory(ref PromptDebugRef) (string, *secureDirectory, error) {
if err := validatePromptDebugRef(ref); err != nil {
return "", nil, err
@@ -283,7 +321,7 @@ func promptDebugPreparation(value promptexec.Preparation) PromptDebugPreparation
PromptID: value.PromptID, PromptVersion: value.PromptVersion, PromptHash: value.PromptHash,
RenderedPromptHash: value.RenderedPromptHash, InputHashes: copyPromptDebugMap(value.InputHashes),
ProfileID: value.ProfileID, BackendID: value.BackendID, ModelName: value.ModelName,
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath},
Output: PromptDebugOutput{Format: value.Output.Format, ValidationMode: value.Output.ValidationMode, SchemaPath: value.Output.SchemaPath, RepairAttempts: value.Output.RepairAttempts},
StartedAt: value.StartedAt, EndedAt: value.EndedAt, Duration: value.Duration,
}
}
@@ -300,7 +338,7 @@ func promptDebugExecution(value promptexec.Execution) PromptDebugExecution {
}
func promptDebugValidation(value promptexec.Validation) PromptDebugValidation {
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, Diagnostics: append([]string(nil), value.Diagnostics...)}
return PromptDebugValidation{Status: string(value.Status), Mode: value.Mode, SchemaPath: value.SchemaPath, RepairAttempts: value.RepairAttempts, Diagnostics: append([]string(nil), value.Diagnostics...)}
}
func promptDebugMessages(values []promptexec.RenderedMessage) []PromptDebugMessage {

View File

@@ -44,13 +44,13 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
t.Fatalf("WriteExecution() directory = %q, want %q", executionDir, preparationDir)
}
preparationData := readPromptDebugFile(t, filepath.Join(preparationDir, "preparation.json"))
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v2", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`} {
for _, want := range []string{"weatherreporter.prompt_preparation_debug.v3", "Use the supplied weather facts.", `"type": "object"`, "https://llm.example.test", `"temperature": 0.2`, `"repairAttempts": 0`} {
if !strings.Contains(string(preparationData), want) {
t.Fatalf("preparation debug artifact missing %q:\n%s", want, preparationData)
}
}
executionData := readPromptDebugFile(t, filepath.Join(executionDir, "execution.json"))
for _, want := range []string{"weatherreporter.prompt_execution_debug.v2", "Generated forecast prose.", "validation details", `"status": "passed"`} {
for _, want := range []string{"weatherreporter.prompt_execution_debug.v3", "Generated forecast prose.", "validation details", `"status": "passed"`, `"repairAttempts": 0`} {
if !strings.Contains(string(executionData), want) {
t.Fatalf("execution debug artifact missing %q:\n%s", want, executionData)
}
@@ -66,6 +66,25 @@ func TestPromptDebugWriterWritesIsolatedArtifacts(t *testing.T) {
assertPromptDebugMode(t, filepath.Join(executionDir, "execution.json"), debugFileMode)
}
func TestPromptDebugWriterWritesProviderFailureOnlyToDebugStore(t *testing.T) {
writer, err := NewPromptDebugWriter(filepath.Join(t.TempDir(), "operator-debug"))
if err != nil {
t.Fatalf("NewPromptDebugWriter() error = %v", err)
}
marker := "provider-private-marker"
directory, err := writer.WriteFailure(promptDebugRef(), promptexec.NewGenerationError(429, "rate_limit", "provider_error", marker, nil))
if err != nil {
t.Fatalf("WriteFailure() error = %v", err)
}
data := readPromptDebugFile(t, filepath.Join(directory, "failure.json"))
for _, want := range []string{"weatherreporter.prompt_failure_debug.v1", `"category": "generation"`, `"statusCode": 429`, marker} {
if !strings.Contains(string(data), want) {
t.Fatalf("failure artifact missing %q: %s", want, data)
}
}
assertPromptDebugMode(t, filepath.Join(directory, "failure.json"), debugFileMode)
}
func TestPromptDebugWriterProjectsProviderConfigurationSafely(t *testing.T) {
const marker = "private-debug-marker"
tests := []struct {
@@ -349,7 +368,7 @@ func promptDebugExecutionFixture() promptexec.Execution {
InputHashes: map[string]string{"data_package": "input-hash"}, ProfileID: "local", BackendID: "local", ModelName: "weather-model",
GeneratedHash: "generated-hash", Usage: promptexec.TokenUsage{PromptTokens: 10, CompletionTokens: 5, TotalTokens: 15},
StartedAt: startedAt, EndedAt: startedAt.Add(time.Second), Duration: time.Second,
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", []string{"validation details"}),
Validation: promptexec.NewValidation(promptexec.ValidationPassed, "strict", "schemas/daily.json", 0, []string{"validation details"}),
RawOutput: []byte("Generated forecast prose."),
Debug: &promptexec.ExecutionDebug{ValidationDiagnostics: []string{"validation details"}},
}

View File

@@ -78,3 +78,15 @@ func boundText(value string, limit int) string {
}
return value
}
func boundCodePoints(value string, limit int) string {
if limit <= 0 || value == "" {
return ""
}
value = strings.ToValidUTF8(value, "<22>")
runes := []rune(value)
if len(runes) <= limit {
return value
}
return string(runes[:limit])
}

View File

@@ -51,6 +51,7 @@ type OutputContract struct {
Format string
ValidationMode string
SchemaPath string
RepairAttempts int
}
// ProfileInspection describes the safe, selected execution identity for one profile.
@@ -144,15 +145,17 @@ type Validation struct {
Status ValidationStatus
Mode string
SchemaPath string
RepairAttempts int
Diagnostics []string
}
// NewValidation returns a completed validation value with bounded diagnostics.
func NewValidation(status ValidationStatus, mode string, schemaPath string, diagnostics []string) Validation {
func NewValidation(status ValidationStatus, mode string, schemaPath string, repairAttempts int, diagnostics []string) Validation {
return Validation{
Status: status,
Mode: mode,
SchemaPath: schemaPath,
RepairAttempts: repairAttempts,
Diagnostics: boundDiagnostics(diagnostics),
}
}
@@ -235,6 +238,86 @@ func (e *Error) Category() ErrorCategory {
return e.category
}
// GenerationError retains provider failure details for programmatic handling
// without exposing them through routine formatting or serialization.
type GenerationError struct {
statusCode int
providerCode string
providerType string
providerMessage string
err *Error
}
// NewGenerationError returns a classified generation failure with bounded
// provider details. The dependency cause remains reachable through err only.
func NewGenerationError(statusCode int, providerCode string, providerType string, providerMessage string, cause error) *GenerationError {
return &GenerationError{
statusCode: statusCode,
providerCode: boundCodePoints(providerCode, 256),
providerType: boundCodePoints(providerType, 256),
providerMessage: boundCodePoints(providerMessage, 4096),
err: NewError(Generation, "provider generation failed", cause),
}
}
// StatusCode returns the provider HTTP status when one was available.
func (e *GenerationError) StatusCode() int {
if e == nil {
return 0
}
return e.statusCode
}
// ProviderCode returns the bounded provider error code.
func (e *GenerationError) ProviderCode() string {
if e == nil {
return ""
}
return e.providerCode
}
// ProviderType returns the bounded provider error type.
func (e *GenerationError) ProviderType() string {
if e == nil {
return ""
}
return e.providerType
}
// ProviderMessage returns the bounded provider error message.
func (e *GenerationError) ProviderMessage() string {
if e == nil {
return ""
}
return e.providerMessage
}
// Category returns Generation for every generation failure.
func (e *GenerationError) Category() ErrorCategory { return Generation }
// Error intentionally excludes provider details from ordinary error text.
func (e *GenerationError) Error() string {
if e == nil {
return ""
}
message := NewError(Generation, "provider generation failed", nil).Error()
if e.statusCode == 0 {
return message
}
return fmt.Sprintf("%s (HTTP %d)", message, e.statusCode)
}
// GoString keeps %#v formatting as safe as ordinary error formatting.
func (e *GenerationError) GoString() string { return e.Error() }
// Unwrap preserves the project-owned classified error and its hidden cause.
func (e *GenerationError) Unwrap() error {
if e == nil {
return nil
}
return e.err
}
// CapacityError adds the safe backend identity to a capacity failure.
type CapacityError struct {
BackendID string

View File

@@ -3,6 +3,7 @@ package promptexec
import (
"context"
"errors"
"fmt"
"strings"
"testing"
"unicode/utf8"
@@ -172,8 +173,8 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
if bounded[0] != "a<>b" {
t.Fatalf("invalid UTF-8 diagnostic = %q, want replacement", bounded[0])
}
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", values)
if len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
validation := NewValidation(ValidationFailed, "json_schema", "daily.schema.json", 2, values)
if validation.RepairAttempts != 2 || len(validation.Diagnostics) != maxValidationDiagnostics || validation.Diagnostics[0] != "a<>b" {
t.Fatalf("validation = %#v, want bounded diagnostics", validation)
}
@@ -187,8 +188,53 @@ func TestBoundDiagnosticAndErrorText(t *testing.T) {
}
}
func TestGenerationErrorKeepsProviderDetailsOutOfRoutineFormatting(t *testing.T) {
cause := errors.New("provider response must stay hidden")
error := NewGenerationError(
429,
string([]byte{'c', 0xff})+strings.Repeat("界", 300),
strings.Repeat("type", 100),
string([]byte{'m', 0xff})+strings.Repeat("界", 4_200),
cause,
)
if error.StatusCode() != 429 || error.Category() != Generation || CategoryOf(error) != Generation {
t.Fatalf("generation error identity = %#v", error)
}
if utf8.RuneCountInString(error.ProviderCode()) != 256 || utf8.RuneCountInString(error.ProviderType()) != 256 || utf8.RuneCountInString(error.ProviderMessage()) != 4096 {
t.Fatalf("provider detail bounds = %d/%d/%d", utf8.RuneCountInString(error.ProviderCode()), utf8.RuneCountInString(error.ProviderType()), utf8.RuneCountInString(error.ProviderMessage()))
}
if !utf8.ValidString(error.ProviderCode()) || !utf8.ValidString(error.ProviderType()) || !utf8.ValidString(error.ProviderMessage()) {
t.Fatalf("provider details are not valid UTF-8: %#v", error)
}
if !errors.Is(error, cause) {
t.Fatal("errors.Is() = false, want preserved dependency cause")
}
var owned *Error
if !errors.As(error, &owned) || owned.Category() != Generation {
t.Fatalf("errors.As(*Error) = %#v, want project-owned generation error", owned)
}
want := "generation: provider generation failed (HTTP 429)"
if error.Error() != want || fmt.Sprintf("%#v", error) != want {
t.Fatalf("ordinary formatting = %q / %#v, want %q", error.Error(), error, want)
}
for _, private := range []string{cause.Error(), error.ProviderCode(), error.ProviderType(), error.ProviderMessage()} {
if strings.Contains(error.Error(), private) || strings.Contains(fmt.Sprintf("%#v", error), private) {
t.Fatalf("generation error leaks provider detail %q", private)
}
}
statusOnly := NewGenerationError(503, "", "", "", nil)
if statusOnly.Error() != "generation: provider generation failed (HTTP 503)" {
t.Fatalf("status-only error = %q", statusOnly)
}
var nilError *GenerationError
if nilError.StatusCode() != 0 || nilError.ProviderCode() != "" || nilError.ProviderType() != "" || nilError.ProviderMessage() != "" || nilError.Category() != Generation || nilError.Error() != "" || nilError.GoString() != "" || nilError.Unwrap() != nil {
t.Fatalf("nil generation error = %#v", nilError)
}
}
func TestContractCopiesMutableValues(t *testing.T) {
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}}
preparation := Preparation{InputHashes: map[string]string{"data_package": "input-hash"}, Output: OutputContract{RepairAttempts: 3}}
preparationDebug := &PreparationDebug{
RenderedMessages: []RenderedMessage{{Role: "user", Content: "rendered input"}},
StructuredSchema: []byte("schema body"),
@@ -196,7 +242,7 @@ func TestContractCopiesMutableValues(t *testing.T) {
}
execution := Execution{
InputHashes: map[string]string{"data_package": "input-hash"},
Validation: Validation{Diagnostics: []string{"validation detail"}},
Validation: Validation{RepairAttempts: 2, Diagnostics: []string{"validation detail"}},
RawOutput: []byte("generated output"),
Debug: &ExecutionDebug{
RawOutput: []byte("provider output"),
@@ -217,13 +263,13 @@ func TestContractCopiesMutableValues(t *testing.T) {
execution.Debug.RawOutput[0] = 'x'
execution.Debug.ValidationDiagnostics[0] = "changed"
if preparationCopy.InputHashes["data_package"] != "input-hash" {
if preparationCopy.InputHashes["data_package"] != "input-hash" || preparationCopy.Output.RepairAttempts != 3 {
t.Fatalf("preparation copy = %#v", preparationCopy)
}
if debugCopy.RenderedMessages[0].Content != "rendered input" || string(debugCopy.StructuredSchema) != "schema body" || string(debugCopy.ParametersJSON) != `{"temperature":0.2}` {
t.Fatalf("preparation debug copy = %#v", debugCopy)
}
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
if executionCopy.InputHashes["data_package"] != "input-hash" || executionCopy.Validation.RepairAttempts != 2 || executionCopy.Validation.Diagnostics[0] != "validation detail" || string(executionCopy.RawOutput) != "generated output" || string(executionCopy.Debug.RawOutput) != "provider output" || executionCopy.Debug.ValidationDiagnostics[0] != "detailed validation" {
t.Fatalf("execution copy = %#v", executionCopy)
}
}

View File

@@ -12,9 +12,10 @@ func dailyDefinition() Definition {
ID: Daily,
Name: "Daily Report",
PromptID: "weather.daily_generated_text",
PromptVersion: "2.0.0",
PromptVersion: "2.1.0",
TemplateID: "daily",
GeneratedTextSchemaID: "daily",
GeneratedTextRepairAttempts: 1,
ArtifactGroup: "daily",
OutputName: "daily.md",
DistributorPathTemplates: []string{

View File

@@ -33,6 +33,7 @@ type Definition struct {
PromptVersion string
TemplateID string
GeneratedTextSchemaID string
GeneratedTextRepairAttempts int
ArtifactGroup string
OutputName string
DistributorPathTemplates []string

View File

@@ -14,9 +14,10 @@ func hourlyDefinition() Definition {
ID: Hourly,
Name: "Hourly Report",
PromptID: "weather.hourly_generated_text",
PromptVersion: "2.0.0",
PromptVersion: "2.1.0",
TemplateID: "hourly",
GeneratedTextSchemaID: "hourly",
GeneratedTextRepairAttempts: 1,
ArtifactGroup: "hourly",
OutputName: "hourly.md",
DistributorPathTemplates: []string{

View File

@@ -82,8 +82,8 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
}
for _, definition := range definitions {
if definition.PromptVersion != "2.0.0" {
t.Fatalf("%s PromptVersion = %q, want 2.0.0", definition.ID, definition.PromptVersion)
if definition.PromptVersion != "2.1.0" {
t.Fatalf("%s PromptVersion = %q, want 2.1.0", definition.ID, definition.PromptVersion)
}
if definition.PromptID == "" {
t.Fatalf("%s PromptID is empty", definition.ID)
@@ -91,6 +91,9 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
}
if definition.GeneratedTextRepairAttempts != 1 {
t.Fatalf("%s repair attempts = %d, want 1", definition.ID, definition.GeneratedTextRepairAttempts)
}
}
if _, err := registry.Lookup(ID("three_day")); err == nil {

View File

@@ -10,9 +10,10 @@ func todayDefinition() Definition {
ID: Today,
Name: "Today Report",
PromptID: "weather.today_generated_text",
PromptVersion: "2.0.0",
PromptVersion: "2.1.0",
TemplateID: "today",
GeneratedTextSchemaID: "today",
GeneratedTextRepairAttempts: 1,
ArtifactGroup: "today",
OutputName: "today.md",
DistributorPathTemplates: []string{

View File

@@ -10,9 +10,10 @@ func tomorrowDefinition() Definition {
ID: Tomorrow,
Name: "Tomorrow Report",
PromptID: "weather.tomorrow_generated_text",
PromptVersion: "2.0.0",
PromptVersion: "2.1.0",
TemplateID: "tomorrow",
GeneratedTextSchemaID: "tomorrow",
GeneratedTextRepairAttempts: 1,
ArtifactGroup: "tomorrow",
OutputName: "tomorrow.md",
DistributorPathTemplates: []string{