8 Commits

35 changed files with 1613 additions and 602 deletions

View File

@@ -22,8 +22,12 @@ extra report copies for a command and do not change configuration.
collection and generation configuration.
- [config.yml](../examples/config.yml) is a representative production-oriented
configuration using synthetic endpoints and no credentials.
- [weather-light-local-profile.yml](../examples/weather-light-local-profile.yml)
is a complete endpoint-only override for the embedded `weather-light`
profile.
Both files are loaded by the configuration test suite.
The configuration examples are loaded by the configuration test suite. The
profile example is inspected through the Promptkit adapter test suite.
## Minimal Configuration
@@ -146,13 +150,35 @@ individual `generate` or `run` command when explicitly needed.
| Field | Default | Rules |
| --- | --- | --- |
| `profile` | empty | Optional explicit execution profile. Otherwise the prompt's declared default is used. |
| `profile_file` | empty | Optional external profile file. Cannot be combined with `profile_dir`. |
| `profile_dir` | empty | Optional external profile directory. Cannot be combined with `profile_file`. |
| `profile` | empty | Optional global profile selection for every report in one command. When empty, each exact prompt version selects its declared default. |
| `profile_file` | empty | Optional external Promptkit profile file. It cannot be combined with `profile_dir`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
| `profile_dir` | empty | Optional external Promptkit profile directory. It cannot be combined with `profile_file`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
| `timeout` | `2m` | Must be greater than zero. |
| `local.endpoint` | empty | Optional absolute URL for the conventional local backend. A blank endpoint leaves it unregistered. |
| `local.concurrency_limit` | `1` | Maximum local backend concurrency. `0` is unlimited; negative values are invalid. |
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
They are separate decisions. An explicit `profile` applies to every selected
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
Tomorrow select `weather-balanced` through their exact `1.1.0` prompt
definitions.
Promptkit resolves a selected profile definition from a test or embedding
consumer's explicit in-memory profile, then the configured `profile_file` or
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
built-in catalog. Sources provide complete definitions; fields are never
merged. A matching malformed external profile fails rather than using the
embedded definition. The [Promptkit integration guide](integrations/promptkit.md)
owns the catalog and precedence details.
To replace the default Hourly definition with a local OpenAI-compatible
endpoint, set `profile_file` to a copy of
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml).
The example has no credential and should be edited for the local endpoint and
model before use. An alternative profile may use `backend: local`; in that
case `promptkit.local.endpoint` supplies the conventional local backend
endpoint.
### `workspace`
| Field | Default |

View File

@@ -1,16 +1,49 @@
# Promptkit Integration
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are
`weather.daily_generated_text`, `weather.today_generated_text`,
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version
`1.0.1`. Their prompt assets and generated-text JSON Schemas are embedded by
`internal/promptassets`.
Weatherreporter uses Promptkit for all generated-text reports. The four logical
prompts are `weather.daily_generated_text`, `weather.today_generated_text`,
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at
version `1.1.0`. Their prompt assets, generated-text JSON Schemas, and
Weatherreporter profile catalog are embedded by `internal/promptassets`.
Before collection, Weatherreporter inspects the exact prompt version, requires one required
`data_package` input with content type `application/yaml`, and requires the report's JSON
Schema output contract. It selects `promptkit.profile` when configured, otherwise the
prompt's declared default profile. Profiles that require a direct API key are unsupported; a
profile that reports `APIKeyEnv` requires a nonblank value in that environment variable.
## Logical profile catalog
Prompt definitions select a stable Weatherreporter profile ID. The embedded
definitions currently use Promptkit's `openrouter` backend:
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
| --- | --- | --- | --- | --- | --- |
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` | Hourly |
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
profiles intentionally omit endpoints, credentials, temperature, `top_p`, and
output-token limits.
## Selection, lookup, and active execution
Before collection, Weatherreporter inspects the exact prompt version and output
contract. A nonblank `promptkit.profile` selects one profile ID for every
report in the command; otherwise the prompt's declared default selects it.
Promptkit resolves the selected definition in this order:
1. explicit in-memory profiles used by an embedding consumer or test;
2. the configured `profile_file` or `profile_dir`;
3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's built-in catalog.
A source falls through only when the selected ID is absent. Each source
supplies a complete definition, so profile fields are not merged. A malformed
matching operator definition is an error and does not fall back.
Profiles that require a direct API key are unsupported; a profile that reports
`APIKeyEnv` requires a nonblank value in that environment variable. Inspection,
preparation, and execution retain the selected logical profile ID and resolved
backend and model through Weatherreporter's project-owned contract. Ordinary
errors, summaries, logs, and workspace state exclude endpoints, credentials,
rendered messages, schemas, request bodies, response bodies, and complete
parameter maps.
Execution receives the already-persisted YAML package, prepares it once, and returns structured
JSON that Weatherreporter validates before rendering its own Markdown template. Preparation and
@@ -21,6 +54,7 @@ The generated-text schemas require `summary`, `forecast_discussion`, and
`precipitation_timing`, and reject additional properties. Prompts return an empty string for
`precipitation_timing` when the deterministic package contains no precipitation windows.
Prompt/profile configuration is owned by the [configuration reference](../config.md). Adapter
construction and mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
Prompt/profile configuration and the maintained local override example are
owned by the [configuration reference](../config.md). Adapter construction and
mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
Durable metadata compatibility is described in [state internals](../internal/state.md).

View File

@@ -7,7 +7,10 @@ notification ordering after the CLI has parsed arguments and loaded configuratio
`GenerateDetailed` resolves one of the four report definitions, initializes an
optional debug root, and inspects the exact Promptkit prompt/profile before it
collects weather or writes managed state. It then builds facts and modules,
collects weather or writes managed state. A configured global profile selects
every report in the action; otherwise the exact prompt selects its default
logical profile. Inspection keeps only the selected profile ID and effective
backend/model needed by the project-owned execution contract. It then builds facts and modules,
saves the YAML data package, persists preparation metadata from the executor
callback, executes the prepared prompt, saves execution provenance and raw
output, validates generated text, renders Markdown, and optionally copies or
@@ -25,8 +28,8 @@ persists raw output and execution provenance but does not render a report.
`RunBatchDetailed` constructs a single debug writer and uses the request's
single executor. Before collection it inspects Today, Tomorrow, and Daily for
morning, or Tomorrow and Daily for evening, deduplicating effective profile
inspection. It then collects once, plans eligible Daily dates, and calls the
morning, or Tomorrow and Daily for evening, deduplicating inspection of a
shared selected profile. It then collects once, plans eligible Daily dates, and calls the
same prompt-generation core sequentially for each planned report. Per-report
notification is suppressed; a failed report does not stop later reports.

View File

@@ -4,10 +4,18 @@
The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor
per action. Promptkit dependency types do not escape the adapter.
The adapter exposes exact prompt and profile inspection plus prepared execution. It maps Promptkit
inspection values to project-owned prompt input, output-contract, profile, preparation, execution,
validation, and optional debug values. It classifies adapter failures without copying provider secrets
or unbounded response bodies into application errors or normal state.
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback
profile filesystems to each engine. Promptkit remains responsible for resolving
the configured operator profile source, application fallback catalog, and its
built-in catalog; the adapter does not parse profile YAML, merge sources, or
probe endpoints.
The adapter exposes exact prompt and profile inspection plus prepared execution.
It maps Promptkit inspection values to project-owned prompt input,
output-contract, logical profile identity, effective backend/model,
preparation, execution, validation, and optional debug values. It classifies
adapter failures without copying provider secrets or unbounded response bodies
into application errors or normal state.
The app calls the executor's preparation callback before provider execution to persist safe preparation
provenance. Completed executions are then persisted as safe execution provenance and raw generated text

View File

@@ -16,16 +16,19 @@ templates, generation eligibility, compatible prior IDs, default modules, and
batch eligibility flags. `Resolved` combines that definition with the valid
period and run metadata for one invocation.
| Report ID | Prompt version | Period policy | Comparison | Registry batch flag | Output copy |
| --- | --- | --- | --- | --- | --- |
| `daily` | `1.0.1` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
| `today` | `1.0.1` | Selected or current local civil day | Same valid date | Morning | `today.md` |
| `tomorrow` | `1.0.1` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
| `hourly` | `1.0.1` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
| Report ID | Prompt version | Default profile | Period policy | Comparison | Registry batch flag | Output copy |
| --- | --- | --- | --- | --- | --- | --- |
| `daily` | `1.1.0` | `weather-balanced` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
| `today` | `1.1.0` | `weather-balanced` | Selected or current local civil day | Same valid date | Morning | `today.md` |
| `tomorrow` | `1.1.0` | `weather-balanced` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
| `hourly` | `1.1.0` | `weather-light` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
Each report pairs its ID and prompt version with matching template and schema
IDs. Exact template fields and schema assets belong to [report templates](../templates.md)
and [generated-text internals](generatedtext.md).
and [generated-text internals](generatedtext.md). Prompt assets own default
profile selection; the registry deliberately stores no provider setting. The
[Promptkit integration guide](../integrations/promptkit.md) owns profile
definitions and resolution.
All valid periods are half-open.

View File

@@ -34,8 +34,10 @@ runs never write V1 records.
Prompt preparation and execution records are validated on both save and load.
They require exact report/prompt identity, complete timing, internally
consistent provenance, and status-appropriate validation or bounded classified
errors. Completed execution provenance keeps Promptkit's run identity distinct
from the Weatherreporter run identity.
errors. Completed preparation and execution provenance retain the selected
logical profile ID and resolved backend/model, but never profile endpoints or
credentials. Completed execution provenance keeps Promptkit's run identity
distinct from the Weatherreporter run identity.
For a completed prompt run, the execution record is atomically replaced after
each downstream artifact is saved. Its path set therefore records the raw and

View File

@@ -19,12 +19,38 @@ persists the module snapshot and prompt data package, records Promptkit
preparation provenance before provider execution, then persists raw output and
execution provenance, validates the structured generated text, and renders the
managed Markdown report from the validated text and deterministic values.
The current receipts are transitional workspace state, not a cross-version
profile-provenance contract.
The managed report and its final metadata are saved before single-report
Distributor notification is attempted. `--out` writes an extra operator copy;
it never changes the managed report or upload source. A successful generate
command prints its summary to stdout unless `--quiet` is used.
## Local Prompt Profile Override
Hourly normally selects the embedded `weather-light` profile. To use a local
OpenAI-compatible model without changing prompts or application code, copy
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml),
set its `endpoint` and `model` for the local server, and configure the copy as
`promptkit.profile_file`. The profile file's `weather-light` definition
completely replaces the embedded definition; it does not affect a report that
selects another profile ID.
For example, install the profile file at a known absolute path and set:
```yaml
promptkit:
profile_file: /etc/weatherreporter/weather-light-local-profile.yml
```
Prompt inspection occurs before weather collection. A malformed profile file,
missing required credential, or unsupported selected backend stops the command
before collection. A reachable profile can still fail later if its local model
endpoint is unavailable; Weatherreporter does not switch to a remote profile.
See the [configuration reference](config.md) for field definitions and the
[troubleshooting guide](troubleshooting.md) for recovery.
## Optional Prompt Debug Capture
Use `--llm-debug-dir` only when content-rich prompt diagnostics are required:
@@ -87,8 +113,11 @@ The generated-text and render-context artifacts are written for every completed
single-report generation.
A report's metadata links the module snapshot, data package, preparation and
execution receipts, managed report, generated-text artifacts, and any available single-report
notification artifact. Batch notification artifacts are separate batch-level
records under `notifications/batches`.
notification artifact. These current-version receipts remain transitional; use
the active command's classified error and explicit secure debug capture for
prompt diagnosis rather than relying on them as a durable interface. Batch
notification artifacts are separate batch-level records under
`notifications/batches`.
RunIDs begin with the UTC generation timestamp and report ID. A Daily RunID
also contains its local valid date so multiple Daily reports in one batch have
@@ -162,10 +191,12 @@ remain available where they can be safely persisted.
- A batch notification failure preserves each report's artifacts and adds the
top-level batch notification artifact.
Use the RunID from the action summary with the inspection commands above. For
a batch failure, inspect the summary first, then inspect the affected report
RunIDs or the batch notification path. Do not remove the whole workspace as a
first response; retain it until the failure is understood.
Use the action summary and its classified error first. For prompt or provider
diagnosis, prefer an explicitly enabled secure debug capture; current-version
receipt paths may provide supplemental context when available. For a batch
failure, inspect the summary first, then inspect the affected report RunIDs or
the batch notification path. Do not remove the whole workspace as a first
response; retain it until the failure is understood.
## Operational Caveats
@@ -174,5 +205,7 @@ first response; retain it until the failure is understood.
not publish them unintentionally.
- Weatherreporter uses one configured Weather API endpoint and local workspace
state.
- Promptkit profile resolution does not discover local endpoints or fail over
between local and remote profiles.
- It does not provide automatic resume, cleanup, archival, remote state, daemon
operation, or automatic storm monitoring.

View File

@@ -1,6 +1,6 @@
# Domain-Specific Prompt Profiles Roadmap
Status: Accepted feature direction; implementation has not started.
Status: Implemented.
## Purpose
@@ -10,9 +10,9 @@ These logical profiles should give each report an appropriate default while
allowing operators to replace any definition through the existing configured
profile source.
This roadmap defines the scope, policy, and intended end state. A later staged
implementation plan will translate it into implementation work. This document
does not prescribe implementation stages.
This roadmap records the scope, policy, and implemented end state. The
companion [implementation plan](implementation.md) records the ordered work
and verification used to reach it.
## User Intent
@@ -29,35 +29,34 @@ The feature is intended to provide three related benefits:
guarantee. A locally hosted lightweight model may still generate slowly on the
available hardware.
## Current State
## Pre-Implementation Baseline
Daily, Today, Tomorrow, and Hourly each declare Promptkit's
`gemini-flash-latest` profile as their prompt default. The optional
`promptkit.profile` setting overrides that default for every selected report in
Before implementation, Daily, Today, Tomorrow, and Hourly each declared
Promptkit's `gemini-flash-latest` profile as their prompt default. The optional
`promptkit.profile` setting overrode that default for every selected report in
an invocation.
Weatherreporter accepts either `promptkit.profile_file` or
`promptkit.profile_dir` and passes that source to Promptkit. A matching external
profile can override a Promptkit built-in profile, and the configured local
backend can support profiles that select `backend: local`. Endpoint-only
OpenAI-compatible profiles can also provide their own endpoint.
Weatherreporter accepted either `promptkit.profile_file` or
`promptkit.profile_dir` and passed that source to Promptkit. A matching external
profile could override a Promptkit built-in profile, and the configured local
backend could support profiles that select `backend: local`. Endpoint-only
OpenAI-compatible profiles could also provide their own endpoint.
Weatherreporter does not currently own or embed execution profiles. Promptkit's
current public source precedence also has no layer where application-owned
fallback profiles can remain overridable by the ordinary configured source.
Weatherreporter did not own or embed execution profiles. Promptkit v0.5.0
provided the fallback-profile layer used to add them without changing the
existing operator-source precedence.
## Prerequisite
Promptkit must first provide the application fallback profile capability
defined in the companion
[upstream feature request](promptkit-fallback-profiles-feature-request.md).
Promptkit v0.5.0 provides the application fallback profile capability defined
in the companion
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
Weatherreporter now depends on that tagged release. The dependency upgrade has
passed the repository test suite and an operator smoke test. Weatherreporter
must continue to use only Promptkit's public API rather than depending on its
internal packages or reproducing its profile repository behavior.
Weatherreporter should adopt a tagged Promptkit release containing that public
capability. It should not depend on Promptkit internal packages, reproduce
Promptkit's profile repository behavior, or commit a local module replacement
as the production dependency.
## Desired End State
## Implemented End State
Weatherreporter embeds usable definitions for these exact logical profile IDs:
@@ -148,20 +147,21 @@ selected model and supported by its backend. Avoid incidental generation
parameters that reduce portability or trigger provider-specific request
failures without a demonstrated quality benefit.
The initial model candidates are:
The initial profile definitions are:
- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B
instruction model;
- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and
- `weather-deep`: Promptkit's maintained Gemini Pro family target.
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
| --- | --- | --- | --- | --- |
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
The exact identifiers and generation settings must be finalized against the
available tagged Promptkit catalog and a representative Weatherreporter
evaluation corpus before the staged implementation plan is approved. The
logical profile IDs and report assignments do not depend on those concrete
choices.
These settings deliberately match the corresponding Promptkit v0.5.0
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
part of each OpenRouter rolling-alias identifier. The profiles do not set
temperature, `top_p`, or output-token limits; omission preserves provider
defaults and avoids unsupported incidental parameters.
## Prompt And Provenance Contract
## Prompt And Active Execution Contract
Changing a prompt's `default_profile` is a material prompt-definition change.
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
@@ -172,10 +172,18 @@ Prompt inspection must continue to occur before weather collection. It should
report the selected logical profile ID and the resolved backend and model
without exposing endpoints or credentials.
Preparation, execution, and failure artifacts should retain both the logical
profile identity and effective model provenance already available through the
project-owned execution contract. This feature should not replace a logical
profile ID with only the concrete model name in durable records.
The active execution contract should retain both the selected logical profile
identity and the resolved backend and model through inspection, preparation,
execution, errors, and command results where those values are already exposed.
This feature must not add a new durable-provenance or cross-version artifact
contract.
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
prompt provenance a non-goal. Existing workspace persistence may remain while
this feature lands, but it is transitional behavior and must not be expanded or
treated as part of the profile feature's desired end state. Prompt preparation
and execution artifacts written at `1.0.1` are not required to remain readable
after the prompt definitions advance to `1.1.0`.
## Evaluation Policy
@@ -196,7 +204,7 @@ The purpose is to choose an appropriate default for each tier, not to add a
permanent benchmark framework or live-provider requirement to the ordinary
test suite. Repository tests remain offline and deterministic.
## Scope
## Implemented Scope
The completed feature includes:
@@ -211,7 +219,7 @@ The completed feature includes:
- local-backend and endpoint-only override coverage;
- fail-fast inspection of missing, malformed, or unusable selected profiles;
- offline tests for selection, source precedence, effective model inspection,
batch reuse, and durable provenance;
batch reuse, and active execution behavior;
- maintained operator examples for overriding `weather-light` locally; and
- updates to the canonical configuration, Promptkit integration, report
registry, operations, troubleshooting, internal adapter, and release
@@ -243,13 +251,18 @@ observe new per-report defaults. This is a user-visible model-selection and
cost change and must be called out in release notes. Operators who require the
old all-report model can preserve it by setting an explicit global profile.
The prompt-version transition does not provide backward compatibility for
historical prompt preparation or execution artifacts. This is consistent with
the accepted ephemeral-state direction; the profile feature does not otherwise
redesign or remove the current workspace layout.
An external same-ID override is an operator-owned compatibility commitment.
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
silently merge an operator file.
## Completion Criteria
## Completion Record
The roadmap is complete when:
The following conditions are satisfied:
- a tagged Promptkit dependency supports the required fallback layer;
- every operational prompt selects its assigned logical profile at exact
@@ -262,33 +275,9 @@ The roadmap is complete when:
- `weather-light` can resolve through an endpoint-only or configured-local
override without requiring code or prompt changes;
- global `promptkit.profile` still overrides every report in an invocation;
- prompt preparation and execution artifacts record the selected logical
profile and effective model;
- active inspection and execution preserve the selected logical profile and
effective model through the project-owned execution contract;
- morning and evening batch preflight deduplicates inspection of shared
effective profile IDs as it does today;
- the default test suite remains offline and deterministic; and
- implemented behavior is documented by its canonical current-state owners.
## Open Questions
### Exact initial model identifiers and settings
Recommendation: evaluate the current Gemma 4 31B instruction target for
`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and
the maintained Gemini Pro target for `weather-deep`. Pin the exact profile
contents in this roadmap before deriving the staged implementation plan.
Alternative: duplicate three current Promptkit built-in definitions without a
Weatherreporter-specific evaluation. This is faster, but it provides less
evidence that their cost, schema reliability, and weather reasoning match the
tier policy.
### Future default use of `weather-deep`
Recommendation: ship the profile as an explicit override with no default
report assignment. Assign it later only when a report or evaluation shows a
material benefit over `weather-balanced`.
Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may
improve difficult synthesis, but it increases routine cost before the benefit
has been measured.

View File

@@ -0,0 +1,382 @@
# Ephemeral Operational State Roadmap
Status: Accepted feature direction; implementation has not started.
## Purpose
Weatherreporter should treat generated weather reports and their intermediate
artifacts as short-lived operational material rather than a durable audit
history. Forecasts and current conditions change continuously, and the normal
response to an old or failed report is to generate a new report, not to
reconstruct the provenance of the old one.
The application should retain only the bounded state needed to publish the
current report, calculate Recent Changes against the last successfully
published report for the same valid period, and complete the current
invocation safely. Detailed LLM diagnostics should remain an explicit,
operator-controlled exception outside ordinary workspace state.
This roadmap defines the intended state lifecycle, compatibility policy, and
architectural boundaries. A separate implementation plan will define the
ordered work after the roadmap is complete.
## User Intent
The state model should reflect these product expectations:
- weather reports are ephemeral products, not business records;
- old report provenance has no continuing operational value once conditions
and forecasts have changed;
- regenerating is preferable to recovering, replaying, or inspecting an old
generation;
- routine operation should not accumulate unbounded run-addressed artifacts;
- Recent Changes remains useful, but needs only one prior successful snapshot
for the same report and valid period; and
- sensitive prompt and response capture remains opt-in and explicitly managed
by the operator.
## Current State
Each generation currently writes a run-addressed collection containing a
module snapshot, data package, prompt preparation receipt, prompt execution
receipt, raw generated text, validated generated text, render context, managed
report, metadata, and optional notification receipt. Successful and failed
runs accumulate beneath the workspace.
Metadata links the collection and supports lookup by RunID. The CLI can list
historical runs and inspect their metadata, modules, data packages, prior
snapshots, and source provenance. New metadata uses the V2 format while the
reader retains V1 compatibility. Prompt artifacts are validated against
current report and prompt definitions when saved and loaded.
Most of this persistence exists for retrospective inspection and failure
recovery. Dedicated prompt preparation and execution load operations have no
ordinary production consumer. The important exception is module snapshot
state: generation actively loads the most recent compatible snapshot to build
the deterministic Recent Changes input for Daily, Today, and Tomorrow.
## Desired End State
Weatherreporter has three distinct state classes:
| State class | Lifecycle | Purpose |
| --- | --- | --- |
| Invocation workspace | Temporary and unpublished | Hold intermediate values while one report or batch is running. |
| Current published state | Bounded and replaceable | Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation. |
| Secure LLM debug capture | Explicitly enabled and operator-managed | Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture. |
Ordinary generation uses an invocation-scoped temporary directory on the same
filesystem as the managed workspace when atomic publication requires it.
Prompt preparation, prompt execution, raw generated text, validated generated
text, render contexts, data packages, and notification receipts may exist
there while needed, but they are not published as durable historical
artifacts.
A successful report atomically replaces the current published state for its
logical report key and valid period. A failed attempt leaves the last
successfully published report and comparison snapshot unchanged. Ordinary
temporary artifacts are removed after both success and handled failure;
cleanup failure is reported safely but must not replace the primary generation
error.
RunIDs remain useful as in-process correlation identifiers in action results,
logs, provider provenance, and optional debug paths. They no longer identify a
durable collection that Weatherreporter promises to locate or decode later.
## Published Report Policy
The managed Markdown report remains the authoritative upload source during an
invocation. The intended default is to retain only the current managed report
for each logical report key and valid period, replacing it atomically after a
new report has been fully rendered and validated.
An explicit `--out` or `--out-dir` copy remains operator-owned output outside
the managed-state lifecycle. Weatherreporter does not delete, rotate, or
rewrite those copies except when the same explicit destination is selected by
a later invocation.
Distributor continues to receive only a completed managed Markdown report.
Notification success or failure does not create a durable notification
history. A notification failure leaves the newly published report available
and returns a safe error through the current action result.
## Recent Changes State
Recent Changes must be preserved without preserving general report history.
For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible
module snapshot for each logical report key and valid period.
The retained snapshot represents the last successfully published report. A
new invocation reads it before constructing Recent Changes and replaces it
only when the new managed report has been successfully validated, rendered,
and published. A failed generation therefore does not become the baseline for
the next report and cannot hide changes that the user has not yet seen.
Hourly does not currently use the comparison strategy and should not retain a
comparison snapshot solely for symmetry. State whose valid period has ended
and can no longer participate in a supported comparison is eligible for safe
cleanup.
## Temporary Workspace And Failure Semantics
Temporary state must remain beneath a narrowly owned application directory and
use safe path construction, restrictive permissions where content is
sensitive, and atomic writes where practical. Publication must not expose a
partially rendered report or a snapshot that does not correspond to the
published report.
Normal results retain bounded error information and paths only for artifacts
that remain meaningful after the command: a previously or newly published
report, an explicit operator output, or an enabled secure debug capture.
Temporary intermediate paths are not emitted as if they were durable recovery
locations. A failed command is retried by starting a new generation.
Process interruption may leave an uncommitted temporary directory. Such a
directory is never considered published state, is never selected for Recent
Changes, and may be removed by a documented safe cleanup mechanism. Cleanup
must distinguish inactive temporary directories from concurrent active
invocations and must never recursively target the workspace root or an
unresolved configuration path.
## Inspection And Metadata Policy
Run-history discovery and inspection are not part of the desired product
contract. The historical `inspect reports`, `inspect metadata`, `inspect
modules`, `inspect data-package`, `inspect prior`, and `inspect sources`
surfaces are candidates for removal together rather than preservation through
a new storage representation.
Any manifest retained for atomic publication or Recent Changes is current
operational state, not an archival metadata record. It should contain only the
identity, valid period, safe paths, and deterministic snapshot information
needed to validate and use that current state. It does not need to preserve
prompt messages, generated prose intermediates, source provenance, provider
provenance, notification history, or a catalog of prior runs.
The application does not promise cross-version decoding of ordinary workspace
state. A new release may replace or ignore incompatible current-state files,
provided it fails safely, never mistakes stale state for a compatible Recent
Changes baseline, and documents any operator action required during upgrade.
## Prompt Execution And Debugging
Prompt inspection before weather collection and prepared execution remain
runtime safety requirements. They do not require durable preparation or
execution receipts.
The selected logical profile, effective backend and model, validation outcome,
and safe classified error remain available to the active workflow and its CLI
summary where useful. Weatherreporter does not retain them as long-term report
provenance after the invocation completes.
The existing explicit secure debug root remains outside ordinary state and may
retain rendered prompts, schemas, input bodies, generated bodies, and effective
parameters according to its documented contract. Weatherreporter does not
automatically clean that operator-selected location. Credentials must remain
excluded from debug capture.
## Compatibility And Upgrade Policy
This is an intentional breaking change to the workspace and inspection
contracts. Weatherreporter does not need to migrate historical V1 or V2
metadata, prompt receipts, intermediate generated-text artifacts, or managed
reports into the new representation.
Legacy workspace trees must not be silently interpreted as current published
state. They also must not be deleted automatically merely because a new
version starts: an operator may have placed or referenced files there despite
the absence of a continuing application compatibility promise. Release notes
and operations documentation must explain whether legacy data can be removed
manually and identify the exact safe target.
The change should land in a release whose notes clearly identify removed CLI
commands, obsolete paths and schemas, the new bounded state behavior, and any
upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic
version policy may carry this breaking change without inventing a migration
framework.
## Required Architecture Decision Record
The implemented feature must include an Accepted ADR recording the durable
architectural decision to use ephemeral operational state. The ADR is not part
of this roadmap-writing pass and should not be created until implementation is
being prepared.
The ADR should record:
- the mismatch between run-addressed provenance storage and the ephemeral
weather-report lifecycle;
- the decision to retain bounded current report and comparison state rather
than historical runs;
- the distinction between temporary invocation state, published operational
state, explicit output copies, and secure debug capture;
- the removal of historical inspection and backward-compatibility guarantees;
- atomic publication and failed-run behavior;
- the alternatives considered, including retaining the current archive,
adding time-based retention, or keeping a bounded run history; and
- consequences for CLI compatibility, workspace layout, testing, operations,
and future schema changes.
Once accepted, the ADR owns the decision rationale. The architecture policy
owns the resulting current invariant, while focused state, CLI, operations,
and integration documents own the implemented contracts.
## Scope
The completed feature includes:
- an invocation-scoped temporary workspace for intermediate generation state;
- atomic publication of the current managed report and its minimal operational
state;
- a bounded comparison snapshot representing the last successfully published
report for each supported report key and valid period;
- safe cleanup behavior for normal completion, handled failure, and abandoned
temporary workspaces;
- removal of durable prompt preparation, prompt execution, generated-text,
render-context, data-package, notification, and run-metadata history;
- removal of run-history inspection commands and their application/state
contracts;
- removal of V1 metadata compatibility and current-version coupling for
historical prompt artifacts by removing the historical artifact contract;
- preservation of active-command partial status, safe errors, and paths to
genuinely retained published, operator-owned, or debug outputs;
- preservation of explicit output copies, Distributor upload behavior, and
opt-in secure LLM debug capture;
- risk-appropriate offline tests for atomic publication, comparison baselines,
failure isolation, cleanup safety, concurrent invocation safety, and absence
of unbounded state growth;
- an Accepted ADR documenting the architectural decision; and
- updates to canonical architecture, CLI, operations, configuration,
troubleshooting, integration, internal, testing, and release documentation
where their contracts change.
## Non-Goals
This feature does not include:
- a general-purpose cache, database, archival service, or retention engine;
- replaying or resuming interrupted generation;
- migrating legacy artifacts into the new representation;
- retaining a bounded number of historical runs for convenience;
- automatic upload or archival of state to remote storage;
- collecting additional provider telemetry or weather-source provenance;
- changing report content, prompt text, schemas, profile selection, weather
derivation, or batch membership;
- deleting operator-owned `--out`, `--out-dir`, or secure debug files;
- changing Distributor's report-content contract; or
- making live external services part of the default test suite.
## Safety And Testing Policy
The state refactoring must preserve Weatherreporter's existing path-safety and
atomicity expectations while reducing the amount of durable state. Tests
should emphasize observable lifecycle guarantees rather than private file
choreography.
Important risks requiring durable offline coverage include:
- a failed or canceled generation replacing a previously published report or
comparison baseline;
- a partially written report becoming visible as current;
- Recent Changes selecting an incompatible report, valid period, or failed
attempt;
- cleanup deleting published, operator-owned, debug, or concurrently active
files;
- batch partial success corrupting the state of another report;
- notification failure rolling back or obscuring a successfully published
report;
- stale or incompatible current state being treated as valid; and
- repeated successful and failed runs causing unbounded ordinary workspace
growth.
Tests remain deterministic, offline, credential-free, and based on real
temporary filesystems plus narrow external-boundary fakes. Race-enabled tests
are required where publication, cleanup, or concurrent invocation behavior
shares mutable filesystem state.
## Relationship To Domain-Specific Profiles
The domain-specific profile feature can be implemented before this refactor,
but it should not add new historical compatibility or durable-provenance
commitments. Profile inspection, selection, override precedence, and effective
model resolution remain active-workflow behavior and survive the state change.
The domain-profile roadmap and implementation plan should acknowledge that
prompt artifacts from version `1.0.1` need not remain readable after prompts
advance to `1.1.0`. Existing state persistence may remain temporarily while
the profile feature lands, but it should not be expanded or treated as the
target architecture.
## Completion Criteria
The roadmap's target state is achieved when:
- ordinary runs no longer create durable run-addressed artifact collections;
- a successful report atomically replaces only the corresponding current
published state;
- failed and canceled attempts leave the prior published report and Recent
Changes baseline unchanged;
- Daily, Today, and Tomorrow compare against at most one compatible snapshot
from the last successfully published report;
- expired comparison and published state can be removed safely without
touching operator-owned or active files;
- Hourly does not retain an unused comparison snapshot;
- historical inspection commands and V1/V2 archival compatibility code are
removed;
- temporary, published, explicit-output, and debug paths have distinct and
documented ownership and cleanup rules;
- Distributor and active-command summaries continue to receive the completed
report and safe status information they require;
- the default suite proves atomicity, bounded growth, cleanup safety, batch
isolation, and comparison correctness offline;
- an Accepted ADR records the architectural decision and alternatives; and
- canonical current-state documentation describes only the implemented
lifecycle.
## Open Questions
### Lifetime of the current managed report
Recommendation: retain one current managed report per logical report key and
valid period until it is replaced or its valid period expires. This preserves
the current default behavior for invocations without `--out` while bounding
growth.
Alternative: treat the managed report as temporary and retain output only when
the operator supplies `--out` or `--out-dir`. This minimizes state further but
makes a successful default invocation produce no durable report for the user
and complicates Distributor sequencing.
### Historical inspection replacement
Recommendation: remove the run-history inspection commands without adding a
replacement initially. Current command summaries, current managed files, and
opt-in debug capture cover the remaining supported workflows.
Alternative: add a narrow `inspect current REPORT` command backed only by the
current operational manifest. This provides discoverability without history,
but it creates a new public surface and may preserve metadata complexity that
the refactor is intended to remove.
### Abandoned temporary workspace cleanup
Recommendation: use an explicitly owned temporary subtree with per-invocation
ownership markers and a conservative age threshold. Normal cleanup removes the
current invocation synchronously; opportunistic cleanup removes only marked,
inactive directories old enough that they cannot reasonably belong to a live
invocation.
Alternative: perform only synchronous cleanup and document manual removal of
directories left by process termination. This minimizes destructive code and
concurrency risk, but crashed processes can still accumulate unbounded files.
### Legacy workspace cleanup
Recommendation: ignore legacy run-addressed trees and document a precise,
manual one-time cleanup procedure. Do not automatically delete them during
startup or upgrade.
Alternative: add an explicit cleanup command that previews and then removes
recognized legacy artifacts. This is more convenient for large installations
but introduces a destructive command and a legacy-format classifier that must
be maintained and tested.

File diff suppressed because it is too large Load Diff

View File

@@ -1,6 +1,11 @@
# Promptkit Feature Request: Application Fallback Profiles
Status: Proposed upstream capability.
Status: Implemented upstream in Promptkit v0.5.0.
Promptkit v0.5.0 resolved this request with the public
`WithFallbackProfileFS` engine option and the precedence and error semantics
specified below. This document is retained as the downstream rationale for
the capability.
## Purpose

View File

@@ -1,7 +1,10 @@
# Troubleshooting
Keep failed workspace artifacts in place. When a RunID is available, start
with `weatherreporter inspect metadata RUN_ID` and use the paths in its result.
Start with the command's classified error. When content-rich prompt diagnostics
are needed, enable a new run with `--llm-debug-dir` and handle the resulting
secure capture as sensitive. Current-version workspace receipts can provide
additional context when present, but are transitional state rather than a
long-term troubleshooting interface.
## Prompt inspection or credentials fail before collection
@@ -11,13 +14,37 @@ the configured `promptkit` profile or profile source, confirm the exact
Promptkit asset is available, and supply any reported environment credential.
Do not add provider keys to YAML. See [configuration](config.md).
## Local profile override is malformed or selects an unexpected model
`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile
definitions. A same-ID definition replaces the embedded profile, and a malformed
matching definition fails before collection instead of falling back. Validate
the selected profile's YAML, ID, backend or endpoint, and model. If the model
is unexpected, first check the global `promptkit.profile` selection and then
look for a same-ID definition in the configured file or directory.
Current-version preparation and execution receipts may retain the selected
profile ID and effective backend/model, but not an endpoint or credential.
Use them only as supplemental context after the active command error or an
explicit secure debug capture. See the maintained
[local `weather-light` profile example](../examples/weather-light-local-profile.yml).
## Local model endpoint is unavailable
An endpoint-only `weather-light` override can pass preflight and still fail
during provider preparation or execution when the local server is unavailable
or does not accept the configured model. Start the local server, correct the
endpoint or model in the profile, and run the command again. Weatherreporter
does not probe endpoints or automatically use a remote profile instead.
## Preparation, capacity, or execution fails
A preparation failure occurs before provider work; an execution failure occurs
after preparation. Both leave safe provenance and metadata when reached. A
capacity error for one batch report does not retry that report or prevent later
independent reports. Inspect the preparation or execution path, correct the
profile/backend condition, and create a new run. See [operations](operations.md).
after preparation. A capacity error for one batch report does not retry that
report or prevent later independent reports. Correct the profile or backend
condition identified by the bounded command error, then create a new run.
Use explicit secure debug capture only when additional content-rich diagnostics
are necessary. See [operations](operations.md).
## Generated text fails validation

View File

@@ -0,0 +1,4 @@
id: weather-light
endpoint: http://127.0.0.1:11434/v1
model: weather-local
timeout_seconds: 180

View File

@@ -45,6 +45,7 @@ func newAdapter(config Config, additionalOptions ...promptkit.Option) (*Adapter,
options := []promptkit.Option{
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
}
if config.ProfileFile != "" {
options = append(options, promptkit.WithProfileFile(config.ProfileFile))

View File

@@ -68,11 +68,11 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
func TestInspectPromptAndProfile(t *testing.T) {
adapter := newTestAdapter(t, &fakeClient{})
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.0.1")
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.1.0")
if err != nil {
t.Fatalf("InspectPrompt() error = %v", err)
}
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.0.1" || inspection.DefaultProfileID != "gemini-flash-latest" {
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != "weather-balanced" {
t.Fatalf("inspection = %#v", inspection)
}
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
@@ -102,6 +102,107 @@ func TestInspectPromptAndProfile(t *testing.T) {
}
}
func TestEmbeddedProfilesAreAvailableToProductionAndTestAdapters(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
for _, want := range []struct {
id string
backend string
model string
}{
{"weather-light", "openrouter", "deepseek/deepseek-v4-flash"},
{"weather-balanced", "openrouter", "~google/gemini-flash-latest"},
{"weather-deep", "openrouter", "~anthropic/claude-sonnet-latest"},
} {
t.Run(want.id, func(t *testing.T) {
assertProfile(t, adapter, want.id, want.backend, want.model)
})
}
testAdapter, err := newAdapterForTest(Config{}, &fakeClient{})
if err != nil {
t.Fatalf("newAdapterForTest() error = %v", err)
}
assertProfile(t, testAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
}
func TestConfiguredProfilesOverrideEmbeddedFallbacks(t *testing.T) {
file := writeProfileFile(t, `id: weather-light
endpoint: https://local-file.example/v1
model: file-light
`)
fileAdapter, err := New(Config{ProfileFile: file})
if err != nil {
t.Fatalf("New(profile file) error = %v", err)
}
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
directory := testProfileDirectory(t, `id: weather-light
backend: local
model: directory-light
`)
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil {
t.Fatalf("New(profile directory) error = %v", err)
}
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
}
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
adapter, err := New(Config{ProfileFile: filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml")})
if err != nil {
t.Fatalf("New() error = %v", err)
}
assertProfile(t, adapter, "weather-light", "", "weather-local")
}
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile
backend: openrouter
model: other-model
`)})
if err != nil {
t.Fatalf("New(absent profile) error = %v", err)
}
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light
backend: openrouter
`)})
if err != nil {
t.Fatalf("New(malformed profile) error = %v", err)
}
if _, err := malformedAdapter.InspectProfile(context.Background(), "weather-light"); err == nil {
t.Fatal("InspectProfile() error = nil, want malformed configured profile error")
}
}
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
builtin, err := adapter.InspectProfile(context.Background(), "gemini-flash-latest")
if err != nil {
t.Fatalf("InspectProfile(builtin) error = %v", err)
}
if builtin.ProfileID != "gemini-flash-latest" || builtin.BackendID != "openrouter" || builtin.ModelName == "" {
t.Fatalf("builtin profile = %#v", builtin)
}
explicit, err := newAdapter(Config{}, promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
Endpoint: "https://explicit.example/v1",
Model: "explicit-light",
}))
if err != nil {
t.Fatalf("newAdapter(explicit profile) error = %v", err)
}
assertProfile(t, explicit, "weather-light", "", "explicit-light")
}
func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
client := &fakeClient{response: validResponse()}
adapter := newTestAdapter(t, client)
@@ -141,6 +242,44 @@ func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
}
}
func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
t.Setenv("OPENROUTER_API_KEY", "test-openrouter-key")
client := &fakeClient{response: hourlyValidResponse()}
adapter, err := newAdapter(Config{}, promptkit.WithLLMClient(client))
if err != nil {
t.Fatalf("newAdapter() error = %v", err)
}
request := promptexec.ExecuteRequest{
PromptID: "weather.hourly_generated_text",
PromptVersion: "1.1.0",
ProfileID: "weather-light",
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
DataPackagePath: "data-packages/hourly/data_package.yaml",
}
var preparation promptexec.Preparation
prepared := false
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
if client.callCount() != 0 {
t.Fatal("provider was called before preparation completed")
}
preparation = value
prepared = true
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if !prepared || preparation.ProfileID != "weather-light" || preparation.BackendID != "openrouter" || preparation.ModelName != "deepseek/deepseek-v4-flash" {
t.Fatalf("preparation = %#v", preparation)
}
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "openrouter" || result.ModelName != "deepseek/deepseek-v4-flash" || result.Validation.Status != promptexec.ValidationPassed {
t.Fatalf("execution = %#v", result)
}
if client.callCount() != 1 || client.request().Target.Model != "deepseek/deepseek-v4-flash" {
t.Fatalf("provider calls/request = %d/%#v", client.callCount(), client.request())
}
}
func TestExecuteUsesExactInlineDataPackageProvenance(t *testing.T) {
client := &fakeClient{response: validResponse()}
reader := &recordingReader{}
@@ -339,6 +478,17 @@ func newTestAdapter(t *testing.T, client promptkit.LLMClient) *Adapter {
return newTestAdapterWithOptions(t, client)
}
func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, model string) {
t.Helper()
profile, err := adapter.InspectProfile(context.Background(), id)
if err != nil {
t.Fatalf("InspectProfile(%q) error = %v", id, err)
}
if profile.ProfileID != id || profile.BackendID != backend || profile.ModelName != model {
t.Fatalf("profile = %#v, want %q with backend/model %q/%q", profile, id, backend, model)
}
}
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
t.Helper()
profiles := testProfileDirectory(t, `id: test-profile
@@ -366,10 +516,19 @@ func testProfileDirectory(t *testing.T, profile string) string {
return profiles
}
func writeProfileFile(t *testing.T, profile string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "profile.yml")
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
return path
}
func testExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{
PromptID: "weather.daily_generated_text",
PromptVersion: "1.0.1",
PromptVersion: "1.1.0",
ProfileID: "test-profile",
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
DataPackagePath: "data-packages/daily/data_package.yaml",
@@ -382,3 +541,10 @@ func validResponse() *promptkit.GenerateResponse {
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
}
}
func hourlyValidResponse() *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
}
}

View File

@@ -33,10 +33,10 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
t.Fatalf("batchInspectionCandidates() error = %v", err)
}
executor := &inspectionExecutor{profiles: map[string]promptexec.ProfileInspection{
"default-profile": {ProfileID: "default-profile", BackendID: "local", ModelName: "model"},
"weather-balanced": {ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
}, prompts: map[string]promptexec.PromptInspection{}}
for _, candidate := range candidates {
executor.prompts[candidate.Definition.PromptID] = validPromptInspection(candidate.Definition)
executor.prompts[candidate.Definition.PromptID] = logicalPromptInspection(candidate.Definition)
}
collector := collectorFunc(func(context.Context, collect.Request) (*collect.Result, error) {
return nil, errors.New("collection reached")
@@ -47,7 +47,7 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
if err == nil || err.Error() != "collection reached" {
t.Fatalf("RunBatchDetailed() error = %v, want collection error", err)
}
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 {
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
t.Fatalf("inspection calls = prompts %#v profiles %#v", executor.promptRequests, executor.profileRequests)
}
})

View File

@@ -44,7 +44,7 @@ func (e *assembledBatchExecutor) InspectPrompt(_ context.Context, id, version st
if !ok || definition.PromptVersion != version {
return promptexec.PromptInspection{}, errors.New("unexpected prompt inspection")
}
return validPromptInspection(definition), nil
return logicalPromptInspection(definition), nil
}
func (e *assembledBatchExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
@@ -151,8 +151,8 @@ func TestRunBatchDetailedExecutesRetainedReportsSequentially(t *testing.T) {
if len(executor.executeRequests) != len(test.wantIDs) || executor.maxActive != 1 {
t.Fatalf("executor calls/max active = %d/%d, want %d/1", len(executor.executeRequests), executor.maxActive, len(test.wantIDs))
}
if len(executor.profileRequests) != 1 {
t.Fatalf("profile inspections = %#v, want one shared profile inspection", executor.profileRequests)
if len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
t.Fatalf("profile inspections = %#v, want one shared weather-balanced inspection", executor.profileRequests)
}
for index, item := range result.Reports {
if item.ReportID != test.wantIDs[index] || item.Status != "succeeded" {

View File

@@ -189,3 +189,13 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
}
}
func logicalPromptInspection(definition report.Definition) promptexec.PromptInspection {
inspection := validPromptInspection(definition)
if definition.ID == report.Hourly {
inspection.DefaultProfileID = "weather-light"
} else {
inspection.DefaultProfileID = "weather-balanced"
}
return inspection
}

View File

@@ -0,0 +1,73 @@
package app_test
import (
"context"
"os"
"path/filepath"
"testing"
"time"
promptkitadapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/promptkit"
"gitea.maximumdirect.net/eric/weatherreporter/internal/app"
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
)
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
lookupEnv := func(string) (string, bool) { return "test-key", true }
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
t.Helper()
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, id),
Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: profile},
LookupEnv: lookupEnv,
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if result.ProfileID != wantID || result.BackendID != wantBackend || result.ModelName != wantModel {
t.Fatalf("inspection = %#v, want profile/backend/model %q/%q/%q", result, wantID, wantBackend, wantModel)
}
}
embedded, err := promptkitadapter.New(promptkitadapter.Config{})
if err != nil {
t.Fatalf("New(embedded) error = %v", err)
}
inspect(t, embedded, report.Hourly, "", "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
inspect(t, embedded, report.Daily, "", "weather-balanced", "openrouter", "~google/gemini-flash-latest")
inspect(t, embedded, report.Daily, "weather-deep", "weather-deep", "openrouter", "~anthropic/claude-sonnet-latest")
override, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: writeProfileFile(t, `id: weather-light
endpoint: https://local.example/v1
model: local-weather
`)})
if err != nil {
t.Fatalf("New(override) error = %v", err)
}
inspect(t, override, report.Hourly, "", "weather-light", "", "local-weather")
}
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
t.Helper()
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)
request := report.ResolveRequest{Now: now, Location: time.UTC}
if id == report.Daily {
request.Date = now
}
resolved, err := report.DefaultRegistry().Resolve(id, request)
if err != nil {
t.Fatalf("Resolve(%q) error = %v", id, err)
}
return resolved
}
func writeProfileFile(t *testing.T, profile string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "profile.yml")
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
return path
}

View File

@@ -37,8 +37,10 @@ func (c *workflowCollector) Run(context.Context, collect.Request) (*collect.Resu
type workflowExecutor struct {
definition report.Definition
raw []byte
prompt promptexec.PromptInspection
inspectionErr error
profile promptexec.ProfileInspection
profileErr error
beforePreparationErr error
afterCallbackErr error
afterPreparationErr error
@@ -49,6 +51,8 @@ type workflowExecutor struct {
beforeProvider func()
preparationDebug *promptexec.PreparationDebug
executionDebug *promptexec.ExecutionDebug
preparation *promptexec.Preparation
execution *promptexec.Execution
}
func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string) (promptexec.PromptInspection, error) {
@@ -58,10 +62,16 @@ func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string)
if id != e.definition.PromptID || version != e.definition.PromptVersion {
return promptexec.PromptInspection{}, errors.New("unexpected prompt identity")
}
if e.prompt.PromptID != "" {
return e.prompt, nil
}
return validPromptInspection(e.definition), nil
}
func (e *workflowExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
if e.profileErr != nil {
return promptexec.ProfileInspection{}, e.profileErr
}
profile := e.profile
if profile.ProfileID == "" {
profile = promptexec.ProfileInspection{ProfileID: id, BackendID: "fixture", ModelName: "fixture-model"}
@@ -76,11 +86,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
return nil, e.beforePreparationErr
}
stamp := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
profile := e.profile
if profile.ProfileID == "" {
profile = promptexec.ProfileInspection{ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model"}
}
preparation := promptexec.Preparation{
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: "prompt-hash",
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture",
ModelName: "fixture-model", DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: profile.BackendID,
ModelName: profile.ModelName, DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
}
e.preparation = &preparation
if err := callback(preparation, e.preparationDebug); err != nil {
return nil, err
}
@@ -98,14 +113,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
if validation == "" {
validation = promptexec.ValidationPassed
}
return &promptexec.Execution{
execution := &promptexec.Execution{
RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion,
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID,
BackendID: "fixture", ModelName: "fixture-model", GeneratedHash: "generated-hash",
BackendID: profile.BackendID, ModelName: profile.ModelName, GeneratedHash: "generated-hash",
StartedAt: stamp, EndedAt: stamp, DataPackagePath: req.DataPackagePath, RawOutput: e.raw,
Debug: e.executionDebug,
Validation: promptexec.NewValidation(validation, "json_schema", e.definition.GeneratedTextSchemaID+".generated_text.schema.json", nil),
}, nil
}
e.execution = execution
return execution, nil
}
type workflowNotifier struct {
@@ -208,6 +225,57 @@ func TestGenerateDetailedCompletesRetainedReportWorkflows(t *testing.T) {
}
}
func TestGenerateDetailedPreservesSelectedProfileThroughExecution(t *testing.T) {
tests := []struct {
name string
kind ReportKind
id report.ID
raw string
override string
profile promptexec.ProfileInspection
}{
{
name: "hourly default", kind: ReportHourly, id: report.Hourly, raw: validHourlyWorkflowJSON(),
profile: promptexec.ProfileInspection{ProfileID: "weather-light", BackendID: "openrouter", ModelName: "deepseek/deepseek-v4-flash"},
},
{
name: "daily default", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(),
profile: promptexec.ProfileInspection{ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
},
{
name: "global override", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(), override: "operator-profile",
profile: promptexec.ProfileInspection{ProfileID: "operator-profile", BackendID: "local", ModelName: "local-weather-model"},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
cfg := workflowConfig(t)
cfg.Promptkit.Profile = test.override
definition := report.DefaultRegistry().MustLookup(test.id)
executor := &workflowExecutor{
definition: definition, prompt: logicalPromptInspection(definition), profile: test.profile, raw: []byte(test.raw),
}
bundle := workflowBundle(t)
_, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: cfg, Report: test.kind, Date: workflowTime("2026-05-29T12:00:00-05:00"), Now: workflowTime("2026-05-29T08:30:00-05:00"),
Collector: &workflowCollector{result: &collect.Result{Bundle: &bundle}}, Executor: executor, Notifier: &workflowNotifier{},
})
if err != nil {
t.Fatalf("GenerateDetailed() error = %v", err)
}
if executor.request.ProfileID != test.profile.ProfileID {
t.Fatalf("execution profile = %q, want %q", executor.request.ProfileID, test.profile.ProfileID)
}
if executor.preparation == nil || executor.preparation.ProfileID != test.profile.ProfileID || executor.preparation.BackendID != test.profile.BackendID || executor.preparation.ModelName != test.profile.ModelName {
t.Fatalf("prepared profile = %#v, want %q/%q/%q", executor.preparation, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
}
if executor.execution == nil || executor.execution.ProfileID != test.profile.ProfileID || executor.execution.BackendID != test.profile.BackendID || executor.execution.ModelName != test.profile.ModelName {
t.Fatalf("executed profile = %#v, want %q/%q/%q", executor.execution, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
}
})
}
}
type preparationFailingStore struct {
state.Store
}
@@ -299,6 +367,11 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
wantCategory promptexec.ErrorCategory
}{
{name: "inspection", configure: func(e *workflowExecutor) { e.inspectionErr = errors.New("inspection unavailable") }, wantCategory: promptexec.InvalidConfiguration},
{name: "unknown profile", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unknown selected profile") }, wantCategory: promptexec.InvalidConfiguration},
{name: "malformed profile", configure: func(e *workflowExecutor) {
e.profileErr = errors.New("malformed profile at https://operator.example/v1 api_key=secret")
}, wantCategory: promptexec.InvalidConfiguration},
{name: "unusable backend", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unsupported backend") }, wantCategory: promptexec.InvalidConfiguration},
{name: "credential", configure: func(e *workflowExecutor) {
e.profile = promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true}
}, wantCategory: promptexec.MissingCredential},
@@ -317,6 +390,9 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
if err == nil || result != nil || promptexec.CategoryOf(err) != test.wantCategory || collector.calls != 0 || executor.executeCalls != 0 {
t.Fatalf("result/error/category/collect/execute = %#v/%v/%q/%d/%d", result, err, promptexec.CategoryOf(err), collector.calls, executor.executeCalls)
}
if strings.Contains(err.Error(), "operator.example") || strings.Contains(err.Error(), "secret") {
t.Fatalf("error leaks profile details: %v", err)
}
entries, readErr := os.ReadDir(cfg.Workspace.Root)
if readErr != nil || len(entries) != 0 {
t.Fatalf("workspace entries/error = %#v/%v, want no writes before collection", entries, readErr)

View File

@@ -0,0 +1,6 @@
id: weather-balanced
backend: openrouter
model: "~google/gemini-flash-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -0,0 +1,6 @@
id: weather-deep
backend: openrouter
model: "~anthropic/claude-sonnet-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -0,0 +1,5 @@
id: weather-light
backend: openrouter
model: deepseek/deepseek-v4-flash
timeout_seconds: 180
service_tier: flex

View File

@@ -1,6 +1,6 @@
id: weather.daily_generated_text
version: "1.0.1"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-balanced
description: Daily weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -1,6 +1,6 @@
id: weather.hourly_generated_text
version: "1.0.1"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-light
description: Hourly weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -1,6 +1,6 @@
id: weather.today_generated_text
version: "1.0.1"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-balanced
description: Today's weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -1,6 +1,6 @@
id: weather.tomorrow_generated_text
version: "1.0.1"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-balanced
description: Tomorrow's weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -7,7 +7,7 @@ import (
"io/fs"
)
//go:embed assets/prompts assets/schemas
//go:embed assets/prompts assets/profiles assets/schemas
var assets embed.FS
var schemaPaths = map[string]string{
@@ -35,6 +35,15 @@ func SchemaFS() fs.FS {
return fsys
}
// ProfileFS returns the embedded Weatherreporter Promptkit profile definitions.
func ProfileFS() fs.FS {
fsys, err := fs.Sub(assets, "assets/profiles")
if err != nil {
panic(fmt.Sprintf("embedded profile assets: %v", err))
}
return fsys
}
// Schema returns an independent copy of the canonical schema for id.
func Schema(id string) ([]byte, error) {
path, ok := schemaPaths[id]

View File

@@ -34,11 +34,12 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
path string
id string
schemaID string
profile string
}{
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily"},
{"today/today_generated_text.yml", "weather.today_generated_text", "today"},
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow"},
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly"},
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily", "weather-balanced"},
{"today/today_generated_text.yml", "weather.today_generated_text", "today", "weather-balanced"},
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow", "weather-balanced"},
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly", "weather-light"},
}
definitions := 0
@@ -67,8 +68,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("decode prompt definition: %v", err)
}
if definition.ID != tc.id || definition.Version != "1.0.1" || definition.DefaultProfile != "gemini-flash-latest" {
t.Fatalf("definition = %#v, want %s version 1.0.1 and gemini-flash-latest", definition, tc.id)
if definition.ID != tc.id || definition.Version != "1.1.0" || definition.DefaultProfile != tc.profile {
t.Fatalf("definition = %#v, want %s version 1.1.0 and profile %s", definition, tc.id, tc.profile)
}
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
@@ -123,21 +124,116 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
)
if err != nil {
t.Fatalf("NewEngine() error = %v", err)
}
for _, id := range []string{"weather.daily_generated_text", "weather.today_generated_text", "weather.tomorrow_generated_text", "weather.hourly_generated_text"} {
t.Run(id, func(t *testing.T) {
inspection, err := engine.InspectPrompt(context.Background(), id, "1.0.1")
for _, want := range []struct {
id string
profile string
model string
}{
{"weather.daily_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
{"weather.today_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
{"weather.tomorrow_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
} {
t.Run(want.id, func(t *testing.T) {
inspection, err := engine.InspectPrompt(context.Background(), want.id, "1.1.0")
if err != nil {
t.Fatalf("InspectPrompt() error = %v", err)
}
if inspection.PromptID != id || inspection.PromptVersion != "1.0.1" || inspection.DefaultProfileID != "gemini-flash-latest" {
if inspection.PromptID != want.id || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != want.profile {
t.Fatalf("inspection = %#v", inspection)
}
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
if err != nil || profile.EffectiveModelParams.Model != want.model {
t.Fatalf("profile/error = %#v/%v, want model %q", profile, err, want.model)
}
})
}
profile, err := engine.InspectProfile(context.Background(), "weather-deep")
if err != nil || profile.EffectiveModelParams.Model != "~anthropic/claude-sonnet-latest" {
t.Fatalf("weather-deep profile/error = %#v/%v", profile, err)
}
}
func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
wantPaths := map[string]bool{
"weather-balanced.yml": false,
"weather-deep.yml": false,
"weather-light.yml": false,
}
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
if _, ok := wantPaths[path]; !ok {
t.Fatalf("unexpected embedded profile asset %q", path)
}
wantPaths[path] = true
return nil
}); err != nil {
t.Fatalf("walk embedded profiles: %v", err)
}
for path, found := range wantPaths {
if !found {
t.Errorf("missing embedded profile asset %q", path)
}
}
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
)
if err != nil {
t.Fatalf("NewEngine() error = %v", err)
}
profiles := []struct {
id string
model string
timeoutSeconds int
reasoningEffort string
}{
{"weather-light", "deepseek/deepseek-v4-flash", 180, ""},
{"weather-balanced", "~google/gemini-flash-latest", 240, "high"},
{"weather-deep", "~anthropic/claude-sonnet-latest", 240, "high"},
}
for _, want := range profiles {
t.Run(want.id, func(t *testing.T) {
inspection, err := engine.InspectProfile(context.Background(), want.id)
if err != nil {
t.Fatalf("InspectProfile() error = %v", err)
}
got := inspection.EffectiveModelParams
if inspection.ProfileID != want.id || got.BackendID != "openrouter" || got.Model != want.model || got.TimeoutSeconds != want.timeoutSeconds || got.ServiceTier != "flex" || got.ReasoningEffort != want.reasoningEffort {
t.Fatalf("inspection = %#v, want %q using openrouter model %q", inspection, want.id, want.model)
}
})
}
}
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
if err != nil {
return err
}
for _, setting := range forbidden {
if strings.Contains(string(data), setting) {
t.Fatalf("%s contains forbidden profile setting %q", path, setting)
}
}
return nil
}); err != nil {
t.Fatalf("walk embedded profiles: %v", err)
}
}
func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {

View File

@@ -12,7 +12,7 @@ func dailyDefinition() Definition {
ID: Daily,
Name: "Daily Report",
PromptID: "weather.daily_generated_text",
PromptVersion: "1.0.1",
PromptVersion: "1.1.0",
TemplateID: "daily",
GeneratedTextSchemaID: "daily",
ComparisonStrategy: CompareSameValidDate,

View File

@@ -14,7 +14,7 @@ func hourlyDefinition() Definition {
ID: Hourly,
Name: "Hourly Report",
PromptID: "weather.hourly_generated_text",
PromptVersion: "1.0.1",
PromptVersion: "1.1.0",
TemplateID: "hourly",
GeneratedTextSchemaID: "hourly",
ComparisonStrategy: CompareRollingWindow,

View File

@@ -51,8 +51,11 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
}
for _, definition := range definitions {
if definition.PromptVersion != "1.0.1" {
t.Fatalf("%s PromptVersion = %q, want 1.0.1", definition.ID, definition.PromptVersion)
if definition.PromptVersion != "1.1.0" {
t.Fatalf("%s PromptVersion = %q, want 1.1.0", definition.ID, definition.PromptVersion)
}
if definition.PromptID == "" {
t.Fatalf("%s PromptID is empty", definition.ID)
}
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)

View File

@@ -10,7 +10,7 @@ func todayDefinition() Definition {
ID: Today,
Name: "Today Report",
PromptID: "weather.today_generated_text",
PromptVersion: "1.0.1",
PromptVersion: "1.1.0",
TemplateID: "today",
GeneratedTextSchemaID: "today",
ComparisonStrategy: CompareSameValidDate,

View File

@@ -10,7 +10,7 @@ func tomorrowDefinition() Definition {
ID: Tomorrow,
Name: "Tomorrow Report",
PromptID: "weather.tomorrow_generated_text",
PromptVersion: "1.0.1",
PromptVersion: "1.1.0",
TemplateID: "tomorrow",
GeneratedTextSchemaID: "tomorrow",
ComparisonStrategy: CompareSameValidDate,

View File

@@ -172,9 +172,9 @@ func validPreparationArtifact() PromptPreparationArtifact {
return PromptPreparationArtifact{
SchemaVersion: PromptPreparationSchemaVersion, Status: PromptPreparationSucceeded,
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text",
PromptVersion: "1.0.1", DataPackagePath: "/workspace/data.yaml",
PromptVersion: "1.1.0", DataPackagePath: "/workspace/data.yaml",
Preparation: &promptexec.Preparation{
PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
DataPackagePath: "/workspace/data.yaml",
},
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
@@ -186,9 +186,9 @@ func validExecutionArtifact() PromptExecutionArtifact {
validation := promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", "daily.generated_text.schema.json", nil)
return PromptExecutionArtifact{
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionSucceeded,
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
Provenance: &PromptExecutionProvenance{
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: "profile",
BackendID: "backend", ModelName: "model", DataPackagePath: "/workspace/data.yaml",
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
@@ -201,7 +201,7 @@ func validFailedExecutionArtifact() PromptExecutionArtifact {
started := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
return PromptExecutionArtifact{
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionFailed,
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
StartedAt: started, EndedAt: started, Error: &PromptArtifactError{Category: promptexec.Generation, Message: "provider unavailable"},
}
}