11 Commits

62 changed files with 2132 additions and 641 deletions

View File

@@ -22,8 +22,12 @@ extra report copies for a command and do not change configuration.
collection and generation configuration.
- [config.yml](../examples/config.yml) is a representative production-oriented
configuration using synthetic endpoints and no credentials.
- [weather-light-local-profile.yml](../examples/weather-light-local-profile.yml)
is a complete endpoint-only override for the embedded `weather-light`
profile.
Both files are loaded by the configuration test suite.
The configuration examples are loaded by the configuration test suite. The
profile example is inspected through the Promptkit adapter test suite.
## Minimal Configuration
@@ -146,13 +150,35 @@ individual `generate` or `run` command when explicitly needed.
| Field | Default | Rules |
| --- | --- | --- |
| `profile` | empty | Optional explicit execution profile. Otherwise the prompt's declared default is used. |
| `profile_file` | empty | Optional external profile file. Cannot be combined with `profile_dir`. |
| `profile_dir` | empty | Optional external profile directory. Cannot be combined with `profile_file`. |
| `profile` | empty | Optional global profile selection for every report in one command. When empty, each exact prompt version selects its declared default. |
| `profile_file` | empty | Optional external Promptkit profile file. It cannot be combined with `profile_dir`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
| `profile_dir` | empty | Optional external Promptkit profile directory. It cannot be combined with `profile_file`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
| `timeout` | `2m` | Must be greater than zero. |
| `local.endpoint` | empty | Optional absolute URL for the conventional local backend. A blank endpoint leaves it unregistered. |
| `local.concurrency_limit` | `1` | Maximum local backend concurrency. `0` is unlimited; negative values are invalid. |
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
They are separate decisions. An explicit `profile` applies to every selected
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
Tomorrow select `weather-balanced` through their exact `1.1.0` prompt
definitions.
Promptkit resolves a selected profile definition from a test or embedding
consumer's explicit in-memory profile, then the configured `profile_file` or
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
built-in catalog. Sources provide complete definitions; fields are never
merged. A matching malformed external profile fails rather than using the
embedded definition. The [Promptkit integration guide](integrations/promptkit.md)
owns the catalog and precedence details.
To replace the default Hourly definition with a local OpenAI-compatible
endpoint, set `profile_file` to a copy of
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml).
The example has no credential and should be edited for the local endpoint and
model before use. An alternative profile may use `backend: local`; in that
case `promptkit.local.endpoint` supplies the conventional local backend
endpoint.
### `workspace`
| Field | Default |

View File

@@ -1,22 +1,60 @@
# Promptkit Integration
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are
`weather.daily_generated_text`, `weather.today_generated_text`,
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version
`1.0.0`. Their prompt assets and generated-text JSON Schemas are embedded by
`internal/promptassets`.
Weatherreporter uses Promptkit for all generated-text reports. The four logical
prompts are `weather.daily_generated_text`, `weather.today_generated_text`,
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at
version `1.1.0`. Their prompt assets, generated-text JSON Schemas, and
Weatherreporter profile catalog are embedded by `internal/promptassets`.
Before collection, Weatherreporter inspects the exact prompt version, requires one required
`data_package` input with content type `application/yaml`, and requires the report's JSON
Schema output contract. It selects `promptkit.profile` when configured, otherwise the
prompt's declared default profile. Profiles that require a direct API key are unsupported; a
profile that reports `APIKeyEnv` requires a nonblank value in that environment variable.
## Logical profile catalog
Prompt definitions select a stable Weatherreporter profile ID. The embedded
definitions currently use Promptkit's `openrouter` backend:
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
| --- | --- | --- | --- | --- | --- |
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` | Hourly |
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
profiles intentionally omit endpoints, credentials, temperature, `top_p`, and
output-token limits.
## Selection, lookup, and active execution
Before collection, Weatherreporter inspects the exact prompt version and output
contract. A nonblank `promptkit.profile` selects one profile ID for every
report in the command; otherwise the prompt's declared default selects it.
Promptkit resolves the selected definition in this order:
1. explicit in-memory profiles used by an embedding consumer or test;
2. the configured `profile_file` or `profile_dir`;
3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's built-in catalog.
A source falls through only when the selected ID is absent. Each source
supplies a complete definition, so profile fields are not merged. A malformed
matching operator definition is an error and does not fall back.
Profiles that require a direct API key are unsupported; a profile that reports
`APIKeyEnv` requires a nonblank value in that environment variable. Inspection,
preparation, and execution retain the selected logical profile ID and resolved
backend and model through Weatherreporter's project-owned contract. Ordinary
errors, summaries, logs, and workspace state exclude endpoints, credentials,
rendered messages, schemas, request bodies, response bodies, and complete
parameter maps.
Execution receives the already-persisted YAML package, prepares it once, and returns structured
JSON that Weatherreporter validates before rendering its own Markdown template. Preparation and
execution receipts are project-owned, safe provenance records. Content-rich diagnostics are
opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
Prompt/profile configuration is owned by the [configuration reference](../config.md). Adapter
construction and mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
The generated-text schemas require `summary`, `forecast_discussion`, and
`precipitation_timing`, and reject additional properties. Prompts return an empty string for
`precipitation_timing` when the deterministic package contains no precipitation windows.
Prompt/profile configuration and the maintained local override example are
owned by the [configuration reference](../config.md). Adapter construction and
mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
Durable metadata compatibility is described in [state internals](../internal/state.md).

View File

@@ -7,7 +7,10 @@ notification ordering after the CLI has parsed arguments and loaded configuratio
`GenerateDetailed` resolves one of the four report definitions, initializes an
optional debug root, and inspects the exact Promptkit prompt/profile before it
collects weather or writes managed state. It then builds facts and modules,
collects weather or writes managed state. A configured global profile selects
every report in the action; otherwise the exact prompt selects its default
logical profile. Inspection keeps only the selected profile ID and effective
backend/model needed by the project-owned execution contract. It then builds facts and modules,
saves the YAML data package, persists preparation metadata from the executor
callback, executes the prepared prompt, saves execution provenance and raw
output, validates generated text, renders Markdown, and optionally copies or
@@ -25,8 +28,8 @@ persists raw output and execution provenance but does not render a report.
`RunBatchDetailed` constructs a single debug writer and uses the request's
single executor. Before collection it inspects Today, Tomorrow, and Daily for
morning, or Tomorrow and Daily for evening, deduplicating effective profile
inspection. It then collects once, plans eligible Daily dates, and calls the
morning, or Tomorrow and Daily for evening, deduplicating inspection of a
shared selected profile. It then collects once, plans eligible Daily dates, and calls the
same prompt-generation core sequentially for each planned report. Per-report
notification is suppressed; a failed report does not stop later reports.

View File

@@ -17,8 +17,9 @@ value and canonical normalized JSON, loads its canonical schema through
Daily, Today, and Tomorrow use a day-style value with required trimmed summary
and one or more nonblank discussion paragraphs. Hourly requires trimmed summary
and a single trimmed discussion string. Each form permits optional trimmed
precipitation-timing and confidence prose. Typed decoding rejects unknown JSON
and a single trimmed discussion string. Every form also requires the
`precipitation_timing` field; an empty string means there is no supported timing
prose to render. Typed decoding rejects missing required fields and unknown JSON
fields; no general-purpose JSON Schema engine is used at runtime.
## Render contexts

View File

@@ -4,10 +4,18 @@
The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor
per action. Promptkit dependency types do not escape the adapter.
The adapter exposes exact prompt and profile inspection plus prepared execution. It maps Promptkit
inspection values to project-owned prompt input, output-contract, profile, preparation, execution,
validation, and optional debug values. It classifies adapter failures without copying provider secrets
or unbounded response bodies into application errors or normal state.
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback
profile filesystems to each engine. Promptkit remains responsible for resolving
the configured operator profile source, application fallback catalog, and its
built-in catalog; the adapter does not parse profile YAML, merge sources, or
probe endpoints.
The adapter exposes exact prompt and profile inspection plus prepared execution.
It maps Promptkit inspection values to project-owned prompt input,
output-contract, logical profile identity, effective backend/model,
preparation, execution, validation, and optional debug values. It classifies
adapter failures without copying provider secrets or unbounded response bodies
into application errors or normal state.
The app calls the executor's preparation callback before provider execution to persist safe preparation
provenance. Completed executions are then persisted as safe execution provenance and raw generated text

View File

@@ -16,16 +16,19 @@ templates, generation eligibility, compatible prior IDs, default modules, and
batch eligibility flags. `Resolved` combines that definition with the valid
period and run metadata for one invocation.
| Report ID | Prompt version | Period policy | Comparison | Registry batch flag | Output copy |
| --- | --- | --- | --- | --- | --- |
| `daily` | `1.0.0` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
| `today` | `1.0.0` | Selected or current local civil day | Same valid date | Morning | `today.md` |
| `tomorrow` | `1.0.0` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
| `hourly` | `1.0.0` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
| Report ID | Prompt version | Default profile | Period policy | Comparison | Registry batch flag | Output copy |
| --- | --- | --- | --- | --- | --- | --- |
| `daily` | `1.1.0` | `weather-balanced` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
| `today` | `1.1.0` | `weather-balanced` | Selected or current local civil day | Same valid date | Morning | `today.md` |
| `tomorrow` | `1.1.0` | `weather-balanced` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
| `hourly` | `1.1.0` | `weather-light` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
Each report pairs its ID and prompt version with matching template and schema
IDs. Exact template fields and schema assets belong to [report templates](../templates.md)
and [generated-text internals](generatedtext.md).
and [generated-text internals](generatedtext.md). Prompt assets own default
profile selection; the registry deliberately stores no provider setting. The
[Promptkit integration guide](../integrations/promptkit.md) owns profile
definitions and resolution.
All valid periods are half-open.

View File

@@ -34,8 +34,10 @@ runs never write V1 records.
Prompt preparation and execution records are validated on both save and load.
They require exact report/prompt identity, complete timing, internally
consistent provenance, and status-appropriate validation or bounded classified
errors. Completed execution provenance keeps Promptkit's run identity distinct
from the Weatherreporter run identity.
errors. Completed preparation and execution provenance retain the selected
logical profile ID and resolved backend/model, but never profile endpoints or
credentials. Completed execution provenance keeps Promptkit's run identity
distinct from the Weatherreporter run identity.
For a completed prompt run, the execution record is atomically replaced after
each downstream artifact is saved. Its path set therefore records the raw and

View File

@@ -19,12 +19,38 @@ persists the module snapshot and prompt data package, records Promptkit
preparation provenance before provider execution, then persists raw output and
execution provenance, validates the structured generated text, and renders the
managed Markdown report from the validated text and deterministic values.
The current receipts are transitional workspace state, not a cross-version
profile-provenance contract.
The managed report and its final metadata are saved before single-report
Distributor notification is attempted. `--out` writes an extra operator copy;
it never changes the managed report or upload source. A successful generate
command prints its summary to stdout unless `--quiet` is used.
## Local Prompt Profile Override
Hourly normally selects the embedded `weather-light` profile. To use a local
OpenAI-compatible model without changing prompts or application code, copy
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml),
set its `endpoint` and `model` for the local server, and configure the copy as
`promptkit.profile_file`. The profile file's `weather-light` definition
completely replaces the embedded definition; it does not affect a report that
selects another profile ID.
For example, install the profile file at a known absolute path and set:
```yaml
promptkit:
profile_file: /etc/weatherreporter/weather-light-local-profile.yml
```
Prompt inspection occurs before weather collection. A malformed profile file,
missing required credential, or unsupported selected backend stops the command
before collection. A reachable profile can still fail later if its local model
endpoint is unavailable; Weatherreporter does not switch to a remote profile.
See the [configuration reference](config.md) for field definitions and the
[troubleshooting guide](troubleshooting.md) for recovery.
## Optional Prompt Debug Capture
Use `--llm-debug-dir` only when content-rich prompt diagnostics are required:
@@ -87,8 +113,11 @@ The generated-text and render-context artifacts are written for every completed
single-report generation.
A report's metadata links the module snapshot, data package, preparation and
execution receipts, managed report, generated-text artifacts, and any available single-report
notification artifact. Batch notification artifacts are separate batch-level
records under `notifications/batches`.
notification artifact. These current-version receipts remain transitional; use
the active command's classified error and explicit secure debug capture for
prompt diagnosis rather than relying on them as a durable interface. Batch
notification artifacts are separate batch-level records under
`notifications/batches`.
RunIDs begin with the UTC generation timestamp and report ID. A Daily RunID
also contains its local valid date so multiple Daily reports in one batch have
@@ -162,10 +191,12 @@ remain available where they can be safely persisted.
- A batch notification failure preserves each report's artifacts and adds the
top-level batch notification artifact.
Use the RunID from the action summary with the inspection commands above. For
a batch failure, inspect the summary first, then inspect the affected report
RunIDs or the batch notification path. Do not remove the whole workspace as a
first response; retain it until the failure is understood.
Use the action summary and its classified error first. For prompt or provider
diagnosis, prefer an explicitly enabled secure debug capture; current-version
receipt paths may provide supplemental context when available. For a batch
failure, inspect the summary first, then inspect the affected report RunIDs or
the batch notification path. Do not remove the whole workspace as a first
response; retain it until the failure is understood.
## Operational Caveats
@@ -174,5 +205,7 @@ first response; retain it until the failure is understood.
not publish them unintentionally.
- Weatherreporter uses one configured Weather API endpoint and local workspace
state.
- Promptkit profile resolution does not discover local endpoints or fail over
between local and remote profiles.
- It does not provide automatic resume, cleanup, archival, remote state, daemon
operation, or automatic storm monitoring.

View File

@@ -0,0 +1,283 @@
# Domain-Specific Prompt Profiles Roadmap
Status: Implemented.
## Purpose
Weatherreporter should provide stable, domain-specific Promptkit profile IDs
that express the relative resource and analysis needs of its report products.
These logical profiles should give each report an appropriate default while
allowing operators to replace any definition through the existing configured
profile source.
This roadmap records the scope, policy, and implemented end state. The
companion [implementation plan](implementation.md) records the ordered work
and verification used to reach it.
## User Intent
The feature is intended to provide three related benefits:
- frequent reports can use a cost-effective model by default;
- reports needing broader synthesis can select a stronger default without
forcing the same cost on every invocation; and
- an installation can map a stable Weatherreporter profile ID to a model on a
local network endpoint without modifying embedded prompts or application
code.
`weather-light` describes the profile's intended resource tier, not a latency
guarantee. A locally hosted lightweight model may still generate slowly on the
available hardware.
## Pre-Implementation Baseline
Before implementation, Daily, Today, Tomorrow, and Hourly each declared
Promptkit's `gemini-flash-latest` profile as their prompt default. The optional
`promptkit.profile` setting overrode that default for every selected report in
an invocation.
Weatherreporter accepted either `promptkit.profile_file` or
`promptkit.profile_dir` and passed that source to Promptkit. A matching external
profile could override a Promptkit built-in profile, and the configured local
backend could support profiles that select `backend: local`. Endpoint-only
OpenAI-compatible profiles could also provide their own endpoint.
Weatherreporter did not own or embed execution profiles. Promptkit v0.5.0
provided the fallback-profile layer used to add them without changing the
existing operator-source precedence.
## Prerequisite
Promptkit v0.5.0 provides the application fallback profile capability defined
in the companion
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
Weatherreporter now depends on that tagged release. The dependency upgrade has
passed the repository test suite and an operator smoke test. Weatherreporter
must continue to use only Promptkit's public API rather than depending on its
internal packages or reproducing its profile repository behavior.
## Implemented End State
Weatherreporter embeds usable definitions for these exact logical profile IDs:
- `weather-light`
- `weather-balanced`
- `weather-deep`
The profiles are Weatherreporter-owned assets and remain behind the existing
Promptkit adapter boundary. Prompt definitions select the logical IDs, while
Promptkit resolves the effective backend, endpoint, model, and generation
settings.
An operator can place a profile with the same ID in `profile_file` or
`profile_dir`. The operator definition completely replaces the embedded
Weatherreporter definition for that ID. If the external source does not contain
the selected ID, lookup falls through to Weatherreporter's embedded profile and
then to Promptkit's built-in catalog.
The existing global `promptkit.profile` setting remains available as an
explicit all-report override. No new configuration field is required for the
initial feature.
## Profile Catalog And Report Assignment
| Profile | Meaning | Initial default reports |
| --- | --- | --- |
| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly |
| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow |
| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially |
The initial assignment recognizes that Weatherreporter's deterministic modules
already perform most weather selection and calculation. A higher-capability
model should not become a default merely because it is available. Moving an
existing report to `weather-deep` requires evidence that the stronger tier
materially improves supported reasoning or output quality.
The three profile IDs are capability policies, not permanent aliases for one
provider or model family. Their embedded definitions may change in a future
Weatherreporter release, with the change disclosed through normal release and
compatibility documentation.
## Selection And Definition Precedence
Profile ID selection and profile definition lookup are separate decisions.
Weatherreporter selects the profile ID in this order:
1. nonblank `promptkit.profile`; or
2. the exact prompt version's `default_profile`.
Promptkit then resolves the selected profile definition in this order:
1. explicit in-memory profiles, when used by an embedding consumer or test;
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
3. Weatherreporter's embedded fallback profiles; and
4. Promptkit's embedded built-in profiles.
A higher-precedence source falls through only when the selected ID is absent.
A matching but malformed operator profile fails before weather collection and
must not silently use the embedded definition.
## Local Endpoint Experience
An operator should be able to override `weather-light` with an endpoint-only
profile whose model name is understood by the local OpenAI-compatible server.
This path does not require a separate Weatherreporter local-backend setting.
Alternatively, an override may select `backend: local`; in that case the
existing `promptkit.local.endpoint` and concurrency settings continue to own
the shared local backend definition.
The selected local profile is deterministic configuration, not a preference
hint. Weatherreporter will not probe for a local model and will not
automatically fall back to a remote or paid profile when the endpoint is
unavailable. The failure remains visible and attributable to the selected
profile.
## Embedded Profile Policy
Each embedded profile must be a complete, valid Promptkit profile and must be
usable in a default installation with the documented credential mechanism. The
initial embedded profiles are expected to use Promptkit's `openrouter` backend,
allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment
variable without embedding credentials.
Embedded definitions should include only settings that are intentional for the
selected model and supported by its backend. Avoid incidental generation
parameters that reduce portability or trigger provider-specific request
failures without a demonstrated quality benefit.
The initial profile definitions are:
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
| --- | --- | --- | --- | --- |
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
These settings deliberately match the corresponding Promptkit v0.5.0
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
part of each OpenRouter rolling-alias identifier. The profiles do not set
temperature, `top_p`, or output-token limits; omission preserves provider
defaults and avoids unsupported incidental parameters.
## Prompt And Active Execution Contract
Changing a prompt's `default_profile` is a material prompt-definition change.
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
defaults are introduced. Prompt content and generated-text schemas need not
change solely for this feature.
Prompt inspection must continue to occur before weather collection. It should
report the selected logical profile ID and the resolved backend and model
without exposing endpoints or credentials.
The active execution contract should retain both the selected logical profile
identity and the resolved backend and model through inspection, preparation,
execution, errors, and command results where those values are already exposed.
This feature must not add a new durable-provenance or cross-version artifact
contract.
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
prompt provenance a non-goal. Existing workspace persistence may remain while
this feature lands, but it is transitional behavior and must not be expanded or
treated as part of the profile feature's desired end state. Prompt preparation
and execution artifacts written at `1.0.1` are not required to remain readable
after the prompt definitions advance to `1.1.0`.
## Evaluation Policy
Concrete model assignments should be evaluated with representative,
secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should
consider:
- strict-schema success rate;
- unsupported or invented weather claims;
- precipitation-timing accuracy and empty-string behavior;
- correct use of deterministic hazards, periods, and uncertainty;
- summary and forecast-discussion usefulness;
- generation latency;
- token use and provider cost; and
- behavior through a representative local OpenAI-compatible endpoint.
The purpose is to choose an appropriate default for each tier, not to add a
permanent benchmark framework or live-provider requirement to the ordinary
test suite. Repository tests remain offline and deterministic.
## Implemented Scope
The completed feature includes:
- Weatherreporter-owned embedded profile assets for all three logical IDs;
- Promptkit adapter wiring that supplies those assets as the application
fallback profile source;
- per-prompt default-profile assignments matching the catalog above;
- an exact prompt-version update for the changed definitions;
- preservation of the global profile override;
- same-ID override behavior through both supported external profile-source
forms;
- local-backend and endpoint-only override coverage;
- fail-fast inspection of missing, malformed, or unusable selected profiles;
- offline tests for selection, source precedence, effective model inspection,
batch reuse, and active execution behavior;
- maintained operator examples for overriding `weather-light` locally; and
- updates to the canonical configuration, Promptkit integration, report
registry, operations, troubleshooting, internal adapter, and release
documentation as applicable when implementation lands.
## Non-Goals
The feature does not include:
- automatic discovery, health checking, or benchmarking of local endpoints;
- implicit failover between local and remote profiles;
- retries with a more expensive tier after provider or validation failure;
- per-report profile configuration fields outside prompt defaults;
- profile inheritance, aliases, or field-level merging;
- runtime model selection based on weather severity, token count, or report
content;
- moving Weatherreporter profile policy into Promptkit's built-in catalog;
- exposing Promptkit types outside the adapter boundary; or
- making live provider calls part of the default repository test suite.
## Compatibility And Operational Policy
Existing configurations with a nonblank `promptkit.profile` retain their
all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and
credential configuration fields retain their meanings.
Configurations that rely on the omitted profile setting will intentionally
observe new per-report defaults. This is a user-visible model-selection and
cost change and must be called out in release notes. Operators who require the
old all-report model can preserve it by setting an explicit global profile.
The prompt-version transition does not provide backward compatibility for
historical prompt preparation or execution artifacts. This is consistent with
the accepted ephemeral-state direction; the profile feature does not otherwise
redesign or remove the current workspace layout.
An external same-ID override is an operator-owned compatibility commitment.
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
silently merge an operator file.
## Completion Record
The following conditions are satisfied:
- a tagged Promptkit dependency supports the required fallback layer;
- every operational prompt selects its assigned logical profile at exact
version `1.1.0`;
- all three embedded profiles inspect successfully without an external profile
source;
- configured same-ID definitions override embedded definitions through both
`profile_file` and `profile_dir`;
- an invalid matching external definition fails without fallback;
- `weather-light` can resolve through an endpoint-only or configured-local
override without requiring code or prompt changes;
- global `promptkit.profile` still overrides every report in an invocation;
- active inspection and execution preserve the selected logical profile and
effective model through the project-owned execution contract;
- morning and evening batch preflight deduplicates inspection of shared
effective profile IDs as it does today;
- the default test suite remains offline and deterministic; and
- implemented behavior is documented by its canonical current-state owners.

View File

@@ -0,0 +1,382 @@
# Ephemeral Operational State Roadmap
Status: Accepted feature direction; implementation has not started.
## Purpose
Weatherreporter should treat generated weather reports and their intermediate
artifacts as short-lived operational material rather than a durable audit
history. Forecasts and current conditions change continuously, and the normal
response to an old or failed report is to generate a new report, not to
reconstruct the provenance of the old one.
The application should retain only the bounded state needed to publish the
current report, calculate Recent Changes against the last successfully
published report for the same valid period, and complete the current
invocation safely. Detailed LLM diagnostics should remain an explicit,
operator-controlled exception outside ordinary workspace state.
This roadmap defines the intended state lifecycle, compatibility policy, and
architectural boundaries. A separate implementation plan will define the
ordered work after the roadmap is complete.
## User Intent
The state model should reflect these product expectations:
- weather reports are ephemeral products, not business records;
- old report provenance has no continuing operational value once conditions
and forecasts have changed;
- regenerating is preferable to recovering, replaying, or inspecting an old
generation;
- routine operation should not accumulate unbounded run-addressed artifacts;
- Recent Changes remains useful, but needs only one prior successful snapshot
for the same report and valid period; and
- sensitive prompt and response capture remains opt-in and explicitly managed
by the operator.
## Current State
Each generation currently writes a run-addressed collection containing a
module snapshot, data package, prompt preparation receipt, prompt execution
receipt, raw generated text, validated generated text, render context, managed
report, metadata, and optional notification receipt. Successful and failed
runs accumulate beneath the workspace.
Metadata links the collection and supports lookup by RunID. The CLI can list
historical runs and inspect their metadata, modules, data packages, prior
snapshots, and source provenance. New metadata uses the V2 format while the
reader retains V1 compatibility. Prompt artifacts are validated against
current report and prompt definitions when saved and loaded.
Most of this persistence exists for retrospective inspection and failure
recovery. Dedicated prompt preparation and execution load operations have no
ordinary production consumer. The important exception is module snapshot
state: generation actively loads the most recent compatible snapshot to build
the deterministic Recent Changes input for Daily, Today, and Tomorrow.
## Desired End State
Weatherreporter has three distinct state classes:
| State class | Lifecycle | Purpose |
| --- | --- | --- |
| Invocation workspace | Temporary and unpublished | Hold intermediate values while one report or batch is running. |
| Current published state | Bounded and replaceable | Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation. |
| Secure LLM debug capture | Explicitly enabled and operator-managed | Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture. |
Ordinary generation uses an invocation-scoped temporary directory on the same
filesystem as the managed workspace when atomic publication requires it.
Prompt preparation, prompt execution, raw generated text, validated generated
text, render contexts, data packages, and notification receipts may exist
there while needed, but they are not published as durable historical
artifacts.
A successful report atomically replaces the current published state for its
logical report key and valid period. A failed attempt leaves the last
successfully published report and comparison snapshot unchanged. Ordinary
temporary artifacts are removed after both success and handled failure;
cleanup failure is reported safely but must not replace the primary generation
error.
RunIDs remain useful as in-process correlation identifiers in action results,
logs, provider provenance, and optional debug paths. They no longer identify a
durable collection that Weatherreporter promises to locate or decode later.
## Published Report Policy
The managed Markdown report remains the authoritative upload source during an
invocation. The intended default is to retain only the current managed report
for each logical report key and valid period, replacing it atomically after a
new report has been fully rendered and validated.
An explicit `--out` or `--out-dir` copy remains operator-owned output outside
the managed-state lifecycle. Weatherreporter does not delete, rotate, or
rewrite those copies except when the same explicit destination is selected by
a later invocation.
Distributor continues to receive only a completed managed Markdown report.
Notification success or failure does not create a durable notification
history. A notification failure leaves the newly published report available
and returns a safe error through the current action result.
## Recent Changes State
Recent Changes must be preserved without preserving general report history.
For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible
module snapshot for each logical report key and valid period.
The retained snapshot represents the last successfully published report. A
new invocation reads it before constructing Recent Changes and replaces it
only when the new managed report has been successfully validated, rendered,
and published. A failed generation therefore does not become the baseline for
the next report and cannot hide changes that the user has not yet seen.
Hourly does not currently use the comparison strategy and should not retain a
comparison snapshot solely for symmetry. State whose valid period has ended
and can no longer participate in a supported comparison is eligible for safe
cleanup.
## Temporary Workspace And Failure Semantics
Temporary state must remain beneath a narrowly owned application directory and
use safe path construction, restrictive permissions where content is
sensitive, and atomic writes where practical. Publication must not expose a
partially rendered report or a snapshot that does not correspond to the
published report.
Normal results retain bounded error information and paths only for artifacts
that remain meaningful after the command: a previously or newly published
report, an explicit operator output, or an enabled secure debug capture.
Temporary intermediate paths are not emitted as if they were durable recovery
locations. A failed command is retried by starting a new generation.
Process interruption may leave an uncommitted temporary directory. Such a
directory is never considered published state, is never selected for Recent
Changes, and may be removed by a documented safe cleanup mechanism. Cleanup
must distinguish inactive temporary directories from concurrent active
invocations and must never recursively target the workspace root or an
unresolved configuration path.
## Inspection And Metadata Policy
Run-history discovery and inspection are not part of the desired product
contract. The historical `inspect reports`, `inspect metadata`, `inspect
modules`, `inspect data-package`, `inspect prior`, and `inspect sources`
surfaces are candidates for removal together rather than preservation through
a new storage representation.
Any manifest retained for atomic publication or Recent Changes is current
operational state, not an archival metadata record. It should contain only the
identity, valid period, safe paths, and deterministic snapshot information
needed to validate and use that current state. It does not need to preserve
prompt messages, generated prose intermediates, source provenance, provider
provenance, notification history, or a catalog of prior runs.
The application does not promise cross-version decoding of ordinary workspace
state. A new release may replace or ignore incompatible current-state files,
provided it fails safely, never mistakes stale state for a compatible Recent
Changes baseline, and documents any operator action required during upgrade.
## Prompt Execution And Debugging
Prompt inspection before weather collection and prepared execution remain
runtime safety requirements. They do not require durable preparation or
execution receipts.
The selected logical profile, effective backend and model, validation outcome,
and safe classified error remain available to the active workflow and its CLI
summary where useful. Weatherreporter does not retain them as long-term report
provenance after the invocation completes.
The existing explicit secure debug root remains outside ordinary state and may
retain rendered prompts, schemas, input bodies, generated bodies, and effective
parameters according to its documented contract. Weatherreporter does not
automatically clean that operator-selected location. Credentials must remain
excluded from debug capture.
## Compatibility And Upgrade Policy
This is an intentional breaking change to the workspace and inspection
contracts. Weatherreporter does not need to migrate historical V1 or V2
metadata, prompt receipts, intermediate generated-text artifacts, or managed
reports into the new representation.
Legacy workspace trees must not be silently interpreted as current published
state. They also must not be deleted automatically merely because a new
version starts: an operator may have placed or referenced files there despite
the absence of a continuing application compatibility promise. Release notes
and operations documentation must explain whether legacy data can be removed
manually and identify the exact safe target.
The change should land in a release whose notes clearly identify removed CLI
commands, obsolete paths and schemas, the new bounded state behavior, and any
upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic
version policy may carry this breaking change without inventing a migration
framework.
## Required Architecture Decision Record
The implemented feature must include an Accepted ADR recording the durable
architectural decision to use ephemeral operational state. The ADR is not part
of this roadmap-writing pass and should not be created until implementation is
being prepared.
The ADR should record:
- the mismatch between run-addressed provenance storage and the ephemeral
weather-report lifecycle;
- the decision to retain bounded current report and comparison state rather
than historical runs;
- the distinction between temporary invocation state, published operational
state, explicit output copies, and secure debug capture;
- the removal of historical inspection and backward-compatibility guarantees;
- atomic publication and failed-run behavior;
- the alternatives considered, including retaining the current archive,
adding time-based retention, or keeping a bounded run history; and
- consequences for CLI compatibility, workspace layout, testing, operations,
and future schema changes.
Once accepted, the ADR owns the decision rationale. The architecture policy
owns the resulting current invariant, while focused state, CLI, operations,
and integration documents own the implemented contracts.
## Scope
The completed feature includes:
- an invocation-scoped temporary workspace for intermediate generation state;
- atomic publication of the current managed report and its minimal operational
state;
- a bounded comparison snapshot representing the last successfully published
report for each supported report key and valid period;
- safe cleanup behavior for normal completion, handled failure, and abandoned
temporary workspaces;
- removal of durable prompt preparation, prompt execution, generated-text,
render-context, data-package, notification, and run-metadata history;
- removal of run-history inspection commands and their application/state
contracts;
- removal of V1 metadata compatibility and current-version coupling for
historical prompt artifacts by removing the historical artifact contract;
- preservation of active-command partial status, safe errors, and paths to
genuinely retained published, operator-owned, or debug outputs;
- preservation of explicit output copies, Distributor upload behavior, and
opt-in secure LLM debug capture;
- risk-appropriate offline tests for atomic publication, comparison baselines,
failure isolation, cleanup safety, concurrent invocation safety, and absence
of unbounded state growth;
- an Accepted ADR documenting the architectural decision; and
- updates to canonical architecture, CLI, operations, configuration,
troubleshooting, integration, internal, testing, and release documentation
where their contracts change.
## Non-Goals
This feature does not include:
- a general-purpose cache, database, archival service, or retention engine;
- replaying or resuming interrupted generation;
- migrating legacy artifacts into the new representation;
- retaining a bounded number of historical runs for convenience;
- automatic upload or archival of state to remote storage;
- collecting additional provider telemetry or weather-source provenance;
- changing report content, prompt text, schemas, profile selection, weather
derivation, or batch membership;
- deleting operator-owned `--out`, `--out-dir`, or secure debug files;
- changing Distributor's report-content contract; or
- making live external services part of the default test suite.
## Safety And Testing Policy
The state refactoring must preserve Weatherreporter's existing path-safety and
atomicity expectations while reducing the amount of durable state. Tests
should emphasize observable lifecycle guarantees rather than private file
choreography.
Important risks requiring durable offline coverage include:
- a failed or canceled generation replacing a previously published report or
comparison baseline;
- a partially written report becoming visible as current;
- Recent Changes selecting an incompatible report, valid period, or failed
attempt;
- cleanup deleting published, operator-owned, debug, or concurrently active
files;
- batch partial success corrupting the state of another report;
- notification failure rolling back or obscuring a successfully published
report;
- stale or incompatible current state being treated as valid; and
- repeated successful and failed runs causing unbounded ordinary workspace
growth.
Tests remain deterministic, offline, credential-free, and based on real
temporary filesystems plus narrow external-boundary fakes. Race-enabled tests
are required where publication, cleanup, or concurrent invocation behavior
shares mutable filesystem state.
## Relationship To Domain-Specific Profiles
The domain-specific profile feature can be implemented before this refactor,
but it should not add new historical compatibility or durable-provenance
commitments. Profile inspection, selection, override precedence, and effective
model resolution remain active-workflow behavior and survive the state change.
The domain-profile roadmap and implementation plan should acknowledge that
prompt artifacts from version `1.0.1` need not remain readable after prompts
advance to `1.1.0`. Existing state persistence may remain temporarily while
the profile feature lands, but it should not be expanded or treated as the
target architecture.
## Completion Criteria
The roadmap's target state is achieved when:
- ordinary runs no longer create durable run-addressed artifact collections;
- a successful report atomically replaces only the corresponding current
published state;
- failed and canceled attempts leave the prior published report and Recent
Changes baseline unchanged;
- Daily, Today, and Tomorrow compare against at most one compatible snapshot
from the last successfully published report;
- expired comparison and published state can be removed safely without
touching operator-owned or active files;
- Hourly does not retain an unused comparison snapshot;
- historical inspection commands and V1/V2 archival compatibility code are
removed;
- temporary, published, explicit-output, and debug paths have distinct and
documented ownership and cleanup rules;
- Distributor and active-command summaries continue to receive the completed
report and safe status information they require;
- the default suite proves atomicity, bounded growth, cleanup safety, batch
isolation, and comparison correctness offline;
- an Accepted ADR records the architectural decision and alternatives; and
- canonical current-state documentation describes only the implemented
lifecycle.
## Open Questions
### Lifetime of the current managed report
Recommendation: retain one current managed report per logical report key and
valid period until it is replaced or its valid period expires. This preserves
the current default behavior for invocations without `--out` while bounding
growth.
Alternative: treat the managed report as temporary and retain output only when
the operator supplies `--out` or `--out-dir`. This minimizes state further but
makes a successful default invocation produce no durable report for the user
and complicates Distributor sequencing.
### Historical inspection replacement
Recommendation: remove the run-history inspection commands without adding a
replacement initially. Current command summaries, current managed files, and
opt-in debug capture cover the remaining supported workflows.
Alternative: add a narrow `inspect current REPORT` command backed only by the
current operational manifest. This provides discoverability without history,
but it creates a new public surface and may preserve metadata complexity that
the refactor is intended to remove.
### Abandoned temporary workspace cleanup
Recommendation: use an explicitly owned temporary subtree with per-invocation
ownership markers and a conservative age threshold. Normal cleanup removes the
current invocation synchronously; opportunistic cleanup removes only marked,
inactive directories old enough that they cannot reasonably belong to a live
invocation.
Alternative: perform only synchronous cleanup and document manual removal of
directories left by process termination. This minimizes destructive code and
concurrency risk, but crashed processes can still accumulate unbounded files.
### Legacy workspace cleanup
Recommendation: ignore legacy run-addressed trees and document a precise,
manual one-time cleanup procedure. Do not automatically delete them during
startup or upgrade.
Alternative: add an explicit cleanup command that previews and then removes
recognized legacy artifacts. This is more convenient for large installations
but introduces a destructive command and a legacy-format classifier that must
be maintained and tested.

File diff suppressed because it is too large Load Diff

View File

@@ -0,0 +1,169 @@
# Promptkit Feature Request: Application Fallback Profiles
Status: Implemented upstream in Promptkit v0.5.0.
Promptkit v0.5.0 resolved this request with the public
`WithFallbackProfileFS` engine option and the precedence and error semantics
specified below. This document is retained as the downstream rationale for
the capability.
## Purpose
Promptkit should allow a consuming application to supply an embedded fallback
profile source that sits below operator-configured profiles and above
Promptkit's own built-in profile catalog.
This capability would let an application publish stable, domain-specific
profile IDs with useful defaults while preserving Promptkit's existing
operator-override behavior. The capability must remain application-neutral;
Promptkit should provide the source layer but should not own downstream profile
names, model assignments, or configuration policy.
## Downstream Use Case
Weatherreporter wants to embed profiles such as `weather-light`,
`weather-balanced`, and `weather-deep`. Report prompts would select those
logical profiles instead of naming provider- or model-specific Promptkit
profiles directly.
An installation could then place a profile with the same ID in its configured
profile directory. For example, a local `weather-light` definition could point
to an OpenAI-compatible endpoint on the deployment network. When no operator
definition exists, Weatherreporter's embedded definition would keep the
application usable without additional profile files.
This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want
application-owned execution tiers or workload-specific defaults without
adding domain-specific profiles to Promptkit's general built-in catalog.
## Current Constraint
Promptkit currently resolves matching profile IDs in this order:
1. in-memory profiles supplied through `WithProfiles`;
2. one configured profile file, `fs.FS`, or directory source; and
3. Promptkit's embedded built-in profiles.
These layers do not express the desired application-default relationship:
- `WithProfiles` has higher precedence than the configured source, so it would
prevent an operator file from overriding an application profile with the
same ID.
- `WithProfileFS` can hold embedded application assets, but it occupies the
configured-source layer and therefore replaces rather than sits beneath a
configured profile directory or file.
- adding downstream profile IDs to Promptkit's built-in catalog would make the
library own application-specific policy.
A downstream application could build its own filesystem overlay, but that
would duplicate Promptkit's profile discovery, error, and precedence behavior
at the consumer boundary.
## Requested Capability
Add one optional application fallback profile source to engine construction.
When present, matching profile IDs should resolve in this order:
1. in-memory profiles supplied through `WithProfiles`;
2. the ordinary configured profile source selected through a profile option or
`Config.ProfileDir`;
3. the application fallback profile source; and
4. Promptkit's embedded built-in profiles.
When no application fallback is configured, existing source precedence and
behavior must remain unchanged.
The minimum useful public surface is an `fs.FS`-backed option because consumers
can embed YAML profile assets. A possible API shape is:
```go
promptkit.WithFallbackProfileFS(profileFS, ".")
```
The name is illustrative rather than prescriptive. A companion option for
validated `Profile` values could be added if Promptkit maintainers find it
generally useful, but it is not required for the Weatherreporter use case.
## Required Semantics
- A higher-precedence source falls through only when the requested profile ID
is absent.
- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching
profile is an error and must not silently fall through.
- The fallback source uses the existing strict profile YAML format and profile
validation rules.
- Profile values are selected as a whole. This feature does not merge,
inherit, or partially overlay profile definitions.
- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use
the same profile-source precedence.
- An explicit request profile continues to take precedence over a prompt's
`default_profile`; this request concerns definition lookup after the profile
ID has been selected.
- Repeated fallback-source options should follow Promptkit's documented
same-category option convention, normally with the last value replacing the
earlier value.
- A canceled lookup, invalid fallback asset, or unknown resolved backend should
continue to cross the public facade through Promptkit's existing public error
identities.
- Exact profile inspection must remain side-effect free and must not contact a
model provider.
## Application And Library Boundaries
Promptkit should own:
- the additional repository layer;
- deterministic lookup and fallthrough behavior;
- validation of the supplied source through the existing profile contract;
- consistent use of the layer across inspection and execution; and
- public documentation and tests for the added precedence rule.
The consuming application should continue to own:
- whether it supplies fallback profiles;
- the profile IDs and their domain meaning;
- embedded profile contents and model choices;
- application configuration and override policy;
- report- or workload-to-profile assignment; and
- credential checks and operator-facing errors beyond Promptkit's public
contract.
## Non-Goals
This request does not ask Promptkit to add:
- Weatherreporter-specific profile IDs to its built-in catalog;
- profile inheritance, aliases, or field-level merging;
- automatic endpoint discovery or availability probing;
- provider failover or fallback from a failed selected profile;
- per-request model benchmarking or tier selection;
- application configuration discovery; or
- eager validation of every profile in every source.
## Compatibility
The feature can be additive. Engines that do not configure an application
fallback source should retain their current public behavior and precedence.
Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and
`Config.ProfileDir` should not change meaning.
The application fallback is deliberately lower precedence than every existing
consumer-configured source. This preserves the established expectation that a
custom profile definition can override a packaged default with the same ID.
## Acceptance Criteria
The capability is sufficient for downstream adoption when Promptkit can
demonstrate that:
- a fallback-only profile can be inspected and used for preparation and
execution;
- a configured directory, file, or `fs.FS` profile with the same ID overrides
the fallback profile;
- an absent configured profile falls through to the application fallback;
- an invalid configured match fails instead of falling through;
- an absent application fallback profile continues to resolve from Promptkit's
built-in catalog;
- `WithProfiles` retains highest precedence;
- behavior is identical across inspection, preparation, and execution; and
- omitting the new option preserves existing tests and public contracts.

View File

@@ -128,8 +128,7 @@ It is not a source for deterministic weather facts.
| --- | --- | --- | --- |
| `.GeneratedText.Summary` | `string` | `string` | Required. |
| `.GeneratedText.ForecastDiscussion` | `string` | `[]string` | Required; range over the day-style paragraph slice. |
| `.GeneratedText.PrecipitationTiming` | `string` | `string` | Optional prose used by the precipitation partial when deterministic windows exist. |
| `.GeneratedText.Confidence` | `string` | `string` | Optional validated prose; the current templates do not render it. |
| `.GeneratedText.PrecipitationTiming` | `string` | `string` | Required field; an empty string represents no supported prose. The precipitation partial uses nonempty prose only when deterministic windows exist. |
The JSON schema rejects unknown properties and defines the required fields, but
the schema body and validation behavior are documented in [Generated Text

View File

@@ -1,7 +1,10 @@
# Troubleshooting
Keep failed workspace artifacts in place. When a RunID is available, start
with `weatherreporter inspect metadata RUN_ID` and use the paths in its result.
Start with the command's classified error. When content-rich prompt diagnostics
are needed, enable a new run with `--llm-debug-dir` and handle the resulting
secure capture as sensitive. Current-version workspace receipts can provide
additional context when present, but are transitional state rather than a
long-term troubleshooting interface.
## Prompt inspection or credentials fail before collection
@@ -11,13 +14,37 @@ the configured `promptkit` profile or profile source, confirm the exact
Promptkit asset is available, and supply any reported environment credential.
Do not add provider keys to YAML. See [configuration](config.md).
## Local profile override is malformed or selects an unexpected model
`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile
definitions. A same-ID definition replaces the embedded profile, and a malformed
matching definition fails before collection instead of falling back. Validate
the selected profile's YAML, ID, backend or endpoint, and model. If the model
is unexpected, first check the global `promptkit.profile` selection and then
look for a same-ID definition in the configured file or directory.
Current-version preparation and execution receipts may retain the selected
profile ID and effective backend/model, but not an endpoint or credential.
Use them only as supplemental context after the active command error or an
explicit secure debug capture. See the maintained
[local `weather-light` profile example](../examples/weather-light-local-profile.yml).
## Local model endpoint is unavailable
An endpoint-only `weather-light` override can pass preflight and still fail
during provider preparation or execution when the local server is unavailable
or does not accept the configured model. Start the local server, correct the
endpoint or model in the profile, and run the command again. Weatherreporter
does not probe endpoints or automatically use a remote profile instead.
## Preparation, capacity, or execution fails
A preparation failure occurs before provider work; an execution failure occurs
after preparation. Both leave safe provenance and metadata when reached. A
capacity error for one batch report does not retry that report or prevent later
independent reports. Inspect the preparation or execution path, correct the
profile/backend condition, and create a new run. See [operations](operations.md).
after preparation. A capacity error for one batch report does not retry that
report or prevent later independent reports. Correct the profile or backend
condition identified by the bounded command error, then create a new run.
Use explicit secure debug capture only when additional content-rich diagnostics
are necessary. See [operations](operations.md).
## Generated text fails validation

View File

@@ -0,0 +1,4 @@
id: weather-light
endpoint: http://127.0.0.1:11434/v1
model: weather-local
timeout_seconds: 180

2
go.mod
View File

@@ -6,7 +6,7 @@ require gopkg.in/yaml.v3 v3.0.1
require (
gitea.maximumdirect.net/eric/distributor v0.5.0
gitea.maximumdirect.net/eric/promptkit v0.4.0
gitea.maximumdirect.net/eric/promptkit v0.5.0
)
require (

4
go.sum
View File

@@ -1,7 +1,7 @@
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
gitea.maximumdirect.net/eric/promptkit v0.4.0 h1:WHRQEt3BVBAR7hQePBaGtNXpzrs59mlr/42nQzwgOz4=
gitea.maximumdirect.net/eric/promptkit v0.4.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=

View File

@@ -45,6 +45,7 @@ func newAdapter(config Config, additionalOptions ...promptkit.Option) (*Adapter,
options := []promptkit.Option{
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
}
if config.ProfileFile != "" {
options = append(options, promptkit.WithProfileFile(config.ProfileFile))

View File

@@ -68,11 +68,11 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
func TestInspectPromptAndProfile(t *testing.T) {
adapter := newTestAdapter(t, &fakeClient{})
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.0.0")
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.1.0")
if err != nil {
t.Fatalf("InspectPrompt() error = %v", err)
}
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.0.0" || inspection.DefaultProfileID != "gemini-flash-latest" {
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != "weather-balanced" {
t.Fatalf("inspection = %#v", inspection)
}
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
@@ -102,6 +102,107 @@ func TestInspectPromptAndProfile(t *testing.T) {
}
}
func TestEmbeddedProfilesAreAvailableToProductionAndTestAdapters(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
for _, want := range []struct {
id string
backend string
model string
}{
{"weather-light", "openrouter", "deepseek/deepseek-v4-flash"},
{"weather-balanced", "openrouter", "~google/gemini-flash-latest"},
{"weather-deep", "openrouter", "~anthropic/claude-sonnet-latest"},
} {
t.Run(want.id, func(t *testing.T) {
assertProfile(t, adapter, want.id, want.backend, want.model)
})
}
testAdapter, err := newAdapterForTest(Config{}, &fakeClient{})
if err != nil {
t.Fatalf("newAdapterForTest() error = %v", err)
}
assertProfile(t, testAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
}
func TestConfiguredProfilesOverrideEmbeddedFallbacks(t *testing.T) {
file := writeProfileFile(t, `id: weather-light
endpoint: https://local-file.example/v1
model: file-light
`)
fileAdapter, err := New(Config{ProfileFile: file})
if err != nil {
t.Fatalf("New(profile file) error = %v", err)
}
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
directory := testProfileDirectory(t, `id: weather-light
backend: local
model: directory-light
`)
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
if err != nil {
t.Fatalf("New(profile directory) error = %v", err)
}
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
}
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
adapter, err := New(Config{ProfileFile: filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml")})
if err != nil {
t.Fatalf("New() error = %v", err)
}
assertProfile(t, adapter, "weather-light", "", "weather-local")
}
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile
backend: openrouter
model: other-model
`)})
if err != nil {
t.Fatalf("New(absent profile) error = %v", err)
}
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light
backend: openrouter
`)})
if err != nil {
t.Fatalf("New(malformed profile) error = %v", err)
}
if _, err := malformedAdapter.InspectProfile(context.Background(), "weather-light"); err == nil {
t.Fatal("InspectProfile() error = nil, want malformed configured profile error")
}
}
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
adapter, err := New(Config{})
if err != nil {
t.Fatalf("New() error = %v", err)
}
builtin, err := adapter.InspectProfile(context.Background(), "gemini-flash-latest")
if err != nil {
t.Fatalf("InspectProfile(builtin) error = %v", err)
}
if builtin.ProfileID != "gemini-flash-latest" || builtin.BackendID != "openrouter" || builtin.ModelName == "" {
t.Fatalf("builtin profile = %#v", builtin)
}
explicit, err := newAdapter(Config{}, promptkit.WithProfiles(promptkit.Profile{
ID: "weather-light",
Endpoint: "https://explicit.example/v1",
Model: "explicit-light",
}))
if err != nil {
t.Fatalf("newAdapter(explicit profile) error = %v", err)
}
assertProfile(t, explicit, "weather-light", "", "explicit-light")
}
func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
client := &fakeClient{response: validResponse()}
adapter := newTestAdapter(t, client)
@@ -141,6 +242,44 @@ func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
}
}
func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
t.Setenv("OPENROUTER_API_KEY", "test-openrouter-key")
client := &fakeClient{response: hourlyValidResponse()}
adapter, err := newAdapter(Config{}, promptkit.WithLLMClient(client))
if err != nil {
t.Fatalf("newAdapter() error = %v", err)
}
request := promptexec.ExecuteRequest{
PromptID: "weather.hourly_generated_text",
PromptVersion: "1.1.0",
ProfileID: "weather-light",
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
DataPackagePath: "data-packages/hourly/data_package.yaml",
}
var preparation promptexec.Preparation
prepared := false
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
if client.callCount() != 0 {
t.Fatal("provider was called before preparation completed")
}
preparation = value
prepared = true
return nil
})
if err != nil {
t.Fatalf("Execute() error = %v", err)
}
if !prepared || preparation.ProfileID != "weather-light" || preparation.BackendID != "openrouter" || preparation.ModelName != "deepseek/deepseek-v4-flash" {
t.Fatalf("preparation = %#v", preparation)
}
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "openrouter" || result.ModelName != "deepseek/deepseek-v4-flash" || result.Validation.Status != promptexec.ValidationPassed {
t.Fatalf("execution = %#v", result)
}
if client.callCount() != 1 || client.request().Target.Model != "deepseek/deepseek-v4-flash" {
t.Fatalf("provider calls/request = %d/%#v", client.callCount(), client.request())
}
}
func TestExecuteUsesExactInlineDataPackageProvenance(t *testing.T) {
client := &fakeClient{response: validResponse()}
reader := &recordingReader{}
@@ -339,6 +478,17 @@ func newTestAdapter(t *testing.T, client promptkit.LLMClient) *Adapter {
return newTestAdapterWithOptions(t, client)
}
func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, model string) {
t.Helper()
profile, err := adapter.InspectProfile(context.Background(), id)
if err != nil {
t.Fatalf("InspectProfile(%q) error = %v", id, err)
}
if profile.ProfileID != id || profile.BackendID != backend || profile.ModelName != model {
t.Fatalf("profile = %#v, want %q with backend/model %q/%q", profile, id, backend, model)
}
}
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
t.Helper()
profiles := testProfileDirectory(t, `id: test-profile
@@ -366,10 +516,19 @@ func testProfileDirectory(t *testing.T, profile string) string {
return profiles
}
func writeProfileFile(t *testing.T, profile string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "profile.yml")
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
return path
}
func testExecuteRequest() promptexec.ExecuteRequest {
return promptexec.ExecuteRequest{
PromptID: "weather.daily_generated_text",
PromptVersion: "1.0.0",
PromptVersion: "1.1.0",
ProfileID: "test-profile",
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
DataPackagePath: "data-packages/daily/data_package.yaml",
@@ -378,7 +537,14 @@ func testExecuteRequest() promptexec.ExecuteRequest {
func validResponse() *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{
Content: `{"summary":"A quiet day is expected.","forecast_discussion":["High pressure keeps conditions settled."],"confidence":"High."}`,
Content: `{"summary":"A quiet day is expected.","forecast_discussion":["High pressure keeps conditions settled."],"precipitation_timing":""}`,
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
}
}
func hourlyValidResponse() *promptkit.GenerateResponse {
return &promptkit.GenerateResponse{
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
}
}

View File

@@ -33,10 +33,10 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
t.Fatalf("batchInspectionCandidates() error = %v", err)
}
executor := &inspectionExecutor{profiles: map[string]promptexec.ProfileInspection{
"default-profile": {ProfileID: "default-profile", BackendID: "local", ModelName: "model"},
"weather-balanced": {ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
}, prompts: map[string]promptexec.PromptInspection{}}
for _, candidate := range candidates {
executor.prompts[candidate.Definition.PromptID] = validPromptInspection(candidate.Definition)
executor.prompts[candidate.Definition.PromptID] = logicalPromptInspection(candidate.Definition)
}
collector := collectorFunc(func(context.Context, collect.Request) (*collect.Result, error) {
return nil, errors.New("collection reached")
@@ -47,7 +47,7 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
if err == nil || err.Error() != "collection reached" {
t.Fatalf("RunBatchDetailed() error = %v, want collection error", err)
}
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 {
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
t.Fatalf("inspection calls = prompts %#v profiles %#v", executor.promptRequests, executor.profileRequests)
}
})

View File

@@ -44,7 +44,7 @@ func (e *assembledBatchExecutor) InspectPrompt(_ context.Context, id, version st
if !ok || definition.PromptVersion != version {
return promptexec.PromptInspection{}, errors.New("unexpected prompt inspection")
}
return validPromptInspection(definition), nil
return logicalPromptInspection(definition), nil
}
func (e *assembledBatchExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
@@ -151,8 +151,8 @@ func TestRunBatchDetailedExecutesRetainedReportsSequentially(t *testing.T) {
if len(executor.executeRequests) != len(test.wantIDs) || executor.maxActive != 1 {
t.Fatalf("executor calls/max active = %d/%d, want %d/1", len(executor.executeRequests), executor.maxActive, len(test.wantIDs))
}
if len(executor.profileRequests) != 1 {
t.Fatalf("profile inspections = %#v, want one shared profile inspection", executor.profileRequests)
if len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
t.Fatalf("profile inspections = %#v, want one shared weather-balanced inspection", executor.profileRequests)
}
for index, item := range result.Reports {
if item.ReportID != test.wantIDs[index] || item.Status != "succeeded" {

View File

@@ -119,7 +119,7 @@ func (e artifactPathExecutor) Execute(_ context.Context, req promptexec.ExecuteR
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID,
BackendID: "test", ModelName: "test-model", GeneratedHash: "generated-hash",
StartedAt: now, EndedAt: now, DataPackagePath: req.DataPackagePath,
RawOutput: []byte(`{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`),
RawOutput: []byte(`{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."],"precipitation_timing":"Rain is most likely during the afternoon."}`),
Validation: promptexec.NewValidation(validation, "json_schema", "daily.generated_text.schema.json", nil),
}, nil
}

View File

@@ -189,3 +189,13 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
}
}
func logicalPromptInspection(definition report.Definition) promptexec.PromptInspection {
inspection := validPromptInspection(definition)
if definition.ID == report.Hourly {
inspection.DefaultProfileID = "weather-light"
} else {
inspection.DefaultProfileID = "weather-balanced"
}
return inspection
}

View File

@@ -0,0 +1,73 @@
package app_test
import (
"context"
"os"
"path/filepath"
"testing"
"time"
promptkitadapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/promptkit"
"gitea.maximumdirect.net/eric/weatherreporter/internal/app"
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
)
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
lookupEnv := func(string) (string, bool) { return "test-key", true }
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
t.Helper()
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
Resolved: resolvedPromptProfile(t, id),
Executor: adapter,
Promptkit: config.PromptkitConfig{Profile: profile},
LookupEnv: lookupEnv,
})
if err != nil {
t.Fatalf("InspectPromptExecution() error = %v", err)
}
if result.ProfileID != wantID || result.BackendID != wantBackend || result.ModelName != wantModel {
t.Fatalf("inspection = %#v, want profile/backend/model %q/%q/%q", result, wantID, wantBackend, wantModel)
}
}
embedded, err := promptkitadapter.New(promptkitadapter.Config{})
if err != nil {
t.Fatalf("New(embedded) error = %v", err)
}
inspect(t, embedded, report.Hourly, "", "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
inspect(t, embedded, report.Daily, "", "weather-balanced", "openrouter", "~google/gemini-flash-latest")
inspect(t, embedded, report.Daily, "weather-deep", "weather-deep", "openrouter", "~anthropic/claude-sonnet-latest")
override, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: writeProfileFile(t, `id: weather-light
endpoint: https://local.example/v1
model: local-weather
`)})
if err != nil {
t.Fatalf("New(override) error = %v", err)
}
inspect(t, override, report.Hourly, "", "weather-light", "", "local-weather")
}
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
t.Helper()
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)
request := report.ResolveRequest{Now: now, Location: time.UTC}
if id == report.Daily {
request.Date = now
}
resolved, err := report.DefaultRegistry().Resolve(id, request)
if err != nil {
t.Fatalf("Resolve(%q) error = %v", id, err)
}
return resolved
}
func writeProfileFile(t *testing.T, profile string) string {
t.Helper()
path := filepath.Join(t.TempDir(), "profile.yml")
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
t.Fatalf("write profile: %v", err)
}
return path
}

View File

@@ -37,8 +37,10 @@ func (c *workflowCollector) Run(context.Context, collect.Request) (*collect.Resu
type workflowExecutor struct {
definition report.Definition
raw []byte
prompt promptexec.PromptInspection
inspectionErr error
profile promptexec.ProfileInspection
profileErr error
beforePreparationErr error
afterCallbackErr error
afterPreparationErr error
@@ -49,6 +51,8 @@ type workflowExecutor struct {
beforeProvider func()
preparationDebug *promptexec.PreparationDebug
executionDebug *promptexec.ExecutionDebug
preparation *promptexec.Preparation
execution *promptexec.Execution
}
func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string) (promptexec.PromptInspection, error) {
@@ -58,10 +62,16 @@ func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string)
if id != e.definition.PromptID || version != e.definition.PromptVersion {
return promptexec.PromptInspection{}, errors.New("unexpected prompt identity")
}
if e.prompt.PromptID != "" {
return e.prompt, nil
}
return validPromptInspection(e.definition), nil
}
func (e *workflowExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
if e.profileErr != nil {
return promptexec.ProfileInspection{}, e.profileErr
}
profile := e.profile
if profile.ProfileID == "" {
profile = promptexec.ProfileInspection{ProfileID: id, BackendID: "fixture", ModelName: "fixture-model"}
@@ -76,11 +86,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
return nil, e.beforePreparationErr
}
stamp := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
profile := e.profile
if profile.ProfileID == "" {
profile = promptexec.ProfileInspection{ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model"}
}
preparation := promptexec.Preparation{
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: "prompt-hash",
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture",
ModelName: "fixture-model", DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: profile.BackendID,
ModelName: profile.ModelName, DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
}
e.preparation = &preparation
if err := callback(preparation, e.preparationDebug); err != nil {
return nil, err
}
@@ -98,14 +113,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
if validation == "" {
validation = promptexec.ValidationPassed
}
return &promptexec.Execution{
execution := &promptexec.Execution{
RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion,
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID,
BackendID: "fixture", ModelName: "fixture-model", GeneratedHash: "generated-hash",
BackendID: profile.BackendID, ModelName: profile.ModelName, GeneratedHash: "generated-hash",
StartedAt: stamp, EndedAt: stamp, DataPackagePath: req.DataPackagePath, RawOutput: e.raw,
Debug: e.executionDebug,
Validation: promptexec.NewValidation(validation, "json_schema", e.definition.GeneratedTextSchemaID+".generated_text.schema.json", nil),
}, nil
}
e.execution = execution
return execution, nil
}
type workflowNotifier struct {
@@ -208,6 +225,57 @@ func TestGenerateDetailedCompletesRetainedReportWorkflows(t *testing.T) {
}
}
func TestGenerateDetailedPreservesSelectedProfileThroughExecution(t *testing.T) {
tests := []struct {
name string
kind ReportKind
id report.ID
raw string
override string
profile promptexec.ProfileInspection
}{
{
name: "hourly default", kind: ReportHourly, id: report.Hourly, raw: validHourlyWorkflowJSON(),
profile: promptexec.ProfileInspection{ProfileID: "weather-light", BackendID: "openrouter", ModelName: "deepseek/deepseek-v4-flash"},
},
{
name: "daily default", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(),
profile: promptexec.ProfileInspection{ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
},
{
name: "global override", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(), override: "operator-profile",
profile: promptexec.ProfileInspection{ProfileID: "operator-profile", BackendID: "local", ModelName: "local-weather-model"},
},
}
for _, test := range tests {
t.Run(test.name, func(t *testing.T) {
cfg := workflowConfig(t)
cfg.Promptkit.Profile = test.override
definition := report.DefaultRegistry().MustLookup(test.id)
executor := &workflowExecutor{
definition: definition, prompt: logicalPromptInspection(definition), profile: test.profile, raw: []byte(test.raw),
}
bundle := workflowBundle(t)
_, err := GenerateDetailed(context.Background(), GenerateRequest{
Config: cfg, Report: test.kind, Date: workflowTime("2026-05-29T12:00:00-05:00"), Now: workflowTime("2026-05-29T08:30:00-05:00"),
Collector: &workflowCollector{result: &collect.Result{Bundle: &bundle}}, Executor: executor, Notifier: &workflowNotifier{},
})
if err != nil {
t.Fatalf("GenerateDetailed() error = %v", err)
}
if executor.request.ProfileID != test.profile.ProfileID {
t.Fatalf("execution profile = %q, want %q", executor.request.ProfileID, test.profile.ProfileID)
}
if executor.preparation == nil || executor.preparation.ProfileID != test.profile.ProfileID || executor.preparation.BackendID != test.profile.BackendID || executor.preparation.ModelName != test.profile.ModelName {
t.Fatalf("prepared profile = %#v, want %q/%q/%q", executor.preparation, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
}
if executor.execution == nil || executor.execution.ProfileID != test.profile.ProfileID || executor.execution.BackendID != test.profile.BackendID || executor.execution.ModelName != test.profile.ModelName {
t.Fatalf("executed profile = %#v, want %q/%q/%q", executor.execution, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
}
})
}
}
type preparationFailingStore struct {
state.Store
}
@@ -299,6 +367,11 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
wantCategory promptexec.ErrorCategory
}{
{name: "inspection", configure: func(e *workflowExecutor) { e.inspectionErr = errors.New("inspection unavailable") }, wantCategory: promptexec.InvalidConfiguration},
{name: "unknown profile", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unknown selected profile") }, wantCategory: promptexec.InvalidConfiguration},
{name: "malformed profile", configure: func(e *workflowExecutor) {
e.profileErr = errors.New("malformed profile at https://operator.example/v1 api_key=secret")
}, wantCategory: promptexec.InvalidConfiguration},
{name: "unusable backend", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unsupported backend") }, wantCategory: promptexec.InvalidConfiguration},
{name: "credential", configure: func(e *workflowExecutor) {
e.profile = promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true}
}, wantCategory: promptexec.MissingCredential},
@@ -317,6 +390,9 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
if err == nil || result != nil || promptexec.CategoryOf(err) != test.wantCategory || collector.calls != 0 || executor.executeCalls != 0 {
t.Fatalf("result/error/category/collect/execute = %#v/%v/%q/%d/%d", result, err, promptexec.CategoryOf(err), collector.calls, executor.executeCalls)
}
if strings.Contains(err.Error(), "operator.example") || strings.Contains(err.Error(), "secret") {
t.Fatalf("error leaks profile details: %v", err)
}
entries, readErr := os.ReadDir(cfg.Workspace.Root)
if readErr != nil || len(entries) != 0 {
t.Fatalf("workspace entries/error = %#v/%v, want no writes before collection", entries, readErr)
@@ -643,7 +719,7 @@ func workflowTime(value string) time.Time {
}
func validHourlyWorkflowJSON() string {
return `{"summary":"Storm chances increase through late morning.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"A cold front is moving into the region.","confidence":"Medium"}`
return `{"summary":"Storm chances increase through late morning.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"A cold front is moving into the region."}`
}
func validTomorrowWorkflowJSON() string {
@@ -655,5 +731,5 @@ func validTodayWorkflowJSON() string {
}
func validDailyWorkflowJSON() string {
return `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`
return `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon."}`
}

View File

@@ -147,7 +147,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
}
hourly, normalized, err := hourlyHandler.Validate([]byte(`{
"summary": " Storm chances increase. ",
"forecast_discussion": " A front will keep the region unsettled. "
"forecast_discussion": " A front will keep the region unsettled. ",
"precipitation_timing": ""
}`))
if err != nil {
t.Fatalf("Validate(hourly) error = %v", err)
@@ -165,7 +166,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
}
tomorrow, normalized, err := tomorrowHandler.Validate([]byte(`{
"summary": " Storms become more likely tomorrow. ",
"forecast_discussion": [" A front will keep showers in the forecast. ", ""]
"forecast_discussion": [" A front will keep showers in the forecast. ", ""],
"precipitation_timing": ""
}`))
if err != nil {
t.Fatalf("Validate(tomorrow) error = %v", err)
@@ -187,7 +189,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
}
today, normalized, err := todayHandler.Validate([]byte(`{
"summary": " Showers are likely today. ",
"forecast_discussion": [" A front will keep rain chances elevated. ", ""]
"forecast_discussion": [" A front will keep rain chances elevated. ", ""],
"precipitation_timing": ""
}`))
if err != nil {
t.Fatalf("Validate(today) error = %v", err)
@@ -209,7 +212,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
}
daily, normalized, err := dailyHandler.Validate([]byte(`{
"summary": " Showers are possible during the selected day. ",
"forecast_discussion": [" A front will keep rain chances in the forecast. ", ""]
"forecast_discussion": [" A front will keep rain chances in the forecast. ", ""],
"precipitation_timing": ""
}`))
if err != nil {
t.Fatalf("Validate(daily) error = %v", err)

View File

@@ -3,8 +3,7 @@ package generatedtext
type Daily struct {
Summary string `json:"summary"`
ForecastDiscussion []string `json:"forecast_discussion"`
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
Confidence string `json:"confidence,omitempty"`
PrecipitationTiming string `json:"precipitation_timing"`
}
func ValidateDaily(data []byte) (Daily, []byte, error) {
@@ -16,7 +15,6 @@ func (d *Daily) dayStyleFields() dayStyleFields {
Summary: d.Summary,
ForecastDiscussion: d.ForecastDiscussion,
PrecipitationTiming: d.PrecipitationTiming,
Confidence: d.Confidence,
}
}
@@ -24,5 +22,4 @@ func (d *Daily) setDayStyleFields(fields dayStyleFields) {
d.Summary = fields.Summary
d.ForecastDiscussion = fields.ForecastDiscussion
d.PrecipitationTiming = fields.PrecipitationTiming
d.Confidence = fields.Confidence
}

View File

@@ -13,8 +13,7 @@ func TestValidateDailyNormalizesJSON(t *testing.T) {
"",
" Temperatures stay seasonable by afternoon. "
],
"precipitation_timing": " Rain is most likely during the afternoon. ",
"confidence": " Medium "
"precipitation_timing": " Rain is most likely during the afternoon. "
}`))
if err != nil {
t.Fatalf("ValidateDaily() error = %v", err)
@@ -28,23 +27,22 @@ func TestValidateDailyNormalizesJSON(t *testing.T) {
if value.PrecipitationTiming != "Rain is most likely during the afternoon." {
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
}
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon."}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}
}
func TestValidateDailyOmitsEmptyOptionalFields(t *testing.T) {
func TestValidateDailyPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
_, normalized, err := ValidateDaily([]byte(`{
"summary": "Showers are possible during the selected day.",
"forecast_discussion": ["A front will keep rain chances in the forecast."],
"precipitation_timing": " ",
"confidence": " "
"precipitation_timing": " "
}`))
if err != nil {
t.Fatalf("ValidateDaily() error = %v", err)
}
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."]}`
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."],"precipitation_timing":""}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}

View File

@@ -9,7 +9,6 @@ type dayStyleFields struct {
Summary string
ForecastDiscussion []string
PrecipitationTiming string
Confidence string
}
type dayStyleGeneratedText interface {
@@ -31,7 +30,6 @@ func validateDayStyleGeneratedText[T any, PT interface {
fields := pointer.dayStyleFields()
fields.Summary = strings.TrimSpace(fields.Summary)
fields.PrecipitationTiming = strings.TrimSpace(fields.PrecipitationTiming)
fields.Confidence = strings.TrimSpace(fields.Confidence)
fields.ForecastDiscussion = trimNonEmpty(fields.ForecastDiscussion)
if fields.Summary == "" {
var zero T
@@ -41,6 +39,10 @@ func validateDayStyleGeneratedText[T any, PT interface {
var zero T
return zero, nil, fmt.Errorf("%s generated text forecast discussion is required", name)
}
if err := requireGeneratedTextStringField(data, name, "precipitation_timing"); err != nil {
var zero T
return zero, nil, err
}
pointer.setDayStyleFields(fields)
normalized, err := normalizeGeneratedText(value, name)

View File

@@ -60,8 +60,7 @@ func TestValidateDayStyleGeneratedTextSharedBehavior(t *testing.T) {
"",
" Second paragraph. "
],
"precipitation_timing": " Afternoon. ",
"confidence": " Medium "
"precipitation_timing": " Afternoon. "
}`))
if err != nil {
t.Fatalf("validate() error = %v", err)
@@ -76,31 +75,35 @@ func TestValidateDayStyleGeneratedTextSharedBehavior(t *testing.T) {
if fields.PrecipitationTiming != "Afternoon." {
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", fields.PrecipitationTiming)
}
if fields.Confidence != "Medium" {
t.Fatalf("Confidence = %q, want trimmed confidence", fields.Confidence)
}
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph.","Second paragraph."],"precipitation_timing":"Afternoon.","confidence":"Medium"}`
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph.","Second paragraph."],"precipitation_timing":"Afternoon."}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}
})
t.Run("omits empty optional fields", func(t *testing.T) {
t.Run("preserves required empty precipitation timing", func(t *testing.T) {
_, normalized, err := report.validate([]byte(`{
"summary": "Shared summary.",
"forecast_discussion": ["First paragraph."],
"precipitation_timing": " ",
"confidence": " "
"precipitation_timing": " "
}`))
if err != nil {
t.Fatalf("validate() error = %v", err)
}
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph."]}`
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph."],"precipitation_timing":""}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}
})
t.Run("requires precipitation timing field", func(t *testing.T) {
_, _, err := report.validate([]byte(`{"summary":"Shared summary.","forecast_discussion":["First paragraph."]}`))
want := fmt.Sprintf("%s generated text precipitation timing is required", report.name)
if err == nil || err.Error() != want {
t.Fatalf("validate() error = %v, want %q", err, want)
}
})
t.Run("rejects unknown fields", func(t *testing.T) {
_, _, err := report.validate([]byte(`{"summary":"Shared summary.","forecast_discussion":["First paragraph."],"extra":"value"}`))
if err == nil {
@@ -110,6 +113,13 @@ func TestValidateDayStyleGeneratedTextSharedBehavior(t *testing.T) {
t.Fatalf("validate() error = %v, want unknown field error", err)
}
})
t.Run("rejects retired confidence field", func(t *testing.T) {
_, _, err := report.validate([]byte(`{"summary":"Shared summary.","forecast_discussion":["First paragraph."],"precipitation_timing":"","confidence":"Medium"}`))
if err == nil || !strings.Contains(err.Error(), `unknown field "confidence"`) {
t.Fatalf("validate() error = %v, want retired confidence field rejection", err)
}
})
})
}
}

View File

@@ -9,8 +9,7 @@ import (
type Hourly struct {
Summary string `json:"summary"`
ForecastDiscussion string `json:"forecast_discussion"`
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
Confidence string `json:"confidence,omitempty"`
PrecipitationTiming string `json:"precipitation_timing"`
}
func ValidateHourly(data []byte) (Hourly, []byte, error) {
@@ -22,13 +21,15 @@ func ValidateHourly(data []byte) (Hourly, []byte, error) {
value.Summary = strings.TrimSpace(value.Summary)
value.ForecastDiscussion = strings.TrimSpace(value.ForecastDiscussion)
value.PrecipitationTiming = strings.TrimSpace(value.PrecipitationTiming)
value.Confidence = strings.TrimSpace(value.Confidence)
if value.Summary == "" {
return Hourly{}, nil, fmt.Errorf("hourly generated text summary is required")
}
if value.ForecastDiscussion == "" {
return Hourly{}, nil, fmt.Errorf("hourly generated text forecast discussion is required")
}
if err := requireGeneratedTextStringField(data, "hourly", "precipitation_timing"); err != nil {
return Hourly{}, nil, err
}
normalized, err := normalizeGeneratedText(value, "hourly")
if err != nil {

View File

@@ -9,8 +9,7 @@ func TestValidateHourlyNormalizesJSON(t *testing.T) {
value, normalized, err := ValidateHourly([]byte(`{
"summary": " Storm chances increase. ",
"forecast_discussion": " A front will keep the region unsettled. ",
"precipitation_timing": " Showers are most likely early this afternoon. ",
"confidence": " Medium "
"precipitation_timing": " Showers are most likely early this afternoon. "
}`))
if err != nil {
t.Fatalf("ValidateHourly() error = %v", err)
@@ -24,23 +23,22 @@ func TestValidateHourlyNormalizesJSON(t *testing.T) {
if value.PrecipitationTiming != "Showers are most likely early this afternoon." {
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
}
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"Showers are most likely early this afternoon.","confidence":"Medium"}`
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"Showers are most likely early this afternoon."}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}
}
func TestValidateHourlyOmitsEmptyConfidence(t *testing.T) {
func TestValidateHourlyPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
_, normalized, err := ValidateHourly([]byte(`{
"summary": "Storm chances increase.",
"forecast_discussion": "A front will keep the region unsettled.",
"precipitation_timing": " ",
"confidence": " "
"precipitation_timing": " "
}`))
if err != nil {
t.Fatalf("ValidateHourly() error = %v", err)
}
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled."}`
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":""}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}
@@ -72,6 +70,21 @@ func TestValidateHourlyRejectsInvalidInput(t *testing.T) {
in: `{"summary":"Storm chances increase.","forecast_discussion":" "}`,
want: "forecast discussion is required",
},
{
name: "missing precipitation timing",
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled."}`,
want: "precipitation timing is required",
},
{
name: "null precipitation timing",
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":null}`,
want: "precipitation timing must be a string",
},
{
name: "retired confidence field rejected",
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"","confidence":"Medium"}`,
want: `unknown field "confidence"`,
},
{
name: "old timing field rejected",
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","timing":"Late morning."}`,

View File

@@ -5,6 +5,7 @@ import (
"encoding/json"
"fmt"
"io"
"strings"
)
func decodeGeneratedText[T any](data []byte, name string) (T, error) {
@@ -24,6 +25,21 @@ func decodeGeneratedText[T any](data []byte, name string) (T, error) {
return value, fmt.Errorf("decode %s generated text: multiple JSON values", name)
}
func requireGeneratedTextStringField(data []byte, name, field string) error {
var fields map[string]json.RawMessage
if err := json.Unmarshal(data, &fields); err != nil {
return fmt.Errorf("decode %s generated text: %w", name, err)
}
raw, ok := fields[field]
if !ok {
return fmt.Errorf("%s generated text %s is required", name, strings.ReplaceAll(field, "_", " "))
}
if bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
return fmt.Errorf("%s generated text %s must be a string", name, strings.ReplaceAll(field, "_", " "))
}
return nil
}
func normalizeGeneratedText[T any](value T, name string) ([]byte, error) {
normalized, err := json.Marshal(value)
if err != nil {

View File

@@ -21,7 +21,6 @@ func TestBuildHourlyRenderContext(t *testing.T) {
Summary: "Storm chances increase through late morning.",
ForecastDiscussion: "A front will keep the region unsettled.",
PrecipitationTiming: "A cold front is moving into the region.",
Confidence: "Medium confidence in timing.",
}
collected := testCollected()
derived := testDerived()

View File

@@ -3,8 +3,7 @@ package generatedtext
type Today struct {
Summary string `json:"summary"`
ForecastDiscussion []string `json:"forecast_discussion"`
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
Confidence string `json:"confidence,omitempty"`
PrecipitationTiming string `json:"precipitation_timing"`
}
func ValidateToday(data []byte) (Today, []byte, error) {
@@ -16,7 +15,6 @@ func (t *Today) dayStyleFields() dayStyleFields {
Summary: t.Summary,
ForecastDiscussion: t.ForecastDiscussion,
PrecipitationTiming: t.PrecipitationTiming,
Confidence: t.Confidence,
}
}
@@ -24,5 +22,4 @@ func (t *Today) setDayStyleFields(fields dayStyleFields) {
t.Summary = fields.Summary
t.ForecastDiscussion = fields.ForecastDiscussion
t.PrecipitationTiming = fields.PrecipitationTiming
t.Confidence = fields.Confidence
}

View File

@@ -13,8 +13,7 @@ func TestValidateTodayNormalizesJSON(t *testing.T) {
"",
" Temperatures stay mild through the afternoon. "
],
"precipitation_timing": " Rain is most likely during the afternoon. ",
"confidence": " Medium "
"precipitation_timing": " Rain is most likely during the afternoon. "
}`))
if err != nil {
t.Fatalf("ValidateToday() error = %v", err)
@@ -28,23 +27,22 @@ func TestValidateTodayNormalizesJSON(t *testing.T) {
if value.PrecipitationTiming != "Rain is most likely during the afternoon." {
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
}
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated.","Temperatures stay mild through the afternoon."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated.","Temperatures stay mild through the afternoon."],"precipitation_timing":"Rain is most likely during the afternoon."}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}
}
func TestValidateTodayOmitsEmptyOptionalFields(t *testing.T) {
func TestValidateTodayPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
_, normalized, err := ValidateToday([]byte(`{
"summary": "Showers are likely today.",
"forecast_discussion": ["A front will keep rain chances elevated."],
"precipitation_timing": " ",
"confidence": " "
"precipitation_timing": " "
}`))
if err != nil {
t.Fatalf("ValidateToday() error = %v", err)
}
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated."]}`
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated."],"precipitation_timing":""}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}

View File

@@ -3,8 +3,7 @@ package generatedtext
type Tomorrow struct {
Summary string `json:"summary"`
ForecastDiscussion []string `json:"forecast_discussion"`
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
Confidence string `json:"confidence,omitempty"`
PrecipitationTiming string `json:"precipitation_timing"`
}
func ValidateTomorrow(data []byte) (Tomorrow, []byte, error) {
@@ -16,7 +15,6 @@ func (t *Tomorrow) dayStyleFields() dayStyleFields {
Summary: t.Summary,
ForecastDiscussion: t.ForecastDiscussion,
PrecipitationTiming: t.PrecipitationTiming,
Confidence: t.Confidence,
}
}
@@ -24,5 +22,4 @@ func (t *Tomorrow) setDayStyleFields(fields dayStyleFields) {
t.Summary = fields.Summary
t.ForecastDiscussion = fields.ForecastDiscussion
t.PrecipitationTiming = fields.PrecipitationTiming
t.Confidence = fields.Confidence
}

View File

@@ -13,8 +13,7 @@ func TestValidateTomorrowNormalizesJSON(t *testing.T) {
"",
" Temperatures stay seasonable by afternoon. "
],
"precipitation_timing": " Rain is most likely before sunrise. ",
"confidence": " Medium "
"precipitation_timing": " Rain is most likely before sunrise. "
}`))
if err != nil {
t.Fatalf("ValidateTomorrow() error = %v", err)
@@ -28,23 +27,22 @@ func TestValidateTomorrowNormalizesJSON(t *testing.T) {
if value.PrecipitationTiming != "Rain is most likely before sunrise." {
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
}
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely before sunrise.","confidence":"Medium"}`
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely before sunrise."}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}
}
func TestValidateTomorrowOmitsEmptyOptionalFields(t *testing.T) {
func TestValidateTomorrowPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
_, normalized, err := ValidateTomorrow([]byte(`{
"summary": "Storms become more likely tomorrow.",
"forecast_discussion": ["A front will keep showers in the forecast."],
"precipitation_timing": " ",
"confidence": " "
"precipitation_timing": " "
}`))
if err != nil {
t.Fatalf("ValidateTomorrow() error = %v", err)
}
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast."]}`
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast."],"precipitation_timing":""}`
if string(normalized) != want {
t.Fatalf("normalized = %s, want %s", normalized, want)
}

View File

@@ -0,0 +1,6 @@
id: weather-balanced
backend: openrouter
model: "~google/gemini-flash-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -0,0 +1,6 @@
id: weather-deep
backend: openrouter
model: "~anthropic/claude-sonnet-latest"
reasoning_effort: high
timeout_seconds: 240
service_tier: flex

View File

@@ -0,0 +1,5 @@
id: weather-light
backend: openrouter
model: deepseek/deepseek-v4-flash
timeout_seconds: 180
service_tier: flex

View File

@@ -8,8 +8,7 @@ Return these fields:
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
- `forecast_discussion`: required. Three paragraphs explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
Return JSON only.
@@ -27,7 +26,7 @@ In most cases, include three paragraphs: a two-to-four sentence relevant local o
# Precipitation timing
Include this only if precipitation is forecast. Use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
When precipitation windows are present, use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.
# Narrative source selection

View File

@@ -1,6 +1,6 @@
id: weather.daily_generated_text
version: "1.0.0"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-balanced
description: Daily weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -8,8 +8,7 @@ Return these fields:
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
- `forecast_discussion`: required. Two or three sentences explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
Return JSON only.
@@ -25,4 +24,4 @@ Use narrative products to explain the “why” behind the local forecast when u
# Precipitation timing
Include this only if precipitation is forecast. Use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
When precipitation windows are present, use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.

View File

@@ -1,6 +1,6 @@
id: weather.hourly_generated_text
version: "1.0.0"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-light
description: Hourly weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -8,8 +8,7 @@ Return these fields:
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
- `forecast_discussion`: required. Three paragraphs explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
Return JSON only.
@@ -27,4 +26,4 @@ In most cases, include three paragraphs: a two-to-four sentence relevant local o
# Precipitation timing
Include this only if precipitation is forecast. Use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
When precipitation windows are present, use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.

View File

@@ -1,6 +1,6 @@
id: weather.today_generated_text
version: "1.0.0"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-balanced
description: Today's weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -8,8 +8,7 @@ Return these fields:
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
- `forecast_discussion`: required. Three paragraphs explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
Return JSON only.
@@ -27,4 +26,4 @@ In most cases, include three paragraphs: a two-to-four sentence relevant local o
# Precipitation timing
Use one or two sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
When precipitation windows are present, use one or two sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.

View File

@@ -1,6 +1,6 @@
id: weather.tomorrow_generated_text
version: "1.0.0"
default_profile: gemini-flash-latest
version: "1.1.0"
default_profile: weather-balanced
description: Tomorrow's weather report analysis prompt.
inputs:
- name: data_package

View File

@@ -4,11 +4,10 @@
"title": "Daily GeneratedText",
"type": "object",
"additionalProperties": false,
"required": ["summary", "forecast_discussion"],
"required": ["summary", "forecast_discussion", "precipitation_timing"],
"properties": {
"summary": {"type": "string"},
"forecast_discussion": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"precipitation_timing": {"type": "string"},
"confidence": {"type": "string"}
"precipitation_timing": {"type": "string"}
}
}

View File

@@ -4,11 +4,10 @@
"title": "Hourly GeneratedText",
"type": "object",
"additionalProperties": false,
"required": ["summary", "forecast_discussion"],
"required": ["summary", "forecast_discussion", "precipitation_timing"],
"properties": {
"summary": {"type": "string"},
"forecast_discussion": {"type": "string"},
"precipitation_timing": {"type": "string"},
"confidence": {"type": "string"}
"precipitation_timing": {"type": "string"}
}
}

View File

@@ -4,11 +4,10 @@
"title": "Today GeneratedText",
"type": "object",
"additionalProperties": false,
"required": ["summary", "forecast_discussion"],
"required": ["summary", "forecast_discussion", "precipitation_timing"],
"properties": {
"summary": {"type": "string"},
"forecast_discussion": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"precipitation_timing": {"type": "string"},
"confidence": {"type": "string"}
"precipitation_timing": {"type": "string"}
}
}

View File

@@ -4,11 +4,10 @@
"title": "Tomorrow GeneratedText",
"type": "object",
"additionalProperties": false,
"required": ["summary", "forecast_discussion"],
"required": ["summary", "forecast_discussion", "precipitation_timing"],
"properties": {
"summary": {"type": "string"},
"forecast_discussion": {"type": "array", "items": {"type": "string"}, "minItems": 1},
"precipitation_timing": {"type": "string"},
"confidence": {"type": "string"}
"precipitation_timing": {"type": "string"}
}
}

View File

@@ -7,7 +7,7 @@ import (
"io/fs"
)
//go:embed assets/prompts assets/schemas
//go:embed assets/prompts assets/profiles assets/schemas
var assets embed.FS
var schemaPaths = map[string]string{
@@ -35,6 +35,15 @@ func SchemaFS() fs.FS {
return fsys
}
// ProfileFS returns the embedded Weatherreporter Promptkit profile definitions.
func ProfileFS() fs.FS {
fsys, err := fs.Sub(assets, "assets/profiles")
if err != nil {
panic(fmt.Sprintf("embedded profile assets: %v", err))
}
return fsys
}
// Schema returns an independent copy of the canonical schema for id.
func Schema(id string) ([]byte, error) {
path, ok := schemaPaths[id]

View File

@@ -34,11 +34,12 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
path string
id string
schemaID string
profile string
}{
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily"},
{"today/today_generated_text.yml", "weather.today_generated_text", "today"},
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow"},
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly"},
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily", "weather-balanced"},
{"today/today_generated_text.yml", "weather.today_generated_text", "today", "weather-balanced"},
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow", "weather-balanced"},
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly", "weather-light"},
}
definitions := 0
@@ -67,8 +68,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
if err := yaml.Unmarshal(data, &definition); err != nil {
t.Fatalf("decode prompt definition: %v", err)
}
if definition.ID != tc.id || definition.Version != "1.0.0" || definition.DefaultProfile != "gemini-flash-latest" {
t.Fatalf("definition = %#v, want %s version 1.0.0 and gemini-flash-latest", definition, tc.id)
if definition.ID != tc.id || definition.Version != "1.1.0" || definition.DefaultProfile != tc.profile {
t.Fatalf("definition = %#v, want %s version 1.1.0 and profile %s", definition, tc.id, tc.profile)
}
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
@@ -101,11 +102,11 @@ func TestSchemasAreCanonicalAndIndependent(t *testing.T) {
if err := json.Unmarshal(data, &schema); err != nil {
t.Fatalf("decode schema: %v", err)
}
if schema.Type != "object" || schema.AdditionalProperties || strings.Join(schema.Required, ",") != "summary,forecast_discussion" {
if schema.Type != "object" || schema.AdditionalProperties || strings.Join(schema.Required, ",") != "summary,forecast_discussion,precipitation_timing" {
t.Fatalf("schema = %#v, want strict generated-text object", schema)
}
if _, ok := schema.Properties["confidence"]; !ok {
t.Fatalf("schema properties = %#v, want confidence", schema.Properties)
if _, ok := schema.Properties["confidence"]; ok {
t.Fatalf("schema properties = %#v, do not want retired confidence field", schema.Properties)
}
if id == "daily" && (schema.ID != "weatherreporter.daily.generated_text.schema.json" || schema.Title != "Daily GeneratedText") {
t.Fatalf("daily schema identity = %q/%q, want corrected Daily identity", schema.ID, schema.Title)
@@ -123,21 +124,116 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
)
if err != nil {
t.Fatalf("NewEngine() error = %v", err)
}
for _, id := range []string{"weather.daily_generated_text", "weather.today_generated_text", "weather.tomorrow_generated_text", "weather.hourly_generated_text"} {
t.Run(id, func(t *testing.T) {
inspection, err := engine.InspectPrompt(context.Background(), id, "1.0.0")
for _, want := range []struct {
id string
profile string
model string
}{
{"weather.daily_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
{"weather.today_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
{"weather.tomorrow_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
} {
t.Run(want.id, func(t *testing.T) {
inspection, err := engine.InspectPrompt(context.Background(), want.id, "1.1.0")
if err != nil {
t.Fatalf("InspectPrompt() error = %v", err)
}
if inspection.PromptID != id || inspection.PromptVersion != "1.0.0" || inspection.DefaultProfileID != "gemini-flash-latest" {
if inspection.PromptID != want.id || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != want.profile {
t.Fatalf("inspection = %#v", inspection)
}
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
if err != nil || profile.EffectiveModelParams.Model != want.model {
t.Fatalf("profile/error = %#v/%v, want model %q", profile, err, want.model)
}
})
}
profile, err := engine.InspectProfile(context.Background(), "weather-deep")
if err != nil || profile.EffectiveModelParams.Model != "~anthropic/claude-sonnet-latest" {
t.Fatalf("weather-deep profile/error = %#v/%v", profile, err)
}
}
func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
wantPaths := map[string]bool{
"weather-balanced.yml": false,
"weather-deep.yml": false,
"weather-light.yml": false,
}
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
if _, ok := wantPaths[path]; !ok {
t.Fatalf("unexpected embedded profile asset %q", path)
}
wantPaths[path] = true
return nil
}); err != nil {
t.Fatalf("walk embedded profiles: %v", err)
}
for path, found := range wantPaths {
if !found {
t.Errorf("missing embedded profile asset %q", path)
}
}
engine, err := promptkit.NewEngine(promptkit.Config{},
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
)
if err != nil {
t.Fatalf("NewEngine() error = %v", err)
}
profiles := []struct {
id string
model string
timeoutSeconds int
reasoningEffort string
}{
{"weather-light", "deepseek/deepseek-v4-flash", 180, ""},
{"weather-balanced", "~google/gemini-flash-latest", 240, "high"},
{"weather-deep", "~anthropic/claude-sonnet-latest", 240, "high"},
}
for _, want := range profiles {
t.Run(want.id, func(t *testing.T) {
inspection, err := engine.InspectProfile(context.Background(), want.id)
if err != nil {
t.Fatalf("InspectProfile() error = %v", err)
}
got := inspection.EffectiveModelParams
if inspection.ProfileID != want.id || got.BackendID != "openrouter" || got.Model != want.model || got.TimeoutSeconds != want.timeoutSeconds || got.ServiceTier != "flex" || got.ReasoningEffort != want.reasoningEffort {
t.Fatalf("inspection = %#v, want %q using openrouter model %q", inspection, want.id, want.model)
}
})
}
}
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
if err != nil || entry.IsDir() {
return err
}
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
if err != nil {
return err
}
for _, setting := range forbidden {
if strings.Contains(string(data), setting) {
t.Fatalf("%s contains forbidden profile setting %q", path, setting)
}
}
return nil
}); err != nil {
t.Fatalf("walk embedded profiles: %v", err)
}
}
func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {

View File

@@ -12,7 +12,7 @@ func dailyDefinition() Definition {
ID: Daily,
Name: "Daily Report",
PromptID: "weather.daily_generated_text",
PromptVersion: "1.0.0",
PromptVersion: "1.1.0",
TemplateID: "daily",
GeneratedTextSchemaID: "daily",
ComparisonStrategy: CompareSameValidDate,

View File

@@ -14,7 +14,7 @@ func hourlyDefinition() Definition {
ID: Hourly,
Name: "Hourly Report",
PromptID: "weather.hourly_generated_text",
PromptVersion: "1.0.0",
PromptVersion: "1.1.0",
TemplateID: "hourly",
GeneratedTextSchemaID: "hourly",
ComparisonStrategy: CompareRollingWindow,

View File

@@ -51,8 +51,11 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
}
for _, definition := range definitions {
if definition.PromptVersion != "1.0.0" {
t.Fatalf("%s PromptVersion = %q, want 1.0.0", definition.ID, definition.PromptVersion)
if definition.PromptVersion != "1.1.0" {
t.Fatalf("%s PromptVersion = %q, want 1.1.0", definition.ID, definition.PromptVersion)
}
if definition.PromptID == "" {
t.Fatalf("%s PromptID is empty", definition.ID)
}
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)

View File

@@ -10,7 +10,7 @@ func todayDefinition() Definition {
ID: Today,
Name: "Today Report",
PromptID: "weather.today_generated_text",
PromptVersion: "1.0.0",
PromptVersion: "1.1.0",
TemplateID: "today",
GeneratedTextSchemaID: "today",
ComparisonStrategy: CompareSameValidDate,

View File

@@ -10,7 +10,7 @@ func tomorrowDefinition() Definition {
ID: Tomorrow,
Name: "Tomorrow Report",
PromptID: "weather.tomorrow_generated_text",
PromptVersion: "1.0.0",
PromptVersion: "1.1.0",
TemplateID: "tomorrow",
GeneratedTextSchemaID: "tomorrow",
ComparisonStrategy: CompareSameValidDate,

View File

@@ -64,7 +64,7 @@ func TestSchemaLookup(t *testing.T) {
if err != nil {
t.Fatalf("Schema() error = %v", err)
}
assertStringSchema(t, data, "summary,forecast_discussion", []string{"summary", "forecast_discussion", "precipitation_timing", "confidence"})
assertStringSchema(t, data, "summary,forecast_discussion,precipitation_timing", []string{"summary", "forecast_discussion", "precipitation_timing"})
}
func TestTomorrowSchemaLookup(t *testing.T) {
@@ -72,7 +72,7 @@ func TestTomorrowSchemaLookup(t *testing.T) {
if err != nil {
t.Fatalf("Schema() error = %v", err)
}
schema := assertSchema(t, data, "summary,forecast_discussion")
schema := assertSchema(t, data, "summary,forecast_discussion,precipitation_timing")
property, ok := schema.Properties["forecast_discussion"].(map[string]any)
if !ok {
t.Fatal("schema property forecast_discussion missing or invalid")
@@ -87,7 +87,7 @@ func TestTomorrowSchemaLookup(t *testing.T) {
if !ok || items["type"] != "string" {
t.Fatalf("forecast_discussion items = %#v, want string items", property["items"])
}
for _, field := range []string{"summary", "precipitation_timing", "confidence"} {
for _, field := range []string{"summary", "precipitation_timing"} {
property, ok := schema.Properties[field].(map[string]any)
if !ok {
t.Fatalf("schema property %q missing or invalid", field)
@@ -103,7 +103,7 @@ func TestDailySchemaLookup(t *testing.T) {
if err != nil {
t.Fatalf("Schema() error = %v", err)
}
schema := assertSchema(t, data, "summary,forecast_discussion")
schema := assertSchema(t, data, "summary,forecast_discussion,precipitation_timing")
property, ok := schema.Properties["forecast_discussion"].(map[string]any)
if !ok {
t.Fatal("schema property forecast_discussion missing or invalid")
@@ -118,7 +118,7 @@ func TestDailySchemaLookup(t *testing.T) {
if !ok || items["type"] != "string" {
t.Fatalf("forecast_discussion items = %#v, want string items", property["items"])
}
for _, field := range []string{"summary", "precipitation_timing", "confidence"} {
for _, field := range []string{"summary", "precipitation_timing"} {
property, ok := schema.Properties[field].(map[string]any)
if !ok {
t.Fatalf("schema property %q missing or invalid", field)
@@ -134,7 +134,7 @@ func TestTodaySchemaLookup(t *testing.T) {
if err != nil {
t.Fatalf("Schema() error = %v", err)
}
schema := assertSchema(t, data, "summary,forecast_discussion")
schema := assertSchema(t, data, "summary,forecast_discussion,precipitation_timing")
property, ok := schema.Properties["forecast_discussion"].(map[string]any)
if !ok {
t.Fatal("schema property forecast_discussion missing or invalid")
@@ -149,7 +149,7 @@ func TestTodaySchemaLookup(t *testing.T) {
if !ok || items["type"] != "string" {
t.Fatalf("forecast_discussion items = %#v, want string items", property["items"])
}
for _, field := range []string{"summary", "precipitation_timing", "confidence"} {
for _, field := range []string{"summary", "precipitation_timing"} {
property, ok := schema.Properties[field].(map[string]any)
if !ok {
t.Fatalf("schema property %q missing or invalid", field)
@@ -214,7 +214,6 @@ func TestRenderHourly(t *testing.T) {
Summary: "Storm chances increase through late morning.",
ForecastDiscussion: "A front will keep the region unsettled.",
PrecipitationTiming: "A cold front is moving into the region.",
Confidence: "Medium confidence in timing.",
},
Modules: testModules{
CurrentConditions: &testCurrentConditions{
@@ -282,7 +281,7 @@ func TestRenderHourly(t *testing.T) {
t.Fatalf("rendered template missing %q:\n%s", want, text)
}
}
if strings.Contains(text, "19%") || strings.Contains(text, "wind S") || strings.Contains(text, "## Confidence") {
if strings.Contains(text, "19%") || strings.Contains(text, "wind S") {
t.Fatalf("rendered template included omitted details:\n%s", text)
}
for _, unwanted := range []string{"Avoid low-water crossings.", "Slight risk for severe thunderstorms"} {
@@ -955,7 +954,6 @@ type testGeneratedText struct {
Summary string
ForecastDiscussion string
PrecipitationTiming string
Confidence string
}
type testTomorrowReportContext struct {
@@ -980,14 +978,12 @@ type testTomorrowGeneratedText struct {
Summary string
ForecastDiscussion []string
PrecipitationTiming string
Confidence string
}
type testDailyGeneratedText struct {
Summary string
ForecastDiscussion []string
PrecipitationTiming string
Confidence string
}
type testModules struct {

View File

@@ -172,9 +172,9 @@ func validPreparationArtifact() PromptPreparationArtifact {
return PromptPreparationArtifact{
SchemaVersion: PromptPreparationSchemaVersion, Status: PromptPreparationSucceeded,
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text",
PromptVersion: "1.0.0", DataPackagePath: "/workspace/data.yaml",
PromptVersion: "1.1.0", DataPackagePath: "/workspace/data.yaml",
Preparation: &promptexec.Preparation{
PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
DataPackagePath: "/workspace/data.yaml",
},
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
@@ -186,9 +186,9 @@ func validExecutionArtifact() PromptExecutionArtifact {
validation := promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", "daily.generated_text.schema.json", nil)
return PromptExecutionArtifact{
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionSucceeded,
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
Provenance: &PromptExecutionProvenance{
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: "profile",
BackendID: "backend", ModelName: "model", DataPackagePath: "/workspace/data.yaml",
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
@@ -201,7 +201,7 @@ func validFailedExecutionArtifact() PromptExecutionArtifact {
started := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
return PromptExecutionArtifact{
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionFailed,
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
StartedAt: started, EndedAt: started, Error: &PromptArtifactError{Category: promptexec.Generation, Message: "provider unavailable"},
}
}