Compare commits
11 Commits
v0.9.0
...
1250247986
| Author | SHA1 | Date | |
|---|---|---|---|
| 1250247986 | |||
| 117c5336ba | |||
| c5ec4f83b2 | |||
| 39c097a710 | |||
| 993120a9f2 | |||
| c20e285d5f | |||
| acbe22dcad | |||
| cc97ae186c | |||
| 51c35f7c22 | |||
| f014a078ee | |||
| 8c19ad763b |
@@ -22,8 +22,12 @@ extra report copies for a command and do not change configuration.
|
||||
collection and generation configuration.
|
||||
- [config.yml](../examples/config.yml) is a representative production-oriented
|
||||
configuration using synthetic endpoints and no credentials.
|
||||
- [weather-light-local-profile.yml](../examples/weather-light-local-profile.yml)
|
||||
is a complete endpoint-only override for the embedded `weather-light`
|
||||
profile.
|
||||
|
||||
Both files are loaded by the configuration test suite.
|
||||
The configuration examples are loaded by the configuration test suite. The
|
||||
profile example is inspected through the Promptkit adapter test suite.
|
||||
|
||||
## Minimal Configuration
|
||||
|
||||
@@ -146,13 +150,35 @@ individual `generate` or `run` command when explicitly needed.
|
||||
|
||||
| Field | Default | Rules |
|
||||
| --- | --- | --- |
|
||||
| `profile` | empty | Optional explicit execution profile. Otherwise the prompt's declared default is used. |
|
||||
| `profile_file` | empty | Optional external profile file. Cannot be combined with `profile_dir`. |
|
||||
| `profile_dir` | empty | Optional external profile directory. Cannot be combined with `profile_file`. |
|
||||
| `profile` | empty | Optional global profile selection for every report in one command. When empty, each exact prompt version selects its declared default. |
|
||||
| `profile_file` | empty | Optional external Promptkit profile file. It cannot be combined with `profile_dir`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
|
||||
| `profile_dir` | empty | Optional external Promptkit profile directory. It cannot be combined with `profile_file`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
|
||||
| `timeout` | `2m` | Must be greater than zero. |
|
||||
| `local.endpoint` | empty | Optional absolute URL for the conventional local backend. A blank endpoint leaves it unregistered. |
|
||||
| `local.concurrency_limit` | `1` | Maximum local backend concurrency. `0` is unlimited; negative values are invalid. |
|
||||
|
||||
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
|
||||
They are separate decisions. An explicit `profile` applies to every selected
|
||||
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
|
||||
Tomorrow select `weather-balanced` through their exact `1.1.0` prompt
|
||||
definitions.
|
||||
|
||||
Promptkit resolves a selected profile definition from a test or embedding
|
||||
consumer's explicit in-memory profile, then the configured `profile_file` or
|
||||
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
|
||||
built-in catalog. Sources provide complete definitions; fields are never
|
||||
merged. A matching malformed external profile fails rather than using the
|
||||
embedded definition. The [Promptkit integration guide](integrations/promptkit.md)
|
||||
owns the catalog and precedence details.
|
||||
|
||||
To replace the default Hourly definition with a local OpenAI-compatible
|
||||
endpoint, set `profile_file` to a copy of
|
||||
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml).
|
||||
The example has no credential and should be edited for the local endpoint and
|
||||
model before use. An alternative profile may use `backend: local`; in that
|
||||
case `promptkit.local.endpoint` supplies the conventional local backend
|
||||
endpoint.
|
||||
|
||||
### `workspace`
|
||||
|
||||
| Field | Default |
|
||||
|
||||
@@ -1,22 +1,60 @@
|
||||
# Promptkit Integration
|
||||
|
||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are
|
||||
`weather.daily_generated_text`, `weather.today_generated_text`,
|
||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version
|
||||
`1.0.0`. Their prompt assets and generated-text JSON Schemas are embedded by
|
||||
`internal/promptassets`.
|
||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical
|
||||
prompts are `weather.daily_generated_text`, `weather.today_generated_text`,
|
||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at
|
||||
version `1.1.0`. Their prompt assets, generated-text JSON Schemas, and
|
||||
Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
||||
|
||||
Before collection, Weatherreporter inspects the exact prompt version, requires one required
|
||||
`data_package` input with content type `application/yaml`, and requires the report's JSON
|
||||
Schema output contract. It selects `promptkit.profile` when configured, otherwise the
|
||||
prompt's declared default profile. Profiles that require a direct API key are unsupported; a
|
||||
profile that reports `APIKeyEnv` requires a nonblank value in that environment variable.
|
||||
## Logical profile catalog
|
||||
|
||||
Prompt definitions select a stable Weatherreporter profile ID. The embedded
|
||||
definitions currently use Promptkit's `openrouter` backend:
|
||||
|
||||
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` | Hourly |
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
|
||||
|
||||
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
|
||||
profiles intentionally omit endpoints, credentials, temperature, `top_p`, and
|
||||
output-token limits.
|
||||
|
||||
## Selection, lookup, and active execution
|
||||
|
||||
Before collection, Weatherreporter inspects the exact prompt version and output
|
||||
contract. A nonblank `promptkit.profile` selects one profile ID for every
|
||||
report in the command; otherwise the prompt's declared default selects it.
|
||||
Promptkit resolves the selected definition in this order:
|
||||
|
||||
1. explicit in-memory profiles used by an embedding consumer or test;
|
||||
2. the configured `profile_file` or `profile_dir`;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's built-in catalog.
|
||||
|
||||
A source falls through only when the selected ID is absent. Each source
|
||||
supplies a complete definition, so profile fields are not merged. A malformed
|
||||
matching operator definition is an error and does not fall back.
|
||||
|
||||
Profiles that require a direct API key are unsupported; a profile that reports
|
||||
`APIKeyEnv` requires a nonblank value in that environment variable. Inspection,
|
||||
preparation, and execution retain the selected logical profile ID and resolved
|
||||
backend and model through Weatherreporter's project-owned contract. Ordinary
|
||||
errors, summaries, logs, and workspace state exclude endpoints, credentials,
|
||||
rendered messages, schemas, request bodies, response bodies, and complete
|
||||
parameter maps.
|
||||
|
||||
Execution receives the already-persisted YAML package, prepares it once, and returns structured
|
||||
JSON that Weatherreporter validates before rendering its own Markdown template. Preparation and
|
||||
execution receipts are project-owned, safe provenance records. Content-rich diagnostics are
|
||||
opt-in through `--llm-debug-dir`; see [operations](../operations.md) for retention and permissions.
|
||||
|
||||
Prompt/profile configuration is owned by the [configuration reference](../config.md). Adapter
|
||||
construction and mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
|
||||
The generated-text schemas require `summary`, `forecast_discussion`, and
|
||||
`precipitation_timing`, and reject additional properties. Prompts return an empty string for
|
||||
`precipitation_timing` when the deterministic package contains no precipitation windows.
|
||||
|
||||
Prompt/profile configuration and the maintained local override example are
|
||||
owned by the [configuration reference](../config.md). Adapter construction and
|
||||
mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
|
||||
Durable metadata compatibility is described in [state internals](../internal/state.md).
|
||||
|
||||
@@ -7,7 +7,10 @@ notification ordering after the CLI has parsed arguments and loaded configuratio
|
||||
|
||||
`GenerateDetailed` resolves one of the four report definitions, initializes an
|
||||
optional debug root, and inspects the exact Promptkit prompt/profile before it
|
||||
collects weather or writes managed state. It then builds facts and modules,
|
||||
collects weather or writes managed state. A configured global profile selects
|
||||
every report in the action; otherwise the exact prompt selects its default
|
||||
logical profile. Inspection keeps only the selected profile ID and effective
|
||||
backend/model needed by the project-owned execution contract. It then builds facts and modules,
|
||||
saves the YAML data package, persists preparation metadata from the executor
|
||||
callback, executes the prepared prompt, saves execution provenance and raw
|
||||
output, validates generated text, renders Markdown, and optionally copies or
|
||||
@@ -25,8 +28,8 @@ persists raw output and execution provenance but does not render a report.
|
||||
|
||||
`RunBatchDetailed` constructs a single debug writer and uses the request's
|
||||
single executor. Before collection it inspects Today, Tomorrow, and Daily for
|
||||
morning, or Tomorrow and Daily for evening, deduplicating effective profile
|
||||
inspection. It then collects once, plans eligible Daily dates, and calls the
|
||||
morning, or Tomorrow and Daily for evening, deduplicating inspection of a
|
||||
shared selected profile. It then collects once, plans eligible Daily dates, and calls the
|
||||
same prompt-generation core sequentially for each planned report. Per-report
|
||||
notification is suppressed; a failed report does not stop later reports.
|
||||
|
||||
|
||||
@@ -17,8 +17,9 @@ value and canonical normalized JSON, loads its canonical schema through
|
||||
|
||||
Daily, Today, and Tomorrow use a day-style value with required trimmed summary
|
||||
and one or more nonblank discussion paragraphs. Hourly requires trimmed summary
|
||||
and a single trimmed discussion string. Each form permits optional trimmed
|
||||
precipitation-timing and confidence prose. Typed decoding rejects unknown JSON
|
||||
and a single trimmed discussion string. Every form also requires the
|
||||
`precipitation_timing` field; an empty string means there is no supported timing
|
||||
prose to render. Typed decoding rejects missing required fields and unknown JSON
|
||||
fields; no general-purpose JSON Schema engine is used at runtime.
|
||||
|
||||
## Render contexts
|
||||
|
||||
@@ -4,10 +4,18 @@
|
||||
The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor
|
||||
per action. Promptkit dependency types do not escape the adapter.
|
||||
|
||||
The adapter exposes exact prompt and profile inspection plus prepared execution. It maps Promptkit
|
||||
inspection values to project-owned prompt input, output-contract, profile, preparation, execution,
|
||||
validation, and optional debug values. It classifies adapter failures without copying provider secrets
|
||||
or unbounded response bodies into application errors or normal state.
|
||||
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback
|
||||
profile filesystems to each engine. Promptkit remains responsible for resolving
|
||||
the configured operator profile source, application fallback catalog, and its
|
||||
built-in catalog; the adapter does not parse profile YAML, merge sources, or
|
||||
probe endpoints.
|
||||
|
||||
The adapter exposes exact prompt and profile inspection plus prepared execution.
|
||||
It maps Promptkit inspection values to project-owned prompt input,
|
||||
output-contract, logical profile identity, effective backend/model,
|
||||
preparation, execution, validation, and optional debug values. It classifies
|
||||
adapter failures without copying provider secrets or unbounded response bodies
|
||||
into application errors or normal state.
|
||||
|
||||
The app calls the executor's preparation callback before provider execution to persist safe preparation
|
||||
provenance. Completed executions are then persisted as safe execution provenance and raw generated text
|
||||
|
||||
@@ -16,16 +16,19 @@ templates, generation eligibility, compatible prior IDs, default modules, and
|
||||
batch eligibility flags. `Resolved` combines that definition with the valid
|
||||
period and run metadata for one invocation.
|
||||
|
||||
| Report ID | Prompt version | Period policy | Comparison | Registry batch flag | Output copy |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `daily` | `1.0.0` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
|
||||
| `today` | `1.0.0` | Selected or current local civil day | Same valid date | Morning | `today.md` |
|
||||
| `tomorrow` | `1.0.0` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
|
||||
| `hourly` | `1.0.0` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
|
||||
| Report ID | Prompt version | Default profile | Period policy | Comparison | Registry batch flag | Output copy |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| `daily` | `1.1.0` | `weather-balanced` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
|
||||
| `today` | `1.1.0` | `weather-balanced` | Selected or current local civil day | Same valid date | Morning | `today.md` |
|
||||
| `tomorrow` | `1.1.0` | `weather-balanced` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
|
||||
| `hourly` | `1.1.0` | `weather-light` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
|
||||
|
||||
Each report pairs its ID and prompt version with matching template and schema
|
||||
IDs. Exact template fields and schema assets belong to [report templates](../templates.md)
|
||||
and [generated-text internals](generatedtext.md).
|
||||
and [generated-text internals](generatedtext.md). Prompt assets own default
|
||||
profile selection; the registry deliberately stores no provider setting. The
|
||||
[Promptkit integration guide](../integrations/promptkit.md) owns profile
|
||||
definitions and resolution.
|
||||
|
||||
All valid periods are half-open.
|
||||
|
||||
|
||||
@@ -34,8 +34,10 @@ runs never write V1 records.
|
||||
Prompt preparation and execution records are validated on both save and load.
|
||||
They require exact report/prompt identity, complete timing, internally
|
||||
consistent provenance, and status-appropriate validation or bounded classified
|
||||
errors. Completed execution provenance keeps Promptkit's run identity distinct
|
||||
from the Weatherreporter run identity.
|
||||
errors. Completed preparation and execution provenance retain the selected
|
||||
logical profile ID and resolved backend/model, but never profile endpoints or
|
||||
credentials. Completed execution provenance keeps Promptkit's run identity
|
||||
distinct from the Weatherreporter run identity.
|
||||
|
||||
For a completed prompt run, the execution record is atomically replaced after
|
||||
each downstream artifact is saved. Its path set therefore records the raw and
|
||||
|
||||
@@ -19,12 +19,38 @@ persists the module snapshot and prompt data package, records Promptkit
|
||||
preparation provenance before provider execution, then persists raw output and
|
||||
execution provenance, validates the structured generated text, and renders the
|
||||
managed Markdown report from the validated text and deterministic values.
|
||||
The current receipts are transitional workspace state, not a cross-version
|
||||
profile-provenance contract.
|
||||
|
||||
The managed report and its final metadata are saved before single-report
|
||||
Distributor notification is attempted. `--out` writes an extra operator copy;
|
||||
it never changes the managed report or upload source. A successful generate
|
||||
command prints its summary to stdout unless `--quiet` is used.
|
||||
|
||||
## Local Prompt Profile Override
|
||||
|
||||
Hourly normally selects the embedded `weather-light` profile. To use a local
|
||||
OpenAI-compatible model without changing prompts or application code, copy
|
||||
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml),
|
||||
set its `endpoint` and `model` for the local server, and configure the copy as
|
||||
`promptkit.profile_file`. The profile file's `weather-light` definition
|
||||
completely replaces the embedded definition; it does not affect a report that
|
||||
selects another profile ID.
|
||||
|
||||
For example, install the profile file at a known absolute path and set:
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
profile_file: /etc/weatherreporter/weather-light-local-profile.yml
|
||||
```
|
||||
|
||||
Prompt inspection occurs before weather collection. A malformed profile file,
|
||||
missing required credential, or unsupported selected backend stops the command
|
||||
before collection. A reachable profile can still fail later if its local model
|
||||
endpoint is unavailable; Weatherreporter does not switch to a remote profile.
|
||||
See the [configuration reference](config.md) for field definitions and the
|
||||
[troubleshooting guide](troubleshooting.md) for recovery.
|
||||
|
||||
## Optional Prompt Debug Capture
|
||||
|
||||
Use `--llm-debug-dir` only when content-rich prompt diagnostics are required:
|
||||
@@ -87,8 +113,11 @@ The generated-text and render-context artifacts are written for every completed
|
||||
single-report generation.
|
||||
A report's metadata links the module snapshot, data package, preparation and
|
||||
execution receipts, managed report, generated-text artifacts, and any available single-report
|
||||
notification artifact. Batch notification artifacts are separate batch-level
|
||||
records under `notifications/batches`.
|
||||
notification artifact. These current-version receipts remain transitional; use
|
||||
the active command's classified error and explicit secure debug capture for
|
||||
prompt diagnosis rather than relying on them as a durable interface. Batch
|
||||
notification artifacts are separate batch-level records under
|
||||
`notifications/batches`.
|
||||
|
||||
RunIDs begin with the UTC generation timestamp and report ID. A Daily RunID
|
||||
also contains its local valid date so multiple Daily reports in one batch have
|
||||
@@ -162,10 +191,12 @@ remain available where they can be safely persisted.
|
||||
- A batch notification failure preserves each report's artifacts and adds the
|
||||
top-level batch notification artifact.
|
||||
|
||||
Use the RunID from the action summary with the inspection commands above. For
|
||||
a batch failure, inspect the summary first, then inspect the affected report
|
||||
RunIDs or the batch notification path. Do not remove the whole workspace as a
|
||||
first response; retain it until the failure is understood.
|
||||
Use the action summary and its classified error first. For prompt or provider
|
||||
diagnosis, prefer an explicitly enabled secure debug capture; current-version
|
||||
receipt paths may provide supplemental context when available. For a batch
|
||||
failure, inspect the summary first, then inspect the affected report RunIDs or
|
||||
the batch notification path. Do not remove the whole workspace as a first
|
||||
response; retain it until the failure is understood.
|
||||
|
||||
## Operational Caveats
|
||||
|
||||
@@ -174,5 +205,7 @@ first response; retain it until the failure is understood.
|
||||
not publish them unintentionally.
|
||||
- Weatherreporter uses one configured Weather API endpoint and local workspace
|
||||
state.
|
||||
- Promptkit profile resolution does not discover local endpoints or fail over
|
||||
between local and remote profiles.
|
||||
- It does not provide automatic resume, cleanup, archival, remote state, daemon
|
||||
operation, or automatic storm monitoring.
|
||||
|
||||
283
docs/roadmap/domain-profiles.md
Normal file
283
docs/roadmap/domain-profiles.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# Domain-Specific Prompt Profiles Roadmap
|
||||
|
||||
Status: Implemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
Weatherreporter should provide stable, domain-specific Promptkit profile IDs
|
||||
that express the relative resource and analysis needs of its report products.
|
||||
These logical profiles should give each report an appropriate default while
|
||||
allowing operators to replace any definition through the existing configured
|
||||
profile source.
|
||||
|
||||
This roadmap records the scope, policy, and implemented end state. The
|
||||
companion [implementation plan](implementation.md) records the ordered work
|
||||
and verification used to reach it.
|
||||
|
||||
## User Intent
|
||||
|
||||
The feature is intended to provide three related benefits:
|
||||
|
||||
- frequent reports can use a cost-effective model by default;
|
||||
- reports needing broader synthesis can select a stronger default without
|
||||
forcing the same cost on every invocation; and
|
||||
- an installation can map a stable Weatherreporter profile ID to a model on a
|
||||
local network endpoint without modifying embedded prompts or application
|
||||
code.
|
||||
|
||||
`weather-light` describes the profile's intended resource tier, not a latency
|
||||
guarantee. A locally hosted lightweight model may still generate slowly on the
|
||||
available hardware.
|
||||
|
||||
## Pre-Implementation Baseline
|
||||
|
||||
Before implementation, Daily, Today, Tomorrow, and Hourly each declared
|
||||
Promptkit's `gemini-flash-latest` profile as their prompt default. The optional
|
||||
`promptkit.profile` setting overrode that default for every selected report in
|
||||
an invocation.
|
||||
|
||||
Weatherreporter accepted either `promptkit.profile_file` or
|
||||
`promptkit.profile_dir` and passed that source to Promptkit. A matching external
|
||||
profile could override a Promptkit built-in profile, and the configured local
|
||||
backend could support profiles that select `backend: local`. Endpoint-only
|
||||
OpenAI-compatible profiles could also provide their own endpoint.
|
||||
|
||||
Weatherreporter did not own or embed execution profiles. Promptkit v0.5.0
|
||||
provided the fallback-profile layer used to add them without changing the
|
||||
existing operator-source precedence.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Promptkit v0.5.0 provides the application fallback profile capability defined
|
||||
in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
|
||||
Weatherreporter now depends on that tagged release. The dependency upgrade has
|
||||
passed the repository test suite and an operator smoke test. Weatherreporter
|
||||
must continue to use only Promptkit's public API rather than depending on its
|
||||
internal packages or reproducing its profile repository behavior.
|
||||
|
||||
## Implemented End State
|
||||
|
||||
Weatherreporter embeds usable definitions for these exact logical profile IDs:
|
||||
|
||||
- `weather-light`
|
||||
- `weather-balanced`
|
||||
- `weather-deep`
|
||||
|
||||
The profiles are Weatherreporter-owned assets and remain behind the existing
|
||||
Promptkit adapter boundary. Prompt definitions select the logical IDs, while
|
||||
Promptkit resolves the effective backend, endpoint, model, and generation
|
||||
settings.
|
||||
|
||||
An operator can place a profile with the same ID in `profile_file` or
|
||||
`profile_dir`. The operator definition completely replaces the embedded
|
||||
Weatherreporter definition for that ID. If the external source does not contain
|
||||
the selected ID, lookup falls through to Weatherreporter's embedded profile and
|
||||
then to Promptkit's built-in catalog.
|
||||
|
||||
The existing global `promptkit.profile` setting remains available as an
|
||||
explicit all-report override. No new configuration field is required for the
|
||||
initial feature.
|
||||
|
||||
## Profile Catalog And Report Assignment
|
||||
|
||||
| Profile | Meaning | Initial default reports |
|
||||
| --- | --- | --- |
|
||||
| `weather-light` | Lowest-cost supported tier for frequent, bounded synthesis. It makes no latency promise. | Hourly |
|
||||
| `weather-balanced` | General-purpose tier for broader day-scale synthesis and forecast discussion. | Daily, Today, Tomorrow |
|
||||
| `weather-deep` | Highest-capability tier for explicit operator use and future products whose measured quality benefit warrants the cost. | None initially |
|
||||
|
||||
The initial assignment recognizes that Weatherreporter's deterministic modules
|
||||
already perform most weather selection and calculation. A higher-capability
|
||||
model should not become a default merely because it is available. Moving an
|
||||
existing report to `weather-deep` requires evidence that the stronger tier
|
||||
materially improves supported reasoning or output quality.
|
||||
|
||||
The three profile IDs are capability policies, not permanent aliases for one
|
||||
provider or model family. Their embedded definitions may change in a future
|
||||
Weatherreporter release, with the change disclosed through normal release and
|
||||
compatibility documentation.
|
||||
|
||||
## Selection And Definition Precedence
|
||||
|
||||
Profile ID selection and profile definition lookup are separate decisions.
|
||||
|
||||
Weatherreporter selects the profile ID in this order:
|
||||
|
||||
1. nonblank `promptkit.profile`; or
|
||||
2. the exact prompt version's `default_profile`.
|
||||
|
||||
Promptkit then resolves the selected profile definition in this order:
|
||||
|
||||
1. explicit in-memory profiles, when used by an embedding consumer or test;
|
||||
2. Weatherreporter's configured `profile_file` or `profile_dir` source;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's embedded built-in profiles.
|
||||
|
||||
A higher-precedence source falls through only when the selected ID is absent.
|
||||
A matching but malformed operator profile fails before weather collection and
|
||||
must not silently use the embedded definition.
|
||||
|
||||
## Local Endpoint Experience
|
||||
|
||||
An operator should be able to override `weather-light` with an endpoint-only
|
||||
profile whose model name is understood by the local OpenAI-compatible server.
|
||||
This path does not require a separate Weatherreporter local-backend setting.
|
||||
|
||||
Alternatively, an override may select `backend: local`; in that case the
|
||||
existing `promptkit.local.endpoint` and concurrency settings continue to own
|
||||
the shared local backend definition.
|
||||
|
||||
The selected local profile is deterministic configuration, not a preference
|
||||
hint. Weatherreporter will not probe for a local model and will not
|
||||
automatically fall back to a remote or paid profile when the endpoint is
|
||||
unavailable. The failure remains visible and attributable to the selected
|
||||
profile.
|
||||
|
||||
## Embedded Profile Policy
|
||||
|
||||
Each embedded profile must be a complete, valid Promptkit profile and must be
|
||||
usable in a default installation with the documented credential mechanism. The
|
||||
initial embedded profiles are expected to use Promptkit's `openrouter` backend,
|
||||
allowing them to inherit its endpoint and `OPENROUTER_API_KEY` environment
|
||||
variable without embedding credentials.
|
||||
|
||||
Embedded definitions should include only settings that are intentional for the
|
||||
selected model and supported by its backend. Avoid incidental generation
|
||||
parameters that reduce portability or trigger provider-specific request
|
||||
failures without a demonstrated quality benefit.
|
||||
|
||||
The initial profile definitions are:
|
||||
|
||||
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
These settings deliberately match the corresponding Promptkit v0.5.0
|
||||
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
|
||||
part of each OpenRouter rolling-alias identifier. The profiles do not set
|
||||
temperature, `top_p`, or output-token limits; omission preserves provider
|
||||
defaults and avoids unsupported incidental parameters.
|
||||
|
||||
## Prompt And Active Execution Contract
|
||||
|
||||
Changing a prompt's `default_profile` is a material prompt-definition change.
|
||||
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
||||
defaults are introduced. Prompt content and generated-text schemas need not
|
||||
change solely for this feature.
|
||||
|
||||
Prompt inspection must continue to occur before weather collection. It should
|
||||
report the selected logical profile ID and the resolved backend and model
|
||||
without exposing endpoints or credentials.
|
||||
|
||||
The active execution contract should retain both the selected logical profile
|
||||
identity and the resolved backend and model through inspection, preparation,
|
||||
execution, errors, and command results where those values are already exposed.
|
||||
This feature must not add a new durable-provenance or cross-version artifact
|
||||
contract.
|
||||
|
||||
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
|
||||
prompt provenance a non-goal. Existing workspace persistence may remain while
|
||||
this feature lands, but it is transitional behavior and must not be expanded or
|
||||
treated as part of the profile feature's desired end state. Prompt preparation
|
||||
and execution artifacts written at `1.0.1` are not required to remain readable
|
||||
after the prompt definitions advance to `1.1.0`.
|
||||
|
||||
## Evaluation Policy
|
||||
|
||||
Concrete model assignments should be evaluated with representative,
|
||||
secret-free Daily, Today, Tomorrow, and Hourly data packages. Evaluation should
|
||||
consider:
|
||||
|
||||
- strict-schema success rate;
|
||||
- unsupported or invented weather claims;
|
||||
- precipitation-timing accuracy and empty-string behavior;
|
||||
- correct use of deterministic hazards, periods, and uncertainty;
|
||||
- summary and forecast-discussion usefulness;
|
||||
- generation latency;
|
||||
- token use and provider cost; and
|
||||
- behavior through a representative local OpenAI-compatible endpoint.
|
||||
|
||||
The purpose is to choose an appropriate default for each tier, not to add a
|
||||
permanent benchmark framework or live-provider requirement to the ordinary
|
||||
test suite. Repository tests remain offline and deterministic.
|
||||
|
||||
## Implemented Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
- Weatherreporter-owned embedded profile assets for all three logical IDs;
|
||||
- Promptkit adapter wiring that supplies those assets as the application
|
||||
fallback profile source;
|
||||
- per-prompt default-profile assignments matching the catalog above;
|
||||
- an exact prompt-version update for the changed definitions;
|
||||
- preservation of the global profile override;
|
||||
- same-ID override behavior through both supported external profile-source
|
||||
forms;
|
||||
- local-backend and endpoint-only override coverage;
|
||||
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
||||
- offline tests for selection, source precedence, effective model inspection,
|
||||
batch reuse, and active execution behavior;
|
||||
- maintained operator examples for overriding `weather-light` locally; and
|
||||
- updates to the canonical configuration, Promptkit integration, report
|
||||
registry, operations, troubleshooting, internal adapter, and release
|
||||
documentation as applicable when implementation lands.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
The feature does not include:
|
||||
|
||||
- automatic discovery, health checking, or benchmarking of local endpoints;
|
||||
- implicit failover between local and remote profiles;
|
||||
- retries with a more expensive tier after provider or validation failure;
|
||||
- per-report profile configuration fields outside prompt defaults;
|
||||
- profile inheritance, aliases, or field-level merging;
|
||||
- runtime model selection based on weather severity, token count, or report
|
||||
content;
|
||||
- moving Weatherreporter profile policy into Promptkit's built-in catalog;
|
||||
- exposing Promptkit types outside the adapter boundary; or
|
||||
- making live provider calls part of the default repository test suite.
|
||||
|
||||
## Compatibility And Operational Policy
|
||||
|
||||
Existing configurations with a nonblank `promptkit.profile` retain their
|
||||
all-report behavior. Existing `profile_file`, `profile_dir`, local-backend, and
|
||||
credential configuration fields retain their meanings.
|
||||
|
||||
Configurations that rely on the omitted profile setting will intentionally
|
||||
observe new per-report defaults. This is a user-visible model-selection and
|
||||
cost change and must be called out in release notes. Operators who require the
|
||||
old all-report model can preserve it by setting an explicit global profile.
|
||||
|
||||
The prompt-version transition does not provide backward compatibility for
|
||||
historical prompt preparation or execution artifacts. This is consistent with
|
||||
the accepted ephemeral-state direction; the profile feature does not otherwise
|
||||
redesign or remove the current workspace layout.
|
||||
|
||||
An external same-ID override is an operator-owned compatibility commitment.
|
||||
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
||||
silently merge an operator file.
|
||||
|
||||
## Completion Record
|
||||
|
||||
The following conditions are satisfied:
|
||||
|
||||
- a tagged Promptkit dependency supports the required fallback layer;
|
||||
- every operational prompt selects its assigned logical profile at exact
|
||||
version `1.1.0`;
|
||||
- all three embedded profiles inspect successfully without an external profile
|
||||
source;
|
||||
- configured same-ID definitions override embedded definitions through both
|
||||
`profile_file` and `profile_dir`;
|
||||
- an invalid matching external definition fails without fallback;
|
||||
- `weather-light` can resolve through an endpoint-only or configured-local
|
||||
override without requiring code or prompt changes;
|
||||
- global `promptkit.profile` still overrides every report in an invocation;
|
||||
- active inspection and execution preserve the selected logical profile and
|
||||
effective model through the project-owned execution contract;
|
||||
- morning and evening batch preflight deduplicates inspection of shared
|
||||
effective profile IDs as it does today;
|
||||
- the default test suite remains offline and deterministic; and
|
||||
- implemented behavior is documented by its canonical current-state owners.
|
||||
382
docs/roadmap/ephemeral-state.md
Normal file
382
docs/roadmap/ephemeral-state.md
Normal file
@@ -0,0 +1,382 @@
|
||||
# Ephemeral Operational State Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
|
||||
## Purpose
|
||||
|
||||
Weatherreporter should treat generated weather reports and their intermediate
|
||||
artifacts as short-lived operational material rather than a durable audit
|
||||
history. Forecasts and current conditions change continuously, and the normal
|
||||
response to an old or failed report is to generate a new report, not to
|
||||
reconstruct the provenance of the old one.
|
||||
|
||||
The application should retain only the bounded state needed to publish the
|
||||
current report, calculate Recent Changes against the last successfully
|
||||
published report for the same valid period, and complete the current
|
||||
invocation safely. Detailed LLM diagnostics should remain an explicit,
|
||||
operator-controlled exception outside ordinary workspace state.
|
||||
|
||||
This roadmap defines the intended state lifecycle, compatibility policy, and
|
||||
architectural boundaries. A separate implementation plan will define the
|
||||
ordered work after the roadmap is complete.
|
||||
|
||||
## User Intent
|
||||
|
||||
The state model should reflect these product expectations:
|
||||
|
||||
- weather reports are ephemeral products, not business records;
|
||||
- old report provenance has no continuing operational value once conditions
|
||||
and forecasts have changed;
|
||||
- regenerating is preferable to recovering, replaying, or inspecting an old
|
||||
generation;
|
||||
- routine operation should not accumulate unbounded run-addressed artifacts;
|
||||
- Recent Changes remains useful, but needs only one prior successful snapshot
|
||||
for the same report and valid period; and
|
||||
- sensitive prompt and response capture remains opt-in and explicitly managed
|
||||
by the operator.
|
||||
|
||||
## Current State
|
||||
|
||||
Each generation currently writes a run-addressed collection containing a
|
||||
module snapshot, data package, prompt preparation receipt, prompt execution
|
||||
receipt, raw generated text, validated generated text, render context, managed
|
||||
report, metadata, and optional notification receipt. Successful and failed
|
||||
runs accumulate beneath the workspace.
|
||||
|
||||
Metadata links the collection and supports lookup by RunID. The CLI can list
|
||||
historical runs and inspect their metadata, modules, data packages, prior
|
||||
snapshots, and source provenance. New metadata uses the V2 format while the
|
||||
reader retains V1 compatibility. Prompt artifacts are validated against
|
||||
current report and prompt definitions when saved and loaded.
|
||||
|
||||
Most of this persistence exists for retrospective inspection and failure
|
||||
recovery. Dedicated prompt preparation and execution load operations have no
|
||||
ordinary production consumer. The important exception is module snapshot
|
||||
state: generation actively loads the most recent compatible snapshot to build
|
||||
the deterministic Recent Changes input for Daily, Today, and Tomorrow.
|
||||
|
||||
## Desired End State
|
||||
|
||||
Weatherreporter has three distinct state classes:
|
||||
|
||||
| State class | Lifecycle | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Invocation workspace | Temporary and unpublished | Hold intermediate values while one report or batch is running. |
|
||||
| Current published state | Bounded and replaceable | Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation. |
|
||||
| Secure LLM debug capture | Explicitly enabled and operator-managed | Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture. |
|
||||
|
||||
Ordinary generation uses an invocation-scoped temporary directory on the same
|
||||
filesystem as the managed workspace when atomic publication requires it.
|
||||
Prompt preparation, prompt execution, raw generated text, validated generated
|
||||
text, render contexts, data packages, and notification receipts may exist
|
||||
there while needed, but they are not published as durable historical
|
||||
artifacts.
|
||||
|
||||
A successful report atomically replaces the current published state for its
|
||||
logical report key and valid period. A failed attempt leaves the last
|
||||
successfully published report and comparison snapshot unchanged. Ordinary
|
||||
temporary artifacts are removed after both success and handled failure;
|
||||
cleanup failure is reported safely but must not replace the primary generation
|
||||
error.
|
||||
|
||||
RunIDs remain useful as in-process correlation identifiers in action results,
|
||||
logs, provider provenance, and optional debug paths. They no longer identify a
|
||||
durable collection that Weatherreporter promises to locate or decode later.
|
||||
|
||||
## Published Report Policy
|
||||
|
||||
The managed Markdown report remains the authoritative upload source during an
|
||||
invocation. The intended default is to retain only the current managed report
|
||||
for each logical report key and valid period, replacing it atomically after a
|
||||
new report has been fully rendered and validated.
|
||||
|
||||
An explicit `--out` or `--out-dir` copy remains operator-owned output outside
|
||||
the managed-state lifecycle. Weatherreporter does not delete, rotate, or
|
||||
rewrite those copies except when the same explicit destination is selected by
|
||||
a later invocation.
|
||||
|
||||
Distributor continues to receive only a completed managed Markdown report.
|
||||
Notification success or failure does not create a durable notification
|
||||
history. A notification failure leaves the newly published report available
|
||||
and returns a safe error through the current action result.
|
||||
|
||||
## Recent Changes State
|
||||
|
||||
Recent Changes must be preserved without preserving general report history.
|
||||
For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible
|
||||
module snapshot for each logical report key and valid period.
|
||||
|
||||
The retained snapshot represents the last successfully published report. A
|
||||
new invocation reads it before constructing Recent Changes and replaces it
|
||||
only when the new managed report has been successfully validated, rendered,
|
||||
and published. A failed generation therefore does not become the baseline for
|
||||
the next report and cannot hide changes that the user has not yet seen.
|
||||
|
||||
Hourly does not currently use the comparison strategy and should not retain a
|
||||
comparison snapshot solely for symmetry. State whose valid period has ended
|
||||
and can no longer participate in a supported comparison is eligible for safe
|
||||
cleanup.
|
||||
|
||||
## Temporary Workspace And Failure Semantics
|
||||
|
||||
Temporary state must remain beneath a narrowly owned application directory and
|
||||
use safe path construction, restrictive permissions where content is
|
||||
sensitive, and atomic writes where practical. Publication must not expose a
|
||||
partially rendered report or a snapshot that does not correspond to the
|
||||
published report.
|
||||
|
||||
Normal results retain bounded error information and paths only for artifacts
|
||||
that remain meaningful after the command: a previously or newly published
|
||||
report, an explicit operator output, or an enabled secure debug capture.
|
||||
Temporary intermediate paths are not emitted as if they were durable recovery
|
||||
locations. A failed command is retried by starting a new generation.
|
||||
|
||||
Process interruption may leave an uncommitted temporary directory. Such a
|
||||
directory is never considered published state, is never selected for Recent
|
||||
Changes, and may be removed by a documented safe cleanup mechanism. Cleanup
|
||||
must distinguish inactive temporary directories from concurrent active
|
||||
invocations and must never recursively target the workspace root or an
|
||||
unresolved configuration path.
|
||||
|
||||
## Inspection And Metadata Policy
|
||||
|
||||
Run-history discovery and inspection are not part of the desired product
|
||||
contract. The historical `inspect reports`, `inspect metadata`, `inspect
|
||||
modules`, `inspect data-package`, `inspect prior`, and `inspect sources`
|
||||
surfaces are candidates for removal together rather than preservation through
|
||||
a new storage representation.
|
||||
|
||||
Any manifest retained for atomic publication or Recent Changes is current
|
||||
operational state, not an archival metadata record. It should contain only the
|
||||
identity, valid period, safe paths, and deterministic snapshot information
|
||||
needed to validate and use that current state. It does not need to preserve
|
||||
prompt messages, generated prose intermediates, source provenance, provider
|
||||
provenance, notification history, or a catalog of prior runs.
|
||||
|
||||
The application does not promise cross-version decoding of ordinary workspace
|
||||
state. A new release may replace or ignore incompatible current-state files,
|
||||
provided it fails safely, never mistakes stale state for a compatible Recent
|
||||
Changes baseline, and documents any operator action required during upgrade.
|
||||
|
||||
## Prompt Execution And Debugging
|
||||
|
||||
Prompt inspection before weather collection and prepared execution remain
|
||||
runtime safety requirements. They do not require durable preparation or
|
||||
execution receipts.
|
||||
|
||||
The selected logical profile, effective backend and model, validation outcome,
|
||||
and safe classified error remain available to the active workflow and its CLI
|
||||
summary where useful. Weatherreporter does not retain them as long-term report
|
||||
provenance after the invocation completes.
|
||||
|
||||
The existing explicit secure debug root remains outside ordinary state and may
|
||||
retain rendered prompts, schemas, input bodies, generated bodies, and effective
|
||||
parameters according to its documented contract. Weatherreporter does not
|
||||
automatically clean that operator-selected location. Credentials must remain
|
||||
excluded from debug capture.
|
||||
|
||||
## Compatibility And Upgrade Policy
|
||||
|
||||
This is an intentional breaking change to the workspace and inspection
|
||||
contracts. Weatherreporter does not need to migrate historical V1 or V2
|
||||
metadata, prompt receipts, intermediate generated-text artifacts, or managed
|
||||
reports into the new representation.
|
||||
|
||||
Legacy workspace trees must not be silently interpreted as current published
|
||||
state. They also must not be deleted automatically merely because a new
|
||||
version starts: an operator may have placed or referenced files there despite
|
||||
the absence of a continuing application compatibility promise. Release notes
|
||||
and operations documentation must explain whether legacy data can be removed
|
||||
manually and identify the exact safe target.
|
||||
|
||||
The change should land in a release whose notes clearly identify removed CLI
|
||||
commands, obsolete paths and schemas, the new bounded state behavior, and any
|
||||
upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic
|
||||
version policy may carry this breaking change without inventing a migration
|
||||
framework.
|
||||
|
||||
## Required Architecture Decision Record
|
||||
|
||||
The implemented feature must include an Accepted ADR recording the durable
|
||||
architectural decision to use ephemeral operational state. The ADR is not part
|
||||
of this roadmap-writing pass and should not be created until implementation is
|
||||
being prepared.
|
||||
|
||||
The ADR should record:
|
||||
|
||||
- the mismatch between run-addressed provenance storage and the ephemeral
|
||||
weather-report lifecycle;
|
||||
- the decision to retain bounded current report and comparison state rather
|
||||
than historical runs;
|
||||
- the distinction between temporary invocation state, published operational
|
||||
state, explicit output copies, and secure debug capture;
|
||||
- the removal of historical inspection and backward-compatibility guarantees;
|
||||
- atomic publication and failed-run behavior;
|
||||
- the alternatives considered, including retaining the current archive,
|
||||
adding time-based retention, or keeping a bounded run history; and
|
||||
- consequences for CLI compatibility, workspace layout, testing, operations,
|
||||
and future schema changes.
|
||||
|
||||
Once accepted, the ADR owns the decision rationale. The architecture policy
|
||||
owns the resulting current invariant, while focused state, CLI, operations,
|
||||
and integration documents own the implemented contracts.
|
||||
|
||||
## Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
- an invocation-scoped temporary workspace for intermediate generation state;
|
||||
- atomic publication of the current managed report and its minimal operational
|
||||
state;
|
||||
- a bounded comparison snapshot representing the last successfully published
|
||||
report for each supported report key and valid period;
|
||||
- safe cleanup behavior for normal completion, handled failure, and abandoned
|
||||
temporary workspaces;
|
||||
- removal of durable prompt preparation, prompt execution, generated-text,
|
||||
render-context, data-package, notification, and run-metadata history;
|
||||
- removal of run-history inspection commands and their application/state
|
||||
contracts;
|
||||
- removal of V1 metadata compatibility and current-version coupling for
|
||||
historical prompt artifacts by removing the historical artifact contract;
|
||||
- preservation of active-command partial status, safe errors, and paths to
|
||||
genuinely retained published, operator-owned, or debug outputs;
|
||||
- preservation of explicit output copies, Distributor upload behavior, and
|
||||
opt-in secure LLM debug capture;
|
||||
- risk-appropriate offline tests for atomic publication, comparison baselines,
|
||||
failure isolation, cleanup safety, concurrent invocation safety, and absence
|
||||
of unbounded state growth;
|
||||
- an Accepted ADR documenting the architectural decision; and
|
||||
- updates to canonical architecture, CLI, operations, configuration,
|
||||
troubleshooting, integration, internal, testing, and release documentation
|
||||
where their contracts change.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This feature does not include:
|
||||
|
||||
- a general-purpose cache, database, archival service, or retention engine;
|
||||
- replaying or resuming interrupted generation;
|
||||
- migrating legacy artifacts into the new representation;
|
||||
- retaining a bounded number of historical runs for convenience;
|
||||
- automatic upload or archival of state to remote storage;
|
||||
- collecting additional provider telemetry or weather-source provenance;
|
||||
- changing report content, prompt text, schemas, profile selection, weather
|
||||
derivation, or batch membership;
|
||||
- deleting operator-owned `--out`, `--out-dir`, or secure debug files;
|
||||
- changing Distributor's report-content contract; or
|
||||
- making live external services part of the default test suite.
|
||||
|
||||
## Safety And Testing Policy
|
||||
|
||||
The state refactoring must preserve Weatherreporter's existing path-safety and
|
||||
atomicity expectations while reducing the amount of durable state. Tests
|
||||
should emphasize observable lifecycle guarantees rather than private file
|
||||
choreography.
|
||||
|
||||
Important risks requiring durable offline coverage include:
|
||||
|
||||
- a failed or canceled generation replacing a previously published report or
|
||||
comparison baseline;
|
||||
- a partially written report becoming visible as current;
|
||||
- Recent Changes selecting an incompatible report, valid period, or failed
|
||||
attempt;
|
||||
- cleanup deleting published, operator-owned, debug, or concurrently active
|
||||
files;
|
||||
- batch partial success corrupting the state of another report;
|
||||
- notification failure rolling back or obscuring a successfully published
|
||||
report;
|
||||
- stale or incompatible current state being treated as valid; and
|
||||
- repeated successful and failed runs causing unbounded ordinary workspace
|
||||
growth.
|
||||
|
||||
Tests remain deterministic, offline, credential-free, and based on real
|
||||
temporary filesystems plus narrow external-boundary fakes. Race-enabled tests
|
||||
are required where publication, cleanup, or concurrent invocation behavior
|
||||
shares mutable filesystem state.
|
||||
|
||||
## Relationship To Domain-Specific Profiles
|
||||
|
||||
The domain-specific profile feature can be implemented before this refactor,
|
||||
but it should not add new historical compatibility or durable-provenance
|
||||
commitments. Profile inspection, selection, override precedence, and effective
|
||||
model resolution remain active-workflow behavior and survive the state change.
|
||||
|
||||
The domain-profile roadmap and implementation plan should acknowledge that
|
||||
prompt artifacts from version `1.0.1` need not remain readable after prompts
|
||||
advance to `1.1.0`. Existing state persistence may remain temporarily while
|
||||
the profile feature lands, but it should not be expanded or treated as the
|
||||
target architecture.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The roadmap's target state is achieved when:
|
||||
|
||||
- ordinary runs no longer create durable run-addressed artifact collections;
|
||||
- a successful report atomically replaces only the corresponding current
|
||||
published state;
|
||||
- failed and canceled attempts leave the prior published report and Recent
|
||||
Changes baseline unchanged;
|
||||
- Daily, Today, and Tomorrow compare against at most one compatible snapshot
|
||||
from the last successfully published report;
|
||||
- expired comparison and published state can be removed safely without
|
||||
touching operator-owned or active files;
|
||||
- Hourly does not retain an unused comparison snapshot;
|
||||
- historical inspection commands and V1/V2 archival compatibility code are
|
||||
removed;
|
||||
- temporary, published, explicit-output, and debug paths have distinct and
|
||||
documented ownership and cleanup rules;
|
||||
- Distributor and active-command summaries continue to receive the completed
|
||||
report and safe status information they require;
|
||||
- the default suite proves atomicity, bounded growth, cleanup safety, batch
|
||||
isolation, and comparison correctness offline;
|
||||
- an Accepted ADR records the architectural decision and alternatives; and
|
||||
- canonical current-state documentation describes only the implemented
|
||||
lifecycle.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Lifetime of the current managed report
|
||||
|
||||
Recommendation: retain one current managed report per logical report key and
|
||||
valid period until it is replaced or its valid period expires. This preserves
|
||||
the current default behavior for invocations without `--out` while bounding
|
||||
growth.
|
||||
|
||||
Alternative: treat the managed report as temporary and retain output only when
|
||||
the operator supplies `--out` or `--out-dir`. This minimizes state further but
|
||||
makes a successful default invocation produce no durable report for the user
|
||||
and complicates Distributor sequencing.
|
||||
|
||||
### Historical inspection replacement
|
||||
|
||||
Recommendation: remove the run-history inspection commands without adding a
|
||||
replacement initially. Current command summaries, current managed files, and
|
||||
opt-in debug capture cover the remaining supported workflows.
|
||||
|
||||
Alternative: add a narrow `inspect current REPORT` command backed only by the
|
||||
current operational manifest. This provides discoverability without history,
|
||||
but it creates a new public surface and may preserve metadata complexity that
|
||||
the refactor is intended to remove.
|
||||
|
||||
### Abandoned temporary workspace cleanup
|
||||
|
||||
Recommendation: use an explicitly owned temporary subtree with per-invocation
|
||||
ownership markers and a conservative age threshold. Normal cleanup removes the
|
||||
current invocation synchronously; opportunistic cleanup removes only marked,
|
||||
inactive directories old enough that they cannot reasonably belong to a live
|
||||
invocation.
|
||||
|
||||
Alternative: perform only synchronous cleanup and document manual removal of
|
||||
directories left by process termination. This minimizes destructive code and
|
||||
concurrency risk, but crashed processes can still accumulate unbounded files.
|
||||
|
||||
### Legacy workspace cleanup
|
||||
|
||||
Recommendation: ignore legacy run-addressed trees and document a precise,
|
||||
manual one-time cleanup procedure. Do not automatically delete them during
|
||||
startup or upgrade.
|
||||
|
||||
Alternative: add an explicit cleanup command that previews and then removes
|
||||
recognized legacy artifacts. This is more convenient for large installations
|
||||
but introduces a destructive command and a legacy-format classifier that must
|
||||
be maintained and tested.
|
||||
File diff suppressed because it is too large
Load Diff
169
docs/roadmap/promptkit-fallback-profiles-feature-request.md
Normal file
169
docs/roadmap/promptkit-fallback-profiles-feature-request.md
Normal file
@@ -0,0 +1,169 @@
|
||||
# Promptkit Feature Request: Application Fallback Profiles
|
||||
|
||||
Status: Implemented upstream in Promptkit v0.5.0.
|
||||
|
||||
Promptkit v0.5.0 resolved this request with the public
|
||||
`WithFallbackProfileFS` engine option and the precedence and error semantics
|
||||
specified below. This document is retained as the downstream rationale for
|
||||
the capability.
|
||||
|
||||
## Purpose
|
||||
|
||||
Promptkit should allow a consuming application to supply an embedded fallback
|
||||
profile source that sits below operator-configured profiles and above
|
||||
Promptkit's own built-in profile catalog.
|
||||
|
||||
This capability would let an application publish stable, domain-specific
|
||||
profile IDs with useful defaults while preserving Promptkit's existing
|
||||
operator-override behavior. The capability must remain application-neutral;
|
||||
Promptkit should provide the source layer but should not own downstream profile
|
||||
names, model assignments, or configuration policy.
|
||||
|
||||
## Downstream Use Case
|
||||
|
||||
Weatherreporter wants to embed profiles such as `weather-light`,
|
||||
`weather-balanced`, and `weather-deep`. Report prompts would select those
|
||||
logical profiles instead of naming provider- or model-specific Promptkit
|
||||
profiles directly.
|
||||
|
||||
An installation could then place a profile with the same ID in its configured
|
||||
profile directory. For example, a local `weather-light` definition could point
|
||||
to an OpenAI-compatible endpoint on the deployment network. When no operator
|
||||
definition exists, Weatherreporter's embedded definition would keep the
|
||||
application usable without additional profile files.
|
||||
|
||||
This pattern is useful beyond Weatherreporter. Any Promptkit consumer may want
|
||||
application-owned execution tiers or workload-specific defaults without
|
||||
adding domain-specific profiles to Promptkit's general built-in catalog.
|
||||
|
||||
## Current Constraint
|
||||
|
||||
Promptkit currently resolves matching profile IDs in this order:
|
||||
|
||||
1. in-memory profiles supplied through `WithProfiles`;
|
||||
2. one configured profile file, `fs.FS`, or directory source; and
|
||||
3. Promptkit's embedded built-in profiles.
|
||||
|
||||
These layers do not express the desired application-default relationship:
|
||||
|
||||
- `WithProfiles` has higher precedence than the configured source, so it would
|
||||
prevent an operator file from overriding an application profile with the
|
||||
same ID.
|
||||
- `WithProfileFS` can hold embedded application assets, but it occupies the
|
||||
configured-source layer and therefore replaces rather than sits beneath a
|
||||
configured profile directory or file.
|
||||
- adding downstream profile IDs to Promptkit's built-in catalog would make the
|
||||
library own application-specific policy.
|
||||
|
||||
A downstream application could build its own filesystem overlay, but that
|
||||
would duplicate Promptkit's profile discovery, error, and precedence behavior
|
||||
at the consumer boundary.
|
||||
|
||||
## Requested Capability
|
||||
|
||||
Add one optional application fallback profile source to engine construction.
|
||||
When present, matching profile IDs should resolve in this order:
|
||||
|
||||
1. in-memory profiles supplied through `WithProfiles`;
|
||||
2. the ordinary configured profile source selected through a profile option or
|
||||
`Config.ProfileDir`;
|
||||
3. the application fallback profile source; and
|
||||
4. Promptkit's embedded built-in profiles.
|
||||
|
||||
When no application fallback is configured, existing source precedence and
|
||||
behavior must remain unchanged.
|
||||
|
||||
The minimum useful public surface is an `fs.FS`-backed option because consumers
|
||||
can embed YAML profile assets. A possible API shape is:
|
||||
|
||||
```go
|
||||
promptkit.WithFallbackProfileFS(profileFS, ".")
|
||||
```
|
||||
|
||||
The name is illustrative rather than prescriptive. A companion option for
|
||||
validated `Profile` values could be added if Promptkit maintainers find it
|
||||
generally useful, but it is not required for the Weatherreporter use case.
|
||||
|
||||
## Required Semantics
|
||||
|
||||
- A higher-precedence source falls through only when the requested profile ID
|
||||
is absent.
|
||||
- A malformed, unreadable, duplicate, ambiguous, or otherwise invalid matching
|
||||
profile is an error and must not silently fall through.
|
||||
- The fallback source uses the existing strict profile YAML format and profile
|
||||
validation rules.
|
||||
- Profile values are selected as a whole. This feature does not merge,
|
||||
inherit, or partially overlay profile definitions.
|
||||
- `InspectProfile`, `Prepare`, prepared execution, and ordinary execution use
|
||||
the same profile-source precedence.
|
||||
- An explicit request profile continues to take precedence over a prompt's
|
||||
`default_profile`; this request concerns definition lookup after the profile
|
||||
ID has been selected.
|
||||
- Repeated fallback-source options should follow Promptkit's documented
|
||||
same-category option convention, normally with the last value replacing the
|
||||
earlier value.
|
||||
- A canceled lookup, invalid fallback asset, or unknown resolved backend should
|
||||
continue to cross the public facade through Promptkit's existing public error
|
||||
identities.
|
||||
- Exact profile inspection must remain side-effect free and must not contact a
|
||||
model provider.
|
||||
|
||||
## Application And Library Boundaries
|
||||
|
||||
Promptkit should own:
|
||||
|
||||
- the additional repository layer;
|
||||
- deterministic lookup and fallthrough behavior;
|
||||
- validation of the supplied source through the existing profile contract;
|
||||
- consistent use of the layer across inspection and execution; and
|
||||
- public documentation and tests for the added precedence rule.
|
||||
|
||||
The consuming application should continue to own:
|
||||
|
||||
- whether it supplies fallback profiles;
|
||||
- the profile IDs and their domain meaning;
|
||||
- embedded profile contents and model choices;
|
||||
- application configuration and override policy;
|
||||
- report- or workload-to-profile assignment; and
|
||||
- credential checks and operator-facing errors beyond Promptkit's public
|
||||
contract.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This request does not ask Promptkit to add:
|
||||
|
||||
- Weatherreporter-specific profile IDs to its built-in catalog;
|
||||
- profile inheritance, aliases, or field-level merging;
|
||||
- automatic endpoint discovery or availability probing;
|
||||
- provider failover or fallback from a failed selected profile;
|
||||
- per-request model benchmarking or tier selection;
|
||||
- application configuration discovery; or
|
||||
- eager validation of every profile in every source.
|
||||
|
||||
## Compatibility
|
||||
|
||||
The feature can be additive. Engines that do not configure an application
|
||||
fallback source should retain their current public behavior and precedence.
|
||||
Existing uses of `WithProfiles`, `WithProfileFile`, `WithProfileFS`, and
|
||||
`Config.ProfileDir` should not change meaning.
|
||||
|
||||
The application fallback is deliberately lower precedence than every existing
|
||||
consumer-configured source. This preserves the established expectation that a
|
||||
custom profile definition can override a packaged default with the same ID.
|
||||
|
||||
## Acceptance Criteria
|
||||
|
||||
The capability is sufficient for downstream adoption when Promptkit can
|
||||
demonstrate that:
|
||||
|
||||
- a fallback-only profile can be inspected and used for preparation and
|
||||
execution;
|
||||
- a configured directory, file, or `fs.FS` profile with the same ID overrides
|
||||
the fallback profile;
|
||||
- an absent configured profile falls through to the application fallback;
|
||||
- an invalid configured match fails instead of falling through;
|
||||
- an absent application fallback profile continues to resolve from Promptkit's
|
||||
built-in catalog;
|
||||
- `WithProfiles` retains highest precedence;
|
||||
- behavior is identical across inspection, preparation, and execution; and
|
||||
- omitting the new option preserves existing tests and public contracts.
|
||||
@@ -128,8 +128,7 @@ It is not a source for deterministic weather facts.
|
||||
| --- | --- | --- | --- |
|
||||
| `.GeneratedText.Summary` | `string` | `string` | Required. |
|
||||
| `.GeneratedText.ForecastDiscussion` | `string` | `[]string` | Required; range over the day-style paragraph slice. |
|
||||
| `.GeneratedText.PrecipitationTiming` | `string` | `string` | Optional prose used by the precipitation partial when deterministic windows exist. |
|
||||
| `.GeneratedText.Confidence` | `string` | `string` | Optional validated prose; the current templates do not render it. |
|
||||
| `.GeneratedText.PrecipitationTiming` | `string` | `string` | Required field; an empty string represents no supported prose. The precipitation partial uses nonempty prose only when deterministic windows exist. |
|
||||
|
||||
The JSON schema rejects unknown properties and defines the required fields, but
|
||||
the schema body and validation behavior are documented in [Generated Text
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
# Troubleshooting
|
||||
|
||||
Keep failed workspace artifacts in place. When a RunID is available, start
|
||||
with `weatherreporter inspect metadata RUN_ID` and use the paths in its result.
|
||||
Start with the command's classified error. When content-rich prompt diagnostics
|
||||
are needed, enable a new run with `--llm-debug-dir` and handle the resulting
|
||||
secure capture as sensitive. Current-version workspace receipts can provide
|
||||
additional context when present, but are transitional state rather than a
|
||||
long-term troubleshooting interface.
|
||||
|
||||
## Prompt inspection or credentials fail before collection
|
||||
|
||||
@@ -11,13 +14,37 @@ the configured `promptkit` profile or profile source, confirm the exact
|
||||
Promptkit asset is available, and supply any reported environment credential.
|
||||
Do not add provider keys to YAML. See [configuration](config.md).
|
||||
|
||||
## Local profile override is malformed or selects an unexpected model
|
||||
|
||||
`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile
|
||||
definitions. A same-ID definition replaces the embedded profile, and a malformed
|
||||
matching definition fails before collection instead of falling back. Validate
|
||||
the selected profile's YAML, ID, backend or endpoint, and model. If the model
|
||||
is unexpected, first check the global `promptkit.profile` selection and then
|
||||
look for a same-ID definition in the configured file or directory.
|
||||
|
||||
Current-version preparation and execution receipts may retain the selected
|
||||
profile ID and effective backend/model, but not an endpoint or credential.
|
||||
Use them only as supplemental context after the active command error or an
|
||||
explicit secure debug capture. See the maintained
|
||||
[local `weather-light` profile example](../examples/weather-light-local-profile.yml).
|
||||
|
||||
## Local model endpoint is unavailable
|
||||
|
||||
An endpoint-only `weather-light` override can pass preflight and still fail
|
||||
during provider preparation or execution when the local server is unavailable
|
||||
or does not accept the configured model. Start the local server, correct the
|
||||
endpoint or model in the profile, and run the command again. Weatherreporter
|
||||
does not probe endpoints or automatically use a remote profile instead.
|
||||
|
||||
## Preparation, capacity, or execution fails
|
||||
|
||||
A preparation failure occurs before provider work; an execution failure occurs
|
||||
after preparation. Both leave safe provenance and metadata when reached. A
|
||||
capacity error for one batch report does not retry that report or prevent later
|
||||
independent reports. Inspect the preparation or execution path, correct the
|
||||
profile/backend condition, and create a new run. See [operations](operations.md).
|
||||
after preparation. A capacity error for one batch report does not retry that
|
||||
report or prevent later independent reports. Correct the profile or backend
|
||||
condition identified by the bounded command error, then create a new run.
|
||||
Use explicit secure debug capture only when additional content-rich diagnostics
|
||||
are necessary. See [operations](operations.md).
|
||||
|
||||
## Generated text fails validation
|
||||
|
||||
|
||||
4
examples/weather-light-local-profile.yml
Normal file
4
examples/weather-light-local-profile.yml
Normal file
@@ -0,0 +1,4 @@
|
||||
id: weather-light
|
||||
endpoint: http://127.0.0.1:11434/v1
|
||||
model: weather-local
|
||||
timeout_seconds: 180
|
||||
2
go.mod
2
go.mod
@@ -6,7 +6,7 @@ require gopkg.in/yaml.v3 v3.0.1
|
||||
|
||||
require (
|
||||
gitea.maximumdirect.net/eric/distributor v0.5.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.4.0
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0
|
||||
)
|
||||
|
||||
require (
|
||||
|
||||
4
go.sum
4
go.sum
@@ -1,7 +1,7 @@
|
||||
gitea.maximumdirect.net/eric/distributor v0.5.0 h1:+al7Bw+kMv6V35a3Sm5rUtCTQhwOn5b9x3RsclPMKJk=
|
||||
gitea.maximumdirect.net/eric/distributor v0.5.0/go.mod h1:G03FCFZPHpsUKC6SeMgTdbfNRpPQBdyTtDUj04e1Tu8=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.4.0 h1:WHRQEt3BVBAR7hQePBaGtNXpzrs59mlr/42nQzwgOz4=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.4.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0 h1:jnpazLyyNhWrB2xzwwtUkNUfktkTdkENTwuSPnKiYrc=
|
||||
gitea.maximumdirect.net/eric/promptkit v0.5.0/go.mod h1:R95NM6fbMDGDC0/UomgnSBP6ui2ns+8SZb8bESNvrDQ=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9 h1:/rYeyO2+HrMztAmxAq9++XJtFMqSIpSsNA0yDGALYq4=
|
||||
github.com/aws/aws-sdk-go-v2 v1.41.9/go.mod h1:+HsoOEX80qAVUitj1A2DhCNTjmb3edVyuDypb6LNEeo=
|
||||
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.7.11 h1:h5+3VT69KUBK24grGuuA5saDJTj2IIjLb9au668Fo5I=
|
||||
|
||||
@@ -45,6 +45,7 @@ func newAdapter(config Config, additionalOptions ...promptkit.Option) (*Adapter,
|
||||
options := []promptkit.Option{
|
||||
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
|
||||
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
|
||||
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
|
||||
}
|
||||
if config.ProfileFile != "" {
|
||||
options = append(options, promptkit.WithProfileFile(config.ProfileFile))
|
||||
|
||||
@@ -68,11 +68,11 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
|
||||
|
||||
func TestInspectPromptAndProfile(t *testing.T) {
|
||||
adapter := newTestAdapter(t, &fakeClient{})
|
||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.0.0")
|
||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.1.0")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPrompt() error = %v", err)
|
||||
}
|
||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.0.0" || inspection.DefaultProfileID != "gemini-flash-latest" {
|
||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != "weather-balanced" {
|
||||
t.Fatalf("inspection = %#v", inspection)
|
||||
}
|
||||
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
|
||||
@@ -102,6 +102,107 @@ func TestInspectPromptAndProfile(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesAreAvailableToProductionAndTestAdapters(t *testing.T) {
|
||||
adapter, err := New(Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
for _, want := range []struct {
|
||||
id string
|
||||
backend string
|
||||
model string
|
||||
}{
|
||||
{"weather-light", "openrouter", "deepseek/deepseek-v4-flash"},
|
||||
{"weather-balanced", "openrouter", "~google/gemini-flash-latest"},
|
||||
{"weather-deep", "openrouter", "~anthropic/claude-sonnet-latest"},
|
||||
} {
|
||||
t.Run(want.id, func(t *testing.T) {
|
||||
assertProfile(t, adapter, want.id, want.backend, want.model)
|
||||
})
|
||||
}
|
||||
|
||||
testAdapter, err := newAdapterForTest(Config{}, &fakeClient{})
|
||||
if err != nil {
|
||||
t.Fatalf("newAdapterForTest() error = %v", err)
|
||||
}
|
||||
assertProfile(t, testAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||
}
|
||||
|
||||
func TestConfiguredProfilesOverrideEmbeddedFallbacks(t *testing.T) {
|
||||
file := writeProfileFile(t, `id: weather-light
|
||||
endpoint: https://local-file.example/v1
|
||||
model: file-light
|
||||
`)
|
||||
fileAdapter, err := New(Config{ProfileFile: file})
|
||||
if err != nil {
|
||||
t.Fatalf("New(profile file) error = %v", err)
|
||||
}
|
||||
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
|
||||
|
||||
directory := testProfileDirectory(t, `id: weather-light
|
||||
backend: local
|
||||
model: directory-light
|
||||
`)
|
||||
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
|
||||
if err != nil {
|
||||
t.Fatalf("New(profile directory) error = %v", err)
|
||||
}
|
||||
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
|
||||
}
|
||||
|
||||
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
|
||||
adapter, err := New(Config{ProfileFile: filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml")})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
assertProfile(t, adapter, "weather-light", "", "weather-local")
|
||||
}
|
||||
|
||||
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
|
||||
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile
|
||||
backend: openrouter
|
||||
model: other-model
|
||||
`)})
|
||||
if err != nil {
|
||||
t.Fatalf("New(absent profile) error = %v", err)
|
||||
}
|
||||
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||
|
||||
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light
|
||||
backend: openrouter
|
||||
`)})
|
||||
if err != nil {
|
||||
t.Fatalf("New(malformed profile) error = %v", err)
|
||||
}
|
||||
if _, err := malformedAdapter.InspectProfile(context.Background(), "weather-light"); err == nil {
|
||||
t.Fatal("InspectProfile() error = nil, want malformed configured profile error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
|
||||
adapter, err := New(Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
builtin, err := adapter.InspectProfile(context.Background(), "gemini-flash-latest")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile(builtin) error = %v", err)
|
||||
}
|
||||
if builtin.ProfileID != "gemini-flash-latest" || builtin.BackendID != "openrouter" || builtin.ModelName == "" {
|
||||
t.Fatalf("builtin profile = %#v", builtin)
|
||||
}
|
||||
|
||||
explicit, err := newAdapter(Config{}, promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "weather-light",
|
||||
Endpoint: "https://explicit.example/v1",
|
||||
Model: "explicit-light",
|
||||
}))
|
||||
if err != nil {
|
||||
t.Fatalf("newAdapter(explicit profile) error = %v", err)
|
||||
}
|
||||
assertProfile(t, explicit, "weather-light", "", "explicit-light")
|
||||
}
|
||||
|
||||
func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
|
||||
client := &fakeClient{response: validResponse()}
|
||||
adapter := newTestAdapter(t, client)
|
||||
@@ -141,6 +242,44 @@ func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
|
||||
t.Setenv("OPENROUTER_API_KEY", "test-openrouter-key")
|
||||
client := &fakeClient{response: hourlyValidResponse()}
|
||||
adapter, err := newAdapter(Config{}, promptkit.WithLLMClient(client))
|
||||
if err != nil {
|
||||
t.Fatalf("newAdapter() error = %v", err)
|
||||
}
|
||||
request := promptexec.ExecuteRequest{
|
||||
PromptID: "weather.hourly_generated_text",
|
||||
PromptVersion: "1.1.0",
|
||||
ProfileID: "weather-light",
|
||||
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
|
||||
DataPackagePath: "data-packages/hourly/data_package.yaml",
|
||||
}
|
||||
var preparation promptexec.Preparation
|
||||
prepared := false
|
||||
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
|
||||
if client.callCount() != 0 {
|
||||
t.Fatal("provider was called before preparation completed")
|
||||
}
|
||||
preparation = value
|
||||
prepared = true
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
if !prepared || preparation.ProfileID != "weather-light" || preparation.BackendID != "openrouter" || preparation.ModelName != "deepseek/deepseek-v4-flash" {
|
||||
t.Fatalf("preparation = %#v", preparation)
|
||||
}
|
||||
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "openrouter" || result.ModelName != "deepseek/deepseek-v4-flash" || result.Validation.Status != promptexec.ValidationPassed {
|
||||
t.Fatalf("execution = %#v", result)
|
||||
}
|
||||
if client.callCount() != 1 || client.request().Target.Model != "deepseek/deepseek-v4-flash" {
|
||||
t.Fatalf("provider calls/request = %d/%#v", client.callCount(), client.request())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUsesExactInlineDataPackageProvenance(t *testing.T) {
|
||||
client := &fakeClient{response: validResponse()}
|
||||
reader := &recordingReader{}
|
||||
@@ -339,6 +478,17 @@ func newTestAdapter(t *testing.T, client promptkit.LLMClient) *Adapter {
|
||||
return newTestAdapterWithOptions(t, client)
|
||||
}
|
||||
|
||||
func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, model string) {
|
||||
t.Helper()
|
||||
profile, err := adapter.InspectProfile(context.Background(), id)
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile(%q) error = %v", id, err)
|
||||
}
|
||||
if profile.ProfileID != id || profile.BackendID != backend || profile.ModelName != model {
|
||||
t.Fatalf("profile = %#v, want %q with backend/model %q/%q", profile, id, backend, model)
|
||||
}
|
||||
}
|
||||
|
||||
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
|
||||
t.Helper()
|
||||
profiles := testProfileDirectory(t, `id: test-profile
|
||||
@@ -366,10 +516,19 @@ func testProfileDirectory(t *testing.T, profile string) string {
|
||||
return profiles
|
||||
}
|
||||
|
||||
func writeProfileFile(t *testing.T, profile string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "profile.yml")
|
||||
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
|
||||
t.Fatalf("write profile: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func testExecuteRequest() promptexec.ExecuteRequest {
|
||||
return promptexec.ExecuteRequest{
|
||||
PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "1.0.0",
|
||||
PromptVersion: "1.1.0",
|
||||
ProfileID: "test-profile",
|
||||
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
|
||||
DataPackagePath: "data-packages/daily/data_package.yaml",
|
||||
@@ -378,7 +537,14 @@ func testExecuteRequest() promptexec.ExecuteRequest {
|
||||
|
||||
func validResponse() *promptkit.GenerateResponse {
|
||||
return &promptkit.GenerateResponse{
|
||||
Content: `{"summary":"A quiet day is expected.","forecast_discussion":["High pressure keeps conditions settled."],"confidence":"High."}`,
|
||||
Content: `{"summary":"A quiet day is expected.","forecast_discussion":["High pressure keeps conditions settled."],"precipitation_timing":""}`,
|
||||
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
|
||||
}
|
||||
}
|
||||
|
||||
func hourlyValidResponse() *promptkit.GenerateResponse {
|
||||
return &promptkit.GenerateResponse{
|
||||
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,
|
||||
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,10 +33,10 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
|
||||
t.Fatalf("batchInspectionCandidates() error = %v", err)
|
||||
}
|
||||
executor := &inspectionExecutor{profiles: map[string]promptexec.ProfileInspection{
|
||||
"default-profile": {ProfileID: "default-profile", BackendID: "local", ModelName: "model"},
|
||||
"weather-balanced": {ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
|
||||
}, prompts: map[string]promptexec.PromptInspection{}}
|
||||
for _, candidate := range candidates {
|
||||
executor.prompts[candidate.Definition.PromptID] = validPromptInspection(candidate.Definition)
|
||||
executor.prompts[candidate.Definition.PromptID] = logicalPromptInspection(candidate.Definition)
|
||||
}
|
||||
collector := collectorFunc(func(context.Context, collect.Request) (*collect.Result, error) {
|
||||
return nil, errors.New("collection reached")
|
||||
@@ -47,7 +47,7 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
|
||||
if err == nil || err.Error() != "collection reached" {
|
||||
t.Fatalf("RunBatchDetailed() error = %v, want collection error", err)
|
||||
}
|
||||
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 {
|
||||
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
|
||||
t.Fatalf("inspection calls = prompts %#v profiles %#v", executor.promptRequests, executor.profileRequests)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -44,7 +44,7 @@ func (e *assembledBatchExecutor) InspectPrompt(_ context.Context, id, version st
|
||||
if !ok || definition.PromptVersion != version {
|
||||
return promptexec.PromptInspection{}, errors.New("unexpected prompt inspection")
|
||||
}
|
||||
return validPromptInspection(definition), nil
|
||||
return logicalPromptInspection(definition), nil
|
||||
}
|
||||
|
||||
func (e *assembledBatchExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
||||
@@ -151,8 +151,8 @@ func TestRunBatchDetailedExecutesRetainedReportsSequentially(t *testing.T) {
|
||||
if len(executor.executeRequests) != len(test.wantIDs) || executor.maxActive != 1 {
|
||||
t.Fatalf("executor calls/max active = %d/%d, want %d/1", len(executor.executeRequests), executor.maxActive, len(test.wantIDs))
|
||||
}
|
||||
if len(executor.profileRequests) != 1 {
|
||||
t.Fatalf("profile inspections = %#v, want one shared profile inspection", executor.profileRequests)
|
||||
if len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
|
||||
t.Fatalf("profile inspections = %#v, want one shared weather-balanced inspection", executor.profileRequests)
|
||||
}
|
||||
for index, item := range result.Reports {
|
||||
if item.ReportID != test.wantIDs[index] || item.Status != "succeeded" {
|
||||
|
||||
@@ -119,7 +119,7 @@ func (e artifactPathExecutor) Execute(_ context.Context, req promptexec.ExecuteR
|
||||
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID,
|
||||
BackendID: "test", ModelName: "test-model", GeneratedHash: "generated-hash",
|
||||
StartedAt: now, EndedAt: now, DataPackagePath: req.DataPackagePath,
|
||||
RawOutput: []byte(`{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`),
|
||||
RawOutput: []byte(`{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."],"precipitation_timing":"Rain is most likely during the afternoon."}`),
|
||||
Validation: promptexec.NewValidation(validation, "json_schema", "daily.generated_text.schema.json", nil),
|
||||
}, nil
|
||||
}
|
||||
|
||||
@@ -189,3 +189,13 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
|
||||
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
|
||||
}
|
||||
}
|
||||
|
||||
func logicalPromptInspection(definition report.Definition) promptexec.PromptInspection {
|
||||
inspection := validPromptInspection(definition)
|
||||
if definition.ID == report.Hourly {
|
||||
inspection.DefaultProfileID = "weather-light"
|
||||
} else {
|
||||
inspection.DefaultProfileID = "weather-balanced"
|
||||
}
|
||||
return inspection
|
||||
}
|
||||
|
||||
73
internal/app/prompt_profile_integration_test.go
Normal file
73
internal/app/prompt_profile_integration_test.go
Normal file
@@ -0,0 +1,73 @@
|
||||
package app_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
promptkitadapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/promptkit"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/app"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
|
||||
)
|
||||
|
||||
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
|
||||
lookupEnv := func(string) (string, bool) { return "test-key", true }
|
||||
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
|
||||
t.Helper()
|
||||
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||
Resolved: resolvedPromptProfile(t, id),
|
||||
Executor: adapter,
|
||||
Promptkit: config.PromptkitConfig{Profile: profile},
|
||||
LookupEnv: lookupEnv,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||
}
|
||||
if result.ProfileID != wantID || result.BackendID != wantBackend || result.ModelName != wantModel {
|
||||
t.Fatalf("inspection = %#v, want profile/backend/model %q/%q/%q", result, wantID, wantBackend, wantModel)
|
||||
}
|
||||
}
|
||||
|
||||
embedded, err := promptkitadapter.New(promptkitadapter.Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New(embedded) error = %v", err)
|
||||
}
|
||||
inspect(t, embedded, report.Hourly, "", "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||
inspect(t, embedded, report.Daily, "", "weather-balanced", "openrouter", "~google/gemini-flash-latest")
|
||||
inspect(t, embedded, report.Daily, "weather-deep", "weather-deep", "openrouter", "~anthropic/claude-sonnet-latest")
|
||||
|
||||
override, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: writeProfileFile(t, `id: weather-light
|
||||
endpoint: https://local.example/v1
|
||||
model: local-weather
|
||||
`)})
|
||||
if err != nil {
|
||||
t.Fatalf("New(override) error = %v", err)
|
||||
}
|
||||
inspect(t, override, report.Hourly, "", "weather-light", "", "local-weather")
|
||||
}
|
||||
|
||||
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
|
||||
t.Helper()
|
||||
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)
|
||||
request := report.ResolveRequest{Now: now, Location: time.UTC}
|
||||
if id == report.Daily {
|
||||
request.Date = now
|
||||
}
|
||||
resolved, err := report.DefaultRegistry().Resolve(id, request)
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(%q) error = %v", id, err)
|
||||
}
|
||||
return resolved
|
||||
}
|
||||
|
||||
func writeProfileFile(t *testing.T, profile string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "profile.yml")
|
||||
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
|
||||
t.Fatalf("write profile: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
@@ -37,8 +37,10 @@ func (c *workflowCollector) Run(context.Context, collect.Request) (*collect.Resu
|
||||
type workflowExecutor struct {
|
||||
definition report.Definition
|
||||
raw []byte
|
||||
prompt promptexec.PromptInspection
|
||||
inspectionErr error
|
||||
profile promptexec.ProfileInspection
|
||||
profileErr error
|
||||
beforePreparationErr error
|
||||
afterCallbackErr error
|
||||
afterPreparationErr error
|
||||
@@ -49,6 +51,8 @@ type workflowExecutor struct {
|
||||
beforeProvider func()
|
||||
preparationDebug *promptexec.PreparationDebug
|
||||
executionDebug *promptexec.ExecutionDebug
|
||||
preparation *promptexec.Preparation
|
||||
execution *promptexec.Execution
|
||||
}
|
||||
|
||||
func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string) (promptexec.PromptInspection, error) {
|
||||
@@ -58,10 +62,16 @@ func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string)
|
||||
if id != e.definition.PromptID || version != e.definition.PromptVersion {
|
||||
return promptexec.PromptInspection{}, errors.New("unexpected prompt identity")
|
||||
}
|
||||
if e.prompt.PromptID != "" {
|
||||
return e.prompt, nil
|
||||
}
|
||||
return validPromptInspection(e.definition), nil
|
||||
}
|
||||
|
||||
func (e *workflowExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
||||
if e.profileErr != nil {
|
||||
return promptexec.ProfileInspection{}, e.profileErr
|
||||
}
|
||||
profile := e.profile
|
||||
if profile.ProfileID == "" {
|
||||
profile = promptexec.ProfileInspection{ProfileID: id, BackendID: "fixture", ModelName: "fixture-model"}
|
||||
@@ -76,11 +86,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
|
||||
return nil, e.beforePreparationErr
|
||||
}
|
||||
stamp := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
|
||||
profile := e.profile
|
||||
if profile.ProfileID == "" {
|
||||
profile = promptexec.ProfileInspection{ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model"}
|
||||
}
|
||||
preparation := promptexec.Preparation{
|
||||
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: "prompt-hash",
|
||||
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture",
|
||||
ModelName: "fixture-model", DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
|
||||
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: profile.BackendID,
|
||||
ModelName: profile.ModelName, DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
|
||||
}
|
||||
e.preparation = &preparation
|
||||
if err := callback(preparation, e.preparationDebug); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -98,14 +113,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
|
||||
if validation == "" {
|
||||
validation = promptexec.ValidationPassed
|
||||
}
|
||||
return &promptexec.Execution{
|
||||
execution := &promptexec.Execution{
|
||||
RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion,
|
||||
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID,
|
||||
BackendID: "fixture", ModelName: "fixture-model", GeneratedHash: "generated-hash",
|
||||
BackendID: profile.BackendID, ModelName: profile.ModelName, GeneratedHash: "generated-hash",
|
||||
StartedAt: stamp, EndedAt: stamp, DataPackagePath: req.DataPackagePath, RawOutput: e.raw,
|
||||
Debug: e.executionDebug,
|
||||
Validation: promptexec.NewValidation(validation, "json_schema", e.definition.GeneratedTextSchemaID+".generated_text.schema.json", nil),
|
||||
}, nil
|
||||
}
|
||||
e.execution = execution
|
||||
return execution, nil
|
||||
}
|
||||
|
||||
type workflowNotifier struct {
|
||||
@@ -208,6 +225,57 @@ func TestGenerateDetailedCompletesRetainedReportWorkflows(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateDetailedPreservesSelectedProfileThroughExecution(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
kind ReportKind
|
||||
id report.ID
|
||||
raw string
|
||||
override string
|
||||
profile promptexec.ProfileInspection
|
||||
}{
|
||||
{
|
||||
name: "hourly default", kind: ReportHourly, id: report.Hourly, raw: validHourlyWorkflowJSON(),
|
||||
profile: promptexec.ProfileInspection{ProfileID: "weather-light", BackendID: "openrouter", ModelName: "deepseek/deepseek-v4-flash"},
|
||||
},
|
||||
{
|
||||
name: "daily default", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(),
|
||||
profile: promptexec.ProfileInspection{ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
|
||||
},
|
||||
{
|
||||
name: "global override", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(), override: "operator-profile",
|
||||
profile: promptexec.ProfileInspection{ProfileID: "operator-profile", BackendID: "local", ModelName: "local-weather-model"},
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
cfg := workflowConfig(t)
|
||||
cfg.Promptkit.Profile = test.override
|
||||
definition := report.DefaultRegistry().MustLookup(test.id)
|
||||
executor := &workflowExecutor{
|
||||
definition: definition, prompt: logicalPromptInspection(definition), profile: test.profile, raw: []byte(test.raw),
|
||||
}
|
||||
bundle := workflowBundle(t)
|
||||
_, err := GenerateDetailed(context.Background(), GenerateRequest{
|
||||
Config: cfg, Report: test.kind, Date: workflowTime("2026-05-29T12:00:00-05:00"), Now: workflowTime("2026-05-29T08:30:00-05:00"),
|
||||
Collector: &workflowCollector{result: &collect.Result{Bundle: &bundle}}, Executor: executor, Notifier: &workflowNotifier{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateDetailed() error = %v", err)
|
||||
}
|
||||
if executor.request.ProfileID != test.profile.ProfileID {
|
||||
t.Fatalf("execution profile = %q, want %q", executor.request.ProfileID, test.profile.ProfileID)
|
||||
}
|
||||
if executor.preparation == nil || executor.preparation.ProfileID != test.profile.ProfileID || executor.preparation.BackendID != test.profile.BackendID || executor.preparation.ModelName != test.profile.ModelName {
|
||||
t.Fatalf("prepared profile = %#v, want %q/%q/%q", executor.preparation, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
|
||||
}
|
||||
if executor.execution == nil || executor.execution.ProfileID != test.profile.ProfileID || executor.execution.BackendID != test.profile.BackendID || executor.execution.ModelName != test.profile.ModelName {
|
||||
t.Fatalf("executed profile = %#v, want %q/%q/%q", executor.execution, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type preparationFailingStore struct {
|
||||
state.Store
|
||||
}
|
||||
@@ -299,6 +367,11 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
|
||||
wantCategory promptexec.ErrorCategory
|
||||
}{
|
||||
{name: "inspection", configure: func(e *workflowExecutor) { e.inspectionErr = errors.New("inspection unavailable") }, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "unknown profile", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unknown selected profile") }, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "malformed profile", configure: func(e *workflowExecutor) {
|
||||
e.profileErr = errors.New("malformed profile at https://operator.example/v1 api_key=secret")
|
||||
}, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "unusable backend", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unsupported backend") }, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "credential", configure: func(e *workflowExecutor) {
|
||||
e.profile = promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true}
|
||||
}, wantCategory: promptexec.MissingCredential},
|
||||
@@ -317,6 +390,9 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
|
||||
if err == nil || result != nil || promptexec.CategoryOf(err) != test.wantCategory || collector.calls != 0 || executor.executeCalls != 0 {
|
||||
t.Fatalf("result/error/category/collect/execute = %#v/%v/%q/%d/%d", result, err, promptexec.CategoryOf(err), collector.calls, executor.executeCalls)
|
||||
}
|
||||
if strings.Contains(err.Error(), "operator.example") || strings.Contains(err.Error(), "secret") {
|
||||
t.Fatalf("error leaks profile details: %v", err)
|
||||
}
|
||||
entries, readErr := os.ReadDir(cfg.Workspace.Root)
|
||||
if readErr != nil || len(entries) != 0 {
|
||||
t.Fatalf("workspace entries/error = %#v/%v, want no writes before collection", entries, readErr)
|
||||
@@ -643,7 +719,7 @@ func workflowTime(value string) time.Time {
|
||||
}
|
||||
|
||||
func validHourlyWorkflowJSON() string {
|
||||
return `{"summary":"Storm chances increase through late morning.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"A cold front is moving into the region.","confidence":"Medium"}`
|
||||
return `{"summary":"Storm chances increase through late morning.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"A cold front is moving into the region."}`
|
||||
}
|
||||
|
||||
func validTomorrowWorkflowJSON() string {
|
||||
@@ -655,5 +731,5 @@ func validTodayWorkflowJSON() string {
|
||||
}
|
||||
|
||||
func validDailyWorkflowJSON() string {
|
||||
return `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`
|
||||
return `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon."}`
|
||||
}
|
||||
|
||||
@@ -147,7 +147,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
|
||||
}
|
||||
hourly, normalized, err := hourlyHandler.Validate([]byte(`{
|
||||
"summary": " Storm chances increase. ",
|
||||
"forecast_discussion": " A front will keep the region unsettled. "
|
||||
"forecast_discussion": " A front will keep the region unsettled. ",
|
||||
"precipitation_timing": ""
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("Validate(hourly) error = %v", err)
|
||||
@@ -165,7 +166,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
|
||||
}
|
||||
tomorrow, normalized, err := tomorrowHandler.Validate([]byte(`{
|
||||
"summary": " Storms become more likely tomorrow. ",
|
||||
"forecast_discussion": [" A front will keep showers in the forecast. ", ""]
|
||||
"forecast_discussion": [" A front will keep showers in the forecast. ", ""],
|
||||
"precipitation_timing": ""
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("Validate(tomorrow) error = %v", err)
|
||||
@@ -187,7 +189,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
|
||||
}
|
||||
today, normalized, err := todayHandler.Validate([]byte(`{
|
||||
"summary": " Showers are likely today. ",
|
||||
"forecast_discussion": [" A front will keep rain chances elevated. ", ""]
|
||||
"forecast_discussion": [" A front will keep rain chances elevated. ", ""],
|
||||
"precipitation_timing": ""
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("Validate(today) error = %v", err)
|
||||
@@ -209,7 +212,8 @@ func TestCatalogValidationDispatchSupportsKnownSchemas(t *testing.T) {
|
||||
}
|
||||
daily, normalized, err := dailyHandler.Validate([]byte(`{
|
||||
"summary": " Showers are possible during the selected day. ",
|
||||
"forecast_discussion": [" A front will keep rain chances in the forecast. ", ""]
|
||||
"forecast_discussion": [" A front will keep rain chances in the forecast. ", ""],
|
||||
"precipitation_timing": ""
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("Validate(daily) error = %v", err)
|
||||
|
||||
@@ -3,8 +3,7 @@ package generatedtext
|
||||
type Daily struct {
|
||||
Summary string `json:"summary"`
|
||||
ForecastDiscussion []string `json:"forecast_discussion"`
|
||||
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
|
||||
Confidence string `json:"confidence,omitempty"`
|
||||
PrecipitationTiming string `json:"precipitation_timing"`
|
||||
}
|
||||
|
||||
func ValidateDaily(data []byte) (Daily, []byte, error) {
|
||||
@@ -16,7 +15,6 @@ func (d *Daily) dayStyleFields() dayStyleFields {
|
||||
Summary: d.Summary,
|
||||
ForecastDiscussion: d.ForecastDiscussion,
|
||||
PrecipitationTiming: d.PrecipitationTiming,
|
||||
Confidence: d.Confidence,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,5 +22,4 @@ func (d *Daily) setDayStyleFields(fields dayStyleFields) {
|
||||
d.Summary = fields.Summary
|
||||
d.ForecastDiscussion = fields.ForecastDiscussion
|
||||
d.PrecipitationTiming = fields.PrecipitationTiming
|
||||
d.Confidence = fields.Confidence
|
||||
}
|
||||
|
||||
@@ -13,8 +13,7 @@ func TestValidateDailyNormalizesJSON(t *testing.T) {
|
||||
"",
|
||||
" Temperatures stay seasonable by afternoon. "
|
||||
],
|
||||
"precipitation_timing": " Rain is most likely during the afternoon. ",
|
||||
"confidence": " Medium "
|
||||
"precipitation_timing": " Rain is most likely during the afternoon. "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateDaily() error = %v", err)
|
||||
@@ -28,23 +27,22 @@ func TestValidateDailyNormalizesJSON(t *testing.T) {
|
||||
if value.PrecipitationTiming != "Rain is most likely during the afternoon." {
|
||||
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
|
||||
}
|
||||
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`
|
||||
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely during the afternoon."}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateDailyOmitsEmptyOptionalFields(t *testing.T) {
|
||||
func TestValidateDailyPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
|
||||
_, normalized, err := ValidateDaily([]byte(`{
|
||||
"summary": "Showers are possible during the selected day.",
|
||||
"forecast_discussion": ["A front will keep rain chances in the forecast."],
|
||||
"precipitation_timing": " ",
|
||||
"confidence": " "
|
||||
"precipitation_timing": " "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateDaily() error = %v", err)
|
||||
}
|
||||
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."]}`
|
||||
want := `{"summary":"Showers are possible during the selected day.","forecast_discussion":["A front will keep rain chances in the forecast."],"precipitation_timing":""}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
|
||||
@@ -9,7 +9,6 @@ type dayStyleFields struct {
|
||||
Summary string
|
||||
ForecastDiscussion []string
|
||||
PrecipitationTiming string
|
||||
Confidence string
|
||||
}
|
||||
|
||||
type dayStyleGeneratedText interface {
|
||||
@@ -31,7 +30,6 @@ func validateDayStyleGeneratedText[T any, PT interface {
|
||||
fields := pointer.dayStyleFields()
|
||||
fields.Summary = strings.TrimSpace(fields.Summary)
|
||||
fields.PrecipitationTiming = strings.TrimSpace(fields.PrecipitationTiming)
|
||||
fields.Confidence = strings.TrimSpace(fields.Confidence)
|
||||
fields.ForecastDiscussion = trimNonEmpty(fields.ForecastDiscussion)
|
||||
if fields.Summary == "" {
|
||||
var zero T
|
||||
@@ -41,6 +39,10 @@ func validateDayStyleGeneratedText[T any, PT interface {
|
||||
var zero T
|
||||
return zero, nil, fmt.Errorf("%s generated text forecast discussion is required", name)
|
||||
}
|
||||
if err := requireGeneratedTextStringField(data, name, "precipitation_timing"); err != nil {
|
||||
var zero T
|
||||
return zero, nil, err
|
||||
}
|
||||
pointer.setDayStyleFields(fields)
|
||||
|
||||
normalized, err := normalizeGeneratedText(value, name)
|
||||
|
||||
@@ -60,8 +60,7 @@ func TestValidateDayStyleGeneratedTextSharedBehavior(t *testing.T) {
|
||||
"",
|
||||
" Second paragraph. "
|
||||
],
|
||||
"precipitation_timing": " Afternoon. ",
|
||||
"confidence": " Medium "
|
||||
"precipitation_timing": " Afternoon. "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("validate() error = %v", err)
|
||||
@@ -76,31 +75,35 @@ func TestValidateDayStyleGeneratedTextSharedBehavior(t *testing.T) {
|
||||
if fields.PrecipitationTiming != "Afternoon." {
|
||||
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", fields.PrecipitationTiming)
|
||||
}
|
||||
if fields.Confidence != "Medium" {
|
||||
t.Fatalf("Confidence = %q, want trimmed confidence", fields.Confidence)
|
||||
}
|
||||
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph.","Second paragraph."],"precipitation_timing":"Afternoon.","confidence":"Medium"}`
|
||||
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph.","Second paragraph."],"precipitation_timing":"Afternoon."}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("omits empty optional fields", func(t *testing.T) {
|
||||
t.Run("preserves required empty precipitation timing", func(t *testing.T) {
|
||||
_, normalized, err := report.validate([]byte(`{
|
||||
"summary": "Shared summary.",
|
||||
"forecast_discussion": ["First paragraph."],
|
||||
"precipitation_timing": " ",
|
||||
"confidence": " "
|
||||
"precipitation_timing": " "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("validate() error = %v", err)
|
||||
}
|
||||
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph."]}`
|
||||
want := `{"summary":"Shared summary.","forecast_discussion":["First paragraph."],"precipitation_timing":""}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("requires precipitation timing field", func(t *testing.T) {
|
||||
_, _, err := report.validate([]byte(`{"summary":"Shared summary.","forecast_discussion":["First paragraph."]}`))
|
||||
want := fmt.Sprintf("%s generated text precipitation timing is required", report.name)
|
||||
if err == nil || err.Error() != want {
|
||||
t.Fatalf("validate() error = %v, want %q", err, want)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("rejects unknown fields", func(t *testing.T) {
|
||||
_, _, err := report.validate([]byte(`{"summary":"Shared summary.","forecast_discussion":["First paragraph."],"extra":"value"}`))
|
||||
if err == nil {
|
||||
@@ -110,6 +113,13 @@ func TestValidateDayStyleGeneratedTextSharedBehavior(t *testing.T) {
|
||||
t.Fatalf("validate() error = %v, want unknown field error", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("rejects retired confidence field", func(t *testing.T) {
|
||||
_, _, err := report.validate([]byte(`{"summary":"Shared summary.","forecast_discussion":["First paragraph."],"precipitation_timing":"","confidence":"Medium"}`))
|
||||
if err == nil || !strings.Contains(err.Error(), `unknown field "confidence"`) {
|
||||
t.Fatalf("validate() error = %v, want retired confidence field rejection", err)
|
||||
}
|
||||
})
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
@@ -9,8 +9,7 @@ import (
|
||||
type Hourly struct {
|
||||
Summary string `json:"summary"`
|
||||
ForecastDiscussion string `json:"forecast_discussion"`
|
||||
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
|
||||
Confidence string `json:"confidence,omitempty"`
|
||||
PrecipitationTiming string `json:"precipitation_timing"`
|
||||
}
|
||||
|
||||
func ValidateHourly(data []byte) (Hourly, []byte, error) {
|
||||
@@ -22,13 +21,15 @@ func ValidateHourly(data []byte) (Hourly, []byte, error) {
|
||||
value.Summary = strings.TrimSpace(value.Summary)
|
||||
value.ForecastDiscussion = strings.TrimSpace(value.ForecastDiscussion)
|
||||
value.PrecipitationTiming = strings.TrimSpace(value.PrecipitationTiming)
|
||||
value.Confidence = strings.TrimSpace(value.Confidence)
|
||||
if value.Summary == "" {
|
||||
return Hourly{}, nil, fmt.Errorf("hourly generated text summary is required")
|
||||
}
|
||||
if value.ForecastDiscussion == "" {
|
||||
return Hourly{}, nil, fmt.Errorf("hourly generated text forecast discussion is required")
|
||||
}
|
||||
if err := requireGeneratedTextStringField(data, "hourly", "precipitation_timing"); err != nil {
|
||||
return Hourly{}, nil, err
|
||||
}
|
||||
|
||||
normalized, err := normalizeGeneratedText(value, "hourly")
|
||||
if err != nil {
|
||||
|
||||
@@ -9,8 +9,7 @@ func TestValidateHourlyNormalizesJSON(t *testing.T) {
|
||||
value, normalized, err := ValidateHourly([]byte(`{
|
||||
"summary": " Storm chances increase. ",
|
||||
"forecast_discussion": " A front will keep the region unsettled. ",
|
||||
"precipitation_timing": " Showers are most likely early this afternoon. ",
|
||||
"confidence": " Medium "
|
||||
"precipitation_timing": " Showers are most likely early this afternoon. "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateHourly() error = %v", err)
|
||||
@@ -24,23 +23,22 @@ func TestValidateHourlyNormalizesJSON(t *testing.T) {
|
||||
if value.PrecipitationTiming != "Showers are most likely early this afternoon." {
|
||||
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
|
||||
}
|
||||
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"Showers are most likely early this afternoon.","confidence":"Medium"}`
|
||||
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"Showers are most likely early this afternoon."}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateHourlyOmitsEmptyConfidence(t *testing.T) {
|
||||
func TestValidateHourlyPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
|
||||
_, normalized, err := ValidateHourly([]byte(`{
|
||||
"summary": "Storm chances increase.",
|
||||
"forecast_discussion": "A front will keep the region unsettled.",
|
||||
"precipitation_timing": " ",
|
||||
"confidence": " "
|
||||
"precipitation_timing": " "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateHourly() error = %v", err)
|
||||
}
|
||||
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled."}`
|
||||
want := `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":""}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
@@ -72,6 +70,21 @@ func TestValidateHourlyRejectsInvalidInput(t *testing.T) {
|
||||
in: `{"summary":"Storm chances increase.","forecast_discussion":" "}`,
|
||||
want: "forecast discussion is required",
|
||||
},
|
||||
{
|
||||
name: "missing precipitation timing",
|
||||
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled."}`,
|
||||
want: "precipitation timing is required",
|
||||
},
|
||||
{
|
||||
name: "null precipitation timing",
|
||||
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":null}`,
|
||||
want: "precipitation timing must be a string",
|
||||
},
|
||||
{
|
||||
name: "retired confidence field rejected",
|
||||
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","precipitation_timing":"","confidence":"Medium"}`,
|
||||
want: `unknown field "confidence"`,
|
||||
},
|
||||
{
|
||||
name: "old timing field rejected",
|
||||
in: `{"summary":"Storm chances increase.","forecast_discussion":"A front will keep the region unsettled.","timing":"Late morning."}`,
|
||||
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"io"
|
||||
"strings"
|
||||
)
|
||||
|
||||
func decodeGeneratedText[T any](data []byte, name string) (T, error) {
|
||||
@@ -24,6 +25,21 @@ func decodeGeneratedText[T any](data []byte, name string) (T, error) {
|
||||
return value, fmt.Errorf("decode %s generated text: multiple JSON values", name)
|
||||
}
|
||||
|
||||
func requireGeneratedTextStringField(data []byte, name, field string) error {
|
||||
var fields map[string]json.RawMessage
|
||||
if err := json.Unmarshal(data, &fields); err != nil {
|
||||
return fmt.Errorf("decode %s generated text: %w", name, err)
|
||||
}
|
||||
raw, ok := fields[field]
|
||||
if !ok {
|
||||
return fmt.Errorf("%s generated text %s is required", name, strings.ReplaceAll(field, "_", " "))
|
||||
}
|
||||
if bytes.Equal(bytes.TrimSpace(raw), []byte("null")) {
|
||||
return fmt.Errorf("%s generated text %s must be a string", name, strings.ReplaceAll(field, "_", " "))
|
||||
}
|
||||
return nil
|
||||
}
|
||||
|
||||
func normalizeGeneratedText[T any](value T, name string) ([]byte, error) {
|
||||
normalized, err := json.Marshal(value)
|
||||
if err != nil {
|
||||
|
||||
@@ -21,7 +21,6 @@ func TestBuildHourlyRenderContext(t *testing.T) {
|
||||
Summary: "Storm chances increase through late morning.",
|
||||
ForecastDiscussion: "A front will keep the region unsettled.",
|
||||
PrecipitationTiming: "A cold front is moving into the region.",
|
||||
Confidence: "Medium confidence in timing.",
|
||||
}
|
||||
collected := testCollected()
|
||||
derived := testDerived()
|
||||
|
||||
@@ -3,8 +3,7 @@ package generatedtext
|
||||
type Today struct {
|
||||
Summary string `json:"summary"`
|
||||
ForecastDiscussion []string `json:"forecast_discussion"`
|
||||
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
|
||||
Confidence string `json:"confidence,omitempty"`
|
||||
PrecipitationTiming string `json:"precipitation_timing"`
|
||||
}
|
||||
|
||||
func ValidateToday(data []byte) (Today, []byte, error) {
|
||||
@@ -16,7 +15,6 @@ func (t *Today) dayStyleFields() dayStyleFields {
|
||||
Summary: t.Summary,
|
||||
ForecastDiscussion: t.ForecastDiscussion,
|
||||
PrecipitationTiming: t.PrecipitationTiming,
|
||||
Confidence: t.Confidence,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,5 +22,4 @@ func (t *Today) setDayStyleFields(fields dayStyleFields) {
|
||||
t.Summary = fields.Summary
|
||||
t.ForecastDiscussion = fields.ForecastDiscussion
|
||||
t.PrecipitationTiming = fields.PrecipitationTiming
|
||||
t.Confidence = fields.Confidence
|
||||
}
|
||||
|
||||
@@ -13,8 +13,7 @@ func TestValidateTodayNormalizesJSON(t *testing.T) {
|
||||
"",
|
||||
" Temperatures stay mild through the afternoon. "
|
||||
],
|
||||
"precipitation_timing": " Rain is most likely during the afternoon. ",
|
||||
"confidence": " Medium "
|
||||
"precipitation_timing": " Rain is most likely during the afternoon. "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateToday() error = %v", err)
|
||||
@@ -28,23 +27,22 @@ func TestValidateTodayNormalizesJSON(t *testing.T) {
|
||||
if value.PrecipitationTiming != "Rain is most likely during the afternoon." {
|
||||
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
|
||||
}
|
||||
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated.","Temperatures stay mild through the afternoon."],"precipitation_timing":"Rain is most likely during the afternoon.","confidence":"Medium"}`
|
||||
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated.","Temperatures stay mild through the afternoon."],"precipitation_timing":"Rain is most likely during the afternoon."}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateTodayOmitsEmptyOptionalFields(t *testing.T) {
|
||||
func TestValidateTodayPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
|
||||
_, normalized, err := ValidateToday([]byte(`{
|
||||
"summary": "Showers are likely today.",
|
||||
"forecast_discussion": ["A front will keep rain chances elevated."],
|
||||
"precipitation_timing": " ",
|
||||
"confidence": " "
|
||||
"precipitation_timing": " "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateToday() error = %v", err)
|
||||
}
|
||||
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated."]}`
|
||||
want := `{"summary":"Showers are likely today.","forecast_discussion":["A front will keep rain chances elevated."],"precipitation_timing":""}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
|
||||
@@ -3,8 +3,7 @@ package generatedtext
|
||||
type Tomorrow struct {
|
||||
Summary string `json:"summary"`
|
||||
ForecastDiscussion []string `json:"forecast_discussion"`
|
||||
PrecipitationTiming string `json:"precipitation_timing,omitempty"`
|
||||
Confidence string `json:"confidence,omitempty"`
|
||||
PrecipitationTiming string `json:"precipitation_timing"`
|
||||
}
|
||||
|
||||
func ValidateTomorrow(data []byte) (Tomorrow, []byte, error) {
|
||||
@@ -16,7 +15,6 @@ func (t *Tomorrow) dayStyleFields() dayStyleFields {
|
||||
Summary: t.Summary,
|
||||
ForecastDiscussion: t.ForecastDiscussion,
|
||||
PrecipitationTiming: t.PrecipitationTiming,
|
||||
Confidence: t.Confidence,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -24,5 +22,4 @@ func (t *Tomorrow) setDayStyleFields(fields dayStyleFields) {
|
||||
t.Summary = fields.Summary
|
||||
t.ForecastDiscussion = fields.ForecastDiscussion
|
||||
t.PrecipitationTiming = fields.PrecipitationTiming
|
||||
t.Confidence = fields.Confidence
|
||||
}
|
||||
|
||||
@@ -13,8 +13,7 @@ func TestValidateTomorrowNormalizesJSON(t *testing.T) {
|
||||
"",
|
||||
" Temperatures stay seasonable by afternoon. "
|
||||
],
|
||||
"precipitation_timing": " Rain is most likely before sunrise. ",
|
||||
"confidence": " Medium "
|
||||
"precipitation_timing": " Rain is most likely before sunrise. "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateTomorrow() error = %v", err)
|
||||
@@ -28,23 +27,22 @@ func TestValidateTomorrowNormalizesJSON(t *testing.T) {
|
||||
if value.PrecipitationTiming != "Rain is most likely before sunrise." {
|
||||
t.Fatalf("PrecipitationTiming = %q, want trimmed precipitation timing", value.PrecipitationTiming)
|
||||
}
|
||||
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely before sunrise.","confidence":"Medium"}`
|
||||
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast.","Temperatures stay seasonable by afternoon."],"precipitation_timing":"Rain is most likely before sunrise."}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidateTomorrowOmitsEmptyOptionalFields(t *testing.T) {
|
||||
func TestValidateTomorrowPreservesRequiredEmptyPrecipitationTiming(t *testing.T) {
|
||||
_, normalized, err := ValidateTomorrow([]byte(`{
|
||||
"summary": "Storms become more likely tomorrow.",
|
||||
"forecast_discussion": ["A front will keep showers in the forecast."],
|
||||
"precipitation_timing": " ",
|
||||
"confidence": " "
|
||||
"precipitation_timing": " "
|
||||
}`))
|
||||
if err != nil {
|
||||
t.Fatalf("ValidateTomorrow() error = %v", err)
|
||||
}
|
||||
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast."]}`
|
||||
want := `{"summary":"Storms become more likely tomorrow.","forecast_discussion":["A front will keep showers in the forecast."],"precipitation_timing":""}`
|
||||
if string(normalized) != want {
|
||||
t.Fatalf("normalized = %s, want %s", normalized, want)
|
||||
}
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
id: weather-balanced
|
||||
backend: openrouter
|
||||
model: "~google/gemini-flash-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
6
internal/promptassets/assets/profiles/weather-deep.yml
Normal file
6
internal/promptassets/assets/profiles/weather-deep.yml
Normal file
@@ -0,0 +1,6 @@
|
||||
id: weather-deep
|
||||
backend: openrouter
|
||||
model: "~anthropic/claude-sonnet-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
5
internal/promptassets/assets/profiles/weather-light.yml
Normal file
5
internal/promptassets/assets/profiles/weather-light.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
id: weather-light
|
||||
backend: openrouter
|
||||
model: deepseek/deepseek-v4-flash
|
||||
timeout_seconds: 180
|
||||
service_tier: flex
|
||||
@@ -8,8 +8,7 @@ Return these fields:
|
||||
|
||||
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
|
||||
- `forecast_discussion`: required. Three paragraphs explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
|
||||
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
|
||||
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
|
||||
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
|
||||
|
||||
Return JSON only.
|
||||
|
||||
@@ -27,7 +26,7 @@ In most cases, include three paragraphs: a two-to-four sentence relevant local o
|
||||
|
||||
# Precipitation timing
|
||||
|
||||
Include this only if precipitation is forecast. Use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
|
||||
When precipitation windows are present, use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.
|
||||
|
||||
# Narrative source selection
|
||||
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.daily_generated_text
|
||||
version: "1.0.0"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Daily weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -8,8 +8,7 @@ Return these fields:
|
||||
|
||||
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
|
||||
- `forecast_discussion`: required. Two or three sentences explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
|
||||
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
|
||||
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
|
||||
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
|
||||
|
||||
Return JSON only.
|
||||
|
||||
@@ -25,4 +24,4 @@ Use narrative products to explain the “why” behind the local forecast when u
|
||||
|
||||
# Precipitation timing
|
||||
|
||||
Include this only if precipitation is forecast. Use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
|
||||
When precipitation windows are present, use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.hourly_generated_text
|
||||
version: "1.0.0"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-light
|
||||
description: Hourly weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -8,8 +8,7 @@ Return these fields:
|
||||
|
||||
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
|
||||
- `forecast_discussion`: required. Three paragraphs explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
|
||||
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
|
||||
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
|
||||
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
|
||||
|
||||
Return JSON only.
|
||||
|
||||
@@ -27,4 +26,4 @@ In most cases, include three paragraphs: a two-to-four sentence relevant local o
|
||||
|
||||
# Precipitation timing
|
||||
|
||||
Include this only if precipitation is forecast. Use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
|
||||
When precipitation windows are present, use one to four sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.today_generated_text
|
||||
version: "1.0.0"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Today's weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -8,8 +8,7 @@ Return these fields:
|
||||
|
||||
- `summary`: required. One or two sentences summarizing the main weather story for the valid period.
|
||||
- `forecast_discussion`: required. Three paragraphs explaining the broader setup, trend, or forecast reasoning most relevant to the valid period.
|
||||
- `precipitation_timing`: optional. Include only when the deterministic `precip_timing` module contains precipitation windows.
|
||||
- `confidence`: optional. Include only if uncertainty, timing spread, or conflicting signals materially affect how the reader should interpret the forecast.
|
||||
- `precipitation_timing`: required. When the deterministic `precip_timing` module contains precipitation windows, provide the supported timing prose described below. Otherwise, return an empty string (`""`).
|
||||
|
||||
Return JSON only.
|
||||
|
||||
@@ -27,4 +26,4 @@ In most cases, include three paragraphs: a two-to-four sentence relevant local o
|
||||
|
||||
# Precipitation timing
|
||||
|
||||
Use one or two sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration.
|
||||
When precipitation windows are present, use one or two sentences to give practical context about a supported frontal, convective, or stratiform setup; expected type, intensity, and duration; and uncertainty in onset or duration. When no precipitation windows are present, return an empty string (`""`) for `precipitation_timing`.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.tomorrow_generated_text
|
||||
version: "1.0.0"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Tomorrow's weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -4,11 +4,10 @@
|
||||
"title": "Daily GeneratedText",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["summary", "forecast_discussion"],
|
||||
"required": ["summary", "forecast_discussion", "precipitation_timing"],
|
||||
"properties": {
|
||||
"summary": {"type": "string"},
|
||||
"forecast_discussion": {"type": "array", "items": {"type": "string"}, "minItems": 1},
|
||||
"precipitation_timing": {"type": "string"},
|
||||
"confidence": {"type": "string"}
|
||||
"precipitation_timing": {"type": "string"}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,11 +4,10 @@
|
||||
"title": "Hourly GeneratedText",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["summary", "forecast_discussion"],
|
||||
"required": ["summary", "forecast_discussion", "precipitation_timing"],
|
||||
"properties": {
|
||||
"summary": {"type": "string"},
|
||||
"forecast_discussion": {"type": "string"},
|
||||
"precipitation_timing": {"type": "string"},
|
||||
"confidence": {"type": "string"}
|
||||
"precipitation_timing": {"type": "string"}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,11 +4,10 @@
|
||||
"title": "Today GeneratedText",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["summary", "forecast_discussion"],
|
||||
"required": ["summary", "forecast_discussion", "precipitation_timing"],
|
||||
"properties": {
|
||||
"summary": {"type": "string"},
|
||||
"forecast_discussion": {"type": "array", "items": {"type": "string"}, "minItems": 1},
|
||||
"precipitation_timing": {"type": "string"},
|
||||
"confidence": {"type": "string"}
|
||||
"precipitation_timing": {"type": "string"}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -4,11 +4,10 @@
|
||||
"title": "Tomorrow GeneratedText",
|
||||
"type": "object",
|
||||
"additionalProperties": false,
|
||||
"required": ["summary", "forecast_discussion"],
|
||||
"required": ["summary", "forecast_discussion", "precipitation_timing"],
|
||||
"properties": {
|
||||
"summary": {"type": "string"},
|
||||
"forecast_discussion": {"type": "array", "items": {"type": "string"}, "minItems": 1},
|
||||
"precipitation_timing": {"type": "string"},
|
||||
"confidence": {"type": "string"}
|
||||
"precipitation_timing": {"type": "string"}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"io/fs"
|
||||
)
|
||||
|
||||
//go:embed assets/prompts assets/schemas
|
||||
//go:embed assets/prompts assets/profiles assets/schemas
|
||||
var assets embed.FS
|
||||
|
||||
var schemaPaths = map[string]string{
|
||||
@@ -35,6 +35,15 @@ func SchemaFS() fs.FS {
|
||||
return fsys
|
||||
}
|
||||
|
||||
// ProfileFS returns the embedded Weatherreporter Promptkit profile definitions.
|
||||
func ProfileFS() fs.FS {
|
||||
fsys, err := fs.Sub(assets, "assets/profiles")
|
||||
if err != nil {
|
||||
panic(fmt.Sprintf("embedded profile assets: %v", err))
|
||||
}
|
||||
return fsys
|
||||
}
|
||||
|
||||
// Schema returns an independent copy of the canonical schema for id.
|
||||
func Schema(id string) ([]byte, error) {
|
||||
path, ok := schemaPaths[id]
|
||||
|
||||
@@ -34,11 +34,12 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
path string
|
||||
id string
|
||||
schemaID string
|
||||
profile string
|
||||
}{
|
||||
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily"},
|
||||
{"today/today_generated_text.yml", "weather.today_generated_text", "today"},
|
||||
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow"},
|
||||
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly"},
|
||||
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily", "weather-balanced"},
|
||||
{"today/today_generated_text.yml", "weather.today_generated_text", "today", "weather-balanced"},
|
||||
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow", "weather-balanced"},
|
||||
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly", "weather-light"},
|
||||
}
|
||||
|
||||
definitions := 0
|
||||
@@ -67,8 +68,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
if err := yaml.Unmarshal(data, &definition); err != nil {
|
||||
t.Fatalf("decode prompt definition: %v", err)
|
||||
}
|
||||
if definition.ID != tc.id || definition.Version != "1.0.0" || definition.DefaultProfile != "gemini-flash-latest" {
|
||||
t.Fatalf("definition = %#v, want %s version 1.0.0 and gemini-flash-latest", definition, tc.id)
|
||||
if definition.ID != tc.id || definition.Version != "1.1.0" || definition.DefaultProfile != tc.profile {
|
||||
t.Fatalf("definition = %#v, want %s version 1.1.0 and profile %s", definition, tc.id, tc.profile)
|
||||
}
|
||||
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
||||
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
||||
@@ -101,11 +102,11 @@ func TestSchemasAreCanonicalAndIndependent(t *testing.T) {
|
||||
if err := json.Unmarshal(data, &schema); err != nil {
|
||||
t.Fatalf("decode schema: %v", err)
|
||||
}
|
||||
if schema.Type != "object" || schema.AdditionalProperties || strings.Join(schema.Required, ",") != "summary,forecast_discussion" {
|
||||
if schema.Type != "object" || schema.AdditionalProperties || strings.Join(schema.Required, ",") != "summary,forecast_discussion,precipitation_timing" {
|
||||
t.Fatalf("schema = %#v, want strict generated-text object", schema)
|
||||
}
|
||||
if _, ok := schema.Properties["confidence"]; !ok {
|
||||
t.Fatalf("schema properties = %#v, want confidence", schema.Properties)
|
||||
if _, ok := schema.Properties["confidence"]; ok {
|
||||
t.Fatalf("schema properties = %#v, do not want retired confidence field", schema.Properties)
|
||||
}
|
||||
if id == "daily" && (schema.ID != "weatherreporter.daily.generated_text.schema.json" || schema.Title != "Daily GeneratedText") {
|
||||
t.Fatalf("daily schema identity = %q/%q, want corrected Daily identity", schema.ID, schema.Title)
|
||||
@@ -123,21 +124,116 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
|
||||
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
|
||||
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine() error = %v", err)
|
||||
}
|
||||
for _, id := range []string{"weather.daily_generated_text", "weather.today_generated_text", "weather.tomorrow_generated_text", "weather.hourly_generated_text"} {
|
||||
t.Run(id, func(t *testing.T) {
|
||||
inspection, err := engine.InspectPrompt(context.Background(), id, "1.0.0")
|
||||
for _, want := range []struct {
|
||||
id string
|
||||
profile string
|
||||
model string
|
||||
}{
|
||||
{"weather.daily_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
|
||||
{"weather.today_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
|
||||
{"weather.tomorrow_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
|
||||
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
|
||||
} {
|
||||
t.Run(want.id, func(t *testing.T) {
|
||||
inspection, err := engine.InspectPrompt(context.Background(), want.id, "1.1.0")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPrompt() error = %v", err)
|
||||
}
|
||||
if inspection.PromptID != id || inspection.PromptVersion != "1.0.0" || inspection.DefaultProfileID != "gemini-flash-latest" {
|
||||
if inspection.PromptID != want.id || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != want.profile {
|
||||
t.Fatalf("inspection = %#v", inspection)
|
||||
}
|
||||
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
|
||||
if err != nil || profile.EffectiveModelParams.Model != want.model {
|
||||
t.Fatalf("profile/error = %#v/%v, want model %q", profile, err, want.model)
|
||||
}
|
||||
})
|
||||
}
|
||||
profile, err := engine.InspectProfile(context.Background(), "weather-deep")
|
||||
if err != nil || profile.EffectiveModelParams.Model != "~anthropic/claude-sonnet-latest" {
|
||||
t.Fatalf("weather-deep profile/error = %#v/%v", profile, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
|
||||
wantPaths := map[string]bool{
|
||||
"weather-balanced.yml": false,
|
||||
"weather-deep.yml": false,
|
||||
"weather-light.yml": false,
|
||||
}
|
||||
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
if _, ok := wantPaths[path]; !ok {
|
||||
t.Fatalf("unexpected embedded profile asset %q", path)
|
||||
}
|
||||
wantPaths[path] = true
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("walk embedded profiles: %v", err)
|
||||
}
|
||||
for path, found := range wantPaths {
|
||||
if !found {
|
||||
t.Errorf("missing embedded profile asset %q", path)
|
||||
}
|
||||
}
|
||||
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
|
||||
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
|
||||
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine() error = %v", err)
|
||||
}
|
||||
profiles := []struct {
|
||||
id string
|
||||
model string
|
||||
timeoutSeconds int
|
||||
reasoningEffort string
|
||||
}{
|
||||
{"weather-light", "deepseek/deepseek-v4-flash", 180, ""},
|
||||
{"weather-balanced", "~google/gemini-flash-latest", 240, "high"},
|
||||
{"weather-deep", "~anthropic/claude-sonnet-latest", 240, "high"},
|
||||
}
|
||||
for _, want := range profiles {
|
||||
t.Run(want.id, func(t *testing.T) {
|
||||
inspection, err := engine.InspectProfile(context.Background(), want.id)
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile() error = %v", err)
|
||||
}
|
||||
got := inspection.EffectiveModelParams
|
||||
if inspection.ProfileID != want.id || got.BackendID != "openrouter" || got.Model != want.model || got.TimeoutSeconds != want.timeoutSeconds || got.ServiceTier != "flex" || got.ReasoningEffort != want.reasoningEffort {
|
||||
t.Fatalf("inspection = %#v, want %q using openrouter model %q", inspection, want.id, want.model)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
|
||||
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
|
||||
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, setting := range forbidden {
|
||||
if strings.Contains(string(data), setting) {
|
||||
t.Fatalf("%s contains forbidden profile setting %q", path, setting)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("walk embedded profiles: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
|
||||
|
||||
@@ -12,7 +12,7 @@ func dailyDefinition() Definition {
|
||||
ID: Daily,
|
||||
Name: "Daily Report",
|
||||
PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "1.0.0",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "daily",
|
||||
GeneratedTextSchemaID: "daily",
|
||||
ComparisonStrategy: CompareSameValidDate,
|
||||
|
||||
@@ -14,7 +14,7 @@ func hourlyDefinition() Definition {
|
||||
ID: Hourly,
|
||||
Name: "Hourly Report",
|
||||
PromptID: "weather.hourly_generated_text",
|
||||
PromptVersion: "1.0.0",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "hourly",
|
||||
GeneratedTextSchemaID: "hourly",
|
||||
ComparisonStrategy: CompareRollingWindow,
|
||||
|
||||
@@ -51,8 +51,11 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, definition := range definitions {
|
||||
if definition.PromptVersion != "1.0.0" {
|
||||
t.Fatalf("%s PromptVersion = %q, want 1.0.0", definition.ID, definition.PromptVersion)
|
||||
if definition.PromptVersion != "1.1.0" {
|
||||
t.Fatalf("%s PromptVersion = %q, want 1.1.0", definition.ID, definition.PromptVersion)
|
||||
}
|
||||
if definition.PromptID == "" {
|
||||
t.Fatalf("%s PromptID is empty", definition.ID)
|
||||
}
|
||||
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
|
||||
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
|
||||
|
||||
@@ -10,7 +10,7 @@ func todayDefinition() Definition {
|
||||
ID: Today,
|
||||
Name: "Today Report",
|
||||
PromptID: "weather.today_generated_text",
|
||||
PromptVersion: "1.0.0",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "today",
|
||||
GeneratedTextSchemaID: "today",
|
||||
ComparisonStrategy: CompareSameValidDate,
|
||||
|
||||
@@ -10,7 +10,7 @@ func tomorrowDefinition() Definition {
|
||||
ID: Tomorrow,
|
||||
Name: "Tomorrow Report",
|
||||
PromptID: "weather.tomorrow_generated_text",
|
||||
PromptVersion: "1.0.0",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "tomorrow",
|
||||
GeneratedTextSchemaID: "tomorrow",
|
||||
ComparisonStrategy: CompareSameValidDate,
|
||||
|
||||
@@ -64,7 +64,7 @@ func TestSchemaLookup(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Schema() error = %v", err)
|
||||
}
|
||||
assertStringSchema(t, data, "summary,forecast_discussion", []string{"summary", "forecast_discussion", "precipitation_timing", "confidence"})
|
||||
assertStringSchema(t, data, "summary,forecast_discussion,precipitation_timing", []string{"summary", "forecast_discussion", "precipitation_timing"})
|
||||
}
|
||||
|
||||
func TestTomorrowSchemaLookup(t *testing.T) {
|
||||
@@ -72,7 +72,7 @@ func TestTomorrowSchemaLookup(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Schema() error = %v", err)
|
||||
}
|
||||
schema := assertSchema(t, data, "summary,forecast_discussion")
|
||||
schema := assertSchema(t, data, "summary,forecast_discussion,precipitation_timing")
|
||||
property, ok := schema.Properties["forecast_discussion"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatal("schema property forecast_discussion missing or invalid")
|
||||
@@ -87,7 +87,7 @@ func TestTomorrowSchemaLookup(t *testing.T) {
|
||||
if !ok || items["type"] != "string" {
|
||||
t.Fatalf("forecast_discussion items = %#v, want string items", property["items"])
|
||||
}
|
||||
for _, field := range []string{"summary", "precipitation_timing", "confidence"} {
|
||||
for _, field := range []string{"summary", "precipitation_timing"} {
|
||||
property, ok := schema.Properties[field].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("schema property %q missing or invalid", field)
|
||||
@@ -103,7 +103,7 @@ func TestDailySchemaLookup(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Schema() error = %v", err)
|
||||
}
|
||||
schema := assertSchema(t, data, "summary,forecast_discussion")
|
||||
schema := assertSchema(t, data, "summary,forecast_discussion,precipitation_timing")
|
||||
property, ok := schema.Properties["forecast_discussion"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatal("schema property forecast_discussion missing or invalid")
|
||||
@@ -118,7 +118,7 @@ func TestDailySchemaLookup(t *testing.T) {
|
||||
if !ok || items["type"] != "string" {
|
||||
t.Fatalf("forecast_discussion items = %#v, want string items", property["items"])
|
||||
}
|
||||
for _, field := range []string{"summary", "precipitation_timing", "confidence"} {
|
||||
for _, field := range []string{"summary", "precipitation_timing"} {
|
||||
property, ok := schema.Properties[field].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("schema property %q missing or invalid", field)
|
||||
@@ -134,7 +134,7 @@ func TestTodaySchemaLookup(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Fatalf("Schema() error = %v", err)
|
||||
}
|
||||
schema := assertSchema(t, data, "summary,forecast_discussion")
|
||||
schema := assertSchema(t, data, "summary,forecast_discussion,precipitation_timing")
|
||||
property, ok := schema.Properties["forecast_discussion"].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatal("schema property forecast_discussion missing or invalid")
|
||||
@@ -149,7 +149,7 @@ func TestTodaySchemaLookup(t *testing.T) {
|
||||
if !ok || items["type"] != "string" {
|
||||
t.Fatalf("forecast_discussion items = %#v, want string items", property["items"])
|
||||
}
|
||||
for _, field := range []string{"summary", "precipitation_timing", "confidence"} {
|
||||
for _, field := range []string{"summary", "precipitation_timing"} {
|
||||
property, ok := schema.Properties[field].(map[string]any)
|
||||
if !ok {
|
||||
t.Fatalf("schema property %q missing or invalid", field)
|
||||
@@ -214,7 +214,6 @@ func TestRenderHourly(t *testing.T) {
|
||||
Summary: "Storm chances increase through late morning.",
|
||||
ForecastDiscussion: "A front will keep the region unsettled.",
|
||||
PrecipitationTiming: "A cold front is moving into the region.",
|
||||
Confidence: "Medium confidence in timing.",
|
||||
},
|
||||
Modules: testModules{
|
||||
CurrentConditions: &testCurrentConditions{
|
||||
@@ -282,7 +281,7 @@ func TestRenderHourly(t *testing.T) {
|
||||
t.Fatalf("rendered template missing %q:\n%s", want, text)
|
||||
}
|
||||
}
|
||||
if strings.Contains(text, "19%") || strings.Contains(text, "wind S") || strings.Contains(text, "## Confidence") {
|
||||
if strings.Contains(text, "19%") || strings.Contains(text, "wind S") {
|
||||
t.Fatalf("rendered template included omitted details:\n%s", text)
|
||||
}
|
||||
for _, unwanted := range []string{"Avoid low-water crossings.", "Slight risk for severe thunderstorms"} {
|
||||
@@ -955,7 +954,6 @@ type testGeneratedText struct {
|
||||
Summary string
|
||||
ForecastDiscussion string
|
||||
PrecipitationTiming string
|
||||
Confidence string
|
||||
}
|
||||
|
||||
type testTomorrowReportContext struct {
|
||||
@@ -980,14 +978,12 @@ type testTomorrowGeneratedText struct {
|
||||
Summary string
|
||||
ForecastDiscussion []string
|
||||
PrecipitationTiming string
|
||||
Confidence string
|
||||
}
|
||||
|
||||
type testDailyGeneratedText struct {
|
||||
Summary string
|
||||
ForecastDiscussion []string
|
||||
PrecipitationTiming string
|
||||
Confidence string
|
||||
}
|
||||
|
||||
type testModules struct {
|
||||
|
||||
@@ -172,9 +172,9 @@ func validPreparationArtifact() PromptPreparationArtifact {
|
||||
return PromptPreparationArtifact{
|
||||
SchemaVersion: PromptPreparationSchemaVersion, Status: PromptPreparationSucceeded,
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "1.0.0", DataPackagePath: "/workspace/data.yaml",
|
||||
PromptVersion: "1.1.0", DataPackagePath: "/workspace/data.yaml",
|
||||
Preparation: &promptexec.Preparation{
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
DataPackagePath: "/workspace/data.yaml",
|
||||
},
|
||||
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
|
||||
@@ -186,9 +186,9 @@ func validExecutionArtifact() PromptExecutionArtifact {
|
||||
validation := promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", "daily.generated_text.schema.json", nil)
|
||||
return PromptExecutionArtifact{
|
||||
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionSucceeded,
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
Provenance: &PromptExecutionProvenance{
|
||||
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
|
||||
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: "profile",
|
||||
BackendID: "backend", ModelName: "model", DataPackagePath: "/workspace/data.yaml",
|
||||
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
|
||||
@@ -201,7 +201,7 @@ func validFailedExecutionArtifact() PromptExecutionArtifact {
|
||||
started := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
|
||||
return PromptExecutionArtifact{
|
||||
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionFailed,
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.0",
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
StartedAt: started, EndedAt: started, Error: &PromptArtifactError{Category: promptexec.Generation, Message: "provider unavailable"},
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user