Compare commits
8 Commits
51c35f7c22
...
1250247986
| Author | SHA1 | Date | |
|---|---|---|---|
| 1250247986 | |||
| 117c5336ba | |||
| c5ec4f83b2 | |||
| 39c097a710 | |||
| 993120a9f2 | |||
| c20e285d5f | |||
| acbe22dcad | |||
| cc97ae186c |
@@ -22,8 +22,12 @@ extra report copies for a command and do not change configuration.
|
||||
collection and generation configuration.
|
||||
- [config.yml](../examples/config.yml) is a representative production-oriented
|
||||
configuration using synthetic endpoints and no credentials.
|
||||
- [weather-light-local-profile.yml](../examples/weather-light-local-profile.yml)
|
||||
is a complete endpoint-only override for the embedded `weather-light`
|
||||
profile.
|
||||
|
||||
Both files are loaded by the configuration test suite.
|
||||
The configuration examples are loaded by the configuration test suite. The
|
||||
profile example is inspected through the Promptkit adapter test suite.
|
||||
|
||||
## Minimal Configuration
|
||||
|
||||
@@ -146,13 +150,35 @@ individual `generate` or `run` command when explicitly needed.
|
||||
|
||||
| Field | Default | Rules |
|
||||
| --- | --- | --- |
|
||||
| `profile` | empty | Optional explicit execution profile. Otherwise the prompt's declared default is used. |
|
||||
| `profile_file` | empty | Optional external profile file. Cannot be combined with `profile_dir`. |
|
||||
| `profile_dir` | empty | Optional external profile directory. Cannot be combined with `profile_file`. |
|
||||
| `profile` | empty | Optional global profile selection for every report in one command. When empty, each exact prompt version selects its declared default. |
|
||||
| `profile_file` | empty | Optional external Promptkit profile file. It cannot be combined with `profile_dir`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
|
||||
| `profile_dir` | empty | Optional external Promptkit profile directory. It cannot be combined with `profile_file`. A same-ID profile completely replaces Weatherreporter's embedded definition. |
|
||||
| `timeout` | `2m` | Must be greater than zero. |
|
||||
| `local.endpoint` | empty | Optional absolute URL for the conventional local backend. A blank endpoint leaves it unregistered. |
|
||||
| `local.concurrency_limit` | `1` | Maximum local backend concurrency. `0` is unlimited; negative values are invalid. |
|
||||
|
||||
`profile` selects an ID; `profile_file` and `profile_dir` supply definitions.
|
||||
They are separate decisions. An explicit `profile` applies to every selected
|
||||
report. Otherwise Hourly selects `weather-light`, while Daily, Today, and
|
||||
Tomorrow select `weather-balanced` through their exact `1.1.0` prompt
|
||||
definitions.
|
||||
|
||||
Promptkit resolves a selected profile definition from a test or embedding
|
||||
consumer's explicit in-memory profile, then the configured `profile_file` or
|
||||
`profile_dir`, then Weatherreporter's embedded catalog, and finally Promptkit's
|
||||
built-in catalog. Sources provide complete definitions; fields are never
|
||||
merged. A matching malformed external profile fails rather than using the
|
||||
embedded definition. The [Promptkit integration guide](integrations/promptkit.md)
|
||||
owns the catalog and precedence details.
|
||||
|
||||
To replace the default Hourly definition with a local OpenAI-compatible
|
||||
endpoint, set `profile_file` to a copy of
|
||||
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml).
|
||||
The example has no credential and should be edited for the local endpoint and
|
||||
model before use. An alternative profile may use `backend: local`; in that
|
||||
case `promptkit.local.endpoint` supplies the conventional local backend
|
||||
endpoint.
|
||||
|
||||
### `workspace`
|
||||
|
||||
| Field | Default |
|
||||
|
||||
@@ -1,16 +1,49 @@
|
||||
# Promptkit Integration
|
||||
|
||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical prompts are
|
||||
`weather.daily_generated_text`, `weather.today_generated_text`,
|
||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at version
|
||||
`1.0.1`. Their prompt assets and generated-text JSON Schemas are embedded by
|
||||
`internal/promptassets`.
|
||||
Weatherreporter uses Promptkit for all generated-text reports. The four logical
|
||||
prompts are `weather.daily_generated_text`, `weather.today_generated_text`,
|
||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`, each at
|
||||
version `1.1.0`. Their prompt assets, generated-text JSON Schemas, and
|
||||
Weatherreporter profile catalog are embedded by `internal/promptassets`.
|
||||
|
||||
Before collection, Weatherreporter inspects the exact prompt version, requires one required
|
||||
`data_package` input with content type `application/yaml`, and requires the report's JSON
|
||||
Schema output contract. It selects `promptkit.profile` when configured, otherwise the
|
||||
prompt's declared default profile. Profiles that require a direct API key are unsupported; a
|
||||
profile that reports `APIKeyEnv` requires a nonblank value in that environment variable.
|
||||
## Logical profile catalog
|
||||
|
||||
Prompt definitions select a stable Weatherreporter profile ID. The embedded
|
||||
definitions currently use Promptkit's `openrouter` backend:
|
||||
|
||||
| Profile ID | Model | Reasoning effort | Timeout | Service tier | Default reports |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` | Hourly |
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` | Daily, Today, Tomorrow |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` | None |
|
||||
|
||||
The `~` prefix is part of each OpenRouter rolling-alias model ID. The embedded
|
||||
profiles intentionally omit endpoints, credentials, temperature, `top_p`, and
|
||||
output-token limits.
|
||||
|
||||
## Selection, lookup, and active execution
|
||||
|
||||
Before collection, Weatherreporter inspects the exact prompt version and output
|
||||
contract. A nonblank `promptkit.profile` selects one profile ID for every
|
||||
report in the command; otherwise the prompt's declared default selects it.
|
||||
Promptkit resolves the selected definition in this order:
|
||||
|
||||
1. explicit in-memory profiles used by an embedding consumer or test;
|
||||
2. the configured `profile_file` or `profile_dir`;
|
||||
3. Weatherreporter's embedded fallback profiles; and
|
||||
4. Promptkit's built-in catalog.
|
||||
|
||||
A source falls through only when the selected ID is absent. Each source
|
||||
supplies a complete definition, so profile fields are not merged. A malformed
|
||||
matching operator definition is an error and does not fall back.
|
||||
|
||||
Profiles that require a direct API key are unsupported; a profile that reports
|
||||
`APIKeyEnv` requires a nonblank value in that environment variable. Inspection,
|
||||
preparation, and execution retain the selected logical profile ID and resolved
|
||||
backend and model through Weatherreporter's project-owned contract. Ordinary
|
||||
errors, summaries, logs, and workspace state exclude endpoints, credentials,
|
||||
rendered messages, schemas, request bodies, response bodies, and complete
|
||||
parameter maps.
|
||||
|
||||
Execution receives the already-persisted YAML package, prepares it once, and returns structured
|
||||
JSON that Weatherreporter validates before rendering its own Markdown template. Preparation and
|
||||
@@ -21,6 +54,7 @@ The generated-text schemas require `summary`, `forecast_discussion`, and
|
||||
`precipitation_timing`, and reject additional properties. Prompts return an empty string for
|
||||
`precipitation_timing` when the deterministic package contains no precipitation windows.
|
||||
|
||||
Prompt/profile configuration is owned by the [configuration reference](../config.md). Adapter
|
||||
construction and mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
|
||||
Prompt/profile configuration and the maintained local override example are
|
||||
owned by the [configuration reference](../config.md). Adapter construction and
|
||||
mapping are documented in the [Promptkit adapter internals](../internal/promptkit-adapter.md).
|
||||
Durable metadata compatibility is described in [state internals](../internal/state.md).
|
||||
|
||||
@@ -7,7 +7,10 @@ notification ordering after the CLI has parsed arguments and loaded configuratio
|
||||
|
||||
`GenerateDetailed` resolves one of the four report definitions, initializes an
|
||||
optional debug root, and inspects the exact Promptkit prompt/profile before it
|
||||
collects weather or writes managed state. It then builds facts and modules,
|
||||
collects weather or writes managed state. A configured global profile selects
|
||||
every report in the action; otherwise the exact prompt selects its default
|
||||
logical profile. Inspection keeps only the selected profile ID and effective
|
||||
backend/model needed by the project-owned execution contract. It then builds facts and modules,
|
||||
saves the YAML data package, persists preparation metadata from the executor
|
||||
callback, executes the prepared prompt, saves execution provenance and raw
|
||||
output, validates generated text, renders Markdown, and optionally copies or
|
||||
@@ -25,8 +28,8 @@ persists raw output and execution provenance but does not render a report.
|
||||
|
||||
`RunBatchDetailed` constructs a single debug writer and uses the request's
|
||||
single executor. Before collection it inspects Today, Tomorrow, and Daily for
|
||||
morning, or Tomorrow and Daily for evening, deduplicating effective profile
|
||||
inspection. It then collects once, plans eligible Daily dates, and calls the
|
||||
morning, or Tomorrow and Daily for evening, deduplicating inspection of a
|
||||
shared selected profile. It then collects once, plans eligible Daily dates, and calls the
|
||||
same prompt-generation core sequentially for each planned report. Per-report
|
||||
notification is suppressed; a failed report does not stop later reports.
|
||||
|
||||
|
||||
@@ -4,10 +4,18 @@
|
||||
The CLI maps `promptkit` configuration to a `PromptExecutorConfig` and constructs one executor
|
||||
per action. Promptkit dependency types do not escape the adapter.
|
||||
|
||||
The adapter exposes exact prompt and profile inspection plus prepared execution. It maps Promptkit
|
||||
inspection values to project-owned prompt input, output-contract, profile, preparation, execution,
|
||||
validation, and optional debug values. It classifies adapter failures without copying provider secrets
|
||||
or unbounded response bodies into application errors or normal state.
|
||||
The adapter supplies Weatherreporter's embedded prompt, schema, and fallback
|
||||
profile filesystems to each engine. Promptkit remains responsible for resolving
|
||||
the configured operator profile source, application fallback catalog, and its
|
||||
built-in catalog; the adapter does not parse profile YAML, merge sources, or
|
||||
probe endpoints.
|
||||
|
||||
The adapter exposes exact prompt and profile inspection plus prepared execution.
|
||||
It maps Promptkit inspection values to project-owned prompt input,
|
||||
output-contract, logical profile identity, effective backend/model,
|
||||
preparation, execution, validation, and optional debug values. It classifies
|
||||
adapter failures without copying provider secrets or unbounded response bodies
|
||||
into application errors or normal state.
|
||||
|
||||
The app calls the executor's preparation callback before provider execution to persist safe preparation
|
||||
provenance. Completed executions are then persisted as safe execution provenance and raw generated text
|
||||
|
||||
@@ -16,16 +16,19 @@ templates, generation eligibility, compatible prior IDs, default modules, and
|
||||
batch eligibility flags. `Resolved` combines that definition with the valid
|
||||
period and run metadata for one invocation.
|
||||
|
||||
| Report ID | Prompt version | Period policy | Comparison | Registry batch flag | Output copy |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `daily` | `1.0.1` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
|
||||
| `today` | `1.0.1` | Selected or current local civil day | Same valid date | Morning | `today.md` |
|
||||
| `tomorrow` | `1.0.1` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
|
||||
| `hourly` | `1.0.1` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
|
||||
| Report ID | Prompt version | Default profile | Period policy | Comparison | Registry batch flag | Output copy |
|
||||
| --- | --- | --- | --- | --- | --- | --- |
|
||||
| `daily` | `1.1.0` | `weather-balanced` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
|
||||
| `today` | `1.1.0` | `weather-balanced` | Selected or current local civil day | Same valid date | Morning | `today.md` |
|
||||
| `tomorrow` | `1.1.0` | `weather-balanced` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
|
||||
| `hourly` | `1.1.0` | `weather-light` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
|
||||
|
||||
Each report pairs its ID and prompt version with matching template and schema
|
||||
IDs. Exact template fields and schema assets belong to [report templates](../templates.md)
|
||||
and [generated-text internals](generatedtext.md).
|
||||
and [generated-text internals](generatedtext.md). Prompt assets own default
|
||||
profile selection; the registry deliberately stores no provider setting. The
|
||||
[Promptkit integration guide](../integrations/promptkit.md) owns profile
|
||||
definitions and resolution.
|
||||
|
||||
All valid periods are half-open.
|
||||
|
||||
|
||||
@@ -34,8 +34,10 @@ runs never write V1 records.
|
||||
Prompt preparation and execution records are validated on both save and load.
|
||||
They require exact report/prompt identity, complete timing, internally
|
||||
consistent provenance, and status-appropriate validation or bounded classified
|
||||
errors. Completed execution provenance keeps Promptkit's run identity distinct
|
||||
from the Weatherreporter run identity.
|
||||
errors. Completed preparation and execution provenance retain the selected
|
||||
logical profile ID and resolved backend/model, but never profile endpoints or
|
||||
credentials. Completed execution provenance keeps Promptkit's run identity
|
||||
distinct from the Weatherreporter run identity.
|
||||
|
||||
For a completed prompt run, the execution record is atomically replaced after
|
||||
each downstream artifact is saved. Its path set therefore records the raw and
|
||||
|
||||
@@ -19,12 +19,38 @@ persists the module snapshot and prompt data package, records Promptkit
|
||||
preparation provenance before provider execution, then persists raw output and
|
||||
execution provenance, validates the structured generated text, and renders the
|
||||
managed Markdown report from the validated text and deterministic values.
|
||||
The current receipts are transitional workspace state, not a cross-version
|
||||
profile-provenance contract.
|
||||
|
||||
The managed report and its final metadata are saved before single-report
|
||||
Distributor notification is attempted. `--out` writes an extra operator copy;
|
||||
it never changes the managed report or upload source. A successful generate
|
||||
command prints its summary to stdout unless `--quiet` is used.
|
||||
|
||||
## Local Prompt Profile Override
|
||||
|
||||
Hourly normally selects the embedded `weather-light` profile. To use a local
|
||||
OpenAI-compatible model without changing prompts or application code, copy
|
||||
[weather-light-local-profile.yml](../examples/weather-light-local-profile.yml),
|
||||
set its `endpoint` and `model` for the local server, and configure the copy as
|
||||
`promptkit.profile_file`. The profile file's `weather-light` definition
|
||||
completely replaces the embedded definition; it does not affect a report that
|
||||
selects another profile ID.
|
||||
|
||||
For example, install the profile file at a known absolute path and set:
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
profile_file: /etc/weatherreporter/weather-light-local-profile.yml
|
||||
```
|
||||
|
||||
Prompt inspection occurs before weather collection. A malformed profile file,
|
||||
missing required credential, or unsupported selected backend stops the command
|
||||
before collection. A reachable profile can still fail later if its local model
|
||||
endpoint is unavailable; Weatherreporter does not switch to a remote profile.
|
||||
See the [configuration reference](config.md) for field definitions and the
|
||||
[troubleshooting guide](troubleshooting.md) for recovery.
|
||||
|
||||
## Optional Prompt Debug Capture
|
||||
|
||||
Use `--llm-debug-dir` only when content-rich prompt diagnostics are required:
|
||||
@@ -87,8 +113,11 @@ The generated-text and render-context artifacts are written for every completed
|
||||
single-report generation.
|
||||
A report's metadata links the module snapshot, data package, preparation and
|
||||
execution receipts, managed report, generated-text artifacts, and any available single-report
|
||||
notification artifact. Batch notification artifacts are separate batch-level
|
||||
records under `notifications/batches`.
|
||||
notification artifact. These current-version receipts remain transitional; use
|
||||
the active command's classified error and explicit secure debug capture for
|
||||
prompt diagnosis rather than relying on them as a durable interface. Batch
|
||||
notification artifacts are separate batch-level records under
|
||||
`notifications/batches`.
|
||||
|
||||
RunIDs begin with the UTC generation timestamp and report ID. A Daily RunID
|
||||
also contains its local valid date so multiple Daily reports in one batch have
|
||||
@@ -162,10 +191,12 @@ remain available where they can be safely persisted.
|
||||
- A batch notification failure preserves each report's artifacts and adds the
|
||||
top-level batch notification artifact.
|
||||
|
||||
Use the RunID from the action summary with the inspection commands above. For
|
||||
a batch failure, inspect the summary first, then inspect the affected report
|
||||
RunIDs or the batch notification path. Do not remove the whole workspace as a
|
||||
first response; retain it until the failure is understood.
|
||||
Use the action summary and its classified error first. For prompt or provider
|
||||
diagnosis, prefer an explicitly enabled secure debug capture; current-version
|
||||
receipt paths may provide supplemental context when available. For a batch
|
||||
failure, inspect the summary first, then inspect the affected report RunIDs or
|
||||
the batch notification path. Do not remove the whole workspace as a first
|
||||
response; retain it until the failure is understood.
|
||||
|
||||
## Operational Caveats
|
||||
|
||||
@@ -174,5 +205,7 @@ first response; retain it until the failure is understood.
|
||||
not publish them unintentionally.
|
||||
- Weatherreporter uses one configured Weather API endpoint and local workspace
|
||||
state.
|
||||
- Promptkit profile resolution does not discover local endpoints or fail over
|
||||
between local and remote profiles.
|
||||
- It does not provide automatic resume, cleanup, archival, remote state, daemon
|
||||
operation, or automatic storm monitoring.
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
# Domain-Specific Prompt Profiles Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
Status: Implemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
@@ -10,9 +10,9 @@ These logical profiles should give each report an appropriate default while
|
||||
allowing operators to replace any definition through the existing configured
|
||||
profile source.
|
||||
|
||||
This roadmap defines the scope, policy, and intended end state. A later staged
|
||||
implementation plan will translate it into implementation work. This document
|
||||
does not prescribe implementation stages.
|
||||
This roadmap records the scope, policy, and implemented end state. The
|
||||
companion [implementation plan](implementation.md) records the ordered work
|
||||
and verification used to reach it.
|
||||
|
||||
## User Intent
|
||||
|
||||
@@ -29,35 +29,34 @@ The feature is intended to provide three related benefits:
|
||||
guarantee. A locally hosted lightweight model may still generate slowly on the
|
||||
available hardware.
|
||||
|
||||
## Current State
|
||||
## Pre-Implementation Baseline
|
||||
|
||||
Daily, Today, Tomorrow, and Hourly each declare Promptkit's
|
||||
`gemini-flash-latest` profile as their prompt default. The optional
|
||||
`promptkit.profile` setting overrides that default for every selected report in
|
||||
Before implementation, Daily, Today, Tomorrow, and Hourly each declared
|
||||
Promptkit's `gemini-flash-latest` profile as their prompt default. The optional
|
||||
`promptkit.profile` setting overrode that default for every selected report in
|
||||
an invocation.
|
||||
|
||||
Weatherreporter accepts either `promptkit.profile_file` or
|
||||
`promptkit.profile_dir` and passes that source to Promptkit. A matching external
|
||||
profile can override a Promptkit built-in profile, and the configured local
|
||||
backend can support profiles that select `backend: local`. Endpoint-only
|
||||
OpenAI-compatible profiles can also provide their own endpoint.
|
||||
Weatherreporter accepted either `promptkit.profile_file` or
|
||||
`promptkit.profile_dir` and passed that source to Promptkit. A matching external
|
||||
profile could override a Promptkit built-in profile, and the configured local
|
||||
backend could support profiles that select `backend: local`. Endpoint-only
|
||||
OpenAI-compatible profiles could also provide their own endpoint.
|
||||
|
||||
Weatherreporter does not currently own or embed execution profiles. Promptkit's
|
||||
current public source precedence also has no layer where application-owned
|
||||
fallback profiles can remain overridable by the ordinary configured source.
|
||||
Weatherreporter did not own or embed execution profiles. Promptkit v0.5.0
|
||||
provided the fallback-profile layer used to add them without changing the
|
||||
existing operator-source precedence.
|
||||
|
||||
## Prerequisite
|
||||
|
||||
Promptkit must first provide the application fallback profile capability
|
||||
defined in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md).
|
||||
Promptkit v0.5.0 provides the application fallback profile capability defined
|
||||
in the companion
|
||||
[upstream feature request](promptkit-fallback-profiles-feature-request.md), and
|
||||
Weatherreporter now depends on that tagged release. The dependency upgrade has
|
||||
passed the repository test suite and an operator smoke test. Weatherreporter
|
||||
must continue to use only Promptkit's public API rather than depending on its
|
||||
internal packages or reproducing its profile repository behavior.
|
||||
|
||||
Weatherreporter should adopt a tagged Promptkit release containing that public
|
||||
capability. It should not depend on Promptkit internal packages, reproduce
|
||||
Promptkit's profile repository behavior, or commit a local module replacement
|
||||
as the production dependency.
|
||||
|
||||
## Desired End State
|
||||
## Implemented End State
|
||||
|
||||
Weatherreporter embeds usable definitions for these exact logical profile IDs:
|
||||
|
||||
@@ -148,20 +147,21 @@ selected model and supported by its backend. Avoid incidental generation
|
||||
parameters that reduce portability or trigger provider-specific request
|
||||
failures without a demonstrated quality benefit.
|
||||
|
||||
The initial model candidates are:
|
||||
The initial profile definitions are:
|
||||
|
||||
- `weather-light`: the maintained OpenRouter identifier for the Gemma 4 31B
|
||||
instruction model;
|
||||
- `weather-balanced`: Promptkit's maintained Gemini Flash family target; and
|
||||
- `weather-deep`: Promptkit's maintained Gemini Pro family target.
|
||||
| Profile ID | OpenRouter model | Reasoning effort | Timeout | Service tier |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| `weather-light` | `deepseek/deepseek-v4-flash` | Provider default | 180 seconds | `flex` |
|
||||
| `weather-balanced` | `~google/gemini-flash-latest` | `high` | 240 seconds | `flex` |
|
||||
| `weather-deep` | `~anthropic/claude-sonnet-latest` | `high` | 240 seconds | `flex` |
|
||||
|
||||
The exact identifiers and generation settings must be finalized against the
|
||||
available tagged Promptkit catalog and a representative Weatherreporter
|
||||
evaluation corpus before the staged implementation plan is approved. The
|
||||
logical profile IDs and report assignments do not depend on those concrete
|
||||
choices.
|
||||
These settings deliberately match the corresponding Promptkit v0.5.0
|
||||
built-ins while exposing Weatherreporter-owned logical IDs. The `~` prefix is
|
||||
part of each OpenRouter rolling-alias identifier. The profiles do not set
|
||||
temperature, `top_p`, or output-token limits; omission preserves provider
|
||||
defaults and avoids unsupported incidental parameters.
|
||||
|
||||
## Prompt And Provenance Contract
|
||||
## Prompt And Active Execution Contract
|
||||
|
||||
Changing a prompt's `default_profile` is a material prompt-definition change.
|
||||
The four prompt definitions should advance from `1.0.1` to `1.1.0` when the new
|
||||
@@ -172,10 +172,18 @@ Prompt inspection must continue to occur before weather collection. It should
|
||||
report the selected logical profile ID and the resolved backend and model
|
||||
without exposing endpoints or credentials.
|
||||
|
||||
Preparation, execution, and failure artifacts should retain both the logical
|
||||
profile identity and effective model provenance already available through the
|
||||
project-owned execution contract. This feature should not replace a logical
|
||||
profile ID with only the concrete model name in durable records.
|
||||
The active execution contract should retain both the selected logical profile
|
||||
identity and the resolved backend and model through inspection, preparation,
|
||||
execution, errors, and command results where those values are already exposed.
|
||||
This feature must not add a new durable-provenance or cross-version artifact
|
||||
contract.
|
||||
|
||||
The accepted [ephemeral-state roadmap](ephemeral-state.md) makes historical
|
||||
prompt provenance a non-goal. Existing workspace persistence may remain while
|
||||
this feature lands, but it is transitional behavior and must not be expanded or
|
||||
treated as part of the profile feature's desired end state. Prompt preparation
|
||||
and execution artifacts written at `1.0.1` are not required to remain readable
|
||||
after the prompt definitions advance to `1.1.0`.
|
||||
|
||||
## Evaluation Policy
|
||||
|
||||
@@ -196,7 +204,7 @@ The purpose is to choose an appropriate default for each tier, not to add a
|
||||
permanent benchmark framework or live-provider requirement to the ordinary
|
||||
test suite. Repository tests remain offline and deterministic.
|
||||
|
||||
## Scope
|
||||
## Implemented Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
@@ -211,7 +219,7 @@ The completed feature includes:
|
||||
- local-backend and endpoint-only override coverage;
|
||||
- fail-fast inspection of missing, malformed, or unusable selected profiles;
|
||||
- offline tests for selection, source precedence, effective model inspection,
|
||||
batch reuse, and durable provenance;
|
||||
batch reuse, and active execution behavior;
|
||||
- maintained operator examples for overriding `weather-light` locally; and
|
||||
- updates to the canonical configuration, Promptkit integration, report
|
||||
registry, operations, troubleshooting, internal adapter, and release
|
||||
@@ -243,13 +251,18 @@ observe new per-report defaults. This is a user-visible model-selection and
|
||||
cost change and must be called out in release notes. Operators who require the
|
||||
old all-report model can preserve it by setting an explicit global profile.
|
||||
|
||||
The prompt-version transition does not provide backward compatibility for
|
||||
historical prompt preparation or execution artifacts. This is consistent with
|
||||
the accepted ephemeral-state direction; the profile feature does not otherwise
|
||||
redesign or remove the current workspace layout.
|
||||
|
||||
An external same-ID override is an operator-owned compatibility commitment.
|
||||
Weatherreporter may evolve its embedded definitions, but it must not rewrite or
|
||||
silently merge an operator file.
|
||||
|
||||
## Completion Criteria
|
||||
## Completion Record
|
||||
|
||||
The roadmap is complete when:
|
||||
The following conditions are satisfied:
|
||||
|
||||
- a tagged Promptkit dependency supports the required fallback layer;
|
||||
- every operational prompt selects its assigned logical profile at exact
|
||||
@@ -262,33 +275,9 @@ The roadmap is complete when:
|
||||
- `weather-light` can resolve through an endpoint-only or configured-local
|
||||
override without requiring code or prompt changes;
|
||||
- global `promptkit.profile` still overrides every report in an invocation;
|
||||
- prompt preparation and execution artifacts record the selected logical
|
||||
profile and effective model;
|
||||
- active inspection and execution preserve the selected logical profile and
|
||||
effective model through the project-owned execution contract;
|
||||
- morning and evening batch preflight deduplicates inspection of shared
|
||||
effective profile IDs as it does today;
|
||||
- the default test suite remains offline and deterministic; and
|
||||
- implemented behavior is documented by its canonical current-state owners.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Exact initial model identifiers and settings
|
||||
|
||||
Recommendation: evaluate the current Gemma 4 31B instruction target for
|
||||
`weather-light`, the maintained Gemini Flash target for `weather-balanced`, and
|
||||
the maintained Gemini Pro target for `weather-deep`. Pin the exact profile
|
||||
contents in this roadmap before deriving the staged implementation plan.
|
||||
|
||||
Alternative: duplicate three current Promptkit built-in definitions without a
|
||||
Weatherreporter-specific evaluation. This is faster, but it provides less
|
||||
evidence that their cost, schema reliability, and weather reasoning match the
|
||||
tier policy.
|
||||
|
||||
### Future default use of `weather-deep`
|
||||
|
||||
Recommendation: ship the profile as an explicit override with no default
|
||||
report assignment. Assign it later only when a report or evaluation shows a
|
||||
material benefit over `weather-balanced`.
|
||||
|
||||
Alternative: assign Daily or Tomorrow to `weather-deep` immediately. This may
|
||||
improve difficult synthesis, but it increases routine cost before the benefit
|
||||
has been measured.
|
||||
|
||||
382
docs/roadmap/ephemeral-state.md
Normal file
382
docs/roadmap/ephemeral-state.md
Normal file
@@ -0,0 +1,382 @@
|
||||
# Ephemeral Operational State Roadmap
|
||||
|
||||
Status: Accepted feature direction; implementation has not started.
|
||||
|
||||
## Purpose
|
||||
|
||||
Weatherreporter should treat generated weather reports and their intermediate
|
||||
artifacts as short-lived operational material rather than a durable audit
|
||||
history. Forecasts and current conditions change continuously, and the normal
|
||||
response to an old or failed report is to generate a new report, not to
|
||||
reconstruct the provenance of the old one.
|
||||
|
||||
The application should retain only the bounded state needed to publish the
|
||||
current report, calculate Recent Changes against the last successfully
|
||||
published report for the same valid period, and complete the current
|
||||
invocation safely. Detailed LLM diagnostics should remain an explicit,
|
||||
operator-controlled exception outside ordinary workspace state.
|
||||
|
||||
This roadmap defines the intended state lifecycle, compatibility policy, and
|
||||
architectural boundaries. A separate implementation plan will define the
|
||||
ordered work after the roadmap is complete.
|
||||
|
||||
## User Intent
|
||||
|
||||
The state model should reflect these product expectations:
|
||||
|
||||
- weather reports are ephemeral products, not business records;
|
||||
- old report provenance has no continuing operational value once conditions
|
||||
and forecasts have changed;
|
||||
- regenerating is preferable to recovering, replaying, or inspecting an old
|
||||
generation;
|
||||
- routine operation should not accumulate unbounded run-addressed artifacts;
|
||||
- Recent Changes remains useful, but needs only one prior successful snapshot
|
||||
for the same report and valid period; and
|
||||
- sensitive prompt and response capture remains opt-in and explicitly managed
|
||||
by the operator.
|
||||
|
||||
## Current State
|
||||
|
||||
Each generation currently writes a run-addressed collection containing a
|
||||
module snapshot, data package, prompt preparation receipt, prompt execution
|
||||
receipt, raw generated text, validated generated text, render context, managed
|
||||
report, metadata, and optional notification receipt. Successful and failed
|
||||
runs accumulate beneath the workspace.
|
||||
|
||||
Metadata links the collection and supports lookup by RunID. The CLI can list
|
||||
historical runs and inspect their metadata, modules, data packages, prior
|
||||
snapshots, and source provenance. New metadata uses the V2 format while the
|
||||
reader retains V1 compatibility. Prompt artifacts are validated against
|
||||
current report and prompt definitions when saved and loaded.
|
||||
|
||||
Most of this persistence exists for retrospective inspection and failure
|
||||
recovery. Dedicated prompt preparation and execution load operations have no
|
||||
ordinary production consumer. The important exception is module snapshot
|
||||
state: generation actively loads the most recent compatible snapshot to build
|
||||
the deterministic Recent Changes input for Daily, Today, and Tomorrow.
|
||||
|
||||
## Desired End State
|
||||
|
||||
Weatherreporter has three distinct state classes:
|
||||
|
||||
| State class | Lifecycle | Purpose |
|
||||
| --- | --- | --- |
|
||||
| Invocation workspace | Temporary and unpublished | Hold intermediate values while one report or batch is running. |
|
||||
| Current published state | Bounded and replaceable | Hold the current managed report and the minimal deterministic snapshot or manifest needed for normal operation. |
|
||||
| Secure LLM debug capture | Explicitly enabled and operator-managed | Diagnose prompt rendering or provider output when the operator deliberately requests sensitive capture. |
|
||||
|
||||
Ordinary generation uses an invocation-scoped temporary directory on the same
|
||||
filesystem as the managed workspace when atomic publication requires it.
|
||||
Prompt preparation, prompt execution, raw generated text, validated generated
|
||||
text, render contexts, data packages, and notification receipts may exist
|
||||
there while needed, but they are not published as durable historical
|
||||
artifacts.
|
||||
|
||||
A successful report atomically replaces the current published state for its
|
||||
logical report key and valid period. A failed attempt leaves the last
|
||||
successfully published report and comparison snapshot unchanged. Ordinary
|
||||
temporary artifacts are removed after both success and handled failure;
|
||||
cleanup failure is reported safely but must not replace the primary generation
|
||||
error.
|
||||
|
||||
RunIDs remain useful as in-process correlation identifiers in action results,
|
||||
logs, provider provenance, and optional debug paths. They no longer identify a
|
||||
durable collection that Weatherreporter promises to locate or decode later.
|
||||
|
||||
## Published Report Policy
|
||||
|
||||
The managed Markdown report remains the authoritative upload source during an
|
||||
invocation. The intended default is to retain only the current managed report
|
||||
for each logical report key and valid period, replacing it atomically after a
|
||||
new report has been fully rendered and validated.
|
||||
|
||||
An explicit `--out` or `--out-dir` copy remains operator-owned output outside
|
||||
the managed-state lifecycle. Weatherreporter does not delete, rotate, or
|
||||
rewrite those copies except when the same explicit destination is selected by
|
||||
a later invocation.
|
||||
|
||||
Distributor continues to receive only a completed managed Markdown report.
|
||||
Notification success or failure does not create a durable notification
|
||||
history. A notification failure leaves the newly published report available
|
||||
and returns a safe error through the current action result.
|
||||
|
||||
## Recent Changes State
|
||||
|
||||
Recent Changes must be preserved without preserving general report history.
|
||||
For Daily, Today, and Tomorrow, Weatherreporter retains at most one compatible
|
||||
module snapshot for each logical report key and valid period.
|
||||
|
||||
The retained snapshot represents the last successfully published report. A
|
||||
new invocation reads it before constructing Recent Changes and replaces it
|
||||
only when the new managed report has been successfully validated, rendered,
|
||||
and published. A failed generation therefore does not become the baseline for
|
||||
the next report and cannot hide changes that the user has not yet seen.
|
||||
|
||||
Hourly does not currently use the comparison strategy and should not retain a
|
||||
comparison snapshot solely for symmetry. State whose valid period has ended
|
||||
and can no longer participate in a supported comparison is eligible for safe
|
||||
cleanup.
|
||||
|
||||
## Temporary Workspace And Failure Semantics
|
||||
|
||||
Temporary state must remain beneath a narrowly owned application directory and
|
||||
use safe path construction, restrictive permissions where content is
|
||||
sensitive, and atomic writes where practical. Publication must not expose a
|
||||
partially rendered report or a snapshot that does not correspond to the
|
||||
published report.
|
||||
|
||||
Normal results retain bounded error information and paths only for artifacts
|
||||
that remain meaningful after the command: a previously or newly published
|
||||
report, an explicit operator output, or an enabled secure debug capture.
|
||||
Temporary intermediate paths are not emitted as if they were durable recovery
|
||||
locations. A failed command is retried by starting a new generation.
|
||||
|
||||
Process interruption may leave an uncommitted temporary directory. Such a
|
||||
directory is never considered published state, is never selected for Recent
|
||||
Changes, and may be removed by a documented safe cleanup mechanism. Cleanup
|
||||
must distinguish inactive temporary directories from concurrent active
|
||||
invocations and must never recursively target the workspace root or an
|
||||
unresolved configuration path.
|
||||
|
||||
## Inspection And Metadata Policy
|
||||
|
||||
Run-history discovery and inspection are not part of the desired product
|
||||
contract. The historical `inspect reports`, `inspect metadata`, `inspect
|
||||
modules`, `inspect data-package`, `inspect prior`, and `inspect sources`
|
||||
surfaces are candidates for removal together rather than preservation through
|
||||
a new storage representation.
|
||||
|
||||
Any manifest retained for atomic publication or Recent Changes is current
|
||||
operational state, not an archival metadata record. It should contain only the
|
||||
identity, valid period, safe paths, and deterministic snapshot information
|
||||
needed to validate and use that current state. It does not need to preserve
|
||||
prompt messages, generated prose intermediates, source provenance, provider
|
||||
provenance, notification history, or a catalog of prior runs.
|
||||
|
||||
The application does not promise cross-version decoding of ordinary workspace
|
||||
state. A new release may replace or ignore incompatible current-state files,
|
||||
provided it fails safely, never mistakes stale state for a compatible Recent
|
||||
Changes baseline, and documents any operator action required during upgrade.
|
||||
|
||||
## Prompt Execution And Debugging
|
||||
|
||||
Prompt inspection before weather collection and prepared execution remain
|
||||
runtime safety requirements. They do not require durable preparation or
|
||||
execution receipts.
|
||||
|
||||
The selected logical profile, effective backend and model, validation outcome,
|
||||
and safe classified error remain available to the active workflow and its CLI
|
||||
summary where useful. Weatherreporter does not retain them as long-term report
|
||||
provenance after the invocation completes.
|
||||
|
||||
The existing explicit secure debug root remains outside ordinary state and may
|
||||
retain rendered prompts, schemas, input bodies, generated bodies, and effective
|
||||
parameters according to its documented contract. Weatherreporter does not
|
||||
automatically clean that operator-selected location. Credentials must remain
|
||||
excluded from debug capture.
|
||||
|
||||
## Compatibility And Upgrade Policy
|
||||
|
||||
This is an intentional breaking change to the workspace and inspection
|
||||
contracts. Weatherreporter does not need to migrate historical V1 or V2
|
||||
metadata, prompt receipts, intermediate generated-text artifacts, or managed
|
||||
reports into the new representation.
|
||||
|
||||
Legacy workspace trees must not be silently interpreted as current published
|
||||
state. They also must not be deleted automatically merely because a new
|
||||
version starts: an operator may have placed or referenced files there despite
|
||||
the absence of a continuing application compatibility promise. Release notes
|
||||
and operations documentation must explain whether legacy data can be removed
|
||||
manually and identify the exact safe target.
|
||||
|
||||
The change should land in a release whose notes clearly identify removed CLI
|
||||
commands, obsolete paths and schemas, the new bounded state behavior, and any
|
||||
upgrade action. Because Weatherreporter remains pre-1.0, the ordinary semantic
|
||||
version policy may carry this breaking change without inventing a migration
|
||||
framework.
|
||||
|
||||
## Required Architecture Decision Record
|
||||
|
||||
The implemented feature must include an Accepted ADR recording the durable
|
||||
architectural decision to use ephemeral operational state. The ADR is not part
|
||||
of this roadmap-writing pass and should not be created until implementation is
|
||||
being prepared.
|
||||
|
||||
The ADR should record:
|
||||
|
||||
- the mismatch between run-addressed provenance storage and the ephemeral
|
||||
weather-report lifecycle;
|
||||
- the decision to retain bounded current report and comparison state rather
|
||||
than historical runs;
|
||||
- the distinction between temporary invocation state, published operational
|
||||
state, explicit output copies, and secure debug capture;
|
||||
- the removal of historical inspection and backward-compatibility guarantees;
|
||||
- atomic publication and failed-run behavior;
|
||||
- the alternatives considered, including retaining the current archive,
|
||||
adding time-based retention, or keeping a bounded run history; and
|
||||
- consequences for CLI compatibility, workspace layout, testing, operations,
|
||||
and future schema changes.
|
||||
|
||||
Once accepted, the ADR owns the decision rationale. The architecture policy
|
||||
owns the resulting current invariant, while focused state, CLI, operations,
|
||||
and integration documents own the implemented contracts.
|
||||
|
||||
## Scope
|
||||
|
||||
The completed feature includes:
|
||||
|
||||
- an invocation-scoped temporary workspace for intermediate generation state;
|
||||
- atomic publication of the current managed report and its minimal operational
|
||||
state;
|
||||
- a bounded comparison snapshot representing the last successfully published
|
||||
report for each supported report key and valid period;
|
||||
- safe cleanup behavior for normal completion, handled failure, and abandoned
|
||||
temporary workspaces;
|
||||
- removal of durable prompt preparation, prompt execution, generated-text,
|
||||
render-context, data-package, notification, and run-metadata history;
|
||||
- removal of run-history inspection commands and their application/state
|
||||
contracts;
|
||||
- removal of V1 metadata compatibility and current-version coupling for
|
||||
historical prompt artifacts by removing the historical artifact contract;
|
||||
- preservation of active-command partial status, safe errors, and paths to
|
||||
genuinely retained published, operator-owned, or debug outputs;
|
||||
- preservation of explicit output copies, Distributor upload behavior, and
|
||||
opt-in secure LLM debug capture;
|
||||
- risk-appropriate offline tests for atomic publication, comparison baselines,
|
||||
failure isolation, cleanup safety, concurrent invocation safety, and absence
|
||||
of unbounded state growth;
|
||||
- an Accepted ADR documenting the architectural decision; and
|
||||
- updates to canonical architecture, CLI, operations, configuration,
|
||||
troubleshooting, integration, internal, testing, and release documentation
|
||||
where their contracts change.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
This feature does not include:
|
||||
|
||||
- a general-purpose cache, database, archival service, or retention engine;
|
||||
- replaying or resuming interrupted generation;
|
||||
- migrating legacy artifacts into the new representation;
|
||||
- retaining a bounded number of historical runs for convenience;
|
||||
- automatic upload or archival of state to remote storage;
|
||||
- collecting additional provider telemetry or weather-source provenance;
|
||||
- changing report content, prompt text, schemas, profile selection, weather
|
||||
derivation, or batch membership;
|
||||
- deleting operator-owned `--out`, `--out-dir`, or secure debug files;
|
||||
- changing Distributor's report-content contract; or
|
||||
- making live external services part of the default test suite.
|
||||
|
||||
## Safety And Testing Policy
|
||||
|
||||
The state refactoring must preserve Weatherreporter's existing path-safety and
|
||||
atomicity expectations while reducing the amount of durable state. Tests
|
||||
should emphasize observable lifecycle guarantees rather than private file
|
||||
choreography.
|
||||
|
||||
Important risks requiring durable offline coverage include:
|
||||
|
||||
- a failed or canceled generation replacing a previously published report or
|
||||
comparison baseline;
|
||||
- a partially written report becoming visible as current;
|
||||
- Recent Changes selecting an incompatible report, valid period, or failed
|
||||
attempt;
|
||||
- cleanup deleting published, operator-owned, debug, or concurrently active
|
||||
files;
|
||||
- batch partial success corrupting the state of another report;
|
||||
- notification failure rolling back or obscuring a successfully published
|
||||
report;
|
||||
- stale or incompatible current state being treated as valid; and
|
||||
- repeated successful and failed runs causing unbounded ordinary workspace
|
||||
growth.
|
||||
|
||||
Tests remain deterministic, offline, credential-free, and based on real
|
||||
temporary filesystems plus narrow external-boundary fakes. Race-enabled tests
|
||||
are required where publication, cleanup, or concurrent invocation behavior
|
||||
shares mutable filesystem state.
|
||||
|
||||
## Relationship To Domain-Specific Profiles
|
||||
|
||||
The domain-specific profile feature can be implemented before this refactor,
|
||||
but it should not add new historical compatibility or durable-provenance
|
||||
commitments. Profile inspection, selection, override precedence, and effective
|
||||
model resolution remain active-workflow behavior and survive the state change.
|
||||
|
||||
The domain-profile roadmap and implementation plan should acknowledge that
|
||||
prompt artifacts from version `1.0.1` need not remain readable after prompts
|
||||
advance to `1.1.0`. Existing state persistence may remain temporarily while
|
||||
the profile feature lands, but it should not be expanded or treated as the
|
||||
target architecture.
|
||||
|
||||
## Completion Criteria
|
||||
|
||||
The roadmap's target state is achieved when:
|
||||
|
||||
- ordinary runs no longer create durable run-addressed artifact collections;
|
||||
- a successful report atomically replaces only the corresponding current
|
||||
published state;
|
||||
- failed and canceled attempts leave the prior published report and Recent
|
||||
Changes baseline unchanged;
|
||||
- Daily, Today, and Tomorrow compare against at most one compatible snapshot
|
||||
from the last successfully published report;
|
||||
- expired comparison and published state can be removed safely without
|
||||
touching operator-owned or active files;
|
||||
- Hourly does not retain an unused comparison snapshot;
|
||||
- historical inspection commands and V1/V2 archival compatibility code are
|
||||
removed;
|
||||
- temporary, published, explicit-output, and debug paths have distinct and
|
||||
documented ownership and cleanup rules;
|
||||
- Distributor and active-command summaries continue to receive the completed
|
||||
report and safe status information they require;
|
||||
- the default suite proves atomicity, bounded growth, cleanup safety, batch
|
||||
isolation, and comparison correctness offline;
|
||||
- an Accepted ADR records the architectural decision and alternatives; and
|
||||
- canonical current-state documentation describes only the implemented
|
||||
lifecycle.
|
||||
|
||||
## Open Questions
|
||||
|
||||
### Lifetime of the current managed report
|
||||
|
||||
Recommendation: retain one current managed report per logical report key and
|
||||
valid period until it is replaced or its valid period expires. This preserves
|
||||
the current default behavior for invocations without `--out` while bounding
|
||||
growth.
|
||||
|
||||
Alternative: treat the managed report as temporary and retain output only when
|
||||
the operator supplies `--out` or `--out-dir`. This minimizes state further but
|
||||
makes a successful default invocation produce no durable report for the user
|
||||
and complicates Distributor sequencing.
|
||||
|
||||
### Historical inspection replacement
|
||||
|
||||
Recommendation: remove the run-history inspection commands without adding a
|
||||
replacement initially. Current command summaries, current managed files, and
|
||||
opt-in debug capture cover the remaining supported workflows.
|
||||
|
||||
Alternative: add a narrow `inspect current REPORT` command backed only by the
|
||||
current operational manifest. This provides discoverability without history,
|
||||
but it creates a new public surface and may preserve metadata complexity that
|
||||
the refactor is intended to remove.
|
||||
|
||||
### Abandoned temporary workspace cleanup
|
||||
|
||||
Recommendation: use an explicitly owned temporary subtree with per-invocation
|
||||
ownership markers and a conservative age threshold. Normal cleanup removes the
|
||||
current invocation synchronously; opportunistic cleanup removes only marked,
|
||||
inactive directories old enough that they cannot reasonably belong to a live
|
||||
invocation.
|
||||
|
||||
Alternative: perform only synchronous cleanup and document manual removal of
|
||||
directories left by process termination. This minimizes destructive code and
|
||||
concurrency risk, but crashed processes can still accumulate unbounded files.
|
||||
|
||||
### Legacy workspace cleanup
|
||||
|
||||
Recommendation: ignore legacy run-addressed trees and document a precise,
|
||||
manual one-time cleanup procedure. Do not automatically delete them during
|
||||
startup or upgrade.
|
||||
|
||||
Alternative: add an explicit cleanup command that previews and then removes
|
||||
recognized legacy artifacts. This is more convenient for large installations
|
||||
but introduces a destructive command and a legacy-format classifier that must
|
||||
be maintained and tested.
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,6 +1,11 @@
|
||||
# Promptkit Feature Request: Application Fallback Profiles
|
||||
|
||||
Status: Proposed upstream capability.
|
||||
Status: Implemented upstream in Promptkit v0.5.0.
|
||||
|
||||
Promptkit v0.5.0 resolved this request with the public
|
||||
`WithFallbackProfileFS` engine option and the precedence and error semantics
|
||||
specified below. This document is retained as the downstream rationale for
|
||||
the capability.
|
||||
|
||||
## Purpose
|
||||
|
||||
|
||||
@@ -1,7 +1,10 @@
|
||||
# Troubleshooting
|
||||
|
||||
Keep failed workspace artifacts in place. When a RunID is available, start
|
||||
with `weatherreporter inspect metadata RUN_ID` and use the paths in its result.
|
||||
Start with the command's classified error. When content-rich prompt diagnostics
|
||||
are needed, enable a new run with `--llm-debug-dir` and handle the resulting
|
||||
secure capture as sensitive. Current-version workspace receipts can provide
|
||||
additional context when present, but are transitional state rather than a
|
||||
long-term troubleshooting interface.
|
||||
|
||||
## Prompt inspection or credentials fail before collection
|
||||
|
||||
@@ -11,13 +14,37 @@ the configured `promptkit` profile or profile source, confirm the exact
|
||||
Promptkit asset is available, and supply any reported environment credential.
|
||||
Do not add provider keys to YAML. See [configuration](config.md).
|
||||
|
||||
## Local profile override is malformed or selects an unexpected model
|
||||
|
||||
`promptkit.profile_file` and `promptkit.profile_dir` supply complete profile
|
||||
definitions. A same-ID definition replaces the embedded profile, and a malformed
|
||||
matching definition fails before collection instead of falling back. Validate
|
||||
the selected profile's YAML, ID, backend or endpoint, and model. If the model
|
||||
is unexpected, first check the global `promptkit.profile` selection and then
|
||||
look for a same-ID definition in the configured file or directory.
|
||||
|
||||
Current-version preparation and execution receipts may retain the selected
|
||||
profile ID and effective backend/model, but not an endpoint or credential.
|
||||
Use them only as supplemental context after the active command error or an
|
||||
explicit secure debug capture. See the maintained
|
||||
[local `weather-light` profile example](../examples/weather-light-local-profile.yml).
|
||||
|
||||
## Local model endpoint is unavailable
|
||||
|
||||
An endpoint-only `weather-light` override can pass preflight and still fail
|
||||
during provider preparation or execution when the local server is unavailable
|
||||
or does not accept the configured model. Start the local server, correct the
|
||||
endpoint or model in the profile, and run the command again. Weatherreporter
|
||||
does not probe endpoints or automatically use a remote profile instead.
|
||||
|
||||
## Preparation, capacity, or execution fails
|
||||
|
||||
A preparation failure occurs before provider work; an execution failure occurs
|
||||
after preparation. Both leave safe provenance and metadata when reached. A
|
||||
capacity error for one batch report does not retry that report or prevent later
|
||||
independent reports. Inspect the preparation or execution path, correct the
|
||||
profile/backend condition, and create a new run. See [operations](operations.md).
|
||||
after preparation. A capacity error for one batch report does not retry that
|
||||
report or prevent later independent reports. Correct the profile or backend
|
||||
condition identified by the bounded command error, then create a new run.
|
||||
Use explicit secure debug capture only when additional content-rich diagnostics
|
||||
are necessary. See [operations](operations.md).
|
||||
|
||||
## Generated text fails validation
|
||||
|
||||
|
||||
4
examples/weather-light-local-profile.yml
Normal file
4
examples/weather-light-local-profile.yml
Normal file
@@ -0,0 +1,4 @@
|
||||
id: weather-light
|
||||
endpoint: http://127.0.0.1:11434/v1
|
||||
model: weather-local
|
||||
timeout_seconds: 180
|
||||
@@ -45,6 +45,7 @@ func newAdapter(config Config, additionalOptions ...promptkit.Option) (*Adapter,
|
||||
options := []promptkit.Option{
|
||||
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
|
||||
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
|
||||
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
|
||||
}
|
||||
if config.ProfileFile != "" {
|
||||
options = append(options, promptkit.WithProfileFile(config.ProfileFile))
|
||||
|
||||
@@ -68,11 +68,11 @@ func (client *fakeClient) request() promptkit.GenerateRequest {
|
||||
|
||||
func TestInspectPromptAndProfile(t *testing.T) {
|
||||
adapter := newTestAdapter(t, &fakeClient{})
|
||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.0.1")
|
||||
inspection, err := adapter.InspectPrompt(context.Background(), "weather.daily_generated_text", "1.1.0")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPrompt() error = %v", err)
|
||||
}
|
||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.0.1" || inspection.DefaultProfileID != "gemini-flash-latest" {
|
||||
if inspection.PromptID != "weather.daily_generated_text" || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != "weather-balanced" {
|
||||
t.Fatalf("inspection = %#v", inspection)
|
||||
}
|
||||
if len(inspection.Inputs) != 1 || inspection.Inputs[0].Name != "data_package" || !inspection.Inputs[0].Required || inspection.Inputs[0].ContentType != "application/yaml" {
|
||||
@@ -102,6 +102,107 @@ func TestInspectPromptAndProfile(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesAreAvailableToProductionAndTestAdapters(t *testing.T) {
|
||||
adapter, err := New(Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
for _, want := range []struct {
|
||||
id string
|
||||
backend string
|
||||
model string
|
||||
}{
|
||||
{"weather-light", "openrouter", "deepseek/deepseek-v4-flash"},
|
||||
{"weather-balanced", "openrouter", "~google/gemini-flash-latest"},
|
||||
{"weather-deep", "openrouter", "~anthropic/claude-sonnet-latest"},
|
||||
} {
|
||||
t.Run(want.id, func(t *testing.T) {
|
||||
assertProfile(t, adapter, want.id, want.backend, want.model)
|
||||
})
|
||||
}
|
||||
|
||||
testAdapter, err := newAdapterForTest(Config{}, &fakeClient{})
|
||||
if err != nil {
|
||||
t.Fatalf("newAdapterForTest() error = %v", err)
|
||||
}
|
||||
assertProfile(t, testAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||
}
|
||||
|
||||
func TestConfiguredProfilesOverrideEmbeddedFallbacks(t *testing.T) {
|
||||
file := writeProfileFile(t, `id: weather-light
|
||||
endpoint: https://local-file.example/v1
|
||||
model: file-light
|
||||
`)
|
||||
fileAdapter, err := New(Config{ProfileFile: file})
|
||||
if err != nil {
|
||||
t.Fatalf("New(profile file) error = %v", err)
|
||||
}
|
||||
assertProfile(t, fileAdapter, "weather-light", "", "file-light")
|
||||
|
||||
directory := testProfileDirectory(t, `id: weather-light
|
||||
backend: local
|
||||
model: directory-light
|
||||
`)
|
||||
directoryAdapter, err := New(Config{ProfileDirectory: directory, LocalEndpoint: "https://local-directory.example/v1"})
|
||||
if err != nil {
|
||||
t.Fatalf("New(profile directory) error = %v", err)
|
||||
}
|
||||
assertProfile(t, directoryAdapter, "weather-light", promptkit.BackendLocal, "directory-light")
|
||||
}
|
||||
|
||||
func TestMaintainedWeatherLightLocalProfileExampleInspectsOffline(t *testing.T) {
|
||||
adapter, err := New(Config{ProfileFile: filepath.Join("..", "..", "..", "examples", "weather-light-local-profile.yml")})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
assertProfile(t, adapter, "weather-light", "", "weather-local")
|
||||
}
|
||||
|
||||
func TestProfileResolutionFallsThroughOnlyWhenTheConfiguredIDIsAbsent(t *testing.T) {
|
||||
absentAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: other-profile
|
||||
backend: openrouter
|
||||
model: other-model
|
||||
`)})
|
||||
if err != nil {
|
||||
t.Fatalf("New(absent profile) error = %v", err)
|
||||
}
|
||||
assertProfile(t, absentAdapter, "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||
|
||||
malformedAdapter, err := New(Config{ProfileDirectory: testProfileDirectory(t, `id: weather-light
|
||||
backend: openrouter
|
||||
`)})
|
||||
if err != nil {
|
||||
t.Fatalf("New(malformed profile) error = %v", err)
|
||||
}
|
||||
if _, err := malformedAdapter.InspectProfile(context.Background(), "weather-light"); err == nil {
|
||||
t.Fatal("InspectProfile() error = nil, want malformed configured profile error")
|
||||
}
|
||||
}
|
||||
|
||||
func TestProfileResolutionPreservesBuiltInAndExplicitPrecedence(t *testing.T) {
|
||||
adapter, err := New(Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New() error = %v", err)
|
||||
}
|
||||
builtin, err := adapter.InspectProfile(context.Background(), "gemini-flash-latest")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile(builtin) error = %v", err)
|
||||
}
|
||||
if builtin.ProfileID != "gemini-flash-latest" || builtin.BackendID != "openrouter" || builtin.ModelName == "" {
|
||||
t.Fatalf("builtin profile = %#v", builtin)
|
||||
}
|
||||
|
||||
explicit, err := newAdapter(Config{}, promptkit.WithProfiles(promptkit.Profile{
|
||||
ID: "weather-light",
|
||||
Endpoint: "https://explicit.example/v1",
|
||||
Model: "explicit-light",
|
||||
}))
|
||||
if err != nil {
|
||||
t.Fatalf("newAdapter(explicit profile) error = %v", err)
|
||||
}
|
||||
assertProfile(t, explicit, "weather-light", "", "explicit-light")
|
||||
}
|
||||
|
||||
func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
|
||||
client := &fakeClient{response: validResponse()}
|
||||
adapter := newTestAdapter(t, client)
|
||||
@@ -141,6 +242,44 @@ func TestExecuteUsesPreparedInlineDataPackage(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteEmbeddedHourlyProfileThroughPreparedPath(t *testing.T) {
|
||||
t.Setenv("OPENROUTER_API_KEY", "test-openrouter-key")
|
||||
client := &fakeClient{response: hourlyValidResponse()}
|
||||
adapter, err := newAdapter(Config{}, promptkit.WithLLMClient(client))
|
||||
if err != nil {
|
||||
t.Fatalf("newAdapter() error = %v", err)
|
||||
}
|
||||
request := promptexec.ExecuteRequest{
|
||||
PromptID: "weather.hourly_generated_text",
|
||||
PromptVersion: "1.1.0",
|
||||
ProfileID: "weather-light",
|
||||
DataPackage: []byte("report:\n id: hourly\nbriefing: {}\n"),
|
||||
DataPackagePath: "data-packages/hourly/data_package.yaml",
|
||||
}
|
||||
var preparation promptexec.Preparation
|
||||
prepared := false
|
||||
result, err := adapter.Execute(context.Background(), request, func(value promptexec.Preparation, _ *promptexec.PreparationDebug) error {
|
||||
if client.callCount() != 0 {
|
||||
t.Fatal("provider was called before preparation completed")
|
||||
}
|
||||
preparation = value
|
||||
prepared = true
|
||||
return nil
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Execute() error = %v", err)
|
||||
}
|
||||
if !prepared || preparation.ProfileID != "weather-light" || preparation.BackendID != "openrouter" || preparation.ModelName != "deepseek/deepseek-v4-flash" {
|
||||
t.Fatalf("preparation = %#v", preparation)
|
||||
}
|
||||
if result == nil || result.ProfileID != "weather-light" || result.BackendID != "openrouter" || result.ModelName != "deepseek/deepseek-v4-flash" || result.Validation.Status != promptexec.ValidationPassed {
|
||||
t.Fatalf("execution = %#v", result)
|
||||
}
|
||||
if client.callCount() != 1 || client.request().Target.Model != "deepseek/deepseek-v4-flash" {
|
||||
t.Fatalf("provider calls/request = %d/%#v", client.callCount(), client.request())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteUsesExactInlineDataPackageProvenance(t *testing.T) {
|
||||
client := &fakeClient{response: validResponse()}
|
||||
reader := &recordingReader{}
|
||||
@@ -339,6 +478,17 @@ func newTestAdapter(t *testing.T, client promptkit.LLMClient) *Adapter {
|
||||
return newTestAdapterWithOptions(t, client)
|
||||
}
|
||||
|
||||
func assertProfile(t *testing.T, adapter *Adapter, id string, backend string, model string) {
|
||||
t.Helper()
|
||||
profile, err := adapter.InspectProfile(context.Background(), id)
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile(%q) error = %v", id, err)
|
||||
}
|
||||
if profile.ProfileID != id || profile.BackendID != backend || profile.ModelName != model {
|
||||
t.Fatalf("profile = %#v, want %q with backend/model %q/%q", profile, id, backend, model)
|
||||
}
|
||||
}
|
||||
|
||||
func newTestAdapterWithOptions(t *testing.T, client promptkit.LLMClient, options ...promptkit.Option) *Adapter {
|
||||
t.Helper()
|
||||
profiles := testProfileDirectory(t, `id: test-profile
|
||||
@@ -366,10 +516,19 @@ func testProfileDirectory(t *testing.T, profile string) string {
|
||||
return profiles
|
||||
}
|
||||
|
||||
func writeProfileFile(t *testing.T, profile string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "profile.yml")
|
||||
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
|
||||
t.Fatalf("write profile: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
|
||||
func testExecuteRequest() promptexec.ExecuteRequest {
|
||||
return promptexec.ExecuteRequest{
|
||||
PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "1.0.1",
|
||||
PromptVersion: "1.1.0",
|
||||
ProfileID: "test-profile",
|
||||
DataPackage: []byte("report:\n id: daily\nbriefing: {}\n"),
|
||||
DataPackagePath: "data-packages/daily/data_package.yaml",
|
||||
@@ -382,3 +541,10 @@ func validResponse() *promptkit.GenerateResponse {
|
||||
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
|
||||
}
|
||||
}
|
||||
|
||||
func hourlyValidResponse() *promptkit.GenerateResponse {
|
||||
return &promptkit.GenerateResponse{
|
||||
Content: `{"summary":"A quiet hour is expected.","forecast_discussion":"Conditions remain settled.","precipitation_timing":""}`,
|
||||
Usage: promptkit.TokenUsage{PromptTokens: 12, CompletionTokens: 8, TotalTokens: 20},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -33,10 +33,10 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
|
||||
t.Fatalf("batchInspectionCandidates() error = %v", err)
|
||||
}
|
||||
executor := &inspectionExecutor{profiles: map[string]promptexec.ProfileInspection{
|
||||
"default-profile": {ProfileID: "default-profile", BackendID: "local", ModelName: "model"},
|
||||
"weather-balanced": {ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
|
||||
}, prompts: map[string]promptexec.PromptInspection{}}
|
||||
for _, candidate := range candidates {
|
||||
executor.prompts[candidate.Definition.PromptID] = validPromptInspection(candidate.Definition)
|
||||
executor.prompts[candidate.Definition.PromptID] = logicalPromptInspection(candidate.Definition)
|
||||
}
|
||||
collector := collectorFunc(func(context.Context, collect.Request) (*collect.Result, error) {
|
||||
return nil, errors.New("collection reached")
|
||||
@@ -47,7 +47,7 @@ func TestRunBatchDetailedInspectsEveryCandidateBeforeCollection(t *testing.T) {
|
||||
if err == nil || err.Error() != "collection reached" {
|
||||
t.Fatalf("RunBatchDetailed() error = %v, want collection error", err)
|
||||
}
|
||||
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 {
|
||||
if len(executor.promptRequests) != test.wantPrompts || len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
|
||||
t.Fatalf("inspection calls = prompts %#v profiles %#v", executor.promptRequests, executor.profileRequests)
|
||||
}
|
||||
})
|
||||
|
||||
@@ -44,7 +44,7 @@ func (e *assembledBatchExecutor) InspectPrompt(_ context.Context, id, version st
|
||||
if !ok || definition.PromptVersion != version {
|
||||
return promptexec.PromptInspection{}, errors.New("unexpected prompt inspection")
|
||||
}
|
||||
return validPromptInspection(definition), nil
|
||||
return logicalPromptInspection(definition), nil
|
||||
}
|
||||
|
||||
func (e *assembledBatchExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
||||
@@ -151,8 +151,8 @@ func TestRunBatchDetailedExecutesRetainedReportsSequentially(t *testing.T) {
|
||||
if len(executor.executeRequests) != len(test.wantIDs) || executor.maxActive != 1 {
|
||||
t.Fatalf("executor calls/max active = %d/%d, want %d/1", len(executor.executeRequests), executor.maxActive, len(test.wantIDs))
|
||||
}
|
||||
if len(executor.profileRequests) != 1 {
|
||||
t.Fatalf("profile inspections = %#v, want one shared profile inspection", executor.profileRequests)
|
||||
if len(executor.profileRequests) != 1 || executor.profileRequests[0] != "weather-balanced" {
|
||||
t.Fatalf("profile inspections = %#v, want one shared weather-balanced inspection", executor.profileRequests)
|
||||
}
|
||||
for index, item := range result.Reports {
|
||||
if item.ReportID != test.wantIDs[index] || item.Status != "succeeded" {
|
||||
|
||||
@@ -189,3 +189,13 @@ func validPromptInspection(definition report.Definition) promptexec.PromptInspec
|
||||
Output: promptexec.OutputContract{Format: "json", ValidationMode: "json_schema", SchemaPath: definition.GeneratedTextSchemaID + ".generated_text.schema.json"},
|
||||
}
|
||||
}
|
||||
|
||||
func logicalPromptInspection(definition report.Definition) promptexec.PromptInspection {
|
||||
inspection := validPromptInspection(definition)
|
||||
if definition.ID == report.Hourly {
|
||||
inspection.DefaultProfileID = "weather-light"
|
||||
} else {
|
||||
inspection.DefaultProfileID = "weather-balanced"
|
||||
}
|
||||
return inspection
|
||||
}
|
||||
|
||||
73
internal/app/prompt_profile_integration_test.go
Normal file
73
internal/app/prompt_profile_integration_test.go
Normal file
@@ -0,0 +1,73 @@
|
||||
package app_test
|
||||
|
||||
import (
|
||||
"context"
|
||||
"os"
|
||||
"path/filepath"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
promptkitadapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/promptkit"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/app"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
|
||||
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
|
||||
)
|
||||
|
||||
func TestPromptInspectionResolvesEmbeddedAndOverriddenProfilesOffline(t *testing.T) {
|
||||
lookupEnv := func(string) (string, bool) { return "test-key", true }
|
||||
inspect := func(t *testing.T, adapter *promptkitadapter.Adapter, id report.ID, profile string, wantID string, wantBackend string, wantModel string) {
|
||||
t.Helper()
|
||||
result, err := app.InspectPromptExecution(context.Background(), app.PromptInspectionRequest{
|
||||
Resolved: resolvedPromptProfile(t, id),
|
||||
Executor: adapter,
|
||||
Promptkit: config.PromptkitConfig{Profile: profile},
|
||||
LookupEnv: lookupEnv,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPromptExecution() error = %v", err)
|
||||
}
|
||||
if result.ProfileID != wantID || result.BackendID != wantBackend || result.ModelName != wantModel {
|
||||
t.Fatalf("inspection = %#v, want profile/backend/model %q/%q/%q", result, wantID, wantBackend, wantModel)
|
||||
}
|
||||
}
|
||||
|
||||
embedded, err := promptkitadapter.New(promptkitadapter.Config{})
|
||||
if err != nil {
|
||||
t.Fatalf("New(embedded) error = %v", err)
|
||||
}
|
||||
inspect(t, embedded, report.Hourly, "", "weather-light", "openrouter", "deepseek/deepseek-v4-flash")
|
||||
inspect(t, embedded, report.Daily, "", "weather-balanced", "openrouter", "~google/gemini-flash-latest")
|
||||
inspect(t, embedded, report.Daily, "weather-deep", "weather-deep", "openrouter", "~anthropic/claude-sonnet-latest")
|
||||
|
||||
override, err := promptkitadapter.New(promptkitadapter.Config{ProfileFile: writeProfileFile(t, `id: weather-light
|
||||
endpoint: https://local.example/v1
|
||||
model: local-weather
|
||||
`)})
|
||||
if err != nil {
|
||||
t.Fatalf("New(override) error = %v", err)
|
||||
}
|
||||
inspect(t, override, report.Hourly, "", "weather-light", "", "local-weather")
|
||||
}
|
||||
|
||||
func resolvedPromptProfile(t *testing.T, id report.ID) report.Resolved {
|
||||
t.Helper()
|
||||
now := time.Date(2026, 5, 29, 12, 0, 0, 0, time.UTC)
|
||||
request := report.ResolveRequest{Now: now, Location: time.UTC}
|
||||
if id == report.Daily {
|
||||
request.Date = now
|
||||
}
|
||||
resolved, err := report.DefaultRegistry().Resolve(id, request)
|
||||
if err != nil {
|
||||
t.Fatalf("Resolve(%q) error = %v", id, err)
|
||||
}
|
||||
return resolved
|
||||
}
|
||||
|
||||
func writeProfileFile(t *testing.T, profile string) string {
|
||||
t.Helper()
|
||||
path := filepath.Join(t.TempDir(), "profile.yml")
|
||||
if err := os.WriteFile(path, []byte(profile), 0o600); err != nil {
|
||||
t.Fatalf("write profile: %v", err)
|
||||
}
|
||||
return path
|
||||
}
|
||||
@@ -37,8 +37,10 @@ func (c *workflowCollector) Run(context.Context, collect.Request) (*collect.Resu
|
||||
type workflowExecutor struct {
|
||||
definition report.Definition
|
||||
raw []byte
|
||||
prompt promptexec.PromptInspection
|
||||
inspectionErr error
|
||||
profile promptexec.ProfileInspection
|
||||
profileErr error
|
||||
beforePreparationErr error
|
||||
afterCallbackErr error
|
||||
afterPreparationErr error
|
||||
@@ -49,6 +51,8 @@ type workflowExecutor struct {
|
||||
beforeProvider func()
|
||||
preparationDebug *promptexec.PreparationDebug
|
||||
executionDebug *promptexec.ExecutionDebug
|
||||
preparation *promptexec.Preparation
|
||||
execution *promptexec.Execution
|
||||
}
|
||||
|
||||
func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string) (promptexec.PromptInspection, error) {
|
||||
@@ -58,10 +62,16 @@ func (e *workflowExecutor) InspectPrompt(_ context.Context, id, version string)
|
||||
if id != e.definition.PromptID || version != e.definition.PromptVersion {
|
||||
return promptexec.PromptInspection{}, errors.New("unexpected prompt identity")
|
||||
}
|
||||
if e.prompt.PromptID != "" {
|
||||
return e.prompt, nil
|
||||
}
|
||||
return validPromptInspection(e.definition), nil
|
||||
}
|
||||
|
||||
func (e *workflowExecutor) InspectProfile(_ context.Context, id string) (promptexec.ProfileInspection, error) {
|
||||
if e.profileErr != nil {
|
||||
return promptexec.ProfileInspection{}, e.profileErr
|
||||
}
|
||||
profile := e.profile
|
||||
if profile.ProfileID == "" {
|
||||
profile = promptexec.ProfileInspection{ProfileID: id, BackendID: "fixture", ModelName: "fixture-model"}
|
||||
@@ -76,11 +86,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
|
||||
return nil, e.beforePreparationErr
|
||||
}
|
||||
stamp := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
|
||||
profile := e.profile
|
||||
if profile.ProfileID == "" {
|
||||
profile = promptexec.ProfileInspection{ProfileID: req.ProfileID, BackendID: "fixture", ModelName: "fixture-model"}
|
||||
}
|
||||
preparation := promptexec.Preparation{
|
||||
PromptID: req.PromptID, PromptVersion: req.PromptVersion, PromptHash: "prompt-hash",
|
||||
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: "fixture",
|
||||
ModelName: "fixture-model", DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
|
||||
RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID, BackendID: profile.BackendID,
|
||||
ModelName: profile.ModelName, DataPackagePath: req.DataPackagePath, StartedAt: stamp, EndedAt: stamp,
|
||||
}
|
||||
e.preparation = &preparation
|
||||
if err := callback(preparation, e.preparationDebug); err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -98,14 +113,16 @@ func (e *workflowExecutor) Execute(_ context.Context, req promptexec.ExecuteRequ
|
||||
if validation == "" {
|
||||
validation = promptexec.ValidationPassed
|
||||
}
|
||||
return &promptexec.Execution{
|
||||
execution := &promptexec.Execution{
|
||||
RunID: "provider-run", PromptID: req.PromptID, PromptVersion: req.PromptVersion,
|
||||
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: req.ProfileID,
|
||||
BackendID: "fixture", ModelName: "fixture-model", GeneratedHash: "generated-hash",
|
||||
BackendID: profile.BackendID, ModelName: profile.ModelName, GeneratedHash: "generated-hash",
|
||||
StartedAt: stamp, EndedAt: stamp, DataPackagePath: req.DataPackagePath, RawOutput: e.raw,
|
||||
Debug: e.executionDebug,
|
||||
Validation: promptexec.NewValidation(validation, "json_schema", e.definition.GeneratedTextSchemaID+".generated_text.schema.json", nil),
|
||||
}, nil
|
||||
}
|
||||
e.execution = execution
|
||||
return execution, nil
|
||||
}
|
||||
|
||||
type workflowNotifier struct {
|
||||
@@ -208,6 +225,57 @@ func TestGenerateDetailedCompletesRetainedReportWorkflows(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestGenerateDetailedPreservesSelectedProfileThroughExecution(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
kind ReportKind
|
||||
id report.ID
|
||||
raw string
|
||||
override string
|
||||
profile promptexec.ProfileInspection
|
||||
}{
|
||||
{
|
||||
name: "hourly default", kind: ReportHourly, id: report.Hourly, raw: validHourlyWorkflowJSON(),
|
||||
profile: promptexec.ProfileInspection{ProfileID: "weather-light", BackendID: "openrouter", ModelName: "deepseek/deepseek-v4-flash"},
|
||||
},
|
||||
{
|
||||
name: "daily default", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(),
|
||||
profile: promptexec.ProfileInspection{ProfileID: "weather-balanced", BackendID: "openrouter", ModelName: "~google/gemini-flash-latest"},
|
||||
},
|
||||
{
|
||||
name: "global override", kind: ReportDaily, id: report.Daily, raw: validDailyWorkflowJSON(), override: "operator-profile",
|
||||
profile: promptexec.ProfileInspection{ProfileID: "operator-profile", BackendID: "local", ModelName: "local-weather-model"},
|
||||
},
|
||||
}
|
||||
for _, test := range tests {
|
||||
t.Run(test.name, func(t *testing.T) {
|
||||
cfg := workflowConfig(t)
|
||||
cfg.Promptkit.Profile = test.override
|
||||
definition := report.DefaultRegistry().MustLookup(test.id)
|
||||
executor := &workflowExecutor{
|
||||
definition: definition, prompt: logicalPromptInspection(definition), profile: test.profile, raw: []byte(test.raw),
|
||||
}
|
||||
bundle := workflowBundle(t)
|
||||
_, err := GenerateDetailed(context.Background(), GenerateRequest{
|
||||
Config: cfg, Report: test.kind, Date: workflowTime("2026-05-29T12:00:00-05:00"), Now: workflowTime("2026-05-29T08:30:00-05:00"),
|
||||
Collector: &workflowCollector{result: &collect.Result{Bundle: &bundle}}, Executor: executor, Notifier: &workflowNotifier{},
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("GenerateDetailed() error = %v", err)
|
||||
}
|
||||
if executor.request.ProfileID != test.profile.ProfileID {
|
||||
t.Fatalf("execution profile = %q, want %q", executor.request.ProfileID, test.profile.ProfileID)
|
||||
}
|
||||
if executor.preparation == nil || executor.preparation.ProfileID != test.profile.ProfileID || executor.preparation.BackendID != test.profile.BackendID || executor.preparation.ModelName != test.profile.ModelName {
|
||||
t.Fatalf("prepared profile = %#v, want %q/%q/%q", executor.preparation, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
|
||||
}
|
||||
if executor.execution == nil || executor.execution.ProfileID != test.profile.ProfileID || executor.execution.BackendID != test.profile.BackendID || executor.execution.ModelName != test.profile.ModelName {
|
||||
t.Fatalf("executed profile = %#v, want %q/%q/%q", executor.execution, test.profile.ProfileID, test.profile.BackendID, test.profile.ModelName)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type preparationFailingStore struct {
|
||||
state.Store
|
||||
}
|
||||
@@ -299,6 +367,11 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
|
||||
wantCategory promptexec.ErrorCategory
|
||||
}{
|
||||
{name: "inspection", configure: func(e *workflowExecutor) { e.inspectionErr = errors.New("inspection unavailable") }, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "unknown profile", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unknown selected profile") }, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "malformed profile", configure: func(e *workflowExecutor) {
|
||||
e.profileErr = errors.New("malformed profile at https://operator.example/v1 api_key=secret")
|
||||
}, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "unusable backend", configure: func(e *workflowExecutor) { e.profileErr = errors.New("unsupported backend") }, wantCategory: promptexec.InvalidConfiguration},
|
||||
{name: "credential", configure: func(e *workflowExecutor) {
|
||||
e.profile = promptexec.ProfileInspection{ProfileID: "default-profile", CredentialRequired: true}
|
||||
}, wantCategory: promptexec.MissingCredential},
|
||||
@@ -317,6 +390,9 @@ func TestGenerateDetailedRejectsInspectionAndCredentialsBeforeCollection(t *test
|
||||
if err == nil || result != nil || promptexec.CategoryOf(err) != test.wantCategory || collector.calls != 0 || executor.executeCalls != 0 {
|
||||
t.Fatalf("result/error/category/collect/execute = %#v/%v/%q/%d/%d", result, err, promptexec.CategoryOf(err), collector.calls, executor.executeCalls)
|
||||
}
|
||||
if strings.Contains(err.Error(), "operator.example") || strings.Contains(err.Error(), "secret") {
|
||||
t.Fatalf("error leaks profile details: %v", err)
|
||||
}
|
||||
entries, readErr := os.ReadDir(cfg.Workspace.Root)
|
||||
if readErr != nil || len(entries) != 0 {
|
||||
t.Fatalf("workspace entries/error = %#v/%v, want no writes before collection", entries, readErr)
|
||||
|
||||
@@ -0,0 +1,6 @@
|
||||
id: weather-balanced
|
||||
backend: openrouter
|
||||
model: "~google/gemini-flash-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
6
internal/promptassets/assets/profiles/weather-deep.yml
Normal file
6
internal/promptassets/assets/profiles/weather-deep.yml
Normal file
@@ -0,0 +1,6 @@
|
||||
id: weather-deep
|
||||
backend: openrouter
|
||||
model: "~anthropic/claude-sonnet-latest"
|
||||
reasoning_effort: high
|
||||
timeout_seconds: 240
|
||||
service_tier: flex
|
||||
5
internal/promptassets/assets/profiles/weather-light.yml
Normal file
5
internal/promptassets/assets/profiles/weather-light.yml
Normal file
@@ -0,0 +1,5 @@
|
||||
id: weather-light
|
||||
backend: openrouter
|
||||
model: deepseek/deepseek-v4-flash
|
||||
timeout_seconds: 180
|
||||
service_tier: flex
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.daily_generated_text
|
||||
version: "1.0.1"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Daily weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.hourly_generated_text
|
||||
version: "1.0.1"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-light
|
||||
description: Hourly weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.today_generated_text
|
||||
version: "1.0.1"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Today's weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
id: weather.tomorrow_generated_text
|
||||
version: "1.0.1"
|
||||
default_profile: gemini-flash-latest
|
||||
version: "1.1.0"
|
||||
default_profile: weather-balanced
|
||||
description: Tomorrow's weather report analysis prompt.
|
||||
inputs:
|
||||
- name: data_package
|
||||
|
||||
@@ -7,7 +7,7 @@ import (
|
||||
"io/fs"
|
||||
)
|
||||
|
||||
//go:embed assets/prompts assets/schemas
|
||||
//go:embed assets/prompts assets/profiles assets/schemas
|
||||
var assets embed.FS
|
||||
|
||||
var schemaPaths = map[string]string{
|
||||
@@ -35,6 +35,15 @@ func SchemaFS() fs.FS {
|
||||
return fsys
|
||||
}
|
||||
|
||||
// ProfileFS returns the embedded Weatherreporter Promptkit profile definitions.
|
||||
func ProfileFS() fs.FS {
|
||||
fsys, err := fs.Sub(assets, "assets/profiles")
|
||||
if err != nil {
|
||||
panic(fmt.Sprintf("embedded profile assets: %v", err))
|
||||
}
|
||||
return fsys
|
||||
}
|
||||
|
||||
// Schema returns an independent copy of the canonical schema for id.
|
||||
func Schema(id string) ([]byte, error) {
|
||||
path, ok := schemaPaths[id]
|
||||
|
||||
@@ -34,11 +34,12 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
path string
|
||||
id string
|
||||
schemaID string
|
||||
profile string
|
||||
}{
|
||||
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily"},
|
||||
{"today/today_generated_text.yml", "weather.today_generated_text", "today"},
|
||||
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow"},
|
||||
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly"},
|
||||
{"daily/daily_generated_text.yml", "weather.daily_generated_text", "daily", "weather-balanced"},
|
||||
{"today/today_generated_text.yml", "weather.today_generated_text", "today", "weather-balanced"},
|
||||
{"tomorrow/tomorrow_generated_text.yml", "weather.tomorrow_generated_text", "tomorrow", "weather-balanced"},
|
||||
{"hourly/hourly_generated_text.yml", "weather.hourly_generated_text", "hourly", "weather-light"},
|
||||
}
|
||||
|
||||
definitions := 0
|
||||
@@ -67,8 +68,8 @@ func TestPromptAssetsDeclareTheFourGeneratedTextPrompts(t *testing.T) {
|
||||
if err := yaml.Unmarshal(data, &definition); err != nil {
|
||||
t.Fatalf("decode prompt definition: %v", err)
|
||||
}
|
||||
if definition.ID != tc.id || definition.Version != "1.0.1" || definition.DefaultProfile != "gemini-flash-latest" {
|
||||
t.Fatalf("definition = %#v, want %s version 1.0.1 and gemini-flash-latest", definition, tc.id)
|
||||
if definition.ID != tc.id || definition.Version != "1.1.0" || definition.DefaultProfile != tc.profile {
|
||||
t.Fatalf("definition = %#v, want %s version 1.1.0 and profile %s", definition, tc.id, tc.profile)
|
||||
}
|
||||
if len(definition.Inputs) != 1 || definition.Inputs[0].Name != "data_package" || !definition.Inputs[0].Required || definition.Inputs[0].ContentType != "application/yaml" {
|
||||
t.Fatalf("inputs = %#v, want one required YAML data_package", definition.Inputs)
|
||||
@@ -123,21 +124,116 @@ func TestPromptkitInspectsEmbeddedPromptsOffline(t *testing.T) {
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
|
||||
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
|
||||
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine() error = %v", err)
|
||||
}
|
||||
for _, id := range []string{"weather.daily_generated_text", "weather.today_generated_text", "weather.tomorrow_generated_text", "weather.hourly_generated_text"} {
|
||||
t.Run(id, func(t *testing.T) {
|
||||
inspection, err := engine.InspectPrompt(context.Background(), id, "1.0.1")
|
||||
for _, want := range []struct {
|
||||
id string
|
||||
profile string
|
||||
model string
|
||||
}{
|
||||
{"weather.daily_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
|
||||
{"weather.today_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
|
||||
{"weather.tomorrow_generated_text", "weather-balanced", "~google/gemini-flash-latest"},
|
||||
{"weather.hourly_generated_text", "weather-light", "deepseek/deepseek-v4-flash"},
|
||||
} {
|
||||
t.Run(want.id, func(t *testing.T) {
|
||||
inspection, err := engine.InspectPrompt(context.Background(), want.id, "1.1.0")
|
||||
if err != nil {
|
||||
t.Fatalf("InspectPrompt() error = %v", err)
|
||||
}
|
||||
if inspection.PromptID != id || inspection.PromptVersion != "1.0.1" || inspection.DefaultProfileID != "gemini-flash-latest" {
|
||||
if inspection.PromptID != want.id || inspection.PromptVersion != "1.1.0" || inspection.DefaultProfileID != want.profile {
|
||||
t.Fatalf("inspection = %#v", inspection)
|
||||
}
|
||||
profile, err := engine.InspectProfile(context.Background(), inspection.DefaultProfileID)
|
||||
if err != nil || profile.EffectiveModelParams.Model != want.model {
|
||||
t.Fatalf("profile/error = %#v/%v, want model %q", profile, err, want.model)
|
||||
}
|
||||
})
|
||||
}
|
||||
profile, err := engine.InspectProfile(context.Background(), "weather-deep")
|
||||
if err != nil || profile.EffectiveModelParams.Model != "~anthropic/claude-sonnet-latest" {
|
||||
t.Fatalf("weather-deep profile/error = %#v/%v", profile, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesAreCompleteAndInspectable(t *testing.T) {
|
||||
wantPaths := map[string]bool{
|
||||
"weather-balanced.yml": false,
|
||||
"weather-deep.yml": false,
|
||||
"weather-light.yml": false,
|
||||
}
|
||||
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
if _, ok := wantPaths[path]; !ok {
|
||||
t.Fatalf("unexpected embedded profile asset %q", path)
|
||||
}
|
||||
wantPaths[path] = true
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("walk embedded profiles: %v", err)
|
||||
}
|
||||
for path, found := range wantPaths {
|
||||
if !found {
|
||||
t.Errorf("missing embedded profile asset %q", path)
|
||||
}
|
||||
}
|
||||
|
||||
engine, err := promptkit.NewEngine(promptkit.Config{},
|
||||
promptkit.WithPromptFS(promptassets.PromptFS(), "."),
|
||||
promptkit.WithSchemaFS(promptassets.SchemaFS(), "."),
|
||||
promptkit.WithFallbackProfileFS(promptassets.ProfileFS(), "."),
|
||||
)
|
||||
if err != nil {
|
||||
t.Fatalf("NewEngine() error = %v", err)
|
||||
}
|
||||
profiles := []struct {
|
||||
id string
|
||||
model string
|
||||
timeoutSeconds int
|
||||
reasoningEffort string
|
||||
}{
|
||||
{"weather-light", "deepseek/deepseek-v4-flash", 180, ""},
|
||||
{"weather-balanced", "~google/gemini-flash-latest", 240, "high"},
|
||||
{"weather-deep", "~anthropic/claude-sonnet-latest", 240, "high"},
|
||||
}
|
||||
for _, want := range profiles {
|
||||
t.Run(want.id, func(t *testing.T) {
|
||||
inspection, err := engine.InspectProfile(context.Background(), want.id)
|
||||
if err != nil {
|
||||
t.Fatalf("InspectProfile() error = %v", err)
|
||||
}
|
||||
got := inspection.EffectiveModelParams
|
||||
if inspection.ProfileID != want.id || got.BackendID != "openrouter" || got.Model != want.model || got.TimeoutSeconds != want.timeoutSeconds || got.ServiceTier != "flex" || got.ReasoningEffort != want.reasoningEffort {
|
||||
t.Fatalf("inspection = %#v, want %q using openrouter model %q", inspection, want.id, want.model)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestEmbeddedProfilesExcludeUnsafeOrIncidentalSettings(t *testing.T) {
|
||||
forbidden := []string{"endpoint:", "api_key", "credential", "temperature:", "top_p:", "max_tokens:"}
|
||||
if err := fs.WalkDir(promptassets.ProfileFS(), ".", func(path string, entry fs.DirEntry, err error) error {
|
||||
if err != nil || entry.IsDir() {
|
||||
return err
|
||||
}
|
||||
data, err := fs.ReadFile(promptassets.ProfileFS(), path)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
for _, setting := range forbidden {
|
||||
if strings.Contains(string(data), setting) {
|
||||
t.Fatalf("%s contains forbidden profile setting %q", path, setting)
|
||||
}
|
||||
}
|
||||
return nil
|
||||
}); err != nil {
|
||||
t.Fatalf("walk embedded profiles: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestPromptAssetsExcludeRetiredRuntimeSettings(t *testing.T) {
|
||||
|
||||
@@ -12,7 +12,7 @@ func dailyDefinition() Definition {
|
||||
ID: Daily,
|
||||
Name: "Daily Report",
|
||||
PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "1.0.1",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "daily",
|
||||
GeneratedTextSchemaID: "daily",
|
||||
ComparisonStrategy: CompareSameValidDate,
|
||||
|
||||
@@ -14,7 +14,7 @@ func hourlyDefinition() Definition {
|
||||
ID: Hourly,
|
||||
Name: "Hourly Report",
|
||||
PromptID: "weather.hourly_generated_text",
|
||||
PromptVersion: "1.0.1",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "hourly",
|
||||
GeneratedTextSchemaID: "hourly",
|
||||
ComparisonStrategy: CompareRollingWindow,
|
||||
|
||||
@@ -51,8 +51,11 @@ func TestRegistryContainsOnlyPromptBackedReports(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, definition := range definitions {
|
||||
if definition.PromptVersion != "1.0.1" {
|
||||
t.Fatalf("%s PromptVersion = %q, want 1.0.1", definition.ID, definition.PromptVersion)
|
||||
if definition.PromptVersion != "1.1.0" {
|
||||
t.Fatalf("%s PromptVersion = %q, want 1.1.0", definition.ID, definition.PromptVersion)
|
||||
}
|
||||
if definition.PromptID == "" {
|
||||
t.Fatalf("%s PromptID is empty", definition.ID)
|
||||
}
|
||||
if definition.TemplateID == "" || definition.GeneratedTextSchemaID == "" {
|
||||
t.Fatalf("%s template/schema = %q/%q, want both set", definition.ID, definition.TemplateID, definition.GeneratedTextSchemaID)
|
||||
|
||||
@@ -10,7 +10,7 @@ func todayDefinition() Definition {
|
||||
ID: Today,
|
||||
Name: "Today Report",
|
||||
PromptID: "weather.today_generated_text",
|
||||
PromptVersion: "1.0.1",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "today",
|
||||
GeneratedTextSchemaID: "today",
|
||||
ComparisonStrategy: CompareSameValidDate,
|
||||
|
||||
@@ -10,7 +10,7 @@ func tomorrowDefinition() Definition {
|
||||
ID: Tomorrow,
|
||||
Name: "Tomorrow Report",
|
||||
PromptID: "weather.tomorrow_generated_text",
|
||||
PromptVersion: "1.0.1",
|
||||
PromptVersion: "1.1.0",
|
||||
TemplateID: "tomorrow",
|
||||
GeneratedTextSchemaID: "tomorrow",
|
||||
ComparisonStrategy: CompareSameValidDate,
|
||||
|
||||
@@ -172,9 +172,9 @@ func validPreparationArtifact() PromptPreparationArtifact {
|
||||
return PromptPreparationArtifact{
|
||||
SchemaVersion: PromptPreparationSchemaVersion, Status: PromptPreparationSucceeded,
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text",
|
||||
PromptVersion: "1.0.1", DataPackagePath: "/workspace/data.yaml",
|
||||
PromptVersion: "1.1.0", DataPackagePath: "/workspace/data.yaml",
|
||||
Preparation: &promptexec.Preparation{
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
|
||||
PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
DataPackagePath: "/workspace/data.yaml",
|
||||
},
|
||||
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
|
||||
@@ -186,9 +186,9 @@ func validExecutionArtifact() PromptExecutionArtifact {
|
||||
validation := promptexec.NewValidation(promptexec.ValidationPassed, "json_schema", "daily.generated_text.schema.json", nil)
|
||||
return PromptExecutionArtifact{
|
||||
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionSucceeded,
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
Provenance: &PromptExecutionProvenance{
|
||||
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
|
||||
RunID: "provider-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
PromptHash: "prompt-hash", RenderedPromptHash: "rendered-hash", ProfileID: "profile",
|
||||
BackendID: "backend", ModelName: "model", DataPackagePath: "/workspace/data.yaml",
|
||||
StartedAt: started, EndedAt: started.Add(time.Second), Duration: time.Second,
|
||||
@@ -201,7 +201,7 @@ func validFailedExecutionArtifact() PromptExecutionArtifact {
|
||||
started := time.Date(2026, 5, 29, 15, 0, 0, 0, time.UTC)
|
||||
return PromptExecutionArtifact{
|
||||
SchemaVersion: PromptExecutionSchemaVersion, Status: PromptExecutionFailed,
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.0.1",
|
||||
ReportID: report.Daily, RunID: "weatherreporter-run", PromptID: "weather.daily_generated_text", PromptVersion: "1.1.0",
|
||||
StartedAt: started, EndedAt: started, Error: &PromptArtifactError{Category: promptexec.Generation, Message: "provider unavailable"},
|
||||
}
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user