Prepare reports for Promptkit migration
This commit is contained in:
15
docs/cli.md
15
docs/cli.md
@@ -21,9 +21,6 @@ weatherreporter generate daily --date YYYY-MM-DD [--config PATH] [--units VALUE]
|
||||
weatherreporter generate today [--config PATH] [--units VALUE] [--tz NAME] [--out PATH] [--date YYYY-MM-DD] [--quiet]
|
||||
weatherreporter generate tomorrow [--config PATH] [--units VALUE] [--tz NAME] [--out PATH] [--quiet]
|
||||
weatherreporter generate hourly [--config PATH] [--units VALUE] [--tz NAME] [--out PATH] [--quiet]
|
||||
weatherreporter generate three-day [--config PATH] [--units VALUE] [--tz NAME] [--out PATH] [--quiet]
|
||||
weatherreporter generate weekend [--config PATH] [--units VALUE] [--tz NAME] [--out PATH] [--quiet]
|
||||
weatherreporter generate storm [--config PATH] [--units VALUE] [--tz NAME] [--out PATH] [--quiet] --start TIME --end TIME
|
||||
weatherreporter run morning [--config PATH] [--units VALUE] [--tz NAME] [--out-dir PATH] [--quiet]
|
||||
weatherreporter run evening [--config PATH] [--units VALUE] [--tz NAME] [--out-dir PATH] [--quiet]
|
||||
weatherreporter inspect reports [--config PATH] [--limit N]
|
||||
@@ -38,12 +35,11 @@ weatherreporter inspect sources [--config PATH] RUN_ID
|
||||
| --- | --- |
|
||||
| `generate daily` | Requires `--date YYYY-MM-DD`; the date is interpreted in the effective report timezone. |
|
||||
| `generate today` | Accepts an optional `--date YYYY-MM-DD`; without it, the current local date in the effective report timezone is used. |
|
||||
| `generate tomorrow`, `three-day`, `weekend` | Use their report-defined valid period and accept the common generate flags. |
|
||||
| `generate hourly` | Covers the next six hours in the effective report timezone. It does not accept `--date`, `--start`, `--end`, `--hours`, or `--duration`. |
|
||||
| `generate storm` | Requires both `--start TIME` and `--end TIME`. Each time may be `YYYY-MM-DDTHH:MM` in the effective timezone or an RFC3339 timestamp with an explicit offset. |
|
||||
| `generate tomorrow` | Uses the next local civil day and accepts the common generate flags. |
|
||||
| `generate hourly` | Covers the next six hours in the effective report timezone. It does not accept `--date`, `--hours`, or `--duration`. |
|
||||
| `run morning` and `run evening` | Run their defined report batches. `--out-dir` writes extra Markdown copies; `--out` is not accepted. |
|
||||
|
||||
`generate` accepts all seven report command names shown above. `run` accepts
|
||||
`generate` accepts the four report command names shown above. `run` accepts
|
||||
only `morning` and `evening`. Batch membership, workspace artifacts, and
|
||||
notification sequencing are described in the [operations guide](operations.md).
|
||||
|
||||
@@ -84,8 +80,7 @@ time, valid period, and status:
|
||||
```
|
||||
|
||||
When available, the summary also includes `reportPath`, `metadataPath`,
|
||||
`dataPackagePath`, and `preflightPath`. Generated-text reports additionally
|
||||
include `generatedTextRawPath`, `generatedTextResultPath`,
|
||||
`dataPackagePath`, `preflightPath`, `generatedTextRawPath`, `generatedTextResultPath`,
|
||||
`generatedTextPath`, and `renderContextPath`. `outputPath` is included only
|
||||
when `--out` wrote an extra copy. Distributor notification, when attempted,
|
||||
adds `notificationPath` and may add a compact `notification` object.
|
||||
@@ -116,7 +111,6 @@ batch=morning total=2 succeeded=2 failed=0
|
||||
| `--out-dir PATH` | `run morning`, `run evening` | Write extra Markdown report copies in `PATH`. |
|
||||
| `--quiet` | `generate`, `run` | Suppress action summaries and routine batch status output. |
|
||||
| `--date YYYY-MM-DD` | `generate daily`, `generate today` | Required for Daily; optional for Today. |
|
||||
| `--start TIME`, `--end TIME` | `generate storm` | Required storm-event bounds. |
|
||||
| `--limit N` | `inspect reports` | Maximum runs to list. Defaults to `20`; `0` means no limit. |
|
||||
|
||||
Distributor notification is configured through `notify.distributor`; there are
|
||||
@@ -128,7 +122,6 @@ no Distributor-specific CLI flags. See the [configuration reference](config.md).
|
||||
weatherreporter generate daily --date 2026-05-29 --out ./daily.md
|
||||
weatherreporter generate today --date 2026-05-29 --out ./today.md
|
||||
weatherreporter generate hourly --out ./hourly.md
|
||||
weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00 --out ./storm.md
|
||||
weatherreporter run morning --out-dir ./reports
|
||||
```
|
||||
|
||||
|
||||
@@ -102,10 +102,9 @@ Use `secrets.directory` when a file-backed secret is appropriate.
|
||||
Single-report bundle templates accept `location_id`, `report_id`, `run_id`,
|
||||
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
|
||||
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, `valid_end_stamp`,
|
||||
and `storm_id`. Pipeline and idempotency-key templates may also use
|
||||
`bundle_id`. Dates use `YYYY-MM-DD`; times use `HHMM`; and stamps use
|
||||
`YYYY-MM-DDTHHMM` in the effective report timezone. `storm_id` is
|
||||
`{valid_start_stamp}-{valid_end_stamp}` for Storm Report and empty otherwise.
|
||||
Pipeline and idempotency-key templates may also use `bundle_id`. Dates use
|
||||
`YYYY-MM-DD`; times use `HHMM`; and stamps use `YYYY-MM-DDTHHMM` in the
|
||||
effective report timezone.
|
||||
|
||||
Batch bundle and pipeline templates accept `location_id`, `batch`,
|
||||
`batch_run_id`, and `batch_started_date`; batch idempotency-key templates may
|
||||
@@ -124,9 +123,6 @@ The default paths are:
|
||||
| `daily` | `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md` |
|
||||
| `today` | `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`, `today/index.md` |
|
||||
| `tomorrow` | `daily/{valid_start_date}/{run_id}.md`, `daily/{valid_start_date}/index.md`, `tomorrow/index.md` |
|
||||
| `three_day` | `three-day/{valid_start_date}/{run_id}.md`, `three-day/{valid_start_date}/index.md` |
|
||||
| `weekend` | `weekend/{valid_start_date}/{run_id}.md`, `weekend/{valid_start_date}/index.md` |
|
||||
| `storm` | `storm/{storm_id}/{run_id}.md`, `storm/{storm_id}/index.md` |
|
||||
|
||||
See the [operations guide](operations.md) for notification timing, uploaded
|
||||
artifact selection, and failure handling.
|
||||
@@ -189,9 +185,8 @@ prior comparable module snapshot.
|
||||
`reports` optionally overrides a report's ordered deterministic modules and
|
||||
Distributor path templates. Omit a report entry to retain its defaults.
|
||||
|
||||
Supported report keys are `daily`, `today`, `tomorrow`, `hourly`, `three_day`,
|
||||
`weekend`, and `storm`. Configuration also accepts `three_day_outlook`,
|
||||
`weekend_outlook`, and `storm_report`; hyphens and underscores are equivalent.
|
||||
Supported report keys are `daily`, `today`, `tomorrow`, and `hourly`; hyphens
|
||||
and underscores are equivalent.
|
||||
|
||||
Each report entry can contain:
|
||||
|
||||
|
||||
@@ -30,11 +30,10 @@ scriptorium run [--config <path>] [--profile <profile>] \
|
||||
[<extra_arg> ...]
|
||||
```
|
||||
|
||||
`render` is the preflight command. `run` writes either a Markdown report or a
|
||||
raw generated-text artifact to the supplied `--out` path. The structured
|
||||
generated-text use of `run` has the same argv as Markdown generation; it does
|
||||
not add `--format`, `--schema`, `--schema-path`, or `--json-schema` flags.
|
||||
Prompt configuration selected by `<prompt_id>` controls that output.
|
||||
`render` is the preflight command. `run` writes a structured generated-text
|
||||
artifact to the supplied `--out` path. It does not add `--format`, `--schema`,
|
||||
`--schema-path`, or `--json-schema` flags. Prompt configuration selected by
|
||||
`<prompt_id>` controls that output.
|
||||
|
||||
## Inputs and Outputs
|
||||
|
||||
|
||||
@@ -12,8 +12,8 @@ inspection requests. Generation and batch requests may supply collector,
|
||||
renderer, store, and notifier implementations for tests; production defaults
|
||||
use the focused packages.
|
||||
|
||||
A report result contains the module snapshot, prompt package, available
|
||||
Scriptorium results, generated-text artifacts when used, report and metadata
|
||||
A report result contains the module snapshot, prompt package, Scriptorium
|
||||
results, generated-text artifacts, report and metadata
|
||||
paths, prior snapshot, Recent Changes, and notification information. A batch
|
||||
result contains aggregate counts, per-report outcomes, and an optional batch
|
||||
notification. Inspection returns persisted values only.
|
||||
@@ -39,17 +39,17 @@ ordered work:
|
||||
snapshot and calculate Recent Changes.
|
||||
3. Build and save the prompt data package, run Scriptorium render preflight,
|
||||
save any preflight result, and save initial metadata.
|
||||
4. Produce managed Markdown according to the report generation mode.
|
||||
4. Run structured Scriptorium generation, validate and save generated text,
|
||||
build and save a render context, and render the managed Markdown template.
|
||||
5. Optionally make an output copy, save final metadata, optionally notify
|
||||
Distributor from the managed report path, and save metadata again when a
|
||||
notification path is produced.
|
||||
|
||||
Direct-Markdown reports prepare the managed report and invoke the Scriptorium
|
||||
run boundary. Generated-text-template reports look up their catalog definition,
|
||||
run structured Scriptorium output to the raw artifact, preserve any structured
|
||||
run result, validate and save generated text, build and save a render context,
|
||||
then render the embedded Markdown template. Schema, template, and subprocess
|
||||
details remain in their [generated-text](generatedtext.md),
|
||||
Every report looks up its catalog definition, runs structured Scriptorium
|
||||
output to the raw artifact, preserves the structured run result, validates and
|
||||
saves generated text, builds and saves a render context, then renders the
|
||||
embedded Markdown template. Schema, template, and subprocess details remain in
|
||||
their [generated-text](generatedtext.md),
|
||||
[report-template](reporttemplate.md), and [Scriptorium adapter](scriptorium-adapter.md)
|
||||
owners.
|
||||
|
||||
@@ -104,7 +104,7 @@ Focused tests are in `internal/app/app_test.go` and
|
||||
- A report uses one explicit normalized collection throughout its generation.
|
||||
- Render preflight precedes report generation.
|
||||
- Recent Changes compare structured module snapshots.
|
||||
- Generated-text reports render from a validated typed context, never directly
|
||||
from a raw prompt package.
|
||||
- Reports render from a validated typed context, never directly from a raw
|
||||
prompt package.
|
||||
- Only managed Markdown reports are notification sources; output copies are
|
||||
never uploaded.
|
||||
|
||||
@@ -25,19 +25,14 @@ requires a change between its low, possible, likely, and high categories.
|
||||
| Comparator | Required snapshot data | Compared values |
|
||||
| --- | --- | --- |
|
||||
| `CompareDaily` | `derived_daily_summary`, `derived_daypart_summaries` | Low and high temperature, daily precipitation probability and timing, peak gust, alerts, and aggregate indicators |
|
||||
| `CompareThreeDay` | `derived_daypart_summaries` | Per-day temperatures, precipitation probability and timing, peak gust, indicators, and added or removed outlook days |
|
||||
| `CompareWeekend` | `derived_daypart_summaries` | The three-day values with weekend-prefixed change types |
|
||||
|
||||
For daily comparison, `alert_digest` and `precip_timing` are optional: alerts
|
||||
are compared when present, and timing is compared only when both snapshots
|
||||
contain it. The multi-day comparators build their day map from daypart
|
||||
summaries. A missing or added day becomes a dedicated change rather than a
|
||||
comparison against invented data.
|
||||
contain it.
|
||||
|
||||
The application selects a comparator only after state lookup establishes a
|
||||
compatible prior snapshot. Daily, Today, and Tomorrow use the daily comparator;
|
||||
Three-day and Weekend use their named comparators. Other report types, such as
|
||||
Storm, produce no Recent Changes list.
|
||||
compatible prior snapshot. Daily, Today, and Tomorrow use the daily comparator.
|
||||
Hourly reports do not produce a Recent Changes list.
|
||||
|
||||
## Missing data and failures
|
||||
|
||||
@@ -51,8 +46,8 @@ behavior. It does not decide report compatibility or retain snapshots.
|
||||
|
||||
## Verification and invariants
|
||||
|
||||
Focused tests cover the daily, three-day, and weekend strategies, threshold
|
||||
boundaries, indicator and alert changes, and missing required stanzas:
|
||||
Focused tests cover the daily strategy, threshold boundaries, indicator and
|
||||
alert changes, and missing required stanzas:
|
||||
|
||||
```sh
|
||||
go test ./internal/changes
|
||||
|
||||
@@ -33,10 +33,8 @@ Report identity controls the summary shape:
|
||||
| --- | --- |
|
||||
| Hourly | Rolling-period selections and precipitation timing; no daily or daypart summary |
|
||||
| Daily, Today, Tomorrow | One local civil-day summary and its dayparts |
|
||||
| Three-day, Weekend | One clipped daily summary for each overlapping local day |
|
||||
| Storm | One summary for the explicit report window |
|
||||
|
||||
`DaypartSummaries` is collected from the resulting daily or storm summaries.
|
||||
`DaypartSummaries` is collected from the resulting daily summaries.
|
||||
The detailed grouping, daypart-window, and alert rules are owned by
|
||||
[forecast derivation](forecast-derivation.md).
|
||||
|
||||
@@ -55,8 +53,7 @@ not access the CLI, filesystem, subprocesses, or network.
|
||||
## Verification and invariants
|
||||
|
||||
Focused tests cover collected-fact separation, report-period selection,
|
||||
hourly and storm behavior, daily and partial-day summaries, and convective
|
||||
outlook selection:
|
||||
hourly behavior, daily summaries, and convective outlook selection:
|
||||
|
||||
```sh
|
||||
go test ./internal/facts
|
||||
|
||||
@@ -8,12 +8,12 @@ maintainer-facing context fields belong to [report templates](../templates.md).
|
||||
|
||||
## Catalog and validation
|
||||
|
||||
Only the Daily, Today, Tomorrow, and Hourly report definitions use the
|
||||
generated-text-template mode. `LookupDefinition` rejects a direct-Markdown
|
||||
definition, unknown schema or template IDs, and unsupported schema/template
|
||||
pairs before the run begins. A handler validates raw JSON, returns a typed
|
||||
value and canonical normalized JSON, loads its schema, builds a render context,
|
||||
and renders through `internal/reporttemplate`.
|
||||
The Daily, Today, Tomorrow, and Hourly report definitions each use structured
|
||||
generated text. `LookupDefinition` rejects unknown schema or template IDs and
|
||||
unsupported schema/template pairs before the run begins. A handler validates raw JSON, returns a typed
|
||||
value and canonical normalized JSON, loads its canonical schema through
|
||||
`internal/promptassets`, builds a render context, and renders through
|
||||
`internal/reporttemplate`.
|
||||
|
||||
Daily, Today, and Tomorrow use a day-style value with required trimmed summary
|
||||
and one or more nonblank discussion paragraphs. Hourly requires trimmed summary
|
||||
@@ -47,5 +47,5 @@ go test ./internal/generatedtext
|
||||
```
|
||||
|
||||
Generated text supplies prose slots only; deterministic weather facts remain in
|
||||
module and fact values. Every generated-text definition must resolve to exactly
|
||||
one supported catalog pair.
|
||||
module and fact values. Every report definition must resolve to exactly one
|
||||
supported catalog pair.
|
||||
|
||||
@@ -39,8 +39,6 @@ The registry declares these ordered default compositions:
|
||||
| Today | metadata, current conditions, narrative forecast, daily summary, daypart summaries, precipitation timing, alert digest, SPC outlooks, AFD, SPC discussion, weather story, outdoor windows, hourly forecast, today planning |
|
||||
| Tomorrow | metadata, current conditions, narrative forecast, daily summary, daypart summaries, precipitation timing, alert digest, SPC outlooks, AFD, SPC discussion, weather story, outdoor windows, tomorrow planning, hourly forecast |
|
||||
| Hourly | metadata, current conditions, hourly forecast, precipitation timing, alert digest, SPC outlooks, AFD (key messages and short term), SPC discussion, weather story |
|
||||
| Three-day and Weekend | metadata, current conditions, daypart summaries, precipitation timing, alert digest, SPC outlooks, AFD, SPC discussion, weather story, outdoor windows |
|
||||
| Storm | metadata, current conditions, precipitation timing, alert digest, SPC outlooks, AFD, SPC discussion, weather story |
|
||||
|
||||
The only non-empty default option is the AFD section selection. It accepts a
|
||||
`sections` list; omitted or empty selects all available sections. Report
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
# Report Registry Internals
|
||||
|
||||
`internal/report` owns the registry of report identities and the data declared
|
||||
for each one: resolution, generation mode, prompt identity, comparison policy,
|
||||
for each one: resolution, prompt identity and version, comparison policy,
|
||||
artifact group, output-copy name, default module composition, and Distributor
|
||||
path declarations. The public command syntax is owned by the
|
||||
[CLI reference](../cli.md); configuration aliases and overrides are owned by
|
||||
@@ -10,34 +10,28 @@ the [configuration reference](../config.md).
|
||||
## Definitions and resolution
|
||||
|
||||
Each `Definition` declares a stable ID and display name, prompt ID, generation
|
||||
mode, optional template and generated-text schema IDs, valid-period resolver,
|
||||
version, template and generated-text schema IDs, valid-period resolver,
|
||||
comparison strategy, artifact group, batch-copy filename, Distributor path
|
||||
templates, generation eligibility, compatible prior IDs, default modules, and
|
||||
batch eligibility flags. `Resolved` combines that definition with the valid
|
||||
period and run metadata for one invocation.
|
||||
|
||||
| Report ID | Mode | Period policy | Comparison | Registry batch flag | Output copy |
|
||||
| Report ID | Prompt version | Period policy | Comparison | Registry batch flag | Output copy |
|
||||
| --- | --- | --- | --- | --- | --- |
|
||||
| `daily` | Generated text + template | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
|
||||
| `today` | Generated text + template | Selected or current local civil day | Same valid date | Morning | `today.md` |
|
||||
| `tomorrow` | Generated text + template | Next local civil day | Same valid date | Evening | `tomorrow.md` |
|
||||
| `hourly` | Generated text + template | Rolling six-hour interval | Rolling window | — | `hourly.md` |
|
||||
| `three_day` | Scriptorium Markdown | Generation time through the third following local midnight | Same valid date | Morning | `three-day.md` |
|
||||
| `weekend` | Scriptorium Markdown | Upcoming weekend window | Weekend window | Morning | `weekend.md` |
|
||||
| `storm` | Scriptorium Markdown | Caller-supplied event window | Explicit window | — | `storm.md` |
|
||||
| `daily` | `1.0.0` | Explicit local civil day | Same valid date | Dynamic Daily inclusion is app-owned | `daily.md` |
|
||||
| `today` | `1.0.0` | Selected or current local civil day | Same valid date | Morning | `today.md` |
|
||||
| `tomorrow` | `1.0.0` | Next local civil day | Same valid date | Evening | `tomorrow.md` |
|
||||
| `hourly` | `1.0.0` | Rolling six-hour interval | Rolling window | — | `hourly.md` |
|
||||
|
||||
The four generated-text reports pair their report ID with matching template and
|
||||
schema IDs. The three direct-Markdown reports leave both IDs empty. Exact
|
||||
template fields and schema assets belong to [report templates](../templates.md)
|
||||
Each report pairs its ID and prompt version with matching template and schema
|
||||
IDs. Exact template fields and schema assets belong to [report templates](../templates.md)
|
||||
and [generated-text internals](generatedtext.md).
|
||||
|
||||
All valid periods are half-open. Storm accepts local `YYYY-MM-DDTHH:MM` values
|
||||
in the effective report timezone or offset-bearing RFC3339 values; its end
|
||||
must follow its start. Resolving Weekend directly on Sunday is rejected.
|
||||
All valid periods are half-open.
|
||||
|
||||
## Registry collaborators
|
||||
|
||||
`DefaultRegistry` is the only source of the seven report definitions.
|
||||
`DefaultRegistry` is the only source of the four report definitions.
|
||||
`Lookup`, `Resolve`, and report-name helpers prevent callers from duplicating
|
||||
report identity rules. Registry overrides clone a definition and replace its
|
||||
module list only after the report ID is recognized.
|
||||
@@ -54,13 +48,12 @@ membership and produces the actual batch plan.
|
||||
|
||||
Each definition supplies an ordered `[]module.ConfigItem`; the complete
|
||||
report-to-module mapping is maintained in [module internals](module.md).
|
||||
`ArtifactGroup`, `BatchOutputName`, `Generated`, and comparison compatibility
|
||||
`ArtifactGroup`, `BatchOutputName`, and comparison compatibility
|
||||
are likewise consumed by state and orchestration rather than recomputed there.
|
||||
|
||||
Unknown report IDs or batch names, an invalid weekend resolution, and invalid
|
||||
storm windows return errors. The registry never collects weather data, builds
|
||||
modules, parses CLI flags, writes state, executes Scriptorium, or delivers a
|
||||
report.
|
||||
Unknown report IDs or batch names return errors. The registry never collects
|
||||
weather data, builds modules, parses CLI flags, writes state, executes
|
||||
Scriptorium, or delivers a report.
|
||||
|
||||
## Verification and invariants
|
||||
|
||||
|
||||
@@ -40,7 +40,7 @@ single-report artifact identifiers.
|
||||
## Metadata and durable writes
|
||||
|
||||
`Metadata` is the durable inventory for a run. It records its schema version,
|
||||
run identity, generated and valid timestamps, artifact group and mode, source
|
||||
run identity, generated and valid timestamps, artifact group, source
|
||||
content and provenance, and the module snapshot, data-package, preflight,
|
||||
report, generated-artifact, and notification locations when present.
|
||||
|
||||
@@ -64,7 +64,6 @@ only considers an earlier compatible report in the same artifact group and
|
||||
supports the comparison strategies defined by the report request:
|
||||
|
||||
- `same_valid_date` finds an earlier generated report for the same valid day.
|
||||
- `weekend_window` finds a prior comparable weekend window.
|
||||
|
||||
The newest eligible metadata record wins; the current run is excluded.
|
||||
Unreadable or malformed candidate metadata is ignored so a damaged historical
|
||||
|
||||
@@ -16,10 +16,9 @@ weatherreporter generate today --out ./today.md
|
||||
|
||||
A generation collects weather data, resolves the report period, builds and
|
||||
persists the module snapshot and prompt data package, runs Scriptorium
|
||||
preflight, then produces the managed Markdown report. Daily, Today, Tomorrow,
|
||||
and Hourly reports additionally persist generated-text artifacts, validate the
|
||||
structured generated text, and render Markdown from the validated text and
|
||||
deterministic values.
|
||||
preflight, then persists generated-text artifacts, validates the structured
|
||||
generated text, and renders the managed Markdown report from the validated text
|
||||
and deterministic values.
|
||||
|
||||
The managed report and its final metadata are saved before single-report
|
||||
Distributor notification is attempted. `--out` writes an extra operator copy;
|
||||
@@ -68,11 +67,11 @@ workspace/
|
||||
notifications/batches/<batch>/<YYYY-MM-DD>/distributor.<batch_run_id>.json
|
||||
```
|
||||
|
||||
The generated-text and render-context artifacts are written only by Daily,
|
||||
Today, Tomorrow, and Hourly reports. A report's metadata links the module
|
||||
snapshot, data package, preflight artifact, managed report, and any available
|
||||
generated-text or single-report notification artifact. Batch notification
|
||||
artifacts are separate batch-level records under `notifications/batches`.
|
||||
The generated-text and render-context artifacts are written for every report.
|
||||
A report's metadata links the module snapshot, data package, preflight artifact,
|
||||
managed report, generated-text artifacts, and any available single-report
|
||||
notification artifact. Batch notification artifacts are separate batch-level
|
||||
records under `notifications/batches`.
|
||||
|
||||
RunIDs begin with the UTC generation timestamp and report ID. A Daily RunID
|
||||
also contains its local valid date so multiple Daily reports in one batch have
|
||||
|
||||
@@ -130,12 +130,8 @@ lookup. Keep weather derivation and path construction out of templates.
|
||||
weather facts.
|
||||
- LLM prompts receive curated module-based packages rather than raw,
|
||||
unbounded source payloads.
|
||||
- For generated-text-template reports, generated text is limited to defined
|
||||
prose slots, validated before use, and rendered through typed or otherwise
|
||||
explicit contexts.
|
||||
- Direct-Markdown reports receive the same curated prompt-package boundary but
|
||||
produce managed Markdown directly through Scriptorium rather than the
|
||||
generated-text schema and repository-template workflow.
|
||||
- Generated text is limited to defined prose slots, validated before use, and
|
||||
rendered through typed or otherwise explicit contexts.
|
||||
- Repository-owned templates arrange validated prose and deterministic facts;
|
||||
they do not perform meteorological derivation.
|
||||
|
||||
|
||||
@@ -7,8 +7,7 @@ status; current behavior is documented outside `docs/roadmap/`.
|
||||
|
||||
Status: Proposed and unimplemented.
|
||||
|
||||
Manual Storm Report generation is implemented; see the [CLI reference](../cli.md).
|
||||
Automatic storm-event evaluation remains unimplemented.
|
||||
Storm reporting, whether manual or automatic, is unimplemented.
|
||||
|
||||
Possible direction:
|
||||
|
||||
@@ -16,7 +15,7 @@ Possible direction:
|
||||
story context, hourly thresholds, and material forecast changes.
|
||||
2. Evaluate candidates through Scriptorium or another narrow evaluator adapter.
|
||||
3. Persist storm lifecycle state.
|
||||
4. Generate or update Storm Reports only when a meaningful event is present.
|
||||
4. Generate or update a storm report only when a meaningful event is present.
|
||||
5. Suppress ordinary low-impact thunder or rain chances.
|
||||
|
||||
Possible lifecycle states:
|
||||
@@ -29,8 +28,8 @@ Possible lifecycle states:
|
||||
- `resolved`
|
||||
|
||||
Before implementation, the design must preserve scheduled report behavior,
|
||||
manual Storm Report generation, inspectable evaluator failures, and fixture
|
||||
coverage for deterministic candidate detection.
|
||||
inspectable evaluator failures, and fixture coverage for deterministic
|
||||
candidate detection.
|
||||
|
||||
## Future Report Types
|
||||
|
||||
@@ -59,7 +58,7 @@ Possible future modules:
|
||||
Changes
|
||||
- `weekend_planning` if weekend-specific planning guidance needs a dedicated
|
||||
deterministic stanza
|
||||
- `storm_window_summary` if manual or automatic Storm Reports need a dedicated
|
||||
- `storm_window_summary` if manual or automatic storm reports need a dedicated
|
||||
prompt-facing storm-window module
|
||||
- separate AFD section aliases, such as `afd_key_messages`,
|
||||
`afd_short_term_text`, and `afd_long_term_text`, if separate stanzas prove
|
||||
|
||||
944
docs/roadmap/implementation.md
Normal file
944
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,944 @@
|
||||
# Promptkit Migration Implementation Plan
|
||||
|
||||
Status: Decision-complete implementation plan; unimplemented.
|
||||
|
||||
## Purpose And Authority
|
||||
|
||||
This document defines the ordered implementation procedure for the
|
||||
[Promptkit migration roadmap](promptkit.md). The feature roadmap is
|
||||
authoritative for scope, user intent, policy choices, and the desired end
|
||||
state. This plan is authoritative for sequencing, concrete package ownership,
|
||||
compatibility work, tests, documentation updates, and completion gates.
|
||||
|
||||
Implement the stages in order. Do not reinterpret a roadmap decision merely
|
||||
because the current Scriptorium implementation makes another path shorter.
|
||||
When implementation reveals a conflict with the roadmap, stop and update the
|
||||
roadmap and this plan deliberately rather than introducing an implicit policy
|
||||
change.
|
||||
|
||||
This plan follows the repository's
|
||||
[architecture](../policy/architecture.md),
|
||||
[documentation](../policy/documentation.md), and
|
||||
[testing](../policy/testing.md) policies. All stages are parts of one
|
||||
migration change. Intermediate stages may temporarily retain code needed by a
|
||||
later cutover, but no intermediate state should be released or documented as
|
||||
the final current behavior.
|
||||
|
||||
## Cross-Stage Rules
|
||||
|
||||
- Pin `gitea.maximumdirect.net/eric/promptkit` at exactly `v0.4.0`. Do not
|
||||
commit a `go.work`, local `replace`, pseudo-version, or unpublished commit.
|
||||
- Keep Promptkit types inside `internal/adapters/promptkit`, that package's
|
||||
tests, and the external contract test that verifies `internal/promptassets`
|
||||
with the real Promptkit inspector. App, CLI, report, state, and domain
|
||||
packages use project-owned types.
|
||||
- Keep Promptkit's opaque prepared-execution handle inside its adapter. Do not
|
||||
expose it through an app interface, serialize it, or make it restartable.
|
||||
- Construct one Promptkit engine per `generate` or `run` CLI invocation.
|
||||
Every report in a batch shares that engine. Inspection commands that only
|
||||
read persisted state do not construct an engine.
|
||||
- Keep batches sequential. Do not add retries, output repair, direct Markdown
|
||||
generation, arbitrary backend registration, or live-provider tests.
|
||||
- Preserve the exact persisted `data_package` bytes as the only model input.
|
||||
Promptkit receives those bytes through an inline artifact with the managed
|
||||
YAML path as provenance; it never receives an unrestricted file reference.
|
||||
- Persist successful preparation before `RunPrepared`. If preparation-record
|
||||
persistence or enabled debug persistence fails, discard the handle and do
|
||||
not call the provider.
|
||||
- Keep normal artifacts, errors, logs, and CLI summaries free of credentials,
|
||||
rendered messages, schema bodies, data-package bodies, provider response
|
||||
bodies, and full effective parameter maps.
|
||||
- Use atomic writes for durable state and debug files. Preserve every
|
||||
non-secret artifact reached before a later failure when practical.
|
||||
- Use deterministic Promptkit model-client fakes at the adapter boundary and
|
||||
project-owned executor fakes at the app boundary. The default test suite
|
||||
remains offline and credential-free.
|
||||
- At each stage, update or delete existing tests according to the testing
|
||||
policy. Protect contracts and failure behavior; do not preserve tests that
|
||||
exist only to assert the retired subprocess implementation.
|
||||
- Run the focused checks named by the stage while iterating. Before completing
|
||||
every stage, run `git diff --check`. Run the full repository gate in the
|
||||
final stage.
|
||||
|
||||
## Fixed Package And Contract Decisions
|
||||
|
||||
Use these ownership boundaries so later stages do not need to choose an
|
||||
architecture while editing:
|
||||
|
||||
| Area | Owner | Decision |
|
||||
| --- | --- | --- |
|
||||
| Embedded prompt definitions, referenced prompt content, and provider-facing schemas | `internal/promptassets` | Embed one centralized asset tree and expose read-only prompt and schema `fs.FS` sources plus schema lookup by report schema ID. This package does not import Promptkit. |
|
||||
| Project-owned prompt execution contract | `internal/promptexec` | Own prompt inspection, profile inspection, preparation, execution, validation, usage, debug, and neutral error values. This package does not import Promptkit, app, CLI, report, or state. |
|
||||
| Promptkit integration | `internal/adapters/promptkit` | Construct and own the Promptkit engine, translate project requests and results, classify public Promptkit errors, and enforce prepared-handle lifecycle. |
|
||||
| Workflow ordering | `internal/app` | Resolve reports, inspect prompt/profile selections, collect weather, build and save inputs, record preparation, invoke execution, persist results, perform final generated-text validation, render Markdown, and notify. |
|
||||
| Durable artifacts | `internal/state` | Own v2 paths, preparation and execution schemas, metadata compatibility, atomic writes, and inspection reads. |
|
||||
| Engine composition and flags | `internal/cli` | Construct one executor per action invocation, pass it into app requests, parse `--llm-debug-dir`, and expose project-owned summary paths. |
|
||||
|
||||
`internal/promptexec.Executor` must provide three operations:
|
||||
|
||||
1. inspect an exact prompt ID and version;
|
||||
2. inspect one explicit profile ID; and
|
||||
3. prepare and execute one request.
|
||||
|
||||
The execution operation accepts a preparation callback. The adapter calls that
|
||||
callback exactly once after successful `PrepareExecution` and before
|
||||
`RunPrepared`. The callback receives a safe project-owned preparation value
|
||||
and, only when requested, a project-owned sensitive debug value. A callback
|
||||
error aborts execution. A preparation failure returns a classified
|
||||
project-owned attempt error without calling the callback; app orchestration
|
||||
uses that error to persist a failure receipt.
|
||||
|
||||
The adapter execution request contains the exact prompt ID and version, the
|
||||
optional configured profile override, the exact YAML bytes, the managed
|
||||
data-package path used only as inline provenance, and whether sensitive debug
|
||||
capture is enabled. It does not contain a provider output path.
|
||||
|
||||
Project-owned errors use stable categories for invalid configuration, invalid
|
||||
request, prompt not found/load, profile not found/load, missing credential,
|
||||
artifact load, prompt render, capacity, generation, operational validation,
|
||||
completed validation rejection, cancellation, and deadline. A capacity error
|
||||
also carries the non-secret backend ID. Its `Error` text is
|
||||
Weatherreporter-owned and safe for CLI output; an unexported or non-serialized
|
||||
cause may remain available to `errors.Is` and `errors.As`. Never persist
|
||||
arbitrary dependency error prose.
|
||||
|
||||
## Stage 1: Correct The Implemented Report Surface
|
||||
|
||||
### Goal
|
||||
|
||||
Remove the unfinished three-day, weekend, and storm products before changing
|
||||
the LLM integration. The repository should expose only Daily, Today, Tomorrow,
|
||||
and Hourly while continuing to use Scriptorium temporarily for those four
|
||||
reports.
|
||||
|
||||
### Work
|
||||
|
||||
1. Remove the three report definitions and their files under `internal/report`.
|
||||
Remove their IDs, command names, aliases, period inputs, generation modes,
|
||||
batch flags, registry order, and direct-Markdown-only declarations.
|
||||
2. Add `PromptVersion string` to `report.Definition` and set it to `1.0.0` on
|
||||
all four retained definitions. Since every retained report uses generated
|
||||
text plus a repository template, remove the now-redundant
|
||||
`GenerationMode` and `Generated` fields and their branches.
|
||||
3. Remove three-day, weekend, and storm command parsing, help text, app request
|
||||
fields/constants, resolution branches, Distributor path variables, and
|
||||
configuration report aliases.
|
||||
4. Remove report-specific code whose only caller was one of the retired
|
||||
products. Use graph traces before deletion. This includes the dedicated
|
||||
three-day/weekend Recent Changes comparators, storm-window fact derivation,
|
||||
multi-day report derivation used only by these definitions, storm time
|
||||
parsing, and their focused tests when no retained behavior depends on them.
|
||||
Do not remove general meteorological behavior merely because a fixture or
|
||||
prose string contains words such as “storm” or “weekend.”
|
||||
5. Keep morning and evening planning behavior exactly as implemented:
|
||||
morning plans Today and Tomorrow, evening plans Tomorrow, and both may add
|
||||
eligible future Daily reports from complete hourly coverage.
|
||||
6. Update the current CLI, configuration, report-registry, facts, modules,
|
||||
changes, operations, troubleshooting, and other affected documentation to
|
||||
describe only the four implemented reports. Update `docs/roadmap/future.md`
|
||||
so it no longer claims that a manual Storm Report exists; future versions
|
||||
of these products remain explicitly unimplemented roadmap work.
|
||||
|
||||
### Tests
|
||||
|
||||
- Rewrite report registry and CLI parser tests around exactly four commands.
|
||||
- Remove fake successful-generation tests for nonexistent prompts.
|
||||
- Preserve Daily/Today/Tomorrow/Hourly period, module, batch-planning,
|
||||
Distributor path, and output-name coverage.
|
||||
- Add negative CLI/config tests showing the retired command names and aliases
|
||||
are rejected.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/report ./internal/config ./internal/facts \
|
||||
./internal/changes ./internal/app ./internal/cli
|
||||
go test ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
No implemented registry, CLI, configuration, app, current-state document, or
|
||||
test claims that three-day, weekend, or storm generation exists. The four
|
||||
retained reports still pass through the existing generated-text/template
|
||||
workflow, and each carries exact prompt version `1.0.0`.
|
||||
|
||||
## Stage 2: Promote And Reconcile Runtime Prompt Assets
|
||||
|
||||
### Goal
|
||||
|
||||
Create the single embedded prompt/schema corpus used by Promptkit and by
|
||||
Weatherreporter's final generated-text boundary.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add Promptkit `v0.4.0` to `go.mod` and `go.sum`.
|
||||
2. Create `internal/promptassets` with an embedded tree rooted at:
|
||||
|
||||
```text
|
||||
assets/
|
||||
prompts/
|
||||
common/
|
||||
daily/
|
||||
today/
|
||||
tomorrow/
|
||||
hourly/
|
||||
schemas/
|
||||
```
|
||||
|
||||
Expose copied/read-only prompt and schema filesystem views and a
|
||||
`Schema(id)` lookup for the four report schema IDs. Keep path constants
|
||||
private except where a stable project contract is necessary.
|
||||
3. Promote only the four `*_generated_text` prompt definitions from
|
||||
`docs/roadmap/scriptorium`. Do not promote `weather.daily_report`.
|
||||
4. Use the Scriptorium corpus's common system, data-package guidance, and
|
||||
report-specific files as the behavioral base. Reconcile rather than
|
||||
editorially rewrite:
|
||||
|
||||
- preserve its source weighting, hazard, precipitation, and style rules;
|
||||
- add the optional `confidence` field instruction to Daily, Today, and
|
||||
Hourly so all prompt text agrees with the canonical domain types and
|
||||
schemas;
|
||||
- retain Tomorrow's existing confidence instruction;
|
||||
- prefer the corpus's correct Tomorrow daily framing over the stale
|
||||
hourly-style file currently under `internal/reporttemplate/prompts`;
|
||||
- make no unrelated prompt-tone or product-policy changes.
|
||||
|
||||
5. Every prompt definition must use:
|
||||
|
||||
- its existing prompt ID;
|
||||
- version `1.0.0`;
|
||||
- `default_profile: gemini-flash-latest`;
|
||||
- one required `data_package` input with
|
||||
`content_type: application/yaml`;
|
||||
- JSON output with `validation_mode: json_schema`;
|
||||
- a path inside the embedded schema root; and
|
||||
- zero or omitted `repair_attempts`.
|
||||
|
||||
6. Move the four canonical schemas from `internal/reporttemplate/schemas` into
|
||||
`internal/promptassets/assets/schemas`. Preserve their current application
|
||||
contracts, including optional `confidence`, required summary/discussion,
|
||||
array discussion for day-style reports, string discussion for Hourly,
|
||||
`additionalProperties: false`, and the corrected Daily `$id` and title.
|
||||
7. Change `internal/generatedtext` to load these canonical schemas from
|
||||
`promptassets`. Remove schema and prompt-fragment ownership from
|
||||
`internal/reporttemplate`; it should embed and render Markdown templates and
|
||||
partials only.
|
||||
8. Retain `docs/roadmap/scriptorium` as migration source evidence until the
|
||||
final cleanup stage.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add table-driven asset tests for exactly four prompt IDs and versions,
|
||||
referenced-file resolution, default profiles, YAML input declarations,
|
||||
output contracts, and schema lookup.
|
||||
- Construct a Promptkit engine over the embedded sources in an external test
|
||||
package and call `InspectPrompt` for every report. Do not use a provider or
|
||||
credentials.
|
||||
- Retain generated-text schema and typed-validation tests, now reading the
|
||||
relocated canonical schemas.
|
||||
- Assert that no active embedded prompt uses `local-heavy`,
|
||||
`pipeline-weather/`, JSON input metadata, positive repair attempts, or
|
||||
`weather.daily_report`.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptassets ./internal/generatedtext \
|
||||
./internal/reporttemplate
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
One embedded source contains exactly four valid Promptkit prompt definitions
|
||||
and four canonical schemas. Prompt inspection succeeds offline for every
|
||||
report, and no duplicate runtime provider-facing schema remains.
|
||||
|
||||
## Stage 3: Define The Neutral Prompt Execution Contract
|
||||
|
||||
### Goal
|
||||
|
||||
Define the complete project-owned boundary between Weatherreporter and an LLM
|
||||
execution adapter. This stage introduces no Promptkit imports and makes no
|
||||
production orchestration changes.
|
||||
|
||||
### Work
|
||||
|
||||
1. Create `internal/promptexec` with the neutral contract fixed above. Include:
|
||||
|
||||
- exact prompt and profile inspection values;
|
||||
- safe preparation provenance;
|
||||
- execution result, validation, token usage, and timing;
|
||||
- optional sensitive preparation/execution debug values;
|
||||
- stable error categories and a typed capacity error; and
|
||||
- small helpers for safe diagnostic bounding and copying.
|
||||
|
||||
2. The safe preparation value includes prompt ID/version/hash, rendered prompt
|
||||
hash, input hashes, selected profile/backend, effective model name, output
|
||||
format/validation/schema path, preparation start/end/duration, and the
|
||||
managed data-package path. It excludes endpoints, API-key environment
|
||||
names, full parameters, rendered messages, schema bodies, and input bodies.
|
||||
3. The safe execution value includes Promptkit run ID, repeated prompt and
|
||||
input provenance, selected profile/backend/model, generated-content hash,
|
||||
usage, execution start/end/duration, completed validation status, bounded
|
||||
validation diagnostics, and raw output bytes. It excludes endpoint and full
|
||||
effective parameters.
|
||||
4. Bound routine validation diagnostics to at most 10 entries and at most
|
||||
1,024 bytes per entry, truncating on a valid UTF-8 boundary. Bound a
|
||||
persisted safe error message to 2,048 bytes. Prefer stable categories and
|
||||
fields over diagnostic prose.
|
||||
5. Specify the execution lifecycle in interface comments and tests:
|
||||
|
||||
- inspection is side-effect-free;
|
||||
- preparation invokes the callback exactly once only after successful
|
||||
preparation;
|
||||
- callback failure prevents provider execution;
|
||||
- completed validation rejection is a result, not an operational error;
|
||||
- operational failure does not fabricate a completed result; and
|
||||
- sensitive debug values are populated only when explicitly requested.
|
||||
6. Keep copying and truncation helpers internal to `promptexec`. Callers must
|
||||
not be able to mutate byte slices, maps, or diagnostics retained inside
|
||||
contract values.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add compile-time test fakes that implement the complete interface without
|
||||
importing Promptkit.
|
||||
- Add table-driven tests for every error category, capacity fields, diagnostic
|
||||
count and byte limits, UTF-8-safe truncation, error-message bounds, and
|
||||
defensive copying.
|
||||
- Assert through representative fixtures that safe contract values contain no
|
||||
endpoint, credential, rendered message, schema body, input body, response
|
||||
body, or full parameter map.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptexec
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
`internal/promptexec` provides a stable, documented, offline-tested contract
|
||||
that can represent every inspection, preparation, execution, validation,
|
||||
usage, debug, and failure outcome required by the roadmap. It has no Promptkit,
|
||||
app, CLI, report, or state dependency.
|
||||
|
||||
## Stage 4: Implement The Promptkit Adapter
|
||||
|
||||
### Goal
|
||||
|
||||
Implement and thoroughly test Promptkit behind the Stage 3 contract without
|
||||
modifying application or CLI orchestration.
|
||||
|
||||
### Work
|
||||
|
||||
1. Create `internal/adapters/promptkit`. Its constructor:
|
||||
|
||||
- uses `promptassets` through `WithPromptFS` and `WithSchemaFS`;
|
||||
- selects one external profile directory through Promptkit config or one
|
||||
profile file through `WithProfileFile`;
|
||||
- optionally registers
|
||||
`promptkit.LocalBackend(endpoint, concurrencyLimit)`;
|
||||
- sets the transport timeout;
|
||||
- accepts an injected Promptkit `LLMClient` only through an adapter-local
|
||||
test constructor or option; and
|
||||
- returns project-owned configuration errors.
|
||||
|
||||
2. Implement prompt and profile inspection with exact versions and explicit
|
||||
field-by-field translation into `promptexec` values. Do not return or embed
|
||||
Promptkit values.
|
||||
3. Implement execution:
|
||||
|
||||
- call `PrepareExecution` with the exact requested version and optional
|
||||
profile override;
|
||||
- pass the exact YAML as
|
||||
`InlineWithURI(dataPackagePath, string(bytes))`;
|
||||
- immediately `defer handle.Discard()` after a successful prepare;
|
||||
- map safe preparation details and invoke the preparation callback;
|
||||
- call `RunPrepared` only after the callback succeeds;
|
||||
- treat `ValidationFailed` as a completed result rather than an operational
|
||||
error; and
|
||||
- return no invented execution result for operational errors.
|
||||
|
||||
4. Translate Promptkit's public error sentinels with `errors.Is`, and
|
||||
`CapacityError` with `errors.As`. Preserve caller cancellation and deadline
|
||||
identities. Do not parse error strings.
|
||||
5. Capture sensitive debug values only when requested. Use project-owned debug
|
||||
structs and explicit field mapping rather than serializing whole Promptkit
|
||||
values, so a future upstream field cannot silently enter an artifact.
|
||||
6. Keep the prepared handle entirely within the adapter call. Discard it after
|
||||
every success or failure path and never make it serializable or reusable.
|
||||
|
||||
### Tests
|
||||
|
||||
- Use an injected deterministic Promptkit model client. Never use a live
|
||||
endpoint or credential.
|
||||
- Cover prompt/profile inspection, profile override precedence, built-in and
|
||||
external profiles, endpoint-only profiles, the conventional local backend,
|
||||
local concurrency/capacity rejection, missing credentials, and constructor
|
||||
validation.
|
||||
- Cover preparation callback ordering, callback failure preventing generation,
|
||||
exact inline bytes and URI, exact prompt version, successful execution,
|
||||
completed schema rejection with raw output, operational generation and
|
||||
validation failures with no partial result, cancellation, timeout, and
|
||||
handle discard.
|
||||
- Assert that safe values and errors do not contain rendered messages,
|
||||
endpoints, schema bodies, input bodies, response bodies, credentials, or
|
||||
full parameter maps.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/promptexec ./internal/adapters/promptkit
|
||||
go test -race ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
The Promptkit adapter is a complete offline-tested implementation of the
|
||||
neutral contract. No Promptkit type is referenced outside the adapter, its
|
||||
tests, or the external asset contract test, and no production path uses the
|
||||
adapter yet.
|
||||
|
||||
## Stage 5: Introduce Durable State V2
|
||||
|
||||
### Goal
|
||||
|
||||
Add the final normal-artifact model and backward-compatible metadata reading
|
||||
before app orchestration starts writing the new records. Sensitive debug
|
||||
storage remains out of scope until Stage 6.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add the new-run path members `Preparation` and `Execution`:
|
||||
|
||||
- preparation:
|
||||
`preflight/<group>/<date>/prompt_preparation.<runID>.json`;
|
||||
- execution:
|
||||
`snapshots/<group>/<date>/prompt_execution.<runID>.json`.
|
||||
|
||||
Keep `workspace.preflight_dir` and its default physical directory unchanged.
|
||||
Retain `Preflight` and `GeneratedTextResult` only as temporary legacy
|
||||
write-path members required by the uncut Scriptorium workflows; mark them
|
||||
for deletion in Stage 10. New Promptkit code must use only the new members.
|
||||
2. Add `PromptPreparationArtifact` with schema
|
||||
`weatherreporter.prompt_preparation.v1`. It represents either successful
|
||||
preparation or a failed preparation receipt and includes schema version,
|
||||
status, report/RunID, prompt identity, safe preparation provenance when
|
||||
available, timing, data-package path, and a bounded classified error when
|
||||
failed.
|
||||
3. Add `PromptExecutionArtifact` with schema
|
||||
`weatherreporter.prompt_execution.v1`. It represents success, completed
|
||||
validation rejection, or operational failure and includes safe execution
|
||||
provenance, validation, usage, timing, reached artifact paths, and a bounded
|
||||
classified error when failed. It never embeds generated content.
|
||||
4. Advance newly written metadata to `weatherreporter.metadata.v2`. Replace
|
||||
`preflightPath` and `generatedTextResultPath` with `preparationPath` and
|
||||
`executionPath`. Populate paths only after the corresponding artifact has
|
||||
actually been saved.
|
||||
5. Implement explicit v1/v2 metadata decoding:
|
||||
|
||||
- accept only the known v1 and v2 schema versions;
|
||||
- normalize v1 `preflightPath` and `generatedTextResultPath` internally for
|
||||
inspection;
|
||||
- preserve v1 field names when a loaded v1 record is marshaled by
|
||||
`inspect metadata`;
|
||||
- write only v2 through the new Promptkit state APIs; the temporary
|
||||
Scriptorium path may continue writing v1 until its Stage 10 removal; and
|
||||
- never dual-write legacy aliases.
|
||||
|
||||
Direct inspection of v1 metadata, modules, data packages, sources, and
|
||||
referenced artifacts remains available for any historical report ID.
|
||||
Prior-snapshot reconstruction is required only for the four retained report
|
||||
IDs; do not restore retired definitions solely for legacy comparison.
|
||||
6. Add typed `SavePromptPreparation`, `SavePromptExecution`, and corresponding
|
||||
typed load methods where inspection needs them. Temporarily retain the
|
||||
Scriptorium write methods so the current production paths compile through
|
||||
Stage 9; delete those methods during Stage 10. Keep long-term legacy support
|
||||
read-only.
|
||||
7. Update app/CLI result structs and JSON field names to
|
||||
`preparationPath`, `executionPath`, and optional `llmDebugPath`; temporarily
|
||||
adapt old orchestration so the tree compiles until the cutover stages.
|
||||
Do not write v2 records through the Scriptorium path.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add exact path, schema-defaulting, round-trip, required-field, atomic-write,
|
||||
and unknown-version tests.
|
||||
- Add v1 fixtures covering both generated-text and legacy preflight references;
|
||||
verify list, metadata, modules, data-package, and source inspection.
|
||||
- Verify that re-marshaled v1 inspection uses v1 field names and that v2 output
|
||||
contains no deprecated aliases.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/state ./internal/app ./internal/cli
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
State can read historical v1 runs and write the complete normal v2 artifact
|
||||
contract through the new APIs. The existing production generation path still
|
||||
compiles and remains unchanged until cutover; no new Promptkit artifact uses a
|
||||
Scriptorium-specific filename, schema, or JSON field.
|
||||
|
||||
## Stage 6: Add Secure LLM Debug Persistence
|
||||
|
||||
### Goal
|
||||
|
||||
Implement the explicitly enabled sensitive-debug store as a small, independently
|
||||
auditable boundary before any CLI command can request debug capture.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add a focused debug writer, owned by state or a narrow state-adjacent
|
||||
package, for an explicit operator root outside normal artifact derivation.
|
||||
It must:
|
||||
|
||||
- validate or create the root before collection or provider work;
|
||||
- use `<root>/<report-id>/<valid-date>/<run-id>/`;
|
||||
- write `preparation.json` and `execution.json` atomically;
|
||||
- create directories with `0700` and files with `0600`;
|
||||
- reject symlinks, path escape, non-directory roots, and unsafe
|
||||
run/report/date segments; and
|
||||
- return the per-run debug directory as the project-owned summary path.
|
||||
|
||||
2. Define explicit project-owned debug wire structs. Preparation debug may
|
||||
contain rendered messages, structured-output schema, and effective
|
||||
endpoint/parameters. Execution debug may additionally contain raw generated
|
||||
output and completed validation details.
|
||||
3. Exclude direct credentials, resolved environment credential values, and
|
||||
arbitrary dependency objects. Use field-by-field mapping so future
|
||||
`promptexec` or Promptkit fields are not serialized implicitly.
|
||||
4. Make disabled debug behavior a no-op that performs no filesystem access.
|
||||
A requested debug initialization or write error is returned to the caller;
|
||||
it is never silently downgraded.
|
||||
5. Keep this writer independent of normal state path derivation. Normal state
|
||||
inspection must not discover or serve sensitive debug artifacts.
|
||||
|
||||
### Tests
|
||||
|
||||
- Verify exact grouping, atomic replacement, disabled behavior, and returned
|
||||
per-run paths.
|
||||
- Verify directory `0700` and file `0600` permissions on supported platforms.
|
||||
- Cover traversal, absolute-segment, symlink-root, symlink-component,
|
||||
non-directory, and invalid report/date/run segment rejection.
|
||||
- Marshal representative debug fixtures and verify credentials and resolved
|
||||
secret values are absent while the explicitly allowed diagnostic fields are
|
||||
retained.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/state
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Sensitive debug persistence is secure, explicitly rooted, atomic, isolated
|
||||
from normal state, and fully tested. No production CLI path enables it yet.
|
||||
|
||||
## Stage 7: Add Promptkit Configuration, Composition, And Inspection
|
||||
|
||||
### Goal
|
||||
|
||||
Add the configuration and project-owned composition seams needed for cutover,
|
||||
and centralize pre-collection prompt/profile validation. Retain Scriptorium as
|
||||
the production generator until Stage 8 so this stage does not create a
|
||||
half-cut-over runtime.
|
||||
|
||||
### Work
|
||||
|
||||
1. Add `config.PromptkitConfig` and nested local config with exactly:
|
||||
|
||||
- `profile`;
|
||||
- `profile_file`;
|
||||
- `profile_dir`;
|
||||
- `timeout`, default `2m`;
|
||||
- `local.endpoint`; and
|
||||
- `local.concurrency_limit`, default `1`.
|
||||
|
||||
Scriptorium config remains temporarily because production generation has
|
||||
not yet cut over. Promptkit validation rejects simultaneous profile
|
||||
sources, non-positive transport timeout, an invalid nonblank local
|
||||
endpoint, and negative concurrency. A blank local endpoint leaves `local`
|
||||
unregistered; concurrency zero means unlimited.
|
||||
2. Add an executor factory seam to `cli.Runner` using only project-owned types.
|
||||
Production construction delegates to `internal/adapters/promptkit`; CLI
|
||||
tests inject a fake factory. The factory creates one executor for an action,
|
||||
not one per report.
|
||||
3. Add a project-owned app inspection helper that:
|
||||
|
||||
- inspects the exact prompt ID and `report.Definition.PromptVersion`;
|
||||
- verifies exactly one required `data_package` input with
|
||||
`application/yaml`;
|
||||
- verifies the expected JSON Schema output contract and declared default
|
||||
profile;
|
||||
- selects `promptkit.profile` when nonblank, otherwise the prompt default;
|
||||
- inspects that explicit profile;
|
||||
- rejects a profile requiring a direct API key, because Weatherreporter has
|
||||
no direct-key configuration; and
|
||||
- requires a nonblank environment value for any reported `APIKeyEnv`.
|
||||
|
||||
4. Return only safe, project-owned inspection values and classified errors.
|
||||
The helper performs no collection, provider call, or durable write.
|
||||
5. Unit-test production factory argument mapping, but do not yet wire it into
|
||||
`generate` or `run`. Stage 8 performs that atomic production cutover.
|
||||
|
||||
### Tests
|
||||
|
||||
- Cover exact defaults, mutual profile-source exclusion, local endpoint
|
||||
validation, zero/unlimited and negative concurrency, and timeout validation.
|
||||
- Cover prompt/profile default and override precedence, prompt contract
|
||||
mismatch, unsupported direct-key profiles, missing environment credentials,
|
||||
and safe errors.
|
||||
- Verify the CLI factory maps embedded assets, external profile source, local
|
||||
backend, concurrency, and timeout exactly once without exposing Promptkit
|
||||
types.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/config ./internal/app ./internal/cli \
|
||||
./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Configuration, engine construction, and pre-collection inspection are
|
||||
decision-complete and offline-tested behind project-owned seams. Production
|
||||
generation still follows the old path, so there is no dual execution mode.
|
||||
|
||||
## Stage 8: Cut Over Single-Report Execution And Failure Persistence
|
||||
|
||||
### Goal
|
||||
|
||||
Move all four `generate` commands to Promptkit prepared execution while
|
||||
preserving report output behavior and inspectable partial failure. Debug CLI
|
||||
enablement and final summary presentation are deferred to Stage 9.
|
||||
|
||||
### Work
|
||||
|
||||
1. Construct exactly one executor after configuration is loaded for a
|
||||
`generate` action and pass it through `app.GenerateRequest`.
|
||||
2. Isolate the current shared generation function before changing it:
|
||||
|
||||
- move the Scriptorium implementation behind a temporary, unexported legacy
|
||||
batch helper;
|
||||
- make `GenerateDetailed` call the new Promptkit implementation described
|
||||
below;
|
||||
- leave `RunBatchDetailed` calling only the legacy helper until Stage 10;
|
||||
and
|
||||
- do not add a runtime switch, fallback, or dual invocation for the same
|
||||
report.
|
||||
|
||||
This temporary split is solely a staging seam. Stage 10 moves batches to
|
||||
the Promptkit implementation and deletes the helper.
|
||||
3. Reorder `GenerateDetailed`:
|
||||
|
||||
1. resolve the report and RunID;
|
||||
2. require the injected executor;
|
||||
3. run the Stage 7 exact prompt/profile/credential inspection; and only then
|
||||
4. collect weather.
|
||||
|
||||
Inspection failures occur before managed run artifacts exist.
|
||||
4. Implement the new project-owned report-generation core with the injected
|
||||
executor and the Stage 5 artifact contract:
|
||||
|
||||
- build, serialize once, and save the data package;
|
||||
- use those exact serialized bytes for execution;
|
||||
- persist a failed preparation receipt and v2 metadata when preparation
|
||||
fails;
|
||||
- use the preparation callback to save successful preparation provenance
|
||||
and metadata before provider work;
|
||||
- after execution success or completed validation rejection, save exact raw
|
||||
output, then the execution artifact, then metadata;
|
||||
- after operational execution failure, save a failure execution receipt and
|
||||
metadata without inventing raw output;
|
||||
- on Promptkit validation rejection, return a classified report failure
|
||||
after preserving raw output and bounded diagnostics;
|
||||
- on Promptkit validation success, run the existing typed
|
||||
`internal/generatedtext` validation, normalize/save generated text, build
|
||||
and save render context, and render managed Markdown as before.
|
||||
|
||||
5. Make partial failures return a non-nil `ReportResult` whenever a RunID and
|
||||
inspectable paths exist. Preserve preparation/execution/raw/normalized/
|
||||
context/report paths reached before preparation, capacity, generation,
|
||||
Promptkit validation, domain validation, template, copy, or notification
|
||||
failure.
|
||||
6. Remove Scriptorium result values from the single-report app requests,
|
||||
results, and test fakes used by the generate path. App tests depend only on
|
||||
`promptexec.Executor`.
|
||||
Retain the old adapter and configuration only for the still-uncut batch
|
||||
path.
|
||||
7. Do not expose `--llm-debug-dir` yet. Pass `CaptureDebug: false` during this
|
||||
stage; Stage 9 adds debug orchestration without changing provider execution
|
||||
semantics.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add representative offline app workflows for each of the four
|
||||
reports, using real state, generated-text validation, contexts, and
|
||||
templates with a fake executor.
|
||||
- Verify inspection and credential checks occur before collection.
|
||||
- Verify exact profile override/default precedence and exact prompt versions.
|
||||
- Verify persistence ordering by observing that execution is not called until
|
||||
the successful preparation artifact and metadata exist.
|
||||
- Cover preparation failure, callback/state failure, capacity rejection,
|
||||
credential disappearance at execution, cancellation, deadline, generation
|
||||
failure, operational validation failure, completed schema rejection,
|
||||
generated-text domain failure, template failure, output-copy failure, and
|
||||
notification failure.
|
||||
- Verify partial results retain every reached normal-artifact path without
|
||||
prompt or response content.
|
||||
- Verify single-report commands invoke only Promptkit and batch commands invoke
|
||||
only the temporary legacy helper during this intermediate stage.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/cli
|
||||
go test -race ./internal/app ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Every single-report command uses one Promptkit engine, performs inspection
|
||||
before collection, persists preparation before provider work, and produces the
|
||||
same managed Markdown/template and notification outcomes through project-owned
|
||||
contracts. Sensitive debug capture remains disabled.
|
||||
|
||||
## Stage 9: Add Single-Report Debug And CLI Summary Behavior
|
||||
|
||||
### Goal
|
||||
|
||||
Expose the opt-in debug workflow and finalize single-report CLI success and
|
||||
partial-failure summaries without expanding the normal artifact boundary.
|
||||
|
||||
### Work
|
||||
|
||||
1. Parse `--llm-debug-dir PATH` for all four generate commands. Do not add a
|
||||
YAML debug switch.
|
||||
2. Validate or create the debug root before prompt inspection or weather
|
||||
collection. A requested initialization failure is terminal and occurs
|
||||
before managed run artifacts exist.
|
||||
3. Pass debug intent through project-owned CLI and app request types. When
|
||||
enabled, request sensitive debug values from the executor:
|
||||
|
||||
- write preparation debug inside the preparation callback, after the normal
|
||||
preparation artifact is saved and before metadata is finalized and
|
||||
provider work begins;
|
||||
- if the debug write fails, return the callback error so `RunPrepared` is
|
||||
not called;
|
||||
- write execution debug immediately after receiving a completed execution
|
||||
result and before continuing with downstream validation/rendering; and
|
||||
- treat every requested debug write failure as terminal while preserving
|
||||
already reached normal artifacts.
|
||||
|
||||
4. Set `llmDebugPath` only after the per-run debug directory contains at least
|
||||
one successfully written debug artifact. Never copy sensitive debug content
|
||||
into a normal artifact, log, error, or summary.
|
||||
5. Finalize human and JSON CLI summaries around project-owned
|
||||
`preparationPath`, `executionPath`, and optional `llmDebugPath`. Include
|
||||
reached paths on partial failure; omit absent paths instead of inventing
|
||||
them.
|
||||
6. Preserve existing output behavior: quiet mode suppresses successful human
|
||||
output, JSON output remains machine-readable, and failures return a
|
||||
non-zero status with safe classified text.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add CLI parser and help tests for `--llm-debug-dir` on all four generate
|
||||
commands and its absence from YAML configuration.
|
||||
- Verify debug initialization precedes inspection and collection.
|
||||
- Verify preparation debug persistence precedes provider execution and that a
|
||||
write failure prevents the model-client call.
|
||||
- Cover execution debug success/failure, partial normal-artifact retention,
|
||||
`llmDebugPath` population rules, quiet mode, JSON summaries, and safe
|
||||
failure text.
|
||||
- Assert that summaries and routine logs contain no rendered prompt, schema
|
||||
body, input body, generated body, endpoint, full parameters, or credential.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/app ./internal/cli ./internal/state
|
||||
go test -race ./internal/app ./internal/adapters/promptkit
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All single-report commands provide the complete Promptkit, v2 persistence,
|
||||
opt-in debug, and summary behavior required by the roadmap. Debug failure
|
||||
ordering is enforced without exposing sensitive content.
|
||||
|
||||
## Stage 10: Cut Over Batches And Remove Scriptorium
|
||||
|
||||
### Goal
|
||||
|
||||
Complete the production cutover, share one engine through each sequential
|
||||
batch, and delete the retired dependency boundary.
|
||||
|
||||
### Work
|
||||
|
||||
1. Construct one executor in the CLI for each `run morning` or `run evening`
|
||||
invocation and pass it through `BatchRequest` to every report.
|
||||
2. Parse and initialize `--llm-debug-dir` for run commands using the same
|
||||
policy as generate commands.
|
||||
3. Before collection, inspect the complete candidate set:
|
||||
|
||||
- morning: Today, Tomorrow, and Daily;
|
||||
- evening: Tomorrow and Daily.
|
||||
|
||||
Inspect exact prompt versions, validate declared input/output contracts,
|
||||
resolve unique effective profiles, and enforce credential availability.
|
||||
Daily inspection occurs before its collection-dependent future dates are
|
||||
known because every eligible Daily run uses the same exact prompt contract.
|
||||
4. Collect once, plan the batch as before, and execute every planned report
|
||||
sequentially through the shared executor, Stage 8 generation core, and
|
||||
store. Delete the temporary legacy batch helper after this call site moves.
|
||||
Preserve continuation after independent report failures and existing
|
||||
batch-notification gating.
|
||||
5. Include preparation, execution, and optional debug paths in each batch
|
||||
item, including failed items when those paths were reached.
|
||||
6. Replace Scriptorium configuration entirely with Promptkit configuration.
|
||||
Update defaults, validation, examples, config tests, and all construction
|
||||
sites. Explicitly reject a top-level `scriptorium:` key with an actionable
|
||||
migration error even though the general YAML loader is currently
|
||||
permissive; silently ignoring a former execution configuration is unsafe.
|
||||
Do not translate it or add a dual-run mode. The maintained examples contain
|
||||
only `promptkit:`.
|
||||
7. Delete:
|
||||
|
||||
- `internal/adapters/scriptorium`;
|
||||
- the Scriptorium renderer interface and all result/request types;
|
||||
- the legacy state path members and Scriptorium write methods retained in
|
||||
Stage 5;
|
||||
- subprocess fakes and subprocess-specific tests;
|
||||
- direct-Markdown branches and remaining Scriptorium defaults; and
|
||||
- dead helpers used only by the old adapter.
|
||||
|
||||
8. Run `go mod tidy` and verify the module graph contains Promptkit `v0.4.0`
|
||||
and no Scriptorium module or local replacement.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add batch tests proving one factory call/engine per CLI invocation, one
|
||||
executor shared across all reports, one collection, sequential ordering,
|
||||
later-report continuation, per-report path retention, and notification only
|
||||
after complete success.
|
||||
- Cover a capacity failure for one report followed by a later independent
|
||||
report and confirm no retry occurs.
|
||||
- Cover debug grouping for multiple reports and multiple Daily dates.
|
||||
- Update config tests for exact defaults, mutual exclusivity, local endpoint,
|
||||
zero/unlimited and negative concurrency, and maintained examples.
|
||||
- Run:
|
||||
|
||||
```sh
|
||||
go test ./internal/config ./internal/app ./internal/cli
|
||||
go test -race ./internal/app ./internal/adapters/promptkit
|
||||
go test ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
### Exit Gate
|
||||
|
||||
All production generation paths use Promptkit. A batch owns one engine and
|
||||
continues sequentially under existing failure policy. No Scriptorium code,
|
||||
configuration field, subprocess path, or dependency remains.
|
||||
|
||||
## Stage 11: Update Canonical Documentation And Complete Verification
|
||||
|
||||
### Goal
|
||||
|
||||
Make current-state documentation match the completed implementation, remove
|
||||
migration-only source material, and perform repository-wide verification.
|
||||
|
||||
### Work
|
||||
|
||||
1. Update canonical current-state owners in the same migration change:
|
||||
|
||||
- `docs/development.md` for repository orientation, package map, task
|
||||
routing, and validation language;
|
||||
- `docs/policy/architecture.md` for the Promptkit adapter boundary,
|
||||
generated-text-only flow, prepared execution, and four-report product;
|
||||
- `docs/policy/testing.md` for Promptkit/provider fakes instead of
|
||||
subprocess fakes and the offline external-boundary rule;
|
||||
- `docs/cli.md` for four commands, `--llm-debug-dir`, and renamed summary
|
||||
fields;
|
||||
- `docs/config.md` and `examples/` for the exact Promptkit contract;
|
||||
- `docs/operations.md` for v2 paths, sensitive debug retention and
|
||||
permissions, preparation/execution lifecycle, and v1 inspection;
|
||||
- `docs/troubleshooting.md` for inspection, profile/credential,
|
||||
preparation, capacity, execution, validation, and debug failures;
|
||||
- relevant `docs/internal/` files for app, CLI, report, prompt input,
|
||||
generated text, templates, state, collection, briefing, facts, changes,
|
||||
and package boundaries; and
|
||||
- `docs/templates.md` for Promptkit-generated prose and the relocated schema
|
||||
owner.
|
||||
|
||||
2. Replace `docs/integrations/scriptorium.md` and
|
||||
`docs/internal/scriptorium-adapter.md` with canonical Promptkit integration
|
||||
and adapter documents. The integration document owns the logical
|
||||
prompt/profile/schema and durable compatibility contract; the internal
|
||||
document owns construction, mapping, lifecycle, and tests. Link rather than
|
||||
duplicate the full CLI/config/operations references.
|
||||
3. Search all non-roadmap current-state documentation, examples, Go code,
|
||||
tests, help output, and module metadata for stale `scriptorium`,
|
||||
`local-heavy`, retired report commands, old artifact field names, old
|
||||
filenames, and old metadata versions. Retain old names only in explicit v1
|
||||
compatibility code/tests and historical roadmap discussion.
|
||||
4. Remove `docs/roadmap/scriptorium` after confirming every promoted runtime
|
||||
asset is represented under `internal/promptassets` and covered by asset
|
||||
tests. The unused legacy Daily Markdown prompt is deleted with this source
|
||||
directory.
|
||||
5. Keep the feature roadmap and this plan as temporary migration documents
|
||||
while implementation is under review. Once the implementation is accepted,
|
||||
mark both complete and move every still-useful contract to its canonical
|
||||
current-state owner before later archival or removal.
|
||||
6. Review all added links, fenced examples, flags, field names, defaults,
|
||||
schema IDs, file paths, and version strings against executable sources.
|
||||
Confirm examples contain no credentials or private infrastructure values.
|
||||
|
||||
### Final Verification
|
||||
|
||||
Run:
|
||||
|
||||
```sh
|
||||
gofmt -w <all changed Go files>
|
||||
go mod tidy
|
||||
go test ./...
|
||||
go test -race ./...
|
||||
go run ./cmd/weatherreporter --help
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Then verify explicitly:
|
||||
|
||||
- `go list -m gitea.maximumdirect.net/eric/promptkit` reports `v0.4.0`;
|
||||
- no committed `go.work`, `replace`, secret fixture, or live-provider test
|
||||
exists;
|
||||
- the maintained examples load through config tests;
|
||||
- all four embedded prompts inspect at exact version `1.0.0`;
|
||||
- no runtime prompt requests repair attempts;
|
||||
- no ordinary artifact or CLI summary includes rendered prompts, schema
|
||||
bodies, input bodies, generated bodies, provider endpoints, or credentials;
|
||||
- v1 metadata fixtures remain inspectable and new runs write only v2;
|
||||
- help exposes only Daily, Today, Tomorrow, Hourly, morning, and evening; and
|
||||
- managed Markdown remains the only Distributor upload source.
|
||||
|
||||
### Exit Gate
|
||||
|
||||
Every completion criterion in the feature roadmap is demonstrably satisfied.
|
||||
Current-state documentation describes the implementation rather than the
|
||||
migration, and the repository passes all final verification commands.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and this implementation plan contain all product,
|
||||
architecture, configuration, compatibility, security, sequencing, and
|
||||
verification decisions required to begin implementation.
|
||||
@@ -1,56 +1,87 @@
|
||||
# Promptkit Migration Roadmap
|
||||
|
||||
Status: Accepted migration policy; the migration itself is unimplemented.
|
||||
Status: Accepted feature roadmap; the target state is unimplemented.
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the scope and desired end state for replacing the
|
||||
external Scriptorium CLI integration with the Promptkit Go library. The
|
||||
migration is not yet implemented. Current Scriptorium behavior remains
|
||||
documented in the [Scriptorium integration guide](../integrations/scriptorium.md)
|
||||
until the replacement is complete.
|
||||
external Scriptorium CLI integration with Promptkit. Current implemented
|
||||
behavior remains documented outside `docs/roadmap/` until the replacement is
|
||||
complete.
|
||||
|
||||
A separate staged implementation plan will describe how to move from the
|
||||
current code to this target state. That plan should reference this roadmap
|
||||
rather than redefine its architectural decisions or scope.
|
||||
## Current Baseline
|
||||
|
||||
Status: Current migration input.
|
||||
|
||||
Weatherreporter currently exposes seven report definitions, but only four have
|
||||
complete prompt-backed report implementations:
|
||||
|
||||
- Daily Report: `weather.daily_generated_text`
|
||||
- Today Report: `weather.today_generated_text`
|
||||
- Tomorrow Report: `weather.tomorrow_generated_text`
|
||||
- Hourly Report: `weather.hourly_generated_text`
|
||||
|
||||
The three-day, weekend, and storm commands and registry definitions have no
|
||||
corresponding Scriptorium prompt or schema and never formed complete
|
||||
operational report products. The `weather.daily_report` Scriptorium prompt is
|
||||
legacy source material and is not selected by the current registry.
|
||||
|
||||
The available Scriptorium source corpus is retained temporarily under
|
||||
`docs/roadmap/scriptorium/`. It contains the four operational generated-text
|
||||
prompt definitions, their referenced content, their private response schemas,
|
||||
shared instructions, and the unused legacy Daily Markdown prompt.
|
||||
|
||||
## Desired End State
|
||||
|
||||
Status: Accepted target state; unimplemented.
|
||||
|
||||
Weatherreporter uses a pinned released version of
|
||||
`gitea.maximumdirect.net/eric/promptkit` as its in-process prompt preparation
|
||||
and LLM execution engine. The `scriptorium` executable, subprocess adapter,
|
||||
configuration, runtime dependency, and integration documentation have been
|
||||
removed.
|
||||
Weatherreporter pins
|
||||
`gitea.maximumdirect.net/eric/promptkit` at `v0.4.0` and uses it as the
|
||||
in-process engine for prompt inspection, prepared execution, provider calls,
|
||||
and first-pass output validation.
|
||||
|
||||
The migration does not change weatherreporter's fundamental product behavior.
|
||||
Weather selection, forecast derivation, report periods, module construction,
|
||||
The `scriptorium` executable, subprocess adapter, configuration, runtime
|
||||
dependency, direct-Markdown execution path, and integration documentation have
|
||||
been removed. The four operational reports continue to use structured
|
||||
generated text followed by weatherreporter-owned validation and Markdown
|
||||
templates.
|
||||
|
||||
The unfinished three-day, weekend, and storm reports are not implemented as
|
||||
part of this migration. Their incomplete CLI, registry, documentation, and
|
||||
generation declarations are removed from the implemented surface before the
|
||||
migration is considered complete. Any future implementation of those products
|
||||
requires separate roadmap scope, prompt and schema design, tests, and
|
||||
documentation.
|
||||
|
||||
Weather selection, forecast derivation, valid periods, module construction,
|
||||
Recent Changes, generated-text interpretation, Markdown templates, durable
|
||||
state, inspection, output copies, and distributor notification remain owned by
|
||||
state, inspection, output copies, and Distributor notification remain owned by
|
||||
weatherreporter.
|
||||
|
||||
All report prompts and private response schemas are versioned application
|
||||
assets. Operators may configure Promptkit execution profiles without replacing
|
||||
the report-owned prompt and schema corpus. One Promptkit engine is constructed
|
||||
per CLI invocation and shared by every report in that invocation, including
|
||||
all reports in a morning or evening batch.
|
||||
The four report prompts and private response schemas are versioned embedded
|
||||
application assets. Operators configure Promptkit profiles without replacing
|
||||
the report-owned corpus. One Promptkit engine is constructed per CLI
|
||||
invocation and shared by every report in that invocation, including all
|
||||
reports in a morning or evening batch.
|
||||
|
||||
Promptkit is isolated behind a weatherreporter-owned prompt execution contract.
|
||||
Promptkit is isolated behind a weatherreporter-owned execution contract.
|
||||
Promptkit request, result, validation, error, profile, backend, and provider
|
||||
types do not leak into application orchestration, report definitions, domain
|
||||
packages, CLI summaries, state contracts, or distributor behavior.
|
||||
packages, CLI summaries, durable state contracts, or Distributor behavior.
|
||||
|
||||
## Goals
|
||||
|
||||
Status: Accepted migration scope; unimplemented.
|
||||
|
||||
- Remove the runtime dependency on the `scriptorium` executable.
|
||||
- Replace shell-free subprocess orchestration with typed in-process Promptkit
|
||||
preparation and execution.
|
||||
- Preserve the seven report definitions and their existing prompt IDs.
|
||||
- Preserve both direct-Markdown and generated-text-template report workflows.
|
||||
- Remove the Scriptorium runtime dependency and subprocess boundary.
|
||||
- Migrate the four operational report prompts to Promptkit `v0.4.0`.
|
||||
- Use prepared execution to persist preparation provenance before provider work
|
||||
while executing the exact frozen snapshot.
|
||||
- Validate report prompt and profile selections before weather collection when
|
||||
the required information is available.
|
||||
- Preserve deterministic module snapshots and structured Recent Changes.
|
||||
- Preserve generated-text domain validation and repository-owned Markdown
|
||||
rendering.
|
||||
- Preserve context cancellation, actionable errors, secret redaction, and
|
||||
inspectable failures.
|
||||
- Improve durable prompt provenance with prompt, input, profile, model,
|
||||
@@ -59,6 +90,8 @@ Status: Accepted migration scope; unimplemented.
|
||||
metadata and CLI output.
|
||||
- Keep tests offline and deterministic through injected Promptkit model
|
||||
clients and fixtures.
|
||||
- Remove incomplete report declarations from the implemented product surface
|
||||
rather than creating new report products during an integration migration.
|
||||
|
||||
## Non-Goals
|
||||
|
||||
@@ -66,17 +99,24 @@ Status: Accepted migration scope; unimplemented.
|
||||
|
||||
The migration will not:
|
||||
|
||||
- create prompts, schemas, templates, or completed products for three-day,
|
||||
weekend, or storm reports;
|
||||
- preserve the unused `weather.daily_report` legacy Markdown prompt as an
|
||||
active runtime asset;
|
||||
- preserve a direct-Markdown LLM generation mode;
|
||||
- move meteorological selection, derivation, thresholds, or comparison logic
|
||||
into prompts or Promptkit;
|
||||
- send raw unbounded Weather API responses to the model;
|
||||
- replace weatherreporter's generated-text domain validation or Markdown
|
||||
template rendering;
|
||||
- add a general workflow engine, provider plugin system, or arbitrary backend
|
||||
registry to weatherreporter;
|
||||
- add automatic provider, validation, or capacity retries;
|
||||
- add concurrent report generation to the existing sequential batch workflow;
|
||||
- expose Promptkit types as a weatherreporter component contract;
|
||||
- keep a production-selectable Scriptorium/Promptkit dual-run mode; or
|
||||
registry;
|
||||
- add automatic provider, validation, repair, or capacity retries;
|
||||
- add concurrent report generation to the sequential batch workflow;
|
||||
- expose Promptkit types as weatherreporter contracts;
|
||||
- keep a production-selectable Scriptorium/Promptkit dual-run mode;
|
||||
- require Promptkit eager source validation, structured generation errors, or
|
||||
semantic execution-target fingerprints; or
|
||||
- use an unpublished Promptkit commit, committed Go workspace, or committed
|
||||
local module replacement.
|
||||
|
||||
@@ -84,121 +124,229 @@ The migration will not:
|
||||
|
||||
Status: Accepted decisions for the unimplemented migration.
|
||||
|
||||
### Dependency And Versioning
|
||||
### Dependency And Upgrade Boundary
|
||||
|
||||
- The initial integration will pin Promptkit `v0.3.0`.
|
||||
- The migration pins the tagged Promptkit `v0.4.0` release.
|
||||
- Coordinated local development may temporarily use the sibling Promptkit
|
||||
checkout, but committed module metadata must reference the tagged release.
|
||||
- A future Promptkit upgrade requires an explicit review of the public engine,
|
||||
prompt/profile/schema formats, error identities, validation behavior, and
|
||||
outbound provider contract used by weatherreporter.
|
||||
- The adapter relies on the public root Promptkit package only.
|
||||
- A future Promptkit upgrade requires explicit review of prepared-execution
|
||||
lifecycle, prompt and profile inspection, prompt/profile/schema formats,
|
||||
error identities, validation behavior, capacity behavior, and the outbound
|
||||
provider contract.
|
||||
- Promptkit's deferred eager source validation, structured generation errors,
|
||||
and semantic execution-target fingerprints do not block this migration.
|
||||
|
||||
### Operational Report Scope
|
||||
|
||||
- The migration preserves these prompt IDs:
|
||||
`weather.daily_generated_text`, `weather.today_generated_text`,
|
||||
`weather.tomorrow_generated_text`, and `weather.hourly_generated_text`.
|
||||
- Each operational report definition selects the exact embedded prompt version
|
||||
`1.0.0`; execution does not rely on ambiguous single-version lookup.
|
||||
- Morning and evening batch membership remains based on Today, Tomorrow, and
|
||||
eligible future Daily reports.
|
||||
- Three-day, weekend, and storm are removed from current CLI help, parsing,
|
||||
report registry membership, tests that claim implemented generation, and
|
||||
non-roadmap documentation.
|
||||
- The future product concepts may remain under `docs/roadmap/`, but migration
|
||||
verification does not invent outputs or compare nonexistent prompts.
|
||||
|
||||
### Application Boundary
|
||||
|
||||
- Promptkit remains an adapter boundary even though it runs in process.
|
||||
- A weatherreporter-owned contract will represent preparation, execution,
|
||||
output formats, validation, usage, provenance, and neutral error categories.
|
||||
- The Promptkit adapter will map public Promptkit values into that contract at
|
||||
the boundary.
|
||||
- App orchestration and test fakes will depend on the weatherreporter contract,
|
||||
not on Promptkit.
|
||||
- Existing Scriptorium-specific generation mode names will be replaced with
|
||||
provider-neutral names.
|
||||
- A weatherreporter-owned contract represents prompt identity, preparation,
|
||||
execution, output, validation, usage, provenance, and neutral error
|
||||
categories.
|
||||
- The Promptkit adapter maps public Promptkit values into that contract.
|
||||
- App orchestration and test fakes depend on the project-owned contract, not
|
||||
Promptkit.
|
||||
- Scriptorium-specific request, result, error, and generation-mode types are
|
||||
removed rather than renamed and retained.
|
||||
|
||||
### Prompt And Schema Ownership
|
||||
|
||||
- Weatherreporter will embed all report prompt definitions, prompt content,
|
||||
and private response schemas.
|
||||
- Prompt assets will remain separate files rather than inline Go strings.
|
||||
- The current Scriptorium prompt corpus will be retrieved before the
|
||||
implementation stage that establishes the embedded Promptkit assets.
|
||||
- The retrieved corpus will be reviewed and converted to the pinned Promptkit
|
||||
format without changing report intent or prompt IDs.
|
||||
- The four existing generated-text prompt fragments and schemas under
|
||||
`internal/reporttemplate` will be reconciled with that corpus rather than
|
||||
duplicated.
|
||||
- Direct-Markdown prompt assets for the three-day, weekend, and storm reports
|
||||
will become weatherreporter-owned assets.
|
||||
- Weatherreporter needs one centralized embedded prompt/schema source; it does
|
||||
not need Notarius's multi-module asset-flattening registry.
|
||||
- Weatherreporter embeds the four operational prompt definitions, referenced
|
||||
prompt content, shared prompt content, and private response schemas.
|
||||
- Assets remain separate files rather than inline Go strings.
|
||||
- The temporary corpus under `docs/roadmap/scriptorium/` is migration source
|
||||
material, not the final runtime location.
|
||||
- Weatherreporter's existing generated-text domain types, schemas, and
|
||||
templates remain the canonical application contract. Imported Scriptorium
|
||||
assets are reconciled with that contract rather than copied blindly or kept
|
||||
as duplicate runtime schemas.
|
||||
- The imported Daily schema's incorrect Today `$id` and title are corrected.
|
||||
- `confidence` is handled consistently across each prompt, provider-facing
|
||||
schema, generated-text domain type, and template. The existing optional
|
||||
weatherreporter field remains supported unless a separate domain decision
|
||||
removes it.
|
||||
- Prompt input metadata identifies the serialized data package as YAML rather
|
||||
than JSON.
|
||||
- Imported `pipeline-weather/...` schema paths are replaced with paths valid
|
||||
inside the embedded Promptkit schema source.
|
||||
- Imported `repair_attempts: 2` values are removed or set to zero. The
|
||||
migration does not rely on Promptkit's internal-only repair capability.
|
||||
- The unused `weather.daily_report` prompt is not promoted into runtime assets.
|
||||
- One centralized embedded prompt/schema source is sufficient; Weatherreporter
|
||||
does not need Notarius's multi-module asset-flattening registry.
|
||||
|
||||
### Profiles, Backends, And Credentials
|
||||
|
||||
- Execution profiles remain operator-configurable rather than embedded report
|
||||
policy.
|
||||
- Configuration will support at most one external profile source: a profile
|
||||
directory or a single profile file.
|
||||
- Prompt definitions may provide their normal default profile, while
|
||||
weatherreporter may support an explicit configured profile selection.
|
||||
- Each embedded operational prompt declares Promptkit's built-in
|
||||
`gemini-flash-latest` profile as its default.
|
||||
- `gemini-flash-latest` is intentionally a moving model alias. The execution
|
||||
record captures the effective model identity, but operators who require a
|
||||
pinned model must select an explicit external profile.
|
||||
- Configuration supports at most one external profile source:
|
||||
`promptkit.profile_file` or `promptkit.profile_dir`. The two fields are
|
||||
mutually exclusive.
|
||||
- A nonblank `promptkit.profile` is the explicit request profile for every
|
||||
report in the invocation and takes precedence over each prompt's
|
||||
`default_profile`. A blank value uses the prompt default.
|
||||
- Promptkit's normal profile-source precedence remains intact: an external
|
||||
matching profile takes precedence over an embedded built-in profile, and an
|
||||
invalid matching external profile is an error rather than a reason to fall
|
||||
back.
|
||||
- Weatherreporter exposes Promptkit's conventional `local` backend through the
|
||||
narrow `promptkit.local.endpoint` and
|
||||
`promptkit.local.concurrency_limit` configuration fields. It does not expose
|
||||
arbitrary backend registration.
|
||||
- A configured local endpoint registers the engine-scoped `local` backend. An
|
||||
operator-supplied external profile selects it with `backend: local` and owns
|
||||
the model-specific settings; Weatherreporter does not invent a local model
|
||||
profile.
|
||||
- Local concurrency defaults to one. A value of zero means unlimited, matching
|
||||
Promptkit, and a negative value is invalid. Queue capacity and general
|
||||
backend parameters are not exposed.
|
||||
- Credential values remain in environment variables or file-backed
|
||||
environment secrets. Configuration contains only credential source names.
|
||||
- Provider credentials must not appear in logs, errors, CLI output, durable
|
||||
metadata, preparation artifacts, execution artifacts, or debug summaries.
|
||||
- Weatherreporter will not expose Promptkit's general backend registry as
|
||||
arbitrary application configuration.
|
||||
- Provider credentials never appear in logs, errors, CLI output, durable
|
||||
metadata, preparation records, execution records, or debug summaries.
|
||||
- Promptkit `InspectProfile` reports structural target and credential
|
||||
requirements; Weatherreporter owns policy for checking configured
|
||||
environment availability.
|
||||
- Promptkit revalidates environment credentials at `RunPrepared`; a successful
|
||||
preparation does not promise that execution-time credentials remain
|
||||
available.
|
||||
|
||||
### Engine Lifetime
|
||||
### Configuration Contract
|
||||
|
||||
- One Promptkit engine will be constructed per CLI invocation at the
|
||||
application composition boundary.
|
||||
- Single-report generation will use that engine for preparation and execution.
|
||||
- Morning and evening batches will share the same engine across every planned
|
||||
report.
|
||||
- Per-report orchestration will not construct its own default Promptkit engine.
|
||||
- Promptkit backend capacity state and HTTP transport will therefore be shared
|
||||
consistently for the invocation.
|
||||
The replacement configuration surface is:
|
||||
|
||||
```yaml
|
||||
promptkit:
|
||||
profile: ""
|
||||
profile_file: ""
|
||||
profile_dir: ""
|
||||
timeout: 2m
|
||||
|
||||
local:
|
||||
endpoint: ""
|
||||
concurrency_limit: 1
|
||||
```
|
||||
|
||||
- `timeout` remains the transport-wide provider-call safety cap.
|
||||
- A blank local endpoint leaves the conventional local backend unregistered.
|
||||
- Scriptorium's `binary`, `config_path`, and `extra_args` settings have no
|
||||
Promptkit equivalents and are removed.
|
||||
- Configuration validation rejects simultaneous `profile_file` and
|
||||
`profile_dir` values, invalid local endpoints, negative concurrency, and
|
||||
selected profiles that cannot resolve their backend.
|
||||
|
||||
### Engine Construction And Inspection
|
||||
|
||||
- One Promptkit engine is constructed per CLI invocation at the application
|
||||
composition boundary.
|
||||
- Single-report generation and every report in a batch use that same engine.
|
||||
- Per-report orchestration does not construct a default engine.
|
||||
- Promptkit backend capacity state and HTTP transport are shared consistently
|
||||
for the invocation.
|
||||
- Before collection, `InspectPrompt` checks every selected report's exact ID
|
||||
and version, declared `data_package` input, default-profile metadata, prompt
|
||||
hash availability, and declared output contract.
|
||||
- `InspectPrompt` is a point-in-time structural check. It does not load a JSON
|
||||
Schema, resolve a profile, or freeze later execution.
|
||||
- Explicit profile overrides and relevant prompt defaults are checked with
|
||||
`InspectProfile` before collection when application policy requires them.
|
||||
- `InspectProfile` is also point-in-time and does not check credential values.
|
||||
- Successful `PrepareExecution`, not inspection, is the per-run authority for
|
||||
loaded schema, rendered content, frozen inputs, effective settings, and
|
||||
durable execution provenance.
|
||||
|
||||
### Prompt Input
|
||||
|
||||
- Promptkit will continue to receive the curated `data_package` produced by
|
||||
- Promptkit receives only the curated `data_package` produced by
|
||||
`internal/promptinput`.
|
||||
- Weatherreporter will serialize the data package once, atomically persist
|
||||
those exact bytes, and supply the same bytes as a Promptkit inline artifact.
|
||||
- The managed data-package path may be supplied as non-secret artifact
|
||||
provenance.
|
||||
- Weatherreporter will not delegate unrestricted path loading to Promptkit's
|
||||
- Weatherreporter serializes the package once, atomically persists those exact
|
||||
bytes, and supplies the same bytes with a Promptkit inline artifact.
|
||||
- The managed data-package path may be supplied as non-secret provenance
|
||||
through the inline artifact URI.
|
||||
- Weatherreporter does not delegate unrestricted path loading to Promptkit's
|
||||
default file artifact reader.
|
||||
- The same immutable Promptkit request will be used for preparation and
|
||||
execution so the preflight and run inputs cannot diverge.
|
||||
- Prompt inspection and adapter tests verify that `data_package` is required
|
||||
and declared with the chosen YAML media type.
|
||||
|
||||
### Preparation And Execution
|
||||
### Prepared Execution
|
||||
|
||||
- Promptkit `Prepare` replaces the current Scriptorium render preflight.
|
||||
- Promptkit `Run` performs both Markdown and structured generated-text
|
||||
execution.
|
||||
- Promptkit basic validation will be used where appropriate for direct
|
||||
Markdown output.
|
||||
- Promptkit JSON Schema validation provides the provider-facing and first
|
||||
structured-output check for generated-text reports.
|
||||
- `Engine.PrepareExecution` replaces Scriptorium render preflight.
|
||||
- Weatherreporter obtains `PreparedExecution.Details`, maps a safe subset into
|
||||
its own preparation record, and persists that record before calling
|
||||
`Engine.RunPrepared`.
|
||||
- `RunPrepared` executes the frozen prompt, profile, schema, inputs, rendered
|
||||
messages, target, and validation resources retained by the handle.
|
||||
- Every acquired handle is followed immediately by `defer handle.Discard()`.
|
||||
Discard is safe after execution and releases unused private execution state.
|
||||
- Handles remain adapter-local, engine-bound, one-shot, in-process values.
|
||||
They are never serialized, persisted, copied into app contracts, or treated
|
||||
as restartable jobs.
|
||||
- Preparation and execution use independent contexts. Execution receives the
|
||||
active report workflow context.
|
||||
- Capacity is not reserved during preparation. Capacity rejection can
|
||||
therefore occur after a preparation record has been persisted.
|
||||
- `RunPrepared` consumes the handle on success and every operational failure.
|
||||
- Preparation details remain available from the adapter after execution or
|
||||
discard, but rendered messages are not copied into routine durable state.
|
||||
- Promptkit execution timing excludes preparation and consumer-held delay.
|
||||
Weatherreporter records preparation timing and execution timing separately.
|
||||
|
||||
### Execution And Validation
|
||||
|
||||
- All four operational reports use Promptkit JSON Schema output validation.
|
||||
- A completed Promptkit validation rejection returns a `RunResult`; the
|
||||
adapter retains raw output and bounded validation details before failing the
|
||||
report.
|
||||
- An operational generation or validation error returns no partial
|
||||
`RunResult`.
|
||||
- Weatherreporter's `internal/generatedtext` validation remains the final
|
||||
report-specific domain boundary.
|
||||
- Weatherreporter's `internal/reporttemplate` remains responsible for
|
||||
generated-text Markdown rendering.
|
||||
- Weatherreporter will atomically persist Promptkit output rather than asking
|
||||
the dependency to write managed report files.
|
||||
- The migration will not rely on Promptkit output repair. Promptkit v0.3.0's
|
||||
public engine validates in a single pass even when a prompt declares repair
|
||||
attempts.
|
||||
report-specific decode and domain boundary.
|
||||
- Weatherreporter's `internal/reporttemplate` remains responsible for managed
|
||||
Markdown rendering.
|
||||
- Weatherreporter atomically persists Promptkit raw output and later artifacts
|
||||
rather than asking Promptkit to choose managed filesystem paths.
|
||||
- No Promptkit output-repair behavior is assumed or requested.
|
||||
|
||||
## Durable Artifacts And Observability
|
||||
|
||||
Status: Accepted design constraints; unimplemented.
|
||||
|
||||
Routine durable artifacts should retain useful non-secret provenance without
|
||||
persisting full rendered prompts by default.
|
||||
Routine durable state retains useful non-secret provenance without persisting
|
||||
full rendered prompts.
|
||||
|
||||
The preparation record should contain:
|
||||
The preparation record contains:
|
||||
|
||||
- prompt ID and version;
|
||||
- prompt ID and exact version;
|
||||
- prompt definition hash;
|
||||
- rendered prompt hash;
|
||||
- input hashes;
|
||||
- selected profile and backend identity;
|
||||
- effective model identity;
|
||||
- output contract summary; and
|
||||
- preparation timing.
|
||||
- output contract summary;
|
||||
- preparation start, end, and duration; and
|
||||
- the path of the exact persisted data package.
|
||||
|
||||
The execution record and run metadata should contain, when available:
|
||||
The execution record and run metadata contain, when available:
|
||||
|
||||
- Promptkit run ID;
|
||||
- prompt ID, version, and hashes;
|
||||
@@ -206,122 +354,162 @@ The execution record and run metadata should contain, when available:
|
||||
- selected profile, backend, and model identity;
|
||||
- generated-content hash;
|
||||
- token usage;
|
||||
- start, end, and duration;
|
||||
- execution start, end, and duration;
|
||||
- validation status and bounded diagnostics; and
|
||||
- the path of any separately persisted raw generated output.
|
||||
- paths of separately persisted raw output, normalized generated text, render
|
||||
context, managed Markdown, and other artifacts reached by the workflow.
|
||||
|
||||
Provider endpoints, full effective model parameter maps, rendered messages,
|
||||
schema bodies, data-package contents, and generated content do not belong in
|
||||
routine metadata or CLI summaries.
|
||||
|
||||
Rendered messages and other content-rich preparation or response diagnostics
|
||||
will be available only through an explicitly enabled debug mechanism. Debug
|
||||
artifacts must be documented as potentially sensitive, must not contain
|
||||
credentials, and must have a clear operator-owned retention policy.
|
||||
are available only when the operator supplies
|
||||
`--llm-debug-dir <path>` to a single-report or batch command.
|
||||
|
||||
- There is initially no persistent YAML setting for debug capture.
|
||||
- The debug root is validated or created before weather collection or provider
|
||||
work. A requested destination that cannot be secured or written is an error.
|
||||
- Artifacts are grouped beneath
|
||||
`<path>/<report-id>/<valid-date>/<run-id>/`.
|
||||
- Directories and files use owner-only permissions and atomic writes.
|
||||
- Debug artifacts may contain rendered messages and content-rich preparation
|
||||
or response diagnostics, but never credentials.
|
||||
- The debug path appears in command output only when debug capture is enabled;
|
||||
it is not added to routine durable metadata.
|
||||
- Debug artifacts are not cache or comparison inputs. Their retention is owned
|
||||
by the operator who selected the directory.
|
||||
|
||||
### Artifact Identities And Versions
|
||||
|
||||
Weatherreporter replaces Scriptorium-specific artifact identities rather than
|
||||
reusing names whose meanings have changed:
|
||||
|
||||
- `PromptPreparationArtifact` uses schema version
|
||||
`weatherreporter.prompt_preparation.v1`, is written as
|
||||
`prompt_preparation.<runID>.json`, and is referenced by
|
||||
`preparationPath`.
|
||||
- `PromptExecutionArtifact` uses schema version
|
||||
`weatherreporter.prompt_execution.v1`, is written as
|
||||
`prompt_execution.<runID>.json`, and is referenced by `executionPath`.
|
||||
- Run metadata advances to `weatherreporter.metadata.v2` and uses those new
|
||||
path fields.
|
||||
|
||||
Preparation files remain beneath the existing configurable `preflight/`
|
||||
directory, and execution files remain beneath the existing `snapshots/` tree.
|
||||
The stable physical grouping limits deployment disruption without preserving
|
||||
misleading Scriptorium-era filenames or field names. Raw generated output,
|
||||
normalized generated text, render context, managed Markdown, and other
|
||||
artifacts whose meanings have not changed retain their existing names and
|
||||
locations.
|
||||
|
||||
Run inspection remains able to read `weatherreporter.metadata.v1` and its
|
||||
legacy `preflightPath` and `generatedTextResultPath` references. New runs write
|
||||
only the v2 metadata and new artifact names; Weatherreporter does not
|
||||
dual-write deprecated aliases. CLI summary fields adopt `preparationPath` and
|
||||
`executionPath` as an explicit, documented contract change.
|
||||
|
||||
## Failure Contract
|
||||
|
||||
Status: Accepted design constraints; unimplemented.
|
||||
|
||||
Promptkit returns a completed `RunResult` for output-validation failure but no
|
||||
partial result for operational preparation or execution errors. Weatherreporter
|
||||
will preserve that distinction.
|
||||
|
||||
- A preparation failure produces a redacted weatherreporter-owned failure
|
||||
receipt with report, RunID, prompt, stage, timing, and classified error
|
||||
context. It does not fabricate a Promptkit preparation result.
|
||||
context. It does not fabricate Promptkit preparation details.
|
||||
- An operational execution failure retains the successful preparation record
|
||||
and adds a redacted execution failure receipt. No partial Promptkit result or
|
||||
model output is invented.
|
||||
- A Promptkit validation failure retains the returned result, raw generated
|
||||
- A Promptkit validation rejection retains the returned result, raw generated
|
||||
output, validation details, and safe provenance before the report fails.
|
||||
- A later weatherreporter generated-text decode, domain-validation, or template
|
||||
failure retains every raw and validated artifact reached before that stage.
|
||||
- Context cancellation takes precedence when the caller context is canceled.
|
||||
- Promptkit capacity rejection maps to a weatherreporter-owned error category.
|
||||
It is an operational report failure, not invalid model output.
|
||||
- A later generated-text decode, domain-validation, or template failure
|
||||
retains every raw and validated artifact reached before that stage.
|
||||
- Caller cancellation takes precedence when the active workflow context is
|
||||
canceled.
|
||||
- `promptkit.CapacityError` is recognized with `errors.As`; its backend ID is
|
||||
copied into a weatherreporter-owned capacity error while
|
||||
`ErrCapacityExceeded` remains the classification.
|
||||
- Capacity rejection is an operational report failure, not invalid model
|
||||
output, and does not trigger an automatic retry.
|
||||
- Other Promptkit public error identities are translated into the narrow
|
||||
weatherreporter error categories needed by CLI, metadata, and batch
|
||||
behavior. Diagnostic prose is not parsed as a contract.
|
||||
- Single-report commands return the classified failure with available
|
||||
inspectable paths.
|
||||
- Batch runs continue independent later reports under the existing batch
|
||||
failure policy.
|
||||
- The migration adds no automatic retries. Any future retry policy belongs to
|
||||
app orchestration, not the Promptkit adapter.
|
||||
- Any future retry policy belongs to app orchestration, not the adapter.
|
||||
|
||||
## Compatibility Requirements
|
||||
|
||||
Status: Accepted design constraints; unimplemented.
|
||||
|
||||
- Report IDs, prompt IDs, report selection, valid periods, artifact grouping,
|
||||
output names, and distributor bundle behavior remain stable.
|
||||
- Daily, Today, Tomorrow, and Hourly report IDs, prompt IDs, valid periods,
|
||||
artifact grouping, output names, and Distributor bundle behavior remain
|
||||
stable.
|
||||
- Morning and evening batch collection, planning, ordering, and continuation
|
||||
behavior remains stable.
|
||||
- Module snapshot and Recent Changes behavior remains deterministic.
|
||||
- Promptkit receives only the existing curated prompt-input boundary.
|
||||
- Generated reports continue to use the managed Markdown path as the
|
||||
distributor upload source.
|
||||
- Managed Markdown remains the Distributor upload source.
|
||||
- RunID lookup and inspection remain available for successful and failed runs.
|
||||
- Existing managed state paths remain stable where their meaning is unchanged.
|
||||
Scriptorium-specific artifact names or schemas may change when retaining
|
||||
them would misrepresent the new contract.
|
||||
- Any artifact or metadata schema change is explicit, documented, and covered
|
||||
by state and inspection tests.
|
||||
- Existing managed paths remain stable where their meaning is unchanged.
|
||||
Scriptorium-specific artifact names or schemas change when retaining them
|
||||
would misrepresent the Promptkit contract.
|
||||
- Existing v1 run metadata and referenced artifacts remain inspectable after
|
||||
the migration. New runs use the v2 metadata and Promptkit-era artifact
|
||||
identities without dual-writing deprecated aliases.
|
||||
- Artifact or metadata schema changes are explicit, documented, and covered by
|
||||
state and inspection tests.
|
||||
- Prompt or generated content is not added to routine logs or CLI summaries.
|
||||
- Tests do not require live Promptkit providers or credentials.
|
||||
- Tests do not require live providers or credentials.
|
||||
- Removing incomplete three-day, weekend, and storm surfaces is documented as
|
||||
correction of an unfinished product boundary, not as successful Promptkit
|
||||
migration of those reports.
|
||||
|
||||
## Verification And Completion Criteria
|
||||
|
||||
Status: Proposed completion criteria for the unimplemented migration.
|
||||
Status: Accepted completion criteria for the unimplemented migration.
|
||||
|
||||
The migration is complete when:
|
||||
|
||||
- all seven reports prepare and execute through Promptkit using embedded
|
||||
report-owned assets;
|
||||
- direct-Markdown and generated-text-template paths have deterministic offline
|
||||
adapter and app-level coverage;
|
||||
- preparation, provider failure, capacity rejection, cancellation, timeout,
|
||||
Promptkit validation failure, generated-text validation failure, template
|
||||
failure, and successful generation preserve their specified artifacts;
|
||||
- morning and evening batches construct one shared engine and preserve current
|
||||
- the four operational reports inspect, prepare, and execute through Promptkit
|
||||
`v0.4.0` using embedded report-owned assets;
|
||||
- every report uses exact prompt version `1.0.0`, requires the YAML
|
||||
`data_package`, and declares the expected JSON Schema output contract;
|
||||
- prepared execution persists a safe preparation record before provider work
|
||||
and executes the same frozen snapshot;
|
||||
- deterministic offline adapter and app tests cover success, preparation
|
||||
failure, credential revalidation, capacity rejection, cancellation, timeout,
|
||||
generation failure, Promptkit validation rejection, generated-text domain
|
||||
failure, template failure, and handle discard;
|
||||
- morning and evening batches construct one engine and preserve current
|
||||
collection, planning, ordering, continuation, output, and notification
|
||||
behavior;
|
||||
- the temporary corpus has been reconciled into one runtime prompt/schema
|
||||
source without duplicate provider-facing schemas;
|
||||
- configuration examples load and contain no Scriptorium fields;
|
||||
- CLI summaries and inspection commands expose the new artifact contract
|
||||
without Promptkit dependency types;
|
||||
- CLI summaries and inspection commands expose the new project-owned artifact
|
||||
contract without Promptkit types;
|
||||
- Scriptorium code, configuration, tests, and runtime documentation have been
|
||||
removed;
|
||||
- incomplete three-day, weekend, and storm commands, registry entries, tests,
|
||||
and current-behavior documentation have been removed or moved to roadmap
|
||||
scope;
|
||||
- non-roadmap documentation describes only the implemented Promptkit
|
||||
integration;
|
||||
- `go test ./...`, CLI help validation, and `git diff --check` pass; and
|
||||
- no committed `go.work`, local `replace`, live-provider test, or secret-bearing
|
||||
fixture remains.
|
||||
- `go test ./...`, required focused or race-enabled checks, CLI help
|
||||
validation, and `git diff --check` pass; and
|
||||
- no committed `go.work`, local `replace`, live-provider test, or
|
||||
secret-bearing fixture remains.
|
||||
|
||||
Fixture-based comparison with the current Scriptorium behavior is sufficient
|
||||
for migration verification. A production-selectable dual-run period is not
|
||||
required because model calls are nondeterministic, costly, and difficult to
|
||||
compare meaningfully.
|
||||
Fixture-based comparison with prior Scriptorium behavior is sufficient.
|
||||
Production dual-run is not required because model calls are nondeterministic,
|
||||
costly, and difficult to compare meaningfully.
|
||||
|
||||
## External Prerequisite
|
||||
## Decision Status
|
||||
|
||||
Status: Required and unimplemented.
|
||||
Status: Decision-complete.
|
||||
|
||||
Before implementing the embedded asset stage, the current Scriptorium prompt
|
||||
corpus must be made available in this repository. It should include the seven
|
||||
prompt definitions, referenced content files, private response schemas,
|
||||
relevant default-profile declarations, and any shared prompt fragments needed
|
||||
to reproduce current report behavior.
|
||||
|
||||
## Open Questions
|
||||
|
||||
Status: Open; these require decisions before implementation.
|
||||
|
||||
- What exact `promptkit.*` configuration fields should replace the current
|
||||
Scriptorium fields, including the name and precedence of an optional explicit
|
||||
profile override?
|
||||
- Should weatherreporter expose Promptkit's conventional `local` backend
|
||||
registration as a narrow configuration feature, or rely initially on
|
||||
built-in and endpoint-only profiles?
|
||||
- Should report definitions store an explicit Promptkit prompt version, or
|
||||
should each embedded prompt ID be required to have exactly one version?
|
||||
- What CLI or configuration control enables sensitive prompt/response debug
|
||||
artifacts, and where should those artifacts live?
|
||||
- What final names and schema versions should replace the
|
||||
Scriptorium-specific preflight and run-result artifacts while balancing
|
||||
semantic clarity with existing state-path compatibility?
|
||||
The roadmap has no remaining open product or architecture questions. Any later
|
||||
question that changes migration scope, policy, or the desired end state must be
|
||||
resolved here rather than implicitly in code.
|
||||
|
||||
@@ -14,14 +14,14 @@ source:
|
||||
|
||||
| Report | Template | Schema | Prompt ID and source |
|
||||
| --- | --- | --- | --- |
|
||||
| Daily | `templates/daily.md.tmpl` (`daily`) | `daily` | `weather.daily_generated_text`; `prompts/daily.generated_text.md` |
|
||||
| Today | `templates/today.md.tmpl` (`today`) | `today` | `weather.today_generated_text`; `prompts/today.generated_text.md` |
|
||||
| Tomorrow | `templates/tomorrow.md.tmpl` (`tomorrow`) | `tomorrow` | `weather.tomorrow_generated_text`; `prompts/tomorrow.generated_text.md` |
|
||||
| Hourly | `templates/hourly.md.tmpl` (`hourly`) | `hourly` | `weather.hourly_generated_text`; `prompts/hourly.generated_text.md` |
|
||||
| Daily | `templates/daily.md.tmpl` (`daily`) | `daily` | `weather.daily_generated_text`; `internal/promptassets/assets/prompts/daily/` |
|
||||
| Today | `templates/today.md.tmpl` (`today`) | `today` | `weather.today_generated_text`; `internal/promptassets/assets/prompts/today/` |
|
||||
| Tomorrow | `templates/tomorrow.md.tmpl` (`tomorrow`) | `tomorrow` | `weather.tomorrow_generated_text`; `internal/promptassets/assets/prompts/tomorrow/` |
|
||||
| Hourly | `templates/hourly.md.tmpl` (`hourly`) | `hourly` | `weather.hourly_generated_text`; `internal/promptassets/assets/prompts/hourly/` |
|
||||
|
||||
The matching schema files are under `internal/reporttemplate/schemas/`. The
|
||||
generated-text catalog pairs each schema ID with its template ID; keep the
|
||||
matching report prompt source aligned with that pair.
|
||||
The matching schemas and Promptkit definitions are embedded by
|
||||
`internal/promptassets`. The generated-text catalog pairs each schema ID with
|
||||
its template ID; keep the matching prompt definition aligned with that pair.
|
||||
|
||||
Shared partials are under `internal/reporttemplate/templates/partials/`:
|
||||
|
||||
|
||||
@@ -17,7 +17,7 @@ or required command and configuration values are absent or malformed.
|
||||
|
||||
Diagnostic: Compare the command with [`generate` and `run`](cli.md#commands-and-usage)
|
||||
and review the configured value named in the error. `generate daily` requires
|
||||
`--date`; `generate storm` requires both `--start` and `--end`.
|
||||
`--date`.
|
||||
|
||||
Safe fix: Correct only the reported option or configuration value. Use an
|
||||
absolute Weather API URL and a valid IANA timezone; do not change unrelated
|
||||
|
||||
Reference in New Issue
Block a user