weatherreporter/docs/roadmap/implementation.md

# Modular Data Package Implementation Roadmap

This roadmap is a staged implementation plan for
[`docs/roadmap/modules.md`](modules.md). It is future-work planning only. The
target audience is an LLM coding agent implementing each stage in order.

## Purpose

Implement a pre-release hard cutover from report-shaped briefing packages to a
module-oriented prompt package architecture:

```text
CollectedFacts -> DerivedFacts -> ModuleOutput
```

The implementation should produce YAML `data_package` artifacts with named
stanzas for Scriptorium prompts, persist JSON module snapshots for inspection
and Recent Changes, and make report composition configurable through ordered
module IDs plus typed module options.

## Source Roadmap

[`docs/roadmap/modules.md`](modules.md) is authoritative for the conceptual
policy, user intent, target prompt shape, boundaries, and acceptance criteria
for this refactor. This document is authoritative for implementation order,
stage scope, file/package guidance, and validation commands.

If this implementation plan appears to conflict with `modules.md`, stop and
reconcile the roadmap before changing code. Do not infer a different policy
from stage sequencing.

## Locked Decisions

- Do a clean break. Do not preserve old report-shaped briefing JSON as a
  compatibility layer.
- Introduce `internal/weatherdata` for normalized collected source types.
- Keep `internal/forecast` for forecast-specific derivation algorithms such as
  period selection, daily summaries, daypart grouping, and precipitation
  timing.
- Introduce an internal fact contract for `CollectedFacts` and `DerivedFacts`.
- Introduce a narrow module contract for module IDs, typed options, outputs,
  and snapshots.
- Persist both artifacts:
  - JSON module snapshots for state, inspection, and Recent Changes;
  - YAML prompt data packages passed to Scriptorium as `data_package`.
- Replace `weatherreporter inspect briefing` with
  `weatherreporter inspect modules`.
- Support typed module options from the first configurable composition pass.
- Use named YAML stanzas, not a generic array of module objects.
- Omit QPF fields until a real upstream QPF source is represented in
  `CollectedFacts`.
- Keep public generate/run command names, report IDs, prompt IDs, RunID format,
  managed Markdown report paths, and distributor upload source stable.
- Do not introduce plugins, dynamic loading, generic workflow engines, or
  module-owned upstream fetching.

## Target Packages

Implementation should converge on this package ownership:

- `internal/weatherdata`: normalized collected source facts, source metadata,
  source warnings, and broad weather data types.
- `internal/forecast`: deterministic forecast-specific algorithms over
  `weatherdata` types.
- `internal/facts`: `CollectedFacts`, `DerivedFacts`, and their builders.
- `internal/module`: stable module IDs, module output envelope, module snapshot
  shape, module config item shape, and shared option/output contracts that must
  be imported by both `internal/report` and `internal/briefing`.
- `internal/briefing`: module registry and module builders.
- `internal/report`: report definitions, valid periods, output identity,
  comparison strategy, and default module composition.
- `internal/config`: YAML config structs, defaults, loading, and validation for
  module composition overrides.
- `internal/promptinput`: YAML prompt package assembly, validation, and save
  behavior.
- `internal/changes`: structured comparison over module snapshots.
- `internal/state`: module snapshot paths, YAML data package paths, metadata
  links, and inspection loads.
- `internal/app`: orchestration only.

Avoid import cycles. In particular, `internal/report` may import
`internal/module` for module IDs, but `internal/module` must not import
`internal/report`.

## Target Artifacts

Use explicit schema versions:

- Module snapshot JSON: `weatherreporter.modules.v1`
- YAML prompt data package: `weatherreporter.data_package.v2`

Target workspace paths:

```text
workspace/
  snapshots/<artifact_group>/<valid_date>/<run_id>.modules.json
  snapshots/<artifact_group>/<valid_date>/<run_id>.metadata.json
  data-packages/<artifact_group>/<valid_date>/<run_id>.data_package.yaml
```

Metadata should link the module snapshot and YAML data package paths. Existing
metadata links for preflight, rendered report, source warnings, source hashes,
and distributor notification artifacts should remain.

## Target Module Defaults

Initial implemented default module IDs should cover current behavior without
QPF-specific fields:

- `metadata`
- `current_conditions`
- `derived_daily_summary`
- `derived_daypart_summaries`
- `precip_timing`
- `alert_digest`
- `area_forecast_discussion`
- `weather_story`
- `forecast_delta`
- `outdoor_windows`
- `weekend_planning`
- `storm_window_summary`

Default report composition should be declared in `internal/report`:

- Daily Today:
  `metadata`, `current_conditions`, `derived_daily_summary`,
  `derived_daypart_summaries`, `precip_timing`, `alert_digest`,
  `forecast_delta`, `area_forecast_discussion`, `weather_story`,
  `outdoor_windows`
- Daily Tomorrow:
  same as Daily Today, plus any tomorrow-planning module needed to preserve
  current tomorrow behavior.
- 3-Day:
  `metadata`, `current_conditions`, `derived_daypart_summaries`,
  `precip_timing`, `alert_digest`, `forecast_delta`,
  `area_forecast_discussion`, `weather_story`, `outdoor_windows`
- Weekend:
  `metadata`, `current_conditions`, `derived_daypart_summaries`,
  `precip_timing`, `alert_digest`, `area_forecast_discussion`,
  `weather_story`, `outdoor_windows`, `weekend_planning`
- Storm:
  `metadata`, `current_conditions`, `hourly_table`, `precip_timing`,
  `alert_digest`, `area_forecast_discussion`, `weather_story`,
  `storm_window_summary`

If preserving a current report behavior requires a narrower module, add a
specific module rather than keeping old report-shaped containers.

## Stage 1: Weatherdata Package Split

Goal: separate broad normalized weather data from forecast-specific derivation.

Files/packages to change:

- create `internal/weatherdata`;
- update `internal/forecast`;
- update `internal/adapters/weatherapi`;
- update packages that currently import normalized source types from
  `internal/forecast`.

Implementation guidance:

- Move normalized source/domain types out of `internal/forecast` when they are
  not forecast algorithms:
  - bundle/source metadata/warnings;
  - current conditions;
  - observation run types if present;
  - alert run and alert overlap source types;
  - forecast run and forecast period source types;
  - discussion and discussion section types;
  - weather story types.
- Keep deterministic derivation functions in `internal/forecast`.
- Update Weather API adapter return types to use `weatherdata.Bundle`.
- Keep JSON field names and Weather API fixture behavior unchanged.
- Do not change CLI behavior, artifact paths, or prompt input yet.

Acceptance criteria:

- Weather API adapter tests pass with `weatherdata` types.
- Forecast derivation tests pass using `weatherdata` inputs.
- No external adapter dependency types leak into `weatherdata`.
- Existing generated report behavior is unchanged at this stage.

Validation:

```bash
go test ./internal/weatherdata ./internal/forecast ./internal/adapters/weatherapi
go test ./internal/app ./internal/briefing ./internal/promptinput
```

This stage is suitable for one implementation prompt if kept mechanical.

## Stage 2: Fact Contracts

Goal: add explicit `CollectedFacts` and `DerivedFacts` contracts.

Files/packages to change:

- create `internal/facts`;
- update `internal/app`;
- update `internal/forecast` tests as needed.

Implementation guidance:

- Define `CollectedFacts` as normalized upstream facts collected once per
  report run.
- Define `DerivedFacts` as conservative, reusable, report-scoped
  transformations.
- Add builders:
  - `BuildCollected(bundle *weatherdata.Bundle) CollectedFacts`
  - `BuildDerived(req BuildDerivedRequest) (DerivedFacts, error)`
- `BuildDerivedRequest` should include the resolved report, timezone,
  configured dayparts, and `CollectedFacts`.
- `DerivedFacts` may include:
  - valid-period hourly periods;
  - valid-period narrative periods;
  - alert overlaps;
  - daily summaries;
  - daypart summaries where reusable;
  - precipitation timing if reused by multiple modules.
- Do not put prompt wording, prose strings, module-specific ranking, or
  one-off presentation decisions in `DerivedFacts`.
- Keep source provenance and warnings separate from ordinary fact access.

Acceptance criteria:

- `CollectedFacts` can be built once from a fetched bundle.
- `DerivedFacts` can be built for Daily, Tomorrow, 3-Day, Weekend, and Storm.
- Derived fact builders have tests for valid-period slicing, daypart grouping,
  alert overlaps, and missing optional sources.
- No module or prompt code exists yet that fetches upstream data.

Validation:

```bash
go test ./internal/facts ./internal/forecast ./internal/app
go test ./internal/...
```

This stage is suitable for one implementation prompt.

## Stage 3: Module Core Contracts

Goal: define module IDs, options, outputs, snapshots, and registry mechanics.

Files/packages to change:

- create `internal/module`;
- update `internal/report`;
- update `internal/briefing`.

Implementation guidance:

- Define:
  - `module.ID`;
  - module ID constants;
  - `module.ConfigItem`;
  - `module.Output`;
  - `module.Snapshot`;
  - shared schema version constants.
- `module.Output` should contain module ID, stanza name, and typed value.
- `module.Snapshot` should preserve ordered outputs and support typed stanza
  lookup for comparison code.
- Add duplicate module and duplicate stanza-name validation.
- Add typed option structs for initial modules. Empty option structs are fine
  for modules without options.
- Add a module registry in `internal/briefing` that maps module IDs to builder
  definitions.
- Module definitions should declare:
  - ID;
  - stanza name;
  - option type;
  - default options;
  - required collected facts;
  - required derived facts;
  - supported report IDs or report categories;
  - missing-data behavior.
- Do not execute modules from app orchestration yet unless needed for tests.

Acceptance criteria:

- Report definitions can refer to `module.ID` without import cycles.
- Module registry tests reject unknown modules, duplicate module IDs, duplicate
  stanza names, incompatible reports, and invalid option shapes.
- Module output and snapshot JSON marshal deterministically enough for tests.

Validation:

```bash
go test ./internal/module ./internal/briefing ./internal/report
```

This stage is suitable for one implementation prompt.

## Stage 4: Base Modules

Goal: implement source-oriented modules that mostly pass through normalized or
lightly selected facts.

Files/packages to change:

- `internal/briefing`;
- `internal/module`;
- tests under `internal/briefing`.

Implementation guidance:

- Implement these modules:
  - `metadata`;
  - `current_conditions`;
  - `alert_digest`;
  - `area_forecast_discussion`;
  - `weather_story`.
- The `metadata` module should expose report metadata, configured location,
  units, timezone, valid period, source warnings summary, and alert checked
  status where appropriate.
- `area_forecast_discussion` should expose key messages, short-term text, and
  long-term text when present.
- `weather_story` should expose structured story fields when present and omit
  the stanza when missing/suppressed by missing-source policy.
- `alert_digest` should distinguish checked/no-active-alerts from missing alert
  source data.
- Ordinary modules should not expose endpoint, hash, or transport provenance;
  provenance should remain metadata/source-warning oriented.

Acceptance criteria:

- Each module has focused tests for available data, missing optional data, and
  empty output omission.
- No module fetches upstream data or reads/writes durable state.
- Output field names use YAML-friendly snake_case and unit suffixes where
  needed.

Validation:

```bash
go test ./internal/briefing ./internal/module ./internal/facts
```

This stage is suitable for one implementation prompt.

## Stage 5: Derived Fact Modules

Goal: implement deterministic modules that package reusable forecast
derivations for the LLM.

Files/packages to change:

- `internal/forecast`;
- `internal/facts`;
- `internal/briefing`;
- `internal/module`.

Implementation guidance:

- Implement:
  - `derived_daily_summary`;
  - `derived_daypart_summaries`;
  - `precip_timing`;
  - `outdoor_windows`;
  - any tomorrow-planning module needed to preserve Tomorrow output quality.
- `derived_daily_summary` should include current implementable fields:
  - `high_temp_f`;
  - `low_temp_f`;
  - `max_pop_percent`;
  - `max_pop_window`;
  - `first_precip_hour`;
  - `last_precip_hour`;
  - `thunder_mentioned`;
  - `max_wind_gust_mph`;
  - `heat_index_max_f` when source data supports it.
- Do not implement `measurable_qpf_total_in` or `max_hourly_qpf_in` until QPF
  exists in `CollectedFacts`.
- `derived_daypart_summaries` should expose daypart keyed values using the
  configured daypart definitions.
- Keep broad reusable calculations in `DerivedFacts`; keep prompt-shape
  packaging inside modules.

Acceptance criteria:

- Derived modules have fixture coverage across ordinary, dry, rainy, windy,
  cold/heat, and missing-data scenarios.
- QPF fields are absent unless an upstream QPF source exists.
- Daily and Tomorrow module outputs contain enough data to replace current
  report-shaped daily briefing content.

Validation:

```bash
go test ./internal/forecast ./internal/facts ./internal/briefing ./internal/module
```

This stage may be too large for one prompt if all modules are implemented at
once. Split into Daily-derived modules first, then outlook/storm derived
modules if needed.

## Stage 6: Report Composition And Config Overrides

Goal: make report definitions and config the source of module composition.

Files/packages to change:

- `internal/report`;
- `internal/config`;
- `examples/config.yml`;
- config tests.

Implementation guidance:

- Extend `report.Definition` with default ordered module IDs.
- Keep valid-period resolution, prompt IDs, output naming, generated flag, and
  comparison strategy in `internal/report`.
- Add config support:

```yaml
reports:
  daily:
    deterministic_modules:
      - current_conditions
      - id: area_forecast_discussion
        options:
          sections:
            - short_term
```

- Support both string shorthand and object form for module entries.
- Normalize config into typed `module.ConfigItem` values.
- Decode module options into typed option structs during validation or before
  module execution.
- Reject:
  - unknown report IDs;
  - unknown module IDs;
  - duplicate modules unless explicitly allowed by that module;
  - duplicate stanza names;
  - incompatible report/module combinations;
  - invalid options.
- Built-in defaults should work when no report module config is present.
- Example config may omit module overrides unless an example is needed.

Acceptance criteria:

- Defaults reproduce intended module composition for all implemented reports.
- A config edit can add/remove an implemented module for a report.
- Invalid module config errors are actionable and do not mention raw internal
  panic/details.
- Config examples load.

Validation:

```bash
go test ./internal/report ./internal/config ./internal/briefing
go run ./cmd/weatherreporter --help
```

This stage is suitable for one implementation prompt.

## Stage 7: Module Snapshot State

Goal: persist and inspect JSON module snapshots without changing Scriptorium
input yet.

Files/packages to change:

- `internal/state`;
- `internal/app`;
- `internal/cli`;
- app/state/CLI tests.

Implementation guidance:

- Add state paths for `<run_id>.modules.json`.
- Add save/load methods for module snapshots.
- Update metadata to include `ModuleSnapshotPath`.
- Add `weatherreporter inspect modules [--config PATH] RUN_ID`.
- Remove `inspect briefing` from parser support and help text in this stage.
- Keep old data package generation in place only until Stage 8, but do not
  leave generation without a module snapshot.

Acceptance criteria:

- Generated runs persist module snapshots before prompt package construction.
- `inspect modules` returns the module snapshot.
- `inspect briefing` is gone from help text and parser tests.
- Metadata links the module snapshot path.
- Existing report generation still succeeds with fake Scriptorium.

Validation:

```bash
go test ./internal/state ./internal/cli ./internal/app
go run ./cmd/weatherreporter --help
```

This stage is suitable for one implementation prompt.

## Stage 8: YAML Prompt Package Cutover

Goal: replace JSON prompt data packages with YAML named-stanza data packages.

Files/packages to change:

- `internal/promptinput`;
- `internal/state`;
- `internal/adapters/scriptorium` tests;
- `internal/app`.

Implementation guidance:

- Set prompt package schema version to `weatherreporter.data_package.v2`.
- Build prompt package content from module snapshots, report metadata, recent
  changes, and source warnings.
- Save prompt packages as `.data_package.yaml`.
- Continue passing Scriptorium input as `--input data_package=<path>`.
- Update render/run tests to avoid assuming `.json` filenames.
- Ensure YAML uses named stanzas under `briefing`.
- Omit empty optional fields.
- Keep module snapshot JSON as the comparison/inspection source.
- Update metadata `DataPackagePath` to point to YAML.

Acceptance criteria:

- Scriptorium render and run receive a YAML `data_package` path.
- YAML output is deterministic enough for tests.
- `inspect data-package` returns YAML content or a parsed representation
  chosen consistently in CLI tests.
- No code assumes data package paths end in `.json`.

Validation:

```bash
go test ./internal/promptinput ./internal/adapters/scriptorium ./internal/state ./internal/app ./internal/cli
go test ./...
```

This stage is suitable for one implementation prompt.

## Stage 9: App Orchestration Cutover

Goal: make module execution the only generation path for all implemented
reports.

Files/packages to change:

- `internal/app`;
- `internal/briefing`;
- `internal/facts`;
- app workflow tests.

Implementation guidance:

- In `GenerateReport`, fetch Weather API data once, build `CollectedFacts`,
  build `DerivedFacts`, execute configured modules, save module snapshot, build
  YAML prompt package, then continue preflight/run/metadata/distributor flow.
- Preserve ordering:
  1. resolve prior comparable metadata;
  2. fetch bundle;
  3. build facts;
  4. execute modules;
  5. save module snapshot;
  6. compute Recent Changes;
  7. save YAML data package;
  8. run render preflight;
  9. save metadata;
  10. run Scriptorium;
  11. copy optional output;
  12. save final metadata;
  13. notify distributor if enabled.
- Do not use `--out` or `--out-dir` copies for distributor notification.
- Do not invoke modules after Scriptorium failures.
- Keep batch behavior unchanged: continue independent reports, return nonzero
  aggregate status if any report fails.

Acceptance criteria:

- Daily, Tomorrow, 3-Day, Weekend, and Storm generation all use module
  snapshots and YAML data packages.
- Existing public CLI syntax remains stable except `inspect modules` replacing
  `inspect briefing`.
- Managed Markdown report paths and distributor upload source remain stable.
- App tests assert generated module snapshots and YAML data packages.

Validation:

```bash
go test ./internal/app ./internal/cli ./internal/state ./internal/briefing ./internal/promptinput
go test ./...
```

This stage may be large. Split by report family if needed: Daily/Tomorrow,
Outlooks, then Storm.

## Stage 10: Recent Changes Migration

Goal: compare structured module snapshots instead of report-shaped briefing
packages.

Files/packages to change:

- `internal/changes`;
- `internal/state`;
- `internal/app`;
- changes tests.

Implementation guidance:

- Define which module stanzas each comparison strategy consumes.
- Daily comparison should use `derived_daily_summary`,
  `derived_daypart_summaries`, `alert_digest`, and `precip_timing` where
  present.
- 3-Day and Weekend comparisons should use module snapshot outputs that replace
  current outlook day comparisons.
- Storm comparison should remain explicit-window based and consume storm
  module outputs when implemented.
- Do not compare rendered Markdown or rendered YAML text.
- If a comparison-required module is missing, return an actionable error or an
  inspectable warning according to the report policy chosen in code. Prefer an
  error for required comparison modules and no-op only for optional comparison
  stanzas.

Acceptance criteria:

- Prior snapshot lookup still uses report compatibility and valid-period rules.
- Recent Changes output remains deterministic.
- Tests cover unchanged forecasts, threshold-crossing changes, alert changes,
  precip timing changes, and missing comparison stanzas.
- Old `briefing.Package` comparison code is removed.

Validation:

```bash
go test ./internal/changes ./internal/state ./internal/app
go test ./...
```

This stage is suitable for one implementation prompt if module snapshots are
already available.

## Stage 11: Remove Old Briefing Shapes

Goal: remove obsolete report-shaped briefing containers and stale JSON package
assumptions.

Files/packages to change:

- `internal/briefing`;
- `internal/promptinput`;
- `internal/state`;
- `internal/app`;
- tests throughout `internal`.

Implementation guidance:

- Remove old `Daily`, `ThreeDay`, `Weekend`, and `Storm` briefing container
  structs when no longer used.
- Remove old `briefing.Package` if it no longer represents the module
  snapshot. If the package keeps a `Package` type, it must be module-oriented.
- Remove tests that construct old report-shaped briefing fixtures.
- Remove stale `.data_package.json` assumptions.
- Remove dead helper functions that only supported old report-shaped output.
- Keep generated report Markdown behavior stable.

Acceptance criteria:

- `rg -n "data_package\\.json|inspect briefing|briefing\\.Package" internal docs -g '!docs/roadmap/**'`
  has no production-code matches, except deliberate roadmap/history references
  where appropriate.
- No old report-shaped content structs remain on the generation path.
- All tests pass.

Validation:

```bash
rg -n "data_package\\.json|inspect briefing|briefing\\.Package" internal docs -g '!docs/roadmap/**'
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
```

This stage is suitable for one implementation prompt.

## Stage 12: Documentation And Example Alignment

Goal: align non-roadmap docs with implemented module behavior.

Files to inspect/update:

- `README.md`, only if the orientation or quickstart changed;
- `docs/cli.md`;
- `docs/config.md`;
- `docs/operations.md`;
- `docs/troubleshooting.md`;
- `docs/internal/app-orchestration.md`;
- `docs/internal/briefing.md` or replacement module internals doc;
- `docs/internal/changes.md`;
- `docs/internal/forecast-derivation.md`;
- `docs/internal/prompt-input.md`;
- `docs/internal/state.md`;
- `docs/internal/weather-data.md`;
- `docs/integrations/scriptorium.md`;
- `examples/config.yml`.

Implementation guidance:

- Document only implemented behavior outside `docs/roadmap/`.
- Add or update an internal module contract document if module behavior is now
  implemented.
- Document `inspect modules` and remove `inspect briefing`.
- Document YAML data packages and JSON module snapshots.
- Document report module overrides and typed options only if implemented.
- Keep QPF as future-only unless upstream support was added.
- Keep Scriptorium contract focused on `--input data_package=<path>` and the
  actual file format now passed.

Acceptance criteria:

- Non-roadmap docs no longer describe old report-shaped briefing packages.
- Config examples load.
- CLI examples match `weatherreporter --help`.
- Docs clearly distinguish module snapshots from prompt data packages.

Validation:

```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
rg -n "inspect briefing|data_package\\.json|report-shaped|vars-file|promptvars" README.md docs examples internal -g '!docs/roadmap/**'
```

This stage is suitable for one implementation prompt.

## Stage 13: Final Validation

Goal: run full validation and catch stale assumptions after the cutover.

Required commands:

```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
```

Required grep checks:

```bash
rg -n "inspect briefing|data_package\\.json|briefing\\.Package|Daily struct|ThreeDay struct|Weekend struct|Storm struct" internal docs examples -g '!docs/roadmap/**'
rg -n "measurable_qpf_total_in|max_hourly_qpf_in" internal docs examples -g '!docs/roadmap/**'
```

Expected grep results:

- no production-code references to `inspect briefing`;
- no production-code assumption that prompt packages are JSON;
- no production-code dependence on old report-shaped briefing containers;
- QPF references appear only as future-target docs or omitted-field tests until
  upstream QPF exists.

Manual review:

- Generate command output still writes managed Markdown reports.
- Batch behavior still continues independent reports and returns nonzero on
  aggregate failure.
- Distributor notification still uploads the managed Markdown report, not
  module snapshots or YAML prompt packages.
- Secrets are not printed or persisted.
- YAML prompt package is readable and contains named stanzas.

## Deferred Work

Do not include these in the initial module cutover:

- plugin architecture;
- dynamic module loading;
- YAML-defined module schemas;
- user-authored module code;
- module-owned Weather API fetching;
- QPF fields before upstream QPF exists;
- SPC modules before upstream SPC data exists;
- radar modules before upstream radar data exists;
- event/storm-review reports unless a separate roadmap implements them.

## Open Questions

No blocking open questions remain for this implementation plan. The previously
identified choices are locked above:

- persist JSON module snapshots and YAML prompt data packages;
- split broad normalized source types into `internal/weatherdata`;
- replace `inspect briefing` with `inspect modules`;
- support typed module options from the first config implementation.