weatherreporter/docs/roadmap/implementation.md

Modular Data Package Implementation Roadmap

This roadmap is a staged implementation plan for docs/roadmap/modules.md. It is future-work planning only. The target audience is an LLM coding agent implementing each stage in order.

Purpose

Implement a pre-release hard cutover from report-shaped briefing packages to a module-oriented prompt package architecture:

CollectedFacts -> DerivedFacts -> ModuleOutput

The implementation should produce YAML data_package artifacts with named stanzas for Scriptorium prompts, persist JSON module snapshots for inspection and Recent Changes, and make report composition configurable through ordered module IDs plus typed module options.

Source Roadmap

docs/roadmap/modules.md is authoritative for the conceptual policy, user intent, target prompt shape, boundaries, and acceptance criteria for this refactor. This document is authoritative for implementation order, stage scope, file/package guidance, and validation commands.

If this implementation plan appears to conflict with modules.md, stop and reconcile the roadmap before changing code. Do not infer a different policy from stage sequencing.

Locked Decisions

• Do a clean break. Do not preserve old report-shaped briefing JSON as a compatibility layer.
• Introduce internal/weatherdata for normalized collected source types.
• Keep internal/forecast for forecast-specific derivation algorithms such as period selection, daily summaries, daypart grouping, and precipitation timing.
• Introduce an internal fact contract for CollectedFacts and DerivedFacts.
• Introduce a narrow module contract for module IDs, typed options, outputs, and snapshots.
• Persist both artifacts:
  • JSON module snapshots for state, inspection, and Recent Changes;
  • YAML prompt data packages passed to Scriptorium as data_package.
• Replace weatherreporter inspect briefing with weatherreporter inspect modules.
• Support typed module options from the first configurable composition pass.
• Use named YAML stanzas, not a generic array of module objects.
• Omit QPF fields until a real upstream QPF source is represented in CollectedFacts.
• Keep public generate/run command names, report IDs, prompt IDs, RunID format, managed Markdown report paths, and distributor upload source stable.
• Do not introduce plugins, dynamic loading, generic workflow engines, or module-owned upstream fetching.

Target Packages

Implementation should converge on this package ownership:

• internal/weatherdata: normalized collected source facts, source metadata, source warnings, and broad weather data types.
• internal/forecast: deterministic forecast-specific algorithms over weatherdata types.
• internal/facts: CollectedFacts, DerivedFacts, and their builders.
• internal/module: stable module IDs, module output envelope, module snapshot shape, module config item shape, and shared option/output contracts that must be imported by both internal/report and internal/briefing.
• internal/briefing: module registry and module builders.
• internal/report: report definitions, valid periods, output identity, comparison strategy, and default module composition.
• internal/config: YAML config structs, defaults, loading, and validation for module composition overrides.
• internal/promptinput: YAML prompt package assembly, validation, and save behavior.
• internal/changes: structured comparison over module snapshots.
• internal/state: module snapshot paths, YAML data package paths, metadata links, and inspection loads.
• internal/app: orchestration only.

Avoid import cycles. In particular, internal/report may import internal/module for module IDs, but internal/module must not import internal/report.

Target Artifacts

Use explicit schema versions:

• Module snapshot JSON: weatherreporter.modules.v1
• YAML prompt data package: weatherreporter.data_package.v2

Target workspace paths:

workspace/
  snapshots/<artifact_group>/<valid_date>/<run_id>.modules.json
  snapshots/<artifact_group>/<valid_date>/<run_id>.metadata.json
  data-packages/<artifact_group>/<valid_date>/<run_id>.data_package.yaml

Metadata should link the module snapshot and YAML data package paths. Existing metadata links for preflight, rendered report, source warnings, source hashes, and distributor notification artifacts should remain.

Target Module Defaults

Initial implemented default module IDs should cover current behavior without QPF-specific fields:

• metadata
• current_conditions
• derived_daily_summary
• derived_daypart_summaries
• precip_timing
• alert_digest
• area_forecast_discussion
• weather_story
• forecast_delta
• outdoor_windows
• weekend_planning
• storm_window_summary

Default report composition should be declared in internal/report:

• Daily Today: metadata, current_conditions, derived_daily_summary, derived_daypart_summaries, precip_timing, alert_digest, forecast_delta, area_forecast_discussion, weather_story, outdoor_windows
• Daily Tomorrow: same as Daily Today, plus any tomorrow-planning module needed to preserve current tomorrow behavior.
• 3-Day: metadata, current_conditions, derived_daypart_summaries, precip_timing, alert_digest, forecast_delta, area_forecast_discussion, weather_story, outdoor_windows
• Weekend: metadata, current_conditions, derived_daypart_summaries, precip_timing, alert_digest, area_forecast_discussion, weather_story, outdoor_windows, weekend_planning
• Storm: metadata, current_conditions, hourly_table, precip_timing, alert_digest, area_forecast_discussion, weather_story, storm_window_summary

If preserving a current report behavior requires a narrower module, add a specific module rather than keeping old report-shaped containers.

Stage 1: Weatherdata Package Split

Goal: separate broad normalized weather data from forecast-specific derivation.

Files/packages to change:

• create internal/weatherdata;
• update internal/forecast;
• update internal/adapters/weatherapi;
• update packages that currently import normalized source types from internal/forecast.

Implementation guidance:

• Move normalized source/domain types out of internal/forecast when they are not forecast algorithms:
  • bundle/source metadata/warnings;
  • current conditions;
  • observation run types if present;
  • alert run and alert overlap source types;
  • forecast run and forecast period source types;
  • discussion and discussion section types;
  • weather story types.
• Keep deterministic derivation functions in internal/forecast.
• Update Weather API adapter return types to use weatherdata.Bundle.
• Keep JSON field names and Weather API fixture behavior unchanged.
• Do not change CLI behavior, artifact paths, or prompt input yet.

Acceptance criteria:

• Weather API adapter tests pass with weatherdata types.
• Forecast derivation tests pass using weatherdata inputs.
• No external adapter dependency types leak into weatherdata.
• Existing generated report behavior is unchanged at this stage.

Validation:

go test ./internal/weatherdata ./internal/forecast ./internal/adapters/weatherapi
go test ./internal/app ./internal/briefing ./internal/promptinput

This stage is suitable for one implementation prompt if kept mechanical.

Stage 2: Fact Contracts

Goal: add explicit CollectedFacts and DerivedFacts contracts.

Files/packages to change:

• create internal/facts;
• update internal/app;
• update internal/forecast tests as needed.

Implementation guidance:

• Define CollectedFacts as normalized upstream facts collected once per report run.
• Define DerivedFacts as conservative, reusable, report-scoped transformations.
• Add builders:
  • BuildCollected(bundle *weatherdata.Bundle) CollectedFacts
  • BuildDerived(req BuildDerivedRequest) (DerivedFacts, error)
• BuildDerivedRequest should include the resolved report, timezone, configured dayparts, and CollectedFacts.
• DerivedFacts may include:
  • valid-period hourly periods;
  • valid-period narrative periods;
  • alert overlaps;
  • daily summaries;
  • daypart summaries where reusable;
  • precipitation timing if reused by multiple modules.
• Do not put prompt wording, prose strings, module-specific ranking, or one-off presentation decisions in DerivedFacts.
• Keep source provenance and warnings separate from ordinary fact access.

Acceptance criteria:

• CollectedFacts can be built once from a fetched bundle.
• DerivedFacts can be built for Daily, Tomorrow, 3-Day, Weekend, and Storm.
• Derived fact builders have tests for valid-period slicing, daypart grouping, alert overlaps, and missing optional sources.
• No module or prompt code exists yet that fetches upstream data.

Validation:

go test ./internal/facts ./internal/forecast ./internal/app
go test ./internal/...

This stage is suitable for one implementation prompt.

Stage 3: Module Core Contracts

Goal: define module IDs, options, outputs, snapshots, and registry mechanics.

Files/packages to change:

• create internal/module;
• update internal/report;
• update internal/briefing.

Implementation guidance:

• Define:
  • module.ID;
  • module ID constants;
  • module.ConfigItem;
  • module.Output;
  • module.Snapshot;
  • shared schema version constants.
• module.Output should contain module ID, stanza name, and typed value.
• module.Snapshot should preserve ordered outputs and support typed stanza lookup for comparison code.
• Add duplicate module and duplicate stanza-name validation.
• Add typed option structs for initial modules. Empty option structs are fine for modules without options.
• Add a module registry in internal/briefing that maps module IDs to builder definitions.
• Module definitions should declare:
  • ID;
  • stanza name;
  • option type;
  • default options;
  • required collected facts;
  • required derived facts;
  • supported report IDs or report categories;
  • missing-data behavior.
• Do not execute modules from app orchestration yet unless needed for tests.

Acceptance criteria:

• Report definitions can refer to module.ID without import cycles.
• Module registry tests reject unknown modules, duplicate module IDs, duplicate stanza names, incompatible reports, and invalid option shapes.
• Module output and snapshot JSON marshal deterministically enough for tests.

Validation:

go test ./internal/module ./internal/briefing ./internal/report

This stage is suitable for one implementation prompt.

Stage 4: Base Modules

Goal: implement source-oriented modules that mostly pass through normalized or lightly selected facts.

Files/packages to change:

• internal/briefing;
• internal/module;
• tests under internal/briefing.

Implementation guidance:

• Implement these modules:
  • metadata;
  • current_conditions;
  • alert_digest;
  • area_forecast_discussion;
  • weather_story.
• The metadata module should expose report metadata, configured location, units, timezone, valid period, source warnings summary, and alert checked status where appropriate.
• area_forecast_discussion should expose key messages, short-term text, and long-term text when present.
• weather_story should expose structured story fields when present and omit the stanza when missing/suppressed by missing-source policy.
• alert_digest should distinguish checked/no-active-alerts from missing alert source data.
• Ordinary modules should not expose endpoint, hash, or transport provenance; provenance should remain metadata/source-warning oriented.

Acceptance criteria:

• Each module has focused tests for available data, missing optional data, and empty output omission.
• No module fetches upstream data or reads/writes durable state.
• Output field names use YAML-friendly snake_case and unit suffixes where needed.

Validation:

go test ./internal/briefing ./internal/module ./internal/facts

This stage is suitable for one implementation prompt.

Stage 5: Derived Fact Modules

Goal: implement deterministic modules that package reusable forecast derivations for the LLM.

Files/packages to change:

• internal/forecast;
• internal/facts;
• internal/briefing;
• internal/module.

Implementation guidance:

• Implement:
  • derived_daily_summary;
  • derived_daypart_summaries;
  • precip_timing;
  • outdoor_windows;
  • any tomorrow-planning module needed to preserve Tomorrow output quality.
• derived_daily_summary should include current implementable fields:
  • high_temp_f;
  • low_temp_f;
  • max_pop_percent;
  • max_pop_window;
  • first_precip_hour;
  • last_precip_hour;
  • thunder_mentioned;
  • max_wind_gust_mph;
  • heat_index_max_f when source data supports it.
• Do not implement measurable_qpf_total_in or max_hourly_qpf_in until QPF exists in CollectedFacts.
• derived_daypart_summaries should expose daypart keyed values using the configured daypart definitions.
• Keep broad reusable calculations in DerivedFacts; keep prompt-shape packaging inside modules.

Acceptance criteria:

• Derived modules have fixture coverage across ordinary, dry, rainy, windy, cold/heat, and missing-data scenarios.
• QPF fields are absent unless an upstream QPF source exists.
• Daily and Tomorrow module outputs contain enough data to replace current report-shaped daily briefing content.

Validation:

go test ./internal/forecast ./internal/facts ./internal/briefing ./internal/module

This stage may be too large for one prompt if all modules are implemented at once. Split into Daily-derived modules first, then outlook/storm derived modules if needed.

Stage 6: Report Composition And Config Overrides

Goal: make report definitions and config the source of module composition.

Files/packages to change:

• internal/report;
• internal/config;
• examples/config.yml;
• config tests.

Implementation guidance:

• Extend report.Definition with default ordered module IDs.
• Keep valid-period resolution, prompt IDs, output naming, generated flag, and comparison strategy in internal/report.
• Add config support:

reports:
  daily:
    deterministic_modules:
      - current_conditions
      - id: area_forecast_discussion
        options:
          sections:
            - short_term

• Support both string shorthand and object form for module entries.
• Normalize config into typed module.ConfigItem values.
• Decode module options into typed option structs during validation or before module execution.
• Reject:
  • unknown report IDs;
  • unknown module IDs;
  • duplicate modules unless explicitly allowed by that module;
  • duplicate stanza names;
  • incompatible report/module combinations;
  • invalid options.
• Built-in defaults should work when no report module config is present.
• Example config may omit module overrides unless an example is needed.

Acceptance criteria:

• Defaults reproduce intended module composition for all implemented reports.
• A config edit can add/remove an implemented module for a report.
• Invalid module config errors are actionable and do not mention raw internal panic/details.
• Config examples load.

Validation:

go test ./internal/report ./internal/config ./internal/briefing
go run ./cmd/weatherreporter --help

This stage is suitable for one implementation prompt.

Stage 7: Module Snapshot State

Goal: persist and inspect JSON module snapshots without changing Scriptorium input yet.

Files/packages to change:

• internal/state;
• internal/app;
• internal/cli;
• app/state/CLI tests.

Implementation guidance:

• Add state paths for <run_id>.modules.json.
• Add save/load methods for module snapshots.
• Update metadata to include ModuleSnapshotPath.
• Add weatherreporter inspect modules [--config PATH] RUN_ID.
• Remove inspect briefing from parser support and help text in this stage.
• Keep old data package generation in place only until Stage 8, but do not leave generation without a module snapshot.

Acceptance criteria:

• Generated runs persist module snapshots before prompt package construction.
• inspect modules returns the module snapshot.
• inspect briefing is gone from help text and parser tests.
• Metadata links the module snapshot path.
• Existing report generation still succeeds with fake Scriptorium.

Validation:

go test ./internal/state ./internal/cli ./internal/app
go run ./cmd/weatherreporter --help

This stage is suitable for one implementation prompt.

Stage 8: YAML Prompt Package Cutover

Goal: replace JSON prompt data packages with YAML named-stanza data packages.

Files/packages to change:

• internal/promptinput;
• internal/state;
• internal/adapters/scriptorium tests;
• internal/app.

Implementation guidance:

• Set prompt package schema version to weatherreporter.data_package.v2.
• Build prompt package content from module snapshots, report metadata, recent changes, and source warnings.
• Save prompt packages as .data_package.yaml.
• Continue passing Scriptorium input as --input data_package=<path>.
• Update render/run tests to avoid assuming .json filenames.
• Ensure YAML uses named stanzas under briefing.
• Omit empty optional fields.
• Keep module snapshot JSON as the comparison/inspection source.
• Update metadata DataPackagePath to point to YAML.

Acceptance criteria:

• Scriptorium render and run receive a YAML data_package path.
• YAML output is deterministic enough for tests.
• inspect data-package returns YAML content or a parsed representation chosen consistently in CLI tests.
• No code assumes data package paths end in .json.

Validation:

go test ./internal/promptinput ./internal/adapters/scriptorium ./internal/state ./internal/app ./internal/cli
go test ./...

This stage is suitable for one implementation prompt.

Stage 9: App Orchestration Cutover

Goal: make module execution the only generation path for all implemented reports.

Files/packages to change:

• internal/app;
• internal/briefing;
• internal/facts;
• app workflow tests.

Implementation guidance:

• In GenerateReport, fetch Weather API data once, build CollectedFacts, build DerivedFacts, execute configured modules, save module snapshot, build YAML prompt package, then continue preflight/run/metadata/distributor flow.
• Preserve ordering:
  1. resolve prior comparable metadata;
  2. fetch bundle;
  3. build facts;
  4. execute modules;
  5. save module snapshot;
  6. compute Recent Changes;
  7. save YAML data package;
  8. run render preflight;
  9. save metadata;
  10. run Scriptorium;
  11. copy optional output;
  12. save final metadata;
  13. notify distributor if enabled.
• Do not use --out or --out-dir copies for distributor notification.
• Do not invoke modules after Scriptorium failures.
• Keep batch behavior unchanged: continue independent reports, return nonzero aggregate status if any report fails.

Acceptance criteria:

• Daily, Tomorrow, 3-Day, Weekend, and Storm generation all use module snapshots and YAML data packages.
• Existing public CLI syntax remains stable except inspect modules replacing inspect briefing.
• Managed Markdown report paths and distributor upload source remain stable.
• App tests assert generated module snapshots and YAML data packages.

Validation:

go test ./internal/app ./internal/cli ./internal/state ./internal/briefing ./internal/promptinput
go test ./...

This stage may be large. Split by report family if needed: Daily/Tomorrow, Outlooks, then Storm.

Stage 10: Recent Changes Migration

Goal: compare structured module snapshots instead of report-shaped briefing packages.

Files/packages to change:

• internal/changes;
• internal/state;
• internal/app;
• changes tests.

Implementation guidance:

• Define which module stanzas each comparison strategy consumes.
• Daily comparison should use derived_daily_summary, derived_daypart_summaries, alert_digest, and precip_timing where present.
• 3-Day and Weekend comparisons should use module snapshot outputs that replace current outlook day comparisons.
• Storm comparison should remain explicit-window based and consume storm module outputs when implemented.
• Do not compare rendered Markdown or rendered YAML text.
• If a comparison-required module is missing, return an actionable error or an inspectable warning according to the report policy chosen in code. Prefer an error for required comparison modules and no-op only for optional comparison stanzas.

Acceptance criteria:

• Prior snapshot lookup still uses report compatibility and valid-period rules.
• Recent Changes output remains deterministic.
• Tests cover unchanged forecasts, threshold-crossing changes, alert changes, precip timing changes, and missing comparison stanzas.
• Old briefing.Package comparison code is removed.

Validation:

go test ./internal/changes ./internal/state ./internal/app
go test ./...

This stage is suitable for one implementation prompt if module snapshots are already available.

Stage 11: Remove Old Briefing Shapes

Goal: remove obsolete report-shaped briefing containers and stale JSON package assumptions.

Files/packages to change:

• internal/briefing;
• internal/promptinput;
• internal/state;
• internal/app;
• tests throughout internal.

Implementation guidance:

• Remove old Daily, ThreeDay, Weekend, and Storm briefing container structs when no longer used.
• Remove old briefing.Package if it no longer represents the module snapshot. If the package keeps a Package type, it must be module-oriented.
• Remove tests that construct old report-shaped briefing fixtures.
• Remove stale .data_package.json assumptions.
• Remove dead helper functions that only supported old report-shaped output.
• Keep generated report Markdown behavior stable.

Acceptance criteria:

• rg -n "data_package\\.json|inspect briefing|briefing\\.Package" internal docs -g '!docs/roadmap/**' has no production-code matches, except deliberate roadmap/history references where appropriate.
• No old report-shaped content structs remain on the generation path.
• All tests pass.

Validation:

rg -n "data_package\\.json|inspect briefing|briefing\\.Package" internal docs -g '!docs/roadmap/**'
go test ./...
go run ./cmd/weatherreporter --help
git diff --check

This stage is suitable for one implementation prompt.

Stage 12: Documentation And Example Alignment

Goal: align non-roadmap docs with implemented module behavior.

Files to inspect/update:

• README.md, only if the orientation or quickstart changed;
• docs/cli.md;
• docs/config.md;
• docs/operations.md;
• docs/troubleshooting.md;
• docs/internal/app-orchestration.md;
• docs/internal/briefing.md or replacement module internals doc;
• docs/internal/changes.md;
• docs/internal/forecast-derivation.md;
• docs/internal/prompt-input.md;
• docs/internal/state.md;
• docs/internal/weather-data.md;
• docs/integrations/scriptorium.md;
• examples/config.yml.

Implementation guidance:

• Document only implemented behavior outside docs/roadmap/.
• Add or update an internal module contract document if module behavior is now implemented.
• Document inspect modules and remove inspect briefing.
• Document YAML data packages and JSON module snapshots.
• Document report module overrides and typed options only if implemented.
• Keep QPF as future-only unless upstream support was added.
• Keep Scriptorium contract focused on --input data_package=<path> and the actual file format now passed.

Acceptance criteria:

• Non-roadmap docs no longer describe old report-shaped briefing packages.
• Config examples load.
• CLI examples match weatherreporter --help.
• Docs clearly distinguish module snapshots from prompt data packages.

Validation:

go test ./...
go run ./cmd/weatherreporter --help
git diff --check
rg -n "inspect briefing|data_package\\.json|report-shaped|vars-file|promptvars" README.md docs examples internal -g '!docs/roadmap/**'

This stage is suitable for one implementation prompt.

Stage 13: Final Validation

Goal: run full validation and catch stale assumptions after the cutover.

Required commands:

go test ./...
go run ./cmd/weatherreporter --help
git diff --check

Required grep checks:

rg -n "inspect briefing|data_package\\.json|briefing\\.Package|Daily struct|ThreeDay struct|Weekend struct|Storm struct" internal docs examples -g '!docs/roadmap/**'
rg -n "measurable_qpf_total_in|max_hourly_qpf_in" internal docs examples -g '!docs/roadmap/**'

Expected grep results:

• no production-code references to inspect briefing;
• no production-code assumption that prompt packages are JSON;
• no production-code dependence on old report-shaped briefing containers;
• QPF references appear only as future-target docs or omitted-field tests until upstream QPF exists.

Manual review:

• Generate command output still writes managed Markdown reports.
• Batch behavior still continues independent reports and returns nonzero on aggregate failure.
• Distributor notification still uploads the managed Markdown report, not module snapshots or YAML prompt packages.
• Secrets are not printed or persisted.
• YAML prompt package is readable and contains named stanzas.

Deferred Work

Do not include these in the initial module cutover:

• plugin architecture;
• dynamic module loading;
• YAML-defined module schemas;
• user-authored module code;
• module-owned Weather API fetching;
• QPF fields before upstream QPF exists;
• SPC modules before upstream SPC data exists;
• radar modules before upstream radar data exists;
• event/storm-review reports unless a separate roadmap implements them.

Open Questions

No blocking open questions remain for this implementation plan. The previously identified choices are locked above:

• persist JSON module snapshots and YAML prompt data packages;
• split broad normalized source types into internal/weatherdata;
• replace inspect briefing with inspect modules;
• support typed module options from the first config implementation.