# Modular Data Package Roadmap This roadmap describes planned refactoring work that is not implemented. Current behavior is documented outside `docs/roadmap/`. ## Purpose Move weatherreporter toward deterministic, reusable briefing modules that can be composed per report type. The goal is to make prompt input easier for the LLM to understand, easier for operators to inspect, and easier for developers to change without touching a cross-cutting set of report-builder files. The target outcome is a prompt-facing YAML data package with named stanzas. Each stanza should be built by a self-contained module that derives clear, deterministic facts from normalized forecast inputs. Reports should choose modules by ordered module IDs, so experimenting with a report can be as small as changing one configuration line, plus any matching prompt change outside weatherreporter. The internal target shape is: ```text CollectedFacts -> DerivedFacts -> ModuleOutput ``` Each arrow should be a stable internal contract. `DerivedFacts` should not need to know how `CollectedFacts` were collected. Modules should not need to know the provenance of any collected or derived fact they consume. Report building should not need to know how a module sourced its underlying facts or calculated its output. ## Intent And Context The current application already curates source data before passing it to the LLM. This refactor should strengthen that design. Modules should not expose raw source complexity merely because it is available. They should compute and package the facts the LLM should not have to infer from raw hourly periods, alerts, narrative periods, forecast discussions, or weather stories. The desired module behavior is deterministic. A module should answer a narrow question such as: - what are the current conditions; - what are the key daily forecast facts; - what are the daypart summaries; - when is precipitation most likely; - which alerts overlap the report period; - what short-term AFD text is relevant; - what weather story text is relevant. The primary maintainability goal is local reasoning. For example, updating the derived daily summary should mostly involve one module implementation and its tests. Adding AFD short-term text to a future `next_six_hours` report should be a report composition change, not a copy/paste change across multiple builders. ## Target Prompt Shape The target prompt-facing data package should be YAML with named stanzas under `briefing`. Named stanzas are preferred over an array of generic module objects because they are easier to read, inspect, and reference in prompts. Example target shape: ```yaml report: id: daily_today prompt_id: weather.daily_report generated_at: 2026-06-09T07:15:00-05:00 timezone: America/Chicago current_local_date: 2026-06-09 valid_period: start: 2026-06-09T00:00:00-05:00 end: 2026-06-10T00:00:00-05:00 briefing: metadata: location: id: home name: Brentwood region: St. Louis Metro timezone: America/Chicago current_conditions: condition_text: Partly cloudy temperature_f: 74 apparent_temperature_f: 76 dewpoint_f: 66 relative_humidity_percent: 71 wind_speed_mph: 8 wind_direction_degrees: 190 derived_daily_summary: high_temp_f: 86 low_temp_f: 68 max_pop_percent: 70 max_pop_window: "2 PM-6 PM" measurable_qpf_total_in: 0.35 max_hourly_qpf_in: 0.12 first_precip_hour: "1 PM" last_precip_hour: "8 PM" thunder_mentioned: true max_wind_gust_mph: 28 heat_index_max_f: 91 derived_daypart_summaries: morning: temp_range_f: "70-78" max_pop_percent: 20 dominant_condition: Partly sunny afternoon: temp_range_f: "82-86" max_pop_percent: 70 dominant_condition: Showers and thunderstorms likely alert_digest: checked: true active_count: 0 relevant_count: 0 area_forecast_discussion: key_messages: - Scattered storms are possible this afternoon. short_term: Showers and storms increase during the afternoon. long_term: Periodic rain chances continue into the weekend. weather_story: available: true title: Several Chances for Rain Through Monday description: Scattered showers and thunderstorms remain possible. recent_changes: items: [] ``` Field names should include units where the unit is not obvious: `high_temp_f`, `max_pop_percent`, `measurable_qpf_total_in`, `max_wind_gust_mph`, and similar names are preferred over ambiguous generic names. Time and range strings should be formatted for prompt readability, while machine-oriented timestamps should remain available in report metadata. The QPF fields in the example are target output fields for a future upstream source. They should not be treated as immediately implementable from the current weatherfeeder-backed `CollectedFacts` sources. Until an upstream QPF source exists, QPF fields should be omitted rather than fabricated from precipitation probability or narrative text. ## Architecture Target Keep the current package boundaries: - `internal/forecast` owns normalized source data, deterministic forecast derivation, period slicing, daypart grouping, and weather-signal calculations. - `internal/briefing` owns prompt-facing module builders and module output schemas. - `internal/report` owns report identity, prompt ID, valid-period resolution, output naming, comparison strategy, and default module composition. - `internal/config` owns optional report module composition overrides. - `internal/promptinput` owns final data-package assembly, validation, and prompt-facing serialization. - `internal/app` remains orchestration: resolve report, fetch sources, build module context, execute configured modules, persist artifacts, run Scriptorium, and notify distributor. Do not move source fetching, subprocess execution, distributor upload behavior, or raw external dependency types into module code. ## Layered Fact Contracts Introduce explicit internal contracts for three layers: 1. `CollectedFacts` 2. `DerivedFacts` 3. `ModuleOutput` `CollectedFacts` are normalized upstream inputs collected once per report run. They should be broad and source-oriented, but not tied to Weather API transport details. Examples include current conditions, observations, hourly forecast runs, narrative forecast runs, active alerts, AFD discussion, weather story, future radar inputs, and future historical observation totals. `DerivedFacts` are reusable, report-scoped deterministic products calculated from `CollectedFacts`. They may slice, combine, group, or summarize collected facts when the result is broadly useful to more than one module or needed for consistent behavior across modules. Examples include valid-period hourly periods, valid-period narrative periods, alert overlaps, daily summaries, configured daypart summaries, and reusable precipitation timing windows. `ModuleOutput` is the prompt-facing output contract produced by one module. A module may pass through raw-ish facts, such as AFD text, or expose derived facts, such as daily summary fields. In both cases, the module owns the named stanza shape and should produce stable, readable, unit-explicit prompt fields. Unless implementation discovers a strong reason otherwise, the internal contract for accessing `CollectedFacts` and `DerivedFacts` should have the same shape: - typed Go structs with named fields; - nil pointers, empty slices, or zero values to represent absent facts; - no `map[string]any` or string-keyed fact lookup as the primary API; - immutable-by-convention values once passed to modules; - helper methods only for repeated access patterns that would otherwise be error-prone; - source provenance and warnings stored separately from the primary fact values, available to metadata/source-warning modules but not required by ordinary modules. Illustrative shape: ```go type CollectedFacts struct { Current *weatherdata.CurrentConditions Observations *weatherdata.ObservationRun Alerts *weatherdata.AlertRun Hourly *weatherdata.ForecastRun Narrative *weatherdata.ForecastRun Discussion *weatherdata.Discussion WeatherStory *weatherdata.WeatherStory // Future: radar, historical observations, snow/rain totals, etc. } type DerivedFacts struct { HourlyPeriods []weatherdata.ForecastPeriod NarrativePeriods []weatherdata.ForecastPeriod AlertOverlaps []weatherdata.AlertOverlap DailySummaries []forecast.DailySummary DaypartSummaries []forecast.DaypartSummary PrecipTiming *forecast.PrecipTiming } type ModuleContext struct { Report report.Resolved Collected CollectedFacts Derived DerivedFacts Units string Timezone string Location *LocationContext } ``` The exact package names may differ during implementation. The important boundary is semantic: `CollectedFacts` represent upstream facts after normalization; `DerivedFacts` represent reusable report-scoped transformations; modules represent prompt-facing stanza construction. ### DerivedFacts Boundary Be conservative about what belongs in `DerivedFacts`. Add a value to this layer only when it is: - deterministic; - report-scoped; - reusable by multiple modules or needed to keep modules consistent; - independent of prompt wording and presentation decisions. `DerivedFacts` may: - slice source periods to the report valid period; - group hourly data into configured dayparts; - compute reusable summaries; - compute alert overlaps; - normalize repeated time-window selections. `DerivedFacts` should not: - decide prompt-facing wording; - decide which facts are important for one module only; - format prose-like strings for the LLM; - fetch upstream data; - write artifacts; - depend on Scriptorium or distributor. Module-specific calculations should remain inside the module when they are presentation-specific, used by only one module, or likely to change while tuning prompt behavior. ## Module Model Introduce a typed module model rather than generic maps. A module should have: - stable module ID; - self-contained output struct; - one focused builder function; - fixture or unit tests near the module; - declared input requirements, such as hourly forecast, alerts, discussion, or weather story; - deterministic handling for missing optional source data; - prompt-facing field names that are stable and unit-explicit. The implementation may use a simple function registry rather than a broad interface if that is enough: ```go type ModuleID string type ModuleBuilder func(ModuleContext) (ModuleOutput, error) ``` `ModuleOutput` should include the stable module ID, the YAML stanza name, and a typed value owned by the module: ```go type ModuleOutput struct { ID ModuleID StanzaName string Value any } ``` The module registry should preserve output order from report composition, but the serialized YAML should use named stanzas for clarity. Each module should be able to produce exactly one named stanza. If one source can usefully feed multiple stanzas, split that into multiple modules rather than making one module produce unrelated output. ## Report Composition Target Report definitions should declare default ordered module IDs. Configuration may override the ordered module list for implemented reports. Illustrative future config shape: ```yaml reports: next_6_hours: deterministic_modules: - hourly_table - precip_timing - alert_digest - afd_short_term_text - weather_story_text - spc_products daily: deterministic_modules: - current_conditions - derived_daily_summary - derived_daypart_summaries - precip_timing - alert_digest - forecast_delta - afd_short_term_text - weather_story_text - spc_products ``` Configuration should validate unknown module IDs, duplicate module IDs when duplicates are not meaningful, and modules that are incompatible with the selected report period. Defaults should remain in Go so the application works without report composition config. ## Clean-Break Cutover Policy This project is still pre-release. Prefer a direct cutover to the new internal shape instead of preserving transitional report-shaped briefing structures. Implementation should: - replace report-shaped briefing containers with module-oriented snapshots; - replace JSON prompt package output with YAML prompt package output; - update inspect commands, Recent Changes, tests, and docs in the same cutover; - remove obsolete `Daily`, `ThreeDay`, `Weekend`, and `Storm` briefing container shapes when no longer needed; - avoid compatibility aliases unless they materially reduce implementation risk inside one stage. The public CLI command names, report IDs, prompt IDs, RunID format, managed Markdown report paths, and distributor upload source should remain stable unless a separate roadmap explicitly changes them. ## Artifact And State Target The durable artifacts should reflect the new module-oriented model. Recommended target: - module snapshot artifact: structured JSON for stable inspection, state lookup, and Recent Changes comparisons; - prompt data package artifact: YAML with named stanzas, passed to Scriptorium as `data_package`; - metadata artifact: JSON linking the module snapshot, YAML data package, preflight output, rendered report, source warnings, source hashes, and distributor notification artifact when present. The workspace path names should make the artifact type clear. A future implementation may keep the existing `data-packages/` directory, but file extensions and metadata fields should reflect the real format, such as: ```text workspace/ snapshots///.modules.json data-packages///.data_package.yaml ``` Replace `inspect briefing` with `inspect modules` during the cutover. `inspect modules` should return the module snapshot. `inspect data-package` should return the YAML artifact or a parsed representation of the YAML artifact. Do not leave inspect commands pointed at obsolete report-shaped data. ## Module Options And Compatibility Each module should have a typed options struct, even when initially empty. Configuration may decode module options from YAML, but internal module builders should receive typed options rather than `map[string]any`. Illustrative config shape: ```yaml reports: next_6_hours: deterministic_modules: - id: hourly_table options: range: valid_period fields: - time - temperature_f - pop_percent - wind_gust_mph - id: afd_short_term_text ``` Module definitions should declare: - module ID; - stanza name; - typed options schema; - supported report IDs or report categories; - required collected facts; - required derived facts; - whether missing optional facts omit the stanza, emit an empty stanza, or produce a warning; - whether duplicate use of the module is allowed. Configuration validation should reject: - unknown report IDs; - unknown module IDs; - duplicate module IDs unless explicitly allowed; - two modules that render the same stanza name; - module options that do not match the module's typed option schema; - modules that are incompatible with the report's valid-period strategy or available facts. ## Recent Changes Target Recent Changes must remain structured and deterministic. During the clean-break cutover, move comparison inputs away from report-shaped `briefing.Package` values and toward module-oriented snapshots. Recommended target: - compare `ModuleOutput` values or typed module snapshot stanzas, not rendered YAML and not rendered Markdown; - keep report-compatible matching policy in `internal/report`; - keep threshold configuration in `internal/config`; - keep comparison algorithms in `internal/changes`; - make each comparison explicit about which module stanzas it needs. For example, Daily comparison should primarily consume `derived_daily_summary`, `derived_daypart_summaries`, `alert_digest`, and `precip_timing` if present. If a required stanza is missing, the comparison should return no change with an inspectable warning or an actionable error, depending on the report's configured missing-data policy. ## Package Naming Target The current `internal/forecast` package owns both forecast-specific derivation and broader normalized weather data. Because planned sources include current observations, radar, and historical review inputs, implementation should consider splitting names during the clean-break refactor: - `internal/weatherdata`: normalized collected source facts, source metadata, warnings, and broad weather data types; - `internal/forecast`: forecast-specific algorithms such as period slicing, daily summaries, daypart grouping, and precipitation timing. If this split is too large for the first cutover, introduce `CollectedFacts` in the package that minimizes churn, but avoid expanding the meaning of `internal/forecast` further in new module contracts. ## Initial Module Candidates The first module catalog should start with the modules needed to replace current report-shaped briefing output and should clearly distinguish implemented modules from future-only candidates. Initial candidates: - `metadata` - `current_conditions` - `derived_daily_summary` - `derived_daypart_summaries` - `hourly_table` - `precip_timing` - `alert_digest` - `area_forecast_discussion` - `afd_key_messages` - `afd_short_term_text` - `afd_long_term_text` - `weather_story` - `forecast_delta` - `outdoor_windows` - `weekend_planning` - `storm_window_summary` Each module should declare inputs, outputs, report applicability, missing-data behavior, compatibility behavior, and options. ## Target Derived Daily Summary The intended `derived_daily_summary` shape is: ```yaml derived_daily_summary: high_temp_f: 86 low_temp_f: 68 max_pop_percent: 70 max_pop_window: "2 PM-6 PM" measurable_qpf_total_in: 0.35 max_hourly_qpf_in: 0.12 first_precip_hour: "1 PM" last_precip_hour: "8 PM" thunder_mentioned: true max_wind_gust_mph: 28 heat_index_max_f: 91 ``` `measurable_qpf_total_in` and `max_hourly_qpf_in` are future target fields. They require a real upstream quantitative precipitation source and should be omitted until such a source is represented in `CollectedFacts`. ## Target Derived Daypart Summaries The intended `derived_daypart_summaries` shape is: ```yaml derived_daypart_summaries: morning: temp_range_f: "70-78" max_pop_percent: 20 dominant_condition: Partly sunny afternoon: temp_range_f: "82-86" max_pop_percent: 70 dominant_condition: Showers and thunderstorms likely ``` ## Configurable Composition Target The target configuration model should allow report module composition to be changed without editing cross-cutting report-builder code. Built-in defaults should remain in Go so the application works with no module override config. Illustrative config: ```yaml reports: daily: deterministic_modules: - current_conditions - derived_daily_summary - derived_daypart_summaries - precip_timing - alert_digest - afd_short_term_text - weather_story_text ``` Adding or removing an implemented module from an implemented report should be a single config edit. Unknown modules, invalid options, duplicate stanzas, and incompatible report/module combinations should fail with actionable errors. ## Acceptance Criteria The refactor is complete when: - current implemented reports generate successfully from named-stanza YAML prompt packages; - module snapshots are persisted as structured JSON and linked from metadata; - `inspect modules` returns module snapshots; - `inspect briefing` is removed from CLI help, parser support, and non-roadmap docs; - Recent Changes compares structured module snapshots, not rendered Markdown or YAML text; - report definitions declare default module order in one place; - implemented report module composition can be overridden by config; - implemented modules have typed options and compatibility contracts; - `CollectedFacts` are built once per report run and reused by all modules; - `DerivedFacts` are built from `CollectedFacts` and do not depend on adapter transport details; - module builders do not call Weather API, Scriptorium, distributor, or filesystem state directly; - stale report-shaped briefing containers are removed from the generation path; - QPF output fields remain omitted until upstream QPF exists. ## Design Rules - Keep modules deterministic. - Keep modules self-contained where practical. - Preserve the `CollectedFacts -> DerivedFacts -> ModuleOutput` boundary. - Build `CollectedFacts` once per report run. - Build `DerivedFacts` from `CollectedFacts`, not from adapter-specific transport details. - Keep `DerivedFacts` conservative and reusable. - Keep raw external source details behind adapters and forecast normalization. - Keep report composition centralized and ordered. - Prefer typed outputs over generic maps. - Prefer typed fact contracts over string-keyed fact registries. - Prefer named YAML stanzas over generic module arrays. - Use unit-explicit field names. - Do not require the LLM to calculate obvious derived facts. - Do not let module builders call external services or write durable state. - Do not introduce plugins, dynamic loading, or a generic workflow engine. ## Risks And Mitigations - Prompt contract churn: stage YAML introduction after module outputs are tested and inspectable. - Recent Changes drift: compare stable module outputs and keep snapshot tests. - Over-abstraction: start with simple builders and a registry, not a framework. - Config complexity: expose ordered module selection and typed options only for implemented modules; defer broad parameterization. - Loss of useful context: preserve focused source excerpts and source warnings, but avoid reintroducing raw unbounded payloads. ## Deferred Work These are out of scope for the initial module refactor: - dynamic plugin loading; - user-defined module code; - YAML-defined module schemas; - module parameterization beyond implemented typed options; - replacing Weather API source fetching with module-owned fetches; - moving prompt authoring or Scriptorium prompt changes into weatherreporter; - adding future source modules, such as SPC products, before upstream data and report requirements exist. ## Open Questions ### Should module snapshots and prompt packages both be persisted? Recommended approach: persist module snapshots as JSON and prompt packages as YAML. JSON module snapshots are better for structured Recent Changes, state lookup, and tests. YAML prompt packages are better for prompt readability and LLM consumption. Keeping both artifacts gives each use case the right format without asking comparison code to parse prompt-oriented YAML. Viable alternative: persist only the YAML prompt package and parse it for inspection and Recent Changes. This reduces artifact count, but it couples machine comparison to prompt formatting and makes future prompt-oriented formatting changes riskier. ### Should `internal/forecast` be split during the first cutover? Recommended approach: split broad normalized source types into `internal/weatherdata` during the clean-break cutover if the implementation scope remains manageable. This name fits current conditions, alerts, discussion, weather story, future radar, and future historical data better than `forecast`. Viable alternative: keep existing `internal/forecast` types for the first module implementation and introduce `CollectedFacts` as a wrapper. This reduces short-term churn, but it leaves a package name that will become increasingly misleading as non-forecast sources grow. ### Should module config support typed options immediately? Recommended approach: support typed options for implemented modules from the start, even if most modules use empty options. This establishes the extension point needed for hourly ranges, field selection, and AFD section choices without adding dynamic maps to module builders. Viable alternative: initially support only ordered module IDs and add options later. This is simpler, but it may force another config shape change as soon as hourly range or section-selection experiments begin.