weatherreporter/docs/roadmap/implementation.md

Data Package Export Implementation Roadmap

Purpose

This roadmap defines the staged implementation plan for docs/roadmap/data-package-exports.md. It is written for an LLM coding agent that will implement the curated data-package export feature in order.

The goal is to separate rich module values used by templates from curated prompt-facing values serialized into Scriptorium data packages. The change should make YAML data packages smaller and clearer without reducing deterministic template composability.

This is a planning document only. It may describe unimplemented behavior because it lives under docs/roadmap/.

Source Roadmap

Use docs/roadmap/data-package-exports.md as the feature roadmap and target state authority. That document defines the user intent, module/export boundary, initial cleanup targets, and non-goals.

If this implementation plan and the feature roadmap conflict, update the feature roadmap first so it remains the conceptual source of truth, then update this file.

Locked Decisions

• Keep rich module values available for templates, module snapshots, inspection, Recent Changes, and render contexts.
• Serialize curated prompt-facing module values into data packages.
• Store prompt-facing values on module.Output as runtime-only values.
• Do not persist prompt-facing values in module snapshot JSON.
• Keep internal/promptinput independent from internal/briefing.
• Add prompt export behavior to the module registry/definition path where module output policy already lives.
• Default modules without custom exporters to pass-through prompt values.
• Bump the data-package schema version from weatherreporter.data_package.v2 to weatherreporter.data_package.v3.
• No compatibility aliases are required for removed prompt-facing fields.
• Do not add user-configurable field selection, field allowlists, reflection filters, or prompt-specific field profiles in this implementation.

Implementation Principles

• Preserve existing generated Markdown output.
• Preserve existing template render-context richness.
• Preserve existing module snapshot richness and JSON shape except for any unavoidable schema-version-only change. The preferred approach is to omit prompt values from module snapshot JSON entirely.
• Keep exporter code near the module that owns the shape.
• Use typed prompt-facing structs for modules that need reshaping.
• Keep simple modules on default pass-through behavior.
• Make exporter errors actionable and include module ID/stanza context.
• Keep YAML category ordering unchanged.
• Treat generated-template today, tomorrow, and daily reports as equal consumers of rich template module values.
• Update implemented docs only after code behavior exists.

Stage 1: Runtime Prompt Value Contract

Goal

Add a runtime-only prompt value to module outputs while preserving rich module snapshot persistence.

Files To Inspect

• internal/module/module.go
• internal/module/module_test.go
• internal/state/filesystem.go
• internal/state/filesystem_test.go
• internal/app/app.go
• internal/app/app_test.go

Implementation

• Add a runtime-only field to module.Output, for example:

type Output struct {
    ID          ID     `json:"id"`
    StanzaName  string `json:"stanzaName"`
    Value       any    `json:"value"`
    PromptValue any    `json:"-"`
}

• Add a small helper in internal/module, for example:

func (o Output) DataPackageValue() any

The helper should return PromptValue when non-nil and fall back to Value otherwise. This fallback keeps tests and any manually built snapshots simple.

• Do not require PromptValue in Snapshot.Validate.
• Do not persist PromptValue to module snapshot JSON.
• Do not change StanzaValue; it should continue decoding rich Value.

Acceptance Criteria

• Module snapshots still serialize rich module values under value.
• Module snapshots do not serialize promptValue or any equivalent field.
• Existing rich-module snapshot lookup and StanzaValue behavior remain unchanged.
• Code has a single helper for choosing the data-package value from an output.

Tests

• Add module tests proving:
  • DataPackageValue uses PromptValue when set;
  • DataPackageValue falls back to Value;
  • marshaled snapshot JSON omits PromptValue;
  • StanzaValue continues decoding rich Value.

Suggested focused command:

go test ./internal/module ./internal/state

Prompt Size

Small enough for one implementation prompt.

Stage 2: Module Registry Prompt Exporters

Goal

Teach the module registry to attach prompt-facing values to outputs as modules are built.

Files To Inspect

• internal/briefing/modules.go
• internal/briefing/modules_test.go
• internal/briefing/*_module.go
• internal/module/module.go

Implementation

• Add a prompt exporter type in internal/briefing, for example:

type ModulePromptExporter func(value any) (any, error)

• Add PromptExporter ModulePromptExporter to ModuleDefinition.
• In ModuleRegistry.BuildModule, after builder output ID and stanza validation:
  • if PromptExporter is nil, set output.PromptValue = output.Value;
  • if PromptExporter is present, call it with output.Value;
  • set output.PromptValue to the returned value;
  • wrap exporter errors with module ID and stanza context.
• Add a typed exporter helper if it keeps module exporters concise, for example:

func promptExporter[T any](fn func(T) (any, error)) ModulePromptExporter

The helper should convert any through JSON marshal/unmarshal or direct type assertion only if it meaningfully reduces boilerplate without hiding errors.

• Do not make internal/promptinput import internal/briefing.
• Do not move module-specific export policy into internal/promptinput.

Acceptance Criteria

• Every built module output has a non-nil data-package value.
• Modules without custom exporters pass through rich values.
• Custom exporter errors identify the module and stanza.
• Registry validation remains focused on definitions, duplicate IDs/stanzas, supported reports, options, builders, and missing-data policy.

Tests

• Add or update module registry tests for:
  • default pass-through prompt values;
  • custom exporter prompt values;
  • exporter error wrapping;
  • output ID/stanza validation still runs before or around export behavior;
  • output PromptValue is not persisted in snapshot JSON.

Suggested focused command:

go test ./internal/briefing ./internal/module

Prompt Size

Small enough for one implementation prompt.

Stage 3: Promptinput Uses Exported Values And Schema V3

Goal

Make data-package construction serialize prompt-facing values and bump the data package schema version.

Files To Inspect

• internal/promptinput/package.go
• internal/promptinput/package_test.go
• internal/state/filesystem_test.go
• internal/app/app.go
• internal/app/app_test.go
• docs/roadmap/data-package-exports.md

Implementation

• Change promptinput.SchemaVersion to:

const SchemaVersion = "weatherreporter.data_package.v3"

• Update stanzasFromSnapshot to use output.DataPackageValue() rather than output.Value.
• Keep BriefingStanzas category grouping and ordering unchanged.
• Keep LoadYAML validation strict for the current schema version.
• Update tests and fixtures that assert weatherreporter.data_package.v2.

Acceptance Criteria

• Saved data packages use schema version weatherreporter.data_package.v3.
• Data package stanzas use PromptValue when present.
• Data package stanzas fall back to rich Value when PromptValue is absent, which keeps hand-built test snapshots and loaded rich snapshots usable.
• YAML category ordering remains unchanged.
• Module snapshots remain rich and unaffected by prompt export serialization.

Tests

• Update promptinput tests for:
  • schema version v3;
  • prompt value preferred over rich value;
  • fallback to rich value;
  • deterministic categorized YAML output unchanged apart from stanza values and schema version;
  • unknown/uncategorized stanza behavior unchanged.
• Update state tests that load data packages.

Suggested focused command:

go test ./internal/promptinput ./internal/state

Prompt Size

Small enough for one implementation prompt.

Stage 4: Current Conditions And Hourly Forecast Exports

Goal

Add custom prompt exports for the clearest noisy raw-data modules: current_conditions and hourly_forecast.

Files To Inspect

• internal/briefing/current_conditions_module.go
• internal/briefing/hourly_forecast_module.go
• internal/briefing/base_modules_test.go
• internal/generatedtext/render_context.go
• internal/reporttemplate/templates/*.md.tmpl
• internal/app/app_test.go

Current Conditions Export Shape

The prompt-facing current_conditions export should keep:

• condition_text
• is_day
• temperature_c
• temperature_f
• apparent_temperature_c
• apparent_temperature_f
• dewpoint_c
• dewpoint_f
• relative_humidity_percent
• wind_speed_kmh
• wind_speed_mph
• wind_direction

It should remove:

• condition_text_lower
• wind_direction_text

The rich CurrentConditionsModule should keep those helper fields for templates.

Hourly Forecast Export Shape

The prompt-facing hourly_forecast export should keep module-level fields:

• product
• issued_at
• updated_at
• source_location
• source_location_id
• periods

Each prompt-facing hourly period should keep:

• period_begins
• period_ends
• name
• is_day
• condition_code
• text_description
• temperature fields
• dewpoint fields
• wind speed and gust fields
• wind_direction
• pressure fields
• visibility fields
• apparent temperature fields
• cloud_cover_percent
• probability_of_precipitation_percent
• precipitation amount fields
• snowfall depth fields
• uv_index
• relative_humidity_percent

Each prompt-facing hourly period should remove:

• hour_label
• text_description_lower
• mention_precipitation

The rich HourlyForecastPeriod should keep those helper fields for templates.

Implementation

• Add prompt export structs near each module.
• Add exporter functions near each module.
• Register exporters in the default module definitions.
• Prefer straightforward field copying over reflection.
• Preserve omitempty behavior.

Acceptance Criteria

• Saved data packages no longer include removed fields for current conditions or hourly forecast.
• Rich module snapshots and render contexts still include template helper fields.
• Existing hourly, today, tomorrow, and daily template output remains unchanged wherever those templates consume current conditions or hourly forecast values.

Tests

• Update module tests to prove prompt exports omit stale fields and keep expected factual fields.
• Update template/render-context tests to prove helper fields remain available to templates.
• Update app workflow tests to inspect saved YAML and reject:
  • condition_text_lower;
  • wind_direction_text;
  • hour_label;
  • text_description_lower;
  • mention_precipitation.

Suggested focused command:

go test ./internal/briefing ./internal/generatedtext ./internal/reporttemplate ./internal/app

Prompt Size

Medium. Suitable for one implementation prompt.

Stage 5: Derived Daypart Summary Export

Goal

Add a curated prompt export for derived_daypart_summaries while preserving the rich daypart fields used by Daily, Today, and Tomorrow templates.

Files To Inspect

• internal/briefing/derived_daypart_summaries_module.go
• internal/briefing/derived_modules_test.go
• internal/generatedtext/render_context.go
• internal/reporttemplate/templates/daily.md.tmpl
• internal/reporttemplate/templates/today.md.tmpl
• internal/reporttemplate/templates/tomorrow.md.tmpl
• internal/reporttemplate/reporttemplate_test.go
• internal/app/app_test.go

Export Shape

For each daypart, the prompt-facing export should keep:

• date
• display_name
• period_begins
• period_ends
• temp_range_f
• apparent_temp_range_f
• max_pop_percent
• max_pop_time
• mention_precipitation
• max_wind_gust_mph
• max_wind_gust_time
• dominant_condition
• temperature_trend
• temperature_start_phrase_f
• temperature_end_phrase_f
• temperature_peak_phrase_f
• temperature_steady_phrase_f
• notable_conditions
• snow
• ice
• fog
• heat
• cold
• wind
• relevant_alert_count

The prompt-facing export should remove:

• temperature_phrase_f
• dominant_condition_lower
• dominant_condition_display
• max_pop_time_label

For max_pop_time, use the most readable existing time label. Prefer MaxPopTimeLabel when present, falling back to MaxPopTime, while keeping the prompt-facing field name max_pop_time.

Keep the temperature trend phrase fields even though only some are populated for each trend. They are not duplicates when used according to the trend:

• rising/falling use start and end phrases;
• peaking uses peak phrase;
• steady uses steady phrase.

Implementation

• Add prompt export structs near the daypart module.
• Add an exporter for the map[string]DerivedDaypartSummaryModule value.
• Preserve map keys and values for all emitted dayparts.
• Register the exporter in the default module definitions.
• Do not alter rich DerivedDaypartSummaryModule fields used by templates.

Acceptance Criteria

• Saved data packages no longer include the removed daypart fields.
• The prompt-facing daypart export still gives the LLM enough information to understand condition, temperature trend, precipitation, wind, notable conditions, and alert relevance.
• Daily, Today, and Tomorrow template output remains unchanged.
• Render contexts still expose rich daypart helper fields.

Tests

• Update daypart module tests for:
  • prompt export field presence;
  • removed field absence;
  • rising, falling, peaking, and steady trend values;
  • max PoP time using the friendly label under the stable max_pop_time key.
• Update template tests to prove rich helper fields still render Daily, Today, and Tomorrow daypart wording.
• Update app data-package tests to reject stale daypart fields.

Suggested focused command:

go test ./internal/briefing ./internal/generatedtext ./internal/reporttemplate ./internal/app

Prompt Size

Medium. Suitable for one implementation prompt.

Stage 6: Pass-Through Review And Workflow Regression Tests

Goal

Confirm all other modules export correctly through pass-through behavior and add workflow-level coverage proving the new boundary.

Files To Inspect

• internal/briefing/*_module.go
• internal/briefing/modules.go
• internal/promptinput/package_test.go
• internal/app/app_test.go
• internal/generatedtext/render_context_test.go
• internal/reporttemplate/reporttemplate_test.go

Implementation

• Review remaining modules:
  • metadata
  • narrative_forecast
  • derived_daily_summary
  • precip_timing
  • outdoor_windows
  • alert_digest
  • spc_convective_outlooks
  • spc_convective_discussion
  • area_forecast_discussion
  • weather_story
  • planning modules
• Keep pass-through behavior for modules that are already prompt-appropriate.
• Add custom exporters only if a module contains clear template-only helpers or confusing duplicate fields.
• Do not broaden this stage into a general prompt-schema redesign.
• Add workflow assertions that:
  • module snapshots contain rich values;
  • render contexts contain rich values;
  • data packages contain curated values;
  • Scriptorium receives the curated data package path exactly as before.
• Include workflow coverage for generated-template today, tomorrow, and daily reports when asserting daypart and render-context behavior.

Acceptance Criteria

• Every default module either has a custom exporter or intentionally uses pass-through.
• App workflow tests prove saved data packages omit the cleaned fields.
• App workflow tests prove rich helper fields remain available where templates use them.
• App workflow or render-context tests include daily alongside today and tomorrow for daypart helper coverage.
• Existing report generation behavior remains unchanged except for data-package YAML content and schema version.

Tests

Suggested focused command:

go test ./internal/briefing ./internal/promptinput ./internal/generatedtext ./internal/reporttemplate ./internal/app

Prompt Size

Small to medium. Suitable for one implementation prompt.

Stage 7: Documentation And Final Validation

Goal

Update implemented documentation after the code exists and run full validation.

Files To Inspect

• docs/internal/module.md
• docs/internal/prompt-input.md
• docs/templates.md
• docs/operations.md
• docs/troubleshooting.md
• docs/roadmap/data-package-exports.md
• tests that assert documentation examples or data-package snippets

Implementation

• Update docs/internal/module.md to describe:
  • rich module output;
  • runtime prompt export values;
  • pass-through exporters;
  • module-owned prompt export policy.
• Update docs/internal/prompt-input.md to document:
  • schema version weatherreporter.data_package.v3;
  • curated module export behavior;
  • category ordering unchanged;
  • data packages are not full template render contexts.
• Update docs/templates.md to clarify that templates can use richer fields than the data package exposes.
• Update any Daily template-variable documentation alongside Today and Tomorrow references where the same module fields are discussed.
• Update any implemented docs containing stale examples of removed data-package fields.
• Do not document deferred configurable field profiles or reflection filters as implemented behavior.

Acceptance Criteria

• Non-roadmap docs describe only implemented data-package export behavior.
• Docs do not imply data packages contain template-only helper fields.
• Examples and snippets use schema version v3 when they show data packages.

Validation Commands

go test ./internal/module ./internal/briefing ./internal/promptinput
go test ./internal/generatedtext ./internal/reporttemplate ./internal/app
go test ./...
go run ./cmd/weatherreporter --help
git diff --check

Stale Field Greps

Run focused checks against docs and expected saved data-package fixtures/snippets:

rg -n "condition_text_lower|wind_direction_text|hour_label|text_description_lower|mention_precipitation|dominant_condition_lower|dominant_condition_display|max_pop_time_label|temperature_phrase_f" docs internal/*/*_test.go

Review matches manually. Some matches should remain in rich module structs, template tests, and render-context tests; they should not remain as expected data-package fields.

Prompt Size

Medium. Suitable for one implementation prompt.

Deferred Work

Do not include these in this implementation sequence:

• user-configurable data-package field selection;
• per-prompt custom field profiles;
• reflection-based generic include/exclude lists;
• automatic schema generation for data-package exports;
• changing collected facts or derived facts contracts;
• changing Scriptorium invocation behavior;
• changing generated Markdown templates beyond preserving their output;
• migrating old workspace data-package artifacts.

Open Questions

No open questions block implementation.

The implementation plan intentionally locks in the recommended choices from the feature roadmap: schema version bump to weatherreporter.data_package.v3, and runtime-only prompt export values stored on module.Output.

Global Validation Checklist

• go test ./... passes.
• go run ./cmd/weatherreporter --help is accurate.
• git diff --check passes.
• Saved data packages use schema version weatherreporter.data_package.v3.
• Saved data packages serialize prompt-facing module exports.
• Saved module snapshots serialize rich module values and do not persist prompt-facing values.
• Render contexts continue to expose rich module values.
• Hourly, Today, Tomorrow, and Daily templates continue to render the same Markdown output.
• YAML briefing category order remains unchanged.
• current_conditions data packages omit condition_text_lower and wind_direction_text.
• hourly_forecast.periods[] data packages omit hour_label, text_description_lower, and mention_precipitation.
• derived_daypart_summaries data packages omit temperature_phrase_f, dominant_condition_lower, dominant_condition_display, and max_pop_time_label.
• Non-roadmap docs describe only implemented behavior.