Finalized the initial development roadmap

docs/policy/development.md

@@ -8,12 +8,12 @@ The application should remain a small, explicit, dependency-light Go program. Do
 
 `weatherreporter` is a deterministic weather briefing and report-preparation application. It should:
 
-1. Fetch normalized weather data from an internal weather API backed by `weatherfeeder`.
+1. Fetch normalized weather data from a single configured internal weather API endpoint backed by `weatherfeeder`.
 2. Derive report-specific briefing packages from the normalized forecast bundle.
 3. Compare current briefing snapshots against prior comparable snapshots to produce optional Recent Changes.
-4. Build structured prompt variables for a specific report type.
+4. Build structured prompt input data packages for a specific report type.
 5. Invoke `scriptorium` as an external prompt runner.
-6. Persist the rendered Markdown report, briefing snapshot, and generation metadata.
+6. Persist the rendered Markdown report, briefing snapshot, prompt input package, and generation metadata.
 
 The preferred data flow is:
 
@@ -23,7 +23,7 @@ weatherfeeder-backed internal API
     -> forecast bundle
     -> report-specific briefing builder
     -> recent-change comparison
-    -> prompt variable package
+    -> prompt input data package
     -> scriptorium subprocess adapter
     -> Markdown report + metadata + stored snapshot
 ```
@@ -96,7 +96,7 @@ internal/state/
   filesystem.go
   metadata.go
 
-internal/promptvars/
+internal/promptinput/
   build.go
   schema.go
 
@@ -167,7 +167,8 @@ Responsibilities:
   - `weatherreporter run morning`
   - `weatherreporter run evening`
   - `weatherreporter inspect snapshot`
-- Parse CLI flags and convert them into app-layer request structs.
+- Use the Go standard library for CLI parsing unless future complexity justifies a dependency.
+- Parse flags such as `--config`, `--units`, `--tz`, `--out`, optional Daily `--date`, and storm `--start`/`--end`, then convert them into app-layer request structs.
 - Load configuration through `internal/config`.
 - Present concise user-facing errors.
 
@@ -181,15 +182,19 @@ Non-responsibilities:
 Suggested command shape:
 
 ```text
-weatherreporter generate daily --location home --date today --out ./daily.md
-weatherreporter generate tomorrow --location home --out ./tomorrow.md
-weatherreporter generate three-day --location home --out ./three_day.md
-weatherreporter generate weekend --location home --out ./weekend.md
-weatherreporter generate storm --location home --out ./storm.md
-weatherreporter run morning --location home
-weatherreporter run evening --location home
+weatherreporter generate daily --date 2026-05-29 --out ./daily.md
+weatherreporter generate tomorrow --out ./tomorrow.md
+weatherreporter generate three-day --out ./three_day.md
+weatherreporter generate weekend --out ./weekend.md
+weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00 --out ./storm.md
+weatherreporter run morning
+weatherreporter run evening
 ```
 
+The MVP should not expose location selection. Source `locationId` and `locationName` values returned by the weather API may be retained as provenance.
+
+For `generate daily`, `--date` is optional. When provided, it must use `YYYY-MM-DD`; when omitted, it resolves to the current local date in the configured timezone.
+
 ### `internal/config`
 
 Owns configuration structures, defaults, loading, precedence, and validation.
@@ -199,20 +204,29 @@ Responsibilities:
 - Define application configuration structs.
 - Provide built-in defaults in `defaults.go`.
 - Load YAML configuration from `/usr/local/etc/weatherreporter/config.yml` or a CLI-supplied path.
+- Use `gopkg.in/yaml.v3` for YAML parsing.
 - Apply precedence rules.
 - Validate required settings.
-- Normalize paths, durations, report settings, locations, and daypart definitions.
+- Normalize paths, durations, report settings, weather API units/timezone, missing-source policy, and daypart definitions.
 
 Suggested configuration areas:
 
-- Weather API base URL, timeout, and location endpoints.
-- Locations and time zones.
-- `scriptorium` binary, profile, timeout, and optional extra arguments.
+- Weather API base URL, timeout, units, timezone, precision, and missing-source policy.
+- `scriptorium` binary, config path, profile, timeout, and optional extra arguments.
 - Workspace and output directories.
 - Report enablement and output naming.
 - Daypart definitions.
 - Recent-change thresholds.
 
+Initial defaults:
+
+- Weather API units: `us`.
+- Weather API timezone: `Chicago`.
+- Weather API format: `json`.
+- Missing-source policy: `warn`.
+
+Missing-source policy should support a global default and per-source overrides. Valid policy values are `error`, `warn`, and `none`.
+
 Non-responsibilities:
 
 - No command execution.
@@ -230,7 +244,7 @@ Responsibilities:
   - Run the morning batch.
   - Run the evening batch.
   - Generate a manual storm report.
-- Coordinate config, weather API adapter, report registry, briefing builders, state store, change comparison, prompt variable builder, and `scriptorium` runner.
+- Coordinate config, weather API adapter, report registry, briefing builders, state store, change comparison, prompt input builder, and `scriptorium` runner.
 - Enforce workflow order.
 - Ensure each generation run persists enough artifacts for inspection and future comparison.
 
@@ -238,15 +252,16 @@ The core generation workflow should be approximately:
 
 ```text
 resolve report definition
-resolve location and valid period
+resolve valid period
 fetch current weather bundle
 build current briefing package
 load prior comparable briefing snapshot
 compute recent changes
-build prompt variables
-write vars file
-invoke scriptorium
-persist report metadata and briefing snapshot
+build prompt input data package
+write data package
+run scriptorium render preflight
+invoke scriptorium run
+persist report metadata, briefing snapshot, data package, preflight output, and rendered report
 ```
 
 Non-responsibilities:
@@ -261,25 +276,37 @@ HTTP adapter for the internal weather API backed by `weatherfeeder`.
 
 Responsibilities:
 
-- Fetch normalized weather data for a configured location.
+- Fetch normalized weather data from the configured API base URL.
+- Fan out to multiple weather API endpoints and assemble one internal `forecast.Bundle`.
 - Decode API responses into adapter-owned DTOs or directly into stable internal types if those types are intentionally owned by `weatherreporter`.
 - Apply request timeouts and context cancellation.
+- Apply configured query defaults, including `format=json`, `units=us`, and `tz=Chicago` unless overridden.
+- Fetch full `/forecast/hourly` and `/forecast/narrative` products, not day-slice endpoints, so Go domain code owns report-period selection.
+- Record per-source provenance: endpoint, query, fetch time, issued/updated time when available, SHA-256 over canonical/minified raw `data` JSON, warnings, and missing-source status.
+- Represent source warnings as first-class records with source name, code, severity, message, endpoint, and completeness impact.
+- Require hourly forecast data for normal scheduled reports.
+- Apply missing-source policy for `data:null`, malformed non-required sections, or unavailable upstream products.
 - Return actionable errors containing endpoint and operation context.
 
-Expected data categories:
+Initial data categories:
 
+- Latest observation.
+- Current conditions.
 - Hourly forecast data.
-- Daily forecast data.
 - NWS narrative forecast periods.
 - NWS alerts.
 - NWS forecast discussion.
+
+Stubbed source slots until upstream support exists:
+
+- Daily forecast data.
 - NWS weather story.
 
 Non-responsibilities:
 
 - No daypart grouping.
 - No Recent Changes comparison.
-- No prompt variable construction.
+- No prompt input construction.
 - No `scriptorium` calls.
 
 ### `internal/adapters/scriptorium`
@@ -292,24 +319,32 @@ Responsibilities:
 
 ```go
 type Runner interface {
+    Render(ctx context.Context, req RenderRequest) (*RenderResult, error)
     Run(ctx context.Context, req RunRequest) (*RunResult, error)
 }
 ```
 
-- Execute `scriptorium run` with `exec.CommandContext`.
+- Execute `scriptorium render` for preflight/debug output without LLM generation.
+- Execute `scriptorium run` for report generation.
+- Run `scriptorium render` as an always-on preflight before `scriptorium run` for MVP generated reports.
 - Pass arguments as an argv slice, not through a shell.
-- Prefer a vars file path over large inline JSON.
+- Pass large prompt input as `--input data_package=<path>`.
 - Capture stdout/stderr with reasonable size limits.
-- Treat nonzero exits as actionable errors.
+- Treat nonzero exits as actionable errors, including exit code `2` from `run`, which may still produce output.
 - Keep all `scriptorium`-specific flag details inside the adapter.
 
-Suggested command form:
+Suggested command forms:
 
 ```text
+scriptorium render \
+  --prompt weather.daily_report \
+  --input data_package=./workspace/data-packages/daily/2026-05-29T050000-0500.data_package.json \
+  --format json
+
 scriptorium run \
   --prompt weather.daily_report \
-  --vars-file ./workspace/daily.vars.json \
-  --out ./workspace/daily_report.md
+  --input data_package=./workspace/data-packages/daily/2026-05-29T050000-0500.data_package.json \
+  --out ./workspace/reports/daily/2026-05-29T050000-0500.md
 ```
 
 Non-responsibilities:
@@ -365,12 +400,14 @@ Suggested report definitions:
 
 ```text
 daily_today       -> prompt weather.daily_report
-daily_tomorrow    -> prompt weather.daily_report or weather.tomorrow_report
+daily_tomorrow    -> prompt weather.daily_report
 three_day         -> prompt weather.three_day_outlook
 weekend           -> prompt weather.weekend_outlook
 storm             -> prompt weather.storm_report
 ```
 
+`weather.daily_report` should be the standard prompt for one local civil day, regardless of whether that day is today or tomorrow.
+
 A report definition should describe:
 
 - Report ID.
@@ -395,9 +432,10 @@ Builds report-specific briefing packages from forecast bundles and report defini
 Responsibilities:
 
 - Convert a forecast bundle into a report-specific structured briefing package.
-- Keep each report’s briefing shape explicit and testable.
-- Attach relevant NWS narrative periods, alerts, forecast discussion context, and weather story context.
-- Provide inputs suitable for LLM prompt variables.
+- Keep each report's briefing shape explicit and testable.
+- Attach relevant NWS narrative periods, alerts, forecast discussion context, and weather story context when available.
+- Include metadata such as schema version, configured units/timezone, source warnings, and source provenance.
+- Provide inputs suitable for `scriptorium` data packages.
 
 Report-specific builders should exist for:
 
@@ -425,9 +463,11 @@ Responsibilities:
 
 - Compare current briefing packages against prior comparable snapshots.
 - Apply meaningful-change thresholds.
-- Produce compact structured change summaries for prompt variables.
+- Produce compact structured change summaries for prompt input data packages.
 - Avoid comparison of rendered Markdown report text.
 
+Comparable snapshot matching should be declared by each report definition. Daily Today, Daily Tomorrow, and compatible date slices from multi-day reports may compare by same valid local date when the report registry marks them compatible. Weekend compares by same weekend window. Storm compares by explicit event window.
+
 Meaningful changes may include:
 
 - Temperature changes crossing configured thresholds.
@@ -447,14 +487,17 @@ Non-responsibilities:
 
 ### `internal/state`
 
-Durable state store for reports, snapshots, metadata, and comparison lookup.
+Durable state store for reports, snapshots, data packages, preflight output, metadata, and comparison lookup.
 
 Responsibilities:
 
 - Persist generated report metadata.
 - Persist briefing snapshots.
-- Persist prompt variable files when useful for inspection.
+- Persist prompt input data packages.
+- Persist `scriptorium render` preflight output for generated reports.
 - Locate prior comparable snapshots for Recent Changes.
+- Track RunID as generation timestamp plus report ID.
+- Use timestamped managed report names to avoid overwriting prior runs for the same valid period.
 - Use atomic writes where practical.
 - Keep filesystem layout narrow and predictable.
 
@@ -470,25 +513,26 @@ Suggested state layout:
 
 ```text
 workspace/
-  locations/
-    home/
-      snapshots/
-        daily/
-          2026-05-30/
-            2026-05-29T050000-0500.briefing.json
-            2026-05-29T050000-0500.metadata.json
-        three-day/
-        weekend/
-        storm/
-      reports/
-        daily/
-          2026-05-30.md
-        three-day/
-        weekend/
-        storm/
-      vars/
-        daily/
-          2026-05-29T050000-0500.vars.json
+  snapshots/
+    daily/
+      2026-05-30/
+        2026-05-29T050000-0500.briefing.json
+        2026-05-29T050000-0500.metadata.json
+    three-day/
+    weekend/
+    storm/
+  reports/
+    daily/
+      2026-05-29T050000-0500.md
+    three-day/
+    weekend/
+    storm/
+  data-packages/
+    daily/
+      2026-05-29T050000-0500.data_package.json
+  preflight/
+    daily/
+      2026-05-29T050000-0500.render.json
 ```
 
 Non-responsibilities:
@@ -497,16 +541,16 @@ Non-responsibilities:
 - No report prose generation.
 - No CLI formatting decisions.
 
-### `internal/promptvars`
+### `internal/promptinput`
 
-Builds the final variable payload passed to `scriptorium`.
+Builds the final data package passed to `scriptorium`.
 
 Responsibilities:
 
-- Combine report metadata, briefing package, Recent Changes, and selected source context into a prompt variable document.
-- Validate required prompt variables before invoking `scriptorium`.
-- Keep prompt variable schemas explicit enough to test.
-- Write vars files to the workspace when requested by the app layer.
+- Combine report metadata, briefing package, Recent Changes, selected source context, and source warnings into a prompt input document.
+- Validate required data package fields before invoking `scriptorium`.
+- Keep data package schemas explicit enough to test.
+- Write data package files to the workspace when requested by the app layer.
 
 Non-responsibilities:
 
@@ -521,7 +565,7 @@ Time, clock, and period helpers.
 Responsibilities:
 
 - Provide an injectable clock for deterministic tests.
-- Resolve local dates using the configured location time zone.
+- Resolve local dates using the configured report timezone.
 - Handle daypart spans, including overnight windows.
 - Normalize valid periods.
 - Provide helpers for recurring scheduled batches.
@@ -535,24 +579,38 @@ Non-responsibilities:
 
 Each generated report must be associated with explicit metadata:
 
+- RunID.
 - Report type.
 - Report variant, if applicable.
-- Location ID.
 - Generation time.
+- Configured report timezone.
 - Valid period start.
 - Valid period end.
-- Source product timestamps and/or hashes.
+- Source location ID/name when provided by upstream.
+- Source product timestamps and/or SHA-256 hashes.
+- Source warnings.
 - Briefing snapshot path.
-- Prompt variable path.
+- Prompt input data package path.
+- Preflight output path.
 - Rendered report path.
 
+All valid periods should use the configured local timezone, default `Chicago`, and half-open `[start,end)` intervals.
+
+Initial valid-period rules:
+
+- Daily Today: current local civil day, `[00:00, next 00:00)`.
+- Daily Tomorrow: next local civil day.
+- 3-Day Outlook: generation time through local midnight after the second following local civil day.
+- Weekend Outlook: Monday through Thursday covers Saturday 00:00 to Monday 00:00; Friday and Saturday cover `max(generation time, Friday 18:00)` to Monday 00:00; scheduled Sunday morning skips Weekend Outlook.
+- Manual Storm Report: requires explicit `--start` and `--end`; accept `YYYY-MM-DDTHH:MM` interpreted in the configured timezone and RFC3339 timestamps with explicit offsets.
+
 The valid period should identify what weather period the report covers, independent of when the report was generated.
 
 Examples:
 
 - A 5 PM Tomorrow Planning Brief for Saturday and a 5 AM Saturday Daily Report both cover the same valid date.
 - A Saturday Weekend Outlook covers the remaining weekend, while a Friday Weekend Outlook may cover Friday evening through Sunday night.
-- A Storm Report covers a forecast event window, not a fixed calendar day.
+- A Storm Report covers an explicit forecast event window, not a fixed calendar day.
 
 This identity is required for reliable Recent Changes behavior.
 
@@ -572,6 +630,8 @@ evening:
   - daily_tomorrow
 ```
 
+Scheduled batches should continue independent reports after a report failure. The CLI should return nonzero if any report failed and should emit an aggregate run summary.
+
 External scheduling should be handled by systemd timers, cron, or another orchestrator. `weatherreporter` should simply provide deterministic commands that can be scheduled.
 
 ## Storm Report Direction
@@ -579,7 +639,7 @@ External scheduling should be handled by systemd timers, cron, or another orches
 The initial version should support manual Storm Report generation:
 
 ```text
-weatherreporter generate storm --location home
+weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00
 ```
 
 Future storm monitoring should use a staged design:
@@ -606,14 +666,16 @@ Core tests should not require real external services.
 
 Priority test areas:
 
-- Configuration loading and validation.
+- Configuration loading and validation, including defaults for `units=us`, `tz=Chicago`, and missing-source policy `warn`.
+- Standard-library CLI command parsing, including `--units`, `--tz`, and storm `--start`/`--end`.
+- Weather API fan-out, source provenance, `data:null`, and missing-source policy behavior.
 - Daypart grouping, especially overnight periods.
 - Valid-period resolution for each report type.
 - Briefing package construction from fixtures.
-- Recent Changes threshold behavior.
+- Recent Changes threshold behavior and compatible snapshot matching.
 - Prior snapshot lookup.
-- `scriptorium` adapter behavior using a fake executable or command runner.
-- CLI command parsing for major workflows.
+- `scriptorium` adapter behavior using a fake executable or command runner, including both `render` and `run` with `--input data_package=<path>`.
+- Batch partial-failure behavior and aggregate exit status.
 
 ## Design Invariants