weatherreporter/docs/roadmap/implementation.md

# GeneratedText Hourly Report Implementation Roadmap

## Purpose

This roadmap defines the concrete implementation sequence for
`docs/roadmap/generated-text.md`. It is written for an LLM coding agent that
will implement each stage in order.

This is a future-work roadmap. Until a stage is implemented, non-roadmap docs
must not describe `GeneratedText`, embedded report templates, or other
unimplemented GeneratedText behavior as available.

## Source Feature Roadmap

Use `docs/roadmap/generated-text.md` as the authoritative feature roadmap for
intent, terminology, target behavior, and policy choices. This document is the
step-by-step implementation plan. If target behavior changes, update
`generated-text.md` first, then update this plan.

Future implementation agents should read `generated-text.md` before starting
Stage 1. Treat that file as the conceptual contract for why the feature exists
and what final shape it should have; treat this file as the ordered work plan
for reaching that shape.

## Locked Implementation Decisions

- Implement the first GeneratedText path for the rolling next-hours report,
  named `hourly`.
- Use `hourly` as the only planned public name for the rolling next-hours
  report; do not keep compatibility aliases from retired roadmap terminology.
- Use report ID `hourly`, artifact group `hourly`, batch output name
  `hourly.md`, and CLI command `weatherreporter generate hourly`.
- Keep the rolling report explicit-generation only; do not add scheduled batch
  membership in this implementation.
- Keep the valid-period length as a package-owned constant, initially 6 hours.
- Add report generation modes:
  - `scriptorium_markdown` for existing reports;
  - `generated_text_template` for `hourly`.
- Keep existing reports on the current Scriptorium Markdown path.
- For generated-text reports, keep `Definition.PromptID` as the Scriptorium
  prompt ID and set the hourly value to `weather.hourly_generated_text`.
- Add `TemplateID` and `GeneratedTextSchemaID` to report definitions.
- Use embedded assets under `internal/reporttemplate`.
- Use standard-library `text/template`.
- Let Scriptorium enforce its prompt-associated JSON Schema and retries;
  weatherreporter validates by unmarshalling into a typed GeneratedText struct
  and checking required fields. Do not add a Go JSON Schema dependency in this
  implementation.
- Keep the existing preflight artifact for prompt/input inspection.
- Add managed artifacts for raw GeneratedText JSON, structured Scriptorium run
  result, validated GeneratedText JSON, and render context JSON.
- Distributor notification continues to upload the managed Markdown report only.
- Preserve current CLI behavior, artifact paths, and generated output behavior
  for daily, tomorrow, three-day, weekend, and storm reports.

## Stage 1: Roadmap Reconciliation And Hourly Identity

Goal: remove conflicting future instructions and establish `hourly` as the only
planned public name for the rolling next-hours report.

Files to inspect:

- `docs/roadmap/generated-text.md`
- the superseded roadmap file for the retired rolling-report name
- `docs/roadmap/future.md`
- `docs/roadmap/implementation.md`
- `internal/app/app.go`
- `internal/report/definition.go`
- `internal/report/registry.go`

Implementation:

- Replace the superseded roadmap file for the retired rolling-report name with
  a short note pointing to `docs/roadmap/generated-text.md`, or delete it if
  the repository policy at implementation time prefers removing obsolete
  roadmap files.
- Ensure no roadmap instructs a future agent to implement retired public
  identifiers for the rolling report.
- If partial code using retired identifiers already exists, rename it to
  `hourly` rather than adding aliases:
  - app report kind -> `ReportHourly`;
  - report ID constant -> `report.Hourly`;
  - CLI command -> `hourly`;
  - artifact group -> `hourly`;
  - prompt IDs and tests updated to hourly names.
- Do not add compatibility aliases for retired command or config names.

Acceptance criteria:

- Active roadmap docs use `hourly` for the rolling report.
- No active roadmap describes a retired identifier as the target public report
  ID.
- Existing code compiles after any rename work.

Suggested validation:

```bash
rg -n 'near[_-]term|Near''Term' docs internal
go test ./internal/app ./internal/report ./internal/cli
```

This stage is small enough for one implementation prompt.

## Stage 2: Report Definition Generation Mode

Goal: add generation-mode metadata to report definitions without changing the
current report generation path.

Files to inspect:

- `internal/report/definition.go`
- `internal/report/registry.go`
- `internal/report/*_report.go`
- `internal/report/period_test.go`
- `internal/state/filesystem.go`

Implementation:

- Add a `GenerationMode` type in `internal/report`.
- Add constants:
  - `GenerationModeScriptoriumMarkdown`
  - `GenerationModeGeneratedTextTemplate`
- Add fields to `report.Definition`:
  - `GenerationMode GenerationMode`
  - `TemplateID string`
  - `GeneratedTextSchemaID string`
- Default or explicitly set existing reports to
  `GenerationModeScriptoriumMarkdown`.
- Update report definition tests so every generated report declares a
  generation mode.
- Do not wire app behavior in this stage.
- Keep existing `PromptID` semantics: it is the Scriptorium prompt ID for the
  report's generation mode.

Acceptance criteria:

- Existing report definitions retain current prompt IDs and behavior.
- Registry tests fail if a generated report omits generation mode.
- Generated-text-only fields are empty for existing Markdown reports.

Suggested validation:

```bash
go test ./internal/report ./internal/state
```

This stage is small enough for one implementation prompt.

## Stage 3: Hourly Report Definition, Modules, Facts, And CLI Shell

Goal: add the `hourly` report identity, rolling valid period, module
composition, facts derivation, config override aliases, and CLI command shell.

Files to inspect:

- `internal/report/definition.go`
- `internal/report/registry.go`
- new `internal/report/hourly_report.go`
- `internal/report/period_test.go`
- `internal/briefing/modules.go`
- `internal/facts/facts.go`
- `internal/facts/facts_test.go`
- `internal/config/reports.go`
- `internal/config/config_test.go`
- `internal/app/app.go`
- `internal/cli/root.go`
- `internal/cli/root_test.go`

Implementation:

- Add report ID `Hourly ID = "hourly"`.
- Add comparison strategy `CompareRollingWindow = "rolling_window"`.
- Add `hourlyReportHours` constant, initially `6`.
- Add `hourlyDefinition()`:
  - `Name: "Hourly Report"`
  - `PromptID: "weather.hourly_generated_text"`
  - `GenerationMode: GenerationModeGeneratedTextTemplate`
  - `TemplateID: "hourly"`
  - `GeneratedTextSchemaID: "hourly"`
  - `ComparisonStrategy: CompareRollingWindow`
  - `ArtifactGroup: "hourly"`
  - `BatchOutputName: "hourly.md"`
  - `Generated: true`
  - `CompatiblePriorIDs: []ID{Hourly}`
  - no morning/evening batch membership
- Resolve valid period as `[generation_time, generation_time + 6h)` in the
  effective timezone.
- Add default hourly modules in order:
  1. `metadata`
  2. `current_conditions`
  3. `hourly_forecast`
  4. `precip_timing`
  5. `alert_digest`
  6. `spc_convective_outlooks`
  7. `area_forecast_discussion`
  8. `spc_convective_discussion`
  9. `weather_story`
- Configure hourly `area_forecast_discussion` options to include only:
  - `key_messages`
  - `short_term`
- Update module supported-report lists:
  - include `report.Hourly` in all broadly compatible modules;
  - include `report.Hourly` for `hourly_forecast`;
  - keep daily/daypart modules incompatible.
- Add `facts.BuildDerived` support for hourly:
  - valid-period hourly slice;
  - precipitation timing from that slice;
  - alert overlap filtering;
  - SPC outlook/discussion overlap filtering;
  - no daily summaries, daypart summaries, or storm summary.
- Add config report aliases:
  - `hourly`
  - reject or do not recognize retired rolling-report config keys.
- Add app report kind `ReportHourly = "hourly"` and map it to
  `report.Hourly`.
- Add CLI parsing and help for:

  ```bash
  weatherreporter generate hourly [--config PATH] [--units VALUE] [--tz NAME] [--out PATH]
  ```

- Do not add `--date`, `--start`, `--end`, or duration flags.
- It is acceptable for `generate hourly` to return an explicit
  "generation mode not implemented" error until the app workflow stage, but
  parser and resolution tests should pass.

Acceptance criteria:

- `report.DefaultRegistry().Lookup(report.Hourly)` succeeds.
- hourly valid period is exactly six hours and not civil-day truncated.
- hourly default module composition validates.
- daily-only modules reject hourly.
- facts derivation supports the hourly module list.
- CLI help includes `generate hourly`.
- Existing command behavior is unchanged.

Suggested validation:

```bash
go test ./internal/report ./internal/briefing ./internal/facts
go test ./internal/config ./internal/app ./internal/cli
go run ./cmd/weatherreporter --help
```

This stage is large but still suitable for one implementation prompt if kept
strictly to report identity/facts/CLI shell. If it becomes unwieldy, split it
after report/facts and implement CLI/config aliases in a follow-up prompt.

## Stage 4: Embedded Template And Schema Assets

Goal: create the embedded asset package and first hourly template/schema.

Files to create:

- `internal/reporttemplate/reporttemplate.go`
- `internal/reporttemplate/templates/hourly.md.tmpl`
- `internal/reporttemplate/schemas/hourly.generated_text.schema.json`
- `internal/reporttemplate/reporttemplate_test.go`

Implementation:

- Use `go:embed` for templates and schemas.
- Expose a narrow API:
  - `Template(id string) (string, error)` or equivalent;
  - `Schema(id string) ([]byte, error)` or equivalent;
  - `Render(id string, data any) ([]byte, error)`.
- Use `text/template`.
- Register only `hourly` assets initially.
- Missing or unknown asset IDs must return actionable errors.
- Keep template functions minimal. Prefer prepared render-context strings over
  complex template logic.
- Add an initial hourly JSON Schema with:
  - required `summary`;
  - required `timing`;
  - required `impacts`;
  - optional `confidence`;
  - all fields strings;
  - `additionalProperties: false`.
- Add an initial hourly Markdown template that is readable and deterministic.
  It should include fixed headings, report valid period, selected deterministic
  facts, and GeneratedText slots.

Acceptance criteria:

- schema asset can be looked up for tests, local validation, and out-of-band
  Scriptorium registration;
- template asset can be looked up;
- template renders from a test render context;
- missing template/schema IDs fail clearly;
- no generated schema or template content is inline in Go.

Suggested validation:

```bash
go test ./internal/reporttemplate
```

This stage is small enough for one implementation prompt.

## Stage 5: GeneratedText Contract And Render Context

Goal: add typed GeneratedText validation and a typed hourly render context.

Files to create or inspect:

- new `internal/generatedtext` package, or another narrow package if the
  implementation chooses a better local name
- `internal/reporttemplate`
- `internal/module`
- `internal/briefing`
- `internal/promptinput`

Implementation:

- Add typed hourly GeneratedText:

  ```go
  type Hourly struct {
      Summary    string `json:"summary"`
      Timing     string `json:"timing"`
      Impacts    string `json:"impacts"`
      Confidence string `json:"confidence,omitempty"`
  }
  ```

- Add validation:
  - malformed JSON fails;
  - empty required fields fail after trimming whitespace;
  - unknown fields fail by using `json.Decoder.DisallowUnknownFields`;
  - validated output is normalized to stable JSON.
- Do not add a Go JSON Schema dependency in this implementation.
- Add typed hourly render context.
- Build render context from:
  - report metadata and location;
  - selected module outputs from the module snapshot;
  - validated hourly GeneratedText.
- Use `module.StanzaValue` or equivalent typed decoding rather than ad hoc map
  traversal where practical.
- Keep render context curated; do not pass the full raw data package to the
  template.

Acceptance criteria:

- valid GeneratedText passes and round-trips to stable JSON;
- missing required fields fail;
- unknown fields fail;
- render context has enough deterministic fields for the hourly template;
- render context construction fails clearly when required module stanzas are
  missing.

Suggested validation:

```bash
go test ./internal/generatedtext ./internal/reporttemplate ./internal/module
```

This stage is small enough for one implementation prompt.

## Stage 6: State Artifacts For GeneratedText Reports

Goal: add explicit managed paths and save/load helpers for generated-text
artifacts.

Files to inspect:

- `internal/state/filesystem.go`
- `internal/state/store.go`
- `internal/state/metadata.go`
- `internal/state/filesystem_test.go`
- `internal/fileutil`

Implementation:

- Extend `state.ArtifactPaths` with:
  - `GeneratedTextRaw`
  - `GeneratedTextResult`
  - `GeneratedText`
  - `RenderContext`
- Store all four under the existing snapshots tree:
  - `snapshots/<artifact_group>/<YYYY-MM-DD>/<run_id>.generated_text.raw.json`
  - `snapshots/<artifact_group>/<YYYY-MM-DD>/<run_id>.generated_text.run.json`
  - `snapshots/<artifact_group>/<YYYY-MM-DD>/<run_id>.generated_text.json`
  - `snapshots/<artifact_group>/<YYYY-MM-DD>/<run_id>.render_context.json`
- Do not add new workspace config directories.
- Extend `state.Metadata` with optional path fields for those artifacts.
  Also record `GeneratedTextSchemaID` as metadata for auditability without
  copying schema contents into every run.
- Add store methods as needed, for example:
  - `SaveGeneratedTextRaw`
  - `SaveGeneratedTextResult`
  - `SaveGeneratedText`
  - `SaveRenderContext`
- Use atomic writes through existing `fileutil` helpers.
- Keep existing metadata JSON shape for Markdown-path reports, with new fields
  omitted when empty.

Acceptance criteria:

- path tests cover all new artifact paths;
- save helpers create parent directories and write atomically;
- metadata links generated-text paths for hourly reports;
- existing report metadata tests still pass.

Suggested validation:

```bash
go test ./internal/state ./internal/fileutil
```

This stage is small enough for one implementation prompt.

## Stage 7: Scriptorium GeneratedText Run Adapter

Goal: add a GeneratedText run helper behind the Scriptorium adapter while
preserving the existing `scriptorium run` command shape.

Files to inspect:

- `internal/adapters/scriptorium/runner.go`
- `internal/adapters/scriptorium/runner_test.go`
- `docs/integrations/scriptorium.md` after implementation

Implementation:

- Add weatherreporter-owned request/result types:

  ```go
  type StructuredRunRequest struct {
      PromptID        string
      DataPackagePath string
      OutputPath      string
  }
  ```

- Add `StructuredRun(ctx, req)` to the runner.
- Validate prompt ID, data package path, and output path before
  subprocess execution.
- Preserve current command result capture behavior:
  - full argv;
  - stdout;
  - stderr;
  - truncation flags;
  - exit code;
  - output path.
- Do not change current `Render` or `Run` methods.
- Scriptorium selects the structured output schema from the backend prompt
  configuration associated with `PromptID`; do not pass a schema path at
  runtime.
- Use the same `scriptorium run` command form as Markdown report generation.
  The only material difference is that `--out` points to the raw GeneratedText
  JSON artifact path rather than a Markdown report path.
- Use this argv shape:

  ```bash
  scriptorium run \
    --prompt <prompt_id> \
    --input data_package=<path> \
    --out <generated_text_raw_path>
  ```

- Do not add `--format json` or any schema flag. Structured output is selected
  by Scriptorium's internal prompt configuration.
- If local Scriptorium documentation needs updating after implementation,
  update `docs/integrations/scriptorium.md` to describe this contract.

Acceptance criteria:

- argv construction tests prove `scriptorium run` is invoked without
  `--format` or schema flags and with `--out <generated_text_raw_path>`;
- nonzero exits return captured result plus error;
- missing required fields fail before subprocess execution;
- current Scriptorium render/run tests still pass;
- tokens/secrets are not added to argv or errors.

Suggested validation:

```bash
go test ./internal/adapters/scriptorium
```

This stage is small enough for one implementation prompt.

## Stage 8: GeneratedText App Workflow Success Path

Goal: wire the successful generated-text/template path into app orchestration
for hourly while preserving the existing path for all current reports.

Files to inspect:

- `internal/app/app.go`
- `internal/app/app_test.go`
- `internal/adapters/scriptorium`
- `internal/reporttemplate`
- `internal/generatedtext`
- `internal/state`
- `internal/promptinput`

Implementation:

- Extend the app `Renderer` interface, or introduce a second narrow interface,
  so tests can fake structured generation.
- Branch report generation by `req.Resolved.Definition.GenerationMode`.
- For `GenerationModeScriptoriumMarkdown`, keep the existing flow unchanged.
- For the successful `GenerationModeGeneratedTextTemplate` path:
  1. run existing preflight and save preflight artifact;
  2. call Scriptorium structured generation with raw output path
     `GeneratedTextRaw`;
  3. save structured Scriptorium command result to `GeneratedTextResult`;
  4. validate raw JSON into typed GeneratedText;
  5. save normalized GeneratedText to `GeneratedText`;
  6. build hourly render context from metadata, module snapshot, and
     GeneratedText;
  7. save render context JSON;
  8. render Markdown via embedded template to managed report path;
  9. save final metadata with all generated-text paths and the schema ID;
- Defer distributor notification and optional output copy regression coverage
  to Stage 10.
- Keep failure-specific persistence tests minimal in this stage; Stage 9 owns
  failure artifact behavior.
- Keep Recent Changes empty for hourly.

Acceptance criteria:

- hourly generation reaches Scriptorium structured generation, validates
  GeneratedText, renders Markdown, and persists final metadata;
- daily/tomorrow/three-day/weekend/storm generation still uses existing
  Markdown Scriptorium path;
- generated-text artifacts are linked from metadata for successful hourly runs;
- app tests cover the successful hourly workflow with fake structured
  Scriptorium output.

Suggested validation:

```bash
go test ./internal/app ./internal/state ./internal/reporttemplate ./internal/generatedtext
```

This stage is small enough for one implementation prompt if failure behavior is
kept to the later stages.

## Stage 9: GeneratedText Failure Artifact Persistence

Goal: make generated-text failures inspectable without changing successful
behavior from Stage 8.

Files to inspect:

- `internal/app/app.go`
- `internal/app/app_test.go`
- `internal/adapters/scriptorium`
- `internal/generatedtext`
- `internal/reporttemplate`
- `internal/state`

Implementation:

- Add focused app tests for failures at each generated-text step:
  - preflight failure;
  - structured Scriptorium generation failure;
  - GeneratedText validation failure;
  - render context construction failure;
  - Markdown template rendering failure.
- Ensure failures preserve available artifacts:
  - failed preflight saves preflight output when possible;
  - failed structured generation saves command result when available;
  - validation failures preserve raw GeneratedText output when available;
  - render-context failures preserve validated GeneratedText when already
    available;
  - template failures preserve GeneratedText and render context when already
    built.
- Return actionable errors that include report ID, RunID, and operation context
  without exposing data package contents or secrets.
- Do not run distributor notification after any generated-text failure.
- Do not create optional `--out` copies after any generated-text failure.

Acceptance criteria:

- every generated-text failure point has an app-level regression test;
- partial artifacts are persisted where available and linked from metadata when
  metadata can be safely written;
- failed hourly runs do not notify distributor;
- existing Markdown-generation failure behavior is unchanged.

Suggested validation:

```bash
go test ./internal/app ./internal/state ./internal/generatedtext ./internal/reporttemplate
```

This stage is small enough for one implementation prompt.

## Stage 10: GeneratedText Output Copy And Notification Regression Coverage

Goal: preserve existing user-facing output behavior and distributor boundaries
for generated-text reports.

Files to inspect:

- `internal/app/app.go`
- `internal/app/app_test.go`
- `internal/adapters/distributor`
- `internal/state`

Implementation:

- Keep `--out` copy behavior unchanged after the managed Markdown report
  exists.
- Keep `--out-dir` behavior unchanged for batch paths if hourly is ever run
  through a batch helper during tests.
- Run distributor notification, if enabled, only after:
  - structured generation succeeds;
  - GeneratedText validates;
  - Markdown template rendering succeeds;
  - final metadata is saved.
- Use the managed Markdown report path as the distributor source.
- Never use optional `--out` or `--out-dir` copies as distributor source files.
- Keep distributor request construction template-driven; do not add
  hourly-specific path branching.
- Keep Recent Changes empty for hourly in this implementation.

Acceptance criteria:

- `--out` copies the managed generated-text Markdown report after success;
- distributor notification receives the managed Markdown report path;
- distributor notification is not called on any failure from Stage 9;
- enabled distributor failure makes hourly generation fail with the same
  semantics as existing generated reports;
- existing daily/tomorrow/three-day/weekend/storm distributor behavior is
  unchanged.

Suggested validation:

```bash
go test ./internal/app ./internal/adapters/distributor ./internal/state
```

This stage is small enough for one implementation prompt.

## Stage 11: End-To-End CLI, Config, And Prompt Package Coverage

Goal: prove the public hourly workflow and configured module overrides work
through the CLI/app boundary.

Files to inspect:

- `internal/cli/root.go`
- `internal/cli/root_test.go`
- `internal/config/reports.go`
- `internal/config/config_test.go`
- `internal/app/app_test.go`
- `examples/config.yml`

Implementation:

- Add CLI command tests for `generate hourly`.
- Reject the retired rolling-report command name.
- Reject hourly `--date`, `--start`, `--end`, and duration-like flags.
- Add config override tests for `reports.hourly.deterministic_modules`.
- Ensure retired rolling-report config keys are rejected.
- Add app/CLI workflow tests using fake Scriptorium structured output.
- Update examples only if they enumerate report module overrides.

Acceptance criteria:

- `weatherreporter --help` lists `generate hourly`;
- the retired rolling-report command name is not accepted;
- hourly config overrides validate compatible modules;
- incompatible daily/daypart modules fail clearly for hourly;
- examples load successfully.

Suggested validation:

```bash
go test ./internal/cli ./internal/config ./internal/app
go run ./cmd/weatherreporter --help
```

This stage is small enough for one implementation prompt.

## Stage 12: Implemented Documentation

Goal: document implemented behavior after the code exists.

Files to inspect and update:

- `docs/cli.md`
- `docs/operations.md`
- `docs/config.md`
- `docs/internal/app-orchestration.md`
- `docs/internal/report-registry.md`
- `docs/internal/briefing.md`
- `docs/internal/prompt-input.md`
- `docs/internal/scriptorium-adapter.md`
- `docs/internal/state.md`
- new `docs/internal/reporttemplate.md`
- `docs/integrations/scriptorium.md`
- `docs/roadmap/future.md`
- the superseded roadmap file for the retired rolling-report name
- `examples/config.yml`, only if examples changed

Documentation requirements:

- Non-roadmap docs describe only implemented hourly and GeneratedText behavior.
- `docs/cli.md` documents `generate hourly`, not the retired command name.
- `docs/config.md` documents `reports.hourly` only if override support is
  implemented.
- Internal docs distinguish:
  - CollectedFacts;
  - DerivedFacts;
  - ModuleOutput;
  - GeneratedText;
  - RenderContext.
- Scriptorium integration docs include the structured-output command actually
  used by the adapter.
- State docs list generated-text artifacts and metadata links.
- the superseded roadmap file for the retired rolling-report name is deleted
  or marked superseded.
- Deferred migrations for other report types stay under roadmap docs only.

Acceptance criteria:

- No non-roadmap docs describe unimplemented generated-text migrations.
- No active docs describe retired rolling-report identifiers as the target
  public report.
- Maintained examples remain valid.

Suggested validation:

```bash
go test ./internal/config
rg -n 'near[_-]term|Near''Term' docs internal examples
git diff --check
```

This stage is small enough for one implementation prompt.

## Stage 13: Final Validation

Goal: run full validation and perform targeted manual checks.

Commands:

```bash
go test ./...
go run ./cmd/weatherreporter --help
git diff --check
```

Manual checks:

- `weatherreporter --help` lists `generate hourly`.
- `weatherreporter --help` does not list the retired rolling-report command.
- generated hourly Markdown has deterministic headings and section order.
- GeneratedText JSON is persisted separately from deterministic facts.
- render context JSON is persisted and readable.
- embedded hourly template is readable and easy to edit.
- embedded hourly schema is readable and easy to edit.
- existing Markdown-generation reports still run through the previous
  Scriptorium `run --out <markdown>` path.
- distributor notification, when enabled, uploads the managed Markdown report.

This stage is small enough for one implementation prompt.

## Deferred Work

Do not include these in the first implementation:

- migrating daily, tomorrow, three-day, weekend, or storm to GeneratedText;
- user-configurable template or local schema source overrides;
- user-defined templates or schemas;
- a generic workflow engine;
- template hot reloading;
- dynamic module selection for templates;
- prompt/schema generation from Go structs;
- scheduled hourly batch membership;
- configurable hourly report duration;
- rolling-window Recent Changes for hourly;
- distributor-specific generated-text behavior.

## Open Questions

None block implementation.