Update batch collection documentation

This commit is contained in:
2026-06-17 16:11:09 +00:00
parent 3eccafad6b
commit b913194fb4
5 changed files with 178 additions and 86 deletions

View File

@@ -5,11 +5,12 @@ This document describes the workflow coordinator in `internal/app`.
## Purpose
`internal/app` coordinates the top-level use cases after CLI parsing and config
loading are complete. It resolves report definitions, fetches weather data,
builds collected and derived facts, builds module snapshots and prompt-input
artifacts, invokes Scriptorium through the adapter boundary, optionally
notifies distributor through an app-owned notifier boundary, persists managed
state, runs batches, and reads existing artifacts for inspection.
loading are complete. It resolves report definitions, collects weather data
through `internal/collect`, builds collected and derived facts, builds module
snapshots and prompt-input artifacts, invokes Scriptorium through the adapter
boundary, optionally notifies distributor through an app-owned notifier
boundary, persists managed state, runs batches, and reads existing artifacts
for inspection.
## Inputs And Outputs
@@ -20,9 +21,9 @@ Inputs:
- `FetchBundleRequest` for explicit bundle fetch and save workflows
- `ReportRequest` for single-report generation
- resolved report definitions from `internal/report`
- weather data bundles from `internal/adapters/weatherapi`
- collection results from `internal/collect`
- prior snapshots loaded from `internal/state`
- optional renderer, notifier, and state-store fakes for tests
- optional collector, renderer, notifier, and state-store fakes for tests
Outputs:
@@ -45,15 +46,18 @@ construct Scriptorium argv.
Report selection and report identity policy come from `internal/report`.
Collected and derived fact contracts come from `internal/facts`.
Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium
subprocess behavior stays in `internal/adapters/scriptorium`. Distributor
upload behavior stays in `internal/adapters/distributor`. Filesystem layout and
persisted metadata stay in `internal/state`.
Weather API transport stays in `internal/adapters/weatherapi`, and app-facing
upstream collection stays in `internal/collect`. Scriptorium subprocess
behavior stays in `internal/adapters/scriptorium`. Distributor upload behavior
stays in `internal/adapters/distributor`. Filesystem layout and persisted
metadata stay in `internal/state`.
## Data Flow Terms
- `CollectedFacts` are normalized source facts fetched once from Weather API
and made available to derivation and module builders.
- `collect.Result` is the app-facing upstream collection result. It carries the
normalized `weatherdata.Bundle` used by report generation.
- `CollectedFacts` are normalized source facts derived from a collected Weather
API bundle and made available to derivation and module builders.
- `DerivedFacts` are deterministic calculations over collected facts, the
resolved valid period, daypart configuration, and report-specific windows.
- `module.Output` values are ordered deterministic stanzas built from collected
@@ -77,33 +81,35 @@ defaults.
## Generation Workflow
Single-report generation shares this setup:
Single-report commands validate the report command, collect once through
`internal/collect`, resolve the requested report, and pass the resolved report
plus explicit collection into `GenerateReport`.
1. Resolve the command report to a `report.Resolved` value.
2. Create or use a filesystem store.
3. Locate any prior compatible snapshot through `internal/state`.
4. Fetch a Weather API bundle.
5. Build collected and derived facts once.
6. Execute configured modules and save the module snapshot.
7. Compute Recent Changes from structured prior and current module snapshots.
8. Build and save the YAML Scriptorium `data_package`.
9. Run Scriptorium render preflight.
10. Save preflight JSON when a render result is available.
11. Save metadata for inspection.
`GenerateReport` then uses this setup:
1. Create or use a filesystem store.
2. Locate any prior compatible snapshot through `internal/state`.
3. Build collected and derived facts from the supplied collection.
4. Execute configured modules and save the module snapshot.
5. Compute Recent Changes from structured prior and current module snapshots.
6. Build and save the YAML Scriptorium `data_package`.
7. Run Scriptorium render preflight.
8. Save preflight JSON when a render result is available.
9. Save metadata for inspection.
For `scriptorium_markdown` reports, generation then:
12. Runs Scriptorium report generation to the managed report path.
10. Runs Scriptorium report generation to the managed report path.
For `generated_text_template` reports, generation then:
12. Looks up the generated-text catalog entry for the report schema/template
10. Looks up the generated-text catalog entry for the report schema/template
IDs.
13. Runs structured Scriptorium generation to the raw generated-text JSON path.
14. Saves the structured Scriptorium run result.
15. Validates and saves normalized generated text.
16. Builds and saves a typed render context.
17. Renders Markdown from the embedded template to the managed report path.
11. Runs structured Scriptorium generation to the raw generated-text JSON path.
12. Saves the structured Scriptorium run result.
13. Validates and saves normalized generated text.
14. Builds and saves a typed render context.
15. Renders Markdown from the embedded template to the managed report path.
After either mode has produced a managed Markdown report, shared finalization:
@@ -119,7 +125,7 @@ After either mode has produced a managed Markdown report, shared finalization:
If render preflight returns both a result and an error, preflight JSON and
metadata are persisted before the error is returned. If Scriptorium report
generation returns an error after writing output, the managed report and
metadata remain inspectable. Notification is not attempted after Weather API,
metadata remain inspectable. Notification is not attempted after collection,
module snapshot, prompt input, render, Scriptorium run, or metadata-save
failures.
Generated-text report failures are returned with report ID, RunID, and the
@@ -134,17 +140,21 @@ failure context.
## Batch Workflow
`run morning` resolves Today Report, 3-Day Outlook, and Weekend Outlook except
on Sunday. `run evening` resolves Tomorrow Report. Daily Report is generated
only through `generate daily --date YYYY-MM-DD`; it is not part of scheduled
batches. Batch output copy names come from report definitions. Batch generation
continues independent reports after a failure, records each result, writes
compact status lines to stderr, emits a JSON summary to stdout, and returns an
aggregate error when any report failed. When notification is enabled, each
successfully generated report is notified independently. Notification failure
marks that report failed, records notification fields in the batch result, and
does not stop later reports. `--out-dir` copies are never used as notification
source files.
`run morning` collects once, plans Today Report, Tomorrow Report, and eligible
future Daily Reports from the collected hourly forecast, then passes the same
collection into each report generation. `run evening` uses the same collection
and planning rules, but starts with Tomorrow Report. Future Daily reports start
with the day after tomorrow and require complete hourly forecast coverage for
the target local civil day. Dynamic Daily `--out-dir` copies use
`daily-YYYY-MM-DD.md`; other batch copies use report definition output names.
A collection failure stops the batch before planning or report generation.
After planning succeeds, batch generation continues independent reports after a
failure, records each result, writes compact status lines to stderr, emits a
JSON summary to stdout, and returns an aggregate error when any report failed.
When notification is enabled, each successfully generated report is notified
independently. Notification failure marks that report failed, records
notification fields in the batch result, and does not stop later reports.
`--out-dir` copies are never used as notification source files.
## Inspection Workflow
@@ -155,8 +165,8 @@ inspection view.
## Failure Behavior
- Resolve errors stop the requested workflow before fetching weather data.
- Weather API and module execution errors stop that report before Scriptorium
- Resolve errors stop the requested workflow before collection.
- Collection and module execution errors stop that report before Scriptorium
runs.
- Prompt input validation fails before render preflight.
- Render and run errors preserve Scriptorium stderr and exit-code context.
@@ -173,12 +183,17 @@ inspection view.
Inspect:
- `internal/app/app_test.go`
- `internal/app/batch_plan_test.go`
- `internal/collect/collect_test.go`
- `internal/cli/root_test.go`
- `internal/state/filesystem_test.go`
## Invariants
- Report behavior is resolved through `internal/report`.
- Generate and run commands collect once before report generation.
- Batch planning is app-owned because future Daily membership depends on
collected hourly forecast coverage.
- Generated reports use the same app request and result types regardless of
report ID.
- Render preflight precedes Scriptorium report generation.

58
docs/internal/collect.md Normal file
View File

@@ -0,0 +1,58 @@
# Collection Internals
This document describes the app-facing upstream collection boundary in
`internal/collect`.
## Purpose
`internal/collect` is the canonical package used by app workflows to collect
upstream Weather API data. It constructs the Weather API adapter, fetches a
normalized bundle, and returns that bundle without applying report selection or
batch policy.
## Contract
Inputs:
- `collect.Request`, containing the effective `config.Config`
- `context.Context` for cancellation
Output:
- `collect.Result`, containing `*weatherdata.Bundle`
`Run` returns an actionable error when Weather API adapter construction or
bundle fetch fails. The package does not derive `facts.CollectedFacts`, build
modules, resolve report periods, select reports, write state, invoke
Scriptorium, or notify distributor.
## App Usage
`internal/app` owns a narrow `Collector` interface for orchestration tests. The
default implementation calls `collect.Run`.
Single-report generation collects once, resolves the requested report, and
passes the explicit collection into report generation. Batch generation
collects once before planning and passes the same collection into each planned
report. If collection returns no bundle, app orchestration returns an error
before report generation.
## Boundaries
Weather API HTTP details stay in `internal/adapters/weatherapi`. The collection
package returns normalized `weatherdata` only. It must not know about report
IDs, prompt IDs, batch names, Daily eligibility, module composition, Recent
Changes, state paths, or Scriptorium arguments.
## Tests
Inspect:
- `internal/collect/collect_test.go`
- `internal/app/app_test.go`
## Invariants
- App-facing Weather API collection goes through `internal/collect`.
- Collection returns normalized source data, not report facts or prompt input.
- Report and batch policy belongs outside `internal/collect`.

View File

@@ -1,7 +1,7 @@
# Report Registry Internals
This document describes report identity, valid-period resolution, batch
membership, output naming, artifact grouping, and comparison declarations in
This document describes report identity, valid-period resolution, output
naming, artifact grouping, batch command names, and comparison declarations in
`internal/report`.
## Purpose
@@ -24,7 +24,6 @@ Each report definition declares:
- batch output copy filename
- generated-report eligibility
- prior-report compatibility list
- morning or evening batch membership
- default ordered module composition
Report-owned helpers map public command names and config keys to report IDs.
@@ -69,8 +68,7 @@ All report definitions are eligible for generation.
not a configuration field.
- 3-Day Outlook covers the interval from generation time through local midnight
three days later.
- Weekend Outlook covers the upcoming weekend window and is not scheduled for
Sunday morning batch resolution.
- Weekend Outlook covers the upcoming weekend window.
- Storm Report covers an explicit event window supplied by the caller.
Storm event windows can be parsed from local `YYYY-MM-DDTHH:MM` timestamps in
@@ -80,8 +78,9 @@ must be after start time.
## Boundaries
`internal/report` defines report metadata, public report names, batch command
names, and time coverage. It does not fetch weather data, build module values,
compare snapshot contents, write state, parse CLI flags, or invoke Scriptorium.
names, output naming, and time coverage. It does not fetch weather data, plan
batch membership, build module values, compare snapshot contents, write state,
parse CLI flags, or invoke Scriptorium.
The CLI parses flags and command structure, then uses report-owned helpers for
report and batch command names. Config loading uses report-owned helpers for
@@ -94,11 +93,16 @@ output path copying uses batch output names from report definitions. Report
module overrides can use short keys such as `daily`, `today`, `tomorrow`, and
`hourly`, or descriptive names such as `three_day_outlook`.
## Batch Membership
## Batch Commands
Morning batches include Today Report, 3-Day Outlook, and Weekend Outlook
except on Sunday. Evening batches include Tomorrow Report. Daily Report and
Hourly Report are not part of a scheduled batch.
`internal/report` owns the public batch command names `morning` and `evening`
and validates them through `BatchForCommandName`. Data-dependent batch
membership is owned by `internal/app`, because it depends on collected hourly
forecast coverage.
Report definitions still declare default batch output copy filenames. App
batch planning uses those filenames for fixed report entries and supplies
date-qualified names for dynamic Daily entries.
## State And App Usage
@@ -135,3 +139,5 @@ Inspect:
- Artifact grouping, batch output filenames, generated-report eligibility,
default module composition, comparison compatibility, and comparison strategy
are declared by report definition.
- App-owned batch planning uses report definitions but does not live in the
report registry.