Update batch collection documentation

This commit is contained in:
2026-06-17 16:11:09 +00:00
parent 3eccafad6b
commit b913194fb4
5 changed files with 178 additions and 86 deletions

View File

@@ -45,23 +45,29 @@ notification uses the managed report path, not the extra copy. `generate daily`,
generated-text artifacts, validate structured text from Scriptorium, and render generated-text artifacts, validate structured text from Scriptorium, and render
the managed Markdown report from embedded templates. `generate daily` requires the managed Markdown report from embedded templates. `generate daily` requires
`--date YYYY-MM-DD` for the selected local civil day; omitting `--date` is a `--date YYYY-MM-DD` for the selected local civil day; omitting `--date` is a
command error and stops before weather data is fetched. `generate hourly` command error and stops before weather data is collected. `generate hourly`
covers the next six hours in the effective report timezone and does not accept covers the next six hours in the effective report timezone and does not accept
date or event window flags. `generate storm` requires explicit event-window date or event window flags. `generate storm` requires explicit event-window
bounds with `--start` and `--end`. bounds with `--start` and `--end`.
`run morning` generates Today Report and the 3-Day Outlook, plus Weekend Outlook `run morning` generates Today Report, Tomorrow Report, and a dated Daily Report
except on Sunday. `run evening` generates the Tomorrow Report. Batch for each later future local civil day with complete hourly forecast coverage.
runs continue independent reports after a failure, print a JSON summary to `run evening` generates Tomorrow Report and the same eligible future Daily
stdout, write compact status lines to stderr, and return nonzero when any report reports. Future Daily expansion starts with the day after tomorrow and skips
failed. `--out-dir` writes extra Markdown copies for the operator; distributor days that do not have every hourly forecast period for the local civil day.
notification uses each managed report path, not the extra copies. When Batch commands collect weather data once before planning; a collection failure
notification is enabled, batch summaries and status lines include notification stops the batch before any report is generated. Batch runs continue independent
status, accepted distributor run ID, or notification error fields for each reports after a later report failure, print a JSON summary to stdout, write
attempted report. compact status lines to stderr, and return nonzero when any report failed.
`--out-dir` writes extra Markdown copies for the operator; distributor
notification uses each managed report path, not the extra copies. Today and
Tomorrow use their report default copy names, and dynamic Daily copies use
`daily-YYYY-MM-DD.md`. When notification is enabled, batch summaries and status
lines include notification status, accepted distributor run ID, or notification
error fields for each attempted report.
Hourly Report is explicit only; it is not included in `run morning` or `run Hourly Report, 3-Day Outlook, and Weekend Outlook are explicit only; they are
evening`. not included in `run morning` or `run evening`.
`inspect` commands read existing workspace artifacts and emit JSON to stdout. `inspect` commands read existing workspace artifacts and emit JSON to stdout.
They do not fetch weather data or invoke `scriptorium`. They do not fetch weather data or invoke `scriptorium`.

View File

@@ -5,11 +5,12 @@ This document describes the workflow coordinator in `internal/app`.
## Purpose ## Purpose
`internal/app` coordinates the top-level use cases after CLI parsing and config `internal/app` coordinates the top-level use cases after CLI parsing and config
loading are complete. It resolves report definitions, fetches weather data, loading are complete. It resolves report definitions, collects weather data
builds collected and derived facts, builds module snapshots and prompt-input through `internal/collect`, builds collected and derived facts, builds module
artifacts, invokes Scriptorium through the adapter boundary, optionally snapshots and prompt-input artifacts, invokes Scriptorium through the adapter
notifies distributor through an app-owned notifier boundary, persists managed boundary, optionally notifies distributor through an app-owned notifier
state, runs batches, and reads existing artifacts for inspection. boundary, persists managed state, runs batches, and reads existing artifacts
for inspection.
## Inputs And Outputs ## Inputs And Outputs
@@ -20,9 +21,9 @@ Inputs:
- `FetchBundleRequest` for explicit bundle fetch and save workflows - `FetchBundleRequest` for explicit bundle fetch and save workflows
- `ReportRequest` for single-report generation - `ReportRequest` for single-report generation
- resolved report definitions from `internal/report` - resolved report definitions from `internal/report`
- weather data bundles from `internal/adapters/weatherapi` - collection results from `internal/collect`
- prior snapshots loaded from `internal/state` - prior snapshots loaded from `internal/state`
- optional renderer, notifier, and state-store fakes for tests - optional collector, renderer, notifier, and state-store fakes for tests
Outputs: Outputs:
@@ -45,15 +46,18 @@ construct Scriptorium argv.
Report selection and report identity policy come from `internal/report`. Report selection and report identity policy come from `internal/report`.
Collected and derived fact contracts come from `internal/facts`. Collected and derived fact contracts come from `internal/facts`.
Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium Weather API transport stays in `internal/adapters/weatherapi`, and app-facing
subprocess behavior stays in `internal/adapters/scriptorium`. Distributor upstream collection stays in `internal/collect`. Scriptorium subprocess
upload behavior stays in `internal/adapters/distributor`. Filesystem layout and behavior stays in `internal/adapters/scriptorium`. Distributor upload behavior
persisted metadata stay in `internal/state`. stays in `internal/adapters/distributor`. Filesystem layout and persisted
metadata stay in `internal/state`.
## Data Flow Terms ## Data Flow Terms
- `CollectedFacts` are normalized source facts fetched once from Weather API - `collect.Result` is the app-facing upstream collection result. It carries the
and made available to derivation and module builders. normalized `weatherdata.Bundle` used by report generation.
- `CollectedFacts` are normalized source facts derived from a collected Weather
API bundle and made available to derivation and module builders.
- `DerivedFacts` are deterministic calculations over collected facts, the - `DerivedFacts` are deterministic calculations over collected facts, the
resolved valid period, daypart configuration, and report-specific windows. resolved valid period, daypart configuration, and report-specific windows.
- `module.Output` values are ordered deterministic stanzas built from collected - `module.Output` values are ordered deterministic stanzas built from collected
@@ -77,33 +81,35 @@ defaults.
## Generation Workflow ## Generation Workflow
Single-report generation shares this setup: Single-report commands validate the report command, collect once through
`internal/collect`, resolve the requested report, and pass the resolved report
plus explicit collection into `GenerateReport`.
1. Resolve the command report to a `report.Resolved` value. `GenerateReport` then uses this setup:
2. Create or use a filesystem store.
3. Locate any prior compatible snapshot through `internal/state`. 1. Create or use a filesystem store.
4. Fetch a Weather API bundle. 2. Locate any prior compatible snapshot through `internal/state`.
5. Build collected and derived facts once. 3. Build collected and derived facts from the supplied collection.
6. Execute configured modules and save the module snapshot. 4. Execute configured modules and save the module snapshot.
7. Compute Recent Changes from structured prior and current module snapshots. 5. Compute Recent Changes from structured prior and current module snapshots.
8. Build and save the YAML Scriptorium `data_package`. 6. Build and save the YAML Scriptorium `data_package`.
9. Run Scriptorium render preflight. 7. Run Scriptorium render preflight.
10. Save preflight JSON when a render result is available. 8. Save preflight JSON when a render result is available.
11. Save metadata for inspection. 9. Save metadata for inspection.
For `scriptorium_markdown` reports, generation then: For `scriptorium_markdown` reports, generation then:
12. Runs Scriptorium report generation to the managed report path. 10. Runs Scriptorium report generation to the managed report path.
For `generated_text_template` reports, generation then: For `generated_text_template` reports, generation then:
12. Looks up the generated-text catalog entry for the report schema/template 10. Looks up the generated-text catalog entry for the report schema/template
IDs. IDs.
13. Runs structured Scriptorium generation to the raw generated-text JSON path. 11. Runs structured Scriptorium generation to the raw generated-text JSON path.
14. Saves the structured Scriptorium run result. 12. Saves the structured Scriptorium run result.
15. Validates and saves normalized generated text. 13. Validates and saves normalized generated text.
16. Builds and saves a typed render context. 14. Builds and saves a typed render context.
17. Renders Markdown from the embedded template to the managed report path. 15. Renders Markdown from the embedded template to the managed report path.
After either mode has produced a managed Markdown report, shared finalization: After either mode has produced a managed Markdown report, shared finalization:
@@ -119,7 +125,7 @@ After either mode has produced a managed Markdown report, shared finalization:
If render preflight returns both a result and an error, preflight JSON and If render preflight returns both a result and an error, preflight JSON and
metadata are persisted before the error is returned. If Scriptorium report metadata are persisted before the error is returned. If Scriptorium report
generation returns an error after writing output, the managed report and generation returns an error after writing output, the managed report and
metadata remain inspectable. Notification is not attempted after Weather API, metadata remain inspectable. Notification is not attempted after collection,
module snapshot, prompt input, render, Scriptorium run, or metadata-save module snapshot, prompt input, render, Scriptorium run, or metadata-save
failures. failures.
Generated-text report failures are returned with report ID, RunID, and the Generated-text report failures are returned with report ID, RunID, and the
@@ -134,17 +140,21 @@ failure context.
## Batch Workflow ## Batch Workflow
`run morning` resolves Today Report, 3-Day Outlook, and Weekend Outlook except `run morning` collects once, plans Today Report, Tomorrow Report, and eligible
on Sunday. `run evening` resolves Tomorrow Report. Daily Report is generated future Daily Reports from the collected hourly forecast, then passes the same
only through `generate daily --date YYYY-MM-DD`; it is not part of scheduled collection into each report generation. `run evening` uses the same collection
batches. Batch output copy names come from report definitions. Batch generation and planning rules, but starts with Tomorrow Report. Future Daily reports start
continues independent reports after a failure, records each result, writes with the day after tomorrow and require complete hourly forecast coverage for
compact status lines to stderr, emits a JSON summary to stdout, and returns an the target local civil day. Dynamic Daily `--out-dir` copies use
aggregate error when any report failed. When notification is enabled, each `daily-YYYY-MM-DD.md`; other batch copies use report definition output names.
successfully generated report is notified independently. Notification failure A collection failure stops the batch before planning or report generation.
marks that report failed, records notification fields in the batch result, and After planning succeeds, batch generation continues independent reports after a
does not stop later reports. `--out-dir` copies are never used as notification failure, records each result, writes compact status lines to stderr, emits a
source files. JSON summary to stdout, and returns an aggregate error when any report failed.
When notification is enabled, each successfully generated report is notified
independently. Notification failure marks that report failed, records
notification fields in the batch result, and does not stop later reports.
`--out-dir` copies are never used as notification source files.
## Inspection Workflow ## Inspection Workflow
@@ -155,8 +165,8 @@ inspection view.
## Failure Behavior ## Failure Behavior
- Resolve errors stop the requested workflow before fetching weather data. - Resolve errors stop the requested workflow before collection.
- Weather API and module execution errors stop that report before Scriptorium - Collection and module execution errors stop that report before Scriptorium
runs. runs.
- Prompt input validation fails before render preflight. - Prompt input validation fails before render preflight.
- Render and run errors preserve Scriptorium stderr and exit-code context. - Render and run errors preserve Scriptorium stderr and exit-code context.
@@ -173,12 +183,17 @@ inspection view.
Inspect: Inspect:
- `internal/app/app_test.go` - `internal/app/app_test.go`
- `internal/app/batch_plan_test.go`
- `internal/collect/collect_test.go`
- `internal/cli/root_test.go` - `internal/cli/root_test.go`
- `internal/state/filesystem_test.go` - `internal/state/filesystem_test.go`
## Invariants ## Invariants
- Report behavior is resolved through `internal/report`. - Report behavior is resolved through `internal/report`.
- Generate and run commands collect once before report generation.
- Batch planning is app-owned because future Daily membership depends on
collected hourly forecast coverage.
- Generated reports use the same app request and result types regardless of - Generated reports use the same app request and result types regardless of
report ID. report ID.
- Render preflight precedes Scriptorium report generation. - Render preflight precedes Scriptorium report generation.

58
docs/internal/collect.md Normal file
View File

@@ -0,0 +1,58 @@
# Collection Internals
This document describes the app-facing upstream collection boundary in
`internal/collect`.
## Purpose
`internal/collect` is the canonical package used by app workflows to collect
upstream Weather API data. It constructs the Weather API adapter, fetches a
normalized bundle, and returns that bundle without applying report selection or
batch policy.
## Contract
Inputs:
- `collect.Request`, containing the effective `config.Config`
- `context.Context` for cancellation
Output:
- `collect.Result`, containing `*weatherdata.Bundle`
`Run` returns an actionable error when Weather API adapter construction or
bundle fetch fails. The package does not derive `facts.CollectedFacts`, build
modules, resolve report periods, select reports, write state, invoke
Scriptorium, or notify distributor.
## App Usage
`internal/app` owns a narrow `Collector` interface for orchestration tests. The
default implementation calls `collect.Run`.
Single-report generation collects once, resolves the requested report, and
passes the explicit collection into report generation. Batch generation
collects once before planning and passes the same collection into each planned
report. If collection returns no bundle, app orchestration returns an error
before report generation.
## Boundaries
Weather API HTTP details stay in `internal/adapters/weatherapi`. The collection
package returns normalized `weatherdata` only. It must not know about report
IDs, prompt IDs, batch names, Daily eligibility, module composition, Recent
Changes, state paths, or Scriptorium arguments.
## Tests
Inspect:
- `internal/collect/collect_test.go`
- `internal/app/app_test.go`
## Invariants
- App-facing Weather API collection goes through `internal/collect`.
- Collection returns normalized source data, not report facts or prompt input.
- Report and batch policy belongs outside `internal/collect`.

View File

@@ -1,7 +1,7 @@
# Report Registry Internals # Report Registry Internals
This document describes report identity, valid-period resolution, batch This document describes report identity, valid-period resolution, output
membership, output naming, artifact grouping, and comparison declarations in naming, artifact grouping, batch command names, and comparison declarations in
`internal/report`. `internal/report`.
## Purpose ## Purpose
@@ -24,7 +24,6 @@ Each report definition declares:
- batch output copy filename - batch output copy filename
- generated-report eligibility - generated-report eligibility
- prior-report compatibility list - prior-report compatibility list
- morning or evening batch membership
- default ordered module composition - default ordered module composition
Report-owned helpers map public command names and config keys to report IDs. Report-owned helpers map public command names and config keys to report IDs.
@@ -69,8 +68,7 @@ All report definitions are eligible for generation.
not a configuration field. not a configuration field.
- 3-Day Outlook covers the interval from generation time through local midnight - 3-Day Outlook covers the interval from generation time through local midnight
three days later. three days later.
- Weekend Outlook covers the upcoming weekend window and is not scheduled for - Weekend Outlook covers the upcoming weekend window.
Sunday morning batch resolution.
- Storm Report covers an explicit event window supplied by the caller. - Storm Report covers an explicit event window supplied by the caller.
Storm event windows can be parsed from local `YYYY-MM-DDTHH:MM` timestamps in Storm event windows can be parsed from local `YYYY-MM-DDTHH:MM` timestamps in
@@ -80,8 +78,9 @@ must be after start time.
## Boundaries ## Boundaries
`internal/report` defines report metadata, public report names, batch command `internal/report` defines report metadata, public report names, batch command
names, and time coverage. It does not fetch weather data, build module values, names, output naming, and time coverage. It does not fetch weather data, plan
compare snapshot contents, write state, parse CLI flags, or invoke Scriptorium. batch membership, build module values, compare snapshot contents, write state,
parse CLI flags, or invoke Scriptorium.
The CLI parses flags and command structure, then uses report-owned helpers for The CLI parses flags and command structure, then uses report-owned helpers for
report and batch command names. Config loading uses report-owned helpers for report and batch command names. Config loading uses report-owned helpers for
@@ -94,11 +93,16 @@ output path copying uses batch output names from report definitions. Report
module overrides can use short keys such as `daily`, `today`, `tomorrow`, and module overrides can use short keys such as `daily`, `today`, `tomorrow`, and
`hourly`, or descriptive names such as `three_day_outlook`. `hourly`, or descriptive names such as `three_day_outlook`.
## Batch Membership ## Batch Commands
Morning batches include Today Report, 3-Day Outlook, and Weekend Outlook `internal/report` owns the public batch command names `morning` and `evening`
except on Sunday. Evening batches include Tomorrow Report. Daily Report and and validates them through `BatchForCommandName`. Data-dependent batch
Hourly Report are not part of a scheduled batch. membership is owned by `internal/app`, because it depends on collected hourly
forecast coverage.
Report definitions still declare default batch output copy filenames. App
batch planning uses those filenames for fixed report entries and supplies
date-qualified names for dynamic Daily entries.
## State And App Usage ## State And App Usage
@@ -135,3 +139,5 @@ Inspect:
- Artifact grouping, batch output filenames, generated-report eligibility, - Artifact grouping, batch output filenames, generated-report eligibility,
default module composition, comparison compatibility, and comparison strategy default module composition, comparison compatibility, and comparison strategy
are declared by report definition. are declared by report definition.
- App-owned batch planning uses report definitions but does not live in the
report registry.

View File

@@ -18,11 +18,11 @@ weatherreporter generate weekend
weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00 weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00
``` ```
Generation commands resolve a report period, fetch a Weather API bundle, build Generation commands resolve a report period, collect a Weather API bundle,
a rich JSON module snapshot, build a curated YAML prompt input data package, run build a rich JSON module snapshot, build a curated YAML prompt input data
`scriptorium render`, and write managed artifacts under the configured package, run `scriptorium render`, and write managed artifacts under the
workspace. Markdown-path reports then run `scriptorium run` directly to the configured workspace. Markdown-path reports then run `scriptorium run` directly
managed Markdown report path. to the managed Markdown report path.
`generate daily`, `generate today`, `generate tomorrow`, and `generate hourly` `generate daily`, `generate today`, `generate tomorrow`, and `generate hourly`
use the generated-text-template workflow. They run structured `scriptorium run` use the generated-text-template workflow. They run structured `scriptorium run`
@@ -31,8 +31,8 @@ and render the managed Markdown report from embedded templates. `generate
daily` requires `--date YYYY-MM-DD` for the selected local civil day. daily` requires `--date YYYY-MM-DD` for the selected local civil day.
`generate today` covers the selected or current local civil day. `generate `generate today` covers the selected or current local civil day. `generate
hourly` covers the six-hour rolling period from generation time in the hourly` covers the six-hour rolling period from generation time in the
effective report timezone and is not part of scheduled morning or evening effective report timezone and is not included in `run morning` or
batches. `run evening`.
When distributor notification is enabled, weatherreporter uploads the managed When distributor notification is enabled, weatherreporter uploads the managed
Markdown report after report rendering succeeds and final metadata is saved. Markdown report after report rendering succeeds and final metadata is saved.
@@ -46,16 +46,23 @@ weatherreporter run morning
weatherreporter run evening weatherreporter run evening
``` ```
`run morning` generates Today Report and the 3-Day Outlook, plus Weekend Outlook `run morning` generates Today Report, Tomorrow Report, and a dated Daily Report
except on Sunday. `run evening` generates the Tomorrow Report. Batch for each later future local civil day with complete hourly forecast coverage.
commands print a JSON summary to stdout, write compact per-report status lines `run evening` generates Tomorrow Report and the same eligible future Daily
to stderr, continue independent reports after one report fails, and return reports. Future Daily expansion starts with the day after tomorrow. A Daily
nonzero when any report failed. When notification is configured, the summary and report is eligible only when the collected hourly forecast contains every
status lines include notification status, accepted distributor run ID, or hourly period for that local civil day; partial days are skipped. Batch commands
notification error fields for each attempted report. `--out-dir PATH` writes collect weather data once before planning, and a collection failure stops the
extra Markdown copies using report default filenames such as `today.md`, batch before any report is generated.
`three-day.md`, `weekend.md`, and `tomorrow.md`; these copies are not used as
distributor upload sources. After planning succeeds, batch commands print a JSON summary to stdout, write
compact per-report status lines to stderr, continue independent reports after
one report fails, and return nonzero when any report failed. When notification
is configured, the summary and status lines include notification status,
accepted distributor run ID, or notification error fields for each attempted
report. `--out-dir PATH` writes extra Markdown copies using report default
filenames such as `today.md` and `tomorrow.md`; dynamic Daily copies use
`daily-YYYY-MM-DD.md`. These copies are not used as distributor upload sources.
## Filesystem Layout ## Filesystem Layout