Update batch collection documentation
This commit is contained in:
30
docs/cli.md
30
docs/cli.md
@@ -45,23 +45,29 @@ notification uses the managed report path, not the extra copy. `generate daily`,
|
||||
generated-text artifacts, validate structured text from Scriptorium, and render
|
||||
the managed Markdown report from embedded templates. `generate daily` requires
|
||||
`--date YYYY-MM-DD` for the selected local civil day; omitting `--date` is a
|
||||
command error and stops before weather data is fetched. `generate hourly`
|
||||
command error and stops before weather data is collected. `generate hourly`
|
||||
covers the next six hours in the effective report timezone and does not accept
|
||||
date or event window flags. `generate storm` requires explicit event-window
|
||||
bounds with `--start` and `--end`.
|
||||
|
||||
`run morning` generates Today Report and the 3-Day Outlook, plus Weekend Outlook
|
||||
except on Sunday. `run evening` generates the Tomorrow Report. Batch
|
||||
runs continue independent reports after a failure, print a JSON summary to
|
||||
stdout, write compact status lines to stderr, and return nonzero when any report
|
||||
failed. `--out-dir` writes extra Markdown copies for the operator; distributor
|
||||
notification uses each managed report path, not the extra copies. When
|
||||
notification is enabled, batch summaries and status lines include notification
|
||||
status, accepted distributor run ID, or notification error fields for each
|
||||
attempted report.
|
||||
`run morning` generates Today Report, Tomorrow Report, and a dated Daily Report
|
||||
for each later future local civil day with complete hourly forecast coverage.
|
||||
`run evening` generates Tomorrow Report and the same eligible future Daily
|
||||
reports. Future Daily expansion starts with the day after tomorrow and skips
|
||||
days that do not have every hourly forecast period for the local civil day.
|
||||
Batch commands collect weather data once before planning; a collection failure
|
||||
stops the batch before any report is generated. Batch runs continue independent
|
||||
reports after a later report failure, print a JSON summary to stdout, write
|
||||
compact status lines to stderr, and return nonzero when any report failed.
|
||||
`--out-dir` writes extra Markdown copies for the operator; distributor
|
||||
notification uses each managed report path, not the extra copies. Today and
|
||||
Tomorrow use their report default copy names, and dynamic Daily copies use
|
||||
`daily-YYYY-MM-DD.md`. When notification is enabled, batch summaries and status
|
||||
lines include notification status, accepted distributor run ID, or notification
|
||||
error fields for each attempted report.
|
||||
|
||||
Hourly Report is explicit only; it is not included in `run morning` or `run
|
||||
evening`.
|
||||
Hourly Report, 3-Day Outlook, and Weekend Outlook are explicit only; they are
|
||||
not included in `run morning` or `run evening`.
|
||||
|
||||
`inspect` commands read existing workspace artifacts and emit JSON to stdout.
|
||||
They do not fetch weather data or invoke `scriptorium`.
|
||||
|
||||
@@ -5,11 +5,12 @@ This document describes the workflow coordinator in `internal/app`.
|
||||
## Purpose
|
||||
|
||||
`internal/app` coordinates the top-level use cases after CLI parsing and config
|
||||
loading are complete. It resolves report definitions, fetches weather data,
|
||||
builds collected and derived facts, builds module snapshots and prompt-input
|
||||
artifacts, invokes Scriptorium through the adapter boundary, optionally
|
||||
notifies distributor through an app-owned notifier boundary, persists managed
|
||||
state, runs batches, and reads existing artifacts for inspection.
|
||||
loading are complete. It resolves report definitions, collects weather data
|
||||
through `internal/collect`, builds collected and derived facts, builds module
|
||||
snapshots and prompt-input artifacts, invokes Scriptorium through the adapter
|
||||
boundary, optionally notifies distributor through an app-owned notifier
|
||||
boundary, persists managed state, runs batches, and reads existing artifacts
|
||||
for inspection.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
@@ -20,9 +21,9 @@ Inputs:
|
||||
- `FetchBundleRequest` for explicit bundle fetch and save workflows
|
||||
- `ReportRequest` for single-report generation
|
||||
- resolved report definitions from `internal/report`
|
||||
- weather data bundles from `internal/adapters/weatherapi`
|
||||
- collection results from `internal/collect`
|
||||
- prior snapshots loaded from `internal/state`
|
||||
- optional renderer, notifier, and state-store fakes for tests
|
||||
- optional collector, renderer, notifier, and state-store fakes for tests
|
||||
|
||||
Outputs:
|
||||
|
||||
@@ -45,15 +46,18 @@ construct Scriptorium argv.
|
||||
|
||||
Report selection and report identity policy come from `internal/report`.
|
||||
Collected and derived fact contracts come from `internal/facts`.
|
||||
Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium
|
||||
subprocess behavior stays in `internal/adapters/scriptorium`. Distributor
|
||||
upload behavior stays in `internal/adapters/distributor`. Filesystem layout and
|
||||
persisted metadata stay in `internal/state`.
|
||||
Weather API transport stays in `internal/adapters/weatherapi`, and app-facing
|
||||
upstream collection stays in `internal/collect`. Scriptorium subprocess
|
||||
behavior stays in `internal/adapters/scriptorium`. Distributor upload behavior
|
||||
stays in `internal/adapters/distributor`. Filesystem layout and persisted
|
||||
metadata stay in `internal/state`.
|
||||
|
||||
## Data Flow Terms
|
||||
|
||||
- `CollectedFacts` are normalized source facts fetched once from Weather API
|
||||
and made available to derivation and module builders.
|
||||
- `collect.Result` is the app-facing upstream collection result. It carries the
|
||||
normalized `weatherdata.Bundle` used by report generation.
|
||||
- `CollectedFacts` are normalized source facts derived from a collected Weather
|
||||
API bundle and made available to derivation and module builders.
|
||||
- `DerivedFacts` are deterministic calculations over collected facts, the
|
||||
resolved valid period, daypart configuration, and report-specific windows.
|
||||
- `module.Output` values are ordered deterministic stanzas built from collected
|
||||
@@ -77,33 +81,35 @@ defaults.
|
||||
|
||||
## Generation Workflow
|
||||
|
||||
Single-report generation shares this setup:
|
||||
Single-report commands validate the report command, collect once through
|
||||
`internal/collect`, resolve the requested report, and pass the resolved report
|
||||
plus explicit collection into `GenerateReport`.
|
||||
|
||||
1. Resolve the command report to a `report.Resolved` value.
|
||||
2. Create or use a filesystem store.
|
||||
3. Locate any prior compatible snapshot through `internal/state`.
|
||||
4. Fetch a Weather API bundle.
|
||||
5. Build collected and derived facts once.
|
||||
6. Execute configured modules and save the module snapshot.
|
||||
7. Compute Recent Changes from structured prior and current module snapshots.
|
||||
8. Build and save the YAML Scriptorium `data_package`.
|
||||
9. Run Scriptorium render preflight.
|
||||
10. Save preflight JSON when a render result is available.
|
||||
11. Save metadata for inspection.
|
||||
`GenerateReport` then uses this setup:
|
||||
|
||||
1. Create or use a filesystem store.
|
||||
2. Locate any prior compatible snapshot through `internal/state`.
|
||||
3. Build collected and derived facts from the supplied collection.
|
||||
4. Execute configured modules and save the module snapshot.
|
||||
5. Compute Recent Changes from structured prior and current module snapshots.
|
||||
6. Build and save the YAML Scriptorium `data_package`.
|
||||
7. Run Scriptorium render preflight.
|
||||
8. Save preflight JSON when a render result is available.
|
||||
9. Save metadata for inspection.
|
||||
|
||||
For `scriptorium_markdown` reports, generation then:
|
||||
|
||||
12. Runs Scriptorium report generation to the managed report path.
|
||||
10. Runs Scriptorium report generation to the managed report path.
|
||||
|
||||
For `generated_text_template` reports, generation then:
|
||||
|
||||
12. Looks up the generated-text catalog entry for the report schema/template
|
||||
10. Looks up the generated-text catalog entry for the report schema/template
|
||||
IDs.
|
||||
13. Runs structured Scriptorium generation to the raw generated-text JSON path.
|
||||
14. Saves the structured Scriptorium run result.
|
||||
15. Validates and saves normalized generated text.
|
||||
16. Builds and saves a typed render context.
|
||||
17. Renders Markdown from the embedded template to the managed report path.
|
||||
11. Runs structured Scriptorium generation to the raw generated-text JSON path.
|
||||
12. Saves the structured Scriptorium run result.
|
||||
13. Validates and saves normalized generated text.
|
||||
14. Builds and saves a typed render context.
|
||||
15. Renders Markdown from the embedded template to the managed report path.
|
||||
|
||||
After either mode has produced a managed Markdown report, shared finalization:
|
||||
|
||||
@@ -119,7 +125,7 @@ After either mode has produced a managed Markdown report, shared finalization:
|
||||
If render preflight returns both a result and an error, preflight JSON and
|
||||
metadata are persisted before the error is returned. If Scriptorium report
|
||||
generation returns an error after writing output, the managed report and
|
||||
metadata remain inspectable. Notification is not attempted after Weather API,
|
||||
metadata remain inspectable. Notification is not attempted after collection,
|
||||
module snapshot, prompt input, render, Scriptorium run, or metadata-save
|
||||
failures.
|
||||
Generated-text report failures are returned with report ID, RunID, and the
|
||||
@@ -134,17 +140,21 @@ failure context.
|
||||
|
||||
## Batch Workflow
|
||||
|
||||
`run morning` resolves Today Report, 3-Day Outlook, and Weekend Outlook except
|
||||
on Sunday. `run evening` resolves Tomorrow Report. Daily Report is generated
|
||||
only through `generate daily --date YYYY-MM-DD`; it is not part of scheduled
|
||||
batches. Batch output copy names come from report definitions. Batch generation
|
||||
continues independent reports after a failure, records each result, writes
|
||||
compact status lines to stderr, emits a JSON summary to stdout, and returns an
|
||||
aggregate error when any report failed. When notification is enabled, each
|
||||
successfully generated report is notified independently. Notification failure
|
||||
marks that report failed, records notification fields in the batch result, and
|
||||
does not stop later reports. `--out-dir` copies are never used as notification
|
||||
source files.
|
||||
`run morning` collects once, plans Today Report, Tomorrow Report, and eligible
|
||||
future Daily Reports from the collected hourly forecast, then passes the same
|
||||
collection into each report generation. `run evening` uses the same collection
|
||||
and planning rules, but starts with Tomorrow Report. Future Daily reports start
|
||||
with the day after tomorrow and require complete hourly forecast coverage for
|
||||
the target local civil day. Dynamic Daily `--out-dir` copies use
|
||||
`daily-YYYY-MM-DD.md`; other batch copies use report definition output names.
|
||||
A collection failure stops the batch before planning or report generation.
|
||||
After planning succeeds, batch generation continues independent reports after a
|
||||
failure, records each result, writes compact status lines to stderr, emits a
|
||||
JSON summary to stdout, and returns an aggregate error when any report failed.
|
||||
When notification is enabled, each successfully generated report is notified
|
||||
independently. Notification failure marks that report failed, records
|
||||
notification fields in the batch result, and does not stop later reports.
|
||||
`--out-dir` copies are never used as notification source files.
|
||||
|
||||
## Inspection Workflow
|
||||
|
||||
@@ -155,8 +165,8 @@ inspection view.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
- Resolve errors stop the requested workflow before fetching weather data.
|
||||
- Weather API and module execution errors stop that report before Scriptorium
|
||||
- Resolve errors stop the requested workflow before collection.
|
||||
- Collection and module execution errors stop that report before Scriptorium
|
||||
runs.
|
||||
- Prompt input validation fails before render preflight.
|
||||
- Render and run errors preserve Scriptorium stderr and exit-code context.
|
||||
@@ -173,12 +183,17 @@ inspection view.
|
||||
Inspect:
|
||||
|
||||
- `internal/app/app_test.go`
|
||||
- `internal/app/batch_plan_test.go`
|
||||
- `internal/collect/collect_test.go`
|
||||
- `internal/cli/root_test.go`
|
||||
- `internal/state/filesystem_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Report behavior is resolved through `internal/report`.
|
||||
- Generate and run commands collect once before report generation.
|
||||
- Batch planning is app-owned because future Daily membership depends on
|
||||
collected hourly forecast coverage.
|
||||
- Generated reports use the same app request and result types regardless of
|
||||
report ID.
|
||||
- Render preflight precedes Scriptorium report generation.
|
||||
|
||||
58
docs/internal/collect.md
Normal file
58
docs/internal/collect.md
Normal file
@@ -0,0 +1,58 @@
|
||||
# Collection Internals
|
||||
|
||||
This document describes the app-facing upstream collection boundary in
|
||||
`internal/collect`.
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/collect` is the canonical package used by app workflows to collect
|
||||
upstream Weather API data. It constructs the Weather API adapter, fetches a
|
||||
normalized bundle, and returns that bundle without applying report selection or
|
||||
batch policy.
|
||||
|
||||
## Contract
|
||||
|
||||
Inputs:
|
||||
|
||||
- `collect.Request`, containing the effective `config.Config`
|
||||
- `context.Context` for cancellation
|
||||
|
||||
Output:
|
||||
|
||||
- `collect.Result`, containing `*weatherdata.Bundle`
|
||||
|
||||
`Run` returns an actionable error when Weather API adapter construction or
|
||||
bundle fetch fails. The package does not derive `facts.CollectedFacts`, build
|
||||
modules, resolve report periods, select reports, write state, invoke
|
||||
Scriptorium, or notify distributor.
|
||||
|
||||
## App Usage
|
||||
|
||||
`internal/app` owns a narrow `Collector` interface for orchestration tests. The
|
||||
default implementation calls `collect.Run`.
|
||||
|
||||
Single-report generation collects once, resolves the requested report, and
|
||||
passes the explicit collection into report generation. Batch generation
|
||||
collects once before planning and passes the same collection into each planned
|
||||
report. If collection returns no bundle, app orchestration returns an error
|
||||
before report generation.
|
||||
|
||||
## Boundaries
|
||||
|
||||
Weather API HTTP details stay in `internal/adapters/weatherapi`. The collection
|
||||
package returns normalized `weatherdata` only. It must not know about report
|
||||
IDs, prompt IDs, batch names, Daily eligibility, module composition, Recent
|
||||
Changes, state paths, or Scriptorium arguments.
|
||||
|
||||
## Tests
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/collect/collect_test.go`
|
||||
- `internal/app/app_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- App-facing Weather API collection goes through `internal/collect`.
|
||||
- Collection returns normalized source data, not report facts or prompt input.
|
||||
- Report and batch policy belongs outside `internal/collect`.
|
||||
@@ -1,7 +1,7 @@
|
||||
# Report Registry Internals
|
||||
|
||||
This document describes report identity, valid-period resolution, batch
|
||||
membership, output naming, artifact grouping, and comparison declarations in
|
||||
This document describes report identity, valid-period resolution, output
|
||||
naming, artifact grouping, batch command names, and comparison declarations in
|
||||
`internal/report`.
|
||||
|
||||
## Purpose
|
||||
@@ -24,7 +24,6 @@ Each report definition declares:
|
||||
- batch output copy filename
|
||||
- generated-report eligibility
|
||||
- prior-report compatibility list
|
||||
- morning or evening batch membership
|
||||
- default ordered module composition
|
||||
|
||||
Report-owned helpers map public command names and config keys to report IDs.
|
||||
@@ -69,8 +68,7 @@ All report definitions are eligible for generation.
|
||||
not a configuration field.
|
||||
- 3-Day Outlook covers the interval from generation time through local midnight
|
||||
three days later.
|
||||
- Weekend Outlook covers the upcoming weekend window and is not scheduled for
|
||||
Sunday morning batch resolution.
|
||||
- Weekend Outlook covers the upcoming weekend window.
|
||||
- Storm Report covers an explicit event window supplied by the caller.
|
||||
|
||||
Storm event windows can be parsed from local `YYYY-MM-DDTHH:MM` timestamps in
|
||||
@@ -80,8 +78,9 @@ must be after start time.
|
||||
## Boundaries
|
||||
|
||||
`internal/report` defines report metadata, public report names, batch command
|
||||
names, and time coverage. It does not fetch weather data, build module values,
|
||||
compare snapshot contents, write state, parse CLI flags, or invoke Scriptorium.
|
||||
names, output naming, and time coverage. It does not fetch weather data, plan
|
||||
batch membership, build module values, compare snapshot contents, write state,
|
||||
parse CLI flags, or invoke Scriptorium.
|
||||
|
||||
The CLI parses flags and command structure, then uses report-owned helpers for
|
||||
report and batch command names. Config loading uses report-owned helpers for
|
||||
@@ -94,11 +93,16 @@ output path copying uses batch output names from report definitions. Report
|
||||
module overrides can use short keys such as `daily`, `today`, `tomorrow`, and
|
||||
`hourly`, or descriptive names such as `three_day_outlook`.
|
||||
|
||||
## Batch Membership
|
||||
## Batch Commands
|
||||
|
||||
Morning batches include Today Report, 3-Day Outlook, and Weekend Outlook
|
||||
except on Sunday. Evening batches include Tomorrow Report. Daily Report and
|
||||
Hourly Report are not part of a scheduled batch.
|
||||
`internal/report` owns the public batch command names `morning` and `evening`
|
||||
and validates them through `BatchForCommandName`. Data-dependent batch
|
||||
membership is owned by `internal/app`, because it depends on collected hourly
|
||||
forecast coverage.
|
||||
|
||||
Report definitions still declare default batch output copy filenames. App
|
||||
batch planning uses those filenames for fixed report entries and supplies
|
||||
date-qualified names for dynamic Daily entries.
|
||||
|
||||
## State And App Usage
|
||||
|
||||
@@ -135,3 +139,5 @@ Inspect:
|
||||
- Artifact grouping, batch output filenames, generated-report eligibility,
|
||||
default module composition, comparison compatibility, and comparison strategy
|
||||
are declared by report definition.
|
||||
- App-owned batch planning uses report definitions but does not live in the
|
||||
report registry.
|
||||
|
||||
@@ -18,11 +18,11 @@ weatherreporter generate weekend
|
||||
weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00
|
||||
```
|
||||
|
||||
Generation commands resolve a report period, fetch a Weather API bundle, build
|
||||
a rich JSON module snapshot, build a curated YAML prompt input data package, run
|
||||
`scriptorium render`, and write managed artifacts under the configured
|
||||
workspace. Markdown-path reports then run `scriptorium run` directly to the
|
||||
managed Markdown report path.
|
||||
Generation commands resolve a report period, collect a Weather API bundle,
|
||||
build a rich JSON module snapshot, build a curated YAML prompt input data
|
||||
package, run `scriptorium render`, and write managed artifacts under the
|
||||
configured workspace. Markdown-path reports then run `scriptorium run` directly
|
||||
to the managed Markdown report path.
|
||||
|
||||
`generate daily`, `generate today`, `generate tomorrow`, and `generate hourly`
|
||||
use the generated-text-template workflow. They run structured `scriptorium run`
|
||||
@@ -31,8 +31,8 @@ and render the managed Markdown report from embedded templates. `generate
|
||||
daily` requires `--date YYYY-MM-DD` for the selected local civil day.
|
||||
`generate today` covers the selected or current local civil day. `generate
|
||||
hourly` covers the six-hour rolling period from generation time in the
|
||||
effective report timezone and is not part of scheduled morning or evening
|
||||
batches.
|
||||
effective report timezone and is not included in `run morning` or
|
||||
`run evening`.
|
||||
|
||||
When distributor notification is enabled, weatherreporter uploads the managed
|
||||
Markdown report after report rendering succeeds and final metadata is saved.
|
||||
@@ -46,16 +46,23 @@ weatherreporter run morning
|
||||
weatherreporter run evening
|
||||
```
|
||||
|
||||
`run morning` generates Today Report and the 3-Day Outlook, plus Weekend Outlook
|
||||
except on Sunday. `run evening` generates the Tomorrow Report. Batch
|
||||
commands print a JSON summary to stdout, write compact per-report status lines
|
||||
to stderr, continue independent reports after one report fails, and return
|
||||
nonzero when any report failed. When notification is configured, the summary and
|
||||
status lines include notification status, accepted distributor run ID, or
|
||||
notification error fields for each attempted report. `--out-dir PATH` writes
|
||||
extra Markdown copies using report default filenames such as `today.md`,
|
||||
`three-day.md`, `weekend.md`, and `tomorrow.md`; these copies are not used as
|
||||
distributor upload sources.
|
||||
`run morning` generates Today Report, Tomorrow Report, and a dated Daily Report
|
||||
for each later future local civil day with complete hourly forecast coverage.
|
||||
`run evening` generates Tomorrow Report and the same eligible future Daily
|
||||
reports. Future Daily expansion starts with the day after tomorrow. A Daily
|
||||
report is eligible only when the collected hourly forecast contains every
|
||||
hourly period for that local civil day; partial days are skipped. Batch commands
|
||||
collect weather data once before planning, and a collection failure stops the
|
||||
batch before any report is generated.
|
||||
|
||||
After planning succeeds, batch commands print a JSON summary to stdout, write
|
||||
compact per-report status lines to stderr, continue independent reports after
|
||||
one report fails, and return nonzero when any report failed. When notification
|
||||
is configured, the summary and status lines include notification status,
|
||||
accepted distributor run ID, or notification error fields for each attempted
|
||||
report. `--out-dir PATH` writes extra Markdown copies using report default
|
||||
filenames such as `today.md` and `tomorrow.md`; dynamic Daily copies use
|
||||
`daily-YYYY-MM-DD.md`. These copies are not used as distributor upload sources.
|
||||
|
||||
## Filesystem Layout
|
||||
|
||||
|
||||
Reference in New Issue
Block a user