Refresh CLI collection and app internals
This commit is contained in:
@@ -1,233 +1,109 @@
|
||||
# App Orchestration Internals
|
||||
# Application Orchestration Internals
|
||||
|
||||
This document describes the workflow coordinator in `internal/app`.
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/app` coordinates the top-level use cases after CLI parsing and config
|
||||
loading are complete. It resolves report definitions, collects weather data
|
||||
through `internal/collect`, builds collected and derived facts, builds module
|
||||
snapshots and prompt-input artifacts, invokes Scriptorium through the adapter
|
||||
boundary, optionally notifies distributor through an app-owned notifier
|
||||
boundary, persists managed state, runs batches, and reads existing artifacts
|
||||
for inspection.
|
||||
`internal/app` composes top-level generation, batch, collection-save, and
|
||||
inspection workflows after CLI parsing and configuration loading. It owns
|
||||
workflow ordering, request composition, partial-result handling, and the
|
||||
application-facing interfaces used for tests.
|
||||
|
||||
## Inputs And Outputs
|
||||
|
||||
Inputs:
|
||||
The package accepts generate, resolved-report, batch, explicit-collection, and
|
||||
inspection requests. Generation and batch requests may supply collector,
|
||||
renderer, store, and notifier implementations for tests; production defaults
|
||||
use the focused packages.
|
||||
|
||||
- `GenerateRequest` for one report command
|
||||
- `BatchRequest` for morning or evening batch commands
|
||||
- `FetchBundleRequest` for explicit bundle collection and save workflows
|
||||
- `ReportRequest` for single-report generation
|
||||
- resolved report definitions from `internal/report`
|
||||
- collection results from `internal/collect`
|
||||
- prior snapshots loaded from `internal/state`
|
||||
- optional collector, renderer, notifier, and state-store fakes for tests
|
||||
A report result contains the module snapshot, prompt package, available
|
||||
Scriptorium results, generated-text artifacts when used, report and metadata
|
||||
paths, prior snapshot, Recent Changes, and notification information. A batch
|
||||
result contains aggregate counts, per-report outcomes, and an optional batch
|
||||
notification. Inspection returns persisted values only.
|
||||
|
||||
Outputs:
|
||||
Exact public command syntax, configuration fields, workspace layout, external
|
||||
protocols, and report definitions belong in [the CLI reference](../cli.md),
|
||||
[the configuration reference](../config.md), [operations](../operations.md),
|
||||
and their focused integration and internal documents.
|
||||
|
||||
- generated report results with JSON module snapshot, YAML data package,
|
||||
preflight, report, metadata, prior snapshot, Recent Changes, Scriptorium
|
||||
result details, generated-text artifact paths when applicable, and
|
||||
notification result when attempted
|
||||
- batch summaries with per-report status, artifact paths, error text, and
|
||||
one top-level batch notification result when attempted or skipped
|
||||
- saved Weather API bundle JSON for explicit bundle collection workflows
|
||||
- inspection JSON values for reports, metadata, module snapshots, data
|
||||
packages, prior snapshots, and source provenance
|
||||
## Single-Report Workflow
|
||||
|
||||
## Boundaries
|
||||
`GenerateDetailed` first collects weather data, then resolves the requested
|
||||
report using the configured registry and current time, and finally calls
|
||||
`GenerateReport` with that explicit collection. It returns no result when
|
||||
collection or resolution fails.
|
||||
|
||||
`internal/app` owns workflow order and request composition. It does not parse
|
||||
CLI flags, load YAML files directly, implement HTTP transport, own fact
|
||||
derivation algorithms, define report periods, compare rendered Markdown, or
|
||||
construct Scriptorium argv.
|
||||
`GenerateReport` requires a non-nil normalized bundle and then performs this
|
||||
ordered work:
|
||||
|
||||
Report selection and report identity policy come from `internal/report`.
|
||||
Collected and derived fact contracts come from `internal/facts`.
|
||||
Weather API transport stays in `internal/adapters/weatherapi`, and app-facing
|
||||
upstream collection stays in `internal/collect`. Scriptorium subprocess
|
||||
behavior stays in `internal/adapters/scriptorium`. Distributor upload behavior
|
||||
stays in `internal/adapters/distributor`. Filesystem layout and persisted
|
||||
metadata stay in `internal/state`.
|
||||
1. Select a state store, determine artifact destinations, and locate a prior
|
||||
compatible snapshot.
|
||||
2. Build report facts and deterministic module snapshots, then save the module
|
||||
snapshot and calculate Recent Changes.
|
||||
3. Build and save the prompt data package, run Scriptorium render preflight,
|
||||
save any preflight result, and save initial metadata.
|
||||
4. Produce managed Markdown according to the report generation mode.
|
||||
5. Finalize metadata, optionally make an output copy, and optionally notify
|
||||
Distributor from the managed report path.
|
||||
|
||||
## Data Flow Terms
|
||||
Direct-Markdown reports prepare the managed report and invoke the Scriptorium
|
||||
run boundary. Generated-text-template reports look up their catalog definition,
|
||||
run structured Scriptorium output to the raw artifact, preserve any structured
|
||||
run result, validate and save generated text, build and save a render context,
|
||||
then render the embedded Markdown template. Schema, template, and subprocess
|
||||
details remain in their [generated-text](generatedtext.md),
|
||||
[report-template](reporttemplate.md), and [Scriptorium adapter](scriptorium-adapter.md)
|
||||
owners.
|
||||
|
||||
- `collect.Result` is the app-facing upstream collection result. It carries the
|
||||
normalized `weatherdata.Bundle` used by report generation.
|
||||
- `CollectedFacts` are normalized source facts derived from a collected Weather
|
||||
API bundle and made available to derivation and module builders.
|
||||
- `DerivedFacts` are deterministic calculations over collected facts, the
|
||||
resolved valid period, daypart configuration, and report-specific windows.
|
||||
- `module.Output` values are ordered deterministic stanzas built from collected
|
||||
and derived facts for prompt input and inspection.
|
||||
- `GeneratedText` is structured prose returned by Scriptorium for
|
||||
generated-text-template reports and validated by `internal/generatedtext`.
|
||||
- `RenderContext` is the typed template input built from report metadata,
|
||||
module outputs, and validated generated text before Markdown rendering.
|
||||
If preflight returns a result with an error, the result and initial metadata are
|
||||
saved before the error returns. If report generation fails after a managed path
|
||||
is prepared, metadata still records that path; output copies and notification
|
||||
are skipped. Generated-text failures preserve the latest artifact reached
|
||||
before failure when it was saved.
|
||||
|
||||
## Config Fields Used
|
||||
## Batch And Inspection Workflows
|
||||
|
||||
- `weather_api.*` for Weather API client construction and module metadata
|
||||
- `scriptorium.*` for renderer construction
|
||||
- `workspace.*` for filesystem state
|
||||
- `dayparts` for daily and outlook summarization
|
||||
- `recent_change.*` for structured Recent Changes thresholds
|
||||
- `notify.distributor.*` for optional single-report and batch notification
|
||||
after report generation
|
||||
`RunBatchDetailed` collects once, asks the report registry to plan the batch
|
||||
from that collection, and invokes `GenerateReport` independently for every
|
||||
planned report using the same collection and state store. Per-report
|
||||
notification is suppressed. A failed report is recorded and does not prevent
|
||||
later planned reports from running.
|
||||
|
||||
Output copy flags are command request fields. They are not configuration
|
||||
defaults.
|
||||
After report generation, the batch notifier is considered once. It is omitted
|
||||
when Distributor or batch notification is disabled, skipped when any report
|
||||
failed, and otherwise receives one multi-file request. A batch notification
|
||||
failure increments the aggregate failure count but does not rewrite successful
|
||||
report items. Notification identities, path mappings, polling, and redaction
|
||||
are owned by the [Distributor adapter](distributor-adapter.md).
|
||||
|
||||
## Generation Workflow
|
||||
Inspection methods create a state store and load existing report records,
|
||||
metadata, module snapshots, prompt packages, prior snapshots, or source
|
||||
provenance. They neither collect data nor invoke Scriptorium or Distributor.
|
||||
|
||||
Single-report commands validate the report command, collect once through
|
||||
`internal/collect`, resolve the requested report, and pass the resolved report
|
||||
plus explicit collection into `GenerateReport`. `GenerateDetailed` returns the
|
||||
resulting `ReportResult`; `Generate` wraps the same workflow for error-only
|
||||
callers.
|
||||
## Boundaries And Failure Propagation
|
||||
|
||||
`GenerateReport` then uses this setup:
|
||||
The app layer does not parse flags, load configuration files, implement Weather
|
||||
API transport, construct Scriptorium argv, or define report registry policy. It
|
||||
coordinates the relevant collaborators and preserves their error context.
|
||||
|
||||
1. Create or use a filesystem store.
|
||||
2. Locate any prior compatible snapshot through `internal/state`.
|
||||
3. Build collected and derived facts from the supplied collection.
|
||||
4. Execute configured modules and save the module snapshot.
|
||||
5. Compute Recent Changes from structured prior and current module snapshots.
|
||||
6. Build and save the YAML Scriptorium `data_package`.
|
||||
7. Run Scriptorium render preflight.
|
||||
8. Save preflight JSON when a render result is available.
|
||||
9. Save metadata for inspection.
|
||||
- Collection failure stops a single report or batch before resolution or
|
||||
planning completes.
|
||||
- State, fact, module, prompt-input, or preflight failures stop that report
|
||||
before report generation.
|
||||
- A terminal Distributor failure is returned with the saved notification
|
||||
information when available.
|
||||
- Batch failures are represented per report and through aggregate batch status.
|
||||
- Persisted artifact paths are carried in results so callers can inspect work
|
||||
completed before a later failure.
|
||||
|
||||
For `scriptorium_markdown` reports, generation then:
|
||||
## Tests And Invariants
|
||||
|
||||
10. Runs Scriptorium report generation to the managed report path.
|
||||
Focused tests are in `internal/app/app_test.go` and
|
||||
`internal/app/batch_plan_test.go`, with collection coverage in
|
||||
`internal/collect/collect_test.go`.
|
||||
|
||||
For `generated_text_template` reports, generation then:
|
||||
|
||||
10. Looks up the generated-text catalog entry for the report schema/template
|
||||
IDs.
|
||||
11. Runs structured Scriptorium generation to the raw generated-text JSON path.
|
||||
12. Saves the structured Scriptorium run result.
|
||||
13. Validates and saves normalized generated text.
|
||||
14. Builds and saves a typed render context.
|
||||
15. Renders Markdown from the embedded template to the managed report path.
|
||||
|
||||
After either mode has produced a managed Markdown report, shared finalization:
|
||||
|
||||
1. Copies the managed report to the requested `--out` or `--out-dir` path when
|
||||
provided.
|
||||
2. Saves final metadata with the managed report path and any generated-text
|
||||
artifact paths already produced.
|
||||
3. If distributor notification is enabled, notifies using the managed report
|
||||
path as the source file.
|
||||
4. Saves a distributor notification debug artifact and updates metadata with
|
||||
its path.
|
||||
|
||||
If render preflight returns both a result and an error, preflight JSON and
|
||||
metadata are persisted before the error is returned. If Scriptorium report
|
||||
generation returns an error after writing output, the managed report and
|
||||
metadata remain inspectable. Notification is not attempted after collection,
|
||||
module snapshot, prompt input, render, Scriptorium run, or metadata-save
|
||||
failures.
|
||||
Generated-text report failures are returned with report ID, RunID, and the
|
||||
failed operation. When available, the app preserves the latest generated-text
|
||||
artifacts already reached by the workflow: preflight output, structured run
|
||||
result, raw generated text, validated generated text, and render context.
|
||||
When notification is attempted, the debug artifact records request identity,
|
||||
including rendered pipeline ID, bundle paths, accepted upload fields,
|
||||
distributor status fields, raw status report JSON when available, and redacted
|
||||
failure context.
|
||||
`--out` copies are never used as notification source files.
|
||||
|
||||
## Batch Workflow
|
||||
|
||||
`run morning` collects once, plans Today Report, Tomorrow Report, and eligible
|
||||
future Daily Reports from the collected hourly forecast, then passes the same
|
||||
collection into each report generation. `run evening` uses the same collection
|
||||
and planning rules, but starts with Tomorrow Report. Future Daily reports start
|
||||
with the day after tomorrow and require complete hourly forecast coverage for
|
||||
the target local civil day. Dynamic Daily `--out-dir` copies use
|
||||
`daily-YYYY-MM-DD.md`; other batch copies use report definition output names.
|
||||
A collection failure stops the batch before planning or report generation.
|
||||
After planning succeeds, batch generation continues independent reports after a
|
||||
failure, records each result, writes compact status lines to stderr, emits a
|
||||
JSON summary to stdout, and returns an aggregate error when any report failed.
|
||||
|
||||
Batch report generation suppresses per-report distributor notification. After
|
||||
all planned reports finish, app orchestration evaluates batch notification:
|
||||
|
||||
1. If distributor notification is disabled, the batch notification result is
|
||||
omitted.
|
||||
2. If batch notification is disabled, the batch notification result is omitted
|
||||
and there is no per-report fallback upload.
|
||||
3. If any planned report failed, the batch notification result is `skipped`
|
||||
with reason `one or more reports failed`, and distributor is not called.
|
||||
4. If every report succeeded, app orchestration renders batch pipeline, bundle
|
||||
ID, and idempotency key templates, renders report-specific distributor
|
||||
paths for each included report, validates every managed source path and
|
||||
bundle path, checks duplicate bundle paths across the batch, calls the
|
||||
notifier once with a multi-file request, and saves a batch notification
|
||||
debug artifact.
|
||||
|
||||
Batch notification failure records a top-level failed notification, increments
|
||||
the aggregate batch failure count, and returns an aggregate batch error without
|
||||
marking individual report items failed. `--out-dir` copies are never used as
|
||||
notification source files.
|
||||
|
||||
## Inspection Workflow
|
||||
|
||||
Inspection workflows load existing filesystem state only. They do not fetch
|
||||
weather data or invoke Scriptorium. Run-specific inspect commands share the same
|
||||
store and metadata lookup path, then load the requested artifact or derived
|
||||
inspection view.
|
||||
|
||||
## Failure Behavior
|
||||
|
||||
- Resolve errors stop the requested workflow before collection.
|
||||
- Collection and module execution errors stop that report before Scriptorium
|
||||
runs.
|
||||
- Prompt input validation fails before render preflight.
|
||||
- Render and run errors preserve Scriptorium stderr and exit-code context.
|
||||
- Generated-text report errors preserve available intermediate artifacts and do
|
||||
not create extra output copies.
|
||||
- Single-report notification errors are wrapped with report ID, RunID, and
|
||||
managed report path context. Detailed generation returns the inspectable
|
||||
report, metadata, and notification artifact paths when finalization has
|
||||
already saved them.
|
||||
- Batch notification errors are recorded on the top-level batch notification
|
||||
result and do not change individual report item status.
|
||||
- Metadata and artifact path errors include filesystem context.
|
||||
- Batch failures are recorded per report and surfaced through an aggregate
|
||||
batch error.
|
||||
|
||||
## Tests
|
||||
|
||||
Inspect:
|
||||
|
||||
- `internal/app/app_test.go`
|
||||
- `internal/app/batch_plan_test.go`
|
||||
- `internal/collect/collect_test.go`
|
||||
- `internal/cli/root_test.go`
|
||||
- `internal/state/filesystem_test.go`
|
||||
|
||||
## Invariants
|
||||
|
||||
- Report behavior is resolved through `internal/report`.
|
||||
- Generate and run commands collect once before report generation.
|
||||
- Batch planning is app-owned because future Daily membership depends on
|
||||
collected hourly forecast coverage.
|
||||
- Generated reports use the same app request and result types regardless of
|
||||
report ID.
|
||||
- Render preflight precedes Scriptorium report generation.
|
||||
- Generated-text reports render Markdown from a curated render context, not from
|
||||
a raw data package.
|
||||
- Recent Changes are computed from structured module snapshots.
|
||||
- Metadata links artifacts produced for a run.
|
||||
- Single-report distributor notification maps the managed Markdown report path
|
||||
to configured bundle paths.
|
||||
- Batch distributor notification maps each included managed Markdown report
|
||||
path to bundle paths rendered for that report and uploads once for the
|
||||
batch.
|
||||
- Extra output copies are not upload sources.
|
||||
- Production workflows collect through `internal/collect`.
|
||||
- A report uses one explicit normalized collection throughout its generation.
|
||||
- Render preflight precedes report generation.
|
||||
- Recent Changes compare structured module snapshots.
|
||||
- Generated-text reports render from a validated typed context, never directly
|
||||
from a raw prompt package.
|
||||
- Only managed Markdown reports are notification sources; output copies are
|
||||
never uploaded.
|
||||
|
||||
Reference in New Issue
Block a user