Refresh CLI collection and app internals

This commit is contained in:
2026-07-31 01:21:23 +00:00
parent 1130d807dc
commit c6f8570474
3 changed files with 166 additions and 307 deletions

View File

@@ -1,233 +1,109 @@
# App Orchestration Internals
# Application Orchestration Internals
This document describes the workflow coordinator in `internal/app`.
## Purpose
`internal/app` coordinates the top-level use cases after CLI parsing and config
loading are complete. It resolves report definitions, collects weather data
through `internal/collect`, builds collected and derived facts, builds module
snapshots and prompt-input artifacts, invokes Scriptorium through the adapter
boundary, optionally notifies distributor through an app-owned notifier
boundary, persists managed state, runs batches, and reads existing artifacts
for inspection.
`internal/app` composes top-level generation, batch, collection-save, and
inspection workflows after CLI parsing and configuration loading. It owns
workflow ordering, request composition, partial-result handling, and the
application-facing interfaces used for tests.
## Inputs And Outputs
Inputs:
The package accepts generate, resolved-report, batch, explicit-collection, and
inspection requests. Generation and batch requests may supply collector,
renderer, store, and notifier implementations for tests; production defaults
use the focused packages.
- `GenerateRequest` for one report command
- `BatchRequest` for morning or evening batch commands
- `FetchBundleRequest` for explicit bundle collection and save workflows
- `ReportRequest` for single-report generation
- resolved report definitions from `internal/report`
- collection results from `internal/collect`
- prior snapshots loaded from `internal/state`
- optional collector, renderer, notifier, and state-store fakes for tests
A report result contains the module snapshot, prompt package, available
Scriptorium results, generated-text artifacts when used, report and metadata
paths, prior snapshot, Recent Changes, and notification information. A batch
result contains aggregate counts, per-report outcomes, and an optional batch
notification. Inspection returns persisted values only.
Outputs:
Exact public command syntax, configuration fields, workspace layout, external
protocols, and report definitions belong in [the CLI reference](../cli.md),
[the configuration reference](../config.md), [operations](../operations.md),
and their focused integration and internal documents.
- generated report results with JSON module snapshot, YAML data package,
preflight, report, metadata, prior snapshot, Recent Changes, Scriptorium
result details, generated-text artifact paths when applicable, and
notification result when attempted
- batch summaries with per-report status, artifact paths, error text, and
one top-level batch notification result when attempted or skipped
- saved Weather API bundle JSON for explicit bundle collection workflows
- inspection JSON values for reports, metadata, module snapshots, data
packages, prior snapshots, and source provenance
## Single-Report Workflow
## Boundaries
`GenerateDetailed` first collects weather data, then resolves the requested
report using the configured registry and current time, and finally calls
`GenerateReport` with that explicit collection. It returns no result when
collection or resolution fails.
`internal/app` owns workflow order and request composition. It does not parse
CLI flags, load YAML files directly, implement HTTP transport, own fact
derivation algorithms, define report periods, compare rendered Markdown, or
construct Scriptorium argv.
`GenerateReport` requires a non-nil normalized bundle and then performs this
ordered work:
Report selection and report identity policy come from `internal/report`.
Collected and derived fact contracts come from `internal/facts`.
Weather API transport stays in `internal/adapters/weatherapi`, and app-facing
upstream collection stays in `internal/collect`. Scriptorium subprocess
behavior stays in `internal/adapters/scriptorium`. Distributor upload behavior
stays in `internal/adapters/distributor`. Filesystem layout and persisted
metadata stay in `internal/state`.
1. Select a state store, determine artifact destinations, and locate a prior
compatible snapshot.
2. Build report facts and deterministic module snapshots, then save the module
snapshot and calculate Recent Changes.
3. Build and save the prompt data package, run Scriptorium render preflight,
save any preflight result, and save initial metadata.
4. Produce managed Markdown according to the report generation mode.
5. Finalize metadata, optionally make an output copy, and optionally notify
Distributor from the managed report path.
## Data Flow Terms
Direct-Markdown reports prepare the managed report and invoke the Scriptorium
run boundary. Generated-text-template reports look up their catalog definition,
run structured Scriptorium output to the raw artifact, preserve any structured
run result, validate and save generated text, build and save a render context,
then render the embedded Markdown template. Schema, template, and subprocess
details remain in their [generated-text](generatedtext.md),
[report-template](reporttemplate.md), and [Scriptorium adapter](scriptorium-adapter.md)
owners.
- `collect.Result` is the app-facing upstream collection result. It carries the
normalized `weatherdata.Bundle` used by report generation.
- `CollectedFacts` are normalized source facts derived from a collected Weather
API bundle and made available to derivation and module builders.
- `DerivedFacts` are deterministic calculations over collected facts, the
resolved valid period, daypart configuration, and report-specific windows.
- `module.Output` values are ordered deterministic stanzas built from collected
and derived facts for prompt input and inspection.
- `GeneratedText` is structured prose returned by Scriptorium for
generated-text-template reports and validated by `internal/generatedtext`.
- `RenderContext` is the typed template input built from report metadata,
module outputs, and validated generated text before Markdown rendering.
If preflight returns a result with an error, the result and initial metadata are
saved before the error returns. If report generation fails after a managed path
is prepared, metadata still records that path; output copies and notification
are skipped. Generated-text failures preserve the latest artifact reached
before failure when it was saved.
## Config Fields Used
## Batch And Inspection Workflows
- `weather_api.*` for Weather API client construction and module metadata
- `scriptorium.*` for renderer construction
- `workspace.*` for filesystem state
- `dayparts` for daily and outlook summarization
- `recent_change.*` for structured Recent Changes thresholds
- `notify.distributor.*` for optional single-report and batch notification
after report generation
`RunBatchDetailed` collects once, asks the report registry to plan the batch
from that collection, and invokes `GenerateReport` independently for every
planned report using the same collection and state store. Per-report
notification is suppressed. A failed report is recorded and does not prevent
later planned reports from running.
Output copy flags are command request fields. They are not configuration
defaults.
After report generation, the batch notifier is considered once. It is omitted
when Distributor or batch notification is disabled, skipped when any report
failed, and otherwise receives one multi-file request. A batch notification
failure increments the aggregate failure count but does not rewrite successful
report items. Notification identities, path mappings, polling, and redaction
are owned by the [Distributor adapter](distributor-adapter.md).
## Generation Workflow
Inspection methods create a state store and load existing report records,
metadata, module snapshots, prompt packages, prior snapshots, or source
provenance. They neither collect data nor invoke Scriptorium or Distributor.
Single-report commands validate the report command, collect once through
`internal/collect`, resolve the requested report, and pass the resolved report
plus explicit collection into `GenerateReport`. `GenerateDetailed` returns the
resulting `ReportResult`; `Generate` wraps the same workflow for error-only
callers.
## Boundaries And Failure Propagation
`GenerateReport` then uses this setup:
The app layer does not parse flags, load configuration files, implement Weather
API transport, construct Scriptorium argv, or define report registry policy. It
coordinates the relevant collaborators and preserves their error context.
1. Create or use a filesystem store.
2. Locate any prior compatible snapshot through `internal/state`.
3. Build collected and derived facts from the supplied collection.
4. Execute configured modules and save the module snapshot.
5. Compute Recent Changes from structured prior and current module snapshots.
6. Build and save the YAML Scriptorium `data_package`.
7. Run Scriptorium render preflight.
8. Save preflight JSON when a render result is available.
9. Save metadata for inspection.
- Collection failure stops a single report or batch before resolution or
planning completes.
- State, fact, module, prompt-input, or preflight failures stop that report
before report generation.
- A terminal Distributor failure is returned with the saved notification
information when available.
- Batch failures are represented per report and through aggregate batch status.
- Persisted artifact paths are carried in results so callers can inspect work
completed before a later failure.
For `scriptorium_markdown` reports, generation then:
## Tests And Invariants
10. Runs Scriptorium report generation to the managed report path.
Focused tests are in `internal/app/app_test.go` and
`internal/app/batch_plan_test.go`, with collection coverage in
`internal/collect/collect_test.go`.
For `generated_text_template` reports, generation then:
10. Looks up the generated-text catalog entry for the report schema/template
IDs.
11. Runs structured Scriptorium generation to the raw generated-text JSON path.
12. Saves the structured Scriptorium run result.
13. Validates and saves normalized generated text.
14. Builds and saves a typed render context.
15. Renders Markdown from the embedded template to the managed report path.
After either mode has produced a managed Markdown report, shared finalization:
1. Copies the managed report to the requested `--out` or `--out-dir` path when
provided.
2. Saves final metadata with the managed report path and any generated-text
artifact paths already produced.
3. If distributor notification is enabled, notifies using the managed report
path as the source file.
4. Saves a distributor notification debug artifact and updates metadata with
its path.
If render preflight returns both a result and an error, preflight JSON and
metadata are persisted before the error is returned. If Scriptorium report
generation returns an error after writing output, the managed report and
metadata remain inspectable. Notification is not attempted after collection,
module snapshot, prompt input, render, Scriptorium run, or metadata-save
failures.
Generated-text report failures are returned with report ID, RunID, and the
failed operation. When available, the app preserves the latest generated-text
artifacts already reached by the workflow: preflight output, structured run
result, raw generated text, validated generated text, and render context.
When notification is attempted, the debug artifact records request identity,
including rendered pipeline ID, bundle paths, accepted upload fields,
distributor status fields, raw status report JSON when available, and redacted
failure context.
`--out` copies are never used as notification source files.
## Batch Workflow
`run morning` collects once, plans Today Report, Tomorrow Report, and eligible
future Daily Reports from the collected hourly forecast, then passes the same
collection into each report generation. `run evening` uses the same collection
and planning rules, but starts with Tomorrow Report. Future Daily reports start
with the day after tomorrow and require complete hourly forecast coverage for
the target local civil day. Dynamic Daily `--out-dir` copies use
`daily-YYYY-MM-DD.md`; other batch copies use report definition output names.
A collection failure stops the batch before planning or report generation.
After planning succeeds, batch generation continues independent reports after a
failure, records each result, writes compact status lines to stderr, emits a
JSON summary to stdout, and returns an aggregate error when any report failed.
Batch report generation suppresses per-report distributor notification. After
all planned reports finish, app orchestration evaluates batch notification:
1. If distributor notification is disabled, the batch notification result is
omitted.
2. If batch notification is disabled, the batch notification result is omitted
and there is no per-report fallback upload.
3. If any planned report failed, the batch notification result is `skipped`
with reason `one or more reports failed`, and distributor is not called.
4. If every report succeeded, app orchestration renders batch pipeline, bundle
ID, and idempotency key templates, renders report-specific distributor
paths for each included report, validates every managed source path and
bundle path, checks duplicate bundle paths across the batch, calls the
notifier once with a multi-file request, and saves a batch notification
debug artifact.
Batch notification failure records a top-level failed notification, increments
the aggregate batch failure count, and returns an aggregate batch error without
marking individual report items failed. `--out-dir` copies are never used as
notification source files.
## Inspection Workflow
Inspection workflows load existing filesystem state only. They do not fetch
weather data or invoke Scriptorium. Run-specific inspect commands share the same
store and metadata lookup path, then load the requested artifact or derived
inspection view.
## Failure Behavior
- Resolve errors stop the requested workflow before collection.
- Collection and module execution errors stop that report before Scriptorium
runs.
- Prompt input validation fails before render preflight.
- Render and run errors preserve Scriptorium stderr and exit-code context.
- Generated-text report errors preserve available intermediate artifacts and do
not create extra output copies.
- Single-report notification errors are wrapped with report ID, RunID, and
managed report path context. Detailed generation returns the inspectable
report, metadata, and notification artifact paths when finalization has
already saved them.
- Batch notification errors are recorded on the top-level batch notification
result and do not change individual report item status.
- Metadata and artifact path errors include filesystem context.
- Batch failures are recorded per report and surfaced through an aggregate
batch error.
## Tests
Inspect:
- `internal/app/app_test.go`
- `internal/app/batch_plan_test.go`
- `internal/collect/collect_test.go`
- `internal/cli/root_test.go`
- `internal/state/filesystem_test.go`
## Invariants
- Report behavior is resolved through `internal/report`.
- Generate and run commands collect once before report generation.
- Batch planning is app-owned because future Daily membership depends on
collected hourly forecast coverage.
- Generated reports use the same app request and result types regardless of
report ID.
- Render preflight precedes Scriptorium report generation.
- Generated-text reports render Markdown from a curated render context, not from
a raw data package.
- Recent Changes are computed from structured module snapshots.
- Metadata links artifacts produced for a run.
- Single-report distributor notification maps the managed Markdown report path
to configured bundle paths.
- Batch distributor notification maps each included managed Markdown report
path to bundle paths rendered for that report and uploads once for the
batch.
- Extra output copies are not upload sources.
- Production workflows collect through `internal/collect`.
- A report uses one explicit normalized collection throughout its generation.
- Render preflight precedes report generation.
- Recent Changes compare structured module snapshots.
- Generated-text reports render from a validated typed context, never directly
from a raw prompt package.
- Only managed Markdown reports are notification sources; output copies are
never uploaded.

View File

@@ -1,67 +1,65 @@
# CLI Internals
This document describes command output ownership in `internal/cli`.
`internal/cli` turns process arguments into application requests and translates
application results into terminal output. The user-facing command, flag, and
output contract belongs in the [CLI reference](../cli.md).
## Purpose
## Responsibilities
`internal/cli` owns command parsing, app request construction, help text, and
presentation of command results. It converts app-layer results into stable CLI
summaries and writes stdout/stderr through shared output helpers.
`Runner.Run` dispatches the top-level action or inspection request. For actions,
the package parses command-specific and common flags, loads configuration with
CLI overrides, obtains the current time, and constructs either an
`app.GenerateRequest` or an `app.BatchRequest`. It delegates generation and
batch execution to `internal/app`.
## Command Categories
For inspection, it loads configuration, builds the appropriate app inspection
request, and writes the returned value. Inspection is read-only; the inspected
artifact types and user invocation remain owned by the [CLI reference](../cli.md)
and [operations guide](../operations.md).
- Action commands: `generate` and `run`. These perform work, write artifacts,
and return compact summaries.
- Inspection commands: `inspect reports`, `inspect metadata`, `inspect
modules`, `inspect data-package`, `inspect prior`, and `inspect sources`.
These read existing artifacts and return requested data.
## Result Translation
Future commands must declare which category they belong to before adding output
behavior.
Action results become CLI-safe JSON summaries in `result.go`. Generate summaries
carry report identity, status, relevant artifact paths, and notification
summary data. Batch summaries carry aggregate counts, per-report outcomes, and
the optional batch notification result. The translation deliberately excludes
full module snapshots, prompt packages, raw generated text, Scriptorium output,
and complete Distributor payloads.
## Stdout And Stderr
When an action returns both a result and an error, the CLI writes the failed
summary before returning that error. Parse, configuration-load, and other
failures that produce no application result return without a summary.
Action commands write JSON summaries to stdout by default. `run` also writes
compact status lines to stderr through `writeBatchStatus`. `generate` does not
write routine stderr today. Pre-run errors return without partial JSON.
`writeActionResult` writes action status information to stderr first, then JSON
to stdout. Batch execution supplies the status writer; single-report generation
does not emit routine stderr output. Quiet action requests suppress both normal
streams but still return errors. Inspection writes its JSON value to stdout and
does not accept quiet mode because stdout is the inspection result.
Inspection commands write requested JSON data to stdout with `writeJSON`. They
do not use action output helpers and do not support quiet mode.
## Boundaries
Returned errors are not hidden by output helpers. The caller remains
responsible for displaying command errors.
The package owns argument parsing, request adaptation, help text, and terminal
presentation. It does not implement report selection, collection, state
persistence, external transport, subprocess execution, or notification policy.
Those concerns remain in [application orchestration](app-orchestration.md) and
their focused owners.
## Quiet Mode
## Failure Behavior
`--quiet` is supported only by action commands. It suppresses successful stdout
and routine stderr by passing `outputOptions{Quiet: true}` to
`writeActionResult`. It does not suppress returned errors.
- Invalid command names, flags, dates, and configuration fail before an app
request is executed.
- Application errors retain their application context; output helpers do not
hide or replace them.
- JSON-encoding errors are returned directly.
- A failed batch summary causes the CLI to return an aggregate batch error even
when the detailed batch call has already returned its result.
Quiet mode is intentionally not accepted by inspection commands because
inspection stdout is the command result.
## Tests And Invariants
## Summary Ownership
Focused tests are in `internal/cli/root_test.go`, `internal/cli/output_test.go`,
and `internal/cli/result_test.go`.
CLI-safe summary structs live in `internal/cli/result.go`.
- `newGenerateSummary` converts `*app.ReportResult` plus an optional error into
the generate JSON contract.
- `newBatchSummary` converts `*app.BatchResult` into the run JSON contract and
derives the top-level run status.
Summary types must not expose full app internals, module contents, data package
contents, raw generated text, Scriptorium result bodies, or full distributor
payloads.
## Helper Path
New action commands should:
1. parse command-specific flags into CLI option structs;
2. call the app-layer use case;
3. convert app results into a CLI summary type;
4. write through `writeActionResult`;
5. use a status writer only for routine stderr status lines.
New inspection commands should call the app inspection use case and write the
returned data through `writeJSON`.
- CLI summaries are stable, bounded views of app results.
- Routine batch status lines precede the batch JSON summary.
- A quiet action produces no successful or failure summary output.
- Inspection never invokes action-output helpers.

View File

@@ -1,58 +1,43 @@
# Collection Internals
This document describes the app-facing upstream collection boundary in
`internal/collect`.
## Purpose
`internal/collect` is the canonical package used by app workflows to collect
upstream Weather API data. It constructs the Weather API adapter, fetches a
normalized bundle, and returns that bundle without applying report selection or
batch policy.
`internal/collect` is the application-facing boundary for collecting the
normalized Weather API bundle. The external HTTP contract belongs in the
[Weather API integration guide](../integrations/weatherapi.md); normalized data
semantics belong in [weather-data internals](weather-data.md).
## Contract
Inputs:
`Run` accepts a `context.Context` and a `Request` containing effective
`config.Config`. It constructs the Weather API adapter from that configuration,
calls `FetchBundle`, and returns `Result{Bundle: *weatherdata.Bundle}`.
- `collect.Request`, containing the effective `config.Config`
- `context.Context` for cancellation
The package wraps adapter construction failures as weather-collection setup
errors and fetch failures as bundle-collection errors. It does not retry,
persist, select reports, derive facts, build modules, invoke Scriptorium, or
notify Distributor.
Output:
## Application Composition
- `collect.Result`, containing `*weatherdata.Bundle`
`internal/app` owns the narrow `Collector` interface used by workflow tests;
the production implementation delegates to `collect.Run`. Generation, batch
execution, and explicit bundle fetching all use this boundary. Application
orchestration rejects a nil collector result or a nil bundle before report work
can continue.
`Run` returns an actionable error when Weather API adapter construction or
bundle fetch fails. The package does not derive `facts.CollectedFacts`, build
modules, resolve report periods, select reports, write state, invoke
Scriptorium, or notify distributor.
Single-report generation and a batch each collect once. A batch passes the same
normalized collection to planning and to every report it generates. Collection
failure prevents later workflow work for that request.
## App Usage
## Boundaries And Invariants
`internal/app` owns a narrow `Collector` interface for orchestration tests. The
default implementation calls `collect.Run`.
Collection owns adapter creation and retrieval of one normalized bundle. It
must not make report, period, batch, prompt, module, filesystem, or notification
decisions.
Single-report generation collects once, resolves the requested report, and
passes the explicit collection into report generation. Batch generation
collects once before planning and passes the same collection into each planned
report. If collection returns no bundle, app orchestration returns an error
before report generation.
- App-facing Weather API collection always passes through this package.
- The returned value is normalized source data, not facts or prompt input.
- Context cancellation is passed to the Weather API adapter.
- Errors retain whether setup or fetching failed.
## Boundaries
Weather API HTTP details stay in `internal/adapters/weatherapi`. The collection
package returns normalized `weatherdata` only. It must not know about report
IDs, prompt IDs, batch names, Daily eligibility, module composition, Recent
Changes, state paths, or Scriptorium arguments.
## Tests
Inspect:
- `internal/collect/collect_test.go`
- `internal/app/app_test.go`
## Invariants
- App-facing Weather API collection goes through `internal/collect`.
- Collection returns normalized source data, not report facts or prompt input.
- Report and batch policy belongs outside `internal/collect`.
Focused tests are in `internal/collect/collect_test.go`; orchestration use is
also covered by `internal/app/app_test.go`.