diff --git a/docs/internal/app-orchestration.md b/docs/internal/app-orchestration.md index 8865cb1..34dcbee 100644 --- a/docs/internal/app-orchestration.md +++ b/docs/internal/app-orchestration.md @@ -1,233 +1,109 @@ -# App Orchestration Internals +# Application Orchestration Internals -This document describes the workflow coordinator in `internal/app`. - -## Purpose - -`internal/app` coordinates the top-level use cases after CLI parsing and config -loading are complete. It resolves report definitions, collects weather data -through `internal/collect`, builds collected and derived facts, builds module -snapshots and prompt-input artifacts, invokes Scriptorium through the adapter -boundary, optionally notifies distributor through an app-owned notifier -boundary, persists managed state, runs batches, and reads existing artifacts -for inspection. +`internal/app` composes top-level generation, batch, collection-save, and +inspection workflows after CLI parsing and configuration loading. It owns +workflow ordering, request composition, partial-result handling, and the +application-facing interfaces used for tests. ## Inputs And Outputs -Inputs: +The package accepts generate, resolved-report, batch, explicit-collection, and +inspection requests. Generation and batch requests may supply collector, +renderer, store, and notifier implementations for tests; production defaults +use the focused packages. -- `GenerateRequest` for one report command -- `BatchRequest` for morning or evening batch commands -- `FetchBundleRequest` for explicit bundle collection and save workflows -- `ReportRequest` for single-report generation -- resolved report definitions from `internal/report` -- collection results from `internal/collect` -- prior snapshots loaded from `internal/state` -- optional collector, renderer, notifier, and state-store fakes for tests +A report result contains the module snapshot, prompt package, available +Scriptorium results, generated-text artifacts when used, report and metadata +paths, prior snapshot, Recent Changes, and notification information. A batch +result contains aggregate counts, per-report outcomes, and an optional batch +notification. Inspection returns persisted values only. -Outputs: +Exact public command syntax, configuration fields, workspace layout, external +protocols, and report definitions belong in [the CLI reference](../cli.md), +[the configuration reference](../config.md), [operations](../operations.md), +and their focused integration and internal documents. -- generated report results with JSON module snapshot, YAML data package, - preflight, report, metadata, prior snapshot, Recent Changes, Scriptorium - result details, generated-text artifact paths when applicable, and - notification result when attempted -- batch summaries with per-report status, artifact paths, error text, and - one top-level batch notification result when attempted or skipped -- saved Weather API bundle JSON for explicit bundle collection workflows -- inspection JSON values for reports, metadata, module snapshots, data - packages, prior snapshots, and source provenance +## Single-Report Workflow -## Boundaries +`GenerateDetailed` first collects weather data, then resolves the requested +report using the configured registry and current time, and finally calls +`GenerateReport` with that explicit collection. It returns no result when +collection or resolution fails. -`internal/app` owns workflow order and request composition. It does not parse -CLI flags, load YAML files directly, implement HTTP transport, own fact -derivation algorithms, define report periods, compare rendered Markdown, or -construct Scriptorium argv. +`GenerateReport` requires a non-nil normalized bundle and then performs this +ordered work: -Report selection and report identity policy come from `internal/report`. -Collected and derived fact contracts come from `internal/facts`. -Weather API transport stays in `internal/adapters/weatherapi`, and app-facing -upstream collection stays in `internal/collect`. Scriptorium subprocess -behavior stays in `internal/adapters/scriptorium`. Distributor upload behavior -stays in `internal/adapters/distributor`. Filesystem layout and persisted -metadata stay in `internal/state`. +1. Select a state store, determine artifact destinations, and locate a prior + compatible snapshot. +2. Build report facts and deterministic module snapshots, then save the module + snapshot and calculate Recent Changes. +3. Build and save the prompt data package, run Scriptorium render preflight, + save any preflight result, and save initial metadata. +4. Produce managed Markdown according to the report generation mode. +5. Finalize metadata, optionally make an output copy, and optionally notify + Distributor from the managed report path. -## Data Flow Terms +Direct-Markdown reports prepare the managed report and invoke the Scriptorium +run boundary. Generated-text-template reports look up their catalog definition, +run structured Scriptorium output to the raw artifact, preserve any structured +run result, validate and save generated text, build and save a render context, +then render the embedded Markdown template. Schema, template, and subprocess +details remain in their [generated-text](generatedtext.md), +[report-template](reporttemplate.md), and [Scriptorium adapter](scriptorium-adapter.md) +owners. -- `collect.Result` is the app-facing upstream collection result. It carries the - normalized `weatherdata.Bundle` used by report generation. -- `CollectedFacts` are normalized source facts derived from a collected Weather - API bundle and made available to derivation and module builders. -- `DerivedFacts` are deterministic calculations over collected facts, the - resolved valid period, daypart configuration, and report-specific windows. -- `module.Output` values are ordered deterministic stanzas built from collected - and derived facts for prompt input and inspection. -- `GeneratedText` is structured prose returned by Scriptorium for - generated-text-template reports and validated by `internal/generatedtext`. -- `RenderContext` is the typed template input built from report metadata, - module outputs, and validated generated text before Markdown rendering. +If preflight returns a result with an error, the result and initial metadata are +saved before the error returns. If report generation fails after a managed path +is prepared, metadata still records that path; output copies and notification +are skipped. Generated-text failures preserve the latest artifact reached +before failure when it was saved. -## Config Fields Used +## Batch And Inspection Workflows -- `weather_api.*` for Weather API client construction and module metadata -- `scriptorium.*` for renderer construction -- `workspace.*` for filesystem state -- `dayparts` for daily and outlook summarization -- `recent_change.*` for structured Recent Changes thresholds -- `notify.distributor.*` for optional single-report and batch notification - after report generation +`RunBatchDetailed` collects once, asks the report registry to plan the batch +from that collection, and invokes `GenerateReport` independently for every +planned report using the same collection and state store. Per-report +notification is suppressed. A failed report is recorded and does not prevent +later planned reports from running. -Output copy flags are command request fields. They are not configuration -defaults. +After report generation, the batch notifier is considered once. It is omitted +when Distributor or batch notification is disabled, skipped when any report +failed, and otherwise receives one multi-file request. A batch notification +failure increments the aggregate failure count but does not rewrite successful +report items. Notification identities, path mappings, polling, and redaction +are owned by the [Distributor adapter](distributor-adapter.md). -## Generation Workflow +Inspection methods create a state store and load existing report records, +metadata, module snapshots, prompt packages, prior snapshots, or source +provenance. They neither collect data nor invoke Scriptorium or Distributor. -Single-report commands validate the report command, collect once through -`internal/collect`, resolve the requested report, and pass the resolved report -plus explicit collection into `GenerateReport`. `GenerateDetailed` returns the -resulting `ReportResult`; `Generate` wraps the same workflow for error-only -callers. +## Boundaries And Failure Propagation -`GenerateReport` then uses this setup: +The app layer does not parse flags, load configuration files, implement Weather +API transport, construct Scriptorium argv, or define report registry policy. It +coordinates the relevant collaborators and preserves their error context. -1. Create or use a filesystem store. -2. Locate any prior compatible snapshot through `internal/state`. -3. Build collected and derived facts from the supplied collection. -4. Execute configured modules and save the module snapshot. -5. Compute Recent Changes from structured prior and current module snapshots. -6. Build and save the YAML Scriptorium `data_package`. -7. Run Scriptorium render preflight. -8. Save preflight JSON when a render result is available. -9. Save metadata for inspection. +- Collection failure stops a single report or batch before resolution or + planning completes. +- State, fact, module, prompt-input, or preflight failures stop that report + before report generation. +- A terminal Distributor failure is returned with the saved notification + information when available. +- Batch failures are represented per report and through aggregate batch status. +- Persisted artifact paths are carried in results so callers can inspect work + completed before a later failure. -For `scriptorium_markdown` reports, generation then: +## Tests And Invariants -10. Runs Scriptorium report generation to the managed report path. +Focused tests are in `internal/app/app_test.go` and +`internal/app/batch_plan_test.go`, with collection coverage in +`internal/collect/collect_test.go`. -For `generated_text_template` reports, generation then: - -10. Looks up the generated-text catalog entry for the report schema/template - IDs. -11. Runs structured Scriptorium generation to the raw generated-text JSON path. -12. Saves the structured Scriptorium run result. -13. Validates and saves normalized generated text. -14. Builds and saves a typed render context. -15. Renders Markdown from the embedded template to the managed report path. - -After either mode has produced a managed Markdown report, shared finalization: - -1. Copies the managed report to the requested `--out` or `--out-dir` path when - provided. -2. Saves final metadata with the managed report path and any generated-text - artifact paths already produced. -3. If distributor notification is enabled, notifies using the managed report - path as the source file. -4. Saves a distributor notification debug artifact and updates metadata with - its path. - -If render preflight returns both a result and an error, preflight JSON and -metadata are persisted before the error is returned. If Scriptorium report -generation returns an error after writing output, the managed report and -metadata remain inspectable. Notification is not attempted after collection, -module snapshot, prompt input, render, Scriptorium run, or metadata-save -failures. -Generated-text report failures are returned with report ID, RunID, and the -failed operation. When available, the app preserves the latest generated-text -artifacts already reached by the workflow: preflight output, structured run -result, raw generated text, validated generated text, and render context. -When notification is attempted, the debug artifact records request identity, -including rendered pipeline ID, bundle paths, accepted upload fields, -distributor status fields, raw status report JSON when available, and redacted -failure context. -`--out` copies are never used as notification source files. - -## Batch Workflow - -`run morning` collects once, plans Today Report, Tomorrow Report, and eligible -future Daily Reports from the collected hourly forecast, then passes the same -collection into each report generation. `run evening` uses the same collection -and planning rules, but starts with Tomorrow Report. Future Daily reports start -with the day after tomorrow and require complete hourly forecast coverage for -the target local civil day. Dynamic Daily `--out-dir` copies use -`daily-YYYY-MM-DD.md`; other batch copies use report definition output names. -A collection failure stops the batch before planning or report generation. -After planning succeeds, batch generation continues independent reports after a -failure, records each result, writes compact status lines to stderr, emits a -JSON summary to stdout, and returns an aggregate error when any report failed. - -Batch report generation suppresses per-report distributor notification. After -all planned reports finish, app orchestration evaluates batch notification: - -1. If distributor notification is disabled, the batch notification result is - omitted. -2. If batch notification is disabled, the batch notification result is omitted - and there is no per-report fallback upload. -3. If any planned report failed, the batch notification result is `skipped` - with reason `one or more reports failed`, and distributor is not called. -4. If every report succeeded, app orchestration renders batch pipeline, bundle - ID, and idempotency key templates, renders report-specific distributor - paths for each included report, validates every managed source path and - bundle path, checks duplicate bundle paths across the batch, calls the - notifier once with a multi-file request, and saves a batch notification - debug artifact. - -Batch notification failure records a top-level failed notification, increments -the aggregate batch failure count, and returns an aggregate batch error without -marking individual report items failed. `--out-dir` copies are never used as -notification source files. - -## Inspection Workflow - -Inspection workflows load existing filesystem state only. They do not fetch -weather data or invoke Scriptorium. Run-specific inspect commands share the same -store and metadata lookup path, then load the requested artifact or derived -inspection view. - -## Failure Behavior - -- Resolve errors stop the requested workflow before collection. -- Collection and module execution errors stop that report before Scriptorium - runs. -- Prompt input validation fails before render preflight. -- Render and run errors preserve Scriptorium stderr and exit-code context. -- Generated-text report errors preserve available intermediate artifacts and do - not create extra output copies. -- Single-report notification errors are wrapped with report ID, RunID, and - managed report path context. Detailed generation returns the inspectable - report, metadata, and notification artifact paths when finalization has - already saved them. -- Batch notification errors are recorded on the top-level batch notification - result and do not change individual report item status. -- Metadata and artifact path errors include filesystem context. -- Batch failures are recorded per report and surfaced through an aggregate - batch error. - -## Tests - -Inspect: - -- `internal/app/app_test.go` -- `internal/app/batch_plan_test.go` -- `internal/collect/collect_test.go` -- `internal/cli/root_test.go` -- `internal/state/filesystem_test.go` - -## Invariants - -- Report behavior is resolved through `internal/report`. -- Generate and run commands collect once before report generation. -- Batch planning is app-owned because future Daily membership depends on - collected hourly forecast coverage. -- Generated reports use the same app request and result types regardless of - report ID. -- Render preflight precedes Scriptorium report generation. -- Generated-text reports render Markdown from a curated render context, not from - a raw data package. -- Recent Changes are computed from structured module snapshots. -- Metadata links artifacts produced for a run. -- Single-report distributor notification maps the managed Markdown report path - to configured bundle paths. -- Batch distributor notification maps each included managed Markdown report - path to bundle paths rendered for that report and uploads once for the - batch. -- Extra output copies are not upload sources. +- Production workflows collect through `internal/collect`. +- A report uses one explicit normalized collection throughout its generation. +- Render preflight precedes report generation. +- Recent Changes compare structured module snapshots. +- Generated-text reports render from a validated typed context, never directly + from a raw prompt package. +- Only managed Markdown reports are notification sources; output copies are + never uploaded. diff --git a/docs/internal/cli.md b/docs/internal/cli.md index 50c4e13..8a5365b 100644 --- a/docs/internal/cli.md +++ b/docs/internal/cli.md @@ -1,67 +1,65 @@ # CLI Internals -This document describes command output ownership in `internal/cli`. +`internal/cli` turns process arguments into application requests and translates +application results into terminal output. The user-facing command, flag, and +output contract belongs in the [CLI reference](../cli.md). -## Purpose +## Responsibilities -`internal/cli` owns command parsing, app request construction, help text, and -presentation of command results. It converts app-layer results into stable CLI -summaries and writes stdout/stderr through shared output helpers. +`Runner.Run` dispatches the top-level action or inspection request. For actions, +the package parses command-specific and common flags, loads configuration with +CLI overrides, obtains the current time, and constructs either an +`app.GenerateRequest` or an `app.BatchRequest`. It delegates generation and +batch execution to `internal/app`. -## Command Categories +For inspection, it loads configuration, builds the appropriate app inspection +request, and writes the returned value. Inspection is read-only; the inspected +artifact types and user invocation remain owned by the [CLI reference](../cli.md) +and [operations guide](../operations.md). -- Action commands: `generate` and `run`. These perform work, write artifacts, - and return compact summaries. -- Inspection commands: `inspect reports`, `inspect metadata`, `inspect - modules`, `inspect data-package`, `inspect prior`, and `inspect sources`. - These read existing artifacts and return requested data. +## Result Translation -Future commands must declare which category they belong to before adding output -behavior. +Action results become CLI-safe JSON summaries in `result.go`. Generate summaries +carry report identity, status, relevant artifact paths, and notification +summary data. Batch summaries carry aggregate counts, per-report outcomes, and +the optional batch notification result. The translation deliberately excludes +full module snapshots, prompt packages, raw generated text, Scriptorium output, +and complete Distributor payloads. -## Stdout And Stderr +When an action returns both a result and an error, the CLI writes the failed +summary before returning that error. Parse, configuration-load, and other +failures that produce no application result return without a summary. -Action commands write JSON summaries to stdout by default. `run` also writes -compact status lines to stderr through `writeBatchStatus`. `generate` does not -write routine stderr today. Pre-run errors return without partial JSON. +`writeActionResult` writes action status information to stderr first, then JSON +to stdout. Batch execution supplies the status writer; single-report generation +does not emit routine stderr output. Quiet action requests suppress both normal +streams but still return errors. Inspection writes its JSON value to stdout and +does not accept quiet mode because stdout is the inspection result. -Inspection commands write requested JSON data to stdout with `writeJSON`. They -do not use action output helpers and do not support quiet mode. +## Boundaries -Returned errors are not hidden by output helpers. The caller remains -responsible for displaying command errors. +The package owns argument parsing, request adaptation, help text, and terminal +presentation. It does not implement report selection, collection, state +persistence, external transport, subprocess execution, or notification policy. +Those concerns remain in [application orchestration](app-orchestration.md) and +their focused owners. -## Quiet Mode +## Failure Behavior -`--quiet` is supported only by action commands. It suppresses successful stdout -and routine stderr by passing `outputOptions{Quiet: true}` to -`writeActionResult`. It does not suppress returned errors. +- Invalid command names, flags, dates, and configuration fail before an app + request is executed. +- Application errors retain their application context; output helpers do not + hide or replace them. +- JSON-encoding errors are returned directly. +- A failed batch summary causes the CLI to return an aggregate batch error even + when the detailed batch call has already returned its result. -Quiet mode is intentionally not accepted by inspection commands because -inspection stdout is the command result. +## Tests And Invariants -## Summary Ownership +Focused tests are in `internal/cli/root_test.go`, `internal/cli/output_test.go`, +and `internal/cli/result_test.go`. -CLI-safe summary structs live in `internal/cli/result.go`. - -- `newGenerateSummary` converts `*app.ReportResult` plus an optional error into - the generate JSON contract. -- `newBatchSummary` converts `*app.BatchResult` into the run JSON contract and - derives the top-level run status. - -Summary types must not expose full app internals, module contents, data package -contents, raw generated text, Scriptorium result bodies, or full distributor -payloads. - -## Helper Path - -New action commands should: - -1. parse command-specific flags into CLI option structs; -2. call the app-layer use case; -3. convert app results into a CLI summary type; -4. write through `writeActionResult`; -5. use a status writer only for routine stderr status lines. - -New inspection commands should call the app inspection use case and write the -returned data through `writeJSON`. +- CLI summaries are stable, bounded views of app results. +- Routine batch status lines precede the batch JSON summary. +- A quiet action produces no successful or failure summary output. +- Inspection never invokes action-output helpers. diff --git a/docs/internal/collect.md b/docs/internal/collect.md index 71cf40f..ed6f29d 100644 --- a/docs/internal/collect.md +++ b/docs/internal/collect.md @@ -1,58 +1,43 @@ # Collection Internals -This document describes the app-facing upstream collection boundary in -`internal/collect`. - -## Purpose - -`internal/collect` is the canonical package used by app workflows to collect -upstream Weather API data. It constructs the Weather API adapter, fetches a -normalized bundle, and returns that bundle without applying report selection or -batch policy. +`internal/collect` is the application-facing boundary for collecting the +normalized Weather API bundle. The external HTTP contract belongs in the +[Weather API integration guide](../integrations/weatherapi.md); normalized data +semantics belong in [weather-data internals](weather-data.md). ## Contract -Inputs: +`Run` accepts a `context.Context` and a `Request` containing effective +`config.Config`. It constructs the Weather API adapter from that configuration, +calls `FetchBundle`, and returns `Result{Bundle: *weatherdata.Bundle}`. -- `collect.Request`, containing the effective `config.Config` -- `context.Context` for cancellation +The package wraps adapter construction failures as weather-collection setup +errors and fetch failures as bundle-collection errors. It does not retry, +persist, select reports, derive facts, build modules, invoke Scriptorium, or +notify Distributor. -Output: +## Application Composition -- `collect.Result`, containing `*weatherdata.Bundle` +`internal/app` owns the narrow `Collector` interface used by workflow tests; +the production implementation delegates to `collect.Run`. Generation, batch +execution, and explicit bundle fetching all use this boundary. Application +orchestration rejects a nil collector result or a nil bundle before report work +can continue. -`Run` returns an actionable error when Weather API adapter construction or -bundle fetch fails. The package does not derive `facts.CollectedFacts`, build -modules, resolve report periods, select reports, write state, invoke -Scriptorium, or notify distributor. +Single-report generation and a batch each collect once. A batch passes the same +normalized collection to planning and to every report it generates. Collection +failure prevents later workflow work for that request. -## App Usage +## Boundaries And Invariants -`internal/app` owns a narrow `Collector` interface for orchestration tests. The -default implementation calls `collect.Run`. +Collection owns adapter creation and retrieval of one normalized bundle. It +must not make report, period, batch, prompt, module, filesystem, or notification +decisions. -Single-report generation collects once, resolves the requested report, and -passes the explicit collection into report generation. Batch generation -collects once before planning and passes the same collection into each planned -report. If collection returns no bundle, app orchestration returns an error -before report generation. +- App-facing Weather API collection always passes through this package. +- The returned value is normalized source data, not facts or prompt input. +- Context cancellation is passed to the Weather API adapter. +- Errors retain whether setup or fetching failed. -## Boundaries - -Weather API HTTP details stay in `internal/adapters/weatherapi`. The collection -package returns normalized `weatherdata` only. It must not know about report -IDs, prompt IDs, batch names, Daily eligibility, module composition, Recent -Changes, state paths, or Scriptorium arguments. - -## Tests - -Inspect: - -- `internal/collect/collect_test.go` -- `internal/app/app_test.go` - -## Invariants - -- App-facing Weather API collection goes through `internal/collect`. -- Collection returns normalized source data, not report facts or prompt input. -- Report and batch policy belongs outside `internal/collect`. +Focused tests are in `internal/collect/collect_test.go`; orchestration use is +also covered by `internal/app/app_test.go`.