diff --git a/docs/config.md b/docs/config.md index c4dc862..da79701 100644 --- a/docs/config.md +++ b/docs/config.md @@ -78,8 +78,10 @@ missing directories, and unreadable files fail config loading. ### `notify` -`notify.distributor` is validated configuration for distributor notification. -It is disabled by default. This configuration does not add CLI flags. +`notify.distributor` controls distributor notification after successful report +generation. It is disabled by default and does not add CLI flags. When enabled, +weatherreporter uploads one distributor bundle per generated report after +`scriptorium run` succeeds and final metadata is saved. - `enabled`: whether distributor notification config is active. Default: `false`. @@ -105,6 +107,10 @@ Rendered report paths must be relative paths with `/` separators. They must not contain backslashes, empty path segments, `.`, `..`, `manifest.json`, or `.distributor.json`. +The upload token is read from the environment variable named by `token_env` +after config loading and `secrets.directory` processing. Config files should +name the variable only; they should not contain the token value. + ### `missing_source` - `default`: missing-source behavior for optional sources. One of `error`, `warn`, or `none`. Default: `warn`. @@ -158,7 +164,11 @@ snapshot exists and a threshold is crossed. Configuration files should not contain raw secrets. Use `secrets.directory` to load secret values from files into environment variables for integrations that -read credentials from the environment. +read credentials from the environment. Secret file names become environment +variable names, and secret file contents become values. For distributor +notification, this allows a file such as +`/DISTRIBUTOR_UPLOAD_TOKEN` to supply the token referenced by +`notify.distributor.token_env`. ## Maintained Examples diff --git a/docs/internal/app-orchestration.md b/docs/internal/app-orchestration.md index 1550406..689de7b 100644 --- a/docs/internal/app-orchestration.md +++ b/docs/internal/app-orchestration.md @@ -7,8 +7,9 @@ This document describes the implemented workflow coordinator in `internal/app`. `internal/app` coordinates the top-level use cases after CLI parsing and config loading are complete. It resolves report definitions, fetches weather data, builds briefing and prompt-input artifacts, invokes Scriptorium through the -adapter boundary, persists managed state, runs batches, and reads existing -artifacts for inspection. +adapter boundary, optionally notifies distributor through an app-owned notifier +boundary, persists managed state, runs batches, and reads existing artifacts for +inspection. ## Inputs And Outputs @@ -22,13 +23,15 @@ Inputs: - resolved report definitions from `internal/report` - forecast bundles from `internal/adapters/weatherapi` - prior snapshots loaded from `internal/state` -- optional renderer and state-store fakes for tests +- optional renderer, notifier, and state-store fakes for tests Outputs: - generated report results with briefing, data package, preflight, report, - metadata, prior snapshot, Recent Changes, and Scriptorium result details -- batch summaries with per-report status, artifact paths, and error text + metadata, prior snapshot, Recent Changes, Scriptorium result details, and + notification result when attempted +- batch summaries with per-report status, artifact paths, error text, and + notification outcome when attempted - saved Weather API bundle JSON for fetch workflows - inspection JSON values for reports, metadata, briefings, data packages, prior snapshots, and source provenance @@ -42,8 +45,9 @@ Scriptorium argv. Report selection and report identity policy come from `internal/report`. Weather API transport stays in `internal/adapters/weatherapi`. Scriptorium -subprocess behavior stays in `internal/adapters/scriptorium`. Filesystem layout -and persisted metadata stay in `internal/state`. +subprocess behavior stays in `internal/adapters/scriptorium`. Distributor +upload behavior stays in `internal/adapters/distributor`. Filesystem layout and +persisted metadata stay in `internal/state`. ## Config Fields Used @@ -52,6 +56,7 @@ and persisted metadata stay in `internal/state`. - `workspace.*` for filesystem state - `dayparts` for daily and outlook summarization - `recent_change.*` for structured Recent Changes thresholds +- `notify.distributor.*` for optional notification after report generation Output copy flags are command request fields. They are not configuration defaults. @@ -74,11 +79,15 @@ Single-report generation follows this order: 12. Run Scriptorium report generation to the managed report path. 13. Copy the managed report to the requested `--out` path when provided. 14. Save metadata with the managed report path. +15. If distributor notification is enabled, notify using the managed report + path as the source file. If render preflight returns both a result and an error, preflight JSON and metadata are persisted before the error is returned. If Scriptorium report generation returns an error after writing output, the managed report and -metadata remain inspectable. +metadata remain inspectable. Notification is not attempted after Weather API, +briefing, prompt input, render, Scriptorium run, or metadata-save failures. +`--out` copies are never used as notification source files. ## Batch Workflow @@ -87,6 +96,10 @@ on Sunday. `run evening` resolves Daily Tomorrow. Batch output copy names come from report definitions. Batch generation continues independent reports after a failure, records each result, writes compact status lines to stderr, emits a JSON summary to stdout, and returns an aggregate error when any report failed. +When notification is enabled, each successfully generated report is notified +independently. Notification failure marks that report failed, records +notification fields in the batch result, and does not stop later reports. +`--out-dir` copies are never used as notification source files. ## Inspection Workflow @@ -101,6 +114,8 @@ inspection view. - Weather API and briefing errors stop that report before Scriptorium runs. - Prompt input validation fails before render preflight. - Render and run errors preserve Scriptorium stderr and exit-code context. +- Notification errors are wrapped with report ID, RunID, and managed report path + context and are recorded separately in batch results. - Metadata and artifact path errors include filesystem context. - Batch failures are recorded per report and surfaced through an aggregate batch error. @@ -121,3 +136,5 @@ Inspect: - Render preflight precedes Scriptorium report generation. - Recent Changes are computed from structured briefing snapshots. - Metadata links artifacts produced for a run. +- Distributor notification uses the managed Markdown report path, not extra + output copies. diff --git a/docs/internal/distributor-adapter.md b/docs/internal/distributor-adapter.md new file mode 100644 index 0000000..df0bfda --- /dev/null +++ b/docs/internal/distributor-adapter.md @@ -0,0 +1,106 @@ +# Distributor Adapter Internals + +This document describes the distributor upload adapter in +`internal/adapters/distributor`. + +## Purpose + +The adapter submits generated weatherreporter Markdown reports to a configured +distributor HTTP upload endpoint. It isolates distributor package types, +token-env lookup, upload client construction, source-bundle file mapping, +timeout handling, and upload error wrapping from app orchestration. + +## Inputs And Outputs + +Inputs: + +- distributor endpoint URL +- token environment variable name +- upload timeout +- bundle ID +- idempotency key +- source Markdown report path +- bundle-relative Markdown path +- context for cancellation + +Outputs: + +- accepted distributor run ID +- accepted distributor status +- weatherreporter-owned idempotency conflict error when applicable + +## Boundaries + +`internal/adapters/distributor` is the only weatherreporter package that imports +`gitea.maximumdirect.net/eric/distributor/pkg/upload` or +`gitea.maximumdirect.net/eric/distributor/pkg/bundle`. + +The app layer passes weatherreporter-owned request values to the adapter. The +adapter does not choose report types, render templates, select output copies, +configure destinations, wait for downstream publication, transform Markdown, or +persist notification state. + +Full upstream distributor package and HTTP contract details stay under +`docs/integrations/distributor/`. + +## Config Fields Used + +The adapter is built from `notify.distributor` config: + +- `endpoint` +- `token_env` +- `timeout` + +The app layer renders bundle ID, idempotency key, and bundle path from: + +- `bundle_id_template` +- `idempotency_key_template` +- `report_path_template` + +The token value is read from the environment variable named by `token_env` +after config loading and `secrets.directory` processing. + +## Upload Behavior + +The adapter calls distributor `UploadFiles` with exactly one file: + +- source path: the managed Markdown report path selected by app orchestration +- bundle path: the rendered bundle-relative report path + +The adapter creates a distributor upload client with the configured endpoint, +bearer token, and timeout-backed HTTP client. It also wraps the upload context +with the configured timeout when the timeout is greater than zero. + +## Failure Behavior + +The adapter validates required endpoint, token env name, token value, bundle ID, +idempotency key, source path, bundle path, and upload client inputs before +uploading. + +Upload failures include endpoint, bundle ID, idempotency key, source path, and +bundle path context. Token values are redacted from adapter errors. + +Distributor idempotency conflicts are exposed as a weatherreporter-owned +`IdempotencyConflictError`, so callers do not depend on upstream distributor +types. + +## Tests + +Inspect: + +- `internal/adapters/distributor/client_test.go` +- `internal/app/app_test.go` +- `internal/cli/root_test.go` + +Adapter tests use a fake upload client factory and do not require a live +distributor service. + +## Invariants + +- Distributor package types do not leak outside the adapter. +- Only the managed Markdown report is uploaded. +- The adapter never scans the workspace. +- Token values are not included in errors, CLI output, metadata, docs, or + examples. +- Destination routing and Markdown-to-HTML transformation belong to + distributor, not weatherreporter. diff --git a/docs/operations.md b/docs/operations.md index 6607ecf..b43ef66 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -19,7 +19,10 @@ weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00 Each command resolves a report period, fetches a Weather API bundle, builds a briefing, builds a prompt input data package, runs `scriptorium render`, runs `scriptorium run`, and writes managed artifacts under the configured workspace. -`--out PATH` writes an extra Markdown copy for the current generated report. +When distributor notification is enabled, weatherreporter uploads the managed +Markdown report after `scriptorium run` succeeds and final metadata is saved. +`--out PATH` writes an extra Markdown copy for the current generated report; it +is not used as the distributor upload source. Implemented batch commands: @@ -36,7 +39,8 @@ nonzero when any report failed. When notification is configured, the summary and status lines include notification status, accepted distributor run ID, or notification error fields for each attempted report. `--out-dir PATH` writes extra Markdown copies using report default filenames such as `daily.md`, -`three-day.md`, `weekend.md`, and `tomorrow.md`. +`three-day.md`, `weekend.md`, and `tomorrow.md`; these copies are not used as +distributor upload sources. ## Filesystem Layout @@ -121,7 +125,38 @@ Each generated report writes metadata that links: Batch summaries include report status, error text when applicable, notification outcome when attempted, valid period, and known artifact paths for each -attempted report. +attempted report. Notification fields are `notificationStatus`, +`notificationRunId`, and `notificationError`. + +## Distributor Notification + +Distributor notification is configured with `notify.distributor` and is disabled +by default. When enabled, weatherreporter uploads exactly one file per +successfully generated report: the managed Markdown report path recorded in the +report result and metadata. Extra copies written by `--out` or `--out-dir` are +operator conveniences only. + +The default bundle ID is derived from producer name, location ID, report ID, and +RunID: + +```text +weatherreporter.{location_id}.{report_id}.{run_id} +``` + +The default idempotency key is the rendered bundle ID. The default bundle path +for the Markdown file is the report definition's batch output name, such as +`daily.md`, `tomorrow.md`, `three-day.md`, or `weekend.md`. + +Notification happens after final metadata save. Weather API, briefing, +data-package, render preflight, Scriptorium run, and metadata-save failures do +not trigger notification. A notification failure fails that report. In a batch, +other reports continue, the failed report includes notification fields in the +JSON summary, and the batch returns nonzero. + +Weatherreporter is responsible for selecting the managed Markdown report, +constructing a source bundle, and submitting it to the configured distributor +HTTP endpoint. Distributor remains responsible for destination routing, +publication, and any downstream Markdown-to-HTML transformation. ## Inspection @@ -167,6 +202,8 @@ A failed generation run may still leave useful artifacts: preflight JSON and metadata are written for inspection. - If `scriptorium run` exits nonzero after writing a report, the managed report and metadata remain available. +- If distributor notification fails, report artifacts and final metadata remain + available, but the report or batch command returns nonzero. - For batch commands, inspect the stdout JSON summary first, then inspect the artifact paths for each failed report. diff --git a/docs/troubleshooting.md b/docs/troubleshooting.md index 0c2759e..c8da3e4 100644 --- a/docs/troubleshooting.md +++ b/docs/troubleshooting.md @@ -199,6 +199,101 @@ metadata, sources, briefing, and data package for that RunID. Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md). +## Invalid Secrets Directory + +Symptom: config loading fails with `read secrets directory`, `secret file`, or +environment variable name context. + +Likely cause: `secrets.directory` points to a missing directory or contains an +invalid entry. Secret entries must be regular files directly under the +configured directory, and file basenames must match +`[A-Za-z_][A-Za-z0-9_]*`. + +Diagnostic: list the configured directory and inspect entry names and file +types. Do not print secret file contents. + +Safe fix: create the directory, remove subdirectories or symlinks, fix invalid +filenames, and ensure the weatherreporter process can read each secret file. + +Relevant docs: [Configuration reference](config.md). + +## Distributor Token Is Missing + +Symptom: notification fails with a message that the distributor token +environment variable is not set. + +Likely cause: `notify.distributor.enabled` is true, but the environment +variable named by `notify.distributor.token_env` was not populated directly or +through `secrets.directory`. + +Diagnostic: check `notify.distributor.token_env`, then verify a matching secret +file exists under `secrets.directory` or that the process environment includes +the variable. Do not print the token value. + +Safe fix: create a readable secret file whose basename matches `token_env`, or +set the environment variable through the service manager. + +Relevant docs: [Configuration reference](config.md), +[Operations guide](operations.md). + +## Distributor Upload Conflict + +Symptom: notification fails with idempotency conflict context. + +Likely cause: the same idempotency key was reused for different bundle content. +By default the key derives from bundle ID, which includes location ID, report +ID, and RunID. + +Diagnostic: inspect the failed batch JSON or stderr line for bundle and +idempotency context. Compare the configured templates with the report RunID and +report path. + +Safe fix: keep idempotency templates stable for retries of the same generated +report, but do not reuse the same rendered key for different generated report +content. + +Relevant docs: [Operations guide](operations.md), +[Distributor adapter internals](internal/distributor-adapter.md). + +## Distributor Upload Rejected + +Symptom: notification fails with distributor upload rejection, HTTP status, or +bundle validation context. + +Likely cause: the distributor endpoint rejected the token, bundle ID, +idempotency key, source file, or bundle path. + +Diagnostic: inspect stdout JSON or stderr status lines for +`notificationError`. Confirm `notify.distributor.endpoint`, +`notify.distributor.report_path_template`, and token configuration. Token values +are redacted from weatherreporter errors. + +Safe fix: fix the endpoint, token, templates, or distributor-side upload +configuration. The weatherreporter upload source is the managed Markdown report, +not `--out` or `--out-dir` copies. + +Relevant docs: [Configuration reference](config.md), +[Operations guide](operations.md), +[Distributor adapter internals](internal/distributor-adapter.md). + +## Distributor Unavailable + +Symptom: notification fails with network, timeout, or service unavailable +context. + +Likely cause: the configured distributor endpoint is unreachable, slow, or +temporarily unavailable. + +Diagnostic: check network access from the weatherreporter host to +`notify.distributor.endpoint`. For batch runs, inspect which reports have +`notificationStatus: "failed"`. + +Safe fix: restore distributor service availability and rerun the affected +report or batch. Stable idempotency keys make retrying the same generated report +safe unless the distributor reports a conflict. + +Relevant docs: [Operations guide](operations.md). + ## Unknown RunID Symptom: an inspect command fails with `metadata for run id ... was not found`.