Document batch distributor uploads
This commit is contained in:
10
docs/cli.md
10
docs/cli.md
@@ -60,11 +60,13 @@ stops the batch before any report is generated. Batch runs continue independent
|
|||||||
reports after a later report failure, print a JSON summary to stdout, write
|
reports after a later report failure, print a JSON summary to stdout, write
|
||||||
compact status lines to stderr, and return nonzero when any report failed.
|
compact status lines to stderr, and return nonzero when any report failed.
|
||||||
`--out-dir` writes extra Markdown copies for the operator; distributor
|
`--out-dir` writes extra Markdown copies for the operator; distributor
|
||||||
notification uses each managed report path, not the extra copies. Today and
|
notification uses managed report paths, not the extra copies. Today and
|
||||||
Tomorrow use their report default copy names, and dynamic Daily copies use
|
Tomorrow use their report default copy names, and dynamic Daily copies use
|
||||||
`daily-YYYY-MM-DD.md`. When notification is enabled, batch summaries and status
|
`daily-YYYY-MM-DD.md`. When distributor and batch notification are enabled, a
|
||||||
lines include notification status, accepted distributor run ID, or notification
|
fully successful batch uploads one distributor bundle after report generation
|
||||||
error fields for each attempted report.
|
finishes. The JSON summary exposes that upload as a top-level `notification`
|
||||||
|
object, and stderr includes one `batchNotification` status line. If any planned
|
||||||
|
report fails, the batch notification is skipped for the whole batch.
|
||||||
|
|
||||||
Hourly Report, 3-Day Outlook, and Weekend Outlook are explicit only; they are
|
Hourly Report, 3-Day Outlook, and Weekend Outlook are explicit only; they are
|
||||||
not included in `run morning` or `run evening`.
|
not included in `run morning` or `run evening`.
|
||||||
|
|||||||
@@ -78,10 +78,13 @@ missing directories, and unreadable files fail config loading.
|
|||||||
|
|
||||||
### `notify`
|
### `notify`
|
||||||
|
|
||||||
`notify.distributor` controls distributor notification after successful report
|
`notify.distributor` controls distributor uploads after successful report
|
||||||
generation. It is disabled by default and does not add CLI flags. When enabled,
|
rendering. It is disabled by default and does not add CLI flags. When enabled,
|
||||||
weatherreporter uploads one distributor bundle per generated report after
|
`generate <report>` uploads one distributor bundle for the generated report
|
||||||
report rendering succeeds and final metadata is saved.
|
after final metadata is saved. `run morning` and `run evening` use
|
||||||
|
`notify.distributor.batch`: when batch notification is enabled and every
|
||||||
|
planned report succeeds, weatherreporter uploads one distributor bundle that
|
||||||
|
contains all managed Markdown reports from that batch.
|
||||||
|
|
||||||
- `enabled`: whether distributor notification config is active. Default:
|
- `enabled`: whether distributor notification config is active. Default:
|
||||||
`false`.
|
`false`.
|
||||||
@@ -92,15 +95,18 @@ report rendering succeeds and final metadata is saved.
|
|||||||
- `timeout`: distributor operation timeout. Must be greater than zero when
|
- `timeout`: distributor operation timeout. Must be greater than zero when
|
||||||
enabled. Default: `30s`.
|
enabled. Default: `30s`.
|
||||||
- `failure_policy`: must be `error` when enabled. Default: `error`.
|
- `failure_policy`: must be `error` when enabled. Default: `error`.
|
||||||
- `pipeline_id_template`: template for the distributor pipeline ID. Required
|
- `pipeline_id_template`: template for single-report distributor pipeline IDs.
|
||||||
when enabled. Default: empty.
|
Required when enabled. Default: empty.
|
||||||
- `bundle_id_template`: template for distributor bundle IDs. Default:
|
- `bundle_id_template`: template for single-report distributor bundle IDs.
|
||||||
`weatherreporter.{location_id}.{report_id}`.
|
Default: `weatherreporter.{location_id}.{report_id}`.
|
||||||
- `idempotency_key_template`: template for distributor idempotency keys.
|
- `idempotency_key_template`: template for single-report distributor
|
||||||
Default: `{bundle_id}.{run_id}`.
|
idempotency keys. Default: `{bundle_id}.{run_id}`.
|
||||||
- `report_path_templates`: ordered list of templates for Markdown report paths
|
- `report_path_templates`: ordered list of templates for Markdown report paths
|
||||||
inside the distributor bundle. Each rendered path maps to the same managed
|
inside distributor bundles. For `generate <report>`, each rendered path maps
|
||||||
Markdown report source. Default:
|
to the generated report's managed Markdown source. For `run morning` and
|
||||||
|
`run evening`, the templates are rendered once per included report and each
|
||||||
|
rendered path maps that report's managed Markdown source into the one batch
|
||||||
|
bundle. Default:
|
||||||
```yaml
|
```yaml
|
||||||
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
|
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
|
||||||
```
|
```
|
||||||
@@ -116,7 +122,7 @@ report rendering succeeds and final metadata is saved.
|
|||||||
keys. Required when distributor notification and batch notification are
|
keys. Required when distributor notification and batch notification are
|
||||||
enabled. Default: `{bundle_id}.{batch_run_id}`.
|
enabled. Default: `{bundle_id}.{batch_run_id}`.
|
||||||
|
|
||||||
Supported template variables are `location_id`, `report_id`, `run_id`,
|
Single-report templates support `location_id`, `report_id`, `run_id`,
|
||||||
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
|
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
|
||||||
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, and
|
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, and
|
||||||
`valid_end_stamp`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
|
`valid_end_stamp`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
|
||||||
@@ -129,12 +135,16 @@ report stream. The rendered idempotency key is the per-run retry identity.
|
|||||||
|
|
||||||
Batch templates support `location_id`, `batch`, `batch_run_id`, and
|
Batch templates support `location_id`, `batch`, `batch_run_id`, and
|
||||||
`batch_started_date`. Batch idempotency templates may also use `bundle_id`.
|
`batch_started_date`. Batch idempotency templates may also use `bundle_id`.
|
||||||
|
`batch_started_date` is the batch start date in the effective report timezone.
|
||||||
Batch bundle IDs identify a logical batch stream; batch idempotency keys
|
Batch bundle IDs identify a logical batch stream; batch idempotency keys
|
||||||
identify a specific retryable batch attempt.
|
identify a specific retryable batch attempt.
|
||||||
|
|
||||||
Rendered report paths must be unique relative paths with `/` separators. They
|
Rendered report paths must be unique relative paths with `/` separators. They
|
||||||
must not contain backslashes, empty path segments, `.`, `..`, `manifest.json`,
|
must not contain backslashes, empty path segments, `.`, `..`, `manifest.json`,
|
||||||
or `.distributor.json`.
|
or `.distributor.json`. In a batch upload, uniqueness is checked across every
|
||||||
|
rendered bundle path for every included report before distributor is called.
|
||||||
|
Managed Markdown report paths are the only upload source files; copies written
|
||||||
|
with `--out` or `--out-dir` are never uploaded.
|
||||||
|
|
||||||
The upload token is read from the environment variable named by `token_env`
|
The upload token is read from the environment variable named by `token_env`
|
||||||
after config loading and `secrets.directory` processing. Config files should
|
after config loading and `secrets.directory` processing. Config files should
|
||||||
|
|||||||
@@ -32,7 +32,7 @@ Outputs:
|
|||||||
result details, generated-text artifact paths when applicable, and
|
result details, generated-text artifact paths when applicable, and
|
||||||
notification result when attempted
|
notification result when attempted
|
||||||
- batch summaries with per-report status, artifact paths, error text, and
|
- batch summaries with per-report status, artifact paths, error text, and
|
||||||
notification outcome when attempted
|
one top-level batch notification result when attempted or skipped
|
||||||
- saved Weather API bundle JSON for explicit bundle collection workflows
|
- saved Weather API bundle JSON for explicit bundle collection workflows
|
||||||
- inspection JSON values for reports, metadata, module snapshots, data
|
- inspection JSON values for reports, metadata, module snapshots, data
|
||||||
packages, prior snapshots, and source provenance
|
packages, prior snapshots, and source provenance
|
||||||
@@ -74,7 +74,8 @@ metadata stay in `internal/state`.
|
|||||||
- `workspace.*` for filesystem state
|
- `workspace.*` for filesystem state
|
||||||
- `dayparts` for daily and outlook summarization
|
- `dayparts` for daily and outlook summarization
|
||||||
- `recent_change.*` for structured Recent Changes thresholds
|
- `recent_change.*` for structured Recent Changes thresholds
|
||||||
- `notify.distributor.*` for optional notification after report generation
|
- `notify.distributor.*` for optional single-report and batch notification
|
||||||
|
after report generation
|
||||||
|
|
||||||
Output copy flags are command request fields. They are not configuration
|
Output copy flags are command request fields. They are not configuration
|
||||||
defaults.
|
defaults.
|
||||||
@@ -151,10 +152,26 @@ A collection failure stops the batch before planning or report generation.
|
|||||||
After planning succeeds, batch generation continues independent reports after a
|
After planning succeeds, batch generation continues independent reports after a
|
||||||
failure, records each result, writes compact status lines to stderr, emits a
|
failure, records each result, writes compact status lines to stderr, emits a
|
||||||
JSON summary to stdout, and returns an aggregate error when any report failed.
|
JSON summary to stdout, and returns an aggregate error when any report failed.
|
||||||
When notification is enabled, each successfully generated report is notified
|
|
||||||
independently. Notification failure marks that report failed, records
|
Batch report generation suppresses per-report distributor notification. After
|
||||||
notification fields in the batch result, and does not stop later reports.
|
all planned reports finish, app orchestration evaluates batch notification:
|
||||||
`--out-dir` copies are never used as notification source files.
|
|
||||||
|
1. If distributor notification is disabled, the batch notification result is
|
||||||
|
omitted.
|
||||||
|
2. If batch notification is disabled, the batch notification result is omitted
|
||||||
|
and there is no per-report fallback upload.
|
||||||
|
3. If any planned report failed, the batch notification result is `skipped`
|
||||||
|
with reason `one or more reports failed`, and distributor is not called.
|
||||||
|
4. If every report succeeded, app orchestration renders batch pipeline, bundle
|
||||||
|
ID, and idempotency key templates, renders `report_path_templates` for each
|
||||||
|
included report, validates every managed source path and bundle path, checks
|
||||||
|
duplicate bundle paths across the batch, calls the notifier once with a
|
||||||
|
multi-file request, and saves a batch notification debug artifact.
|
||||||
|
|
||||||
|
Batch notification failure records a top-level failed notification, increments
|
||||||
|
the aggregate batch failure count, and returns an aggregate batch error without
|
||||||
|
marking individual report items failed. `--out-dir` copies are never used as
|
||||||
|
notification source files.
|
||||||
|
|
||||||
## Inspection Workflow
|
## Inspection Workflow
|
||||||
|
|
||||||
@@ -172,8 +189,10 @@ inspection view.
|
|||||||
- Render and run errors preserve Scriptorium stderr and exit-code context.
|
- Render and run errors preserve Scriptorium stderr and exit-code context.
|
||||||
- Generated-text report errors preserve available intermediate artifacts and do
|
- Generated-text report errors preserve available intermediate artifacts and do
|
||||||
not create extra output copies.
|
not create extra output copies.
|
||||||
- Notification errors are wrapped with report ID, RunID, and managed report path
|
- Single-report notification errors are wrapped with report ID, RunID, and
|
||||||
context and are recorded separately in batch results.
|
managed report path context.
|
||||||
|
- Batch notification errors are recorded on the top-level batch notification
|
||||||
|
result and do not change individual report item status.
|
||||||
- Metadata and artifact path errors include filesystem context.
|
- Metadata and artifact path errors include filesystem context.
|
||||||
- Batch failures are recorded per report and surfaced through an aggregate
|
- Batch failures are recorded per report and surfaced through an aggregate
|
||||||
batch error.
|
batch error.
|
||||||
@@ -201,5 +220,9 @@ Inspect:
|
|||||||
a raw data package.
|
a raw data package.
|
||||||
- Recent Changes are computed from structured module snapshots.
|
- Recent Changes are computed from structured module snapshots.
|
||||||
- Metadata links artifacts produced for a run.
|
- Metadata links artifacts produced for a run.
|
||||||
- Distributor notification maps the managed Markdown report path to configured
|
- Single-report distributor notification maps the managed Markdown report path
|
||||||
bundle paths; extra output copies are not upload sources.
|
to configured bundle paths.
|
||||||
|
- Batch distributor notification maps each included managed Markdown report
|
||||||
|
path to bundle paths rendered for that report and uploads once for the
|
||||||
|
batch.
|
||||||
|
- Extra output copies are not upload sources.
|
||||||
|
|||||||
@@ -6,10 +6,10 @@ This document describes the distributor upload adapter in
|
|||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
The adapter submits generated weatherreporter Markdown reports to a configured
|
The adapter submits generated weatherreporter Markdown reports to a configured
|
||||||
distributor HTTP upload endpoint. It isolates distributor package types,
|
distributor HTTP upload endpoint. It supports one or more file mappings per
|
||||||
token-env lookup, upload client construction, source-bundle file mapping,
|
upload request. It isolates distributor package types, token-env lookup, upload
|
||||||
timeout handling, status polling, and upload error wrapping from app
|
client construction, source-bundle file mapping, timeout handling, status
|
||||||
orchestration.
|
polling, and upload error wrapping from app orchestration.
|
||||||
|
|
||||||
## Inputs And Outputs
|
## Inputs And Outputs
|
||||||
|
|
||||||
@@ -21,7 +21,7 @@ Inputs:
|
|||||||
- pipeline ID
|
- pipeline ID
|
||||||
- bundle ID
|
- bundle ID
|
||||||
- idempotency key
|
- idempotency key
|
||||||
- source Markdown report path and bundle-relative path mappings
|
- source Markdown report paths and bundle-relative path mappings
|
||||||
- bundle created timestamp
|
- bundle created timestamp
|
||||||
- context for cancellation
|
- context for cancellation
|
||||||
|
|
||||||
@@ -40,8 +40,9 @@ Outputs:
|
|||||||
|
|
||||||
The app layer passes weatherreporter-owned request values to the adapter. The
|
The app layer passes weatherreporter-owned request values to the adapter. The
|
||||||
adapter does not choose report types, render templates, select output copies,
|
adapter does not choose report types, render templates, select output copies,
|
||||||
configure destinations, wait for downstream publication, transform Markdown, or
|
decide whether an upload represents one report or a batch, configure
|
||||||
persist notification state.
|
destinations, wait for downstream publication, transform Markdown, or persist
|
||||||
|
notification state.
|
||||||
|
|
||||||
Full upstream distributor package and HTTP contract details stay under
|
Full upstream distributor package and HTTP contract details stay under
|
||||||
`docs/integrations/distributor/`.
|
`docs/integrations/distributor/`.
|
||||||
@@ -54,14 +55,19 @@ The adapter is built from `notify.distributor` config:
|
|||||||
- `token_env`
|
- `token_env`
|
||||||
- `timeout`
|
- `timeout`
|
||||||
|
|
||||||
The app layer renders pipeline ID, bundle ID, idempotency key, and bundle paths
|
The app layer renders single-report pipeline ID, bundle ID, idempotency key,
|
||||||
from:
|
and bundle paths from:
|
||||||
|
|
||||||
- `pipeline_id_template`
|
- `pipeline_id_template`
|
||||||
- `bundle_id_template`
|
- `bundle_id_template`
|
||||||
- `idempotency_key_template`
|
- `idempotency_key_template`
|
||||||
- `report_path_templates`
|
- `report_path_templates`
|
||||||
|
|
||||||
|
For batch uploads, the app layer renders pipeline ID, bundle ID, and
|
||||||
|
idempotency key from `notify.distributor.batch.*`, renders
|
||||||
|
`report_path_templates` once per included report, and passes the resulting
|
||||||
|
multi-file request to this adapter.
|
||||||
|
|
||||||
The token value is read from the environment variable named by `token_env`
|
The token value is read from the environment variable named by `token_env`
|
||||||
after config loading and `secrets.directory` processing.
|
after config loading and `secrets.directory` processing.
|
||||||
|
|
||||||
@@ -70,9 +76,9 @@ after config loading and `secrets.directory` processing.
|
|||||||
The adapter calls distributor `UploadFiles` with one or more file mappings:
|
The adapter calls distributor `UploadFiles` with one or more file mappings:
|
||||||
|
|
||||||
- pipeline ID: the rendered distributor workflow selector
|
- pipeline ID: the rendered distributor workflow selector
|
||||||
- source path: the managed Markdown report path selected by app orchestration
|
- source paths: managed Markdown report paths selected by app orchestration
|
||||||
- bundle paths: rendered bundle-relative report paths
|
- bundle paths: rendered bundle-relative report paths for each source
|
||||||
- created: the report generation timestamp
|
- created: the report or batch generation timestamp
|
||||||
|
|
||||||
The adapter creates a distributor upload client with the configured endpoint,
|
The adapter creates a distributor upload client with the configured endpoint,
|
||||||
bearer token, and timeout-backed HTTP client. It also wraps the upload context
|
bearer token, and timeout-backed HTTP client. It also wraps the upload context
|
||||||
@@ -115,7 +121,8 @@ distributor service.
|
|||||||
## Invariants
|
## Invariants
|
||||||
|
|
||||||
- Distributor package types do not leak outside the adapter.
|
- Distributor package types do not leak outside the adapter.
|
||||||
- Only the managed Markdown report is uploaded.
|
- Only managed Markdown report paths selected by app orchestration are
|
||||||
|
uploaded.
|
||||||
- The adapter never scans the workspace.
|
- The adapter never scans the workspace.
|
||||||
- Token values are not included in errors, CLI output, metadata, docs, or
|
- Token values are not included in errors, CLI output, metadata, docs, or
|
||||||
examples.
|
examples.
|
||||||
|
|||||||
@@ -31,6 +31,7 @@ Outputs:
|
|||||||
- render context JSON path
|
- render context JSON path
|
||||||
- managed Markdown report path
|
- managed Markdown report path
|
||||||
- metadata JSON path
|
- metadata JSON path
|
||||||
|
- distributor notification debug artifact paths
|
||||||
- prior comparable snapshot metadata
|
- prior comparable snapshot metadata
|
||||||
- loaded module snapshot, data package, generated text, generated-text run
|
- loaded module snapshot, data package, generated text, generated-text run
|
||||||
result, or render context
|
result, or render context
|
||||||
@@ -74,6 +75,7 @@ valid-period start date for dated artifacts, and the RunID.
|
|||||||
data-packages/<artifact_group>/<YYYY-MM-DD>/<run_id>.data_package.yaml
|
data-packages/<artifact_group>/<YYYY-MM-DD>/<run_id>.data_package.yaml
|
||||||
preflight/<artifact_group>/<YYYY-MM-DD>/<run_id>.render.json
|
preflight/<artifact_group>/<YYYY-MM-DD>/<run_id>.render.json
|
||||||
notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json
|
notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json
|
||||||
|
notifications/batches/<batch>/<YYYY-MM-DD>/<batch_run_id>.distributor.json
|
||||||
reports/<artifact_group>/<run_id>.md
|
reports/<artifact_group>/<run_id>.md
|
||||||
```
|
```
|
||||||
|
|
||||||
@@ -85,6 +87,10 @@ Scriptorium run result, validated generated text, and render context artifacts.
|
|||||||
Markdown-report metadata omits those generated-text fields. Report listing
|
Markdown-report metadata omits those generated-text fields. Report listing
|
||||||
walks metadata files under the snapshots directory.
|
walks metadata files under the snapshots directory.
|
||||||
|
|
||||||
|
Batch notification artifacts are stored under the notifications tree rather
|
||||||
|
than report metadata because they describe a batch-level upload. The date
|
||||||
|
directory is the batch start date in the effective report timezone.
|
||||||
|
|
||||||
## Prior Lookup
|
## Prior Lookup
|
||||||
|
|
||||||
Prior snapshot lookup reads stored metadata through the shared lookup path and
|
Prior snapshot lookup reads stored metadata through the shared lookup path and
|
||||||
@@ -118,6 +124,27 @@ atomically when notification is attempted and include rendered distributor
|
|||||||
pipeline ID, bundle ID, idempotency key, bundle paths, upload status, latest
|
pipeline ID, bundle ID, idempotency key, bundle paths, upload status, latest
|
||||||
run status, and redacted errors.
|
run status, and redacted errors.
|
||||||
|
|
||||||
|
Single-report notification artifacts use schema version
|
||||||
|
`weatherreporter.distributor_notification.v1` and record one managed source
|
||||||
|
path plus that source's bundle paths. Batch notification artifacts use schema
|
||||||
|
version `weatherreporter.batch_distributor_notification.v1` and record:
|
||||||
|
|
||||||
|
- `batch`
|
||||||
|
- `batchRunId`
|
||||||
|
- `attemptedAt`
|
||||||
|
- `endpoint`
|
||||||
|
- `pipelineId`
|
||||||
|
- `bundleId`
|
||||||
|
- `idempotencyKey`
|
||||||
|
- `bundleCreated`
|
||||||
|
- `includedReports`, each with `reportId`, `runId`, `sourcePath`, and
|
||||||
|
`bundlePaths`
|
||||||
|
- `status`
|
||||||
|
- `upload`
|
||||||
|
- `runStatus`
|
||||||
|
- `statusError`
|
||||||
|
- `error`
|
||||||
|
|
||||||
Inspection helpers read existing metadata, module snapshot, data package,
|
Inspection helpers read existing metadata, module snapshot, data package,
|
||||||
generated text, generated-text run result, and render context files. Missing
|
generated text, generated-text run result, and render context files. Missing
|
||||||
metadata directories return no inspection records or no prior snapshot rather
|
metadata directories return no inspection records or no prior snapshot rather
|
||||||
@@ -145,4 +172,6 @@ Inspect:
|
|||||||
- Metadata links artifacts produced for a run.
|
- Metadata links artifacts produced for a run.
|
||||||
- Generated-text artifacts live under the snapshots tree beside module
|
- Generated-text artifacts live under the snapshots tree beside module
|
||||||
snapshots and metadata.
|
snapshots and metadata.
|
||||||
|
- Batch notification artifacts live under `notifications/batches` and are not
|
||||||
|
linked from report metadata.
|
||||||
- Prior lookup is based on structured metadata, not rendered report text.
|
- Prior lookup is based on structured metadata, not rendered report text.
|
||||||
|
|||||||
@@ -57,12 +57,14 @@ batch before any report is generated.
|
|||||||
|
|
||||||
After planning succeeds, batch commands print a JSON summary to stdout, write
|
After planning succeeds, batch commands print a JSON summary to stdout, write
|
||||||
compact per-report status lines to stderr, continue independent reports after
|
compact per-report status lines to stderr, continue independent reports after
|
||||||
one report fails, and return nonzero when any report failed. When notification
|
one report fails, and return nonzero when any report failed. Batch commands do
|
||||||
is configured, the summary and status lines include notification status,
|
not upload each report independently. When distributor notification and batch
|
||||||
accepted distributor run ID, or notification error fields for each attempted
|
notification are enabled, weatherreporter uploads one distributor bundle only
|
||||||
report. `--out-dir PATH` writes extra Markdown copies using report default
|
after every planned report succeeds. If any report fails, the batch upload is
|
||||||
filenames such as `today.md` and `tomorrow.md`; dynamic Daily copies use
|
skipped for the whole batch. `--out-dir PATH` writes extra Markdown copies
|
||||||
`daily-YYYY-MM-DD.md`. These copies are not used as distributor upload sources.
|
using report default filenames such as `today.md` and `tomorrow.md`; dynamic
|
||||||
|
Daily copies use `daily-YYYY-MM-DD.md`. These copies are not used as
|
||||||
|
distributor upload sources.
|
||||||
|
|
||||||
## Filesystem Layout
|
## Filesystem Layout
|
||||||
|
|
||||||
@@ -160,6 +162,13 @@ workspace/
|
|||||||
YYYY-MM-DD/
|
YYYY-MM-DD/
|
||||||
<run_id>.render.json
|
<run_id>.render.json
|
||||||
notifications/
|
notifications/
|
||||||
|
batches/
|
||||||
|
morning/
|
||||||
|
YYYY-MM-DD/
|
||||||
|
<batch_run_id>.distributor.json
|
||||||
|
evening/
|
||||||
|
YYYY-MM-DD/
|
||||||
|
<batch_run_id>.distributor.json
|
||||||
daily/
|
daily/
|
||||||
YYYY-MM-DD/
|
YYYY-MM-DD/
|
||||||
<run_id>.distributor.json
|
<run_id>.distributor.json
|
||||||
@@ -206,14 +215,21 @@ period do not overwrite each other.
|
|||||||
RunIDs are based on generation time plus report ID. Reports that can be
|
RunIDs are based on generation time plus report ID. Reports that can be
|
||||||
generated more than once in a single command may append a report-specific
|
generated more than once in a single command may append a report-specific
|
||||||
disambiguator. Daily appends the local valid date so multiple dynamic Daily
|
disambiguator. Daily appends the local valid date so multiple dynamic Daily
|
||||||
reports in one batch have distinct managed artifacts and notification
|
reports in one batch have distinct managed artifacts:
|
||||||
idempotency keys:
|
|
||||||
|
|
||||||
```text
|
```text
|
||||||
20260529T100000.123456789Z_daily_2026-05-31
|
20260529T100000.123456789Z_daily_2026-05-31
|
||||||
20260529T100000.123456789Z_today
|
20260529T100000.123456789Z_today
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Batch notification RunIDs use the batch start timestamp plus the batch command
|
||||||
|
name:
|
||||||
|
|
||||||
|
```text
|
||||||
|
20260529T100000.123456789Z_morning
|
||||||
|
20260529T220000.123456789Z_evening
|
||||||
|
```
|
||||||
|
|
||||||
Each generated report writes metadata that links:
|
Each generated report writes metadata that links:
|
||||||
|
|
||||||
- RunID, report ID, variant, and prompt ID
|
- RunID, report ID, variant, and prompt ID
|
||||||
@@ -227,19 +243,33 @@ Each generated report writes metadata that links:
|
|||||||
generated-text-template reports
|
generated-text-template reports
|
||||||
- distributor notification debug artifact path, when notification is attempted
|
- distributor notification debug artifact path, when notification is attempted
|
||||||
|
|
||||||
Batch summaries include report status, error text when applicable, notification
|
Batch summaries include report status, error text when applicable, valid
|
||||||
outcome when attempted, valid period, and known artifact paths for each
|
period, and known artifact paths for each attempted report. Single-report
|
||||||
attempted report. Notification fields are `notificationStatus`,
|
notification fields on report items are empty for batch commands. When a batch
|
||||||
`notificationRunId`, and `notificationError`.
|
notification is attempted, skipped, or fails, the summary includes one
|
||||||
|
top-level `notification` object with fields such as `status`, `reason`,
|
||||||
|
`runId`, `pipelineId`, `bundleId`, `idempotencyKey`, `path`,
|
||||||
|
`includedReports`, and `error`.
|
||||||
|
|
||||||
## Distributor Notification
|
## Distributor Notification
|
||||||
|
|
||||||
Distributor notification is configured with `notify.distributor` and is
|
Distributor notification is configured with `notify.distributor` and is
|
||||||
disabled by default. When enabled, weatherreporter uploads the managed Markdown
|
disabled by default. For `generate <report>`, weatherreporter uploads the
|
||||||
report path recorded in the report result and metadata. That single source file
|
managed Markdown report path recorded in the report result and metadata. That
|
||||||
can be mapped to one or more configured bundle paths. By default, it is mapped
|
single source file can be mapped to one or more configured bundle paths. By
|
||||||
to one dated report path. Extra copies written by `--out` or `--out-dir` are
|
default, it is mapped to one dated report path. Extra copies written by `--out`
|
||||||
operator conveniences only.
|
or `--out-dir` are operator conveniences only.
|
||||||
|
|
||||||
|
For `run morning` and `run evening`, per-report notification is suppressed. If
|
||||||
|
`notify.distributor.enabled` and `notify.distributor.batch.enabled` are both
|
||||||
|
true, the batch uploads once after all reports finish successfully. The upload
|
||||||
|
contains one file mapping set per included report. Each mapping uses the
|
||||||
|
managed Markdown report as the source and renders `report_path_templates` using
|
||||||
|
that report's normal template values. All rendered bundle paths across the
|
||||||
|
batch must be unique. If any report fails, weatherreporter records a top-level
|
||||||
|
notification status of `skipped` with reason `one or more reports failed` and
|
||||||
|
does not call distributor. If batch notification is disabled, run commands do
|
||||||
|
not fall back to per-report uploads.
|
||||||
|
|
||||||
The rendered pipeline ID selects the configured distributor `http_upload`
|
The rendered pipeline ID selects the configured distributor `http_upload`
|
||||||
workflow. The default bundle ID is a stable logical source identity derived from
|
workflow. The default bundle ID is a stable logical source identity derived from
|
||||||
@@ -249,31 +279,40 @@ producer name, location ID, and report ID:
|
|||||||
weatherreporter.{location_id}.{report_id}
|
weatherreporter.{location_id}.{report_id}
|
||||||
```
|
```
|
||||||
|
|
||||||
The default idempotency key appends RunID to the rendered bundle ID so each
|
The default single-report idempotency key appends RunID to the rendered bundle
|
||||||
report generation has a distinct retry identity. The default bundle path uses
|
ID so each report generation has a distinct retry identity. The default bundle
|
||||||
the valid-period start date, artifact group, and RunID. Distributor owns
|
path uses the valid-period start date, artifact group, and RunID. Batch bundle
|
||||||
destination merge, retention, and derived snapshot behavior such as `latest`.
|
IDs default to `weatherreporter.{location_id}.{batch}`, and batch idempotency
|
||||||
For Daily, the default report ID and artifact group values are both `daily`,
|
keys default to `{bundle_id}.{batch_run_id}`. Distributor owns destination
|
||||||
and the default output filename value is `daily.md`.
|
merge, retention, and derived snapshot behavior such as `latest`. For Daily,
|
||||||
For Today, the default report ID and artifact group values are both `today`,
|
the default report ID and artifact group values are both `daily`, and the
|
||||||
and the batch output filename value is `today.md`.
|
default output filename value is `daily.md`. For Today, the default report ID
|
||||||
|
and artifact group values are both `today`, and the batch output filename value
|
||||||
|
is `today.md`.
|
||||||
|
|
||||||
Notification happens after final metadata save for generated reports.
|
Single-report notification happens after final metadata save for generated
|
||||||
Collection, module snapshot, data-package, render preflight, Scriptorium run,
|
reports. Batch notification happens after all planned reports have finished and
|
||||||
generated-text validation, template rendering, and metadata-save failures do
|
only when all report generations succeeded. Collection, module snapshot,
|
||||||
not trigger notification. A notification failure fails that report.
|
data-package, render preflight, Scriptorium run, generated-text validation,
|
||||||
In a batch, other reports continue, the failed report includes notification
|
template rendering, and metadata-save failures do not trigger notification. A
|
||||||
fields in the JSON summary, and the batch returns nonzero.
|
single-report notification failure fails that report. A batch notification
|
||||||
|
failure makes the batch return nonzero and increments the aggregate failure
|
||||||
|
count, but individual report items remain succeeded.
|
||||||
|
|
||||||
Each notification attempt writes a debug artifact under `notifications/`. The
|
Each notification attempt writes a debug artifact under `notifications/`.
|
||||||
artifact records the rendered pipeline ID, bundle ID, idempotency key, managed
|
Single-report artifacts live under
|
||||||
source path, bundle-relative paths, bundle created timestamp, accepted upload
|
`notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json`. Batch
|
||||||
response, and the latest distributor run status response when available.
|
artifacts live under
|
||||||
Weatherreporter polls status until distributor reports `succeeded` or `failed`,
|
`notifications/batches/<batch>/<YYYY-MM-DD>/<batch_run_id>.distributor.json`,
|
||||||
or until the configured notification timeout expires. The run status includes
|
where the date directory is the batch start date in the effective report
|
||||||
the distributor status, error text, and raw run report JSON, which can show
|
timezone. The artifact records the rendered pipeline ID, bundle ID,
|
||||||
actions such as `replace_older`, `skip_same`, `skip_destination_newer`, or
|
idempotency key, managed source paths, bundle-relative paths, bundle created
|
||||||
`failed`. Token values are not written.
|
timestamp, accepted upload response, and the latest distributor run status
|
||||||
|
response when available. Weatherreporter polls status until distributor reports
|
||||||
|
`succeeded` or `failed`, or until the configured notification timeout expires.
|
||||||
|
The run status includes the distributor status, error text, and raw run report
|
||||||
|
JSON, which can show actions such as `replace_older`, `skip_same`,
|
||||||
|
`skip_destination_newer`, or `failed`. Token values are not written.
|
||||||
|
|
||||||
Weatherreporter is responsible for selecting the managed Markdown report,
|
Weatherreporter is responsible for selecting the managed Markdown report,
|
||||||
constructing a source bundle, and submitting it to the configured distributor
|
constructing a source bundle, and submitting it to the configured distributor
|
||||||
@@ -333,10 +372,13 @@ A failed generation run may still leave useful artifacts:
|
|||||||
available intermediate artifacts, such as the structured run result, raw
|
available intermediate artifacts, such as the structured run result, raw
|
||||||
generated-text JSON, validated generated text, and render context. Metadata
|
generated-text JSON, validated generated text, and render context. Metadata
|
||||||
links those paths when it can be safely written.
|
links those paths when it can be safely written.
|
||||||
- If distributor notification fails, report artifacts and final metadata remain
|
- If single-report distributor notification fails, report artifacts and final
|
||||||
available, but the report or batch command returns nonzero.
|
metadata remain available, but the report command returns nonzero.
|
||||||
|
- If batch distributor notification fails, report artifacts and final metadata
|
||||||
|
remain available, the top-level batch notification links the debug artifact,
|
||||||
|
and the batch command returns nonzero.
|
||||||
- For batch commands, inspect the stdout JSON summary first, then inspect the
|
- For batch commands, inspect the stdout JSON summary first, then inspect the
|
||||||
artifact paths for each failed report.
|
artifact paths for each failed report or the top-level notification path.
|
||||||
|
|
||||||
For a bad report, start with:
|
For a bad report, start with:
|
||||||
|
|
||||||
|
|||||||
@@ -236,20 +236,107 @@ Relevant docs: [Report Templates](templates.md),
|
|||||||
Symptom: `run morning` or `run evening` returns nonzero.
|
Symptom: `run morning` or `run evening` returns nonzero.
|
||||||
|
|
||||||
Likely cause: weather collection failed before planning, or at least one
|
Likely cause: weather collection failed before planning, or at least one
|
||||||
planned report failed after planning succeeded.
|
planned report failed after planning succeeded, or every report succeeded but
|
||||||
|
the top-level batch distributor notification failed.
|
||||||
|
|
||||||
Diagnostic: if stdout contains a JSON summary, inspect each failed report item
|
Diagnostic: if stdout contains a JSON summary, inspect each failed report item
|
||||||
and stderr status line. If no summary was emitted, inspect the command error;
|
and the top-level `notification` object. Stderr includes one
|
||||||
configuration, Weather API collection, or batch validation failed before any
|
`batchNotification` line when batch notification is attempted, skipped, or
|
||||||
report artifacts were created.
|
fails. If no summary was emitted, inspect the command error; configuration,
|
||||||
|
Weather API collection, or batch validation failed before any report artifacts
|
||||||
|
were created.
|
||||||
|
|
||||||
Safe fix: for collection failures, fix the configuration or upstream Weather
|
Safe fix: for collection failures, fix the configuration or upstream Weather
|
||||||
API availability and rerun the batch. For report failures, use the failed
|
API availability and rerun the batch. For report failures, use the failed
|
||||||
report's artifact paths from the summary, then inspect metadata, sources,
|
report's artifact paths from the summary, then inspect metadata, sources,
|
||||||
module snapshot, and data package for that RunID.
|
module snapshot, and data package for that RunID. For a batch notification
|
||||||
|
failure, inspect the notification artifact path from the top-level
|
||||||
|
`notification.path`.
|
||||||
|
|
||||||
Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md).
|
Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md).
|
||||||
|
|
||||||
|
## Batch Upload Skipped
|
||||||
|
|
||||||
|
Symptom: a batch JSON summary contains
|
||||||
|
`"notification":{"status":"skipped","reason":"one or more reports failed"}`.
|
||||||
|
|
||||||
|
Likely cause: at least one planned report failed, so weatherreporter did not
|
||||||
|
call distributor for the batch.
|
||||||
|
|
||||||
|
Diagnostic: inspect the failed report items in the batch JSON summary and the
|
||||||
|
matching stderr report lines. A skipped batch notification has no distributor
|
||||||
|
run ID and no notification artifact path.
|
||||||
|
|
||||||
|
Safe fix: fix the report-generation failure first, then rerun the batch. The
|
||||||
|
batch upload is all-or-nothing.
|
||||||
|
|
||||||
|
Relevant docs: [Operations guide](operations.md).
|
||||||
|
|
||||||
|
## Batch Upload Fails
|
||||||
|
|
||||||
|
Symptom: every report item in a batch summary is succeeded, but the batch
|
||||||
|
returns nonzero and the top-level notification has `status: "failed"`.
|
||||||
|
|
||||||
|
Likely cause: the distributor upload was rejected, the distributor service was
|
||||||
|
unavailable, status polling reached a terminal distributor failure, or
|
||||||
|
weatherreporter rejected the batch file mapping before upload.
|
||||||
|
|
||||||
|
Diagnostic: inspect `notification.error`, `notification.pipelineId`,
|
||||||
|
`notification.bundleId`, `notification.idempotencyKey`, and
|
||||||
|
`notification.path` in stdout. Then inspect the notification artifact; it
|
||||||
|
records included report source paths, bundle paths, upload status, distributor
|
||||||
|
run status, status lookup error, and raw status report JSON when available.
|
||||||
|
|
||||||
|
Safe fix: fix the endpoint, token, distributor pipeline, batch identity
|
||||||
|
templates, or report path templates indicated by the error, then rerun the
|
||||||
|
batch. Individual report artifacts from the failed batch notification remain
|
||||||
|
available and do not need to be regenerated for diagnosis.
|
||||||
|
|
||||||
|
Relevant docs: [Configuration reference](config.md),
|
||||||
|
[Operations guide](operations.md).
|
||||||
|
|
||||||
|
## Duplicate Batch Bundle Path
|
||||||
|
|
||||||
|
Symptom: a batch returns nonzero with duplicate bundle path context before a
|
||||||
|
distributor run ID is accepted.
|
||||||
|
|
||||||
|
Likely cause: `notify.distributor.report_path_templates` rendered the same
|
||||||
|
bundle-relative path for two included reports in the same batch.
|
||||||
|
|
||||||
|
Diagnostic: inspect the error in stdout or stderr. The validation error
|
||||||
|
includes the duplicate bundle path plus the report IDs, RunIDs, and managed
|
||||||
|
source paths involved.
|
||||||
|
|
||||||
|
Safe fix: change `notify.distributor.report_path_templates` so every report in
|
||||||
|
a batch renders a unique path. Include values such as `{artifact_group}`,
|
||||||
|
`{valid_start_date}`, `{batch_output_name}`, or `{run_id}` when needed.
|
||||||
|
|
||||||
|
Relevant docs: [Configuration reference](config.md),
|
||||||
|
[Operations guide](operations.md).
|
||||||
|
|
||||||
|
## Distributor Source Conflict
|
||||||
|
|
||||||
|
Symptom: distributor accepts or rejects an upload with conflict context for a
|
||||||
|
source, destination, digest, or idempotency key.
|
||||||
|
|
||||||
|
Likely cause: the rendered bundle ID or idempotency key does not match the
|
||||||
|
intended producer identity. A bundle ID identifies the logical source stream;
|
||||||
|
an idempotency key identifies a retry of the same upload request.
|
||||||
|
|
||||||
|
Diagnostic: inspect the report notification artifact linked from metadata or
|
||||||
|
the batch notification artifact linked from the top-level notification path.
|
||||||
|
Compare the rendered pipeline ID, bundle ID, idempotency key, included source
|
||||||
|
paths, and bundle paths with `notify.distributor.*` templates and distributor
|
||||||
|
pipeline state.
|
||||||
|
|
||||||
|
Safe fix: keep bundle ID templates stable for the source stream that should be
|
||||||
|
updated, and keep idempotency keys stable only for retries of the same generated
|
||||||
|
content. Do not reuse one idempotency key for different report or batch
|
||||||
|
content.
|
||||||
|
|
||||||
|
Relevant docs: [Operations guide](operations.md),
|
||||||
|
[Distributor adapter internals](internal/distributor-adapter.md).
|
||||||
|
|
||||||
## Invalid Secrets Directory
|
## Invalid Secrets Directory
|
||||||
|
|
||||||
Symptom: config loading fails with `read secrets directory`, `secret file`, or
|
Symptom: config loading fails with `read secrets directory`, `secret file`, or
|
||||||
@@ -296,12 +383,14 @@ within the same distributor token and pipeline. By default the bundle ID is a
|
|||||||
stable report-stream identity and the idempotency key appends RunID.
|
stable report-stream identity and the idempotency key appends RunID.
|
||||||
|
|
||||||
Diagnostic: inspect the failed batch JSON or stderr line for pipeline, bundle,
|
Diagnostic: inspect the failed batch JSON or stderr line for pipeline, bundle,
|
||||||
and idempotency context. Compare the configured templates with the report RunID
|
and idempotency context. For batch commands, use the top-level notification
|
||||||
and report path.
|
object rather than per-report notification fields. Compare the configured
|
||||||
|
templates with the report RunID or batch RunID and report path.
|
||||||
|
|
||||||
Also inspect the notification artifact linked from metadata. It records the
|
Also inspect the notification artifact linked from metadata or from the
|
||||||
rendered pipeline ID, bundle ID, idempotency key, upload result, distributor run
|
top-level batch notification path. It records the rendered pipeline ID, bundle
|
||||||
status, status error, and raw run report JSON when available.
|
ID, idempotency key, upload result, distributor run status, status error, and
|
||||||
|
raw run report JSON when available.
|
||||||
|
|
||||||
Safe fix: keep idempotency templates stable for retries of the same generated
|
Safe fix: keep idempotency templates stable for retries of the same generated
|
||||||
report, but do not reuse the same rendered key for different generated report
|
report, but do not reuse the same rendered key for different generated report
|
||||||
@@ -319,7 +408,8 @@ Likely cause: the distributor endpoint rejected the token, pipeline ID, bundle
|
|||||||
ID, idempotency key, source file, or one of the rendered bundle paths.
|
ID, idempotency key, source file, or one of the rendered bundle paths.
|
||||||
|
|
||||||
Diagnostic: inspect stdout JSON or stderr status lines for
|
Diagnostic: inspect stdout JSON or stderr status lines for
|
||||||
`notificationError`. Confirm `notify.distributor.endpoint`,
|
`notificationError` or the top-level batch notification `error`. Confirm
|
||||||
|
`notify.distributor.endpoint`,
|
||||||
`notify.distributor.pipeline_id_template`,
|
`notify.distributor.pipeline_id_template`,
|
||||||
`notify.distributor.report_path_templates`, and token configuration. Token
|
`notify.distributor.report_path_templates`, and token configuration. Token
|
||||||
values are redacted from weatherreporter errors.
|
values are redacted from weatherreporter errors.
|
||||||
@@ -346,8 +436,8 @@ Likely cause: the configured distributor endpoint is unreachable, slow, or
|
|||||||
temporarily unavailable.
|
temporarily unavailable.
|
||||||
|
|
||||||
Diagnostic: check network access from the weatherreporter host to
|
Diagnostic: check network access from the weatherreporter host to
|
||||||
`notify.distributor.endpoint`. For batch runs, inspect which reports have
|
`notify.distributor.endpoint`. For batch runs, inspect the top-level
|
||||||
`notificationStatus: "failed"`.
|
notification object and the artifact linked by `notification.path`.
|
||||||
|
|
||||||
Safe fix: restore distributor service availability and rerun the affected
|
Safe fix: restore distributor service availability and rerun the affected
|
||||||
report or batch. Stable idempotency keys make retrying the same generated report
|
report or batch. Stable idempotency keys make retrying the same generated report
|
||||||
|
|||||||
Reference in New Issue
Block a user