Document batch distributor uploads

This commit is contained in:
2026-06-17 21:11:28 +00:00
parent 662db5e511
commit b3637cddd6
7 changed files with 300 additions and 97 deletions

View File

@@ -32,7 +32,7 @@ Outputs:
result details, generated-text artifact paths when applicable, and
notification result when attempted
- batch summaries with per-report status, artifact paths, error text, and
notification outcome when attempted
one top-level batch notification result when attempted or skipped
- saved Weather API bundle JSON for explicit bundle collection workflows
- inspection JSON values for reports, metadata, module snapshots, data
packages, prior snapshots, and source provenance
@@ -74,7 +74,8 @@ metadata stay in `internal/state`.
- `workspace.*` for filesystem state
- `dayparts` for daily and outlook summarization
- `recent_change.*` for structured Recent Changes thresholds
- `notify.distributor.*` for optional notification after report generation
- `notify.distributor.*` for optional single-report and batch notification
after report generation
Output copy flags are command request fields. They are not configuration
defaults.
@@ -151,10 +152,26 @@ A collection failure stops the batch before planning or report generation.
After planning succeeds, batch generation continues independent reports after a
failure, records each result, writes compact status lines to stderr, emits a
JSON summary to stdout, and returns an aggregate error when any report failed.
When notification is enabled, each successfully generated report is notified
independently. Notification failure marks that report failed, records
notification fields in the batch result, and does not stop later reports.
`--out-dir` copies are never used as notification source files.
Batch report generation suppresses per-report distributor notification. After
all planned reports finish, app orchestration evaluates batch notification:
1. If distributor notification is disabled, the batch notification result is
omitted.
2. If batch notification is disabled, the batch notification result is omitted
and there is no per-report fallback upload.
3. If any planned report failed, the batch notification result is `skipped`
with reason `one or more reports failed`, and distributor is not called.
4. If every report succeeded, app orchestration renders batch pipeline, bundle
ID, and idempotency key templates, renders `report_path_templates` for each
included report, validates every managed source path and bundle path, checks
duplicate bundle paths across the batch, calls the notifier once with a
multi-file request, and saves a batch notification debug artifact.
Batch notification failure records a top-level failed notification, increments
the aggregate batch failure count, and returns an aggregate batch error without
marking individual report items failed. `--out-dir` copies are never used as
notification source files.
## Inspection Workflow
@@ -172,8 +189,10 @@ inspection view.
- Render and run errors preserve Scriptorium stderr and exit-code context.
- Generated-text report errors preserve available intermediate artifacts and do
not create extra output copies.
- Notification errors are wrapped with report ID, RunID, and managed report path
context and are recorded separately in batch results.
- Single-report notification errors are wrapped with report ID, RunID, and
managed report path context.
- Batch notification errors are recorded on the top-level batch notification
result and do not change individual report item status.
- Metadata and artifact path errors include filesystem context.
- Batch failures are recorded per report and surfaced through an aggregate
batch error.
@@ -201,5 +220,9 @@ Inspect:
a raw data package.
- Recent Changes are computed from structured module snapshots.
- Metadata links artifacts produced for a run.
- Distributor notification maps the managed Markdown report path to configured
bundle paths; extra output copies are not upload sources.
- Single-report distributor notification maps the managed Markdown report path
to configured bundle paths.
- Batch distributor notification maps each included managed Markdown report
path to bundle paths rendered for that report and uploads once for the
batch.
- Extra output copies are not upload sources.

View File

@@ -6,10 +6,10 @@ This document describes the distributor upload adapter in
## Purpose
The adapter submits generated weatherreporter Markdown reports to a configured
distributor HTTP upload endpoint. It isolates distributor package types,
token-env lookup, upload client construction, source-bundle file mapping,
timeout handling, status polling, and upload error wrapping from app
orchestration.
distributor HTTP upload endpoint. It supports one or more file mappings per
upload request. It isolates distributor package types, token-env lookup, upload
client construction, source-bundle file mapping, timeout handling, status
polling, and upload error wrapping from app orchestration.
## Inputs And Outputs
@@ -21,7 +21,7 @@ Inputs:
- pipeline ID
- bundle ID
- idempotency key
- source Markdown report path and bundle-relative path mappings
- source Markdown report paths and bundle-relative path mappings
- bundle created timestamp
- context for cancellation
@@ -40,8 +40,9 @@ Outputs:
The app layer passes weatherreporter-owned request values to the adapter. The
adapter does not choose report types, render templates, select output copies,
configure destinations, wait for downstream publication, transform Markdown, or
persist notification state.
decide whether an upload represents one report or a batch, configure
destinations, wait for downstream publication, transform Markdown, or persist
notification state.
Full upstream distributor package and HTTP contract details stay under
`docs/integrations/distributor/`.
@@ -54,14 +55,19 @@ The adapter is built from `notify.distributor` config:
- `token_env`
- `timeout`
The app layer renders pipeline ID, bundle ID, idempotency key, and bundle paths
from:
The app layer renders single-report pipeline ID, bundle ID, idempotency key,
and bundle paths from:
- `pipeline_id_template`
- `bundle_id_template`
- `idempotency_key_template`
- `report_path_templates`
For batch uploads, the app layer renders pipeline ID, bundle ID, and
idempotency key from `notify.distributor.batch.*`, renders
`report_path_templates` once per included report, and passes the resulting
multi-file request to this adapter.
The token value is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing.
@@ -70,9 +76,9 @@ after config loading and `secrets.directory` processing.
The adapter calls distributor `UploadFiles` with one or more file mappings:
- pipeline ID: the rendered distributor workflow selector
- source path: the managed Markdown report path selected by app orchestration
- bundle paths: rendered bundle-relative report paths
- created: the report generation timestamp
- source paths: managed Markdown report paths selected by app orchestration
- bundle paths: rendered bundle-relative report paths for each source
- created: the report or batch generation timestamp
The adapter creates a distributor upload client with the configured endpoint,
bearer token, and timeout-backed HTTP client. It also wraps the upload context
@@ -115,7 +121,8 @@ distributor service.
## Invariants
- Distributor package types do not leak outside the adapter.
- Only the managed Markdown report is uploaded.
- Only managed Markdown report paths selected by app orchestration are
uploaded.
- The adapter never scans the workspace.
- Token values are not included in errors, CLI output, metadata, docs, or
examples.

View File

@@ -31,6 +31,7 @@ Outputs:
- render context JSON path
- managed Markdown report path
- metadata JSON path
- distributor notification debug artifact paths
- prior comparable snapshot metadata
- loaded module snapshot, data package, generated text, generated-text run
result, or render context
@@ -74,6 +75,7 @@ valid-period start date for dated artifacts, and the RunID.
data-packages/<artifact_group>/<YYYY-MM-DD>/<run_id>.data_package.yaml
preflight/<artifact_group>/<YYYY-MM-DD>/<run_id>.render.json
notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json
notifications/batches/<batch>/<YYYY-MM-DD>/<batch_run_id>.distributor.json
reports/<artifact_group>/<run_id>.md
```
@@ -85,6 +87,10 @@ Scriptorium run result, validated generated text, and render context artifacts.
Markdown-report metadata omits those generated-text fields. Report listing
walks metadata files under the snapshots directory.
Batch notification artifacts are stored under the notifications tree rather
than report metadata because they describe a batch-level upload. The date
directory is the batch start date in the effective report timezone.
## Prior Lookup
Prior snapshot lookup reads stored metadata through the shared lookup path and
@@ -118,6 +124,27 @@ atomically when notification is attempted and include rendered distributor
pipeline ID, bundle ID, idempotency key, bundle paths, upload status, latest
run status, and redacted errors.
Single-report notification artifacts use schema version
`weatherreporter.distributor_notification.v1` and record one managed source
path plus that source's bundle paths. Batch notification artifacts use schema
version `weatherreporter.batch_distributor_notification.v1` and record:
- `batch`
- `batchRunId`
- `attemptedAt`
- `endpoint`
- `pipelineId`
- `bundleId`
- `idempotencyKey`
- `bundleCreated`
- `includedReports`, each with `reportId`, `runId`, `sourcePath`, and
`bundlePaths`
- `status`
- `upload`
- `runStatus`
- `statusError`
- `error`
Inspection helpers read existing metadata, module snapshot, data package,
generated text, generated-text run result, and render context files. Missing
metadata directories return no inspection records or no prior snapshot rather
@@ -145,4 +172,6 @@ Inspect:
- Metadata links artifacts produced for a run.
- Generated-text artifacts live under the snapshots tree beside module
snapshots and metadata.
- Batch notification artifacts live under `notifications/batches` and are not
linked from report metadata.
- Prior lookup is based on structured metadata, not rendered report text.