10 Commits

21 changed files with 2205 additions and 243 deletions

View File

@@ -60,11 +60,13 @@ stops the batch before any report is generated. Batch runs continue independent
reports after a later report failure, print a JSON summary to stdout, write
compact status lines to stderr, and return nonzero when any report failed.
`--out-dir` writes extra Markdown copies for the operator; distributor
notification uses each managed report path, not the extra copies. Today and
notification uses managed report paths, not the extra copies. Today and
Tomorrow use their report default copy names, and dynamic Daily copies use
`daily-YYYY-MM-DD.md`. When notification is enabled, batch summaries and status
lines include notification status, accepted distributor run ID, or notification
error fields for each attempted report.
`daily-YYYY-MM-DD.md`. When distributor and batch notification are enabled, a
fully successful batch uploads one distributor bundle after report generation
finishes. The JSON summary exposes that upload as a top-level `notification`
object, and stderr includes one `batchNotification` status line. If any planned
report fails, the batch notification is skipped for the whole batch.
Hourly Report, 3-Day Outlook, and Weekend Outlook are explicit only; they are
not included in `run morning` or `run evening`.

View File

@@ -78,10 +78,13 @@ missing directories, and unreadable files fail config loading.
### `notify`
`notify.distributor` controls distributor notification after successful report
generation. It is disabled by default and does not add CLI flags. When enabled,
weatherreporter uploads one distributor bundle per generated report after
report rendering succeeds and final metadata is saved.
`notify.distributor` controls distributor uploads after successful report
rendering. It is disabled by default and does not add CLI flags. When enabled,
`generate <report>` uploads one distributor bundle for the generated report
after final metadata is saved. `run morning` and `run evening` use
`notify.distributor.batch`: when batch notification is enabled and every
planned report succeeds, weatherreporter uploads one distributor bundle that
contains all managed Markdown reports from that batch.
- `enabled`: whether distributor notification config is active. Default:
`false`.
@@ -92,20 +95,34 @@ report rendering succeeds and final metadata is saved.
- `timeout`: distributor operation timeout. Must be greater than zero when
enabled. Default: `30s`.
- `failure_policy`: must be `error` when enabled. Default: `error`.
- `pipeline_id_template`: template for the distributor pipeline ID. Required
when enabled. Default: empty.
- `bundle_id_template`: template for distributor bundle IDs. Default:
`weatherreporter.{location_id}.{report_id}`.
- `idempotency_key_template`: template for distributor idempotency keys.
Default: `{bundle_id}.{run_id}`.
- `pipeline_id_template`: template for single-report distributor pipeline IDs.
Required when enabled. Default: empty.
- `bundle_id_template`: template for single-report distributor bundle IDs.
Default: `weatherreporter.{location_id}.{report_id}`.
- `idempotency_key_template`: template for single-report distributor
idempotency keys. Default: `{bundle_id}.{run_id}`.
- `report_path_templates`: ordered list of templates for Markdown report paths
inside the distributor bundle. Each rendered path maps to the same managed
Markdown report source. Default:
inside distributor bundles. For `generate <report>`, each rendered path maps
to the generated report's managed Markdown source. For `run morning` and
`run evening`, the templates are rendered once per included report and each
rendered path maps that report's managed Markdown source into the one batch
bundle. Default:
```yaml
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
```
- `batch.enabled`: whether batch distributor notification config is active
when distributor notification is enabled. Default: `true`.
- `batch.pipeline_id_template`: template for batch distributor pipeline IDs.
Required when distributor notification and batch notification are enabled.
Default: `weatherreporter`.
- `batch.bundle_id_template`: template for batch distributor bundle IDs.
Required when distributor notification and batch notification are enabled.
Default: `weatherreporter.{location_id}.{batch}`.
- `batch.idempotency_key_template`: template for batch distributor idempotency
keys. Required when distributor notification and batch notification are
enabled. Default: `{bundle_id}.{batch_run_id}`.
Supported template variables are `location_id`, `report_id`, `run_id`,
Single-report templates support `location_id`, `report_id`, `run_id`,
`artifact_group`, `batch_output_name`, `valid_start_date`, `valid_end_date`,
`valid_start_time`, `valid_end_time`, `valid_start_stamp`, and
`valid_end_stamp`. Date values use `YYYY-MM-DD`, time values use `HHMM`, and
@@ -116,9 +133,18 @@ The rendered pipeline ID selects the configured distributor `http_upload`
workflow. The rendered bundle ID is the stable logical source identity for the
report stream. The rendered idempotency key is the per-run retry identity.
Batch templates support `location_id`, `batch`, `batch_run_id`, and
`batch_started_date`. Batch idempotency templates may also use `bundle_id`.
`batch_started_date` is the batch start date in the effective report timezone.
Batch bundle IDs identify a logical batch stream; batch idempotency keys
identify a specific retryable batch attempt.
Rendered report paths must be unique relative paths with `/` separators. They
must not contain backslashes, empty path segments, `.`, `..`, `manifest.json`,
or `.distributor.json`.
or `.distributor.json`. In a batch upload, uniqueness is checked across every
rendered bundle path for every included report before distributor is called.
Managed Markdown report paths are the only upload source files; copies written
with `--out` or `--out-dir` are never uploaded.
The upload token is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing. Config files should

View File

@@ -32,7 +32,7 @@ Outputs:
result details, generated-text artifact paths when applicable, and
notification result when attempted
- batch summaries with per-report status, artifact paths, error text, and
notification outcome when attempted
one top-level batch notification result when attempted or skipped
- saved Weather API bundle JSON for explicit bundle collection workflows
- inspection JSON values for reports, metadata, module snapshots, data
packages, prior snapshots, and source provenance
@@ -74,7 +74,8 @@ metadata stay in `internal/state`.
- `workspace.*` for filesystem state
- `dayparts` for daily and outlook summarization
- `recent_change.*` for structured Recent Changes thresholds
- `notify.distributor.*` for optional notification after report generation
- `notify.distributor.*` for optional single-report and batch notification
after report generation
Output copy flags are command request fields. They are not configuration
defaults.
@@ -151,10 +152,26 @@ A collection failure stops the batch before planning or report generation.
After planning succeeds, batch generation continues independent reports after a
failure, records each result, writes compact status lines to stderr, emits a
JSON summary to stdout, and returns an aggregate error when any report failed.
When notification is enabled, each successfully generated report is notified
independently. Notification failure marks that report failed, records
notification fields in the batch result, and does not stop later reports.
`--out-dir` copies are never used as notification source files.
Batch report generation suppresses per-report distributor notification. After
all planned reports finish, app orchestration evaluates batch notification:
1. If distributor notification is disabled, the batch notification result is
omitted.
2. If batch notification is disabled, the batch notification result is omitted
and there is no per-report fallback upload.
3. If any planned report failed, the batch notification result is `skipped`
with reason `one or more reports failed`, and distributor is not called.
4. If every report succeeded, app orchestration renders batch pipeline, bundle
ID, and idempotency key templates, renders `report_path_templates` for each
included report, validates every managed source path and bundle path, checks
duplicate bundle paths across the batch, calls the notifier once with a
multi-file request, and saves a batch notification debug artifact.
Batch notification failure records a top-level failed notification, increments
the aggregate batch failure count, and returns an aggregate batch error without
marking individual report items failed. `--out-dir` copies are never used as
notification source files.
## Inspection Workflow
@@ -172,8 +189,10 @@ inspection view.
- Render and run errors preserve Scriptorium stderr and exit-code context.
- Generated-text report errors preserve available intermediate artifacts and do
not create extra output copies.
- Notification errors are wrapped with report ID, RunID, and managed report path
context and are recorded separately in batch results.
- Single-report notification errors are wrapped with report ID, RunID, and
managed report path context.
- Batch notification errors are recorded on the top-level batch notification
result and do not change individual report item status.
- Metadata and artifact path errors include filesystem context.
- Batch failures are recorded per report and surfaced through an aggregate
batch error.
@@ -201,5 +220,9 @@ Inspect:
a raw data package.
- Recent Changes are computed from structured module snapshots.
- Metadata links artifacts produced for a run.
- Distributor notification maps the managed Markdown report path to configured
bundle paths; extra output copies are not upload sources.
- Single-report distributor notification maps the managed Markdown report path
to configured bundle paths.
- Batch distributor notification maps each included managed Markdown report
path to bundle paths rendered for that report and uploads once for the
batch.
- Extra output copies are not upload sources.

View File

@@ -6,10 +6,10 @@ This document describes the distributor upload adapter in
## Purpose
The adapter submits generated weatherreporter Markdown reports to a configured
distributor HTTP upload endpoint. It isolates distributor package types,
token-env lookup, upload client construction, source-bundle file mapping,
timeout handling, status polling, and upload error wrapping from app
orchestration.
distributor HTTP upload endpoint. It supports one or more file mappings per
upload request. It isolates distributor package types, token-env lookup, upload
client construction, source-bundle file mapping, timeout handling, status
polling, and upload error wrapping from app orchestration.
## Inputs And Outputs
@@ -21,7 +21,7 @@ Inputs:
- pipeline ID
- bundle ID
- idempotency key
- source Markdown report path and bundle-relative path mappings
- source Markdown report paths and bundle-relative path mappings
- bundle created timestamp
- context for cancellation
@@ -40,8 +40,9 @@ Outputs:
The app layer passes weatherreporter-owned request values to the adapter. The
adapter does not choose report types, render templates, select output copies,
configure destinations, wait for downstream publication, transform Markdown, or
persist notification state.
decide whether an upload represents one report or a batch, configure
destinations, wait for downstream publication, transform Markdown, or persist
notification state.
Full upstream distributor package and HTTP contract details stay under
`docs/integrations/distributor/`.
@@ -54,14 +55,19 @@ The adapter is built from `notify.distributor` config:
- `token_env`
- `timeout`
The app layer renders pipeline ID, bundle ID, idempotency key, and bundle paths
from:
The app layer renders single-report pipeline ID, bundle ID, idempotency key,
and bundle paths from:
- `pipeline_id_template`
- `bundle_id_template`
- `idempotency_key_template`
- `report_path_templates`
For batch uploads, the app layer renders pipeline ID, bundle ID, and
idempotency key from `notify.distributor.batch.*`, renders
`report_path_templates` once per included report, and passes the resulting
multi-file request to this adapter.
The token value is read from the environment variable named by `token_env`
after config loading and `secrets.directory` processing.
@@ -70,9 +76,9 @@ after config loading and `secrets.directory` processing.
The adapter calls distributor `UploadFiles` with one or more file mappings:
- pipeline ID: the rendered distributor workflow selector
- source path: the managed Markdown report path selected by app orchestration
- bundle paths: rendered bundle-relative report paths
- created: the report generation timestamp
- source paths: managed Markdown report paths selected by app orchestration
- bundle paths: rendered bundle-relative report paths for each source
- created: the report or batch generation timestamp
The adapter creates a distributor upload client with the configured endpoint,
bearer token, and timeout-backed HTTP client. It also wraps the upload context
@@ -115,7 +121,8 @@ distributor service.
## Invariants
- Distributor package types do not leak outside the adapter.
- Only the managed Markdown report is uploaded.
- Only managed Markdown report paths selected by app orchestration are
uploaded.
- The adapter never scans the workspace.
- Token values are not included in errors, CLI output, metadata, docs, or
examples.

View File

@@ -31,6 +31,7 @@ Outputs:
- render context JSON path
- managed Markdown report path
- metadata JSON path
- distributor notification debug artifact paths
- prior comparable snapshot metadata
- loaded module snapshot, data package, generated text, generated-text run
result, or render context
@@ -74,6 +75,7 @@ valid-period start date for dated artifacts, and the RunID.
data-packages/<artifact_group>/<YYYY-MM-DD>/<run_id>.data_package.yaml
preflight/<artifact_group>/<YYYY-MM-DD>/<run_id>.render.json
notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json
notifications/batches/<batch>/<YYYY-MM-DD>/<batch_run_id>.distributor.json
reports/<artifact_group>/<run_id>.md
```
@@ -85,6 +87,10 @@ Scriptorium run result, validated generated text, and render context artifacts.
Markdown-report metadata omits those generated-text fields. Report listing
walks metadata files under the snapshots directory.
Batch notification artifacts are stored under the notifications tree rather
than report metadata because they describe a batch-level upload. The date
directory is the batch start date in the effective report timezone.
## Prior Lookup
Prior snapshot lookup reads stored metadata through the shared lookup path and
@@ -118,6 +124,27 @@ atomically when notification is attempted and include rendered distributor
pipeline ID, bundle ID, idempotency key, bundle paths, upload status, latest
run status, and redacted errors.
Single-report notification artifacts use schema version
`weatherreporter.distributor_notification.v1` and record one managed source
path plus that source's bundle paths. Batch notification artifacts use schema
version `weatherreporter.batch_distributor_notification.v1` and record:
- `batch`
- `batchRunId`
- `attemptedAt`
- `endpoint`
- `pipelineId`
- `bundleId`
- `idempotencyKey`
- `bundleCreated`
- `includedReports`, each with `reportId`, `runId`, `sourcePath`, and
`bundlePaths`
- `status`
- `upload`
- `runStatus`
- `statusError`
- `error`
Inspection helpers read existing metadata, module snapshot, data package,
generated text, generated-text run result, and render context files. Missing
metadata directories return no inspection records or no prior snapshot rather
@@ -145,4 +172,6 @@ Inspect:
- Metadata links artifacts produced for a run.
- Generated-text artifacts live under the snapshots tree beside module
snapshots and metadata.
- Batch notification artifacts live under `notifications/batches` and are not
linked from report metadata.
- Prior lookup is based on structured metadata, not rendered report text.

View File

@@ -57,12 +57,14 @@ batch before any report is generated.
After planning succeeds, batch commands print a JSON summary to stdout, write
compact per-report status lines to stderr, continue independent reports after
one report fails, and return nonzero when any report failed. When notification
is configured, the summary and status lines include notification status,
accepted distributor run ID, or notification error fields for each attempted
report. `--out-dir PATH` writes extra Markdown copies using report default
filenames such as `today.md` and `tomorrow.md`; dynamic Daily copies use
`daily-YYYY-MM-DD.md`. These copies are not used as distributor upload sources.
one report fails, and return nonzero when any report failed. Batch commands do
not upload each report independently. When distributor notification and batch
notification are enabled, weatherreporter uploads one distributor bundle only
after every planned report succeeds. If any report fails, the batch upload is
skipped for the whole batch. `--out-dir PATH` writes extra Markdown copies
using report default filenames such as `today.md` and `tomorrow.md`; dynamic
Daily copies use `daily-YYYY-MM-DD.md`. These copies are not used as
distributor upload sources.
## Filesystem Layout
@@ -160,6 +162,13 @@ workspace/
YYYY-MM-DD/
<run_id>.render.json
notifications/
batches/
morning/
YYYY-MM-DD/
<batch_run_id>.distributor.json
evening/
YYYY-MM-DD/
<batch_run_id>.distributor.json
daily/
YYYY-MM-DD/
<run_id>.distributor.json
@@ -206,14 +215,21 @@ period do not overwrite each other.
RunIDs are based on generation time plus report ID. Reports that can be
generated more than once in a single command may append a report-specific
disambiguator. Daily appends the local valid date so multiple dynamic Daily
reports in one batch have distinct managed artifacts and notification
idempotency keys:
reports in one batch have distinct managed artifacts:
```text
20260529T100000.123456789Z_daily_2026-05-31
20260529T100000.123456789Z_today
```
Batch notification RunIDs use the batch start timestamp plus the batch command
name:
```text
20260529T100000.123456789Z_morning
20260529T220000.123456789Z_evening
```
Each generated report writes metadata that links:
- RunID, report ID, variant, and prompt ID
@@ -227,19 +243,33 @@ Each generated report writes metadata that links:
generated-text-template reports
- distributor notification debug artifact path, when notification is attempted
Batch summaries include report status, error text when applicable, notification
outcome when attempted, valid period, and known artifact paths for each
attempted report. Notification fields are `notificationStatus`,
`notificationRunId`, and `notificationError`.
Batch summaries include report status, error text when applicable, valid
period, and known artifact paths for each attempted report. Single-report
notification fields on report items are empty for batch commands. When a batch
notification is attempted, skipped, or fails, the summary includes one
top-level `notification` object with fields such as `status`, `reason`,
`runId`, `pipelineId`, `bundleId`, `idempotencyKey`, `path`,
`includedReports`, and `error`.
## Distributor Notification
Distributor notification is configured with `notify.distributor` and is
disabled by default. When enabled, weatherreporter uploads the managed Markdown
report path recorded in the report result and metadata. That single source file
can be mapped to one or more configured bundle paths. By default, it is mapped
to one dated report path. Extra copies written by `--out` or `--out-dir` are
operator conveniences only.
disabled by default. For `generate <report>`, weatherreporter uploads the
managed Markdown report path recorded in the report result and metadata. That
single source file can be mapped to one or more configured bundle paths. By
default, it is mapped to one dated report path. Extra copies written by `--out`
or `--out-dir` are operator conveniences only.
For `run morning` and `run evening`, per-report notification is suppressed. If
`notify.distributor.enabled` and `notify.distributor.batch.enabled` are both
true, the batch uploads once after all reports finish successfully. The upload
contains one file mapping set per included report. Each mapping uses the
managed Markdown report as the source and renders `report_path_templates` using
that report's normal template values. All rendered bundle paths across the
batch must be unique. If any report fails, weatherreporter records a top-level
notification status of `skipped` with reason `one or more reports failed` and
does not call distributor. If batch notification is disabled, run commands do
not fall back to per-report uploads.
The rendered pipeline ID selects the configured distributor `http_upload`
workflow. The default bundle ID is a stable logical source identity derived from
@@ -249,31 +279,40 @@ producer name, location ID, and report ID:
weatherreporter.{location_id}.{report_id}
```
The default idempotency key appends RunID to the rendered bundle ID so each
report generation has a distinct retry identity. The default bundle path uses
the valid-period start date, artifact group, and RunID. Distributor owns
destination merge, retention, and derived snapshot behavior such as `latest`.
For Daily, the default report ID and artifact group values are both `daily`,
and the default output filename value is `daily.md`.
For Today, the default report ID and artifact group values are both `today`,
and the batch output filename value is `today.md`.
The default single-report idempotency key appends RunID to the rendered bundle
ID so each report generation has a distinct retry identity. The default bundle
path uses the valid-period start date, artifact group, and RunID. Batch bundle
IDs default to `weatherreporter.{location_id}.{batch}`, and batch idempotency
keys default to `{bundle_id}.{batch_run_id}`. Distributor owns destination
merge, retention, and derived snapshot behavior such as `latest`. For Daily,
the default report ID and artifact group values are both `daily`, and the
default output filename value is `daily.md`. For Today, the default report ID
and artifact group values are both `today`, and the batch output filename value
is `today.md`.
Notification happens after final metadata save for generated reports.
Collection, module snapshot, data-package, render preflight, Scriptorium run,
generated-text validation, template rendering, and metadata-save failures do
not trigger notification. A notification failure fails that report.
In a batch, other reports continue, the failed report includes notification
fields in the JSON summary, and the batch returns nonzero.
Single-report notification happens after final metadata save for generated
reports. Batch notification happens after all planned reports have finished and
only when all report generations succeeded. Collection, module snapshot,
data-package, render preflight, Scriptorium run, generated-text validation,
template rendering, and metadata-save failures do not trigger notification. A
single-report notification failure fails that report. A batch notification
failure makes the batch return nonzero and increments the aggregate failure
count, but individual report items remain succeeded.
Each notification attempt writes a debug artifact under `notifications/`. The
artifact records the rendered pipeline ID, bundle ID, idempotency key, managed
source path, bundle-relative paths, bundle created timestamp, accepted upload
response, and the latest distributor run status response when available.
Weatherreporter polls status until distributor reports `succeeded` or `failed`,
or until the configured notification timeout expires. The run status includes
the distributor status, error text, and raw run report JSON, which can show
actions such as `replace_older`, `skip_same`, `skip_destination_newer`, or
`failed`. Token values are not written.
Each notification attempt writes a debug artifact under `notifications/`.
Single-report artifacts live under
`notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json`. Batch
artifacts live under
`notifications/batches/<batch>/<YYYY-MM-DD>/<batch_run_id>.distributor.json`,
where the date directory is the batch start date in the effective report
timezone. The artifact records the rendered pipeline ID, bundle ID,
idempotency key, managed source paths, bundle-relative paths, bundle created
timestamp, accepted upload response, and the latest distributor run status
response when available. Weatherreporter polls status until distributor reports
`succeeded` or `failed`, or until the configured notification timeout expires.
The run status includes the distributor status, error text, and raw run report
JSON, which can show actions such as `replace_older`, `skip_same`,
`skip_destination_newer`, or `failed`. Token values are not written.
Weatherreporter is responsible for selecting the managed Markdown report,
constructing a source bundle, and submitting it to the configured distributor
@@ -333,10 +372,13 @@ A failed generation run may still leave useful artifacts:
available intermediate artifacts, such as the structured run result, raw
generated-text JSON, validated generated text, and render context. Metadata
links those paths when it can be safely written.
- If distributor notification fails, report artifacts and final metadata remain
available, but the report or batch command returns nonzero.
- If single-report distributor notification fails, report artifacts and final
metadata remain available, but the report command returns nonzero.
- If batch distributor notification fails, report artifacts and final metadata
remain available, the top-level batch notification links the debug artifact,
and the batch command returns nonzero.
- For batch commands, inspect the stdout JSON summary first, then inspect the
artifact paths for each failed report.
artifact paths for each failed report or the top-level notification path.
For a bad report, start with:

View File

@@ -236,20 +236,107 @@ Relevant docs: [Report Templates](templates.md),
Symptom: `run morning` or `run evening` returns nonzero.
Likely cause: weather collection failed before planning, or at least one
planned report failed after planning succeeded.
planned report failed after planning succeeded, or every report succeeded but
the top-level batch distributor notification failed.
Diagnostic: if stdout contains a JSON summary, inspect each failed report item
and stderr status line. If no summary was emitted, inspect the command error;
configuration, Weather API collection, or batch validation failed before any
report artifacts were created.
and the top-level `notification` object. Stderr includes one
`batchNotification` line when batch notification is attempted, skipped, or
fails. If no summary was emitted, inspect the command error; configuration,
Weather API collection, or batch validation failed before any report artifacts
were created.
Safe fix: for collection failures, fix the configuration or upstream Weather
API availability and rerun the batch. For report failures, use the failed
report's artifact paths from the summary, then inspect metadata, sources,
module snapshot, and data package for that RunID.
module snapshot, and data package for that RunID. For a batch notification
failure, inspect the notification artifact path from the top-level
`notification.path`.
Relevant docs: [CLI reference](cli.md), [Operations guide](operations.md).
## Batch Upload Skipped
Symptom: a batch JSON summary contains
`"notification":{"status":"skipped","reason":"one or more reports failed"}`.
Likely cause: at least one planned report failed, so weatherreporter did not
call distributor for the batch.
Diagnostic: inspect the failed report items in the batch JSON summary and the
matching stderr report lines. A skipped batch notification has no distributor
run ID and no notification artifact path.
Safe fix: fix the report-generation failure first, then rerun the batch. The
batch upload is all-or-nothing.
Relevant docs: [Operations guide](operations.md).
## Batch Upload Fails
Symptom: every report item in a batch summary is succeeded, but the batch
returns nonzero and the top-level notification has `status: "failed"`.
Likely cause: the distributor upload was rejected, the distributor service was
unavailable, status polling reached a terminal distributor failure, or
weatherreporter rejected the batch file mapping before upload.
Diagnostic: inspect `notification.error`, `notification.pipelineId`,
`notification.bundleId`, `notification.idempotencyKey`, and
`notification.path` in stdout. Then inspect the notification artifact; it
records included report source paths, bundle paths, upload status, distributor
run status, status lookup error, and raw status report JSON when available.
Safe fix: fix the endpoint, token, distributor pipeline, batch identity
templates, or report path templates indicated by the error, then rerun the
batch. Individual report artifacts from the failed batch notification remain
available and do not need to be regenerated for diagnosis.
Relevant docs: [Configuration reference](config.md),
[Operations guide](operations.md).
## Duplicate Batch Bundle Path
Symptom: a batch returns nonzero with duplicate bundle path context before a
distributor run ID is accepted.
Likely cause: `notify.distributor.report_path_templates` rendered the same
bundle-relative path for two included reports in the same batch.
Diagnostic: inspect the error in stdout or stderr. The validation error
includes the duplicate bundle path plus the report IDs, RunIDs, and managed
source paths involved.
Safe fix: change `notify.distributor.report_path_templates` so every report in
a batch renders a unique path. Include values such as `{artifact_group}`,
`{valid_start_date}`, `{batch_output_name}`, or `{run_id}` when needed.
Relevant docs: [Configuration reference](config.md),
[Operations guide](operations.md).
## Distributor Source Conflict
Symptom: distributor accepts or rejects an upload with conflict context for a
source, destination, digest, or idempotency key.
Likely cause: the rendered bundle ID or idempotency key does not match the
intended producer identity. A bundle ID identifies the logical source stream;
an idempotency key identifies a retry of the same upload request.
Diagnostic: inspect the report notification artifact linked from metadata or
the batch notification artifact linked from the top-level notification path.
Compare the rendered pipeline ID, bundle ID, idempotency key, included source
paths, and bundle paths with `notify.distributor.*` templates and distributor
pipeline state.
Safe fix: keep bundle ID templates stable for the source stream that should be
updated, and keep idempotency keys stable only for retries of the same generated
content. Do not reuse one idempotency key for different report or batch
content.
Relevant docs: [Operations guide](operations.md),
[Distributor adapter internals](internal/distributor-adapter.md).
## Invalid Secrets Directory
Symptom: config loading fails with `read secrets directory`, `secret file`, or
@@ -296,12 +383,14 @@ within the same distributor token and pipeline. By default the bundle ID is a
stable report-stream identity and the idempotency key appends RunID.
Diagnostic: inspect the failed batch JSON or stderr line for pipeline, bundle,
and idempotency context. Compare the configured templates with the report RunID
and report path.
and idempotency context. For batch commands, use the top-level notification
object rather than per-report notification fields. Compare the configured
templates with the report RunID or batch RunID and report path.
Also inspect the notification artifact linked from metadata. It records the
rendered pipeline ID, bundle ID, idempotency key, upload result, distributor run
status, status error, and raw run report JSON when available.
Also inspect the notification artifact linked from metadata or from the
top-level batch notification path. It records the rendered pipeline ID, bundle
ID, idempotency key, upload result, distributor run status, status error, and
raw run report JSON when available.
Safe fix: keep idempotency templates stable for retries of the same generated
report, but do not reuse the same rendered key for different generated report
@@ -319,7 +408,8 @@ Likely cause: the distributor endpoint rejected the token, pipeline ID, bundle
ID, idempotency key, source file, or one of the rendered bundle paths.
Diagnostic: inspect stdout JSON or stderr status lines for
`notificationError`. Confirm `notify.distributor.endpoint`,
`notificationError` or the top-level batch notification `error`. Confirm
`notify.distributor.endpoint`,
`notify.distributor.pipeline_id_template`,
`notify.distributor.report_path_templates`, and token configuration. Token
values are redacted from weatherreporter errors.
@@ -346,8 +436,8 @@ Likely cause: the configured distributor endpoint is unreachable, slow, or
temporarily unavailable.
Diagnostic: check network access from the weatherreporter host to
`notify.distributor.endpoint`. For batch runs, inspect which reports have
`notificationStatus: "failed"`.
`notify.distributor.endpoint`. For batch runs, inspect the top-level
notification object and the artifact linked by `notification.path`.
Safe fix: restore distributor service availability and rerun the affected
report or batch. Stable idempotency keys make retrying the same generated report

View File

@@ -1,5 +1,5 @@
weather_api:
base_url: https://weather.api.rakestrawhome.com/
base_url: https://weather.api.example.com/
timeout: 15s
precision: 1
units: us
@@ -26,6 +26,11 @@ notify:
idempotency_key_template: "{bundle_id}.{run_id}"
report_path_templates:
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
batch:
enabled: true
pipeline_id_template: "weatherreporter"
bundle_id_template: "weatherreporter.{location_id}.{batch}"
idempotency_key_template: "{bundle_id}.{batch_run_id}"
missing_source:
default: warn

View File

@@ -91,6 +91,7 @@ type ReportRequest struct {
Renderer Renderer
Store state.Store
Notifier Notifier
noNotify bool
}
type ReportResult struct {
@@ -117,13 +118,33 @@ type ReportResult struct {
}
type BatchResult struct {
Batch BatchKind `json:"batch"`
StartedAt time.Time `json:"startedAt"`
FinishedAt time.Time `json:"finishedAt"`
Total int `json:"total"`
Succeeded int `json:"succeeded"`
Failed int `json:"failed"`
Reports []BatchReportResult `json:"reports"`
Batch BatchKind `json:"batch"`
StartedAt time.Time `json:"startedAt"`
FinishedAt time.Time `json:"finishedAt"`
Total int `json:"total"`
Succeeded int `json:"succeeded"`
Failed int `json:"failed"`
Notification *BatchNotificationResult `json:"notification,omitempty"`
Reports []BatchReportResult `json:"reports"`
}
type BatchNotificationResult struct {
Status string `json:"status"`
Reason string `json:"reason,omitempty"`
RunID string `json:"runId,omitempty"`
PipelineID string `json:"pipelineId,omitempty"`
BundleID string `json:"bundleId,omitempty"`
IdempotencyKey string `json:"idempotencyKey,omitempty"`
Path string `json:"path,omitempty"`
IncludedReports []BatchNotificationReport `json:"includedReports,omitempty"`
Error string `json:"error,omitempty"`
}
type BatchNotificationReport struct {
ReportID report.ID `json:"reportId"`
RunID string `json:"runId"`
SourcePath string `json:"sourcePath"`
BundlePaths []string `json:"bundlePaths"`
}
type BatchReportResult struct {
@@ -155,9 +176,32 @@ func (e BatchError) Error() string {
if e.Result == nil {
return "batch failed"
}
if batchNotificationFailed(e.Result) && batchReportFailures(e.Result) == 0 {
if e.Result.Notification.Error != "" {
return fmt.Sprintf("batch %s notification failed: %s", e.Result.Batch, e.Result.Notification.Error)
}
return fmt.Sprintf("batch %s notification failed", e.Result.Batch)
}
return fmt.Sprintf("batch %s failed: %d of %d reports failed", e.Result.Batch, e.Result.Failed, e.Result.Total)
}
func batchNotificationFailed(result *BatchResult) bool {
return result != nil && result.Notification != nil && result.Notification.Status == "failed"
}
func batchReportFailures(result *BatchResult) int {
if result == nil {
return 0
}
failures := 0
for _, item := range result.Reports {
if item.Status == "failed" {
failures++
}
}
return failures
}
type Renderer interface {
Render(context.Context, scriptorium.RenderRequest) (*scriptorium.RenderResult, error)
Run(context.Context, scriptorium.RunRequest) (*scriptorium.RunResult, error)
@@ -311,6 +355,7 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
Renderer: req.Renderer,
Store: store,
Notifier: req.Notifier,
noNotify: true,
})
if err != nil {
item.Status = "failed"
@@ -343,6 +388,13 @@ func RunBatchDetailed(ctx context.Context, req BatchRequest) (*BatchResult, erro
result.Reports = append(result.Reports, item)
}
result.Total = len(result.Reports)
batchNotification, err := notifyBatch(ctx, req.Config, req.Batch, batchRunID(startedAt, req.Batch), startedAt, result, plannedReports, store, req.Notifier)
if batchNotification != nil {
result.Notification = batchNotification
}
if err != nil {
result.Failed++
}
result.FinishedAt = time.Now()
return result, nil
}
@@ -599,6 +651,7 @@ func GenerateReport(ctx context.Context, req ReportRequest) (*ReportResult, erro
OutputPath: req.OutputPath,
Notifier: req.Notifier,
GenerationErr: runErr,
noNotify: req.noNotify,
})
if err != nil {
return nil, err
@@ -721,6 +774,7 @@ func generateTextTemplateReport(ctx context.Context, req generatedReportRequest)
ManagedReportPath: reportPath,
OutputPath: req.OutputPath,
Notifier: req.Notifier,
noNotify: req.noNotify,
})
if err != nil {
return nil, err
@@ -758,6 +812,7 @@ type finalizeRenderedReportRequest struct {
OutputPath string
Notifier Notifier
GenerationErr error
noNotify bool
}
type finalizeRenderedReportResult struct {
@@ -800,6 +855,9 @@ func finalizeRenderedReport(ctx context.Context, req finalizeRenderedReportReque
if req.GenerationErr != nil {
return result, req.GenerationErr
}
if req.noNotify {
return result, nil
}
notification, notificationPath, err := notifyReport(ctx, req.Config, req.Resolved, req.ManagedReportPath, metadata, req.Notifier, req.Store)
if notificationPath != "" {
@@ -859,14 +917,8 @@ func reportNotifier(cfg config.Config, notifier Notifier) (Notifier, bool) {
}
func buildNotificationRequest(cfg config.Config, resolved report.Resolved, reportPath string, metadata state.Metadata) (NotificationRequest, error) {
values := config.DistributorTemplateValues{
LocationID: cfg.Location.ID,
ReportID: string(resolved.Definition.ID),
RunID: metadata.RunID,
ArtifactGroup: resolved.Definition.ArtifactGroup,
BatchOutputName: resolved.Definition.BatchOutputName,
}
if err := addDistributorValidPeriodValues(&values, resolved.ValidPeriod, cfg.WeatherAPI.Timezone); err != nil {
values, err := distributorTemplateValuesForReport(cfg, resolved, metadata.RunID, resolved.Definition.BatchOutputName)
if err != nil {
return NotificationRequest{}, err
}
bundleID, err := config.RenderDistributorBundleID(cfg.Notify.Distributor.BundleIDTemplate, values)
@@ -898,6 +950,23 @@ func buildNotificationRequest(cfg config.Config, resolved report.Resolved, repor
}, nil
}
func distributorTemplateValuesForReport(cfg config.Config, resolved report.Resolved, runID string, batchOutputName string) (config.DistributorTemplateValues, error) {
values := config.DistributorTemplateValues{
LocationID: cfg.Location.ID,
ReportID: string(resolved.Definition.ID),
RunID: runID,
ArtifactGroup: resolved.Definition.ArtifactGroup,
BatchOutputName: batchOutputName,
}
if values.BatchOutputName == "" {
values.BatchOutputName = resolved.Definition.BatchOutputName
}
if err := addDistributorValidPeriodValues(&values, resolved.ValidPeriod, cfg.WeatherAPI.Timezone); err != nil {
return config.DistributorTemplateValues{}, err
}
return values, nil
}
func addDistributorValidPeriodValues(values *config.DistributorTemplateValues, period timeutil.Period, timezone string) error {
location, err := timeutil.LoadLocation(timezone)
if err != nil {
@@ -1005,6 +1074,38 @@ func (n distributorNotifier) Notify(ctx context.Context, req NotificationRequest
return notification, nil
}
func (n distributorNotifier) NotifyBatch(ctx context.Context, req batchNotificationRequest) (*NotificationResult, error) {
result, err := n.client.Upload(ctx, batchDistributorUploadRequest(req))
notification := notificationResultFromUpload(req.PipelineID, req.BundleID, req.IdempotencyKey, result)
if err != nil {
return notification, err
}
return notification, nil
}
func notificationResultFromUpload(pipelineID string, bundleID string, idempotencyKey string, result distributoradapter.UploadResult) *NotificationResult {
notification := &NotificationResult{
PipelineID: pipelineID,
BundleID: bundleID,
IdempotencyKey: idempotencyKey,
RunID: result.RunID,
Status: result.Status,
UploadStatus: result.UploadStatus,
StatusError: result.StatusError,
}
if result.RunStatus != nil {
if result.RunStatus.PipelineID != "" {
notification.PipelineID = result.RunStatus.PipelineID
}
notification.AcceptedAt = result.RunStatus.AcceptedAt
notification.StartedAt = result.RunStatus.StartedAt
notification.FinishedAt = result.RunStatus.FinishedAt
notification.Report = append([]byte(nil), result.RunStatus.Report...)
notification.Error = result.RunStatus.Error
}
return notification
}
func distributorUploadFiles(sourcePath string, bundlePaths []string) []distributoradapter.UploadFile {
files := make([]distributoradapter.UploadFile, 0, len(bundlePaths))
for _, bundlePath := range bundlePaths {

View File

@@ -2214,12 +2214,244 @@ func TestResolveGenerateStorm(t *testing.T) {
}
}
func TestBatchRunIDUsesUTCStartAndBatchName(t *testing.T) {
tests := []struct {
name string
startedAt time.Time
batch BatchKind
want string
}{
{
name: "morning",
startedAt: mustParse("2026-05-29T05:00:00-05:00"),
batch: BatchMorning,
want: "20260529T100000.000000000Z_morning",
},
{
name: "evening",
startedAt: mustParse("2026-05-29T18:30:45-05:00"),
batch: BatchEvening,
want: "20260529T233045.000000000Z_evening",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if got := batchRunID(tt.startedAt, tt.batch); got != tt.want {
t.Fatalf("batchRunID() = %q, want %q", got, tt.want)
}
})
}
}
func TestRenderBatchNotificationIdentity(t *testing.T) {
cfg := config.Defaults()
cfg.Location.ID = "home"
cfg.WeatherAPI.Timezone = "America/Chicago"
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "weatherreporter.{batch_started_date}"
cfg.Notify.Distributor.Batch.BundleIDTemplate = "weatherreporter.{location_id}.{batch}"
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{bundle_id}.{batch_run_id}"
startedAt := mustParse("2026-05-30T03:30:00Z")
runID := batchRunID(startedAt, BatchEvening)
identity, err := renderBatchNotificationIdentity(cfg, BatchEvening, runID, startedAt)
if err != nil {
t.Fatalf("renderBatchNotificationIdentity() error = %v", err)
}
if identity.PipelineID != "weatherreporter.2026-05-29" {
t.Fatalf("PipelineID = %q, want local batch date", identity.PipelineID)
}
if identity.BundleID != "weatherreporter.home.evening" {
t.Fatalf("BundleID = %q, want rendered bundle id", identity.BundleID)
}
wantKey := "weatherreporter.home.evening.20260530T033000.000000000Z_evening"
if identity.IdempotencyKey != wantKey {
t.Fatalf("IdempotencyKey = %q, want %q", identity.IdempotencyKey, wantKey)
}
}
func TestBatchResultJSONOmitsNilNotification(t *testing.T) {
data, err := json.Marshal(BatchResult{
Batch: BatchMorning,
Reports: []BatchReportResult{},
})
if err != nil {
t.Fatalf("Marshal() error = %v", err)
}
if strings.Contains(string(data), "notification") {
t.Fatalf("BatchResult JSON = %s, want no notification field", data)
}
}
func TestBatchResultJSONIncludesNotification(t *testing.T) {
result := BatchResult{
Batch: BatchEvening,
Notification: &BatchNotificationResult{
Status: "accepted",
RunID: "distributor-run",
PipelineID: "weatherreporter",
BundleID: "weatherreporter.home.evening",
IdempotencyKey: "weatherreporter.home.evening.20260529T233000.000000000Z_evening",
Path: "notifications/batches/evening/2026-05-29/20260529T233000.000000000Z_evening.distributor.json",
IncludedReports: []BatchNotificationReport{
{
ReportID: report.Tomorrow,
RunID: "20260529T233000.000000000Z_tomorrow",
SourcePath: "reports/tomorrow.md",
BundlePaths: []string{"tomorrow/index.md"},
},
},
},
Reports: []BatchReportResult{},
}
data, err := json.Marshal(result)
if err != nil {
t.Fatalf("Marshal() error = %v", err)
}
for _, want := range []string{
`"notification":{`,
`"status":"accepted"`,
`"runId":"distributor-run"`,
`"pipelineId":"weatherreporter"`,
`"bundleId":"weatherreporter.home.evening"`,
`"idempotencyKey":"weatherreporter.home.evening.20260529T233000.000000000Z_evening"`,
`"path":"notifications/batches/evening/2026-05-29/20260529T233000.000000000Z_evening.distributor.json"`,
`"includedReports":[`,
`"reportId":"tomorrow"`,
`"sourcePath":"reports/tomorrow.md"`,
`"bundlePaths":["tomorrow/index.md"]`,
} {
if !strings.Contains(string(data), want) {
t.Fatalf("BatchResult JSON = %s, want %s", data, want)
}
}
}
func TestBuildBatchNotificationRequestIncludesEveningReports(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
cfg.Notify.Distributor.ReportPathTemplates = []string{
"archive/{valid_start_date}/{artifact_group}/{run_id}.md",
"latest/{batch_output_name}",
}
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "weatherreporter.{batch}.{batch_started_date}"
cfg.Notify.Distributor.Batch.BundleIDTemplate = "weatherreporter.{location_id}.{batch}"
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{bundle_id}.{batch_run_id}"
startedAt := mustParse("2026-05-29T18:00:00-05:00")
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31", "2026-06-01")
runID := batchRunID(startedAt, BatchEvening)
req, err := buildBatchNotificationRequest(cfg, BatchEvening, runID, startedAt, reports, planned)
if err != nil {
t.Fatalf("buildBatchNotificationRequest() error = %v", err)
}
if req.Batch != BatchEvening || req.RunID != runID {
t.Fatalf("batch identity = %s/%s, want %s/%s", req.Batch, req.RunID, BatchEvening, runID)
}
if req.PipelineID != "weatherreporter.evening.2026-05-29" {
t.Fatalf("PipelineID = %q, want rendered batch pipeline", req.PipelineID)
}
if req.BundleID != "weatherreporter.home.evening" {
t.Fatalf("BundleID = %q, want rendered batch bundle id", req.BundleID)
}
wantKey := "weatherreporter.home.evening." + runID
if req.IdempotencyKey != wantKey {
t.Fatalf("IdempotencyKey = %q, want %q", req.IdempotencyKey, wantKey)
}
if !req.CreatedAt.Equal(startedAt) {
t.Fatalf("CreatedAt = %s, want %s", req.CreatedAt, startedAt)
}
if len(req.IncludedReports) != 3 {
t.Fatalf("IncludedReports = %d, want 3", len(req.IncludedReports))
}
if len(req.Files) != 6 {
t.Fatalf("Files = %d, want two mappings per report", len(req.Files))
}
wantBundlePaths := map[string]struct{}{}
for _, plannedReport := range planned {
resolved := plannedReport.Resolved
runID := resolved.Metadata().RunID
outputName := plannedReport.OutputCopyName
if outputName == "" {
outputName = resolved.Definition.BatchOutputName
}
wantBundlePaths[fmt.Sprintf("archive/%s/%s/%s.md", resolved.ValidPeriod.Start.In(mustLoadTestLocation(t, cfg.WeatherAPI.Timezone)).Format(timeutil.DateLayout), resolved.Definition.ArtifactGroup, runID)] = struct{}{}
wantBundlePaths["latest/"+outputName] = struct{}{}
}
gotBundlePaths := map[string]struct{}{}
gotSourcePaths := map[string]struct{}{}
for _, file := range req.Files {
gotBundlePaths[file.BundlePath] = struct{}{}
gotSourcePaths[file.SourcePath] = struct{}{}
}
for want := range wantBundlePaths {
if _, ok := gotBundlePaths[want]; !ok {
t.Fatalf("bundle paths = %#v, missing %q", gotBundlePaths, want)
}
}
for _, item := range reports {
if _, ok := gotSourcePaths[item.ReportPath]; !ok {
t.Fatalf("source paths = %#v, missing managed report path %q", gotSourcePaths, item.ReportPath)
}
}
uploadReq := batchDistributorUploadRequest(req)
if uploadReq.PipelineID != req.PipelineID || uploadReq.BundleID != req.BundleID || uploadReq.IdempotencyKey != req.IdempotencyKey || !uploadReq.CreatedAt.Equal(startedAt) {
t.Fatalf("upload request = %#v, want batch notification identity", uploadReq)
}
if len(uploadReq.Files) != len(req.Files) {
t.Fatalf("upload files = %d, want %d", len(uploadReq.Files), len(req.Files))
}
}
func TestBuildBatchNotificationRequestRejectsDuplicateBundlePaths(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
cfg.Notify.Distributor.ReportPathTemplates = []string{"index.md"}
startedAt := mustParse("2026-05-29T18:00:00-05:00")
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31")
_, err := buildBatchNotificationRequest(cfg, BatchEvening, batchRunID(startedAt, BatchEvening), startedAt, reports, planned)
if err == nil {
t.Fatal("buildBatchNotificationRequest() error = nil, want duplicate path error")
}
for _, want := range []string{"duplicate bundle path", "index.md", "report", "run", "source path"} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("error = %q, want %q", err.Error(), want)
}
}
}
func TestBuildBatchNotificationRequestRejectsMissingReportPath(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
startedAt := mustParse("2026-05-29T18:00:00-05:00")
planned, reports := plannedBatchNotificationReports(t, cfg, BatchEvening, startedAt, "2026-05-31")
reports[0].ReportPath = ""
_, err := buildBatchNotificationRequest(cfg, BatchEvening, batchRunID(startedAt, BatchEvening), startedAt, reports, planned)
if err == nil {
t.Fatal("buildBatchNotificationRequest() error = nil, want missing path error")
}
for _, want := range []string{"missing managed report path", string(reports[0].ReportID), reports[0].RunID} {
if !strings.Contains(err.Error(), want) {
t.Fatalf("error = %q, want %q", err.Error(), want)
}
}
}
func TestRunBatchContinuesAfterReportFailure(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyWorkspaceConfig(t, server)
cfg := dailyNotificationConfig(t, server)
collection := collectionWithFutureDailyForTest(t, cfg, "2026-05-31")
cfg.WeatherAPI.BaseURL = ""
collector := &recordingCollector{result: &collection}
notifier := &recordingNotifier{}
renderer := &selectiveRenderer{
failRenderPrompt: "weather.tomorrow_generated_text",
runBody: "# Batch Report\n",
@@ -2231,6 +2463,7 @@ func TestRunBatchContinuesAfterReportFailure(t *testing.T) {
Now: mustParse("2026-05-29T05:00:00-05:00"),
Collector: collector,
Renderer: renderer,
Notifier: notifier,
})
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
@@ -2245,6 +2478,12 @@ func TestRunBatchContinuesAfterReportFailure(t *testing.T) {
if renderer.runCalls != 0 || renderer.structuredRunCalls != 2 {
t.Fatalf("renderer calls run=%d structured=%d, want successful reports to continue", renderer.runCalls, renderer.structuredRunCalls)
}
if len(notifier.requests) != 0 || len(notifier.batchRequests) != 0 {
t.Fatalf("notification requests report=%d batch=%d, want none after report failure", len(notifier.requests), len(notifier.batchRequests))
}
if result.Notification == nil || result.Notification.Status != "skipped" || result.Notification.Reason != "one or more reports failed" {
t.Fatalf("batch notification = %#v, want skipped after report failure", result.Notification)
}
var failedTomorrow bool
var succeededDaily bool
for _, item := range result.Reports {
@@ -2266,17 +2505,13 @@ func TestRunBatchContinuesAfterReportFailure(t *testing.T) {
}
}
func TestRunBatchContinuesAfterNotificationFailure(t *testing.T) {
func TestRunBatchMorningSendsOneBatchNotification(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
collection := collectionWithFutureDailyForTest(t, cfg, "2026-05-31")
cfg.WeatherAPI.BaseURL = ""
collector := &recordingCollector{result: &collection}
notifier := &recordingNotifier{
errByReport: map[report.ID]error{
report.Tomorrow: errors.New("distributor unavailable"),
},
}
notifier := &recordingNotifier{batchResult: successfulBatchNotificationResult()}
result, err := RunBatchDetailed(context.Background(), BatchRequest{
Config: cfg,
@@ -2289,69 +2524,237 @@ func TestRunBatchContinuesAfterNotificationFailure(t *testing.T) {
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
}
if result.Total != 3 || result.Succeeded != 3 || result.Failed != 0 {
t.Fatalf("summary total/succeeded/failed = %d/%d/%d, want 3/3/0", result.Total, result.Succeeded, result.Failed)
}
if len(notifier.requests) != 0 {
t.Fatalf("per-report notification requests = %#v, want none", notifier.requests)
}
if len(notifier.batchRequests) != 1 {
t.Fatalf("batch notification requests = %d, want 1", len(notifier.batchRequests))
}
req := notifier.batchRequests[0]
if req.Batch != BatchMorning || req.RunID != "20260529T100000.000000000Z_morning" {
t.Fatalf("batch request identity = %s/%s, want morning run id", req.Batch, req.RunID)
}
if len(req.IncludedReports) != 3 || len(req.Files) != 3 {
t.Fatalf("batch request reports/files = %d/%d, want 3/3", len(req.IncludedReports), len(req.Files))
}
for _, file := range req.Files {
if file.SourcePath == "" || file.BundlePath == "" {
t.Fatalf("batch file = %#v, want source and bundle path", file)
}
if !strings.Contains(file.BundlePath, file.RunID) {
t.Fatalf("bundle path %q does not include report run id %q", file.BundlePath, file.RunID)
}
}
if result.Notification == nil || result.Notification.Status != "succeeded" || result.Notification.RunID != "batch-distributor-run" || result.Notification.Path == "" {
t.Fatalf("batch notification = %#v, want succeeded result with artifact path", result.Notification)
}
if len(result.Notification.IncludedReports) != 3 {
t.Fatalf("batch notification included reports = %d, want 3", len(result.Notification.IncludedReports))
}
artifact := readBatchNotificationForTest(t, result.Notification.Path)
if artifact.Status != "succeeded" || artifact.Upload == nil || artifact.Upload.RunID != "batch-distributor-run" || artifact.RunStatus == nil {
t.Fatalf("batch notification artifact = %#v, want succeeded upload and run status", artifact)
}
if len(artifact.Reports) != 3 {
t.Fatalf("artifact included reports = %d, want 3", len(artifact.Reports))
}
}
if result.Total != 3 || result.Succeeded != 2 || result.Failed != 1 {
t.Fatalf("summary total/succeeded/failed = %d/%d/%d, want 3/2/1", result.Total, result.Succeeded, result.Failed)
func TestRunBatchSuppressesPerReportNotification(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
collection := collectionWithFutureDailyForTest(t, cfg, "2026-05-31")
cfg.WeatherAPI.BaseURL = ""
collector := &recordingCollector{result: &collection}
store := recordingFilesystemStore(t, cfg)
notifier := &recordingNotifier{err: errors.New("distributor unavailable")}
result, err := RunBatchDetailed(context.Background(), BatchRequest{
Config: cfg,
Batch: BatchEvening,
Now: mustParse("2026-05-29T18:00:00-05:00"),
Collector: collector,
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
Store: store,
Notifier: notifier,
})
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
}
if result.Total != 2 || result.Succeeded != 2 || result.Failed != 0 {
t.Fatalf("summary total/succeeded/failed = %d/%d/%d, want 2/2/0", result.Total, result.Succeeded, result.Failed)
}
if len(collector.requests) != 1 {
t.Fatalf("collector requests = %d, want one collection for batch", len(collector.requests))
}
if len(notifier.requests) != 3 {
t.Fatalf("notification requests = %d, want one per generated report", len(notifier.requests))
if len(notifier.requests) != 0 {
t.Fatalf("notification requests = %#v, want none for batch-generated reports", notifier.requests)
}
if len(notifier.batchRequests) != 1 {
t.Fatalf("batch notification requests = %d, want 1", len(notifier.batchRequests))
}
var failedTomorrow bool
for _, item := range result.Reports {
if item.ReportID == report.Tomorrow {
if item.Status == "failed" && strings.Contains(item.Error, "notify report") && strings.Contains(item.Error, "distributor unavailable") {
failedTomorrow = true
}
if item.NotificationStatus != "failed" {
t.Fatalf("Tomorrow notification status = %q, want failed", item.NotificationStatus)
}
if item.NotificationPipelineID != "weatherreporter.tomorrow" {
t.Fatalf("Tomorrow notification pipeline = %q, want weatherreporter.tomorrow", item.NotificationPipelineID)
}
if !strings.Contains(item.NotificationError, "distributor unavailable") {
t.Fatalf("Tomorrow notification error = %q, want distributor unavailable", item.NotificationError)
}
continue
}
if item.Status != "succeeded" {
t.Fatalf("report %s status = %s, want succeeded", item.ReportID, item.Status)
}
if item.NotificationStatus != "accepted" {
t.Fatalf("report %s notification status = %q, want accepted", item.ReportID, item.NotificationStatus)
if item.NotificationStatus != "" || item.NotificationRunID != "" || item.NotificationPipelineID != "" || item.NotificationError != "" || item.NotificationPath != "" {
t.Fatalf("report %s notification fields = %#v, want empty per-report notification fields", item.ReportID, item)
}
if item.NotificationPipelineID == "" {
t.Fatalf("report %s notification pipeline is empty", item.ReportID)
metadata := readMetadataForTest(t, item.MetadataPath)
if metadata.NotificationPath != "" {
t.Fatalf("report %s metadata NotificationPath = %q, want empty", item.ReportID, metadata.NotificationPath)
}
}
if !failedTomorrow {
t.Fatalf("reports = %#v, want notification failure on Tomorrow item", result.Reports)
for _, item := range result.Reports {
reportNotificationDir := filepath.Join(cfg.Workspace.Root, cfg.Workspace.NotificationsDir, string(item.ReportID))
if _, err := os.Stat(reportNotificationDir); err == nil {
t.Fatalf("per-report notification directory %q exists, want none", reportNotificationDir)
} else if !os.IsNotExist(err) {
t.Fatalf("stat per-report notification directory %q: %v", reportNotificationDir, err)
}
}
}
func TestRunBatchNotificationFailureKeepsReportItemsSucceeded(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
collection := collectionWithFutureDailyForTest(t, cfg, "2026-05-31")
cfg.WeatherAPI.BaseURL = ""
notifier := &recordingNotifier{batchErr: errors.New("batch upload rejected")}
result, err := RunBatchDetailed(context.Background(), BatchRequest{
Config: cfg,
Batch: BatchEvening,
Now: mustParse("2026-05-29T18:00:00-05:00"),
Collector: &recordingCollector{result: &collection},
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
Notifier: notifier,
})
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
}
if result.Failed != 1 || result.Succeeded != 2 {
t.Fatalf("summary succeeded/failed = %d/%d, want report successes plus notification failure", result.Succeeded, result.Failed)
}
for _, item := range result.Reports {
if item.Status != "succeeded" {
t.Fatalf("report %s status = %s, want succeeded despite batch notification failure", item.ReportID, item.Status)
}
}
if result.Notification == nil || result.Notification.Status != "failed" || !strings.Contains(result.Notification.Error, "batch upload rejected") || result.Notification.Path == "" {
t.Fatalf("batch notification = %#v, want failed upload result", result.Notification)
}
artifact := readBatchNotificationForTest(t, result.Notification.Path)
if artifact.Status != "failed" || !strings.Contains(artifact.Error, "batch upload rejected") {
t.Fatalf("batch notification artifact = %#v, want failed upload error", artifact)
}
cfg.Workspace.Root = t.TempDir()
collector = &recordingCollector{result: &collection}
err = RunBatch(context.Background(), BatchRequest{
Config: cfg,
Batch: BatchMorning,
Now: mustParse("2026-05-29T05:00:00-05:00"),
Collector: collector,
Renderer: &selectiveRenderer{
runBody: "# Batch Report\n",
},
Notifier: &recordingNotifier{
errByReport: map[report.ID]error{
report.Tomorrow: errors.New("distributor unavailable"),
},
},
Batch: BatchEvening,
Now: mustParse("2026-05-29T18:00:00-05:00"),
Collector: &recordingCollector{result: &collection},
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
Notifier: &recordingNotifier{batchErr: errors.New("batch upload rejected")},
})
var batchErr BatchError
if !errors.As(err, &batchErr) {
t.Fatalf("RunBatch() error = %T %v, want BatchError", err, err)
}
if batchErr.Result == nil || batchErr.Result.Failed != 1 || batchErr.Result.Succeeded != 2 {
t.Fatalf("RunBatch() result = %#v, want notification failure aggregate", batchErr.Result)
if batchErr.Result == nil || !batchNotificationFailed(batchErr.Result) || batchReportFailures(batchErr.Result) != 0 {
t.Fatalf("RunBatch() result = %#v, want notification-only batch failure", batchErr.Result)
}
}
func TestRunBatchNotificationStatusErrorPersistsStatusReport(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
collection := collectionWithFutureDailyForTest(t, cfg, "2026-05-31")
cfg.WeatherAPI.BaseURL = ""
notifier := &recordingNotifier{
batchResult: &NotificationResult{
RunID: "batch-distributor-run",
Status: "accepted",
UploadStatus: "accepted",
StatusError: "status lookup unavailable",
Report: []byte(`{"actions":[{"action":"replace_older"}]}`),
},
}
result, err := RunBatchDetailed(context.Background(), BatchRequest{
Config: cfg,
Batch: BatchMorning,
Now: mustParse("2026-05-29T05:00:00-05:00"),
Collector: &recordingCollector{result: &collection},
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
Notifier: notifier,
})
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
}
if result.Failed != 0 || result.Notification == nil || result.Notification.Path == "" {
t.Fatalf("result = %#v, want status-error notification artifact without batch failure", result)
}
artifact := readBatchNotificationForTest(t, result.Notification.Path)
if artifact.StatusError != "status lookup unavailable" || artifact.RunStatus == nil || !strings.Contains(string(artifact.RunStatus.Report), "replace_older") {
t.Fatalf("batch notification artifact = %#v, want status error and raw status report", artifact)
}
}
func TestRunBatchDisabledDistributorSkipsBatchNotification(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyWorkspaceConfig(t, server)
collection := collectionWithFutureDailyForTest(t, cfg, "2026-05-31")
cfg.WeatherAPI.BaseURL = ""
notifier := &recordingNotifier{}
result, err := RunBatchDetailed(context.Background(), BatchRequest{
Config: cfg,
Batch: BatchEvening,
Now: mustParse("2026-05-29T18:00:00-05:00"),
Collector: &recordingCollector{result: &collection},
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
Notifier: notifier,
})
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
}
if result.Notification != nil || result.Failed != 0 {
t.Fatalf("result notification/failed = %#v/%d, want disabled notification omitted", result.Notification, result.Failed)
}
if len(notifier.requests) != 0 || len(notifier.batchRequests) != 0 {
t.Fatalf("notification requests report=%d batch=%d, want none when distributor disabled", len(notifier.requests), len(notifier.batchRequests))
}
}
func TestRunBatchDisabledBatchNotificationSkipsNotifier(t *testing.T) {
server := dailyBundleServer(t)
cfg := dailyNotificationConfig(t, server)
cfg.Notify.Distributor.Batch.Enabled = false
collection := collectionWithFutureDailyForTest(t, cfg, "2026-05-31")
cfg.WeatherAPI.BaseURL = ""
notifier := &recordingNotifier{}
result, err := RunBatchDetailed(context.Background(), BatchRequest{
Config: cfg,
Batch: BatchEvening,
Now: mustParse("2026-05-29T18:00:00-05:00"),
Collector: &recordingCollector{result: &collection},
Renderer: &selectiveRenderer{runBody: "# Batch Report\n"},
Notifier: notifier,
})
if err != nil {
t.Fatalf("RunBatchDetailed() error = %v", err)
}
if result.Notification != nil || result.Failed != 0 {
t.Fatalf("result notification/failed = %#v/%d, want disabled batch notification omitted", result.Notification, result.Failed)
}
if len(notifier.requests) != 0 || len(notifier.batchRequests) != 0 {
t.Fatalf("notification requests report=%d batch=%d, want none when batch notification disabled", len(notifier.requests), len(notifier.batchRequests))
}
}
@@ -2433,8 +2836,11 @@ func TestRunBatchDynamicDailyReportsHaveDistinctIdentity(t *testing.T) {
if result.Failed != 0 || len(result.Reports) != 3 {
t.Fatalf("summary = %#v, want three successful reports", result)
}
if len(notifier.requests) != 3 {
t.Fatalf("notification requests = %d, want one per report", len(notifier.requests))
if len(notifier.requests) != 0 {
t.Fatalf("notification requests = %#v, want none for batch-generated reports", notifier.requests)
}
if len(notifier.batchRequests) != 1 {
t.Fatalf("batch notification requests = %d, want 1", len(notifier.batchRequests))
}
dailyByDate := map[string]BatchReportResult{}
@@ -2482,22 +2888,18 @@ func TestRunBatchDynamicDailyReportsHaveDistinctIdentity(t *testing.T) {
}
}
dailyKeys := map[string]struct{}{}
for _, req := range notifier.requests {
if req.ReportID != report.Daily {
continue
}
if req.IdempotencyKey != req.BundleID+"."+req.RunID {
t.Fatalf("Daily idempotency key = %q, want bundle id plus run id", req.IdempotencyKey)
}
if _, ok := dailyKeys[req.IdempotencyKey]; ok {
t.Fatalf("duplicate Daily idempotency key: %q", req.IdempotencyKey)
}
dailyKeys[req.IdempotencyKey] = struct{}{}
if len(notifier.batchRequests[0].IncludedReports) != len(result.Reports) {
t.Fatalf("batch included reports = %d, want %d", len(notifier.batchRequests[0].IncludedReports), len(result.Reports))
}
if len(dailyKeys) != 2 {
t.Fatalf("Daily notification keys = %#v, want two distinct keys", dailyKeys)
for _, included := range notifier.batchRequests[0].IncludedReports {
if strings.Contains(included.SourcePath, outputDir) {
t.Fatalf("batch notification source path = %q, want managed report path outside output dir", included.SourcePath)
}
if _, ok := reportPaths[included.SourcePath]; !ok {
t.Fatalf("batch notification source path = %q, want one of %#v", included.SourcePath, reportPaths)
}
}
}
func TestRunBatchMorningUsesTodayOutputName(t *testing.T) {
@@ -2596,6 +2998,27 @@ func dailyNotificationConfig(t *testing.T, server *httptest.Server) config.Confi
return cfg
}
func plannedBatchNotificationReports(t *testing.T, cfg config.Config, batch BatchKind, now time.Time, futureDailyDates ...string) ([]plannedBatchReport, []BatchReportResult) {
t.Helper()
collection := collectionWithFutureDailyForTest(t, cfg, futureDailyDates...)
planned, err := planBatchRun(BatchRequest{Config: cfg, Batch: batch}, now, collection)
if err != nil {
t.Fatalf("planBatchRun() error = %v", err)
}
reportDir := filepath.Join(t.TempDir(), "managed-reports")
results := make([]BatchReportResult, 0, len(planned))
for _, item := range planned {
metadata := item.Resolved.Metadata()
results = append(results, BatchReportResult{
ReportID: item.Resolved.Definition.ID,
RunID: metadata.RunID,
Status: "succeeded",
ReportPath: filepath.Join(reportDir, string(item.Resolved.Definition.ID), metadata.RunID+".md"),
})
}
return planned, results
}
func collectionForTest(t *testing.T, cfg config.Config) collect.Result {
t.Helper()
bundle, err := FetchBundle(context.Background(), FetchBundleRequest{Config: cfg})
@@ -2644,6 +3067,15 @@ func successfulNotificationResult() *NotificationResult {
}
}
func successfulBatchNotificationResult() *NotificationResult {
return &NotificationResult{
RunID: "batch-distributor-run",
Status: "succeeded",
UploadStatus: "accepted",
Report: []byte(`{"actions":[{"action":"replace_older"}]}`),
}
}
func successfulGeneratedTextRenderer(body string) *recordingRenderer {
return &recordingRenderer{
renderResult: &scriptorium.RenderResult{ExitCode: 0},
@@ -2927,6 +3359,19 @@ func readMetadataForTest(t *testing.T, path string) state.Metadata {
return metadata
}
func readBatchNotificationForTest(t *testing.T, path string) state.BatchDistributorNotificationArtifact {
t.Helper()
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read batch notification %q: %v", path, err)
}
var artifact state.BatchDistributorNotificationArtifact
if err := json.Unmarshal(data, &artifact); err != nil {
t.Fatalf("decode batch notification %q: %v", path, err)
}
return artifact
}
func assertGeneratedReportError(t *testing.T, err error, resolved report.Resolved, operation string) {
t.Helper()
if err == nil {
@@ -3139,10 +3584,13 @@ type selectiveRenderer struct {
}
type recordingNotifier struct {
requests []NotificationRequest
result *NotificationResult
err error
errByReport map[report.ID]error
requests []NotificationRequest
batchRequests []batchNotificationRequest
result *NotificationResult
batchResult *NotificationResult
err error
batchErr error
errByReport map[report.ID]error
}
func (n *recordingNotifier) Notify(_ context.Context, req NotificationRequest) (*NotificationResult, error) {
@@ -3175,6 +3623,34 @@ func (n *recordingNotifier) Notify(_ context.Context, req NotificationRequest) (
}, nil
}
func (n *recordingNotifier) NotifyBatch(_ context.Context, req batchNotificationRequest) (*NotificationResult, error) {
n.batchRequests = append(n.batchRequests, req)
if n.batchErr != nil {
return nil, n.batchErr
}
if n.batchResult != nil {
result := *n.batchResult
if result.BundleID == "" {
result.BundleID = req.BundleID
}
if result.IdempotencyKey == "" {
result.IdempotencyKey = req.IdempotencyKey
}
if result.PipelineID == "" {
result.PipelineID = req.PipelineID
}
return &result, nil
}
return &NotificationResult{
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
RunID: "batch-distributor-run",
Status: "accepted",
UploadStatus: "accepted",
}, nil
}
func (r *selectiveRenderer) Render(_ context.Context, req scriptorium.RenderRequest) (*scriptorium.RenderResult, error) {
r.renderCalls++
if req.PromptID == r.failRenderPrompt {

View File

@@ -0,0 +1,379 @@
package app
import (
"context"
"fmt"
"time"
distributoradapter "gitea.maximumdirect.net/eric/weatherreporter/internal/adapters/distributor"
"gitea.maximumdirect.net/eric/weatherreporter/internal/config"
"gitea.maximumdirect.net/eric/weatherreporter/internal/report"
"gitea.maximumdirect.net/eric/weatherreporter/internal/state"
"gitea.maximumdirect.net/eric/weatherreporter/internal/timeutil"
)
const runIDTimestampLayout = "20060102T150405.000000000Z"
type batchNotificationIdentity struct {
PipelineID string
BundleID string
IdempotencyKey string
}
type batchNotificationRequest struct {
Batch BatchKind
RunID string
PipelineID string
BundleID string
IdempotencyKey string
Files []batchNotificationFile
IncludedReports []BatchNotificationReport
CreatedAt time.Time
}
type batchNotificationFile struct {
ReportID report.ID
RunID string
SourcePath string
BundlePath string
}
type batchNotifier interface {
NotifyBatch(context.Context, batchNotificationRequest) (*NotificationResult, error)
}
func batchRunID(startedAt time.Time, batch BatchKind) string {
return startedAt.UTC().Format(runIDTimestampLayout) + "_" + string(batch)
}
func notifyBatch(ctx context.Context, cfg config.Config, batch BatchKind, runID string, startedAt time.Time, result *BatchResult, planned []plannedBatchReport, store state.Store, notifier Notifier) (*BatchNotificationResult, error) {
if !cfg.Notify.Distributor.Enabled {
return nil, nil
}
if !cfg.Notify.Distributor.Batch.Enabled {
return nil, nil
}
if result == nil {
return nil, fmt.Errorf("batch result is required")
}
if result.Failed > 0 {
return &BatchNotificationResult{
Status: "skipped",
Reason: "one or more reports failed",
}, nil
}
req, err := buildBatchNotificationRequest(cfg, batch, runID, startedAt, result.Reports, planned)
if err != nil {
path, saveErr := saveBatchNotificationArtifact(ctx, store, cfg, batch, runID, startedAt, batchNotificationRequest{}, nil, err)
if saveErr != nil {
return nil, saveErr
}
return failedBatchNotificationResult(batchNotificationRequest{}, path, err), err
}
batchNotifier, err := resolveBatchNotifier(cfg, notifier)
if err != nil {
path, saveErr := saveBatchNotificationArtifact(ctx, store, cfg, batch, runID, startedAt, req, nil, err)
if saveErr != nil {
return nil, saveErr
}
return failedBatchNotificationResult(req, path, err), err
}
notification, notifyErr := batchNotifier.NotifyBatch(ctx, req)
wrappedErr := notifyErr
if notifyErr != nil {
wrappedErr = fmt.Errorf("notify batch %q run %q bundle %q: %w", batch, runID, req.BundleID, notifyErr)
}
path, saveErr := saveBatchNotificationArtifact(ctx, store, cfg, batch, runID, startedAt, req, notification, wrappedErr)
if saveErr != nil {
return nil, saveErr
}
batchResult := batchNotificationResult(req, notification, path)
if wrappedErr != nil {
batchResult.Status = "failed"
batchResult.Error = wrappedErr.Error()
return batchResult, wrappedErr
}
return batchResult, nil
}
func resolveBatchNotifier(cfg config.Config, notifier Notifier) (batchNotifier, error) {
if notifier != nil {
if batchNotifier, ok := notifier.(batchNotifier); ok {
return batchNotifier, nil
}
return nil, fmt.Errorf("batch distributor notifier is required")
}
return distributorNotifier{
client: distributoradapter.New(cfg.Notify.Distributor),
}, nil
}
func buildBatchNotificationRequest(cfg config.Config, batch BatchKind, runID string, startedAt time.Time, reports []BatchReportResult, planned []plannedBatchReport) (batchNotificationRequest, error) {
if len(reports) == 0 {
return batchNotificationRequest{}, fmt.Errorf("batch notification requires at least one report")
}
identity, err := renderBatchNotificationIdentity(cfg, batch, runID, startedAt)
if err != nil {
return batchNotificationRequest{}, err
}
if identity.PipelineID == "" {
return batchNotificationRequest{}, fmt.Errorf("batch notification pipeline id is required")
}
if identity.BundleID == "" {
return batchNotificationRequest{}, fmt.Errorf("batch notification bundle id is required")
}
if identity.IdempotencyKey == "" {
return batchNotificationRequest{}, fmt.Errorf("batch notification idempotency key is required for bundle %q", identity.BundleID)
}
plannedByRunID, err := plannedReportsByRunID(planned)
if err != nil {
return batchNotificationRequest{}, err
}
req := batchNotificationRequest{
Batch: batch,
RunID: runID,
PipelineID: identity.PipelineID,
BundleID: identity.BundleID,
IdempotencyKey: identity.IdempotencyKey,
CreatedAt: startedAt,
}
seenBundlePaths := map[string]batchNotificationFile{}
for _, item := range reports {
plannedReport, ok := plannedByRunID[item.RunID]
if !ok {
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q has no matching planned report", item.ReportID, item.RunID)
}
if item.ReportID != plannedReport.Resolved.Definition.ID {
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q does not match planned report %q", item.ReportID, item.RunID, plannedReport.Resolved.Definition.ID)
}
if item.ReportPath == "" {
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q is missing managed report path", item.ReportID, item.RunID)
}
values, err := distributorTemplateValuesForReport(cfg, plannedReport.Resolved, item.RunID, plannedReport.OutputCopyName)
if err != nil {
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q source path %q: %w", item.ReportID, item.RunID, item.ReportPath, err)
}
bundlePaths, err := config.RenderDistributorReportPaths(cfg.Notify.Distributor.ReportPathTemplates, values)
if err != nil {
return batchNotificationRequest{}, fmt.Errorf("batch notification report %q run %q source path %q: %w", item.ReportID, item.RunID, item.ReportPath, err)
}
included := BatchNotificationReport{
ReportID: item.ReportID,
RunID: item.RunID,
SourcePath: item.ReportPath,
BundlePaths: append([]string(nil), bundlePaths...),
}
for _, bundlePath := range bundlePaths {
file := batchNotificationFile{
ReportID: item.ReportID,
RunID: item.RunID,
SourcePath: item.ReportPath,
BundlePath: bundlePath,
}
if previous, ok := seenBundlePaths[bundlePath]; ok {
return batchNotificationRequest{}, fmt.Errorf("batch notification duplicate bundle path %q for report %q run %q source path %q; already used by report %q run %q source path %q", bundlePath, item.ReportID, item.RunID, item.ReportPath, previous.ReportID, previous.RunID, previous.SourcePath)
}
seenBundlePaths[bundlePath] = file
req.Files = append(req.Files, file)
}
req.IncludedReports = append(req.IncludedReports, included)
}
if len(req.Files) == 0 {
return batchNotificationRequest{}, fmt.Errorf("batch notification requires at least one file mapping")
}
return req, nil
}
func plannedReportsByRunID(planned []plannedBatchReport) (map[string]plannedBatchReport, error) {
byRunID := make(map[string]plannedBatchReport, len(planned))
for _, item := range planned {
runID := item.Resolved.Metadata().RunID
if runID == "" {
return nil, fmt.Errorf("planned report %q has empty run id", item.Resolved.Definition.ID)
}
if previous, ok := byRunID[runID]; ok {
return nil, fmt.Errorf("planned reports %q and %q share run id %q", previous.Resolved.Definition.ID, item.Resolved.Definition.ID, runID)
}
byRunID[runID] = item
}
return byRunID, nil
}
func batchDistributorUploadRequest(req batchNotificationRequest) distributoradapter.UploadRequest {
files := make([]distributoradapter.UploadFile, 0, len(req.Files))
for _, file := range req.Files {
files = append(files, distributoradapter.UploadFile{
SourcePath: file.SourcePath,
BundlePath: file.BundlePath,
})
}
return distributoradapter.UploadRequest{
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
Files: files,
CreatedAt: req.CreatedAt,
}
}
func batchNotificationResult(req batchNotificationRequest, result *NotificationResult, path string) *BatchNotificationResult {
notification := &BatchNotificationResult{
Status: "unknown",
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
Path: path,
IncludedReports: append([]BatchNotificationReport(nil), req.IncludedReports...),
}
if result != nil {
notification.Status = result.Status
notification.RunID = result.RunID
if result.PipelineID != "" {
notification.PipelineID = result.PipelineID
}
if result.BundleID != "" {
notification.BundleID = result.BundleID
}
if result.IdempotencyKey != "" {
notification.IdempotencyKey = result.IdempotencyKey
}
if result.Error != "" {
notification.Error = result.Error
}
}
if notification.Status == "" {
notification.Status = "unknown"
}
return notification
}
func failedBatchNotificationResult(req batchNotificationRequest, path string, err error) *BatchNotificationResult {
notification := batchNotificationResult(req, nil, path)
notification.Status = "failed"
if err != nil {
notification.Error = err.Error()
}
return notification
}
func saveBatchNotificationArtifact(ctx context.Context, store state.Store, cfg config.Config, batch BatchKind, runID string, startedAt time.Time, req batchNotificationRequest, result *NotificationResult, notifyErr error) (string, error) {
if store == nil {
return "", fmt.Errorf("state store is required")
}
location, err := timeutil.LoadLocation(cfg.WeatherAPI.Timezone)
if err != nil {
return "", fmt.Errorf("load batch notification timezone: %w", err)
}
artifact := state.BatchDistributorNotificationArtifact{
SchemaVersion: state.BatchDistributorNotificationSchemaVersion,
Batch: string(batch),
BatchRunID: runID,
AttemptedAt: time.Now(),
Endpoint: cfg.Notify.Distributor.Endpoint,
PipelineID: req.PipelineID,
BundleID: req.BundleID,
IdempotencyKey: req.IdempotencyKey,
BundleCreated: req.CreatedAt,
Reports: batchNotificationReportArtifacts(req.IncludedReports),
Status: "attempted",
}
if result != nil {
artifact.Status = result.Status
artifact.Upload = &state.DistributorUploadResult{
RunID: result.RunID,
Status: result.UploadStatus,
}
if result.PipelineID != "" || !result.AcceptedAt.IsZero() || result.StartedAt != nil || result.FinishedAt != nil || len(result.Report) > 0 || result.Error != "" {
artifact.RunStatus = &state.DistributorRunStatus{
RunID: result.RunID,
PipelineID: result.PipelineID,
Status: result.Status,
AcceptedAt: result.AcceptedAt,
StartedAt: result.StartedAt,
FinishedAt: result.FinishedAt,
Report: append([]byte(nil), result.Report...),
Error: result.Error,
}
}
artifact.StatusError = result.StatusError
}
if notifyErr != nil {
artifact.Status = "failed"
artifact.Error = notifyErr.Error()
}
if artifact.Status == "" {
artifact.Status = "unknown"
}
return store.SaveBatchDistributorNotification(ctx, state.BatchDistributorNotificationRef{
Batch: string(batch),
BatchRunID: runID,
StartedAt: startedAt,
Location: location,
}, artifact)
}
func batchNotificationReportArtifacts(reports []BatchNotificationReport) []state.BatchDistributorNotificationReportArtifact {
if len(reports) == 0 {
return nil
}
artifacts := make([]state.BatchDistributorNotificationReportArtifact, 0, len(reports))
for _, item := range reports {
artifacts = append(artifacts, state.BatchDistributorNotificationReportArtifact{
ReportID: item.ReportID,
RunID: item.RunID,
SourcePath: item.SourcePath,
BundlePaths: append([]string(nil), item.BundlePaths...),
})
}
return artifacts
}
func renderBatchNotificationIdentity(cfg config.Config, batch BatchKind, runID string, startedAt time.Time) (batchNotificationIdentity, error) {
values, err := batchNotificationTemplateValues(cfg, batch, runID, startedAt)
if err != nil {
return batchNotificationIdentity{}, err
}
bundleID, err := config.RenderDistributorBatchBundleID(cfg.Notify.Distributor.Batch.BundleIDTemplate, values)
if err != nil {
return batchNotificationIdentity{}, err
}
values.BundleID = bundleID
pipelineID, err := config.RenderDistributorBatchPipelineID(cfg.Notify.Distributor.Batch.PipelineIDTemplate, values)
if err != nil {
return batchNotificationIdentity{}, err
}
idempotencyKey, err := config.RenderDistributorBatchIdempotencyKey(cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate, values)
if err != nil {
return batchNotificationIdentity{}, err
}
return batchNotificationIdentity{
PipelineID: pipelineID,
BundleID: bundleID,
IdempotencyKey: idempotencyKey,
}, nil
}
func batchNotificationTemplateValues(cfg config.Config, batch BatchKind, runID string, startedAt time.Time) (config.DistributorBatchTemplateValues, error) {
location, err := timeutil.LoadLocation(cfg.WeatherAPI.Timezone)
if err != nil {
return config.DistributorBatchTemplateValues{}, fmt.Errorf("load batch notification timezone: %w", err)
}
return config.DistributorBatchTemplateValues{
LocationID: cfg.Location.ID,
Batch: string(batch),
BatchRunID: runID,
BatchStartedDate: startedAt.In(location).Format(timeutil.DateLayout),
}, nil
}

View File

@@ -378,6 +378,28 @@ func writeRunLogs(stderr io.Writer, result *app.BatchResult) {
}
_, _ = fmt.Fprintf(stderr, "report=%s status=succeeded output=%q%s\n", item.ReportID, item.OutputPath, notificationFields)
}
if result.Notification != nil {
_, _ = fmt.Fprintf(stderr, "batchNotification status=%q", result.Notification.Status)
if result.Notification.Reason != "" {
_, _ = fmt.Fprintf(stderr, " reason=%q", result.Notification.Reason)
}
if result.Notification.RunID != "" {
_, _ = fmt.Fprintf(stderr, " runId=%q", result.Notification.RunID)
}
if result.Notification.PipelineID != "" {
_, _ = fmt.Fprintf(stderr, " pipelineId=%q", result.Notification.PipelineID)
}
if result.Notification.BundleID != "" {
_, _ = fmt.Fprintf(stderr, " bundleId=%q", result.Notification.BundleID)
}
if result.Notification.Path != "" {
_, _ = fmt.Fprintf(stderr, " path=%q", result.Notification.Path)
}
if result.Notification.Error != "" {
_, _ = fmt.Fprintf(stderr, " error=%q", result.Notification.Error)
}
_, _ = fmt.Fprintln(stderr)
}
_, _ = fmt.Fprintf(stderr, "batch=%s total=%d succeeded=%d failed=%d\n", result.Batch, result.Total, result.Succeeded, result.Failed)
}

View File

@@ -211,26 +211,33 @@ func TestRunMorningReportsPartialFailureAndContinues(t *testing.T) {
_ = oneArtifact(t, fixture.workspaceRoot, "data-packages", "tomorrow", "2026-05-30", "*.data_package.yaml")
}
func TestBatchOutputIncludesNotificationDetails(t *testing.T) {
func TestBatchOutputIncludesTopLevelNotificationDetails(t *testing.T) {
result := &app.BatchResult{
Batch: app.BatchMorning,
Total: 2,
Succeeded: 1,
Failed: 1,
Succeeded: 2,
Failed: 0,
Notification: &app.BatchNotificationResult{
Status: "succeeded",
RunID: "batch-distributor-run",
PipelineID: "weatherreporter",
BundleID: "weatherreporter.home.morning",
IdempotencyKey: "weatherreporter.home.morning.20260529T120000.000000000Z_morning",
Path: "/tmp/batch.distributor.json",
IncludedReports: []app.BatchNotificationReport{
{ReportID: "daily", RunID: "daily-run", SourcePath: "/tmp/daily.md", BundlePaths: []string{"daily.md"}},
},
},
Reports: []app.BatchReportResult{
{
ReportID: "daily",
Status: "succeeded",
OutputPath: "/tmp/daily.md",
NotificationStatus: "accepted",
NotificationRunID: "distributor-run-1",
ReportID: "daily",
Status: "succeeded",
OutputPath: "/tmp/daily.md",
},
{
ReportID: "three_day",
Status: "failed",
Error: "notify report three_day: upload failed",
NotificationStatus: "failed",
NotificationError: "notify report three_day: upload failed",
ReportID: "tomorrow",
Status: "succeeded",
OutputPath: "/tmp/tomorrow.md",
},
},
}
@@ -246,17 +253,22 @@ func TestBatchOutputIncludesNotificationDetails(t *testing.T) {
if err := json.Unmarshal(stdout.Bytes(), &decoded); err != nil {
t.Fatalf("decode batch JSON: %v\n%s", err, stdout.String())
}
if decoded.Reports[0].NotificationStatus != "accepted" || decoded.Reports[0].NotificationRunID != "distributor-run-1" {
t.Fatalf("success notification fields = %#v", decoded.Reports[0])
if decoded.Notification == nil || decoded.Notification.Status != "succeeded" || decoded.Notification.RunID != "batch-distributor-run" || decoded.Notification.PipelineID != "weatherreporter" || len(decoded.Notification.IncludedReports) != 1 {
t.Fatalf("top-level notification = %#v, want succeeded batch notification", decoded.Notification)
}
if decoded.Reports[1].NotificationStatus != "failed" || !strings.Contains(decoded.Reports[1].NotificationError, "upload failed") {
t.Fatalf("failure notification fields = %#v", decoded.Reports[1])
for _, report := range decoded.Reports {
if report.NotificationStatus != "" || report.NotificationRunID != "" || report.NotificationError != "" {
t.Fatalf("report notification fields = %#v, want empty", report)
}
}
if !strings.Contains(stderr.String(), `notificationStatus="accepted"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
t.Fatalf("stderr missing success notification fields:\n%s", stderr.String())
if count := strings.Count(stderr.String(), "batchNotification "); count != 1 {
t.Fatalf("stderr batch notification lines = %d, want one:\n%s", count, stderr.String())
}
if !strings.Contains(stderr.String(), `notificationStatus="failed"`) || !strings.Contains(stderr.String(), `notificationError="notify report three_day: upload failed"`) {
t.Fatalf("stderr missing failure notification fields:\n%s", stderr.String())
if !strings.Contains(stderr.String(), `batchNotification status="succeeded"`) || !strings.Contains(stderr.String(), `runId="batch-distributor-run"`) || !strings.Contains(stderr.String(), `pipelineId="weatherreporter"`) {
t.Fatalf("stderr missing batch notification details:\n%s", stderr.String())
}
if strings.Contains(stderr.String(), "notificationStatus") || strings.Contains(stderr.String(), "notificationRunId") {
t.Fatalf("stderr includes per-report notification fields:\n%s", stderr.String())
}
}
@@ -265,13 +277,14 @@ func TestBatchOutputDoesNotExposeSecretLikeNotificationErrors(t *testing.T) {
Batch: app.BatchMorning,
Total: 1,
Failed: 1,
Notification: &app.BatchNotificationResult{
Status: "failed",
Error: "notify batch morning: upload failed: [redacted]",
},
Reports: []app.BatchReportResult{
{
ReportID: "daily",
Status: "failed",
Error: "notify report daily: upload failed: [redacted]",
NotificationStatus: "failed",
NotificationError: "notify report daily: upload failed: [redacted]",
ReportID: "daily",
Status: "succeeded",
},
},
}
@@ -293,6 +306,74 @@ func TestBatchOutputDoesNotExposeSecretLikeNotificationErrors(t *testing.T) {
}
}
func TestRunLogsIncludeSkippedBatchNotification(t *testing.T) {
result := &app.BatchResult{
Batch: app.BatchMorning,
Total: 2,
Succeeded: 1,
Failed: 1,
Notification: &app.BatchNotificationResult{
Status: "skipped",
Reason: "one or more reports failed",
},
Reports: []app.BatchReportResult{
{ReportID: "today", Status: "succeeded", OutputPath: "/tmp/today.md"},
{ReportID: "tomorrow", Status: "failed", Error: "render failed"},
},
}
var stderr bytes.Buffer
var stdout bytes.Buffer
if err := writeJSON(&stdout, result); err != nil {
t.Fatalf("writeJSON() error = %v", err)
}
writeRunLogs(&stderr, result)
var decoded app.BatchResult
if err := json.Unmarshal(stdout.Bytes(), &decoded); err != nil {
t.Fatalf("decode batch JSON: %v\n%s", err, stdout.String())
}
if decoded.Notification == nil || decoded.Notification.Status != "skipped" || decoded.Notification.Reason != "one or more reports failed" {
t.Fatalf("top-level notification = %#v, want skipped notification", decoded.Notification)
}
if count := strings.Count(stderr.String(), "batchNotification "); count != 1 {
t.Fatalf("stderr batch notification lines = %d, want one:\n%s", count, stderr.String())
}
if !strings.Contains(stderr.String(), `batchNotification status="skipped" reason="one or more reports failed"`) {
t.Fatalf("stderr missing skipped batch notification:\n%s", stderr.String())
}
}
func TestRunLogsDoNotRepeatBatchNotificationErrorPerReport(t *testing.T) {
result := &app.BatchResult{
Batch: app.BatchEvening,
Total: 1,
Succeeded: 1,
Failed: 1,
Notification: &app.BatchNotificationResult{
Status: "failed",
Error: "notify batch evening: upload failed",
},
Reports: []app.BatchReportResult{
{ReportID: "tomorrow", Status: "succeeded", OutputPath: "/tmp/tomorrow.md"},
},
}
var stderr bytes.Buffer
writeRunLogs(&stderr, result)
if count := strings.Count(stderr.String(), "batchNotification "); count != 1 {
t.Fatalf("stderr batch notification lines = %d, want one:\n%s", count, stderr.String())
}
if count := strings.Count(stderr.String(), "notify batch evening: upload failed"); count != 1 {
t.Fatalf("stderr batch notification error occurrences = %d, want one:\n%s", count, stderr.String())
}
reportLine := firstLineWithPrefix(stderr.String(), "report=tomorrow ")
if strings.Contains(reportLine, "notify batch evening") || strings.Contains(reportLine, "notificationError") {
t.Fatalf("report line repeats batch notification error:\n%s", reportLine)
}
}
func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
fixture := newCLIFixture(t, writeFakeScriptorium)
outputDir := fixture.path("copies")
@@ -321,20 +402,22 @@ func TestRunEveningUsesOutputDirectoryAndSummary(t *testing.T) {
}
}
func TestRunEveningReportsNotificationSuccess(t *testing.T) {
func TestRunEveningReportsOmitsPerReportNotification(t *testing.T) {
server := dailyServer(t)
var uploadCount int
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if r.URL.Path == "/runs/distributor-run-1" {
if r.URL.Path == "/runs/batch-distributor-run" {
w.Header().Set("Content-Type", "application/json")
_, _ = w.Write([]byte(`{"run_id":"distributor-run-1","pipeline_id":"weatherreporter.tomorrow","status":"succeeded","report":{"actions":[{"action":"replace_older"}]}}`))
_, _ = w.Write([]byte(`{"run_id":"batch-distributor-run","pipeline_id":"weatherreporter","status":"succeeded","report":{"actions":[{"action":"replace_older"}]}}`))
return
}
if r.URL.Path != "/v1/pipelines/weatherreporter.tomorrow/upload" {
if r.URL.Path != "/v1/pipelines/weatherreporter/upload" {
http.NotFound(w, r)
return
}
uploadCount++
w.WriteHeader(http.StatusAccepted)
_, _ = w.Write([]byte(`{"run_id":"distributor-run-1","status":"accepted"}`))
_, _ = w.Write([]byte(`{"run_id":"batch-distributor-run","status":"accepted"}`))
}))
t.Cleanup(distributorServer.Close)
tempDir := t.TempDir()
@@ -361,29 +444,39 @@ func TestRunEveningReportsNotificationSuccess(t *testing.T) {
if len(summary.Reports) != 1 {
t.Fatalf("reports = %#v, want one report", summary.Reports)
}
if summary.Reports[0].NotificationStatus != "succeeded" || summary.Reports[0].NotificationRunID != "distributor-run-1" || summary.Reports[0].NotificationPipelineID != "weatherreporter.tomorrow" {
t.Fatalf("notification fields = %#v", summary.Reports[0])
if uploadCount != 1 {
t.Fatalf("batch upload count = %d, want 1", uploadCount)
}
if !strings.Contains(stderr.String(), `notificationStatus="succeeded"`) || !strings.Contains(stderr.String(), `notificationRunId="distributor-run-1"`) {
t.Fatalf("stderr missing notification fields:\n%s", stderr.String())
if summary.Notification == nil || summary.Notification.Status != "succeeded" || summary.Notification.RunID != "batch-distributor-run" {
t.Fatalf("batch notification = %#v, want succeeded batch notification", summary.Notification)
}
if count := strings.Count(stderr.String(), "batchNotification "); count != 1 {
t.Fatalf("stderr batch notification lines = %d, want one:\n%s", count, stderr.String())
}
if !strings.Contains(stderr.String(), `batchNotification status="succeeded"`) || !strings.Contains(stderr.String(), `runId="batch-distributor-run"`) {
t.Fatalf("stderr missing batch notification success:\n%s", stderr.String())
}
if summary.Reports[0].NotificationStatus != "" || summary.Reports[0].NotificationRunID != "" || summary.Reports[0].NotificationPipelineID != "" || summary.Reports[0].NotificationError != "" || summary.Reports[0].NotificationPath != "" {
t.Fatalf("notification fields = %#v, want empty per-report notification fields", summary.Reports[0])
}
if strings.Contains(stderr.String(), "notificationStatus") || strings.Contains(stderr.String(), "notificationRunId") {
t.Fatalf("stderr includes per-report notification fields:\n%s", stderr.String())
}
if strings.Contains(stdout.String(), "cli-secret-token") || strings.Contains(stderr.String(), "cli-secret-token") {
t.Fatalf("output contains token value\nstdout=%s\nstderr=%s", stdout.String(), stderr.String())
}
}
func TestRunEveningReportsNotificationFailureWithoutToken(t *testing.T) {
func TestRunEveningReportsDoesNotRequirePerReportDistributorToken(t *testing.T) {
server := dailyServer(t)
distributorServer := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusBadRequest)
_, _ = w.Write([]byte(`{"error":"rejected cli-secret-token","retryable":false}`))
t.Fatalf("unexpected distributor request %s", r.URL.Path)
}))
t.Cleanup(distributorServer.Close)
tempDir := t.TempDir()
scriptoriumPath := writeFakeScriptorium(t, tempDir)
workspaceRoot := filepath.Join(tempDir, "workspace")
configPath := writeTestConfigWithDistributor(t, server, scriptoriumPath, workspaceRoot, distributorServer.URL)
t.Setenv("CLI_DISTRIBUTOR_TOKEN", "cli-secret-token")
configPath := writeTestConfigWithDisabledBatchDistributor(t, server, scriptoriumPath, workspaceRoot, distributorServer.URL)
var stdout bytes.Buffer
var stderr bytes.Buffer
runner := Runner{Clock: fixedClock()}
@@ -392,26 +485,22 @@ func TestRunEveningReportsNotificationFailureWithoutToken(t *testing.T) {
"run", "evening",
"--config", configPath,
}, &stdout, &stderr)
if err == nil {
t.Fatal("Run() error = nil, want notification failure")
if err != nil {
t.Fatalf("Run() error = %v", err)
}
var summary app.BatchResult
if decodeErr := json.Unmarshal(stdout.Bytes(), &summary); decodeErr != nil {
t.Fatalf("decode summary: %v\n%s", decodeErr, stdout.String())
}
if len(summary.Reports) != 1 || summary.Reports[0].NotificationStatus != "failed" {
t.Fatalf("summary reports = %#v, want failed notification", summary.Reports)
if len(summary.Reports) != 1 {
t.Fatalf("summary reports = %#v, want one report", summary.Reports)
}
for _, output := range []string{stdout.String(), stderr.String(), err.Error()} {
if strings.Contains(output, "cli-secret-token") {
t.Fatalf("output contains token value:\n%s", output)
}
if summary.Notification != nil {
t.Fatalf("batch notification = %#v, want omitted when disabled", summary.Notification)
}
for _, output := range []string{stdout.String(), stderr.String()} {
if !strings.Contains(output, "[redacted]") {
t.Fatalf("output missing redaction marker:\n%s", output)
}
if summary.Reports[0].NotificationStatus != "" || summary.Reports[0].NotificationError != "" {
t.Fatalf("notification fields = %#v, want empty per-report notification fields", summary.Reports[0])
}
}
@@ -1163,6 +1252,12 @@ func writeTestConfigWithDistributor(t *testing.T, server *httptest.Server, scrip
return writeConfigFile(t, configBody)
}
func writeTestConfigWithDisabledBatchDistributor(t *testing.T, server *httptest.Server, scriptoriumPath string, workspaceRoot string, distributorEndpoint string) string {
t.Helper()
configBody := "weather_api:\n base_url: " + server.URL + "/\n timezone: America/Chicago\nscriptorium:\n binary: " + scriptoriumPath + "\nworkspace:\n root: " + workspaceRoot + "\nnotify:\n distributor:\n enabled: true\n endpoint: " + distributorEndpoint + "\n token_env: CLI_DISTRIBUTOR_TOKEN\n pipeline_id_template: weatherreporter.{artifact_group}\n batch:\n enabled: false\n"
return writeConfigFile(t, configBody)
}
func writeWorkspaceConfig(t *testing.T, workspaceRoot string) string {
t.Helper()
return writeConfigFile(t, "workspace:\n root: "+workspaceRoot+"\n")
@@ -1191,6 +1286,15 @@ func noArtifacts(t *testing.T, root string, parts ...string) {
}
}
func firstLineWithPrefix(text string, prefix string) string {
for _, line := range strings.Split(text, "\n") {
if strings.HasPrefix(line, prefix) {
return line
}
}
return ""
}
func assertFileContains(t *testing.T, path string, want string) {
t.Helper()
data, err := os.ReadFile(path)

View File

@@ -58,15 +58,23 @@ type NotifyConfig struct {
}
type DistributorNotifyConfig struct {
Enabled bool `yaml:"enabled"`
Endpoint string `yaml:"endpoint"`
TokenEnv string `yaml:"token_env"`
Timeout time.Duration `yaml:"timeout"`
FailurePolicy NotifyFailurePolicy `yaml:"failure_policy"`
PipelineIDTemplate string `yaml:"pipeline_id_template"`
BundleIDTemplate string `yaml:"bundle_id_template"`
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
ReportPathTemplates []string `yaml:"report_path_templates"`
Enabled bool `yaml:"enabled"`
Endpoint string `yaml:"endpoint"`
TokenEnv string `yaml:"token_env"`
Timeout time.Duration `yaml:"timeout"`
FailurePolicy NotifyFailurePolicy `yaml:"failure_policy"`
PipelineIDTemplate string `yaml:"pipeline_id_template"`
BundleIDTemplate string `yaml:"bundle_id_template"`
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
ReportPathTemplates []string `yaml:"report_path_templates"`
Batch DistributorBatchNotifyConfig `yaml:"batch"`
}
type DistributorBatchNotifyConfig struct {
Enabled bool `yaml:"enabled"`
PipelineIDTemplate string `yaml:"pipeline_id_template"`
BundleIDTemplate string `yaml:"bundle_id_template"`
IdempotencyKeyTemplate string `yaml:"idempotency_key_template"`
}
type MissingSourceConfig struct {

View File

@@ -63,6 +63,18 @@ func TestDefaults(t *testing.T) {
if strings.Join(cfg.Notify.Distributor.ReportPathTemplates, "\n") != strings.Join(wantReportPaths, "\n") {
t.Fatalf("Notify.Distributor.ReportPathTemplates = %#v, want %#v", cfg.Notify.Distributor.ReportPathTemplates, wantReportPaths)
}
if !cfg.Notify.Distributor.Batch.Enabled {
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
}
if cfg.Notify.Distributor.Batch.PipelineIDTemplate != "weatherreporter" {
t.Fatalf("Notify.Distributor.Batch.PipelineIDTemplate = %q, want weatherreporter", cfg.Notify.Distributor.Batch.PipelineIDTemplate)
}
if cfg.Notify.Distributor.Batch.BundleIDTemplate != "weatherreporter.{location_id}.{batch}" {
t.Fatalf("Notify.Distributor.Batch.BundleIDTemplate = %q, want default", cfg.Notify.Distributor.Batch.BundleIDTemplate)
}
if cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate != "{bundle_id}.{batch_run_id}" {
t.Fatalf("Notify.Distributor.Batch.IdempotencyKeyTemplate = %q, want default", cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate)
}
if cfg.MissingSource.Default != MissingSourceWarn {
t.Fatalf("MissingSource.Default = %q, want warn", cfg.MissingSource.Default)
}
@@ -74,7 +86,7 @@ func TestLoadExampleConfig(t *testing.T) {
t.Fatalf("LoadFile() error = %v", err)
}
if cfg.WeatherAPI.BaseURL != "https://weather.api.rakestrawhome.com/" {
if cfg.WeatherAPI.BaseURL != "https://weather.api.example.com/" {
t.Fatalf("BaseURL = %q, want configured example URL", cfg.WeatherAPI.BaseURL)
}
if cfg.WeatherAPI.Timeout != 15*time.Second {
@@ -92,6 +104,18 @@ func TestLoadExampleConfig(t *testing.T) {
if len(cfg.Notify.Distributor.ReportPathTemplates) != 1 {
t.Fatalf("ReportPathTemplates = %#v, want example archive path", cfg.Notify.Distributor.ReportPathTemplates)
}
if !cfg.Notify.Distributor.Batch.Enabled {
t.Fatalf("Notify.Distributor.Batch.Enabled = false, want true")
}
if cfg.Notify.Distributor.Batch.PipelineIDTemplate != "weatherreporter" {
t.Fatalf("Batch PipelineIDTemplate = %q, want weatherreporter", cfg.Notify.Distributor.Batch.PipelineIDTemplate)
}
if cfg.Notify.Distributor.Batch.BundleIDTemplate != "weatherreporter.{location_id}.{batch}" {
t.Fatalf("Batch BundleIDTemplate = %q, want example batch bundle template", cfg.Notify.Distributor.Batch.BundleIDTemplate)
}
if cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate != "{bundle_id}.{batch_run_id}" {
t.Fatalf("Batch IdempotencyKeyTemplate = %q, want example batch idempotency template", cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate)
}
overrides, err := cfg.ReportModuleOverrides()
if err != nil {
t.Fatalf("ReportModuleOverrides() error = %v", err)
@@ -782,6 +806,18 @@ func TestDisabledDistributorNotifyAcceptsOmittedFields(t *testing.T) {
}
}
func TestDisabledDistributorNotifyAcceptsMalformedBatchTemplates(t *testing.T) {
cfg := Defaults()
cfg.Notify.Distributor.Enabled = false
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "{unknown}"
cfg.Notify.Distributor.Batch.BundleIDTemplate = "{unknown}"
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{unknown}"
if err := Validate(cfg); err != nil {
t.Fatalf("Validate() error = %v", err)
}
}
func TestEnabledDistributorNotifyValidation(t *testing.T) {
tests := []struct {
name string
@@ -906,6 +942,109 @@ func TestEnabledDistributorNotifyValidation(t *testing.T) {
}
}
func TestEnabledDistributorBatchNotifyValidation(t *testing.T) {
tests := []struct {
name string
mutate func(*Config)
wantErr string
}{
{
name: "PipelineTemplateEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.PipelineIDTemplate = ""
},
wantErr: "notify.distributor.batch.pipeline_id_template",
},
{
name: "PipelineTemplateUnknown",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "{report_id}"
},
wantErr: "notify.distributor.batch.pipeline_id_template",
},
{
name: "PipelineTemplateRenderedEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.PipelineIDTemplate = " "
},
wantErr: "notify.distributor.batch.pipeline_id_template",
},
{
name: "BundleTemplateEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.BundleIDTemplate = ""
},
wantErr: "notify.distributor.batch.bundle_id_template",
},
{
name: "BundleTemplateUnknown",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.BundleIDTemplate = "{run_id}"
},
wantErr: "notify.distributor.batch.bundle_id_template",
},
{
name: "BundleTemplateRenderedEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.BundleIDTemplate = " "
},
wantErr: "notify.distributor.batch.bundle_id_template",
},
{
name: "IdempotencyTemplateEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = ""
},
wantErr: "notify.distributor.batch.idempotency_key_template",
},
{
name: "IdempotencyTemplateUnknown",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{report_id}"
},
wantErr: "notify.distributor.batch.idempotency_key_template",
},
{
name: "IdempotencyTemplateRenderedEmpty",
mutate: func(cfg *Config) {
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = " "
},
wantErr: "notify.distributor.batch.idempotency_key_template",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
cfg := Defaults()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
tt.mutate(&cfg)
err := Validate(cfg)
if err == nil {
t.Fatal("Validate() error = nil, want error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
})
}
}
func TestDisabledDistributorBatchNotifySkipsBatchTemplateValidation(t *testing.T) {
cfg := Defaults()
cfg.Notify.Distributor.Enabled = true
cfg.Notify.Distributor.PipelineIDTemplate = "weatherreporter.{artifact_group}"
cfg.Notify.Distributor.Batch.Enabled = false
cfg.Notify.Distributor.Batch.PipelineIDTemplate = "{unknown}"
cfg.Notify.Distributor.Batch.BundleIDTemplate = "{unknown}"
cfg.Notify.Distributor.Batch.IdempotencyKeyTemplate = "{unknown}"
if err := Validate(cfg); err != nil {
t.Fatalf("Validate() error = %v", err)
}
}
func TestDistributorTemplateRendering(t *testing.T) {
values := DistributorTemplateValues{
LocationID: "home",
@@ -962,6 +1101,69 @@ func TestDistributorTemplateRendering(t *testing.T) {
}
}
func TestDistributorBatchTemplateRendering(t *testing.T) {
values := DistributorBatchTemplateValues{
LocationID: "home",
Batch: "evening",
BatchRunID: "20260617T235037.642224552Z_evening",
BatchStartedDate: "2026-06-17",
}
bundleID, err := RenderDistributorBatchBundleID("weatherreporter.{location_id}.{batch}", values)
if err != nil {
t.Fatalf("RenderDistributorBatchBundleID() error = %v", err)
}
if bundleID != "weatherreporter.home.evening" {
t.Fatalf("bundleID = %q, want batch bundle ID", bundleID)
}
values.BundleID = bundleID
pipelineID, err := RenderDistributorBatchPipelineID("weatherreporter", values)
if err != nil {
t.Fatalf("RenderDistributorBatchPipelineID() error = %v", err)
}
if pipelineID != "weatherreporter" {
t.Fatalf("pipelineID = %q, want weatherreporter", pipelineID)
}
idempotencyKey, err := RenderDistributorBatchIdempotencyKey("{bundle_id}.{batch_run_id}", values)
if err != nil {
t.Fatalf("RenderDistributorBatchIdempotencyKey() error = %v", err)
}
if idempotencyKey != "weatherreporter.home.evening.20260617T235037.642224552Z_evening" {
t.Fatalf("idempotencyKey = %q, want batch retry key", idempotencyKey)
}
bundleID, err = RenderDistributorBatchBundleID("weatherreporter.{batch_started_date}.{batch}", values)
if err != nil {
t.Fatalf("RenderDistributorBatchBundleID() with date error = %v", err)
}
if bundleID != "weatherreporter.2026-06-17.evening" {
t.Fatalf("bundleID = %q, want date-aware batch bundle ID", bundleID)
}
}
func TestDistributorBatchTemplateRejectsUnknownAndMalformedVariables(t *testing.T) {
tests := []struct {
name string
template string
}{
{name: "Unknown", template: "{report_id}"},
{name: "Unclosed", template: "{batch"},
{name: "Unopened", template: "batch}"},
{name: "Empty", template: "{}"},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
_, err := RenderDistributorBatchBundleID(tt.template, DistributorBatchTemplateValues{})
if err == nil {
t.Fatal("RenderDistributorBatchBundleID() error = nil, want error")
}
})
}
}
func TestDistributorTemplateRejectsUnknownAndMalformedVariables(t *testing.T) {
tests := []struct {
name string

View File

@@ -34,6 +34,12 @@ func Defaults() Config {
ReportPathTemplates: []string{
"{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md",
},
Batch: DistributorBatchNotifyConfig{
Enabled: true,
PipelineIDTemplate: "weatherreporter",
BundleIDTemplate: "weatherreporter.{location_id}.{batch}",
IdempotencyKeyTemplate: "{bundle_id}.{batch_run_id}",
},
},
},
MissingSource: MissingSourceConfig{

View File

@@ -21,6 +21,14 @@ type DistributorTemplateValues struct {
BundleID string
}
type DistributorBatchTemplateValues struct {
LocationID string
Batch string
BatchRunID string
BatchStartedDate string
BundleID string
}
var distributorTemplateVariables = map[string]struct{}{
"location_id": {},
"report_id": {},
@@ -52,6 +60,23 @@ var distributorIdempotencyTemplateVariables = map[string]struct{}{
var distributorPipelineTemplateVariables = distributorIdempotencyTemplateVariables
var distributorBatchTemplateVariables = map[string]struct{}{
"location_id": {},
"batch": {},
"batch_run_id": {},
"batch_started_date": {},
}
var distributorBatchIdempotencyTemplateVariables = map[string]struct{}{
"location_id": {},
"batch": {},
"batch_run_id": {},
"batch_started_date": {},
"bundle_id": {},
}
var distributorBatchPipelineTemplateVariables = distributorBatchTemplateVariables
func RenderDistributorBundleID(template string, values DistributorTemplateValues) (string, error) {
return renderDistributorTemplate("notify.distributor.bundle_id_template", template, values, distributorTemplateVariables)
}
@@ -71,6 +96,39 @@ func RenderDistributorIdempotencyKey(template string, values DistributorTemplate
return renderDistributorTemplate("notify.distributor.idempotency_key_template", template, values, distributorIdempotencyTemplateVariables)
}
func RenderDistributorBatchBundleID(template string, values DistributorBatchTemplateValues) (string, error) {
rendered, err := renderDistributorBatchTemplate("notify.distributor.batch.bundle_id_template", template, values, distributorBatchTemplateVariables)
if err != nil {
return "", err
}
if strings.TrimSpace(rendered) == "" {
return "", fmt.Errorf("notify.distributor.batch.bundle_id_template renders an empty bundle id")
}
return rendered, nil
}
func RenderDistributorBatchPipelineID(template string, values DistributorBatchTemplateValues) (string, error) {
rendered, err := renderDistributorBatchTemplate("notify.distributor.batch.pipeline_id_template", template, values, distributorBatchPipelineTemplateVariables)
if err != nil {
return "", err
}
if strings.TrimSpace(rendered) == "" {
return "", fmt.Errorf("notify.distributor.batch.pipeline_id_template renders an empty pipeline id")
}
return rendered, nil
}
func RenderDistributorBatchIdempotencyKey(template string, values DistributorBatchTemplateValues) (string, error) {
rendered, err := renderDistributorBatchTemplate("notify.distributor.batch.idempotency_key_template", template, values, distributorBatchIdempotencyTemplateVariables)
if err != nil {
return "", err
}
if strings.TrimSpace(rendered) == "" {
return "", fmt.Errorf("notify.distributor.batch.idempotency_key_template renders an empty idempotency key")
}
return rendered, nil
}
func RenderDistributorReportPaths(templates []string, values DistributorTemplateValues) ([]string, error) {
if len(templates) == 0 {
return nil, fmt.Errorf("notify.distributor.report_path_templates must contain at least one entry")
@@ -100,6 +158,11 @@ func validateDistributorTemplate(name, template string, allowed map[string]struc
return err
}
func validateDistributorBatchTemplate(name, template string, allowed map[string]struct{}) error {
_, err := renderDistributorBatchTemplate(name, template, DistributorBatchTemplateValues{}, allowed)
return err
}
func renderDistributorTemplate(name, template string, values DistributorTemplateValues, allowed map[string]struct{}) (string, error) {
var rendered strings.Builder
for i := 0; i < len(template); {
@@ -128,6 +191,34 @@ func renderDistributorTemplate(name, template string, values DistributorTemplate
return rendered.String(), nil
}
func renderDistributorBatchTemplate(name, template string, values DistributorBatchTemplateValues, allowed map[string]struct{}) (string, error) {
var rendered strings.Builder
for i := 0; i < len(template); {
switch template[i] {
case '{':
end := strings.IndexByte(template[i+1:], '}')
if end < 0 {
return "", fmt.Errorf("%s contains an unclosed template variable", name)
}
variable := template[i+1 : i+1+end]
if variable == "" {
return "", fmt.Errorf("%s contains an empty template variable", name)
}
if _, ok := allowed[variable]; !ok {
return "", fmt.Errorf("%s contains unknown template variable %q", name, variable)
}
rendered.WriteString(distributorBatchTemplateValue(variable, values))
i += end + 2
case '}':
return "", fmt.Errorf("%s contains an unopened template variable", name)
default:
rendered.WriteByte(template[i])
i++
}
}
return rendered.String(), nil
}
func distributorTemplateValue(variable string, values DistributorTemplateValues) string {
switch variable {
case "location_id":
@@ -159,6 +250,23 @@ func distributorTemplateValue(variable string, values DistributorTemplateValues)
}
}
func distributorBatchTemplateValue(variable string, values DistributorBatchTemplateValues) string {
switch variable {
case "location_id":
return values.LocationID
case "batch":
return values.Batch
case "batch_run_id":
return values.BatchRunID
case "batch_started_date":
return values.BatchStartedDate
case "bundle_id":
return values.BundleID
default:
return ""
}
}
func ValidateDistributorReportPath(name, path string) error {
if path == "" {
return fmt.Errorf("%s renders an empty path", name)

View File

@@ -148,10 +148,56 @@ func validateDistributorNotify(cfg DistributorNotifyConfig) error {
if _, err := RenderDistributorReportPaths(cfg.ReportPathTemplates, values); err != nil {
return err
}
if err := validateDistributorBatchNotify(cfg.Batch); err != nil {
return err
}
return nil
}
func validateDistributorBatchNotify(cfg DistributorBatchNotifyConfig) error {
if !cfg.Enabled {
return nil
}
if cfg.PipelineIDTemplate == "" {
return fmt.Errorf("notify.distributor.batch.pipeline_id_template is required when enabled")
}
if err := validateDistributorBatchTemplate("notify.distributor.batch.pipeline_id_template", cfg.PipelineIDTemplate, distributorBatchPipelineTemplateVariables); err != nil {
return err
}
if cfg.BundleIDTemplate == "" {
return fmt.Errorf("notify.distributor.batch.bundle_id_template is required when enabled")
}
if err := validateDistributorBatchTemplate("notify.distributor.batch.bundle_id_template", cfg.BundleIDTemplate, distributorBatchTemplateVariables); err != nil {
return err
}
if cfg.IdempotencyKeyTemplate == "" {
return fmt.Errorf("notify.distributor.batch.idempotency_key_template is required when enabled")
}
if err := validateDistributorBatchTemplate("notify.distributor.batch.idempotency_key_template", cfg.IdempotencyKeyTemplate, distributorBatchIdempotencyTemplateVariables); err != nil {
return err
}
values := DistributorBatchTemplateValues{
LocationID: "location",
Batch: "morning",
BatchRunID: "20260529T100000.000000000Z_morning",
BatchStartedDate: "2026-05-29",
}
bundleID, err := RenderDistributorBatchBundleID(cfg.BundleIDTemplate, values)
if err != nil {
return err
}
values.BundleID = bundleID
if _, err := RenderDistributorBatchPipelineID(cfg.PipelineIDTemplate, values); err != nil {
return err
}
if _, err := RenderDistributorBatchIdempotencyKey(cfg.IdempotencyKeyTemplate, values); err != nil {
return err
}
return nil
}
func validatePolicy(name string, policy MissingSourcePolicy) error {
switch policy {
case MissingSourceError, MissingSourceWarn, MissingSourceNone:

View File

@@ -141,6 +141,33 @@ func (s *FilesystemStore) SaveDistributorNotification(_ context.Context, resolve
}, artifact)
}
func (s *FilesystemStore) BatchDistributorNotificationPath(ref BatchDistributorNotificationRef) (string, error) {
if s == nil {
return "", fmt.Errorf("state store is required")
}
if err := validateBatchNotificationRef(ref); err != nil {
return "", err
}
localDate := ref.StartedAt.In(ref.Location).Format("2006-01-02")
return s.join(s.notificationsDir, "batches", ref.Batch, localDate, ref.BatchRunID+".distributor.json"), nil
}
func (s *FilesystemStore) SaveBatchDistributorNotification(_ context.Context, ref BatchDistributorNotificationRef, artifact BatchDistributorNotificationArtifact) (string, error) {
path, err := s.BatchDistributorNotificationPath(ref)
if err != nil {
return "", err
}
if artifact.SchemaVersion == "" {
artifact.SchemaVersion = BatchDistributorNotificationSchemaVersion
}
artifact.Batch = ref.Batch
artifact.BatchRunID = ref.BatchRunID
if err := fileutil.WriteJSONAtomic(path, artifact); err != nil {
return "", err
}
return path, nil
}
func (s *FilesystemStore) SaveGeneratedTextRaw(_ context.Context, resolved report.Resolved, data []byte) (string, error) {
return s.saveResolvedBytes(resolved, func(paths ArtifactPaths) string {
return paths.GeneratedTextRaw
@@ -461,6 +488,35 @@ func validateRelativeDir(name string, value string) error {
return nil
}
func validateBatchNotificationRef(ref BatchDistributorNotificationRef) error {
if err := validatePathSegment("batch kind", ref.Batch); err != nil {
return err
}
if err := validatePathSegment("batch run id", ref.BatchRunID); err != nil {
return err
}
if ref.StartedAt.IsZero() {
return fmt.Errorf("batch started time is required")
}
if ref.Location == nil {
return fmt.Errorf("batch location is required")
}
return nil
}
func validatePathSegment(name string, value string) error {
if strings.TrimSpace(value) == "" {
return fmt.Errorf("%s is required", name)
}
if strings.ContainsAny(value, `/\`) {
return fmt.Errorf("%s must not contain path separators", name)
}
if value == "." || value == ".." {
return fmt.Errorf("%s must be a safe path segment", name)
}
return nil
}
func readJSON(path string, target any) error {
data, err := os.ReadFile(path)
if err != nil {

View File

@@ -69,6 +69,193 @@ func TestDailyPathsUseRunIDValidDateDisambiguator(t *testing.T) {
}
}
func TestBatchDistributorNotificationPathUsesWorkspaceBatchDateAndRunID(t *testing.T) {
store := newTestStore(t)
location := mustLoadStateLocation(t, "America/Chicago")
startedAt := time.Date(2026, 6, 18, 3, 30, 0, 123456789, time.UTC)
path, err := store.BatchDistributorNotificationPath(BatchDistributorNotificationRef{
Batch: "evening",
BatchRunID: "20260618T033000.123456789Z_evening",
StartedAt: startedAt,
Location: location,
})
if err != nil {
t.Fatalf("BatchDistributorNotificationPath() error = %v", err)
}
want := filepath.Join("notifications", "batches", "evening", "2026-06-17", "20260618T033000.123456789Z_evening.distributor.json")
if !strings.Contains(path, want) {
t.Fatalf("path = %q, want component %q", path, want)
}
if !strings.HasPrefix(path, store.root) {
t.Fatalf("path = %q, want workspace root prefix %q", path, store.root)
}
}
func TestSaveBatchDistributorNotificationRoundTrip(t *testing.T) {
store := newTestStore(t)
location := mustLoadStateLocation(t, "America/Chicago")
startedAt := time.Date(2026, 6, 17, 12, 0, 0, 0, time.UTC)
bundleCreated := startedAt.Add(2 * time.Second)
attemptedAt := startedAt.Add(3 * time.Second)
acceptedAt := startedAt.Add(4 * time.Second)
finishedAt := startedAt.Add(5 * time.Second)
ref := BatchDistributorNotificationRef{
Batch: "morning",
BatchRunID: "20260617T120000.000000000Z_morning",
StartedAt: startedAt,
Location: location,
}
path, err := store.SaveBatchDistributorNotification(context.Background(), ref, BatchDistributorNotificationArtifact{
AttemptedAt: attemptedAt,
Endpoint: "https://distributor.example.test",
PipelineID: "weatherreporter",
BundleID: "weatherreporter.home.morning",
IdempotencyKey: "weatherreporter.home.morning.20260617T120000.000000000Z_morning",
BundleCreated: bundleCreated,
Reports: []BatchDistributorNotificationReportArtifact{
{
ReportID: report.Today,
RunID: "20260617T120000.000000000Z_today",
SourcePath: "/workspace/reports/today/20260617T120000.000000000Z_today.md",
BundlePaths: []string{"2026-06-17/today/report.md"},
},
{
ReportID: report.Daily,
RunID: "20260617T120000.000000000Z_daily_2026-06-19",
SourcePath: "/workspace/reports/daily/20260617T120000.000000000Z_daily_2026-06-19.md",
BundlePaths: []string{"2026-06-19/daily/report.md"},
},
},
Status: "failed",
Upload: &DistributorUploadResult{
RunID: "distributor-run",
Status: "accepted",
},
RunStatus: &DistributorRunStatus{
RunID: "distributor-run",
PipelineID: "weatherreporter",
Status: "failed",
AcceptedAt: acceptedAt,
FinishedAt: &finishedAt,
Report: json.RawMessage(`{"actions":[{"action":"failed"}]}`),
Error: "destination conflict",
},
StatusError: "status lookup failed",
Error: "batch upload failed",
})
if err != nil {
t.Fatalf("SaveBatchDistributorNotification() error = %v", err)
}
wantPath := filepath.Join("notifications", "batches", "morning", "2026-06-17", "20260617T120000.000000000Z_morning.distributor.json")
if !strings.Contains(path, wantPath) {
t.Fatalf("path = %q, want component %q", path, wantPath)
}
data, err := os.ReadFile(path)
if err != nil {
t.Fatalf("read batch notification: %v", err)
}
var artifact BatchDistributorNotificationArtifact
if err := json.Unmarshal(data, &artifact); err != nil {
t.Fatalf("decode batch notification: %v", err)
}
if artifact.SchemaVersion != BatchDistributorNotificationSchemaVersion {
t.Fatalf("SchemaVersion = %q, want %q", artifact.SchemaVersion, BatchDistributorNotificationSchemaVersion)
}
if artifact.Batch != "morning" || artifact.BatchRunID != ref.BatchRunID {
t.Fatalf("artifact batch identity = %q/%q, want ref values", artifact.Batch, artifact.BatchRunID)
}
if artifact.Endpoint != "https://distributor.example.test" || artifact.PipelineID != "weatherreporter" || artifact.BundleID != "weatherreporter.home.morning" || artifact.IdempotencyKey == "" {
t.Fatalf("artifact identity = %#v, want distributor identity", artifact)
}
if len(artifact.Reports) != 2 || artifact.Reports[0].ReportID != report.Today || strings.Join(artifact.Reports[1].BundlePaths, ",") != "2026-06-19/daily/report.md" {
t.Fatalf("Reports = %#v, want included report records", artifact.Reports)
}
if artifact.Upload == nil || artifact.Upload.RunID != "distributor-run" {
t.Fatalf("Upload = %#v, want accepted upload result", artifact.Upload)
}
if artifact.RunStatus == nil || artifact.RunStatus.Status != "failed" || !strings.Contains(string(artifact.RunStatus.Report), "failed") || artifact.RunStatus.FinishedAt == nil {
t.Fatalf("RunStatus = %#v, want failed run status with raw report", artifact.RunStatus)
}
if artifact.StatusError != "status lookup failed" || artifact.Error != "batch upload failed" {
t.Fatalf("errors = %q/%q, want persisted error fields", artifact.StatusError, artifact.Error)
}
}
func TestBatchDistributorNotificationPathRejectsInvalidIdentity(t *testing.T) {
store := newTestStore(t)
location := mustLoadStateLocation(t, "America/Chicago")
valid := BatchDistributorNotificationRef{
Batch: "morning",
BatchRunID: "20260617T120000.000000000Z_morning",
StartedAt: time.Date(2026, 6, 17, 12, 0, 0, 0, time.UTC),
Location: location,
}
tests := []struct {
name string
mutate func(*BatchDistributorNotificationRef)
wantErr string
}{
{
name: "Batch",
mutate: func(ref *BatchDistributorNotificationRef) {
ref.Batch = ""
},
wantErr: "batch kind is required",
},
{
name: "BatchSeparator",
mutate: func(ref *BatchDistributorNotificationRef) {
ref.Batch = "../morning"
},
wantErr: "batch kind must not contain path separators",
},
{
name: "BatchRunID",
mutate: func(ref *BatchDistributorNotificationRef) {
ref.BatchRunID = ""
},
wantErr: "batch run id is required",
},
{
name: "BatchRunIDSeparator",
mutate: func(ref *BatchDistributorNotificationRef) {
ref.BatchRunID = "nested/run"
},
wantErr: "batch run id must not contain path separators",
},
{
name: "StartedAt",
mutate: func(ref *BatchDistributorNotificationRef) {
ref.StartedAt = time.Time{}
},
wantErr: "batch started time is required",
},
{
name: "Location",
mutate: func(ref *BatchDistributorNotificationRef) {
ref.Location = nil
},
wantErr: "batch location is required",
},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
ref := valid
tt.mutate(&ref)
_, err := store.BatchDistributorNotificationPath(ref)
if err == nil {
t.Fatal("BatchDistributorNotificationPath() error = nil, want error")
}
if !strings.Contains(err.Error(), tt.wantErr) {
t.Fatalf("error = %q, want %q", err.Error(), tt.wantErr)
}
})
}
}
func TestGeneratedTextArtifactPathsUseSnapshotTree(t *testing.T) {
store := newTestStore(t)
tests := []struct {
@@ -595,6 +782,15 @@ func newTestStore(t *testing.T) *FilesystemStore {
return store
}
func mustLoadStateLocation(t *testing.T, name string) *time.Location {
t.Helper()
location, err := time.LoadLocation(name)
if err != nil {
t.Fatalf("LoadLocation(%q) error = %v", name, err)
}
return location
}
func resolveDailyAt(t *testing.T, value string) report.Resolved {
t.Helper()
return resolveDailyForDateAt(t, value, value)

View File

@@ -17,6 +17,7 @@ type Store interface {
SaveDataPackage(context.Context, report.Resolved, promptinput.Package) (string, error)
SavePreflight(context.Context, report.Resolved, PreflightArtifact) (string, error)
SaveDistributorNotification(context.Context, report.Resolved, DistributorNotificationArtifact) (string, error)
SaveBatchDistributorNotification(context.Context, BatchDistributorNotificationRef, BatchDistributorNotificationArtifact) (string, error)
SaveGeneratedTextRaw(context.Context, report.Resolved, []byte) (string, error)
SaveGeneratedTextResult(context.Context, report.Resolved, any) (string, error)
SaveGeneratedText(context.Context, report.Resolved, []byte) (string, error)
@@ -45,6 +46,14 @@ type PreflightArtifact struct {
}
const DistributorNotificationSchemaVersion = "weatherreporter.distributor_notification.v1"
const BatchDistributorNotificationSchemaVersion = "weatherreporter.batch_distributor_notification.v1"
type BatchDistributorNotificationRef struct {
Batch string
BatchRunID string
StartedAt time.Time
Location *time.Location
}
type DistributorNotificationArtifact struct {
SchemaVersion string `json:"schemaVersion"`
@@ -80,3 +89,28 @@ type DistributorRunStatus struct {
Report json.RawMessage `json:"report,omitempty"`
Error string `json:"error,omitempty"`
}
type BatchDistributorNotificationArtifact struct {
SchemaVersion string `json:"schemaVersion"`
Batch string `json:"batch"`
BatchRunID string `json:"batchRunId"`
AttemptedAt time.Time `json:"attemptedAt"`
Endpoint string `json:"endpoint"`
PipelineID string `json:"pipelineId,omitempty"`
BundleID string `json:"bundleId,omitempty"`
IdempotencyKey string `json:"idempotencyKey,omitempty"`
BundleCreated time.Time `json:"bundleCreated,omitempty"`
Reports []BatchDistributorNotificationReportArtifact `json:"includedReports,omitempty"`
Status string `json:"status"`
Upload *DistributorUploadResult `json:"upload,omitempty"`
RunStatus *DistributorRunStatus `json:"runStatus,omitempty"`
StatusError string `json:"statusError,omitempty"`
Error string `json:"error,omitempty"`
}
type BatchDistributorNotificationReportArtifact struct {
ReportID report.ID `json:"reportId"`
RunID string `json:"runId"`
SourcePath string `json:"sourcePath"`
BundlePaths []string `json:"bundlePaths"`
}