361 lines
12 KiB
Markdown
361 lines
12 KiB
Markdown
# Batch Distributor Upload Roadmap
|
|
|
|
## Purpose
|
|
|
|
This roadmap defines the target behavior for distributor notification from
|
|
`weatherreporter run morning` and `weatherreporter run evening`.
|
|
|
|
Batch commands should publish the generated weather reports as one coherent
|
|
distributor source bundle. The batch upload should be atomic from
|
|
weatherreporter's point of view: either every planned report is successfully
|
|
generated and the whole bundle is uploaded, or distributor is not notified.
|
|
|
|
Single-report `weatherreporter generate ...` commands should keep the existing
|
|
per-report distributor notification behavior.
|
|
|
|
## Current Problem
|
|
|
|
Per-report uploads in one batch can create multiple distributor source bundles
|
|
with the same logical destination scope. Depending on configured bundle IDs,
|
|
created timestamps, distributor merge mode, and destination state, distributor
|
|
may reject later uploads as conflicting with an earlier bundle from the same
|
|
batch.
|
|
|
|
The motivating failure was:
|
|
|
|
```text
|
|
destination source has same id and created time but different digest
|
|
```
|
|
|
|
Unique per-report RunIDs reduce one collision class, but they do not provide
|
|
batch-level publication semantics. A batch-level upload is the cleaner
|
|
long-term boundary when the downstream pipeline is intended to publish a
|
|
coherent set of weatherreporter outputs.
|
|
|
|
## Target Behavior
|
|
|
|
- `generate <report>` uploads one distributor bundle for that single generated
|
|
report when distributor notification is enabled.
|
|
- `run morning` and `run evening` generate all planned reports first.
|
|
- If collection, planning, report generation, Scriptorium, template rendering,
|
|
metadata persistence, or output-copy persistence fails for any report, the
|
|
batch distributor upload is skipped for the whole batch.
|
|
- If every planned report succeeds, weatherreporter uploads one distributor
|
|
bundle containing every managed Markdown report from the batch.
|
|
- The batch bundle maps each managed Markdown source path to one or more
|
|
bundle-relative paths rendered from the report's definition, valid period,
|
|
RunID, and configured distributor path templates.
|
|
- Batch upload failure marks the batch as failed and makes the command return a
|
|
nonzero aggregate error, but it does not retroactively mark individual report
|
|
generation as failed.
|
|
- Distributor status polling and debug artifact capture remain required for the
|
|
batch upload, just as they are for per-report uploads today.
|
|
- Optional `--out-dir` copies remain operator conveniences and are never used as
|
|
distributor upload sources.
|
|
|
|
## Locked Decisions
|
|
|
|
- Batch notification is all-or-nothing. If one report fails, skip the
|
|
distributor upload for the entire batch.
|
|
- The initial batch upload uses one distributor pipeline for the whole batch.
|
|
- Do not implement report-type-specific routing inside a single batch upload.
|
|
- Keep distributor package types confined to `internal/adapters/distributor`.
|
|
- Keep single-report notification behavior in place for `generate` commands.
|
|
- Do not implement batch-level publication by asking distributor to scan the
|
|
weatherreporter workspace.
|
|
- Do not introduce a manifest/resume system, workflow engine, plugin system, or
|
|
distributor-specific CLI flags.
|
|
|
|
## Batch Artifact Path
|
|
|
|
Batch notification should have its own debug artifact path rather than being
|
|
attached to one arbitrary report.
|
|
|
|
Target layout:
|
|
|
|
```text
|
|
workspace/
|
|
notifications/
|
|
batches/
|
|
morning/
|
|
2026-06-17/
|
|
20260617T120000.123456789Z_morning.distributor.json
|
|
evening/
|
|
2026-06-17/
|
|
20260617T230000.123456789Z_evening.distributor.json
|
|
```
|
|
|
|
The batch notification artifact should include:
|
|
|
|
- schema version;
|
|
- batch kind;
|
|
- batch run ID;
|
|
- attempted time;
|
|
- endpoint;
|
|
- pipeline ID;
|
|
- bundle ID;
|
|
- idempotency key;
|
|
- bundle created timestamp;
|
|
- included report records, each with report ID, RunID, managed source path, and
|
|
rendered bundle paths;
|
|
- accepted distributor run ID and upload status;
|
|
- latest distributor run status;
|
|
- raw distributor run report JSON when available;
|
|
- status lookup error when available;
|
|
- redacted notification error when upload or downstream run processing fails.
|
|
|
|
This makes batch notification a first-class batch artifact and keeps future
|
|
debugging or inspection behavior straightforward.
|
|
|
|
## Batch JSON Shape
|
|
|
|
Batch notification should be represented as a top-level object on `BatchResult`.
|
|
|
|
Target shape:
|
|
|
|
```json
|
|
{
|
|
"batch": "evening",
|
|
"startedAt": "2026-06-17T18:50:37.642224552-05:00",
|
|
"finishedAt": "2026-06-17T18:51:04.000000000-05:00",
|
|
"total": 3,
|
|
"succeeded": 3,
|
|
"failed": 0,
|
|
"notification": {
|
|
"status": "succeeded",
|
|
"runId": "weatherreporter.20260617T235104Z.77d8f75f",
|
|
"pipelineId": "weatherreporter",
|
|
"bundleId": "weatherreporter.home.evening",
|
|
"idempotencyKey": "weatherreporter.home.evening.20260617T235037.642224552Z_evening",
|
|
"path": "/var/lib/weatherreporter/notifications/batches/evening/2026-06-17/20260617T235037.642224552Z_evening.distributor.json",
|
|
"includedReports": [
|
|
{
|
|
"reportId": "tomorrow",
|
|
"runId": "20260617T235037.642224552Z_tomorrow",
|
|
"sourcePath": "/var/lib/weatherreporter/reports/tomorrow/20260617T235037.642224552Z_tomorrow.md",
|
|
"bundlePaths": ["tomorrow/index.md"]
|
|
},
|
|
{
|
|
"reportId": "daily",
|
|
"runId": "20260617T235037.642224552Z_daily_2026-06-23",
|
|
"sourcePath": "/var/lib/weatherreporter/reports/daily/20260617T235037.642224552Z_daily_2026-06-23.md",
|
|
"bundlePaths": ["archive/2026-06-23/daily/20260617T235037.642224552Z_daily_2026-06-23.md"]
|
|
}
|
|
]
|
|
},
|
|
"reports": []
|
|
}
|
|
```
|
|
|
|
When distributor is disabled, omit the notification object. When notification is
|
|
skipped because a report failed, include:
|
|
|
|
```json
|
|
"notification": {
|
|
"status": "skipped",
|
|
"reason": "one or more reports failed"
|
|
}
|
|
```
|
|
|
|
Do not duplicate the same batch notification status into every report item.
|
|
Per-report notification fields should remain meaningful for single-report
|
|
notification paths.
|
|
|
|
## Configuration Model
|
|
|
|
Use the existing `notify.distributor` block for both single-report and batch
|
|
notification, but add batch-specific identity templates so operators can keep
|
|
single-report and batch source identity separate.
|
|
|
|
Target config shape:
|
|
|
|
```yaml
|
|
notify:
|
|
distributor:
|
|
enabled: false
|
|
endpoint: https://distributor.example.com
|
|
token_env: DISTRIBUTOR_UPLOAD_TOKEN
|
|
timeout: 30s
|
|
failure_policy: error
|
|
pipeline_id_template: "weatherreporter.{report_id}"
|
|
bundle_id_template: "weatherreporter.{location_id}.{report_id}"
|
|
idempotency_key_template: "{bundle_id}.{run_id}"
|
|
report_path_templates:
|
|
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
|
|
batch:
|
|
enabled: true
|
|
pipeline_id_template: "weatherreporter"
|
|
bundle_id_template: "weatherreporter.{location_id}.{batch}"
|
|
idempotency_key_template: "{bundle_id}.{batch_run_id}"
|
|
```
|
|
|
|
Rules:
|
|
|
|
- `notify.distributor.enabled=false` disables both single-report and batch
|
|
notification.
|
|
- `notify.distributor.batch.enabled=true` makes run commands use the new
|
|
batch-level upload behavior.
|
|
- `notify.distributor.batch.enabled=false` disables distributor notification
|
|
for run commands. It does not fall back to legacy per-report batch uploads.
|
|
- Existing report-level `report_path_templates` are reused to map each report
|
|
source into the batch bundle.
|
|
- Each report path template is rendered once per included report using that
|
|
report's normal distributor template values.
|
|
- Batch templates support at least `location_id`, `batch`, `batch_run_id`, and
|
|
`batch_started_date`.
|
|
- Batch idempotency templates may also use `bundle_id`.
|
|
- Batch-level `pipeline_id_template` does not use `report_id` because one
|
|
upload contains multiple reports.
|
|
- Batch bundle IDs are stable for a logical batch stream. The idempotency key,
|
|
not the bundle ID, identifies a specific retryable batch attempt.
|
|
|
|
## Batch Run Identity
|
|
|
|
Batch notification needs an app-level batch run ID for notification identity and
|
|
artifact naming.
|
|
|
|
Format:
|
|
|
|
```text
|
|
<timestamp>_<batch>
|
|
```
|
|
|
|
Examples:
|
|
|
|
```text
|
|
20260617T235037.642224552Z_morning
|
|
20260617T235037.642224552Z_evening
|
|
```
|
|
|
|
The timestamp is based on batch `StartedAt` in UTC with the same nanosecond
|
|
layout used by report RunIDs. This ID is separate from report RunIDs and should
|
|
not replace them.
|
|
|
|
## Bundle Construction
|
|
|
|
Batch uploads use one distributor adapter upload request with multiple file
|
|
mappings.
|
|
|
|
For each successful report:
|
|
|
|
1. Use `BatchReportResult.ReportPath` as the source path.
|
|
2. Render `notify.distributor.report_path_templates` using that report's
|
|
existing distributor template values.
|
|
3. Add one upload file mapping per rendered bundle path.
|
|
|
|
Validation should fail before upload if:
|
|
|
|
- no reports are included;
|
|
- any included report lacks a managed report path;
|
|
- any source path is empty;
|
|
- any bundle path is empty or invalid;
|
|
- two rendered bundle paths in the same batch collide.
|
|
|
|
Errors should include report ID, RunID, source path, and bundle path context
|
|
where useful. Token values must never appear in errors, JSON output, metadata,
|
|
or artifacts.
|
|
|
|
## App Workflow
|
|
|
|
Batch commands should still collect once, plan once, and generate reports
|
|
through the same report generation path used by single-report commands. The
|
|
batch workflow differs only at the notification boundary:
|
|
|
|
- per-report distributor notification is suppressed for `run morning` and
|
|
`run evening`;
|
|
- report generation continues after individual report failures, preserving
|
|
current batch behavior;
|
|
- batch notification is skipped if any report failed;
|
|
- one batch notification is attempted only when all planned reports succeeded;
|
|
- batch notification failure produces an aggregate batch error and a batch
|
|
notification debug artifact.
|
|
|
|
Suppressing per-report notification should be explicit app-layer behavior, not a
|
|
hidden side effect inside `GenerateReport`.
|
|
|
|
## Adapter Boundary
|
|
|
|
The existing distributor adapter already supports multiple file mappings through
|
|
its upload request. Batch uploads should reuse that adapter API rather than
|
|
introducing a second distributor client.
|
|
|
|
The app layer owns:
|
|
|
|
- batch report selection;
|
|
- batch upload identity rendering;
|
|
- source path selection;
|
|
- bundle path rendering;
|
|
- debug artifact persistence;
|
|
- batch JSON shape.
|
|
|
|
The adapter owns:
|
|
|
|
- distributor package type conversion;
|
|
- token lookup;
|
|
- upload client construction;
|
|
- upload request execution;
|
|
- status polling;
|
|
- upload/status error wrapping and redaction.
|
|
|
|
## State And Inspection
|
|
|
|
State should support saving batch notification artifacts without introducing a
|
|
full manifest/resume system.
|
|
|
|
Expected additions:
|
|
|
|
- batch-level artifact path calculation for distributor notification;
|
|
- save support for the batch distributor notification artifact;
|
|
- no required changes to existing report metadata artifact shape.
|
|
|
|
If linking is useful later, prefer adding the batch notification path to the
|
|
top-level batch JSON result first, then consider metadata links.
|
|
|
|
## CLI Output
|
|
|
|
Command syntax remains unchanged:
|
|
|
|
```text
|
|
weatherreporter run morning
|
|
weatherreporter run evening
|
|
```
|
|
|
|
Stderr should keep one compact line per report generation result, then add one
|
|
compact batch notification line when notification is attempted, skipped, or
|
|
failed. The same notification error should not be repeated on every report line.
|
|
|
|
Example:
|
|
|
|
```text
|
|
report=tomorrow status=succeeded output=""
|
|
report=daily status=succeeded output=""
|
|
batchNotification status=succeeded runId="weatherreporter.20260617T235104Z.77d8f75f"
|
|
batch=evening total=2 succeeded=2 failed=0
|
|
```
|
|
|
|
If report generation fails:
|
|
|
|
```text
|
|
report=daily status=failed error="..."
|
|
batchNotification status=skipped reason="one or more reports failed"
|
|
batch=evening total=2 succeeded=1 failed=1
|
|
```
|
|
|
|
## Documentation Boundary
|
|
|
|
Until implemented, this roadmap is the only place this behavior should be
|
|
described. After implementation, update the relevant implemented docs and keep
|
|
future extensions under `docs/roadmap/`.
|
|
|
|
## Deferred Work
|
|
|
|
- Batch-level durable resume or retry queues.
|
|
- Batch-level distributor status inspection command.
|
|
- Multiple distributor pipelines within one batch.
|
|
- Report-type-specific distributor routing inside batch uploads.
|
|
- Uploading data packages, metadata, render contexts, or notification artifacts
|
|
in the batch bundle.
|
|
- Distributor-specific CLI flags.
|
|
- Batch manifest or progress system.
|
|
- Changing distributor destination merge semantics from weatherreporter.
|