Create a roadmap and implementation plan for batch distributor uploads
This commit is contained in:
360
docs/roadmap/batch-distributor.md
Normal file
360
docs/roadmap/batch-distributor.md
Normal file
@@ -0,0 +1,360 @@
|
||||
# Batch Distributor Upload Roadmap
|
||||
|
||||
## Purpose
|
||||
|
||||
This roadmap defines the target behavior for distributor notification from
|
||||
`weatherreporter run morning` and `weatherreporter run evening`.
|
||||
|
||||
Batch commands should publish the generated weather reports as one coherent
|
||||
distributor source bundle. The batch upload should be atomic from
|
||||
weatherreporter's point of view: either every planned report is successfully
|
||||
generated and the whole bundle is uploaded, or distributor is not notified.
|
||||
|
||||
Single-report `weatherreporter generate ...` commands should keep the existing
|
||||
per-report distributor notification behavior.
|
||||
|
||||
## Current Problem
|
||||
|
||||
Per-report uploads in one batch can create multiple distributor source bundles
|
||||
with the same logical destination scope. Depending on configured bundle IDs,
|
||||
created timestamps, distributor merge mode, and destination state, distributor
|
||||
may reject later uploads as conflicting with an earlier bundle from the same
|
||||
batch.
|
||||
|
||||
The motivating failure was:
|
||||
|
||||
```text
|
||||
destination source has same id and created time but different digest
|
||||
```
|
||||
|
||||
Unique per-report RunIDs reduce one collision class, but they do not provide
|
||||
batch-level publication semantics. A batch-level upload is the cleaner
|
||||
long-term boundary when the downstream pipeline is intended to publish a
|
||||
coherent set of weatherreporter outputs.
|
||||
|
||||
## Target Behavior
|
||||
|
||||
- `generate <report>` uploads one distributor bundle for that single generated
|
||||
report when distributor notification is enabled.
|
||||
- `run morning` and `run evening` generate all planned reports first.
|
||||
- If collection, planning, report generation, Scriptorium, template rendering,
|
||||
metadata persistence, or output-copy persistence fails for any report, the
|
||||
batch distributor upload is skipped for the whole batch.
|
||||
- If every planned report succeeds, weatherreporter uploads one distributor
|
||||
bundle containing every managed Markdown report from the batch.
|
||||
- The batch bundle maps each managed Markdown source path to one or more
|
||||
bundle-relative paths rendered from the report's definition, valid period,
|
||||
RunID, and configured distributor path templates.
|
||||
- Batch upload failure marks the batch as failed and makes the command return a
|
||||
nonzero aggregate error, but it does not retroactively mark individual report
|
||||
generation as failed.
|
||||
- Distributor status polling and debug artifact capture remain required for the
|
||||
batch upload, just as they are for per-report uploads today.
|
||||
- Optional `--out-dir` copies remain operator conveniences and are never used as
|
||||
distributor upload sources.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
- Batch notification is all-or-nothing. If one report fails, skip the
|
||||
distributor upload for the entire batch.
|
||||
- The initial batch upload uses one distributor pipeline for the whole batch.
|
||||
- Do not implement report-type-specific routing inside a single batch upload.
|
||||
- Keep distributor package types confined to `internal/adapters/distributor`.
|
||||
- Keep single-report notification behavior in place for `generate` commands.
|
||||
- Do not implement batch-level publication by asking distributor to scan the
|
||||
weatherreporter workspace.
|
||||
- Do not introduce a manifest/resume system, workflow engine, plugin system, or
|
||||
distributor-specific CLI flags.
|
||||
|
||||
## Batch Artifact Path
|
||||
|
||||
Batch notification should have its own debug artifact path rather than being
|
||||
attached to one arbitrary report.
|
||||
|
||||
Target layout:
|
||||
|
||||
```text
|
||||
workspace/
|
||||
notifications/
|
||||
batches/
|
||||
morning/
|
||||
2026-06-17/
|
||||
20260617T120000.123456789Z_morning.distributor.json
|
||||
evening/
|
||||
2026-06-17/
|
||||
20260617T230000.123456789Z_evening.distributor.json
|
||||
```
|
||||
|
||||
The batch notification artifact should include:
|
||||
|
||||
- schema version;
|
||||
- batch kind;
|
||||
- batch run ID;
|
||||
- attempted time;
|
||||
- endpoint;
|
||||
- pipeline ID;
|
||||
- bundle ID;
|
||||
- idempotency key;
|
||||
- bundle created timestamp;
|
||||
- included report records, each with report ID, RunID, managed source path, and
|
||||
rendered bundle paths;
|
||||
- accepted distributor run ID and upload status;
|
||||
- latest distributor run status;
|
||||
- raw distributor run report JSON when available;
|
||||
- status lookup error when available;
|
||||
- redacted notification error when upload or downstream run processing fails.
|
||||
|
||||
This makes batch notification a first-class batch artifact and keeps future
|
||||
debugging or inspection behavior straightforward.
|
||||
|
||||
## Batch JSON Shape
|
||||
|
||||
Batch notification should be represented as a top-level object on `BatchResult`.
|
||||
|
||||
Target shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"batch": "evening",
|
||||
"startedAt": "2026-06-17T18:50:37.642224552-05:00",
|
||||
"finishedAt": "2026-06-17T18:51:04.000000000-05:00",
|
||||
"total": 3,
|
||||
"succeeded": 3,
|
||||
"failed": 0,
|
||||
"notification": {
|
||||
"status": "succeeded",
|
||||
"runId": "weatherreporter.20260617T235104Z.77d8f75f",
|
||||
"pipelineId": "weatherreporter",
|
||||
"bundleId": "weatherreporter.home.evening",
|
||||
"idempotencyKey": "weatherreporter.home.evening.20260617T235037.642224552Z_evening",
|
||||
"path": "/var/lib/weatherreporter/notifications/batches/evening/2026-06-17/20260617T235037.642224552Z_evening.distributor.json",
|
||||
"includedReports": [
|
||||
{
|
||||
"reportId": "tomorrow",
|
||||
"runId": "20260617T235037.642224552Z_tomorrow",
|
||||
"sourcePath": "/var/lib/weatherreporter/reports/tomorrow/20260617T235037.642224552Z_tomorrow.md",
|
||||
"bundlePaths": ["tomorrow/index.md"]
|
||||
},
|
||||
{
|
||||
"reportId": "daily",
|
||||
"runId": "20260617T235037.642224552Z_daily_2026-06-23",
|
||||
"sourcePath": "/var/lib/weatherreporter/reports/daily/20260617T235037.642224552Z_daily_2026-06-23.md",
|
||||
"bundlePaths": ["archive/2026-06-23/daily/20260617T235037.642224552Z_daily_2026-06-23.md"]
|
||||
}
|
||||
]
|
||||
},
|
||||
"reports": []
|
||||
}
|
||||
```
|
||||
|
||||
When distributor is disabled, omit the notification object. When notification is
|
||||
skipped because a report failed, include:
|
||||
|
||||
```json
|
||||
"notification": {
|
||||
"status": "skipped",
|
||||
"reason": "one or more reports failed"
|
||||
}
|
||||
```
|
||||
|
||||
Do not duplicate the same batch notification status into every report item.
|
||||
Per-report notification fields should remain meaningful for single-report
|
||||
notification paths.
|
||||
|
||||
## Configuration Model
|
||||
|
||||
Use the existing `notify.distributor` block for both single-report and batch
|
||||
notification, but add batch-specific identity templates so operators can keep
|
||||
single-report and batch source identity separate.
|
||||
|
||||
Target config shape:
|
||||
|
||||
```yaml
|
||||
notify:
|
||||
distributor:
|
||||
enabled: false
|
||||
endpoint: https://distributor.example.com
|
||||
token_env: DISTRIBUTOR_UPLOAD_TOKEN
|
||||
timeout: 30s
|
||||
failure_policy: error
|
||||
pipeline_id_template: "weatherreporter.{report_id}"
|
||||
bundle_id_template: "weatherreporter.{location_id}.{report_id}"
|
||||
idempotency_key_template: "{bundle_id}.{run_id}"
|
||||
report_path_templates:
|
||||
- "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
|
||||
batch:
|
||||
enabled: true
|
||||
pipeline_id_template: "weatherreporter"
|
||||
bundle_id_template: "weatherreporter.{location_id}.{batch}"
|
||||
idempotency_key_template: "{bundle_id}.{batch_run_id}"
|
||||
```
|
||||
|
||||
Rules:
|
||||
|
||||
- `notify.distributor.enabled=false` disables both single-report and batch
|
||||
notification.
|
||||
- `notify.distributor.batch.enabled=true` makes run commands use the new
|
||||
batch-level upload behavior.
|
||||
- `notify.distributor.batch.enabled=false` disables distributor notification
|
||||
for run commands. It does not fall back to legacy per-report batch uploads.
|
||||
- Existing report-level `report_path_templates` are reused to map each report
|
||||
source into the batch bundle.
|
||||
- Each report path template is rendered once per included report using that
|
||||
report's normal distributor template values.
|
||||
- Batch templates support at least `location_id`, `batch`, `batch_run_id`, and
|
||||
`batch_started_date`.
|
||||
- Batch idempotency templates may also use `bundle_id`.
|
||||
- Batch-level `pipeline_id_template` does not use `report_id` because one
|
||||
upload contains multiple reports.
|
||||
- Batch bundle IDs are stable for a logical batch stream. The idempotency key,
|
||||
not the bundle ID, identifies a specific retryable batch attempt.
|
||||
|
||||
## Batch Run Identity
|
||||
|
||||
Batch notification needs an app-level batch run ID for notification identity and
|
||||
artifact naming.
|
||||
|
||||
Format:
|
||||
|
||||
```text
|
||||
<timestamp>_<batch>
|
||||
```
|
||||
|
||||
Examples:
|
||||
|
||||
```text
|
||||
20260617T235037.642224552Z_morning
|
||||
20260617T235037.642224552Z_evening
|
||||
```
|
||||
|
||||
The timestamp is based on batch `StartedAt` in UTC with the same nanosecond
|
||||
layout used by report RunIDs. This ID is separate from report RunIDs and should
|
||||
not replace them.
|
||||
|
||||
## Bundle Construction
|
||||
|
||||
Batch uploads use one distributor adapter upload request with multiple file
|
||||
mappings.
|
||||
|
||||
For each successful report:
|
||||
|
||||
1. Use `BatchReportResult.ReportPath` as the source path.
|
||||
2. Render `notify.distributor.report_path_templates` using that report's
|
||||
existing distributor template values.
|
||||
3. Add one upload file mapping per rendered bundle path.
|
||||
|
||||
Validation should fail before upload if:
|
||||
|
||||
- no reports are included;
|
||||
- any included report lacks a managed report path;
|
||||
- any source path is empty;
|
||||
- any bundle path is empty or invalid;
|
||||
- two rendered bundle paths in the same batch collide.
|
||||
|
||||
Errors should include report ID, RunID, source path, and bundle path context
|
||||
where useful. Token values must never appear in errors, JSON output, metadata,
|
||||
or artifacts.
|
||||
|
||||
## App Workflow
|
||||
|
||||
Batch commands should still collect once, plan once, and generate reports
|
||||
through the same report generation path used by single-report commands. The
|
||||
batch workflow differs only at the notification boundary:
|
||||
|
||||
- per-report distributor notification is suppressed for `run morning` and
|
||||
`run evening`;
|
||||
- report generation continues after individual report failures, preserving
|
||||
current batch behavior;
|
||||
- batch notification is skipped if any report failed;
|
||||
- one batch notification is attempted only when all planned reports succeeded;
|
||||
- batch notification failure produces an aggregate batch error and a batch
|
||||
notification debug artifact.
|
||||
|
||||
Suppressing per-report notification should be explicit app-layer behavior, not a
|
||||
hidden side effect inside `GenerateReport`.
|
||||
|
||||
## Adapter Boundary
|
||||
|
||||
The existing distributor adapter already supports multiple file mappings through
|
||||
its upload request. Batch uploads should reuse that adapter API rather than
|
||||
introducing a second distributor client.
|
||||
|
||||
The app layer owns:
|
||||
|
||||
- batch report selection;
|
||||
- batch upload identity rendering;
|
||||
- source path selection;
|
||||
- bundle path rendering;
|
||||
- debug artifact persistence;
|
||||
- batch JSON shape.
|
||||
|
||||
The adapter owns:
|
||||
|
||||
- distributor package type conversion;
|
||||
- token lookup;
|
||||
- upload client construction;
|
||||
- upload request execution;
|
||||
- status polling;
|
||||
- upload/status error wrapping and redaction.
|
||||
|
||||
## State And Inspection
|
||||
|
||||
State should support saving batch notification artifacts without introducing a
|
||||
full manifest/resume system.
|
||||
|
||||
Expected additions:
|
||||
|
||||
- batch-level artifact path calculation for distributor notification;
|
||||
- save support for the batch distributor notification artifact;
|
||||
- no required changes to existing report metadata artifact shape.
|
||||
|
||||
If linking is useful later, prefer adding the batch notification path to the
|
||||
top-level batch JSON result first, then consider metadata links.
|
||||
|
||||
## CLI Output
|
||||
|
||||
Command syntax remains unchanged:
|
||||
|
||||
```text
|
||||
weatherreporter run morning
|
||||
weatherreporter run evening
|
||||
```
|
||||
|
||||
Stderr should keep one compact line per report generation result, then add one
|
||||
compact batch notification line when notification is attempted, skipped, or
|
||||
failed. The same notification error should not be repeated on every report line.
|
||||
|
||||
Example:
|
||||
|
||||
```text
|
||||
report=tomorrow status=succeeded output=""
|
||||
report=daily status=succeeded output=""
|
||||
batchNotification status=succeeded runId="weatherreporter.20260617T235104Z.77d8f75f"
|
||||
batch=evening total=2 succeeded=2 failed=0
|
||||
```
|
||||
|
||||
If report generation fails:
|
||||
|
||||
```text
|
||||
report=daily status=failed error="..."
|
||||
batchNotification status=skipped reason="one or more reports failed"
|
||||
batch=evening total=2 succeeded=1 failed=1
|
||||
```
|
||||
|
||||
## Documentation Boundary
|
||||
|
||||
Until implemented, this roadmap is the only place this behavior should be
|
||||
described. After implementation, update the relevant implemented docs and keep
|
||||
future extensions under `docs/roadmap/`.
|
||||
|
||||
## Deferred Work
|
||||
|
||||
- Batch-level durable resume or retry queues.
|
||||
- Batch-level distributor status inspection command.
|
||||
- Multiple distributor pipelines within one batch.
|
||||
- Report-type-specific distributor routing inside batch uploads.
|
||||
- Uploading data packages, metadata, render contexts, or notification artifacts
|
||||
in the batch bundle.
|
||||
- Distributor-specific CLI flags.
|
||||
- Batch manifest or progress system.
|
||||
- Changing distributor destination merge semantics from weatherreporter.
|
||||
Reference in New Issue
Block a user