Files

Eric Rakestraw 133f83f4ce Create a roadmap and implementation plan for batch distributor uploads

2026-06-17 15:31:36 -05:00

12 KiB

Raw Blame History

Batch Distributor Upload Roadmap

Purpose

This roadmap defines the target behavior for distributor notification from weatherreporter run morning and weatherreporter run evening.

Batch commands should publish the generated weather reports as one coherent distributor source bundle. The batch upload should be atomic from weatherreporter's point of view: either every planned report is successfully generated and the whole bundle is uploaded, or distributor is not notified.

Single-report weatherreporter generate ... commands should keep the existing per-report distributor notification behavior.

Current Problem

Per-report uploads in one batch can create multiple distributor source bundles with the same logical destination scope. Depending on configured bundle IDs, created timestamps, distributor merge mode, and destination state, distributor may reject later uploads as conflicting with an earlier bundle from the same batch.

The motivating failure was:

destination source has same id and created time but different digest

Unique per-report RunIDs reduce one collision class, but they do not provide batch-level publication semantics. A batch-level upload is the cleaner long-term boundary when the downstream pipeline is intended to publish a coherent set of weatherreporter outputs.

Target Behavior

generate <report> uploads one distributor bundle for that single generated report when distributor notification is enabled.
run morning and run evening generate all planned reports first.
If collection, planning, report generation, Scriptorium, template rendering, metadata persistence, or output-copy persistence fails for any report, the batch distributor upload is skipped for the whole batch.
If every planned report succeeds, weatherreporter uploads one distributor bundle containing every managed Markdown report from the batch.
The batch bundle maps each managed Markdown source path to one or more bundle-relative paths rendered from the report's definition, valid period, RunID, and configured distributor path templates.
Batch upload failure marks the batch as failed and makes the command return a nonzero aggregate error, but it does not retroactively mark individual report generation as failed.
Distributor status polling and debug artifact capture remain required for the batch upload, just as they are for per-report uploads today.
Optional --out-dir copies remain operator conveniences and are never used as distributor upload sources.

Locked Decisions

Batch notification is all-or-nothing. If one report fails, skip the distributor upload for the entire batch.
The initial batch upload uses one distributor pipeline for the whole batch.
Do not implement report-type-specific routing inside a single batch upload.
Keep distributor package types confined to internal/adapters/distributor.
Keep single-report notification behavior in place for generate commands.
Do not implement batch-level publication by asking distributor to scan the weatherreporter workspace.
Do not introduce a manifest/resume system, workflow engine, plugin system, or distributor-specific CLI flags.

Batch Artifact Path

Batch notification should have its own debug artifact path rather than being attached to one arbitrary report.

Target layout:

workspace/
  notifications/
    batches/
      morning/
        2026-06-17/
          20260617T120000.123456789Z_morning.distributor.json
      evening/
        2026-06-17/
          20260617T230000.123456789Z_evening.distributor.json

The batch notification artifact should include:

schema version;
batch kind;
batch run ID;
attempted time;
endpoint;
pipeline ID;
bundle ID;
idempotency key;
bundle created timestamp;
included report records, each with report ID, RunID, managed source path, and rendered bundle paths;
accepted distributor run ID and upload status;
latest distributor run status;
raw distributor run report JSON when available;
status lookup error when available;
redacted notification error when upload or downstream run processing fails.

This makes batch notification a first-class batch artifact and keeps future debugging or inspection behavior straightforward.

Batch JSON Shape

Batch notification should be represented as a top-level object on BatchResult.

Target shape:

{
  "batch": "evening",
  "startedAt": "2026-06-17T18:50:37.642224552-05:00",
  "finishedAt": "2026-06-17T18:51:04.000000000-05:00",
  "total": 3,
  "succeeded": 3,
  "failed": 0,
  "notification": {
    "status": "succeeded",
    "runId": "weatherreporter.20260617T235104Z.77d8f75f",
    "pipelineId": "weatherreporter",
    "bundleId": "weatherreporter.home.evening",
    "idempotencyKey": "weatherreporter.home.evening.20260617T235037.642224552Z_evening",
    "path": "/var/lib/weatherreporter/notifications/batches/evening/2026-06-17/20260617T235037.642224552Z_evening.distributor.json",
    "includedReports": [
      {
        "reportId": "tomorrow",
        "runId": "20260617T235037.642224552Z_tomorrow",
        "sourcePath": "/var/lib/weatherreporter/reports/tomorrow/20260617T235037.642224552Z_tomorrow.md",
        "bundlePaths": ["tomorrow/index.md"]
      },
      {
        "reportId": "daily",
        "runId": "20260617T235037.642224552Z_daily_2026-06-23",
        "sourcePath": "/var/lib/weatherreporter/reports/daily/20260617T235037.642224552Z_daily_2026-06-23.md",
        "bundlePaths": ["archive/2026-06-23/daily/20260617T235037.642224552Z_daily_2026-06-23.md"]
      }
    ]
  },
  "reports": []
}

When distributor is disabled, omit the notification object. When notification is skipped because a report failed, include:

"notification": {
  "status": "skipped",
  "reason": "one or more reports failed"
}

Do not duplicate the same batch notification status into every report item. Per-report notification fields should remain meaningful for single-report notification paths.

Configuration Model

Use the existing notify.distributor block for both single-report and batch notification, but add batch-specific identity templates so operators can keep single-report and batch source identity separate.

Target config shape:

notify:
  distributor:
    enabled: false
    endpoint: https://distributor.example.com
    token_env: DISTRIBUTOR_UPLOAD_TOKEN
    timeout: 30s
    failure_policy: error
    pipeline_id_template: "weatherreporter.{report_id}"
    bundle_id_template: "weatherreporter.{location_id}.{report_id}"
    idempotency_key_template: "{bundle_id}.{run_id}"
    report_path_templates:
      - "{valid_start_date}/{artifact_group}/{valid_start_date}-{artifact_group}-{run_id}.md"
    batch:
      enabled: true
      pipeline_id_template: "weatherreporter"
      bundle_id_template: "weatherreporter.{location_id}.{batch}"
      idempotency_key_template: "{bundle_id}.{batch_run_id}"

Rules:

notify.distributor.enabled=false disables both single-report and batch notification.
notify.distributor.batch.enabled=true makes run commands use the new batch-level upload behavior.
notify.distributor.batch.enabled=false disables distributor notification for run commands. It does not fall back to legacy per-report batch uploads.
Existing report-level report_path_templates are reused to map each report source into the batch bundle.
Each report path template is rendered once per included report using that report's normal distributor template values.
Batch templates support at least location_id, batch, batch_run_id, and batch_started_date.
Batch idempotency templates may also use bundle_id.
Batch-level pipeline_id_template does not use report_id because one upload contains multiple reports.
Batch bundle IDs are stable for a logical batch stream. The idempotency key, not the bundle ID, identifies a specific retryable batch attempt.

Batch Run Identity

Batch notification needs an app-level batch run ID for notification identity and artifact naming.

Format:

<timestamp>_<batch>

Examples:

20260617T235037.642224552Z_morning
20260617T235037.642224552Z_evening

The timestamp is based on batch StartedAt in UTC with the same nanosecond layout used by report RunIDs. This ID is separate from report RunIDs and should not replace them.

Bundle Construction

Batch uploads use one distributor adapter upload request with multiple file mappings.

For each successful report:

Use BatchReportResult.ReportPath as the source path.
Render notify.distributor.report_path_templates using that report's existing distributor template values.
Add one upload file mapping per rendered bundle path.

Validation should fail before upload if:

no reports are included;
any included report lacks a managed report path;
any source path is empty;
any bundle path is empty or invalid;
two rendered bundle paths in the same batch collide.

Errors should include report ID, RunID, source path, and bundle path context where useful. Token values must never appear in errors, JSON output, metadata, or artifacts.

App Workflow

Batch commands should still collect once, plan once, and generate reports through the same report generation path used by single-report commands. The batch workflow differs only at the notification boundary:

per-report distributor notification is suppressed for run morning and run evening;
report generation continues after individual report failures, preserving current batch behavior;
batch notification is skipped if any report failed;
one batch notification is attempted only when all planned reports succeeded;
batch notification failure produces an aggregate batch error and a batch notification debug artifact.

Suppressing per-report notification should be explicit app-layer behavior, not a hidden side effect inside GenerateReport.

Adapter Boundary

The existing distributor adapter already supports multiple file mappings through its upload request. Batch uploads should reuse that adapter API rather than introducing a second distributor client.

The app layer owns:

batch report selection;
batch upload identity rendering;
source path selection;
bundle path rendering;
debug artifact persistence;
batch JSON shape.

The adapter owns:

distributor package type conversion;
token lookup;
upload client construction;
upload request execution;
status polling;
upload/status error wrapping and redaction.

State And Inspection

State should support saving batch notification artifacts without introducing a full manifest/resume system.

Expected additions:

batch-level artifact path calculation for distributor notification;
save support for the batch distributor notification artifact;
no required changes to existing report metadata artifact shape.

If linking is useful later, prefer adding the batch notification path to the top-level batch JSON result first, then consider metadata links.

CLI Output

Command syntax remains unchanged:

weatherreporter run morning
weatherreporter run evening

Stderr should keep one compact line per report generation result, then add one compact batch notification line when notification is attempted, skipped, or failed. The same notification error should not be repeated on every report line.

Example:

report=tomorrow status=succeeded output=""
report=daily status=succeeded output=""
batchNotification status=succeeded runId="weatherreporter.20260617T235104Z.77d8f75f"
batch=evening total=2 succeeded=2 failed=0

If report generation fails:

report=daily status=failed error="..."
batchNotification status=skipped reason="one or more reports failed"
batch=evening total=2 succeeded=1 failed=1

Documentation Boundary

Until implemented, this roadmap is the only place this behavior should be described. After implementation, update the relevant implemented docs and keep future extensions under docs/roadmap/.

Deferred Work

Batch-level durable resume or retry queues.
Batch-level distributor status inspection command.
Multiple distributor pipelines within one batch.
Report-type-specific distributor routing inside batch uploads.
Uploading data packages, metadata, render contexts, or notification artifacts in the batch bundle.
Distributor-specific CLI flags.
Batch manifest or progress system.
Changing distributor destination merge semantics from weatherreporter.

12 KiB Raw Blame History