Files

Eric Rakestraw b3637cddd6 Document batch distributor uploads

2026-06-17 21:11:28 +00:00

15 KiB

Raw Blame History

Weatherreporter Operations

This guide covers normal operation, generated artifacts, inspection, recovery, and operational caveats. For symptom-specific diagnosis, see Troubleshooting.

Normal Workflow

Generation commands:

weatherreporter generate daily --date 2026-05-29
weatherreporter generate today
weatherreporter generate tomorrow
weatherreporter generate hourly
weatherreporter generate three-day
weatherreporter generate weekend
weatherreporter generate storm --start 2026-05-29T18:00 --end 2026-05-30T06:00

Generation commands resolve a report period, collect a Weather API bundle, build a rich JSON module snapshot, build a curated YAML prompt input data package, run scriptorium render, and write managed artifacts under the configured workspace. Markdown-path reports then run scriptorium run directly to the managed Markdown report path.

generate daily, generate today, generate tomorrow, and generate hourly use the generated-text-template workflow. They run structured scriptorium run to raw GeneratedText JSON, validate the structured text, save a render context, and render the managed Markdown report from embedded templates. generate daily requires --date YYYY-MM-DD for the selected local civil day. generate today covers the selected or current local civil day. generate hourly covers the six-hour rolling period from generation time in the effective report timezone and is not included in run morning or run evening.

When distributor notification is enabled, weatherreporter uploads the managed Markdown report after report rendering succeeds and final metadata is saved. --out PATH writes an extra Markdown copy for generated reports; it is not used as the distributor upload source.

Batch commands:

weatherreporter run morning
weatherreporter run evening

run morning generates Today Report, Tomorrow Report, and a dated Daily Report for each later future local civil day with complete hourly forecast coverage. run evening generates Tomorrow Report and the same eligible future Daily reports. Future Daily expansion starts with the day after tomorrow. A Daily report is eligible only when the collected hourly forecast contains every hourly period for that local civil day; partial days are skipped. Batch commands collect weather data once before planning, and a collection failure stops the batch before any report is generated.

After planning succeeds, batch commands print a JSON summary to stdout, write compact per-report status lines to stderr, continue independent reports after one report fails, and return nonzero when any report failed. Batch commands do not upload each report independently. When distributor notification and batch notification are enabled, weatherreporter uploads one distributor bundle only after every planned report succeeds. If any report fails, the batch upload is skipped for the whole batch. --out-dir PATH writes extra Markdown copies using report default filenames such as today.md and tomorrow.md; dynamic Daily copies use daily-YYYY-MM-DD.md. These copies are not used as distributor upload sources.

Filesystem Layout

The default workspace root is workspace.

workspace/
  snapshots/
    daily/
      YYYY-MM-DD/
        <run_id>.modules.json
        <run_id>.metadata.json
        <run_id>.generated_text.raw.json
        <run_id>.generated_text.run.json
        <run_id>.generated_text.json
        <run_id>.render_context.json
    today/
      YYYY-MM-DD/
        <run_id>.modules.json
        <run_id>.metadata.json
        <run_id>.generated_text.raw.json
        <run_id>.generated_text.run.json
        <run_id>.generated_text.json
        <run_id>.render_context.json
    three-day/
      YYYY-MM-DD/
        <run_id>.modules.json
        <run_id>.metadata.json
    weekend/
      YYYY-MM-DD/
        <run_id>.modules.json
        <run_id>.metadata.json
    hourly/
      YYYY-MM-DD/
        <run_id>.modules.json
        <run_id>.metadata.json
        <run_id>.generated_text.raw.json
        <run_id>.generated_text.run.json
        <run_id>.generated_text.json
        <run_id>.render_context.json
    tomorrow/
      YYYY-MM-DD/
        <run_id>.modules.json
        <run_id>.metadata.json
        <run_id>.generated_text.raw.json
        <run_id>.generated_text.run.json
        <run_id>.generated_text.json
        <run_id>.render_context.json
    storm/
      YYYY-MM-DD/
        <run_id>.modules.json
        <run_id>.metadata.json
  data-packages/
    daily/
      YYYY-MM-DD/
        <run_id>.data_package.yaml
    today/
      YYYY-MM-DD/
        <run_id>.data_package.yaml
    three-day/
      YYYY-MM-DD/
        <run_id>.data_package.yaml
    weekend/
      YYYY-MM-DD/
        <run_id>.data_package.yaml
    hourly/
      YYYY-MM-DD/
        <run_id>.data_package.yaml
    tomorrow/
      YYYY-MM-DD/
        <run_id>.data_package.yaml
    storm/
      YYYY-MM-DD/
        <run_id>.data_package.yaml
  preflight/
    daily/
      YYYY-MM-DD/
        <run_id>.render.json
    today/
      YYYY-MM-DD/
        <run_id>.render.json
    three-day/
      YYYY-MM-DD/
        <run_id>.render.json
    weekend/
      YYYY-MM-DD/
        <run_id>.render.json
    hourly/
      YYYY-MM-DD/
        <run_id>.render.json
    tomorrow/
      YYYY-MM-DD/
        <run_id>.render.json
    storm/
      YYYY-MM-DD/
        <run_id>.render.json
  notifications/
    batches/
      morning/
        YYYY-MM-DD/
          <batch_run_id>.distributor.json
      evening/
        YYYY-MM-DD/
          <batch_run_id>.distributor.json
    daily/
      YYYY-MM-DD/
        <run_id>.distributor.json
    today/
      YYYY-MM-DD/
        <run_id>.distributor.json
    three-day/
      YYYY-MM-DD/
        <run_id>.distributor.json
    weekend/
      YYYY-MM-DD/
        <run_id>.distributor.json
    hourly/
      YYYY-MM-DD/
        <run_id>.distributor.json
    tomorrow/
      YYYY-MM-DD/
        <run_id>.distributor.json
    storm/
      YYYY-MM-DD/
        <run_id>.distributor.json
  reports/
    daily/
      <run_id>.md
    today/
      <run_id>.md
    three-day/
      <run_id>.md
    weekend/
      <run_id>.md
    hourly/
      <run_id>.md
    tomorrow/
      <run_id>.md
    storm/
      <run_id>.md

Managed artifact filenames use the RunID, so repeated runs for the same valid period do not overwrite each other.

RunID And Metadata

RunIDs are based on generation time plus report ID. Reports that can be generated more than once in a single command may append a report-specific disambiguator. Daily appends the local valid date so multiple dynamic Daily reports in one batch have distinct managed artifacts:

20260529T100000.123456789Z_daily_2026-05-31
20260529T100000.123456789Z_today

Batch notification RunIDs use the batch start timestamp plus the batch command name:

20260529T100000.123456789Z_morning
20260529T220000.123456789Z_evening

Each generated report writes metadata that links:

RunID, report ID, variant, and prompt ID
generation time, timezone, and valid period
source location, source hashes, and source warnings
module snapshot path
prompt input data package path
preflight output path
managed Markdown report path
generated text schema ID and generated-text artifact paths for generated-text-template reports
distributor notification debug artifact path, when notification is attempted

Batch summaries include report status, error text when applicable, valid period, and known artifact paths for each attempted report. Single-report notification fields on report items are empty for batch commands. When a batch notification is attempted, skipped, or fails, the summary includes one top-level notification object with fields such as status, reason, runId, pipelineId, bundleId, idempotencyKey, path, includedReports, and error.

Distributor Notification

Distributor notification is configured with notify.distributor and is disabled by default. For generate <report>, weatherreporter uploads the managed Markdown report path recorded in the report result and metadata. That single source file can be mapped to one or more configured bundle paths. By default, it is mapped to one dated report path. Extra copies written by --out or --out-dir are operator conveniences only.

For run morning and run evening, per-report notification is suppressed. If notify.distributor.enabled and notify.distributor.batch.enabled are both true, the batch uploads once after all reports finish successfully. The upload contains one file mapping set per included report. Each mapping uses the managed Markdown report as the source and renders report_path_templates using that report's normal template values. All rendered bundle paths across the batch must be unique. If any report fails, weatherreporter records a top-level notification status of skipped with reason one or more reports failed and does not call distributor. If batch notification is disabled, run commands do not fall back to per-report uploads.

The rendered pipeline ID selects the configured distributor http_upload workflow. The default bundle ID is a stable logical source identity derived from producer name, location ID, and report ID:

weatherreporter.{location_id}.{report_id}

The default single-report idempotency key appends RunID to the rendered bundle ID so each report generation has a distinct retry identity. The default bundle path uses the valid-period start date, artifact group, and RunID. Batch bundle IDs default to weatherreporter.{location_id}.{batch}, and batch idempotency keys default to {bundle_id}.{batch_run_id}. Distributor owns destination merge, retention, and derived snapshot behavior such as latest. For Daily, the default report ID and artifact group values are both daily, and the default output filename value is daily.md. For Today, the default report ID and artifact group values are both today, and the batch output filename value is today.md.

Single-report notification happens after final metadata save for generated reports. Batch notification happens after all planned reports have finished and only when all report generations succeeded. Collection, module snapshot, data-package, render preflight, Scriptorium run, generated-text validation, template rendering, and metadata-save failures do not trigger notification. A single-report notification failure fails that report. A batch notification failure makes the batch return nonzero and increments the aggregate failure count, but individual report items remain succeeded.

Each notification attempt writes a debug artifact under notifications/. Single-report artifacts live under notifications/<artifact_group>/<YYYY-MM-DD>/<run_id>.distributor.json. Batch artifacts live under notifications/batches/<batch>/<YYYY-MM-DD>/<batch_run_id>.distributor.json, where the date directory is the batch start date in the effective report timezone. The artifact records the rendered pipeline ID, bundle ID, idempotency key, managed source paths, bundle-relative paths, bundle created timestamp, accepted upload response, and the latest distributor run status response when available. Weatherreporter polls status until distributor reports succeeded or failed, or until the configured notification timeout expires. The run status includes the distributor status, error text, and raw run report JSON, which can show actions such as replace_older, skip_same, skip_destination_newer, or failed. Token values are not written.

Weatherreporter is responsible for selecting the managed Markdown report, constructing a source bundle, and submitting it to the configured distributor HTTP endpoint. Distributor remains responsible for destination routing, publication, and any downstream Markdown-to-HTML transformation. Distributor leaves destination files alone when they are not tracked by a newly uploaded bundle, so existing uploaded dated report paths can remain available.

Inspection

Inspection commands read existing workspace artifacts and emit JSON to stdout. They do not collect weather data or run scriptorium.

weatherreporter inspect reports --limit 10
weatherreporter inspect metadata RUN_ID
weatherreporter inspect modules RUN_ID
weatherreporter inspect data-package RUN_ID
weatherreporter inspect prior RUN_ID
weatherreporter inspect sources RUN_ID

Use inspect reports to find RunIDs and artifact paths. Use inspect metadata to see the artifact links recorded for a run. Use inspect modules to review the persisted ordered module snapshot with rich template-facing values, and inspect data-package to review the curated prompt package passed to Scriptorium. Use inspect prior to see the prior comparable snapshot selected for Recent Changes, or null when none exists. Use inspect sources to review source provenance and warnings without dumping full weather payloads.

Recent Changes

Recent Changes are computed from structured module snapshots, not rendered Markdown or YAML text.

Daily Report compares with prior Daily Report snapshots for the same valid local date. Today Report compares with prior Today Report snapshots for the same valid local date. Tomorrow Report compares with prior Tomorrow Report snapshots for the same valid local date. 3-Day Outlook compares with prior compatible 3-Day snapshots for the same valid local date. Weekend Outlook compares with prior compatible Weekend snapshots for the same weekend window. Hourly Report and Storm Report leave Recent Changes empty.

When no prior comparable snapshot exists, or no configured threshold is crossed, recentChanges.items is empty.

Recovery

A failed generation run may still leave useful artifacts:

If scriptorium render returns a result with a nonzero exit code, the preflight JSON and metadata are written for inspection.
If scriptorium run exits nonzero after writing a report, the managed report and metadata remain available.
Generated-text failures for Daily, Today, Tomorrow, and Hourly reports preserve available intermediate artifacts, such as the structured run result, raw generated-text JSON, validated generated text, and render context. Metadata links those paths when it can be safely written.
If single-report distributor notification fails, report artifacts and final metadata remain available, but the report command returns nonzero.
If batch distributor notification fails, report artifacts and final metadata remain available, the top-level batch notification links the debug artifact, and the batch command returns nonzero.
For batch commands, inspect the stdout JSON summary first, then inspect the artifact paths for each failed report or the top-level notification path.

For a bad report, start with:

weatherreporter inspect metadata RUN_ID
weatherreporter inspect sources RUN_ID
weatherreporter inspect modules RUN_ID
weatherreporter inspect data-package RUN_ID
weatherreporter inspect prior RUN_ID

Operational Caveats

The application uses one configured Weather API endpoint.
The application writes local filesystem state only.
The application does not implement resume, cleanup, archive, remote storage, daemon operation, or automatic storm monitoring.
Generated reports and Scriptorium stderr can contain sensitive operational context. Store workspace artifacts with appropriate filesystem permissions.

15 KiB Raw Blame History