eric/weatherreporter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Document profile comparison workflow

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-02 06:10:15 +00:00
parent ccf6b66880
commit e0229d9c90
14 changed files with 322 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

70
docs/cli.md
View File

@@ -1,7 +1,8 @@
# Weatherreporter CLI
`weatherreporter` generates Markdown weather reports and runs report batches.
It has no command for inspecting prior runs or application-owned state.
`weatherreporter` generates Markdown weather reports, runs report batches, and
compares explicitly selected Promptkit profiles against one prepared report. It
has no command for inspecting prior runs or application-owned state.
## Shortest Useful Command
@@ -25,6 +26,7 @@ weatherreporter generate tomorrow [--config PATH] [--units VALUE] [--tz NAME] [-
weatherreporter generate hourly [--config PATH] [--units VALUE] [--tz NAME] [--out PATH] [--llm-debug-dir PATH] [--quiet]
weatherreporter run morning [--config PATH] [--units VALUE] [--tz NAME] [--out-dir PATH] [--llm-debug-dir PATH] [--quiet]
weatherreporter run evening [--config PATH] [--units VALUE] [--tz NAME] [--out-dir PATH] [--llm-debug-dir PATH] [--quiet]
weatherreporter compare REPORT --profile PROFILE --profile PROFILE [--config PATH] [--units VALUE] [--tz NAME] [--date YYYY-MM-DD] [--out-dir PATH] [--replace] [--llm-debug-dir PATH] [--quiet]
```
`weatherreporter --version` prints the version embedded in the executable.
@@ -38,10 +40,13 @@ builds report `development`.
| `generate tomorrow` | Uses the next local civil day and writes `tomorrow.md` by default. |
| `generate hourly` | Covers the next six hours in the effective report timezone and writes `hourly.md` by default. It does not accept `--date`, `--hours`, or `--duration`. |
| `run morning` and `run evening` | Run their defined report batches beneath the configured output directory, or the current directory when none is configured. `--out-dir` selects another directory. `--out` is not accepted. |
| `compare REPORT` | Accepts `daily`, `today`, `tomorrow`, or `hourly`. It requires at least two distinct, nonblank `--profile` values in their supplied order. Daily requires `--date`; Today accepts it optionally; Tomorrow and Hourly do not accept it. |
`generate` accepts the four report command names shown above. `run` accepts
only `morning` and `evening`. Batch membership and notification ordering are
described in the [operations guide](operations.md).
only `morning` and `evening`. `compare` always requires explicit profile
selection: `promptkit.profile` is not used as a comparison default. Batch
membership and notification ordering are described in the
[operations guide](operations.md).
## Output, Errors, And Quiet Mode
@@ -50,21 +55,25 @@ For `generate`, the report's default filename is placed beneath
PATH` selects one complete output file instead. A relative path is resolved
from the current directory; an absolute path is used as given. For a batch,
the configured directory has the same role and `--out-dir PATH` selects its
output directory instead. Successful summaries always report the resulting
absolute `outputPath` values. See the [configuration reference](config.md) for
the field's validation and path rules.
output directory instead. For `compare`, `--out-dir PATH` selects one exact
bundle directory; otherwise the report-derived comparison directory is placed
beneath the configured directory or current directory. `--replace` is required
to replace an existing nonempty recognized comparison bundle. See the
[configuration reference](config.md) for the field's validation and path rules
and the [comparison bundle contract](integrations/comparison-bundle.md) for the
bundle format.
Outputs are written atomically. A generation, rendering, write, or cancellation
failure before publication leaves an existing destination unchanged. A
notification failure occurs after publication, so the newly written output
remains available.
Action commands (`generate` and `run`) write a JSON summary to stdout unless
`--quiet` is set. `run` also writes compact per-report and batch status lines
to stderr. A pre-run error, such as an invalid flag, missing required argument,
or configuration-load failure, produces no partial JSON summary. When an
action fails after it has produced a result, its summary has `"status": "failed"`
and an `error` field.
Action commands (`generate`, `run`, and `compare`) write a JSON summary to
stdout unless `--quiet` is set. `run` also writes compact per-report and batch
status lines to stderr. A pre-run error, such as an invalid flag, missing
required argument, or configuration-load failure, produces no partial JSON
summary. When an action fails after it has produced a result, its summary has
`"status": "failed"` and an `error` field.
`--quiet` is supported by action commands only. It suppresses action summaries
and routine batch status output; it does not suppress command errors.
@@ -113,19 +122,41 @@ report=today status=succeeded output="/srv/weather/reports/today.md"
batch=morning total=2 succeeded=2 failed=0
```
### Compare Summary
A comparison summary contains these fields in this order: `command`,
`comparisonId`, `reportId`, `reportName`, `promptId`, `promptVersion`,
`promptHash`, `status`, `startedAt`, `finishedAt`, `timezone`, `validPeriod`,
`outputDirectory`, `manifestPath`, `dataPackagePath`, `total`, `succeeded`,
`failed`, `results`, and optional `error`. Published artifact paths and each
successful `results[].reportPath` are absolute. `results` preserves the
supplied profile order and each item contains `position`, `profileId`, optional
`backendId`, `modelName`, `status`, optional `validationStatus`, optional
`reportPath`, optional `llmDebugPath`, and optional safe `error`.
The comparison status is `succeeded` only when every selected profile succeeds
and the bundle is published. Individual profile failures still publish a
complete partial bundle and return a failed command result. Cancellation or a
failure before publication omits the artifact paths and returns a safe
top-level error; it does not expose provider diagnostics. See the
[comparison bundle contract](integrations/comparison-bundle.md) for durable
artifact fields and failure invariants.
## Flag Reference
| Flag | Accepted by | Meaning |
| --- | --- | --- |
| `-h`, `--help` | top level | Show help. |
| `--config PATH` | all commands | Load `PATH` instead of `/usr/local/etc/weatherreporter/config.yml`. |
| `--units VALUE` | `generate`, `run` | Override `weather_api.units` for this command. |
| `--tz NAME` | `generate`, `run` | Override `weather_api.timezone` for this command. |
| `--units VALUE` | `generate`, `run`, `compare` | Override `weather_api.units` for this command. |
| `--tz NAME` | `generate`, `run`, `compare` | Override `weather_api.timezone` for this command. |
| `--out PATH` | every `generate` command | Write the report to this complete file destination instead of the configured or current-directory default. |
| `--llm-debug-dir PATH` | every `generate` and `run` command | Write requested sensitive prompt diagnostics under this absolute path. |
| `--out-dir PATH` | `run morning`, `run evening` | Write batch reports beneath this directory instead of the configured or current-directory default. |
| `--quiet` | `generate`, `run` | Suppress action summaries and routine batch status output. |
| `--date YYYY-MM-DD` | `generate daily`, `generate today` | Required for Daily; optional for Today. |
| `--llm-debug-dir PATH` | every `generate`, `run`, and `compare` command | Write requested sensitive prompt diagnostics under this absolute path. |
| `--profile PROFILE` | `compare` | Select one explicit profile. Repeat at least twice with distinct, nonblank IDs. |
| `--out-dir PATH` | `run morning`, `run evening`, `compare` | Write batch reports beneath this directory, or select the exact comparison directory. |
| `--replace` | `compare` | Authorize replacement of a recognized nonempty comparison bundle. |
| `--quiet` | `generate`, `run`, `compare` | Suppress successful action output and routine batch status output. |
| `--date YYYY-MM-DD` | `generate daily`, `generate today`, `compare daily`, `compare today` | Required for Daily; optional for Today. |
Distributor notification is configured through `notify.distributor`; there are
no Distributor-specific CLI flags. See the [configuration reference](config.md).
@@ -138,4 +169,5 @@ weatherreporter generate today --out ./reports/today.md
weatherreporter generate hourly --out /srv/weather/hourly.md
weatherreporter generate today --llm-debug-dir /var/tmp/weatherreporter-debug
weatherreporter run morning --out-dir ./reports --llm-debug-dir /var/tmp/weatherreporter-debug
weatherreporter compare daily --date 2026-05-29 --profile weather-light --profile weather-balanced --out-dir ./comparison-daily-2026-05-29
```