From f364ce773d3e49bdce46b90a21ade9af2e0a4a2b Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sun, 2 Aug 2026 01:42:58 +0000 Subject: [PATCH] Complete configurable output directory implementation --- docs/roadmap/implementation.md | 377 +++++++++++++++++++++++++++ docs/roadmap/output-directory.md | 191 ++++++++++++++ docs/roadmap/profile-comparison.md | 397 +++++++++++++++++++++++++++++ 3 files changed, 965 insertions(+) create mode 100644 docs/roadmap/implementation.md create mode 100644 docs/roadmap/output-directory.md create mode 100644 docs/roadmap/profile-comparison.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..c589c1b --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,377 @@ +# Configurable Output Directory Implementation Plan + +Status: Complete. + +## Purpose And Authority + +This plan implements the accepted [Configurable Output Directory +Roadmap](output-directory.md). The feature roadmap is authoritative for user +intent, scope, policy choices, non-goals, and the desired end state. This +document owns implementation order, concrete changes, verification, and exit +gates. + +The target is one optional `output.directory` configuration value that selects +the ordinary publication directory for both single reports and batches while +preserving explicit CLI overrides, current defaults, stateless execution, +atomic publication, and notification ordering. + +This plan follows the repository's [architecture](../policy/architecture.md), +[documentation](../policy/documentation.md), and +[testing](../policy/testing.md) policies. + +## Implementation Rules + +1. Implement the stages in numeric order. Keep the repository buildable and + the default suite passing at every stage. +2. Use `GOWORK=off` for all Go validation so the sibling workspace cannot + supply unpublished dependency changes. +3. Keep this change within the configurable-output-directory scope. Do not + implement any part of the accepted + [LLM profile comparison roadmap](profile-comparison.md). +4. Preserve one operator-owned Markdown output per generated report. Do not + add a workspace, manifest, history, cache, retention policy, directory scan, + or managed cleanup behavior. +5. Preserve current report filenames, report periods, batch membership, + Promptkit inspection and execution, generated-text validation, rendering, + cancellation, atomic file publication, action summaries, and Distributor + notification ordering. +6. Keep CLI flag parsing and current-working-directory capture in + `internal/cli`, configuration loading and validation in `internal/config`, + and effective destination resolution and publication in `internal/app`. +7. Keep tests deterministic, offline, credential-free, and independent of + process-wide `os.Chdir` or machine-specific directories. Use `t.TempDir()` + and injected working directories. +8. Update current-state documentation only when the corresponding behavior is + implemented. Do not create release notes, select a release version, tag, or + publish a release in these stages. +9. Run `gofmt` on changed Go files and `git diff --check` before completing + every stage. Do not weaken unrelated tests or rewrite historical release + notes. + +## Locked Product And Path Decisions + +Implement exactly this optional configuration shape: + +```yaml +output: + directory: /var/lib/weatherreporter/reports +``` + +The following decisions are fixed: + +- Add `config.OutputConfig` with one `Directory string` field and add + `Config.Output` with YAML key `output`. Do not add other output settings. +- The built-in default is the empty string. Omitted and explicitly empty + values preserve current-directory behavior. +- A whitespace-only configured value is invalid. Preserve a nonblank value as + supplied during configuration loading; do not trim it into a different + filesystem path. +- Configuration loading performs no filesystem lookup and creates no + directory. Runtime output preflight owns path cleaning and filesystem + inspection. +- A relative configured directory resolves against the absolute working + directory captured for the invocation, never against the configuration-file + directory. There is no environment-variable or home-directory expansion. +- A nonexistent configured directory is valid and is created only as needed by + ordinary atomic report publication. +- An existing configured path that is not a directory, or cannot be inspected, + fails through the existing output-preflight boundary before prompt + inspection, weather collection, model execution, or output replacement. +- Single-report precedence is explicit `--out` path, then + `output.directory` plus the report's existing default filename, then the + working directory plus that filename. +- Batch precedence is explicit `--out-dir`, then `output.directory`, then the + working directory. +- An explicit `--out` is the complete file destination. A relative `--out` + remains relative to the invocation working directory and is never rebased + beneath `output.directory`. +- Configuration must still pass strict validation before any command runs. + After a valid configuration loads, explicit output flags completely ignore + the configured directory for runtime resolution and filesystem inspection. + They retain their current absolute and relative semantics. +- Successful result and summary paths remain cleaned absolute paths. Default + filenames and all output, cancellation, and notification safety guarantees + remain unchanged. + +## Stage 1: Add The Configuration Contract + +### Goal + +Make `output.directory` a strict, validated configuration value without +changing runtime destination behavior or current-state documentation yet. + +### Work + +1. In `internal/config/config.go`, add: + - `Output OutputConfig` to `Config` with YAML key `output`; and + - `OutputConfig` containing only `Directory string` with YAML key + `directory`. +2. In `internal/config/defaults.go`, initialize `Output.Directory` to the empty + string explicitly so the public default is visible beside the other + configuration defaults. +3. In `internal/config/validate.go`, reject `Output.Directory` only when it is + nonempty and `strings.TrimSpace` finds no non-whitespace character. Use an + actionable error that identifies `output.directory`. Do not clean, make + absolute, stat, create, or otherwise mutate the path during validation. +4. Rely on the existing `yaml.Decoder.KnownFields(true)` boundary for strict + nested decoding. Do not introduce custom YAML unmarshalling or a second + unknown-field mechanism for `OutputConfig`. +5. Add focused configuration tests that prove: + - the exact empty default; + - omission and explicit `directory: ""` are accepted; + - representative absolute and relative values load without mutation; + - a whitespace-only value fails with `output.directory` context; + - an unknown field beneath `output` fails strict loading; and + - constructed and YAML-loaded configurations receive the same validation + result. +6. Keep the maintained example and current-state documentation unchanged in + this stage; they are updated after runtime behavior lands. + +### Tests + +Run: + +```sh +GOWORK=off go test -count=1 ./internal/config +GOWORK=off go test -count=1 ./... +git diff --check +``` + +### Exit Gate + +The configuration package exposes and validates the additive field, strict +loading rejects malformed output configuration, existing configurations remain +valid, and runtime output behavior is still unchanged. + +## Stage 2: Centralize And Apply Effective Destination Resolution + +### Goal + +Apply the locked precedence to single-report and batch orchestration through +one coherent app-owned output-resolution boundary. + +### Work + +1. Create `internal/app/output.go` and move the existing app-owned output + helpers out of `app.go` without changing their established safety behavior: + `prepareBatchOutputs`, `plannedBatchOutputPath`, + `resolveReportOutputPath`, `resolveOutputDir`, `resolveOutputPath`, + `validateWorkingDir`, and `validateOutputPath`. Clean up imports in + `app.go`; do not export these helpers or introduce a new package. +2. Add one narrow directory-precedence helper in `output.go`. It must select a + nonempty explicit directory first, otherwise the configured directory, and + then pass that selected value to the existing directory resolver, whose + empty case returns the validated working directory. Do not inspect the + lower-precedence configured path when an explicit directory is present. +3. Extend the single-report resolver to accept the configured directory and + implement this exact branch: + - obtain the existing default output name from `report.Resolved`; + - when `GenerateRequest.OutputPath` is nonempty, resolve and validate that + complete explicit path against `WorkingDir` and ignore the configured + directory; + - otherwise resolve the configured directory, falling back to + `WorkingDir`, and place the existing default filename beneath it; and + - validate and return one cleaned absolute final file path. +4. Change `GenerateDetailed` to pass `req.Config.Output.Directory` into that + resolver before debug-writer construction, prompt inspection, weather + collection, or model execution. Preserve its current partial-result and + destination-preservation behavior on resolution failure. +5. Change `RunBatchDetailed` to resolve its base directory from + `req.OutputDir`, `req.Config.Output.Directory`, and `req.WorkingDir` in that + order before debug-writer construction, prompt inspection, or collection. + Preserve data-dependent batch planning and the later preflight of every + final report file. +6. Do not add output-directory fields to `GenerateRequest` or `BatchRequest`. + Their existing explicit override fields remain CLI/action inputs, while + the configured fallback remains in `Config`. +7. Add focused `internal/app` tests, using existing real filesystem behavior + and project-owned fakes, that prove: + - omitted configuration retains current-directory destinations for a + representative single report and batch; + - absolute and relative configured directories produce the expected + cleaned absolute destinations; + - single-report `OutputPath` and batch `OutputDir` overrides win and do not + inspect or use a lower-precedence configured path; + - a configured existing non-directory fails before prompt inspection, + collection, model execution, publication, or notification; + - a missing configured directory is created through successful ordinary + publication; + - existing atomic replacement and cancellation tests continue to pass; and + - result paths and Distributor source paths still identify only the final + selected Markdown files. +8. Prefer a compact resolver table plus representative assembled workflows. + Do not duplicate every path case at configuration, CLI, and app layers. + +### Tests + +Run: + +```sh +GOWORK=off go test -count=1 ./internal/app ./internal/fileutil +GOWORK=off go test -count=1 ./... +git diff --check +``` + +### Exit Gate + +Direct app callers, ordinary `generate`, and batches use the locked +CLI/configuration/working-directory precedence; paths are validated before +expensive work; and output, cancellation, and notification invariants remain +intact. + +## Stage 3: Complete CLI Coverage And Current-State Documentation + +### Goal + +Protect the assembled CLI-to-app contract and make every canonical document +and maintained example describe the now-implemented behavior exactly once. + +### Work + +1. Keep `internal/cli.resolveOutputOverride` limited to explicit `--out` and + `--out-dir` values. It must continue resolving relative flag values against + the captured working directory. Do not resolve `output.directory` in the + CLI or replace the raw configured value in `Config`; the app owns final + precedence and runtime path validation. +2. Add or update focused `internal/cli` request-construction tests to prove: + - a configured relative or absolute directory survives configuration + loading and reaches the app request while the absent CLI override remains + empty; + - relative and absolute `--out` and `--out-dir` values retain their current + resolution; and + - an explicit flag coexists with the loaded configured value so app-owned + precedence can ignore the latter. + Keep these tests narrow; app tests own final path and publication outcomes. +3. Add this illustrative stanza to `examples/config.yml`: + + ```yaml + output: + directory: /var/lib/weatherreporter/reports + ``` + + Keep `examples/minimal-config.yml` unchanged so it continues to demonstrate + the empty-default path. Extend `TestLoadExampleConfig` to assert the exact + maintained example value. +4. Update canonical current-state documentation according to + `docs/policy/documentation.md`: + - `docs/config.md` owns the field, empty default, whitespace validation, + absolute and relative semantics, lack of expansion, and configuration + versus flag precedence; + - `docs/cli.md` owns the unchanged flag syntax and the effective + flag/configuration/working-directory selection visible to users; + - `docs/operations.md` owns normal operation with a configured publication + directory, explicit one-command overrides, missing-directory creation, + atomic replacement, and failure handling; + - `README.md` updates only its short quickstart destination summary and + links to the canonical configuration/CLI references; + - `docs/policy/architecture.md` states that configuration or explicit CLI + input selects the operator-owned output destination without changing the + stateless boundary; + - `docs/internal/cli.md` describes capture and transport of the configured + fallback plus explicit overrides without duplicating the field reference; + and + - `docs/internal/app-orchestration.md` describes app-owned effective + destination precedence and the unchanged single and batch preflight + ordering. +5. Do not change historical release notes. Do not duplicate the complete field + contract outside `docs/config.md`, complete flag syntax outside + `docs/cli.md`, or operational procedures outside `docs/operations.md`. +6. Review all changed commands, values, paths, and examples against executable + code. Follow every added or changed repository-relative link and confirm its + target exists. + +### Tests + +Run: + +```sh +GOWORK=off go test -count=1 ./internal/config ./internal/cli ./internal/app +GOWORK=off go run ./cmd/weatherreporter --help +GOWORK=off go test -count=1 ./... +git diff --check +``` + +Confirm the maintained full and minimal examples both load through the +configuration test suite and no current-state document still claims that the +working directory is always the no-flag destination. + +### Exit Gate + +The CLI-to-app boundary is protected without duplicating app behavior tests, +the maintained example is executable configuration, and all canonical +current-state documentation agrees with the implemented contract. + +## Stage 4: Reconcile Roadmaps And Run The Final Gate + +### Goal + +Verify the complete feature in the same module-isolated environment used for +release preparation and leave roadmap status accurate for post-implementation +review. + +### Work + +1. Review the complete diff against the feature roadmap, this plan, and all + documents under `docs/policy/`. Remove incidental refactors, redundant + tests, duplicate documentation, and any behavior outside the accepted + scope. +2. Confirm repository hygiene: + - no tracked `go.work` or `go.work.sum`; + - no `vendor` directory or `go.mod` replacement; + - no generated binaries, test output, credentials, private paths, or + temporary filesystem artifacts; and + - no unintended changes to the profile-comparison feature. +3. After every implementation and documentation gate passes, update + `docs/roadmap/output-directory.md` from `Accepted; unimplemented` to + `Implemented; retained temporarily for post-implementation review`. + Preserve its target-state purpose and policy content until roadmap cleanup + is separately authorized. +4. Update the prerequisite section in + `docs/roadmap/profile-comparison.md` to state that configurable output + directories are implemented. Link its current behavior summary to + `docs/config.md` and `docs/operations.md` rather than duplicating their + contracts. Do not otherwise advance or implement profile comparison. +5. Mark this plan `Status: Complete` only after the final gate succeeds. Do not + delete either roadmap or this plan in this stage; repository roadmap cleanup + follows post-implementation review. + +### Tests + +Run from the repository root: + +```sh +set -eu +test -z "$(git ls-files go.work go.work.sum)" +test ! -e vendor +if grep -Eq '^[[:space:]]*replace([[:space:]]|\()' go.mod; then + printf '%s\n' 'go.mod contains a replacement' >&2 + exit 1 +fi +GOWORK=off go test -count=1 ./... +GOWORK=off go test -race -count=1 ./... +GOWORK=off go vet ./... +GOWORK=off go build ./... +GOWORK=off go mod tidy -diff +unformatted="$(git ls-files '*.go' | while IFS= read -r file; do gofmt -l "$file"; done)" +test -z "$unformatted" +GOWORK=off go run ./cmd/weatherreporter --help >/dev/null +git diff --check +git diff --cached --check +``` + +Follow every added or changed local documentation link and verify the full and +minimal maintained configuration examples through their tests. + +### Exit Gate + +The complete repository passes normal, race-enabled, vet, build, module, +formatting, help, documentation, and hygiene checks with `GOWORK=off`; roadmap +status accurately distinguishes the implemented output-directory prerequisite +from the still-unimplemented profile-comparison feature; and no required work +remains. + +## Open Questions + +None. The feature roadmap and this plan define every product, architecture, +precedence, path, compatibility, testing, documentation, and completion choice +required for implementation. diff --git a/docs/roadmap/output-directory.md b/docs/roadmap/output-directory.md new file mode 100644 index 0000000..f644304 --- /dev/null +++ b/docs/roadmap/output-directory.md @@ -0,0 +1,191 @@ +# Configurable Output Directory Roadmap + +Status: Implemented; retained temporarily for post-implementation review. + +## Purpose + +Weatherreporter currently writes generated reports beneath the present working +directory unless an operator supplies `--out` for a single report or +`--out-dir` for a batch. Scheduled and packaged deployments should be able to +select their ordinary report directory in configuration without repeating a +CLI flag on every invocation. + +This feature adds one optional nested configuration value: + +```yaml +output: + directory: /var/lib/weatherreporter/reports +``` + +The nested `output` namespace leaves room for future output-related settings, +but this roadmap approves only `output.directory`. + +## User Intent + +The configured directory is an operator-owned publication destination. It is +not an application workspace, history store, cache, or source of input for a +later run. The feature must preserve Weatherreporter's stateless execution +model and atomic report publication. + +Operators should be able to establish a stable default once in `config.yml` +while retaining explicit per-invocation control through the existing CLI +flags. + +## Target Configuration Contract + +`output.directory` is an optional string with an empty default. + +- An omitted or empty value preserves the current behavior: default report + names are published beneath the process's present working directory. +- A nonempty absolute path selects that directory directly. +- A nonempty relative path is resolved against the process's present working + directory, matching the existing relative-path behavior of `--out` and + `--out-dir`. It is not resolved relative to the configuration file. +- A whitespace-only value is invalid rather than being treated as an implicit + current-directory selection. +- A nonexistent directory is allowed and may be created as part of normal + atomic output publication. +- An existing path that is not a directory, an inaccessible destination, or + any other unsafe output target fails through the existing output preflight + and publication contracts. + +The configuration remains strict: unknown fields beneath `output` are rejected +in the same way as unknown fields elsewhere in the configuration. + +## Destination Precedence + +Effective output selection follows this precedence, from highest to lowest: + +1. An explicit `--out` path for `generate`, or an explicit `--out-dir` path for + `run`. +2. `output.directory` from the effective configuration. +3. The process's present working directory. + +For single-report generation: + +- `generate ` without `--out` publishes the report's existing default + filename beneath `output.directory` when configured. +- `generate --out PATH` treats `PATH` as the complete destination and + ignores `output.directory`. +- A relative `--out` value remains relative to the present working directory; + it is not placed beneath `output.directory`. + +For batch generation: + +- `run ` without `--out-dir` publishes every selected report beneath + `output.directory` when configured. +- `run --out-dir PATH` ignores `output.directory` for that invocation. + +Default report filenames, batch membership, summaries, and notification +behavior do not change. + +## Architectural End State + +The configuration package owns the `output.directory` field, its empty +default, YAML decoding, normalization, and configuration-level validation. +The CLI continues to own flag parsing and the present working directory used +for relative CLI values. Application orchestration owns the final precedence +decision, output preflight, default filename placement, atomic publication, and +the resolved output paths returned in action results. + +Single-report and batch paths must use one coherent destination-resolution +policy rather than maintaining independent interpretations of the configured +directory. Output destinations must still be fully resolved and validated +before weather collection or Promptkit execution where the existing workflow +provides that guarantee. + +The generated Markdown remains the only ordinary durable artifact. No +directory scan, prior report lookup, manifest, metadata file, or managed +cleanup behavior is introduced. + +## Scope + +The completed feature includes: + +- an `OutputConfig` configuration boundary containing `directory`; +- an empty default that preserves current installations; +- strict YAML loading and validation for the nested configuration; +- configured-directory fallback for both `generate` and `run`; +- explicit CLI-over-configuration precedence; +- consistent absolute and relative path handling; +- preservation of output preflight, atomic replacement, cancellation, and + notification ordering guarantees; +- updates to the maintained example configuration; +- updates to the canonical configuration, CLI, operations, architecture, and + app-orchestration documentation where their owned contracts change; and +- focused configuration, CLI, and application tests protecting the public + behavior and important failure paths. + +## Compatibility + +This is an additive configuration feature. Existing configuration files remain +valid, and installations that do not set `output.directory` retain the current +working-directory behavior. + +Existing `--out` and `--out-dir` syntax and relative-path semantics remain +unchanged. The feature does not alter report contents, default filenames, +Promptkit execution, Distributor payloads, or exit behavior. + +## Testing Expectations + +Tests should provide durable coverage for: + +- the empty default and YAML loading of absolute and relative directories; +- rejection of whitespace-only and unknown output configuration values; +- current-working-directory fallback when the field is omitted; +- configured-directory use by representative single-report and batch actions; +- `--out` and `--out-dir` precedence over the configured directory; +- preservation of existing relative CLI path semantics; +- rejection of an existing non-directory destination before expensive work; +- successful publication beneath a directory that does not yet exist; and +- preservation of atomic destination replacement and action-result paths. + +Tests must remain deterministic, offline, and independent of machine-specific +directories. Filesystem cases should use temporary directories and exercise +behavior through the narrowest stable configuration, CLI, or application +boundary that owns the contract. + +## Documentation End State + +Once implemented, the exact field definition, default, and path semantics +belong in the [configuration reference](../config.md). CLI flag syntax and +precedence summaries belong in the [CLI reference](../cli.md). Normal output +handling belongs in the [operations guide](../operations.md), while ownership +and workflow mechanics belong in the [architecture policy](../policy/architecture.md) +and [app-orchestration internals](../internal/app-orchestration.md), +respectively. + +The maintained configuration under `examples/` should illustrate the field +without turning another document into a duplicate configuration reference. +Current-state documentation must not describe this feature as available until +the implementation lands. + +## Non-Goals + +This roadmap does not introduce: + +- per-report or per-batch configured output directories; +- filename or directory templates; +- a new `--out-dir` flag for single-report generation; +- environment-variable expansion or home-directory expansion in paths; +- paths resolved relative to the configuration file; +- output retention, rotation, cleanup, indexing, or history; +- configuration-time directory creation; +- multiple output destinations or output format selection; or +- changes to debug artifact placement or Distributor notification routing. + +Any future output settings require their own accepted scope and should be +added beneath `output` only when a concrete requirement exists. + +## Completion Criteria + +The feature is complete when operators can set `output.directory` and obtain +the same destination behavior for ordinary single-report and batch generation, +explicit CLI destinations reliably take precedence, omitted configuration is +fully backward compatible, output safety invariants remain intact, and all +affected canonical documentation and maintained examples describe the +implemented contract. + +## Open Questions + +None. The target behavior and policy choices are defined above. diff --git a/docs/roadmap/profile-comparison.md b/docs/roadmap/profile-comparison.md new file mode 100644 index 0000000..24eaffd --- /dev/null +++ b/docs/roadmap/profile-comparison.md @@ -0,0 +1,397 @@ +# LLM Profile Comparison Roadmap + +Status: Accepted; unimplemented. + +## Purpose + +Prompt development currently requires separate Weatherreporter invocations to +compare several LLM profiles. Those invocations may collect different weather +snapshots or rebuild inputs at different times, making model output harder to +compare and slowing prompt iteration. + +Weatherreporter should provide a first-class `compare` command that resolves +one report, prepares one exact data package, executes the same prompt and data +package concurrently through several explicitly selected Promptkit profiles, +and publishes a self-contained local comparison bundle. + +An illustrative invocation is: + +```sh +weatherreporter compare daily \ + --date 2026-08-24 \ + --profile weather-light \ + --profile weather-balanced \ + --profile weather-deep +``` + +This is a prompt-development workflow, not an automated model evaluator. Its +output gives a maintainer consistent evidence for human comparison without +assigning scores or selecting a winner. + +## Prerequisite + +Configurable output directories are implemented. Profile comparison must reuse +the current [configuration reference](../config.md) and [operations +guide](../operations.md) rather than introduce a second destination policy. + +## User Intent + +The command is intended for deliberate evaluation of multiple profiles, +including sets of eight to twelve candidate models. Concurrency is part of the +feature, not a future optimization. Promptkit should retain ownership of +backend-specific capacity, while Weatherreporter owns comparison-wide +coordination, cancellation, deterministic results, and artifact publication. + +Every profile must receive byte-for-byte identical prompt input. Weather data, +derived facts, modules, prompt metadata, and serialized YAML must not be +recollected or rebuilt separately for individual profiles. + +Comparison bundles are explicitly requested, operator-owned development +outputs. They are not Weatherreporter state, are never read implicitly by a +later run, and do not weaken the ordinary stateless execution model. + +## Command Contract + +The command form is: + +```text +weatherreporter compare REPORT [options] +``` + +`REPORT` accepts the implemented generated-text reports: `daily`, `today`, +`tomorrow`, and `hourly`. Report-date behavior matches `generate`: `daily` +requires `--date`, `today` may accept an explicit date or use the current local +date, and the remaining report types retain their existing period policies. + +The command accepts the applicable common generation options, including +`--config`, `--units`, `--tz`, `--date`, `--llm-debug-dir`, and `--quiet`, plus: + +- repeatable `--profile PROFILE_ID` selections; +- `--out-dir PATH` for the exact comparison-bundle directory; and +- `--replace` to authorize guarded replacement of a recognized existing + comparison bundle. + +At least two distinct, nonblank profile IDs are required. Their command-line +order is significant and is preserved in filenames, summaries, and +`comparison.json`. Duplicate profile IDs are rejected rather than silently +deduplicated or executed twice. + +Profiles are always explicit for this command. `promptkit.profile` does not add +or replace a comparison selection, but all other effective Promptkit settings, +profile-source precedence, local backend configuration, credential lookup, and +profile overrides remain in force. + +The initial feature has no Weatherreporter-specific concurrency flag or +artificial profile-count ceiling. The explicit profile list bounds the +comparison, and Promptkit owns capacity enforcement for each selected backend. + +## Preparation And Execution Invariants + +A comparison has this logical lifecycle: + +1. Parse and validate the report, date, profile list, configuration, output + destination, and replacement authorization. +2. Resolve the report definition, valid period, prompt identity, and default + output name once. +3. Inspect the exact prompt once and preflight every selected profile, + including its effective backend, model, and required credential + availability, before weather collection. +4. Collect weather data exactly once. +5. Build collected and derived facts, the module snapshot, briefing metadata, + and the prompt data package exactly once. +6. Marshal the data package to one immutable YAML byte sequence exactly once. +7. Execute the exact prompt version concurrently for every selected profile, + passing the same immutable YAML bytes to every execution. +8. Validate and render each profile result independently from the shared + deterministic inputs. +9. Assemble results in requested-profile order and publish one coherent + comparison bundle. + +This lifecycle describes the required end-state behavior rather than an +implementation-stage sequence. + +No profile execution may cause recollection, report re-resolution, module +rebuilding, or data-package remarshalling. Prompt execution may perform +Promptkit-owned validation or repair behavior, but Weatherreporter does not +retry a failed comparison execution independently. + +## Concurrency And Cancellation + +Weatherreporter starts one execution for each preflighted profile and permits +them to run concurrently through one shared, concurrency-safe Promptkit +executor. Promptkit's engine-local backend pools remain authoritative for +backend concurrency and waiting capacity. Profiles routed to a limited local +backend therefore respect its configured limit, while profiles routed to +other backends may proceed independently. + +Weatherreporter must not add a second semaphore that obscures or overrides +Promptkit's backend policy. It must safely coordinate goroutine lifecycles, +result collection, debug callbacks, and output assembly without data races. + +One profile failure does not cancel its peers. Provider, capacity, validation, +and rendering failures are recorded for that profile while other executions +continue. Cancellation or deadline expiration of the comparison command is +propagated to every outstanding execution, prevents new publication, and is +joined without leaking goroutines. + +Completion order must not affect filenames, manifest order, CLI summaries, or +error aggregation. Those outputs always follow the original `--profile` +order. + +## Output Destination + +Without `--out-dir`, Weatherreporter derives a comparison directory from the +resolved report's existing default Markdown filename by removing `.md` and +prefixing `comparison-`: + +| Report output | Comparison directory | +| --- | --- | +| `today.md` | `comparison-today/` | +| `tomorrow.md` | `comparison-tomorrow/` | +| `hourly.md` | `comparison-hourly/` | +| `daily-2026-08-24.md` | `comparison-daily-2026-08-24/` | + +The derived directory is created beneath `output.directory` when configured, +or beneath the present working directory otherwise. An explicit `--out-dir` +is the exact bundle directory, resolves relative to the present working +directory when necessary, and overrides `output.directory` completely. + +All destination selection and validation completes before weather collection. +The resolved comparison directory is returned in the command's structured +result. + +## Comparison Bundle + +A successful three-profile comparison has a flat layout: + +```text +comparison-daily-2026-08-24/ +├── comparison.json +├── data-package.yml +├── 01-weather-light.md +├── 02-weather-balanced.md +└── 03-weather-deep.md +``` + +`data-package.yml` contains the exact YAML bytes passed to every Promptkit +execution. It is written once and its SHA-256 digest is recorded in the +manifest. + +Each report filename begins with its one-based, zero-padded selection position +and a filesystem-safe representation of the requested logical profile ID. The +safe representation must not permit absolute paths, traversal, separators, or +control characters. The manifest retains the exact case-sensitive profile ID, +so filename normalization never becomes the authority for profile identity. + +`comparison.json` is the authoritative index for the bundle. It uses an +explicit schema version and records safe comparison information including: + +- comparison identity and start and finish timestamps; +- report ID, resolved valid period, and effective timezone; +- prompt ID, version, and inspected prompt hash; +- the relative data-package filename and SHA-256 digest; +- total, succeeded, and failed profile counts; and +- one ordered result per requested profile containing the exact profile ID, + resolved backend and model, relative report filename when present, + execution and validation status, and safe error information when failed. + +The manifest and normal command summary must not contain credentials, provider +request bodies, raw model output, rendered prompts, schemas, provider +endpoints, or other content-rich diagnostics. The explicit data package and +generated reports contain the development material the user requested and +must be handled as operator-owned potentially sensitive output. + +## Failure And Publication Policy + +Failure before concurrent execution, including invalid profiles, missing +credentials, collection failure, preparation failure, or unsafe destination, +publishes no comparison bundle and performs no model calls where the failure +is discoverable during preflight. + +After execution begins, Weatherreporter waits for every non-cancelled profile. +If one or more profiles fail, it still publishes a coherent partial bundle +containing `data-package.yml`, every successfully rendered report, and a +manifest describing all successes and failures. It then returns a non-zero +exit status. A failed profile has no report file unless a future contract +explicitly introduces a separately named diagnostic artifact. + +Bundle contents are staged outside the destination and published only after +the manifest is complete. Ordinary publication accepts only an absent or empty +target directory. A nonempty existing directory fails without modification +unless `--replace` is present. + +`--replace` may replace only the exact resolved target and must reject broad or +unsafe targets such as a filesystem root, the present working directory, a +symlink, or an unrecognized nonempty directory. A recognized prior bundle must +contain a valid Weatherreporter comparison manifest. Replacement publishes the +new complete or coherent partial bundle as a unit, prevents stale reports from +the prior comparison from surviving, and preserves or restores the prior +bundle if the final replacement operation fails. + +An interrupted or cancelled comparison does not replace an existing bundle. +Temporary staging artifacts are cleaned up on ordinary failure and +cancellation without scanning or modifying unrelated directories. + +## Prompt Debugging + +The existing `--llm-debug-dir` mechanism remains available. Concurrent +comparison executions require distinct, deterministic debug identities that +include the comparison and exact profile selection so callbacks cannot collide +or overwrite another profile's artifacts. + +Debug writing must be concurrency-safe and retain the existing permission, +redaction, explicit-opt-in, and path-containment guarantees. Debug artifacts +remain separate from the comparison bundle; the bundle does not implicitly +enable full Promptkit diagnostics. + +## Notification Policy + +Profile comparisons never invoke Distributor notification, even when +notification is enabled in the effective configuration. Comparison reports +are local development artifacts rather than ordinary report publications. + +Adding comparison publication or upload behavior would require a separate +accepted feature scope and explicit operator authorization. + +## Architectural End State + +Application orchestration exposes a reusable prepared-report boundary that +contains the resolved report, shared collected and derived facts, module +snapshot, briefing metadata, generated-text handler, render inputs, and exact +serialized data package. That boundary is immutable during concurrent profile +execution. + +Ordinary `generate` behavior continues to prepare once and execute once. +`compare` prepares once and executes many without duplicating the generation +workflow or calling `GenerateDetailed` in a loop. Shared preparation, +profile-specific Promptkit execution, structured-output validation, rendering, +and artifact publication remain distinct responsibilities. + +The Promptkit adapter remains the only owner of dependency-specific types and +engine calls. The CLI owns parsing and user-facing summaries. The configuration +package owns configuration. Application orchestration owns comparison order, +concurrency lifecycle, failure aggregation, and bundle publication. Domain, +prompt-input, generated-text, and template packages retain their existing +deterministic contracts. + +## Scope + +The completed feature includes: + +- the `compare` CLI command for every implemented generated-text report; +- repeatable explicit profile selection and validation; +- configured and CLI output-directory integration after the prerequisite + feature lands; +- one-time report resolution, collection, deterministic preparation, and YAML + serialization; +- concurrent execution through one Promptkit executor with backend capacity + respected; +- independent validation and rendering with deterministic ordered results; +- the flat, versioned comparison-bundle contract; +- safe filename derivation and data-package hashing; +- coherent partial-result publication and non-zero failure behavior; +- guarded whole-bundle replacement through `--replace`; +- comparison-aware, concurrency-safe optional prompt debugging; +- explicit suppression of Distributor notification; +- structured normal and quiet-mode CLI behavior consistent with existing + commands; +- focused race-safe tests across configuration, CLI, application, + Promptkit-adapter, rendering, and filesystem boundaries; and +- updates to every affected canonical user, operator, architecture, + integration, and internal document. + +## Compatibility + +The feature is additive. Existing `generate` and `run` commands, report +definitions, profile defaults, configuration, output filenames, notification +behavior, and exit contracts remain unchanged. + +The comparison manifest and bundle layout begin as versioned contracts. They +do not become inputs accepted by Weatherreporter, and no backward-compatible +replay or long-term archive guarantee is implied beyond identifying the schema +used to interpret a produced bundle. + +## Testing Expectations + +Tests should provide durable coverage for: + +- report and date parsing consistent with `generate`; +- rejection of fewer than two profiles, blanks, and duplicates; +- inspection of the exact prompt and every profile before collection; +- no collection or model execution after a preflight failure; +- exactly one weather collection and one preparation for several profiles; +- byte-for-byte identical data-package input in every execution; +- observable concurrent execution through a concurrency-safe fake executor; +- respect for Promptkit-owned backend capacity in an assembled adapter test + where that integration adds distinct confidence; +- deterministic filenames, manifest order, summaries, and errors under varied + completion order; +- continuation and coherent partial publication after one profile fails; +- cancellation propagation, goroutine completion, and preservation of an + existing destination; +- destination precedence and each derived default directory; +- safe filename handling for unusual valid profile IDs; +- absent, empty, occupied, symlinked, unsafe, recognized, and unrecognized + replacement targets; +- removal of stale prior report files during authorized whole-bundle + replacement; +- exact package digest and manifest/result consistency; +- concurrency-safe, non-colliding opt-in debug artifacts; and +- absence of Distributor calls for complete and partial comparisons. + +Concurrency and replacement behavior require race-enabled and consequential +failure-path coverage. Tests must remain deterministic, offline, credential +free, and independent of real Promptkit providers or machine-specific paths. + +## Documentation End State + +Once implemented, the [CLI reference](../cli.md) owns command syntax, flags, +summary, and exit behavior. The [operations guide](../operations.md) owns the +bundle lifecycle, replacement procedure, sensitivity guidance, and practical +prompt-comparison workflow. The [architecture policy](../policy/architecture.md) +owns the statelessness, concurrency, notification, and publication invariants. + +The [Promptkit integration guide](../integrations/promptkit.md) should describe +the consumer-visible multi-profile execution boundary without duplicating +Promptkit's backend-capacity reference. App orchestration, prompt input, +generated text, prompt debugging, and any new bundle implementation details +belong in focused documents under `docs/internal/`. + +Current-state documentation must not describe profile comparison as available +until the implementation lands. + +## Non-Goals + +This roadmap does not introduce: + +- automatic model scoring, ranking, recommendation, or winner selection; +- semantic or textual diff generation between reports; +- repeated sampling of one profile or statistical evaluation; +- prompt or profile editing through Weatherreporter; +- replaying a saved data package as command input; +- comparing several report types in one command; +- Weatherreporter-owned backend concurrency or queue configuration; +- automatic retries beyond Promptkit's existing execution contract; +- Distributor upload or other external publication; +- comparison history, indexing, retention, cleanup schedules, or implicit + discovery of prior bundles; or +- changes to ordinary report content or normal generation behavior. + +Any later automated evaluation, replay, sampling, or publication feature +requires a separate accepted roadmap. + +## Completion Criteria + +The feature is complete when a maintainer can select several Promptkit +profiles, have them execute concurrently against one exact prepared report +package, and receive a safe, flat, deterministic comparison bundle whose +manifest accurately describes every success and failure. Configured and +explicit destinations must follow the accepted output policy, replacement must +never mix or silently destroy unrelated contents, cancellation and partial +failure must be race-safe, ordinary notification must remain disabled, and all +affected canonical documentation must describe the implemented behavior. + +## Open Questions + +None. The scope, prerequisites, user intent, and target behavior required for a +future staged implementation plan are defined above.