eric/weatherreporter
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 2 Wiki Activity

Retire completed comparison roadmaps

Browse Source
This commit is contained in:
Eric Rakestraw
2026-08-13 04:28:03 +00:00
parent 88fde0df7f
commit 142f2f92e7
4 changed files with 4 additions and 1310 deletions
Download Patch File Download Diff File Expand all files Collapse all files

2
docs/policy/architecture.md
View File

@@ -87,6 +87,8 @@ directly.
outcomes only; a failed batch notification is represented separately at the outcomes only; a failed batch notification is represented separately at the
batch level. batch level.
- Comparison never invokes Distributor notification. - Comparison never invokes Distributor notification.
- Profile comparison supports operator review only: it does not score, rank,
select, resample, or replay profile executions.
- Default tests are deterministic, offline, and use Promptkit/provider fakes - Default tests are deterministic, offline, and use Promptkit/provider fakes
rather than live provider calls. See the [testing policy](testing.md). rather than live provider calls. See the [testing policy](testing.md).

4
docs/roadmap/audit.md
View File

@@ -222,8 +222,8 @@ investigation stage. Remediation acceptance, priority, and implementation
sequencing remain exclusively Stage 27 work. sequencing remain exclusively Stage 27 work.
Stage 27 accepted all 74 confirmed findings for implementation and allocated Stage 27 accepted all 74 confirmed findings for implementation and allocated
each exactly once across the dependency-ordered each exactly once across the dependency-ordered audit remediation roadmap. No
[audit remediation roadmap](audit-remediation.md). No confirmed finding is confirmed finding is
priority-deferred or blocked on a separate product/architecture decision. The priority-deferred or blocked on a separate product/architecture decision. The
roadmap places generated-forecast integrity, security, correctness, data roadmap places generated-forecast integrity, security, correctness, data
integrity, destructive operations, and cancellation before efficiency, integrity, destructive operations, and cancellation before efficiency,

910
docs/roadmap/implementation.md
View File

@@ -1,910 +0,0 @@
# Repository Audit Remediation Roadmap
Status: Ready for implementation.
## Purpose And Authority
This roadmap translates the confirmed findings in the
[repository audit ledger](audit.md) into dependency-ordered implementation
stages. The ledger owns discovery evidence and final finding disposition. This
document owns remediation scope, ordering, locked behavior, verification, and
completion state until the work is finished.
All 74 confirmed findings are accepted for remediation. None is priority-
deferred and none requires a separate product or architecture decision before
work can begin. Four rejected standalone findings remain consolidated into
their confirmed owners and are not separate work items.
This roadmap describes future work. Until an individual stage is implemented,
canonical current-state documentation and executable behavior remain
authoritative.
## Implementation Rules
Every stage must:
- read `docs/development.md`, all files under `docs/policy/`, this roadmap, and
the task-specific canonical documents identified below;
- implement only its listed findings and prerequisites;
- preserve the architecture policy's stateless pipeline, adapter boundaries,
curated prompt boundary, operator-owned outputs, comparison ordering, and
notification rules unless the stage explicitly corrects that contract;
- add durable tests at the narrowest stable boundary and avoid duplicating
behavior already owned elsewhere;
- update every listed canonical documentation owner in the same change as an
implemented contract change;
- use deterministic, offline, credential-free tests and synthetic fixtures;
- run `gofmt` on changed Go files, the stage's focused commands,
`GOWORK=off go test -count=1 ./...`, and `git diff --check`; and
- commit the completed stage separately when the implementing prompt requests
commits.
Stages affecting concurrent execution, cancellation, filesystem mutation, or
process-global state must also run the affected packages with `-race`. No stage
may weaken a safety invariant merely to make a test pass.
## Priority And Finding Allocation
Each confirmed finding appears exactly once below. Priority describes risk and
the default remediation order, not confidence; dependency prerequisites may
place a lower-priority consolidation mechanism before the P1 correction that
depends on it. Every listed finding has high confidence.
| Priority | Stages | Findings | Decision |
| --- | --- | --- | --- |
| P0: generated forecast integrity | 1 | `AUD-029`, `AUD-030`, `AUD-031`, `AUD-033` | Accepted now |
| P1: security, correctness, and data integrity | 2-14, 16-21, 24-37 | `AUD-002` through `AUD-021` excluding rejected `AUD-022`; `AUD-023` through `AUD-028`; `AUD-034` through `AUD-038`; `AUD-040` through `AUD-058`; `AUD-073`, `AUD-074`, `AUD-077`, and `AUD-078` | Accepted now |
| P2: deterministic test and operational quality | 38-43 | `AUD-059` through `AUD-062`, `AUD-064`, `AUD-065`, and `AUD-066` | Accepted now |
| P3: behavior-preserving consolidation and cleanup | 15, 22-23, 44-52 | `AUD-039`, `AUD-067` through `AUD-072` | Accepted now; Stages 15 and 22-23 run early because P1 corrections depend on them |
| P4: documentation lifecycle and audit retirement | 53-56 | `AUD-075` and `AUD-076`; final audit-record retirement | Accepted now |
Rejected `AUD-001`, `AUD-022`, and `AUD-032` are represented only through
`AUD-072`; rejected `AUD-063` is represented only through `AUD-039`.
## Remediation Stages
### Stage 1: Correct The Curated Prompt Package Contract
- Findings: `AUD-029`, `AUD-030`, `AUD-031`, `AUD-033`.
- Affected areas: `internal/promptinput`, shared embedded prompt assets, and
focused prompt-input/prompt-asset tests.
- Work: remove raw transport-shaped warning details from provider input; make
shared prompt paths match serialized YAML; describe regional versus point-
matched hazard products accurately; and cover stanza serialization failure.
- Locked behavior: deterministic category/stanza order, exact prompt identity,
safe normal diagnostics, and byte-stable valid YAML remain unchanged.
- Documentation owners: `docs/internal/prompt-input.md` and
`docs/integrations/promptkit.md` if their implemented contract changes.
- Exit commands: focused tests for `./internal/promptinput` and
`./internal/promptassets`, then the standard repository commands.
### Stage 2: Make Configuration Keys And Overrides Explicit
- Findings: `AUD-004`, `AUD-006`, `AUD-078`.
- Affected areas: `internal/config`, report-key lookup, timezone parsing,
maintained examples, and configuration tests.
- Work: reject unsupported missing-source keys; make programmatic report
overrides obey the same presence semantics as loaded configuration; and
document accepted report-key normalization and numeric-offset bounds.
- Locked behavior: file/CLI precedence, inactive-integration policy, exact
command-name lookup, and secret redaction remain unchanged.
- Documentation owners: `docs/config.md` and maintained examples only where
executable behavior requires them.
- Exit commands: focused config and CLI tests, followed by the standard
repository commands.
### Stage 3: Make Secret Loading Atomic
- Findings: `AUD-007`.
- Affected areas: configuration loading, secret-directory application, load
failure state, and focused configuration tests.
- Work: stage secret values and apply them to the returned configuration only
after the complete load and validation sequence succeeds.
- Locked behavior: secret precedence, configured secret-directory discovery,
environment independence, actionable errors, and redaction remain unchanged.
- Documentation owners: `docs/config.md` only if the observable load contract
needs clarification.
- Exit commands: focused config tests plus
`GOWORK=off go test -race -count=1 ./internal/config`, followed by the
standard repository commands.
### Stage 4: Require Complete Single-Report Notification Identity
- Findings: `AUD-005`.
- Affected areas: Distributor notification configuration, bundle-ID rendering,
and focused config/app tests.
- Work: reject enabled single-report notification configurations that can
produce blank required identity fields before report work begins.
- Locked behavior: inactive notification settings remain optional; configured
templates and publication-before-notification ordering remain intact.
- Documentation owners: `docs/config.md`, `docs/operations.md`, and Distributor
integration docs if accepted values or failures change.
- Exit commands: focused config and app notification tests, then the standard
repository commands.
### Stage 5: Propagate Process Interrupt Cancellation
- Findings: `AUD-008`.
- Affected areas: `cmd/weatherreporter`, action contexts, CLI wiring, and
interrupt/cancellation tests.
- Work: convert operating-system interrupts into action-context cancellation
and verify active generation, batch, and comparison workflows observe it.
- Locked behavior: command/flag ownership, structured summary shapes, stream
separation, and one executor per action remain unchanged.
- Documentation owners: `docs/cli.md`, `docs/operations.md`, and
`docs/internal/cli.md` if signal behavior needs clarification.
- Exit commands: `GOWORK=off go test -race -count=1 ./internal/cli
./internal/app ./cmd/weatherreporter`, then the standard repository commands.
### Stage 6: Correct CLI Preflight, Coverage, And Visible Contracts
- Findings: `AUD-009`, `AUD-010`, `AUD-074`.
- Affected areas: `internal/cli`, executor construction, assembled CLI tests,
help text, and CLI documentation.
- Work: validate report dates before executor construction; add representative
generate and batch result/exit tests; and align help/internal guidance with
quiet, date, and comparison behavior.
- Locked behavior: command/flag ownership, structured summary shapes, stream
separation, cancellation from Stage 5, and one executor per action remain
unchanged.
- Documentation owners: `docs/cli.md` and `docs/internal/cli.md`.
- Exit commands: focused CLI/app tests, both root and comparison help commands,
then the standard repository commands.
### Stage 7: Validate Weather API Endpoints And Retry Classification
- Findings: `AUD-011`, `AUD-015`.
- Affected areas: Weather API configuration and adapter construction, warmup
retry classification, local HTTP fixtures, and focused tests.
- Work: accept only supported endpoint forms and retry warmup only for
transient failures.
- Locked behavior: all eight source contracts, checked-empty optional products,
cancellation, provenance, and offline testing remain intact.
- Documentation owners: `docs/config.md`, `docs/integrations/weatherapi.md`,
and `docs/internal/collect.md` as applicable.
- Exit commands: focused configuration, adapter, and collection tests, then the
standard repository commands.
### Stage 8: Bound And Redact Weather API Diagnostics
- Findings: `AUD-012`, `AUD-013`.
- Affected areas: Weather API response reading, non-success error projection,
size limits, local HTTP fixtures, and focused adapter tests.
- Work: replace upstream response text with bounded safe diagnostics and reject
rather than silently truncate oversized bodies.
- Locked behavior: HTTP status classification, source-local error context,
cancellation, and offline tests remain intact.
- Documentation owners: `docs/integrations/weatherapi.md` and
`docs/internal/collect.md` if observable limits or diagnostics change.
- Exit commands: focused boundary-size and malicious-response adapter tests,
then the standard repository commands.
### Stage 9: Require Usable Hourly Period Bounds
- Findings: `AUD-014`.
- Affected areas: Weather API hourly decoding, normalized hourly data,
required-source validation, fixtures, and focused tests.
- Work: reject required hourly products whose periods do not contain usable
time bounds before they enter collection and derivation.
- Locked behavior: required/optional source policy, source provenance,
chronological selection, and valid hourly fixtures remain unchanged.
- Documentation owners: `docs/integrations/weatherapi.md`,
`docs/internal/weather-data.md`, and `docs/internal/collect.md`.
- Exit commands: focused weather-data, adapter, and collection tests, then the
standard repository commands.
### Stage 10: Correct Civil-Time Windows And Daily Run Identity
- Findings: `AUD-002`, `AUD-003`.
- Affected areas: `internal/timeutil`, Daily report identity, DST and run-ID
tests, and report-registry documentation.
- Work: construct configured clocks as local civil times across both DST
transitions and add durable protection for Daily run-ID uniqueness.
- Locked behavior: half-open selection, civil-day report periods, rolling
Hourly elapsed time, and source-period identity remain unchanged.
- Documentation owners: `docs/internal/report-registry.md`.
- Exit commands: focused time/report tests with both DST transitions, then the
standard repository commands.
### Stage 11: Preserve Units And Overnight Alert Coverage
- Findings: `AUD-016`, `AUD-017`.
- Affected areas: `internal/forecast`, `internal/facts`, metric fallback
values, overnight dayparts, alerts, and focused tests.
- Work: preserve units in fallback values and include alerts over the complete
overnight interval.
- Locked behavior: half-open overlap semantics, chronological ordering, source
periods, and valid US-customary results remain unchanged.
- Documentation owners: `docs/internal/forecast-derivation.md` and
`docs/internal/facts.md`.
- Exit commands: focused forecast/facts tests with metric and overnight cases,
then the standard repository commands.
### Stage 12: Validate Precipitation And Ice Classification
- Findings: `AUD-018`, `AUD-019`.
- Affected areas: precipitation-domain validation, weather-text
classification, derived summaries, and focused forecast/facts tests.
- Work: reject precipitation probabilities outside the percentage domain and
classify ordinary icy wording consistently.
- Locked behavior: valid percentage handling, summary ordering, and existing
weather-condition categories remain unchanged.
- Documentation owners: `docs/internal/forecast-derivation.md` and
`docs/internal/facts.md`.
- Exit commands: focused forecast/facts boundary tests, then the standard
repository commands.
### Stage 13: Correct Source-Facing Briefing Module Semantics
- Findings: `AUD-020`, `AUD-021`.
- Affected areas: briefing module option validation, weather-story construction,
module registry tests, and normalized source fixtures.
- Work: make accepted AFD option shapes constructible and omit semantically
empty weather stories instead of publishing false availability.
- Locked behavior: report support, default compositions, checked-empty alerts,
and ordered module snapshots remain unchanged.
- Documentation owners: `docs/internal/module.md` and
`docs/internal/briefing.md`.
- Exit commands: focused briefing/module tests, then the standard repository
commands.
### Stage 14: Correct Derived Briefing Language And Ranking
- Findings: `AUD-023`, `AUD-026`, `AUD-027`.
- Affected areas: derived daily summaries, outdoor-window scoring, temperature
band wording, and briefing tests.
- Work: label apparent temperature accurately; include snow, ice, and fog in
outdoor-window ranking; and preserve negative-zero temperature meaning.
- Locked behavior: deterministic ranking, Today-only planning scope, and
existing module/schema identities remain unchanged.
- Documentation owners: `docs/internal/briefing.md` and prompt-facing field
documentation if names or meaning change.
- Exit commands: focused briefing tests with metric, subzero, and hazard cases,
then the standard repository commands.
### Stage 15: Establish One Daypart Canonicalization Mechanism
- Findings: `AUD-067`.
- Affected areas: duplicated canonical-key helpers in briefing and
generated-text packages plus their behavior tests.
- Work: establish one domain-owned canonical key contract and route both
current consumers through it without changing accepted ASCII behavior.
- Locked behavior: display labels, report-specific semantic roles, ordering,
and current valid output remain unchanged.
- Documentation owners: focused internal briefing/generated-text docs only if
ownership needs explanation.
- Exit commands: focused briefing and generated-text tests, then the standard
repository commands.
### Stage 16: Correct Daypart Identity, Collision, And Display Behavior
- Findings: `AUD-024`, `AUD-025`, `AUD-073`.
- Affected areas: derived daypart summaries, planning builders, display
capitalization, configuration fixtures, and rendering tests.
- Work: reject canonical-key collisions; use configured identities consistently
across planning; and make non-ASCII display capitalization rune-safe.
- Locked behavior: Stage 15's canonical identity remains separate from display
text; stable ordering and default dayparts remain unchanged.
- Documentation owners: `docs/config.md`, `docs/internal/briefing.md`, and
generated-text/template docs if visible labels change.
- Exit commands: focused config/briefing/generated-text/reporttemplate tests,
then the standard repository commands.
### Stage 17: Refresh The Official SPC Definition Asset
- Findings: `AUD-028`.
- Affected areas: embedded SPC category definitions, source attribution, and
risk-digest tests.
- Work: reconcile descriptions with the current authoritative SPC table and
record a durable source/review mechanism without changing risk thresholds
accidentally.
- Locked behavior: deterministic category ordering, threshold selection, and
overlap logic remain unchanged unless current official semantics require an
explicit reviewed correction.
- Documentation owners: `docs/internal/briefing.md` or another single
canonical maintainer owner for the external definition source.
- Exit commands: focused briefing asset/risk tests, then the standard
repository commands.
### Stage 18: Harden Prompt-Debug Redaction
- Findings: `AUD-034`.
- Affected areas: `internal/promptdebug`, Promptkit debug callbacks, endpoint
redaction, and adversarial content tests.
- Work: redact common credential aliases and URL credential forms from every
explicit debug artifact and error projection.
- Locked behavior: debug remains explicit opt-in with `0700` directories,
`0600` files, deterministic references, and no sensitive normal output.
- Documentation owners: `docs/operations.md` and
`docs/integrations/promptkit.md`.
- Exit commands: focused promptdebug/Promptkit redaction tests, then the
standard repository commands.
### Stage 19: Make Prompt-Debug Filesystem Traversal Race-Resistant
- Findings: `AUD-035`.
- Affected areas: `internal/promptdebug`, secure directory and file creation,
Promptkit debug callbacks, and adversarial filesystem tests.
- Work: replace pathname check/use gaps with a race-resistant secure traversal
and creation strategy while preserving atomic replacement and actionable
failures.
- Locked behavior: Stage 18 redaction, explicit opt-in, `0700` directories,
`0600` files, deterministic references, containment, and ordinary concurrent
execution remain unchanged.
- Documentation owners: `docs/operations.md` and
`docs/integrations/promptkit.md`.
- Exit commands: focused promptdebug/Promptkit tests plus
`GOWORK=off go test -race -count=1 ./internal/promptdebug
./internal/adapters/promptkit ./internal/app`, then standard commands.
### Stage 20: Enforce Generated-Text Identity And Shape
- Findings: `AUD-036`, `AUD-037`.
- Affected areas: generated-text catalog, decoders, schemas, typed validators,
and focused tests.
- Work: require report-specific schema/template pairs; reject case-variant
properties that schemas reject.
- Locked behavior: typed normalization, exact registered identities, schema
copy isolation, and valid generated content remain unchanged.
- Documentation owners: `docs/internal/generatedtext.md` and template docs.
- Exit commands: focused generated-text and promptassets tests, then the
standard repository commands.
### Stage 21: Bound Generated Content And Diagnostics
- Findings: `AUD-038`.
- Affected areas: generated-text decoders and typed validators, schemas,
Promptkit result handling, safe error projection, and boundary tests.
- Work: bound prose, arrays, total generated content, and diagnostic context
before allocation or normal error projection.
- Locked behavior: Stage 20 identity/shape enforcement, typed normalization,
valid generated content, and content-safe normal summaries remain unchanged.
- Documentation owners: `docs/internal/generatedtext.md`, template docs, and
Promptkit integration limits where externally relevant.
- Exit commands: focused generated-text/promptassets/app tests at and around
every limit, then the standard repository commands.
### Stage 22: Establish The Prepared Metadata Authority
- Findings: `AUD-069`.
- Affected areas: briefing metadata construction, prepared reports, identity
and timing values, and focused app/briefing tests.
- Work: define one prepared identity/timing authority, populate it at the
preparation boundary, and lock its equivalence to current valid output.
- Locked behavior: prompt and output bytes for valid fixtures, immutable
prepared inputs, and module ordering remain unchanged.
- Documentation owners: `docs/internal/prepared-report.md`,
`docs/internal/briefing.md`.
- Exit commands: focused briefing and app tests, then the standard repository
commands.
### Stage 23: Migrate Prepared Metadata Projections
- Findings: none; this completes the remediation begun in Stage 22.
- Affected areas: prompt metadata projection, render-context inputs, superseded
parallel fields, and focused prompt/render tests.
- Work: route prompt and render projections through Stage 22's authority, then
remove or narrow superseded fields without changing serialized YAML or
rendered Markdown.
- Locked behavior: Stage 22's authority, prompt and output bytes for valid
fixtures, immutable prepared inputs, and module ordering remain unchanged.
- Documentation owners: `docs/internal/prepared-report.md`,
`docs/internal/briefing.md`, and `docs/internal/prompt-input.md`.
- Exit commands: focused briefing/app/promptinput/generatedtext tests, then the
standard repository commands.
### Stage 24: Enforce Render Identity And Empty-Section Behavior
- Findings: `AUD-040`, `AUD-042`.
- Affected areas: render-context builders, Today templates/partials, and
focused context/rendering tests.
- Work: reject conflicting report identities and suppress or replace a Today
daypart section that has no rows or fallback.
- Locked behavior: repository templates remain Markdown owners, deterministic
facts remain authoritative, and normal section order remains unchanged.
- Documentation owners: `docs/templates.md`,
`docs/internal/generatedtext.md`, and `docs/internal/reporttemplate.md`.
- Exit commands: focused generated-text/reporttemplate/app tests, then the
standard repository commands.
### Stage 25: Prevent Generated Prose From Impersonating Markdown Structure
- Findings: `AUD-041`.
- Affected areas: generated prose insertion, Markdown templates/partials,
renderer policy, and adversarial rendering tests.
- Work: define and enforce the smallest content-handling rule that prevents
generated prose from creating structure that impersonates deterministic
report sections.
- Locked behavior: repository templates remain Markdown owners, deterministic
facts remain authoritative, ordinary generated prose remains readable, and
normal section order remains unchanged.
- Documentation owners: `docs/templates.md`,
`docs/internal/generatedtext.md`, and `docs/internal/reporttemplate.md`.
- Exit commands: focused generated-text/reporttemplate/app tests with
adversarial headings and related Markdown constructs, then the standard
repository commands.
### Stage 26: Move Prompt Compatibility Checks Before Collection
- Findings: `AUD-043`.
- Affected areas: application prompt inspection and preparation,
generated-text catalog preflight, collection ordering, and app tests.
- Work: validate catalog compatibility before collection or other expensive
report work begins.
- Locked behavior: one inspection/preparation, profile precedence, immutable
execution copies, and partial execution result ownership remain unchanged.
- Documentation owners: `docs/internal/prepared-report.md`,
`docs/internal/app-orchestration.md`, and Promptkit integration docs.
- Exit commands: focused app and generatedtext tests proving failure ordering,
then the standard repository commands.
### Stage 27: Reconcile Prompt Execution Provenance
- Findings: `AUD-044`.
- Affected areas: prompt inspection, prepared execution values, callbacks,
completed execution results, schema validation, and app tests.
- Work: reconcile inspected, prepared, callback, completed, and schema
identities before accepting generated output.
- Locked behavior: Stage 26 preflight ordering, one inspection/preparation,
profile precedence, immutable execution copies, and partial result ownership
remain unchanged.
- Documentation owners: `docs/internal/prepared-report.md`,
`docs/internal/app-orchestration.md`, and Promptkit integration docs.
- Exit commands: focused app/promptexec/generatedtext tests with mismatched
provenance at each boundary, then the standard repository commands.
### Stage 28: Preflight Single-Report Temporary Names
- Findings: `AUD-046`.
- Affected areas: app output planning, bounded temporary naming, destination
validation, directory creation, and focused filesystem tests.
- Work: detect temporary-name and basename limits during preflight without
leaving newly created directories after failure.
- Locked behavior: destination selection, same-directory atomic visibility,
final filenames, and existing regular-file preservation remain unchanged.
- Documentation owners: `docs/operations.md` and app orchestration internals if
preflight behavior needs clarification.
- Exit commands: focused fileutil/app/CLI long-name tests, then the standard
repository commands.
### Stage 29: Make Single-Report Publication Type-Safe And Cancelable
- Findings: `AUD-045`, `AUD-047`.
- Affected areas: `internal/fileutil`, app output planning/publication,
destination error projection, and filesystem tests.
- Work: reject non-regular final destinations; use bounded temporary naming
from Stage 28; and check cancellation at the rename linearization point.
- Locked behavior: same-directory atomic visibility, preservation of prior
regular files, restrictive new-file mode, truthful committed paths, and
notification-after-publication remain unchanged.
- Documentation owners: `docs/operations.md`, architecture policy if an
invariant needs clarification, and app orchestration internals.
- Exit commands: focused fileutil/app/CLI filesystem tests plus
`GOWORK=off go test -race -count=1 ./internal/fileutil ./internal/app
./internal/cli`, then standard commands.
### Stage 30: Validate Distributor Endpoints Before Publication
- Findings: `AUD-048`.
- Affected areas: Distributor configuration validation, app preflight,
endpoint construction, and focused config/app tests.
- Work: reject unsupported endpoint forms before report publication.
- Locked behavior: exactly-once upload/status mapping, published-file source
selection, token redaction, and dependency types remaining adapter-local.
- Documentation owners: `docs/config.md`, `docs/operations.md`, and Distributor
integration docs.
- Exit commands: focused config and app preflight tests, then the standard
repository commands.
### Stage 31: Bound Distributor Diagnostics And Exercise The HTTP Boundary
- Findings: `AUD-049`, `AUD-050`, `AUD-052`.
- Affected areas: Distributor adapter, pinned client interaction, response
reading and classification, local HTTP tests, and safe batch diagnostics.
- Work: bound response reads; redact or classify remote text before normal
results; and exercise the real production HTTP boundary with a local server.
- Locked behavior: Stage 30 endpoint validation, exactly-once upload/status
mapping, published-file source selection, token redaction, and adapter-local
dependency types remain unchanged.
- Documentation owners: `docs/operations.md`,
`docs/internal/distributor-adapter.md`, and Distributor integration docs.
- Exit commands: focused distributor/app tests including oversized and
malicious response bodies, then the standard repository commands.
### Stage 32: Preserve Batch Cancellation As A Distinct Outcome
- Findings: `AUD-051`.
- Affected areas: batch orchestration, result/error types, CLI projection, and
cancellation tests.
- Work: stop starting later reports after cancellation and preserve
cancellation separately from independent report failures and notification
state.
- Locked behavior: collect once, deterministic planned order, continuation
after non-cancellation failures, report-only counters, and all-success
notification gating remain unchanged.
- Documentation owners: `docs/cli.md`, `docs/operations.md`, and
`docs/internal/app-orchestration.md`.
- Exit commands: focused app/CLI tests plus
`GOWORK=off go test -race -count=1 ./internal/app ./internal/cli`, then
standard commands.
### Stage 33: Tighten Comparison Manifest Compatibility
- Findings: `AUD-053`, `AUD-054`.
- Affected areas: comparison JSON decoding, manifest/logical-bundle validation,
filename helpers, and compatibility tests.
- Work: reject duplicate and case-variant fields and require each successful
report path to equal the canonical ordinal/profile filename.
- Locked behavior: current schema version, deterministic encoding/hashing,
flat exact file set, ordered profiles, and bounded recognition errors remain.
- Documentation owners: `docs/integrations/comparison-bundle.md` and
`docs/internal/comparison-publication.md`.
- Exit commands: focused comparison compatibility tests, then the standard
repository commands.
### Stage 34: Preflight Comparison Sibling Names
- Findings: `AUD-057`.
- Affected areas: comparison destination planning, staging and backup sibling
naming, parent directory creation, and focused filesystem tests.
- Work: validate derived sibling-name constraints during preflight and fail
without leaving residual directories.
- Locked behavior: exact target selection, sibling staging, broad-target
rejection, and valid destination names remain unchanged.
- Documentation owners: comparison publication internals and operations if
preflight behavior needs clarification.
- Exit commands: focused comparison/app long-name tests, then the standard
repository commands.
### Stage 35: Honor Cancellation At Comparison Replacement Commit
- Findings: `AUD-055`.
- Affected areas: comparison replacement transaction, cancellation
linearization, rollback, and deterministic filesystem tests.
- Work: honor cancellation after moving the prior bundle but before installing
the replacement, restoring or retaining the prior bundle according to the
transaction's recovery rules.
- Locked behavior: Stage 34 preflight, post-move authorization, no forced
destructive cleanup, rollback, and committed-new-bundle truth remain.
- Documentation owners: comparison bundle/publication docs and operations.
- Exit commands: focused comparison/app cancellation tests plus
`GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app`,
then the standard repository commands.
### Stage 36: Represent Comparison Backup Recovery Truthfully
- Findings: `AUD-056`.
- Affected areas: comparison backup cleanup, publication result/error state,
app/CLI recovery projection, and deterministic filesystem tests.
- Work: distinguish a complete retained recovery bundle from a partially
removed backup and report the actual recoverable state without deleting the
committed new bundle.
- Locked behavior: post-move authorization, rollback, no forced destructive
cleanup, Stage 35 cancellation behavior, committed-new-bundle truth, and
actionable recovery paths remain.
- Documentation owners: comparison bundle/publication docs, operations, and
CLI result docs where state representation changes.
- Exit commands: focused comparison/app/CLI tests plus
`GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app
./internal/cli`, then standard commands.
### Stage 37: State Executor Concurrency And Preserve Profile Failures
- Findings: `AUD-058`, `AUD-077`.
- Affected areas: `promptexec.Executor` contract, comparison execution/result
aggregation, Promptkit adapter, app fakes, and CLI tests.
- Work: document and test concurrent-call substitutability and preserve
already-completed profile failures when later command cancellation occurs.
- Locked behavior: one shared executor, one goroutine per explicit profile,
provider-owned capacity, joined workers, selection ordering, and no
Distributor notification remain unchanged.
- Documentation owners: Promptkit integration and comparison execution/app
orchestration internals.
- Exit commands: focused promptexec/Promptkit/app/CLI tests plus
`GOWORK=off go test -race -count=1 ./internal/adapters/promptkit
./internal/app ./internal/cli`, then standard commands.
### Stage 38: Make Tests Independent Of Ambient Host State
- Findings: `AUD-059`, `AUD-060`.
- Affected areas: Promptkit credential tests and symlink tests in app,
promptdebug, and comparison packages.
- Work: explicitly control the credential environment and treat inability to
create symlinks as a tested capability prerequisite rather than product
failure.
- Locked behavior: real supported-host symlink safety assertions and missing-
credential behavior remain fully protected.
- Documentation owners: testing policy only if supported-platform policy is
deliberately changed; normally none.
- Exit commands: focused packages under scrubbed and ordinary environments,
race-enabled focused tests, then standard commands.
### Stage 39: Remove Redundant Assertions And Production Test Waits
- Findings: `AUD-061`, `AUD-062`.
- Affected areas: briefing SPC tests, collection wrapper tests, Distributor
polling tests, and narrow timing seams.
- Work: remove the private-constant assertion while retaining observable risk
behavior; inject only the wait controls necessary to avoid production delays
in deterministic tests.
- Locked behavior: production retry/poll intervals and observable SPC policy
remain unchanged.
- Documentation owners: none unless a contributor-facing test seam warrants a
focused internal note.
- Exit commands: focused briefing/collect/distributor tests with repeated runs,
then the standard repository commands.
### Stage 40: Make Comparison Test Failure Cleanup Total
- Findings: `AUD-064`.
- Affected areas: comparison execution test barriers and worker cleanup in
`internal/app` tests.
- Work: release and join workers on timeout, early completion, duplicate start,
and every diagnostic exit without weakening liveness detection.
- Locked behavior: production comparison scheduling and five-second deadlock
diagnostics remain unchanged.
- Documentation owners: none.
- Exit commands: repeat focused comparison-execution tests and run them with
`-race`, then the standard repository commands.
### Stage 41: Remove The Redundant Weather Warmup Fetch
- Findings: `AUD-065`.
- Affected areas: Weather API warmup and current-conditions acquisition,
retry semantics from Stage 7, request counts, and adapter/collection tests.
- Work: reuse the successful warmup response or remove the redundant warmup so
current conditions are not fetched and decoded twice.
- Locked behavior: source-local validation/errors, exact source order,
required/optional policy, Stage 7 retry policy, and context cancellation
remain authoritative.
- Documentation owners: Weather API integration and collection internals if
operational request behavior changes materially.
- Exit commands: focused request-count and retry tests, then the standard
repository commands.
### Stage 42: Acquire Independent Weather Sources Concurrently
- Findings: none; this completes the efficiency remediation begun in Stage 41.
- Affected areas: Weather API bundle acquisition, request-result aggregation,
provenance ordering, cancellation, and adapter/collection tests.
- Work: execute independent source requests concurrently and merge their
results in one deterministic source order.
- Locked behavior: Stage 41 current-response handling, source-local
validation/errors, exact source order, required/optional policy, Stage 7
retry policy, and context cancellation remain authoritative.
- Documentation owners: Weather API integration and collection internals if
operational request behavior changes materially.
- Exit commands: focused latency, deterministic-order, failure, and
cancellation tests plus
`GOWORK=off go test -race -count=1 ./internal/adapters/weatherapi
./internal/collect`, then standard commands.
### Stage 43: Eliminate Repeated Comparison Bundle Reads
- Findings: `AUD-066`.
- Affected areas: comparison recognition/planning/publication and app
comparison preflight.
- Work: pass or cache only the recognition evidence needed to avoid four full
reads while retaining reauthorization at every mutation boundary.
- Locked behavior: early rejection, exact file/type validation, post-move
authorization, deterministic hashes, and concurrent destination safety after
Stages 34-36 remain unchanged.
- Documentation owners: comparison publication internals if ownership or
transaction phases change.
- Exit commands: focused read-count and adversarial replacement tests plus
`GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app`, then
standard commands.
### Stage 44: Consolidate Distributor Template Parsing
- Findings: `AUD-068`.
- Affected areas: single and batch notification template rendering and config
tests.
- Work: extract one private grammar/parser with typed callbacks while
preserving each template family's variable allowlist and error context.
- Locked behavior: accepted syntax, rejected malformed forms, exact rendering,
and single-versus-batch policy remain unchanged.
- Documentation owners: none unless the user-visible grammar changes, which is
outside this behavior-preserving stage.
- Exit commands: focused config tests, then the standard repository commands.
### Stage 45: Define One Fact Requirement Vocabulary
- Findings: `AUD-070`.
- Affected areas: module requirement vocabulary, current briefing definitions,
identity/category metadata, and focused module/briefing tests.
- Work: define each requirement's identity and category once, preserve each
module's explicit requirement list, and lock equivalence with current valid
compositions.
- Locked behavior: all current compositions, supported reports, availability,
and diagnostic identity remain byte-for-byte or semantically unchanged.
- Documentation owners: module and briefing internals if ownership changes.
- Exit commands: focused module and briefing catalog tests, then the standard
repository commands.
### Stage 46: Migrate Fact Availability And Constructor Validation
- Findings: none; this completes the remediation begun in Stage 45.
- Affected areas: availability predicates, briefing constructors, legacy
switches, configuration validation, and focused tests.
- Work: route availability and constructor validation through Stage 45's
requirement vocabulary and remove superseded policy implementations.
- Locked behavior: all current compositions, supported reports, missing-data
policy, availability, and diagnostic identity remain unchanged.
- Documentation owners: module and briefing internals if ownership changes.
- Exit commands: focused module/briefing/config tests, then the standard
repository commands.
### Stage 47: Consolidate CLI Report-Date Policy
- Findings: `AUD-071`.
- Affected areas: generate/comparison flag registration and date resolution,
CLI parser/action tests, and private CLI descriptors.
- Work: establish one private report-date policy/resolver used by both actions
after Stage 6 has locked corrected behavior.
- Locked behavior: accepted flags, required/default dates, action-specific
errors, and request types remain unchanged.
- Documentation owners: none unless implementation reveals a current contract
mismatch; CLI docs remain authoritative.
- Exit commands: focused CLI parser/action tests and help commands, then the
standard repository commands.
### Stage 48: Retire Dormant Prompt And Generated-Text Surfaces
- Findings: `AUD-072`.
- Affected areas: unused prompt YAML loading/saving, normalized generated JSON
returns, and their sole-purpose tests and documentation.
- Work: verify these prompt/generated-text compatibility surfaces still have no
production owner, then remove them. If a production consumer appears, stop
and replace this retirement sequence with a separately approved contract.
- Locked behavior: normal Build/Validate/MarshalYAML, report publication,
comparison data packages, prompt debugging, typed normalization, and all
workflows remain unchanged.
- Documentation owners: prompt-input and generated-text internal documents;
remove stale descriptions rather than documenting retired APIs.
- Exit commands: graph/text caller checks, focused promptinput/generatedtext
tests, then the standard repository commands.
### Stage 49: Retire Dormant Weather Persistence Helpers
- Findings: none; this continues the dormant-surface remediation begun in
Stage 48.
- Affected areas: Weather API bundle saving, the JSON atomic helper if it
becomes orphaned, and sole-purpose tests and documentation.
- Work: verify these persistence helpers still have no workflow owner, then
remove them and any helper made unreachable by that removal. If a production
consumer appears, stop and seek a separately approved contract.
- Locked behavior: weather collection, normalized in-memory bundles, prompt
debugging, operator-owned publication, and all workflows remain unchanged.
- Documentation owners: weather-data and collection internals where stale
persistence claims exist.
- Exit commands: graph/text caller checks, focused weatherapi/weatherdata/
fileutil tests, then the standard repository commands.
### Stage 50: Retire Dormant Module And Forecast Compatibility Exports
- Findings: none; this continues the dormant-surface remediation begun in
Stage 48.
- Affected areas: module snapshot compatibility implications, unused forecast
threshold exports, and sole-purpose tests and documentation.
- Work: verify the identified exports still have no production owner, then
remove them without broadening the supported compatibility contract. If a
production consumer appears, stop and seek a separately approved contract.
- Locked behavior: current module snapshots, real forecast decisions, prompt
exports, report rendering, and all workflows remain unchanged.
- Documentation owners: module, briefing, and forecast internals where stale
compatibility claims exist.
- Exit commands: graph/text caller checks, focused module/briefing/forecast
tests, then the standard repository commands.
### Stage 51: Narrow Dormant Alert Envelope Copies
- Findings: none; this completes the dormant-surface remediation begun in
Stage 48.
- Affected areas: unused full alert-envelope copies, retained alert item
provenance, and sole-purpose tests and documentation.
- Work: remove unconsumed alert-envelope data while retaining exactly the item
and provenance values required by current facts, briefing, and report paths.
If a production consumer appears, stop and seek a separately approved
contract.
- Locked behavior: alert selection, alert item provenance, briefing digests,
report rendering, and all workflows remain unchanged.
- Documentation owners: weather-data, facts, and briefing internals where the
narrowed value is described.
- Exit commands: graph/data-flow checks plus focused weatherdata/facts/
briefing tests, then the standard repository commands.
### Stage 52: Consolidate Generated-Text Test Ownership
- Findings: `AUD-039`.
- Affected areas: generated-text day-style and render-context tests,
promptassets schema tests, and reporttemplate tests.
- Work: keep one shared semantic suite plus thin report-identity cases; put
schema ownership in promptassets and Markdown ownership in reporttemplate;
retain one real typed-context render integration.
- Locked behavior: every public validator identity, distinct Hourly behavior,
schema/type agreement, typed projection, template semantics, and composition
remain protected.
- Documentation owners: none.
- Exit commands: focused promptassets/generatedtext/reporttemplate tests, then
the standard repository commands.
### Stage 53: Reconcile Stale Internal Implementation Guides
- Findings: `AUD-075`.
- Affected areas: forecast derivation, collection, report registry, and prompt-
input internal guides plus incoming links.
- Work: replace nonexistent API/flow/test claims with concise descriptions of
the implemented owners and link to canonical contracts instead of duplicating
volatile details.
- Locked behavior: documentation-only; no executable contract changes.
- Documentation owners: the four affected `docs/internal/` files and
`docs/development.md` only if task routing changes.
- Exit commands: repository link/fence checks, focused test inventories used to
verify claims, `GOWORK=off go test -count=1 ./...`, and `git diff --check`.
### Stage 54: Retire Completed Comparison Roadmaps
- Findings: `AUD-076`.
- Affected areas: `docs/roadmap/profile-comparison.md`,
`docs/roadmap/implementation.md`, and all incoming links.
- Work: verify useful implemented contracts have one durable canonical owner,
repair incoming navigation, then remove the completed roadmaps so they no
longer duplicate current behavior.
- Locked behavior: documentation lifecycle only; no code, tests, dependencies,
schemas, or current contracts change.
- Documentation owners: CLI, operations, comparison integration/publication,
comparison execution, Promptkit integration, architecture, and development
docs only where an incoming link or missing durable fact requires correction.
- Exit commands: repository link/fence checks, both CLI help commands,
`GOWORK=off go test -count=1 ./...`, and `git diff --check`.
### Stage 55: Verify Remediation Completion And Durable Ownership
- Findings: none; this is the roadmap lifecycle gate.
- Prerequisite: Stages 1-54 are complete and every referenced finding's tests
and canonical documentation have landed.
- Affected areas: every remediation completion note, canonical documentation
owner, audit finding, retained decision, and incoming audit-document link.
- Work: verify every finding is resolved, superseded, or explicitly moved to a
newly approved roadmap; verify durable rationale lives with its canonical
owner; and produce a concise removal checklist for the temporary audit plan,
ledger, and remediation roadmap. Do not remove those records in this stage.
- Locked behavior: documentation lifecycle only; do not rewrite historical
ADRs or release notes and do not create a second current-state reference.
- Documentation owners: all durable owners touched by earlier stages, solely
to fill a demonstrated ownership gap.
- Exit commands: verify every stage/finding mapping and completion note;
repository link/fence and secret-pattern checks; all focused stage exits;
`GOWORK=off go test -count=1 ./...`,
`GOWORK=off go test -race -count=1 ./...`, both CLI help commands,
`GOWORK=off go vet ./...`, and `git diff --check`.
### Stage 56: Retire The Completed Audit Records
- Findings: none; this is the final documentation lifecycle change.
- Prerequisite: Stage 55 has verified completion and supplied the exact removal
and link-repair checklist.
- Affected areas: `docs/roadmap/audit.md`, `audit-plan.md`, this roadmap, and
their incoming links.
- Work: apply Stage 55's checklist, repair incoming navigation, and remove the
temporary audit plan, ledger, and remediation roadmap without changing
executable behavior or historical records.
- Locked behavior: documentation lifecycle only; durable current contracts and
rationale remain with their canonical owners, and ADRs and release notes are
unchanged.
- Documentation owners: only the navigation or durable owner files identified
by Stage 55.
- Exit commands: repository link/fence and secret-pattern checks;
`GOWORK=off go test -count=1 ./...`,
`GOWORK=off go test -race -count=1 ./...`, both CLI help commands,
`GOWORK=off go vet ./...`, and `git diff --check`.
## Completion Tracking
Update a stage heading or add a concise completion note only after its code,
tests, canonical documentation, and exit commands pass together. A stage that
begins a multi-stage remediation sequence must not mark its finding resolved
until the final continuation stage succeeds. Do not copy implemented contracts
into this roadmap. When all work is complete, follow Stages 55 and 56 rather
than retaining this file as a parallel reference.

398
docs/roadmap/profile-comparison.md
View File

@@ -1,398 +0,0 @@
# LLM Profile Comparison Roadmap
Status: Implemented; retained as the feature roadmap.
## Purpose
Prompt development currently requires separate Weatherreporter invocations to
compare several LLM profiles. Those invocations may collect different weather
snapshots or rebuild inputs at different times, making model output harder to
compare and slowing prompt iteration.
Weatherreporter should provide a first-class `compare` command that resolves
one report, prepares one exact data package, executes the same prompt and data
package concurrently through several explicitly selected Promptkit profiles,
and publishes a self-contained local comparison bundle.
An illustrative invocation is:
```sh
weatherreporter compare daily \
--date 2026-08-24 \
--profile weather-light \
--profile weather-balanced \
--profile weather-deep
```
This is a prompt-development workflow, not an automated model evaluator. Its
output gives a maintainer consistent evidence for human comparison without
assigning scores or selecting a winner.
## Prerequisite
Configurable output directories are implemented. Profile comparison must reuse
the current [configuration reference](../config.md) and [operations
guide](../operations.md) rather than introduce a second destination policy.
## User Intent
The command is intended for deliberate evaluation of multiple profiles,
including sets of eight to twelve candidate models. Concurrency is part of the
feature, not a future optimization. Promptkit should retain ownership of
backend-specific capacity, while Weatherreporter owns comparison-wide
coordination, cancellation, deterministic results, and artifact publication.
Every profile must receive byte-for-byte identical prompt input. Weather data,
derived facts, modules, prompt metadata, and serialized YAML must not be
recollected or rebuilt separately for individual profiles.
Comparison bundles are explicitly requested, operator-owned development
outputs. They are not Weatherreporter state, are never read implicitly by a
later run, and do not weaken the ordinary stateless execution model.
## Command Contract
The command form is:
```text
weatherreporter compare REPORT [options]
```
`REPORT` accepts the implemented generated-text reports: `daily`, `today`,
`tomorrow`, and `hourly`. Report-date behavior matches `generate`: `daily`
requires `--date`, `today` may accept an explicit date or use the current local
date, and the remaining report types retain their existing period policies.
The command accepts the applicable common generation options, including
`--config`, `--units`, `--tz`, `--date`, `--llm-debug-dir`, and `--quiet`, plus:
- repeatable `--profile PROFILE_ID` selections;
- `--out-dir PATH` for the exact comparison-bundle directory; and
- `--replace` to authorize guarded replacement of a recognized existing
comparison bundle.
At least two distinct, nonblank profile IDs are required. Their command-line
order is significant and is preserved in filenames, summaries, and
`comparison.json`. Duplicate profile IDs are rejected rather than silently
deduplicated or executed twice.
Profiles are always explicit for this command. `promptkit.profile` does not add
or replace a comparison selection, but all other effective Promptkit settings,
profile-source precedence, local backend configuration, credential lookup, and
profile overrides remain in force.
The initial feature has no Weatherreporter-specific concurrency flag or
artificial profile-count ceiling. The explicit profile list bounds the
comparison, and Promptkit owns capacity enforcement for each selected backend.
## Preparation And Execution Invariants
A comparison has this logical lifecycle:
1. Parse and validate the report, date, profile list, configuration, output
destination, and replacement authorization.
2. Resolve the report definition, valid period, prompt identity, and default
output name once.
3. Inspect the exact prompt once and preflight every selected profile,
including its effective backend, model, and required credential
availability, before weather collection.
4. Collect weather data exactly once.
5. Build collected and derived facts, the module snapshot, briefing metadata,
and the prompt data package exactly once.
6. Marshal the data package to one immutable YAML byte sequence exactly once.
7. Execute the exact prompt version concurrently for every selected profile,
passing the same immutable YAML bytes to every execution.
8. Validate and render each profile result independently from the shared
deterministic inputs.
9. Assemble results in requested-profile order and publish one coherent
comparison bundle.
This lifecycle describes the required end-state behavior rather than an
implementation-stage sequence.
No profile execution may cause recollection, report re-resolution, module
rebuilding, or data-package remarshalling. Prompt execution may perform
Promptkit-owned validation or repair behavior, but Weatherreporter does not
retry a failed comparison execution independently.
## Concurrency And Cancellation
Weatherreporter starts one execution for each preflighted profile and permits
them to run concurrently through one shared, concurrency-safe Promptkit
executor. Promptkit's engine-local backend pools remain authoritative for
backend concurrency and waiting capacity. Profiles routed to a limited local
backend therefore respect its configured limit, while profiles routed to
other backends may proceed independently.
Weatherreporter must not add a second semaphore that obscures or overrides
Promptkit's backend policy. It must safely coordinate goroutine lifecycles,
result collection, debug callbacks, and output assembly without data races.
One profile failure does not cancel its peers. Provider, capacity, validation,
and rendering failures are recorded for that profile while other executions
continue. Cancellation or deadline expiration of the comparison command is
propagated to every outstanding execution, prevents new publication, and is
joined without leaking goroutines.
Completion order must not affect filenames, manifest order, CLI summaries, or
error aggregation. Those outputs always follow the original `--profile`
order.
## Output Destination
Without `--out-dir`, Weatherreporter derives a comparison directory from the
resolved report's existing default Markdown filename by removing `.md` and
prefixing `comparison-`:
| Report output | Comparison directory |
| --- | --- |
| `today.md` | `comparison-today/` |
| `tomorrow.md` | `comparison-tomorrow/` |
| `hourly.md` | `comparison-hourly/` |
| `daily-2026-08-24.md` | `comparison-daily-2026-08-24/` |
The derived directory is created beneath `output.directory` when configured,
or beneath the present working directory otherwise. An explicit `--out-dir`
is the exact bundle directory, resolves relative to the present working
directory when necessary, and overrides `output.directory` completely.
All destination selection and validation completes before weather collection.
The resolved comparison directory is returned in the command's structured
result.
## Comparison Bundle
A successful three-profile comparison has a flat layout:
```text
comparison-daily-2026-08-24/
├── comparison.json
├── data-package.yml
├── 01-weather-light.md
├── 02-weather-balanced.md
└── 03-weather-deep.md
```
`data-package.yml` contains the exact YAML bytes passed to every Promptkit
execution. It is written once and its SHA-256 digest is recorded in the
manifest.
Each report filename begins with its one-based, zero-padded selection position
and a filesystem-safe representation of the requested logical profile ID. The
safe representation must not permit absolute paths, traversal, separators, or
control characters. The manifest retains the exact case-sensitive profile ID,
so filename normalization never becomes the authority for profile identity.
`comparison.json` is the authoritative index for the bundle. It uses an
explicit schema version and records safe comparison information including:
- comparison identity and start and finish timestamps;
- report ID, resolved valid period, and effective timezone;
- prompt ID, version, and inspected prompt hash;
- the relative data-package filename and SHA-256 digest;
- total, succeeded, and failed profile counts; and
- one ordered result per requested profile containing the exact profile ID,
resolved backend and model, relative report filename when present,
execution and validation status, and safe error information when failed.
The manifest and normal command summary must not contain credentials, provider
request bodies, raw model output, rendered prompts, schemas, provider
endpoints, or other content-rich diagnostics. The explicit data package and
generated reports contain the development material the user requested and
must be handled as operator-owned potentially sensitive output.
## Failure And Publication Policy
Failure before concurrent execution, including invalid profiles, missing
credentials, collection failure, preparation failure, or unsafe destination,
publishes no comparison bundle and performs no model calls where the failure
is discoverable during preflight.
After execution begins, Weatherreporter waits for every non-cancelled profile.
If one or more profiles fail, it still publishes a coherent partial bundle
containing `data-package.yml`, every successfully rendered report, and a
manifest describing all successes and failures. It then returns a non-zero
exit status. A failed profile has no report file unless a future contract
explicitly introduces a separately named diagnostic artifact.
Bundle contents are staged outside the destination and published only after
the manifest is complete. Ordinary publication accepts only an absent or empty
target directory. A nonempty existing directory fails without modification
unless `--replace` is present.
`--replace` may replace only the exact resolved target and must reject broad or
unsafe targets such as a filesystem root, the present working directory, a
symlink, or an unrecognized nonempty directory. A recognized prior bundle must
contain a valid Weatherreporter comparison manifest. Replacement publishes the
new complete or coherent partial bundle as a unit, prevents stale reports from
the prior comparison from surviving, and preserves or restores the prior
bundle if the final replacement operation fails.
An interrupted or cancelled comparison does not replace an existing bundle.
Temporary staging artifacts are cleaned up on ordinary failure and
cancellation without scanning or modifying unrelated directories.
## Prompt Debugging
The existing `--llm-debug-dir` mechanism remains available. Concurrent
comparison executions require distinct, deterministic debug identities that
include the comparison and exact profile selection so callbacks cannot collide
or overwrite another profile's artifacts.
Debug writing must be concurrency-safe and retain the existing permission,
redaction, explicit-opt-in, and path-containment guarantees. Debug artifacts
remain separate from the comparison bundle; the bundle does not implicitly
enable full Promptkit diagnostics.
## Notification Policy
Profile comparisons never invoke Distributor notification, even when
notification is enabled in the effective configuration. Comparison reports
are local development artifacts rather than ordinary report publications.
Adding comparison publication or upload behavior would require a separate
accepted feature scope and explicit operator authorization.
## Architectural End State
Application orchestration exposes a reusable prepared-report boundary that
contains the resolved report, shared collected and derived facts, module
snapshot, briefing metadata, generated-text handler, render inputs, and exact
serialized data package. That boundary is immutable during concurrent profile
execution.
Ordinary `generate` behavior continues to prepare once and execute once.
`compare` prepares once and executes many without duplicating the generation
workflow or calling `GenerateDetailed` in a loop. Shared preparation,
profile-specific Promptkit execution, structured-output validation, rendering,
and artifact publication remain distinct responsibilities.
The Promptkit adapter remains the only owner of dependency-specific types and
engine calls. The CLI owns parsing and user-facing summaries. The configuration
package owns configuration. Application orchestration owns comparison order,
concurrency lifecycle, failure aggregation, and bundle publication. Domain,
prompt-input, generated-text, and template packages retain their existing
deterministic contracts.
## Scope
The completed feature includes:
- the `compare` CLI command for every implemented generated-text report;
- repeatable explicit profile selection and validation;
- configured and CLI output-directory integration through the implemented
destination policy;
- one-time report resolution, collection, deterministic preparation, and YAML
serialization;
- concurrent execution through one Promptkit executor with backend capacity
respected;
- independent validation and rendering with deterministic ordered results;
- the flat, versioned comparison-bundle contract;
- safe filename derivation and data-package hashing;
- coherent partial-result publication and non-zero failure behavior;
- guarded whole-bundle replacement through `--replace`;
- comparison-aware, concurrency-safe optional prompt debugging;
- explicit suppression of Distributor notification;
- structured normal and quiet-mode CLI behavior consistent with existing
commands;
- focused race-safe tests across configuration, CLI, application,
Promptkit-adapter, rendering, and filesystem boundaries; and
- updates to every affected canonical user, operator, architecture,
integration, and internal document.
## Compatibility
The feature is additive. Existing `generate` and `run` commands, report
definitions, profile defaults, configuration, output filenames, notification
behavior, and exit contracts remain unchanged.
The comparison manifest and bundle layout begin as versioned contracts. They
do not become inputs accepted by Weatherreporter, and no backward-compatible
replay or long-term archive guarantee is implied beyond identifying the schema
used to interpret a produced bundle.
## Testing Expectations
Tests should provide durable coverage for:
- report and date parsing consistent with `generate`;
- rejection of fewer than two profiles, blanks, and duplicates;
- inspection of the exact prompt and every profile before collection;
- no collection or model execution after a preflight failure;
- exactly one weather collection and one preparation for several profiles;
- byte-for-byte identical data-package input in every execution;
- observable concurrent execution through a concurrency-safe fake executor;
- respect for Promptkit-owned backend capacity in an assembled adapter test
where that integration adds distinct confidence;
- deterministic filenames, manifest order, summaries, and errors under varied
completion order;
- continuation and coherent partial publication after one profile fails;
- cancellation propagation, goroutine completion, and preservation of an
existing destination;
- destination precedence and each derived default directory;
- safe filename handling for unusual valid profile IDs;
- absent, empty, occupied, symlinked, unsafe, recognized, and unrecognized
replacement targets;
- removal of stale prior report files during authorized whole-bundle
replacement;
- exact package digest and manifest/result consistency;
- concurrency-safe, non-colliding opt-in debug artifacts; and
- absence of Distributor calls for complete and partial comparisons.
Concurrency and replacement behavior require race-enabled and consequential
failure-path coverage. Tests must remain deterministic, offline, credential
free, and independent of real Promptkit providers or machine-specific paths.
## Documentation End State
Once implemented, the [CLI reference](../cli.md) owns command syntax, flags,
summary, and exit behavior. The [operations guide](../operations.md) owns the
bundle lifecycle, replacement procedure, sensitivity guidance, and practical
prompt-comparison workflow. The [architecture policy](../policy/architecture.md)
owns the statelessness, concurrency, notification, and publication invariants.
The [Promptkit integration guide](../integrations/promptkit.md) should describe
the consumer-visible multi-profile execution boundary without duplicating
Promptkit's backend-capacity reference. The versioned manifest and flat bundle
format belong in a focused contract under `docs/integrations/`. App
orchestration, prompt input, generated text, prompt debugging, and bundle
publication mechanics belong in focused documents under `docs/internal/`.
Current-state documentation must not describe profile comparison as available
until the implementation lands.
## Non-Goals
This roadmap does not introduce:
- automatic model scoring, ranking, recommendation, or winner selection;
- semantic or textual diff generation between reports;
- repeated sampling of one profile or statistical evaluation;
- prompt or profile editing through Weatherreporter;
- replaying a saved data package as command input;
- comparing several report types in one command;
- Weatherreporter-owned backend concurrency or queue configuration;
- automatic retries beyond Promptkit's existing execution contract;
- Distributor upload or other external publication;
- comparison history, indexing, retention, cleanup schedules, or implicit
discovery of prior bundles; or
- changes to ordinary report content or normal generation behavior.
Any later automated evaluation, replay, sampling, or publication feature
requires a separate accepted roadmap.
## Completion Criteria
The feature is complete when a maintainer can select several Promptkit
profiles, have them execute concurrently against one exact prepared report
package, and receive a safe, flat, deterministic comparison bundle whose
manifest accurately describes every success and failure. Configured and
explicit destinations must follow the accepted output policy, replacement must
never mix or silently destroy unrelated contents, cancellation and partial
failure must be race-safe, ordinary notification must remain disabled, and all
affected canonical documentation must describe the implemented behavior.
## Open Questions
None. The scope, prerequisites, user intent, and target behavior are defined
above.