diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 01e2fca..1b0d62c 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -87,6 +87,8 @@ directly. outcomes only; a failed batch notification is represented separately at the batch level. - Comparison never invokes Distributor notification. +- Profile comparison supports operator review only: it does not score, rank, + select, resample, or replay profile executions. - Default tests are deterministic, offline, and use Promptkit/provider fakes rather than live provider calls. See the [testing policy](testing.md). diff --git a/docs/roadmap/audit.md b/docs/roadmap/audit.md index 98217fa..2ff2166 100644 --- a/docs/roadmap/audit.md +++ b/docs/roadmap/audit.md @@ -222,8 +222,8 @@ investigation stage. Remediation acceptance, priority, and implementation sequencing remain exclusively Stage 27 work. Stage 27 accepted all 74 confirmed findings for implementation and allocated -each exactly once across the dependency-ordered -[audit remediation roadmap](audit-remediation.md). No confirmed finding is +each exactly once across the dependency-ordered audit remediation roadmap. No +confirmed finding is priority-deferred or blocked on a separate product/architecture decision. The roadmap places generated-forecast integrity, security, correctness, data integrity, destructive operations, and cancellation before efficiency, diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index fb24e1c..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,910 +0,0 @@ -# Repository Audit Remediation Roadmap - -Status: Ready for implementation. - -## Purpose And Authority - -This roadmap translates the confirmed findings in the -[repository audit ledger](audit.md) into dependency-ordered implementation -stages. The ledger owns discovery evidence and final finding disposition. This -document owns remediation scope, ordering, locked behavior, verification, and -completion state until the work is finished. - -All 74 confirmed findings are accepted for remediation. None is priority- -deferred and none requires a separate product or architecture decision before -work can begin. Four rejected standalone findings remain consolidated into -their confirmed owners and are not separate work items. - -This roadmap describes future work. Until an individual stage is implemented, -canonical current-state documentation and executable behavior remain -authoritative. - -## Implementation Rules - -Every stage must: - -- read `docs/development.md`, all files under `docs/policy/`, this roadmap, and - the task-specific canonical documents identified below; -- implement only its listed findings and prerequisites; -- preserve the architecture policy's stateless pipeline, adapter boundaries, - curated prompt boundary, operator-owned outputs, comparison ordering, and - notification rules unless the stage explicitly corrects that contract; -- add durable tests at the narrowest stable boundary and avoid duplicating - behavior already owned elsewhere; -- update every listed canonical documentation owner in the same change as an - implemented contract change; -- use deterministic, offline, credential-free tests and synthetic fixtures; -- run `gofmt` on changed Go files, the stage's focused commands, - `GOWORK=off go test -count=1 ./...`, and `git diff --check`; and -- commit the completed stage separately when the implementing prompt requests - commits. - -Stages affecting concurrent execution, cancellation, filesystem mutation, or -process-global state must also run the affected packages with `-race`. No stage -may weaken a safety invariant merely to make a test pass. - -## Priority And Finding Allocation - -Each confirmed finding appears exactly once below. Priority describes risk and -the default remediation order, not confidence; dependency prerequisites may -place a lower-priority consolidation mechanism before the P1 correction that -depends on it. Every listed finding has high confidence. - -| Priority | Stages | Findings | Decision | -| --- | --- | --- | --- | -| P0: generated forecast integrity | 1 | `AUD-029`, `AUD-030`, `AUD-031`, `AUD-033` | Accepted now | -| P1: security, correctness, and data integrity | 2-14, 16-21, 24-37 | `AUD-002` through `AUD-021` excluding rejected `AUD-022`; `AUD-023` through `AUD-028`; `AUD-034` through `AUD-038`; `AUD-040` through `AUD-058`; `AUD-073`, `AUD-074`, `AUD-077`, and `AUD-078` | Accepted now | -| P2: deterministic test and operational quality | 38-43 | `AUD-059` through `AUD-062`, `AUD-064`, `AUD-065`, and `AUD-066` | Accepted now | -| P3: behavior-preserving consolidation and cleanup | 15, 22-23, 44-52 | `AUD-039`, `AUD-067` through `AUD-072` | Accepted now; Stages 15 and 22-23 run early because P1 corrections depend on them | -| P4: documentation lifecycle and audit retirement | 53-56 | `AUD-075` and `AUD-076`; final audit-record retirement | Accepted now | - -Rejected `AUD-001`, `AUD-022`, and `AUD-032` are represented only through -`AUD-072`; rejected `AUD-063` is represented only through `AUD-039`. - -## Remediation Stages - -### Stage 1: Correct The Curated Prompt Package Contract - -- Findings: `AUD-029`, `AUD-030`, `AUD-031`, `AUD-033`. -- Affected areas: `internal/promptinput`, shared embedded prompt assets, and - focused prompt-input/prompt-asset tests. -- Work: remove raw transport-shaped warning details from provider input; make - shared prompt paths match serialized YAML; describe regional versus point- - matched hazard products accurately; and cover stanza serialization failure. -- Locked behavior: deterministic category/stanza order, exact prompt identity, - safe normal diagnostics, and byte-stable valid YAML remain unchanged. -- Documentation owners: `docs/internal/prompt-input.md` and - `docs/integrations/promptkit.md` if their implemented contract changes. -- Exit commands: focused tests for `./internal/promptinput` and - `./internal/promptassets`, then the standard repository commands. - -### Stage 2: Make Configuration Keys And Overrides Explicit - -- Findings: `AUD-004`, `AUD-006`, `AUD-078`. -- Affected areas: `internal/config`, report-key lookup, timezone parsing, - maintained examples, and configuration tests. -- Work: reject unsupported missing-source keys; make programmatic report - overrides obey the same presence semantics as loaded configuration; and - document accepted report-key normalization and numeric-offset bounds. -- Locked behavior: file/CLI precedence, inactive-integration policy, exact - command-name lookup, and secret redaction remain unchanged. -- Documentation owners: `docs/config.md` and maintained examples only where - executable behavior requires them. -- Exit commands: focused config and CLI tests, followed by the standard - repository commands. - -### Stage 3: Make Secret Loading Atomic - -- Findings: `AUD-007`. -- Affected areas: configuration loading, secret-directory application, load - failure state, and focused configuration tests. -- Work: stage secret values and apply them to the returned configuration only - after the complete load and validation sequence succeeds. -- Locked behavior: secret precedence, configured secret-directory discovery, - environment independence, actionable errors, and redaction remain unchanged. -- Documentation owners: `docs/config.md` only if the observable load contract - needs clarification. -- Exit commands: focused config tests plus - `GOWORK=off go test -race -count=1 ./internal/config`, followed by the - standard repository commands. - -### Stage 4: Require Complete Single-Report Notification Identity - -- Findings: `AUD-005`. -- Affected areas: Distributor notification configuration, bundle-ID rendering, - and focused config/app tests. -- Work: reject enabled single-report notification configurations that can - produce blank required identity fields before report work begins. -- Locked behavior: inactive notification settings remain optional; configured - templates and publication-before-notification ordering remain intact. -- Documentation owners: `docs/config.md`, `docs/operations.md`, and Distributor - integration docs if accepted values or failures change. -- Exit commands: focused config and app notification tests, then the standard - repository commands. - -### Stage 5: Propagate Process Interrupt Cancellation - -- Findings: `AUD-008`. -- Affected areas: `cmd/weatherreporter`, action contexts, CLI wiring, and - interrupt/cancellation tests. -- Work: convert operating-system interrupts into action-context cancellation - and verify active generation, batch, and comparison workflows observe it. -- Locked behavior: command/flag ownership, structured summary shapes, stream - separation, and one executor per action remain unchanged. -- Documentation owners: `docs/cli.md`, `docs/operations.md`, and - `docs/internal/cli.md` if signal behavior needs clarification. -- Exit commands: `GOWORK=off go test -race -count=1 ./internal/cli - ./internal/app ./cmd/weatherreporter`, then the standard repository commands. - -### Stage 6: Correct CLI Preflight, Coverage, And Visible Contracts - -- Findings: `AUD-009`, `AUD-010`, `AUD-074`. -- Affected areas: `internal/cli`, executor construction, assembled CLI tests, - help text, and CLI documentation. -- Work: validate report dates before executor construction; add representative - generate and batch result/exit tests; and align help/internal guidance with - quiet, date, and comparison behavior. -- Locked behavior: command/flag ownership, structured summary shapes, stream - separation, cancellation from Stage 5, and one executor per action remain - unchanged. -- Documentation owners: `docs/cli.md` and `docs/internal/cli.md`. -- Exit commands: focused CLI/app tests, both root and comparison help commands, - then the standard repository commands. - -### Stage 7: Validate Weather API Endpoints And Retry Classification - -- Findings: `AUD-011`, `AUD-015`. -- Affected areas: Weather API configuration and adapter construction, warmup - retry classification, local HTTP fixtures, and focused tests. -- Work: accept only supported endpoint forms and retry warmup only for - transient failures. -- Locked behavior: all eight source contracts, checked-empty optional products, - cancellation, provenance, and offline testing remain intact. -- Documentation owners: `docs/config.md`, `docs/integrations/weatherapi.md`, - and `docs/internal/collect.md` as applicable. -- Exit commands: focused configuration, adapter, and collection tests, then the - standard repository commands. - -### Stage 8: Bound And Redact Weather API Diagnostics - -- Findings: `AUD-012`, `AUD-013`. -- Affected areas: Weather API response reading, non-success error projection, - size limits, local HTTP fixtures, and focused adapter tests. -- Work: replace upstream response text with bounded safe diagnostics and reject - rather than silently truncate oversized bodies. -- Locked behavior: HTTP status classification, source-local error context, - cancellation, and offline tests remain intact. -- Documentation owners: `docs/integrations/weatherapi.md` and - `docs/internal/collect.md` if observable limits or diagnostics change. -- Exit commands: focused boundary-size and malicious-response adapter tests, - then the standard repository commands. - -### Stage 9: Require Usable Hourly Period Bounds - -- Findings: `AUD-014`. -- Affected areas: Weather API hourly decoding, normalized hourly data, - required-source validation, fixtures, and focused tests. -- Work: reject required hourly products whose periods do not contain usable - time bounds before they enter collection and derivation. -- Locked behavior: required/optional source policy, source provenance, - chronological selection, and valid hourly fixtures remain unchanged. -- Documentation owners: `docs/integrations/weatherapi.md`, - `docs/internal/weather-data.md`, and `docs/internal/collect.md`. -- Exit commands: focused weather-data, adapter, and collection tests, then the - standard repository commands. - -### Stage 10: Correct Civil-Time Windows And Daily Run Identity - -- Findings: `AUD-002`, `AUD-003`. -- Affected areas: `internal/timeutil`, Daily report identity, DST and run-ID - tests, and report-registry documentation. -- Work: construct configured clocks as local civil times across both DST - transitions and add durable protection for Daily run-ID uniqueness. -- Locked behavior: half-open selection, civil-day report periods, rolling - Hourly elapsed time, and source-period identity remain unchanged. -- Documentation owners: `docs/internal/report-registry.md`. -- Exit commands: focused time/report tests with both DST transitions, then the - standard repository commands. - -### Stage 11: Preserve Units And Overnight Alert Coverage - -- Findings: `AUD-016`, `AUD-017`. -- Affected areas: `internal/forecast`, `internal/facts`, metric fallback - values, overnight dayparts, alerts, and focused tests. -- Work: preserve units in fallback values and include alerts over the complete - overnight interval. -- Locked behavior: half-open overlap semantics, chronological ordering, source - periods, and valid US-customary results remain unchanged. -- Documentation owners: `docs/internal/forecast-derivation.md` and - `docs/internal/facts.md`. -- Exit commands: focused forecast/facts tests with metric and overnight cases, - then the standard repository commands. - -### Stage 12: Validate Precipitation And Ice Classification - -- Findings: `AUD-018`, `AUD-019`. -- Affected areas: precipitation-domain validation, weather-text - classification, derived summaries, and focused forecast/facts tests. -- Work: reject precipitation probabilities outside the percentage domain and - classify ordinary icy wording consistently. -- Locked behavior: valid percentage handling, summary ordering, and existing - weather-condition categories remain unchanged. -- Documentation owners: `docs/internal/forecast-derivation.md` and - `docs/internal/facts.md`. -- Exit commands: focused forecast/facts boundary tests, then the standard - repository commands. - -### Stage 13: Correct Source-Facing Briefing Module Semantics - -- Findings: `AUD-020`, `AUD-021`. -- Affected areas: briefing module option validation, weather-story construction, - module registry tests, and normalized source fixtures. -- Work: make accepted AFD option shapes constructible and omit semantically - empty weather stories instead of publishing false availability. -- Locked behavior: report support, default compositions, checked-empty alerts, - and ordered module snapshots remain unchanged. -- Documentation owners: `docs/internal/module.md` and - `docs/internal/briefing.md`. -- Exit commands: focused briefing/module tests, then the standard repository - commands. - -### Stage 14: Correct Derived Briefing Language And Ranking - -- Findings: `AUD-023`, `AUD-026`, `AUD-027`. -- Affected areas: derived daily summaries, outdoor-window scoring, temperature - band wording, and briefing tests. -- Work: label apparent temperature accurately; include snow, ice, and fog in - outdoor-window ranking; and preserve negative-zero temperature meaning. -- Locked behavior: deterministic ranking, Today-only planning scope, and - existing module/schema identities remain unchanged. -- Documentation owners: `docs/internal/briefing.md` and prompt-facing field - documentation if names or meaning change. -- Exit commands: focused briefing tests with metric, subzero, and hazard cases, - then the standard repository commands. - -### Stage 15: Establish One Daypart Canonicalization Mechanism - -- Findings: `AUD-067`. -- Affected areas: duplicated canonical-key helpers in briefing and - generated-text packages plus their behavior tests. -- Work: establish one domain-owned canonical key contract and route both - current consumers through it without changing accepted ASCII behavior. -- Locked behavior: display labels, report-specific semantic roles, ordering, - and current valid output remain unchanged. -- Documentation owners: focused internal briefing/generated-text docs only if - ownership needs explanation. -- Exit commands: focused briefing and generated-text tests, then the standard - repository commands. - -### Stage 16: Correct Daypart Identity, Collision, And Display Behavior - -- Findings: `AUD-024`, `AUD-025`, `AUD-073`. -- Affected areas: derived daypart summaries, planning builders, display - capitalization, configuration fixtures, and rendering tests. -- Work: reject canonical-key collisions; use configured identities consistently - across planning; and make non-ASCII display capitalization rune-safe. -- Locked behavior: Stage 15's canonical identity remains separate from display - text; stable ordering and default dayparts remain unchanged. -- Documentation owners: `docs/config.md`, `docs/internal/briefing.md`, and - generated-text/template docs if visible labels change. -- Exit commands: focused config/briefing/generated-text/reporttemplate tests, - then the standard repository commands. - -### Stage 17: Refresh The Official SPC Definition Asset - -- Findings: `AUD-028`. -- Affected areas: embedded SPC category definitions, source attribution, and - risk-digest tests. -- Work: reconcile descriptions with the current authoritative SPC table and - record a durable source/review mechanism without changing risk thresholds - accidentally. -- Locked behavior: deterministic category ordering, threshold selection, and - overlap logic remain unchanged unless current official semantics require an - explicit reviewed correction. -- Documentation owners: `docs/internal/briefing.md` or another single - canonical maintainer owner for the external definition source. -- Exit commands: focused briefing asset/risk tests, then the standard - repository commands. - -### Stage 18: Harden Prompt-Debug Redaction - -- Findings: `AUD-034`. -- Affected areas: `internal/promptdebug`, Promptkit debug callbacks, endpoint - redaction, and adversarial content tests. -- Work: redact common credential aliases and URL credential forms from every - explicit debug artifact and error projection. -- Locked behavior: debug remains explicit opt-in with `0700` directories, - `0600` files, deterministic references, and no sensitive normal output. -- Documentation owners: `docs/operations.md` and - `docs/integrations/promptkit.md`. -- Exit commands: focused promptdebug/Promptkit redaction tests, then the - standard repository commands. - -### Stage 19: Make Prompt-Debug Filesystem Traversal Race-Resistant - -- Findings: `AUD-035`. -- Affected areas: `internal/promptdebug`, secure directory and file creation, - Promptkit debug callbacks, and adversarial filesystem tests. -- Work: replace pathname check/use gaps with a race-resistant secure traversal - and creation strategy while preserving atomic replacement and actionable - failures. -- Locked behavior: Stage 18 redaction, explicit opt-in, `0700` directories, - `0600` files, deterministic references, containment, and ordinary concurrent - execution remain unchanged. -- Documentation owners: `docs/operations.md` and - `docs/integrations/promptkit.md`. -- Exit commands: focused promptdebug/Promptkit tests plus - `GOWORK=off go test -race -count=1 ./internal/promptdebug - ./internal/adapters/promptkit ./internal/app`, then standard commands. - -### Stage 20: Enforce Generated-Text Identity And Shape - -- Findings: `AUD-036`, `AUD-037`. -- Affected areas: generated-text catalog, decoders, schemas, typed validators, - and focused tests. -- Work: require report-specific schema/template pairs; reject case-variant - properties that schemas reject. -- Locked behavior: typed normalization, exact registered identities, schema - copy isolation, and valid generated content remain unchanged. -- Documentation owners: `docs/internal/generatedtext.md` and template docs. -- Exit commands: focused generated-text and promptassets tests, then the - standard repository commands. - -### Stage 21: Bound Generated Content And Diagnostics - -- Findings: `AUD-038`. -- Affected areas: generated-text decoders and typed validators, schemas, - Promptkit result handling, safe error projection, and boundary tests. -- Work: bound prose, arrays, total generated content, and diagnostic context - before allocation or normal error projection. -- Locked behavior: Stage 20 identity/shape enforcement, typed normalization, - valid generated content, and content-safe normal summaries remain unchanged. -- Documentation owners: `docs/internal/generatedtext.md`, template docs, and - Promptkit integration limits where externally relevant. -- Exit commands: focused generated-text/promptassets/app tests at and around - every limit, then the standard repository commands. - -### Stage 22: Establish The Prepared Metadata Authority - -- Findings: `AUD-069`. -- Affected areas: briefing metadata construction, prepared reports, identity - and timing values, and focused app/briefing tests. -- Work: define one prepared identity/timing authority, populate it at the - preparation boundary, and lock its equivalence to current valid output. -- Locked behavior: prompt and output bytes for valid fixtures, immutable - prepared inputs, and module ordering remain unchanged. -- Documentation owners: `docs/internal/prepared-report.md`, - `docs/internal/briefing.md`. -- Exit commands: focused briefing and app tests, then the standard repository - commands. - -### Stage 23: Migrate Prepared Metadata Projections - -- Findings: none; this completes the remediation begun in Stage 22. -- Affected areas: prompt metadata projection, render-context inputs, superseded - parallel fields, and focused prompt/render tests. -- Work: route prompt and render projections through Stage 22's authority, then - remove or narrow superseded fields without changing serialized YAML or - rendered Markdown. -- Locked behavior: Stage 22's authority, prompt and output bytes for valid - fixtures, immutable prepared inputs, and module ordering remain unchanged. -- Documentation owners: `docs/internal/prepared-report.md`, - `docs/internal/briefing.md`, and `docs/internal/prompt-input.md`. -- Exit commands: focused briefing/app/promptinput/generatedtext tests, then the - standard repository commands. - -### Stage 24: Enforce Render Identity And Empty-Section Behavior - -- Findings: `AUD-040`, `AUD-042`. -- Affected areas: render-context builders, Today templates/partials, and - focused context/rendering tests. -- Work: reject conflicting report identities and suppress or replace a Today - daypart section that has no rows or fallback. -- Locked behavior: repository templates remain Markdown owners, deterministic - facts remain authoritative, and normal section order remains unchanged. -- Documentation owners: `docs/templates.md`, - `docs/internal/generatedtext.md`, and `docs/internal/reporttemplate.md`. -- Exit commands: focused generated-text/reporttemplate/app tests, then the - standard repository commands. - -### Stage 25: Prevent Generated Prose From Impersonating Markdown Structure - -- Findings: `AUD-041`. -- Affected areas: generated prose insertion, Markdown templates/partials, - renderer policy, and adversarial rendering tests. -- Work: define and enforce the smallest content-handling rule that prevents - generated prose from creating structure that impersonates deterministic - report sections. -- Locked behavior: repository templates remain Markdown owners, deterministic - facts remain authoritative, ordinary generated prose remains readable, and - normal section order remains unchanged. -- Documentation owners: `docs/templates.md`, - `docs/internal/generatedtext.md`, and `docs/internal/reporttemplate.md`. -- Exit commands: focused generated-text/reporttemplate/app tests with - adversarial headings and related Markdown constructs, then the standard - repository commands. - -### Stage 26: Move Prompt Compatibility Checks Before Collection - -- Findings: `AUD-043`. -- Affected areas: application prompt inspection and preparation, - generated-text catalog preflight, collection ordering, and app tests. -- Work: validate catalog compatibility before collection or other expensive - report work begins. -- Locked behavior: one inspection/preparation, profile precedence, immutable - execution copies, and partial execution result ownership remain unchanged. -- Documentation owners: `docs/internal/prepared-report.md`, - `docs/internal/app-orchestration.md`, and Promptkit integration docs. -- Exit commands: focused app and generatedtext tests proving failure ordering, - then the standard repository commands. - -### Stage 27: Reconcile Prompt Execution Provenance - -- Findings: `AUD-044`. -- Affected areas: prompt inspection, prepared execution values, callbacks, - completed execution results, schema validation, and app tests. -- Work: reconcile inspected, prepared, callback, completed, and schema - identities before accepting generated output. -- Locked behavior: Stage 26 preflight ordering, one inspection/preparation, - profile precedence, immutable execution copies, and partial result ownership - remain unchanged. -- Documentation owners: `docs/internal/prepared-report.md`, - `docs/internal/app-orchestration.md`, and Promptkit integration docs. -- Exit commands: focused app/promptexec/generatedtext tests with mismatched - provenance at each boundary, then the standard repository commands. - -### Stage 28: Preflight Single-Report Temporary Names - -- Findings: `AUD-046`. -- Affected areas: app output planning, bounded temporary naming, destination - validation, directory creation, and focused filesystem tests. -- Work: detect temporary-name and basename limits during preflight without - leaving newly created directories after failure. -- Locked behavior: destination selection, same-directory atomic visibility, - final filenames, and existing regular-file preservation remain unchanged. -- Documentation owners: `docs/operations.md` and app orchestration internals if - preflight behavior needs clarification. -- Exit commands: focused fileutil/app/CLI long-name tests, then the standard - repository commands. - -### Stage 29: Make Single-Report Publication Type-Safe And Cancelable - -- Findings: `AUD-045`, `AUD-047`. -- Affected areas: `internal/fileutil`, app output planning/publication, - destination error projection, and filesystem tests. -- Work: reject non-regular final destinations; use bounded temporary naming - from Stage 28; and check cancellation at the rename linearization point. -- Locked behavior: same-directory atomic visibility, preservation of prior - regular files, restrictive new-file mode, truthful committed paths, and - notification-after-publication remain unchanged. -- Documentation owners: `docs/operations.md`, architecture policy if an - invariant needs clarification, and app orchestration internals. -- Exit commands: focused fileutil/app/CLI filesystem tests plus - `GOWORK=off go test -race -count=1 ./internal/fileutil ./internal/app - ./internal/cli`, then standard commands. - -### Stage 30: Validate Distributor Endpoints Before Publication - -- Findings: `AUD-048`. -- Affected areas: Distributor configuration validation, app preflight, - endpoint construction, and focused config/app tests. -- Work: reject unsupported endpoint forms before report publication. -- Locked behavior: exactly-once upload/status mapping, published-file source - selection, token redaction, and dependency types remaining adapter-local. -- Documentation owners: `docs/config.md`, `docs/operations.md`, and Distributor - integration docs. -- Exit commands: focused config and app preflight tests, then the standard - repository commands. - -### Stage 31: Bound Distributor Diagnostics And Exercise The HTTP Boundary - -- Findings: `AUD-049`, `AUD-050`, `AUD-052`. -- Affected areas: Distributor adapter, pinned client interaction, response - reading and classification, local HTTP tests, and safe batch diagnostics. -- Work: bound response reads; redact or classify remote text before normal - results; and exercise the real production HTTP boundary with a local server. -- Locked behavior: Stage 30 endpoint validation, exactly-once upload/status - mapping, published-file source selection, token redaction, and adapter-local - dependency types remain unchanged. -- Documentation owners: `docs/operations.md`, - `docs/internal/distributor-adapter.md`, and Distributor integration docs. -- Exit commands: focused distributor/app tests including oversized and - malicious response bodies, then the standard repository commands. - -### Stage 32: Preserve Batch Cancellation As A Distinct Outcome - -- Findings: `AUD-051`. -- Affected areas: batch orchestration, result/error types, CLI projection, and - cancellation tests. -- Work: stop starting later reports after cancellation and preserve - cancellation separately from independent report failures and notification - state. -- Locked behavior: collect once, deterministic planned order, continuation - after non-cancellation failures, report-only counters, and all-success - notification gating remain unchanged. -- Documentation owners: `docs/cli.md`, `docs/operations.md`, and - `docs/internal/app-orchestration.md`. -- Exit commands: focused app/CLI tests plus - `GOWORK=off go test -race -count=1 ./internal/app ./internal/cli`, then - standard commands. - -### Stage 33: Tighten Comparison Manifest Compatibility - -- Findings: `AUD-053`, `AUD-054`. -- Affected areas: comparison JSON decoding, manifest/logical-bundle validation, - filename helpers, and compatibility tests. -- Work: reject duplicate and case-variant fields and require each successful - report path to equal the canonical ordinal/profile filename. -- Locked behavior: current schema version, deterministic encoding/hashing, - flat exact file set, ordered profiles, and bounded recognition errors remain. -- Documentation owners: `docs/integrations/comparison-bundle.md` and - `docs/internal/comparison-publication.md`. -- Exit commands: focused comparison compatibility tests, then the standard - repository commands. - -### Stage 34: Preflight Comparison Sibling Names - -- Findings: `AUD-057`. -- Affected areas: comparison destination planning, staging and backup sibling - naming, parent directory creation, and focused filesystem tests. -- Work: validate derived sibling-name constraints during preflight and fail - without leaving residual directories. -- Locked behavior: exact target selection, sibling staging, broad-target - rejection, and valid destination names remain unchanged. -- Documentation owners: comparison publication internals and operations if - preflight behavior needs clarification. -- Exit commands: focused comparison/app long-name tests, then the standard - repository commands. - -### Stage 35: Honor Cancellation At Comparison Replacement Commit - -- Findings: `AUD-055`. -- Affected areas: comparison replacement transaction, cancellation - linearization, rollback, and deterministic filesystem tests. -- Work: honor cancellation after moving the prior bundle but before installing - the replacement, restoring or retaining the prior bundle according to the - transaction's recovery rules. -- Locked behavior: Stage 34 preflight, post-move authorization, no forced - destructive cleanup, rollback, and committed-new-bundle truth remain. -- Documentation owners: comparison bundle/publication docs and operations. -- Exit commands: focused comparison/app cancellation tests plus - `GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app`, - then the standard repository commands. - -### Stage 36: Represent Comparison Backup Recovery Truthfully - -- Findings: `AUD-056`. -- Affected areas: comparison backup cleanup, publication result/error state, - app/CLI recovery projection, and deterministic filesystem tests. -- Work: distinguish a complete retained recovery bundle from a partially - removed backup and report the actual recoverable state without deleting the - committed new bundle. -- Locked behavior: post-move authorization, rollback, no forced destructive - cleanup, Stage 35 cancellation behavior, committed-new-bundle truth, and - actionable recovery paths remain. -- Documentation owners: comparison bundle/publication docs, operations, and - CLI result docs where state representation changes. -- Exit commands: focused comparison/app/CLI tests plus - `GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app - ./internal/cli`, then standard commands. - -### Stage 37: State Executor Concurrency And Preserve Profile Failures - -- Findings: `AUD-058`, `AUD-077`. -- Affected areas: `promptexec.Executor` contract, comparison execution/result - aggregation, Promptkit adapter, app fakes, and CLI tests. -- Work: document and test concurrent-call substitutability and preserve - already-completed profile failures when later command cancellation occurs. -- Locked behavior: one shared executor, one goroutine per explicit profile, - provider-owned capacity, joined workers, selection ordering, and no - Distributor notification remain unchanged. -- Documentation owners: Promptkit integration and comparison execution/app - orchestration internals. -- Exit commands: focused promptexec/Promptkit/app/CLI tests plus - `GOWORK=off go test -race -count=1 ./internal/adapters/promptkit - ./internal/app ./internal/cli`, then standard commands. - -### Stage 38: Make Tests Independent Of Ambient Host State - -- Findings: `AUD-059`, `AUD-060`. -- Affected areas: Promptkit credential tests and symlink tests in app, - promptdebug, and comparison packages. -- Work: explicitly control the credential environment and treat inability to - create symlinks as a tested capability prerequisite rather than product - failure. -- Locked behavior: real supported-host symlink safety assertions and missing- - credential behavior remain fully protected. -- Documentation owners: testing policy only if supported-platform policy is - deliberately changed; normally none. -- Exit commands: focused packages under scrubbed and ordinary environments, - race-enabled focused tests, then standard commands. - -### Stage 39: Remove Redundant Assertions And Production Test Waits - -- Findings: `AUD-061`, `AUD-062`. -- Affected areas: briefing SPC tests, collection wrapper tests, Distributor - polling tests, and narrow timing seams. -- Work: remove the private-constant assertion while retaining observable risk - behavior; inject only the wait controls necessary to avoid production delays - in deterministic tests. -- Locked behavior: production retry/poll intervals and observable SPC policy - remain unchanged. -- Documentation owners: none unless a contributor-facing test seam warrants a - focused internal note. -- Exit commands: focused briefing/collect/distributor tests with repeated runs, - then the standard repository commands. - -### Stage 40: Make Comparison Test Failure Cleanup Total - -- Findings: `AUD-064`. -- Affected areas: comparison execution test barriers and worker cleanup in - `internal/app` tests. -- Work: release and join workers on timeout, early completion, duplicate start, - and every diagnostic exit without weakening liveness detection. -- Locked behavior: production comparison scheduling and five-second deadlock - diagnostics remain unchanged. -- Documentation owners: none. -- Exit commands: repeat focused comparison-execution tests and run them with - `-race`, then the standard repository commands. - -### Stage 41: Remove The Redundant Weather Warmup Fetch - -- Findings: `AUD-065`. -- Affected areas: Weather API warmup and current-conditions acquisition, - retry semantics from Stage 7, request counts, and adapter/collection tests. -- Work: reuse the successful warmup response or remove the redundant warmup so - current conditions are not fetched and decoded twice. -- Locked behavior: source-local validation/errors, exact source order, - required/optional policy, Stage 7 retry policy, and context cancellation - remain authoritative. -- Documentation owners: Weather API integration and collection internals if - operational request behavior changes materially. -- Exit commands: focused request-count and retry tests, then the standard - repository commands. - -### Stage 42: Acquire Independent Weather Sources Concurrently - -- Findings: none; this completes the efficiency remediation begun in Stage 41. -- Affected areas: Weather API bundle acquisition, request-result aggregation, - provenance ordering, cancellation, and adapter/collection tests. -- Work: execute independent source requests concurrently and merge their - results in one deterministic source order. -- Locked behavior: Stage 41 current-response handling, source-local - validation/errors, exact source order, required/optional policy, Stage 7 - retry policy, and context cancellation remain authoritative. -- Documentation owners: Weather API integration and collection internals if - operational request behavior changes materially. -- Exit commands: focused latency, deterministic-order, failure, and - cancellation tests plus - `GOWORK=off go test -race -count=1 ./internal/adapters/weatherapi - ./internal/collect`, then standard commands. - -### Stage 43: Eliminate Repeated Comparison Bundle Reads - -- Findings: `AUD-066`. -- Affected areas: comparison recognition/planning/publication and app - comparison preflight. -- Work: pass or cache only the recognition evidence needed to avoid four full - reads while retaining reauthorization at every mutation boundary. -- Locked behavior: early rejection, exact file/type validation, post-move - authorization, deterministic hashes, and concurrent destination safety after - Stages 34-36 remain unchanged. -- Documentation owners: comparison publication internals if ownership or - transaction phases change. -- Exit commands: focused read-count and adversarial replacement tests plus - `GOWORK=off go test -race -count=1 ./internal/comparison ./internal/app`, then - standard commands. - -### Stage 44: Consolidate Distributor Template Parsing - -- Findings: `AUD-068`. -- Affected areas: single and batch notification template rendering and config - tests. -- Work: extract one private grammar/parser with typed callbacks while - preserving each template family's variable allowlist and error context. -- Locked behavior: accepted syntax, rejected malformed forms, exact rendering, - and single-versus-batch policy remain unchanged. -- Documentation owners: none unless the user-visible grammar changes, which is - outside this behavior-preserving stage. -- Exit commands: focused config tests, then the standard repository commands. - -### Stage 45: Define One Fact Requirement Vocabulary - -- Findings: `AUD-070`. -- Affected areas: module requirement vocabulary, current briefing definitions, - identity/category metadata, and focused module/briefing tests. -- Work: define each requirement's identity and category once, preserve each - module's explicit requirement list, and lock equivalence with current valid - compositions. -- Locked behavior: all current compositions, supported reports, availability, - and diagnostic identity remain byte-for-byte or semantically unchanged. -- Documentation owners: module and briefing internals if ownership changes. -- Exit commands: focused module and briefing catalog tests, then the standard - repository commands. - -### Stage 46: Migrate Fact Availability And Constructor Validation - -- Findings: none; this completes the remediation begun in Stage 45. -- Affected areas: availability predicates, briefing constructors, legacy - switches, configuration validation, and focused tests. -- Work: route availability and constructor validation through Stage 45's - requirement vocabulary and remove superseded policy implementations. -- Locked behavior: all current compositions, supported reports, missing-data - policy, availability, and diagnostic identity remain unchanged. -- Documentation owners: module and briefing internals if ownership changes. -- Exit commands: focused module/briefing/config tests, then the standard - repository commands. - -### Stage 47: Consolidate CLI Report-Date Policy - -- Findings: `AUD-071`. -- Affected areas: generate/comparison flag registration and date resolution, - CLI parser/action tests, and private CLI descriptors. -- Work: establish one private report-date policy/resolver used by both actions - after Stage 6 has locked corrected behavior. -- Locked behavior: accepted flags, required/default dates, action-specific - errors, and request types remain unchanged. -- Documentation owners: none unless implementation reveals a current contract - mismatch; CLI docs remain authoritative. -- Exit commands: focused CLI parser/action tests and help commands, then the - standard repository commands. - -### Stage 48: Retire Dormant Prompt And Generated-Text Surfaces - -- Findings: `AUD-072`. -- Affected areas: unused prompt YAML loading/saving, normalized generated JSON - returns, and their sole-purpose tests and documentation. -- Work: verify these prompt/generated-text compatibility surfaces still have no - production owner, then remove them. If a production consumer appears, stop - and replace this retirement sequence with a separately approved contract. -- Locked behavior: normal Build/Validate/MarshalYAML, report publication, - comparison data packages, prompt debugging, typed normalization, and all - workflows remain unchanged. -- Documentation owners: prompt-input and generated-text internal documents; - remove stale descriptions rather than documenting retired APIs. -- Exit commands: graph/text caller checks, focused promptinput/generatedtext - tests, then the standard repository commands. - -### Stage 49: Retire Dormant Weather Persistence Helpers - -- Findings: none; this continues the dormant-surface remediation begun in - Stage 48. -- Affected areas: Weather API bundle saving, the JSON atomic helper if it - becomes orphaned, and sole-purpose tests and documentation. -- Work: verify these persistence helpers still have no workflow owner, then - remove them and any helper made unreachable by that removal. If a production - consumer appears, stop and seek a separately approved contract. -- Locked behavior: weather collection, normalized in-memory bundles, prompt - debugging, operator-owned publication, and all workflows remain unchanged. -- Documentation owners: weather-data and collection internals where stale - persistence claims exist. -- Exit commands: graph/text caller checks, focused weatherapi/weatherdata/ - fileutil tests, then the standard repository commands. - -### Stage 50: Retire Dormant Module And Forecast Compatibility Exports - -- Findings: none; this continues the dormant-surface remediation begun in - Stage 48. -- Affected areas: module snapshot compatibility implications, unused forecast - threshold exports, and sole-purpose tests and documentation. -- Work: verify the identified exports still have no production owner, then - remove them without broadening the supported compatibility contract. If a - production consumer appears, stop and seek a separately approved contract. -- Locked behavior: current module snapshots, real forecast decisions, prompt - exports, report rendering, and all workflows remain unchanged. -- Documentation owners: module, briefing, and forecast internals where stale - compatibility claims exist. -- Exit commands: graph/text caller checks, focused module/briefing/forecast - tests, then the standard repository commands. - -### Stage 51: Narrow Dormant Alert Envelope Copies - -- Findings: none; this completes the dormant-surface remediation begun in - Stage 48. -- Affected areas: unused full alert-envelope copies, retained alert item - provenance, and sole-purpose tests and documentation. -- Work: remove unconsumed alert-envelope data while retaining exactly the item - and provenance values required by current facts, briefing, and report paths. - If a production consumer appears, stop and seek a separately approved - contract. -- Locked behavior: alert selection, alert item provenance, briefing digests, - report rendering, and all workflows remain unchanged. -- Documentation owners: weather-data, facts, and briefing internals where the - narrowed value is described. -- Exit commands: graph/data-flow checks plus focused weatherdata/facts/ - briefing tests, then the standard repository commands. - -### Stage 52: Consolidate Generated-Text Test Ownership - -- Findings: `AUD-039`. -- Affected areas: generated-text day-style and render-context tests, - promptassets schema tests, and reporttemplate tests. -- Work: keep one shared semantic suite plus thin report-identity cases; put - schema ownership in promptassets and Markdown ownership in reporttemplate; - retain one real typed-context render integration. -- Locked behavior: every public validator identity, distinct Hourly behavior, - schema/type agreement, typed projection, template semantics, and composition - remain protected. -- Documentation owners: none. -- Exit commands: focused promptassets/generatedtext/reporttemplate tests, then - the standard repository commands. - -### Stage 53: Reconcile Stale Internal Implementation Guides - -- Findings: `AUD-075`. -- Affected areas: forecast derivation, collection, report registry, and prompt- - input internal guides plus incoming links. -- Work: replace nonexistent API/flow/test claims with concise descriptions of - the implemented owners and link to canonical contracts instead of duplicating - volatile details. -- Locked behavior: documentation-only; no executable contract changes. -- Documentation owners: the four affected `docs/internal/` files and - `docs/development.md` only if task routing changes. -- Exit commands: repository link/fence checks, focused test inventories used to - verify claims, `GOWORK=off go test -count=1 ./...`, and `git diff --check`. - -### Stage 54: Retire Completed Comparison Roadmaps - -- Findings: `AUD-076`. -- Affected areas: `docs/roadmap/profile-comparison.md`, - `docs/roadmap/implementation.md`, and all incoming links. -- Work: verify useful implemented contracts have one durable canonical owner, - repair incoming navigation, then remove the completed roadmaps so they no - longer duplicate current behavior. -- Locked behavior: documentation lifecycle only; no code, tests, dependencies, - schemas, or current contracts change. -- Documentation owners: CLI, operations, comparison integration/publication, - comparison execution, Promptkit integration, architecture, and development - docs only where an incoming link or missing durable fact requires correction. -- Exit commands: repository link/fence checks, both CLI help commands, - `GOWORK=off go test -count=1 ./...`, and `git diff --check`. - -### Stage 55: Verify Remediation Completion And Durable Ownership - -- Findings: none; this is the roadmap lifecycle gate. -- Prerequisite: Stages 1-54 are complete and every referenced finding's tests - and canonical documentation have landed. -- Affected areas: every remediation completion note, canonical documentation - owner, audit finding, retained decision, and incoming audit-document link. -- Work: verify every finding is resolved, superseded, or explicitly moved to a - newly approved roadmap; verify durable rationale lives with its canonical - owner; and produce a concise removal checklist for the temporary audit plan, - ledger, and remediation roadmap. Do not remove those records in this stage. -- Locked behavior: documentation lifecycle only; do not rewrite historical - ADRs or release notes and do not create a second current-state reference. -- Documentation owners: all durable owners touched by earlier stages, solely - to fill a demonstrated ownership gap. -- Exit commands: verify every stage/finding mapping and completion note; - repository link/fence and secret-pattern checks; all focused stage exits; - `GOWORK=off go test -count=1 ./...`, - `GOWORK=off go test -race -count=1 ./...`, both CLI help commands, - `GOWORK=off go vet ./...`, and `git diff --check`. - -### Stage 56: Retire The Completed Audit Records - -- Findings: none; this is the final documentation lifecycle change. -- Prerequisite: Stage 55 has verified completion and supplied the exact removal - and link-repair checklist. -- Affected areas: `docs/roadmap/audit.md`, `audit-plan.md`, this roadmap, and - their incoming links. -- Work: apply Stage 55's checklist, repair incoming navigation, and remove the - temporary audit plan, ledger, and remediation roadmap without changing - executable behavior or historical records. -- Locked behavior: documentation lifecycle only; durable current contracts and - rationale remain with their canonical owners, and ADRs and release notes are - unchanged. -- Documentation owners: only the navigation or durable owner files identified - by Stage 55. -- Exit commands: repository link/fence and secret-pattern checks; - `GOWORK=off go test -count=1 ./...`, - `GOWORK=off go test -race -count=1 ./...`, both CLI help commands, - `GOWORK=off go vet ./...`, and `git diff --check`. - -## Completion Tracking - -Update a stage heading or add a concise completion note only after its code, -tests, canonical documentation, and exit commands pass together. A stage that -begins a multi-stage remediation sequence must not mark its finding resolved -until the final continuation stage succeeds. Do not copy implemented contracts -into this roadmap. When all work is complete, follow Stages 55 and 56 rather -than retaining this file as a parallel reference. diff --git a/docs/roadmap/profile-comparison.md b/docs/roadmap/profile-comparison.md deleted file mode 100644 index 009671c..0000000 --- a/docs/roadmap/profile-comparison.md +++ /dev/null @@ -1,398 +0,0 @@ -# LLM Profile Comparison Roadmap - -Status: Implemented; retained as the feature roadmap. - -## Purpose - -Prompt development currently requires separate Weatherreporter invocations to -compare several LLM profiles. Those invocations may collect different weather -snapshots or rebuild inputs at different times, making model output harder to -compare and slowing prompt iteration. - -Weatherreporter should provide a first-class `compare` command that resolves -one report, prepares one exact data package, executes the same prompt and data -package concurrently through several explicitly selected Promptkit profiles, -and publishes a self-contained local comparison bundle. - -An illustrative invocation is: - -```sh -weatherreporter compare daily \ - --date 2026-08-24 \ - --profile weather-light \ - --profile weather-balanced \ - --profile weather-deep -``` - -This is a prompt-development workflow, not an automated model evaluator. Its -output gives a maintainer consistent evidence for human comparison without -assigning scores or selecting a winner. - -## Prerequisite - -Configurable output directories are implemented. Profile comparison must reuse -the current [configuration reference](../config.md) and [operations -guide](../operations.md) rather than introduce a second destination policy. - -## User Intent - -The command is intended for deliberate evaluation of multiple profiles, -including sets of eight to twelve candidate models. Concurrency is part of the -feature, not a future optimization. Promptkit should retain ownership of -backend-specific capacity, while Weatherreporter owns comparison-wide -coordination, cancellation, deterministic results, and artifact publication. - -Every profile must receive byte-for-byte identical prompt input. Weather data, -derived facts, modules, prompt metadata, and serialized YAML must not be -recollected or rebuilt separately for individual profiles. - -Comparison bundles are explicitly requested, operator-owned development -outputs. They are not Weatherreporter state, are never read implicitly by a -later run, and do not weaken the ordinary stateless execution model. - -## Command Contract - -The command form is: - -```text -weatherreporter compare REPORT [options] -``` - -`REPORT` accepts the implemented generated-text reports: `daily`, `today`, -`tomorrow`, and `hourly`. Report-date behavior matches `generate`: `daily` -requires `--date`, `today` may accept an explicit date or use the current local -date, and the remaining report types retain their existing period policies. - -The command accepts the applicable common generation options, including -`--config`, `--units`, `--tz`, `--date`, `--llm-debug-dir`, and `--quiet`, plus: - -- repeatable `--profile PROFILE_ID` selections; -- `--out-dir PATH` for the exact comparison-bundle directory; and -- `--replace` to authorize guarded replacement of a recognized existing - comparison bundle. - -At least two distinct, nonblank profile IDs are required. Their command-line -order is significant and is preserved in filenames, summaries, and -`comparison.json`. Duplicate profile IDs are rejected rather than silently -deduplicated or executed twice. - -Profiles are always explicit for this command. `promptkit.profile` does not add -or replace a comparison selection, but all other effective Promptkit settings, -profile-source precedence, local backend configuration, credential lookup, and -profile overrides remain in force. - -The initial feature has no Weatherreporter-specific concurrency flag or -artificial profile-count ceiling. The explicit profile list bounds the -comparison, and Promptkit owns capacity enforcement for each selected backend. - -## Preparation And Execution Invariants - -A comparison has this logical lifecycle: - -1. Parse and validate the report, date, profile list, configuration, output - destination, and replacement authorization. -2. Resolve the report definition, valid period, prompt identity, and default - output name once. -3. Inspect the exact prompt once and preflight every selected profile, - including its effective backend, model, and required credential - availability, before weather collection. -4. Collect weather data exactly once. -5. Build collected and derived facts, the module snapshot, briefing metadata, - and the prompt data package exactly once. -6. Marshal the data package to one immutable YAML byte sequence exactly once. -7. Execute the exact prompt version concurrently for every selected profile, - passing the same immutable YAML bytes to every execution. -8. Validate and render each profile result independently from the shared - deterministic inputs. -9. Assemble results in requested-profile order and publish one coherent - comparison bundle. - -This lifecycle describes the required end-state behavior rather than an -implementation-stage sequence. - -No profile execution may cause recollection, report re-resolution, module -rebuilding, or data-package remarshalling. Prompt execution may perform -Promptkit-owned validation or repair behavior, but Weatherreporter does not -retry a failed comparison execution independently. - -## Concurrency And Cancellation - -Weatherreporter starts one execution for each preflighted profile and permits -them to run concurrently through one shared, concurrency-safe Promptkit -executor. Promptkit's engine-local backend pools remain authoritative for -backend concurrency and waiting capacity. Profiles routed to a limited local -backend therefore respect its configured limit, while profiles routed to -other backends may proceed independently. - -Weatherreporter must not add a second semaphore that obscures or overrides -Promptkit's backend policy. It must safely coordinate goroutine lifecycles, -result collection, debug callbacks, and output assembly without data races. - -One profile failure does not cancel its peers. Provider, capacity, validation, -and rendering failures are recorded for that profile while other executions -continue. Cancellation or deadline expiration of the comparison command is -propagated to every outstanding execution, prevents new publication, and is -joined without leaking goroutines. - -Completion order must not affect filenames, manifest order, CLI summaries, or -error aggregation. Those outputs always follow the original `--profile` -order. - -## Output Destination - -Without `--out-dir`, Weatherreporter derives a comparison directory from the -resolved report's existing default Markdown filename by removing `.md` and -prefixing `comparison-`: - -| Report output | Comparison directory | -| --- | --- | -| `today.md` | `comparison-today/` | -| `tomorrow.md` | `comparison-tomorrow/` | -| `hourly.md` | `comparison-hourly/` | -| `daily-2026-08-24.md` | `comparison-daily-2026-08-24/` | - -The derived directory is created beneath `output.directory` when configured, -or beneath the present working directory otherwise. An explicit `--out-dir` -is the exact bundle directory, resolves relative to the present working -directory when necessary, and overrides `output.directory` completely. - -All destination selection and validation completes before weather collection. -The resolved comparison directory is returned in the command's structured -result. - -## Comparison Bundle - -A successful three-profile comparison has a flat layout: - -```text -comparison-daily-2026-08-24/ -├── comparison.json -├── data-package.yml -├── 01-weather-light.md -├── 02-weather-balanced.md -└── 03-weather-deep.md -``` - -`data-package.yml` contains the exact YAML bytes passed to every Promptkit -execution. It is written once and its SHA-256 digest is recorded in the -manifest. - -Each report filename begins with its one-based, zero-padded selection position -and a filesystem-safe representation of the requested logical profile ID. The -safe representation must not permit absolute paths, traversal, separators, or -control characters. The manifest retains the exact case-sensitive profile ID, -so filename normalization never becomes the authority for profile identity. - -`comparison.json` is the authoritative index for the bundle. It uses an -explicit schema version and records safe comparison information including: - -- comparison identity and start and finish timestamps; -- report ID, resolved valid period, and effective timezone; -- prompt ID, version, and inspected prompt hash; -- the relative data-package filename and SHA-256 digest; -- total, succeeded, and failed profile counts; and -- one ordered result per requested profile containing the exact profile ID, - resolved backend and model, relative report filename when present, - execution and validation status, and safe error information when failed. - -The manifest and normal command summary must not contain credentials, provider -request bodies, raw model output, rendered prompts, schemas, provider -endpoints, or other content-rich diagnostics. The explicit data package and -generated reports contain the development material the user requested and -must be handled as operator-owned potentially sensitive output. - -## Failure And Publication Policy - -Failure before concurrent execution, including invalid profiles, missing -credentials, collection failure, preparation failure, or unsafe destination, -publishes no comparison bundle and performs no model calls where the failure -is discoverable during preflight. - -After execution begins, Weatherreporter waits for every non-cancelled profile. -If one or more profiles fail, it still publishes a coherent partial bundle -containing `data-package.yml`, every successfully rendered report, and a -manifest describing all successes and failures. It then returns a non-zero -exit status. A failed profile has no report file unless a future contract -explicitly introduces a separately named diagnostic artifact. - -Bundle contents are staged outside the destination and published only after -the manifest is complete. Ordinary publication accepts only an absent or empty -target directory. A nonempty existing directory fails without modification -unless `--replace` is present. - -`--replace` may replace only the exact resolved target and must reject broad or -unsafe targets such as a filesystem root, the present working directory, a -symlink, or an unrecognized nonempty directory. A recognized prior bundle must -contain a valid Weatherreporter comparison manifest. Replacement publishes the -new complete or coherent partial bundle as a unit, prevents stale reports from -the prior comparison from surviving, and preserves or restores the prior -bundle if the final replacement operation fails. - -An interrupted or cancelled comparison does not replace an existing bundle. -Temporary staging artifacts are cleaned up on ordinary failure and -cancellation without scanning or modifying unrelated directories. - -## Prompt Debugging - -The existing `--llm-debug-dir` mechanism remains available. Concurrent -comparison executions require distinct, deterministic debug identities that -include the comparison and exact profile selection so callbacks cannot collide -or overwrite another profile's artifacts. - -Debug writing must be concurrency-safe and retain the existing permission, -redaction, explicit-opt-in, and path-containment guarantees. Debug artifacts -remain separate from the comparison bundle; the bundle does not implicitly -enable full Promptkit diagnostics. - -## Notification Policy - -Profile comparisons never invoke Distributor notification, even when -notification is enabled in the effective configuration. Comparison reports -are local development artifacts rather than ordinary report publications. - -Adding comparison publication or upload behavior would require a separate -accepted feature scope and explicit operator authorization. - -## Architectural End State - -Application orchestration exposes a reusable prepared-report boundary that -contains the resolved report, shared collected and derived facts, module -snapshot, briefing metadata, generated-text handler, render inputs, and exact -serialized data package. That boundary is immutable during concurrent profile -execution. - -Ordinary `generate` behavior continues to prepare once and execute once. -`compare` prepares once and executes many without duplicating the generation -workflow or calling `GenerateDetailed` in a loop. Shared preparation, -profile-specific Promptkit execution, structured-output validation, rendering, -and artifact publication remain distinct responsibilities. - -The Promptkit adapter remains the only owner of dependency-specific types and -engine calls. The CLI owns parsing and user-facing summaries. The configuration -package owns configuration. Application orchestration owns comparison order, -concurrency lifecycle, failure aggregation, and bundle publication. Domain, -prompt-input, generated-text, and template packages retain their existing -deterministic contracts. - -## Scope - -The completed feature includes: - -- the `compare` CLI command for every implemented generated-text report; -- repeatable explicit profile selection and validation; -- configured and CLI output-directory integration through the implemented - destination policy; -- one-time report resolution, collection, deterministic preparation, and YAML - serialization; -- concurrent execution through one Promptkit executor with backend capacity - respected; -- independent validation and rendering with deterministic ordered results; -- the flat, versioned comparison-bundle contract; -- safe filename derivation and data-package hashing; -- coherent partial-result publication and non-zero failure behavior; -- guarded whole-bundle replacement through `--replace`; -- comparison-aware, concurrency-safe optional prompt debugging; -- explicit suppression of Distributor notification; -- structured normal and quiet-mode CLI behavior consistent with existing - commands; -- focused race-safe tests across configuration, CLI, application, - Promptkit-adapter, rendering, and filesystem boundaries; and -- updates to every affected canonical user, operator, architecture, - integration, and internal document. - -## Compatibility - -The feature is additive. Existing `generate` and `run` commands, report -definitions, profile defaults, configuration, output filenames, notification -behavior, and exit contracts remain unchanged. - -The comparison manifest and bundle layout begin as versioned contracts. They -do not become inputs accepted by Weatherreporter, and no backward-compatible -replay or long-term archive guarantee is implied beyond identifying the schema -used to interpret a produced bundle. - -## Testing Expectations - -Tests should provide durable coverage for: - -- report and date parsing consistent with `generate`; -- rejection of fewer than two profiles, blanks, and duplicates; -- inspection of the exact prompt and every profile before collection; -- no collection or model execution after a preflight failure; -- exactly one weather collection and one preparation for several profiles; -- byte-for-byte identical data-package input in every execution; -- observable concurrent execution through a concurrency-safe fake executor; -- respect for Promptkit-owned backend capacity in an assembled adapter test - where that integration adds distinct confidence; -- deterministic filenames, manifest order, summaries, and errors under varied - completion order; -- continuation and coherent partial publication after one profile fails; -- cancellation propagation, goroutine completion, and preservation of an - existing destination; -- destination precedence and each derived default directory; -- safe filename handling for unusual valid profile IDs; -- absent, empty, occupied, symlinked, unsafe, recognized, and unrecognized - replacement targets; -- removal of stale prior report files during authorized whole-bundle - replacement; -- exact package digest and manifest/result consistency; -- concurrency-safe, non-colliding opt-in debug artifacts; and -- absence of Distributor calls for complete and partial comparisons. - -Concurrency and replacement behavior require race-enabled and consequential -failure-path coverage. Tests must remain deterministic, offline, credential -free, and independent of real Promptkit providers or machine-specific paths. - -## Documentation End State - -Once implemented, the [CLI reference](../cli.md) owns command syntax, flags, -summary, and exit behavior. The [operations guide](../operations.md) owns the -bundle lifecycle, replacement procedure, sensitivity guidance, and practical -prompt-comparison workflow. The [architecture policy](../policy/architecture.md) -owns the statelessness, concurrency, notification, and publication invariants. - -The [Promptkit integration guide](../integrations/promptkit.md) should describe -the consumer-visible multi-profile execution boundary without duplicating -Promptkit's backend-capacity reference. The versioned manifest and flat bundle -format belong in a focused contract under `docs/integrations/`. App -orchestration, prompt input, generated text, prompt debugging, and bundle -publication mechanics belong in focused documents under `docs/internal/`. - -Current-state documentation must not describe profile comparison as available -until the implementation lands. - -## Non-Goals - -This roadmap does not introduce: - -- automatic model scoring, ranking, recommendation, or winner selection; -- semantic or textual diff generation between reports; -- repeated sampling of one profile or statistical evaluation; -- prompt or profile editing through Weatherreporter; -- replaying a saved data package as command input; -- comparing several report types in one command; -- Weatherreporter-owned backend concurrency or queue configuration; -- automatic retries beyond Promptkit's existing execution contract; -- Distributor upload or other external publication; -- comparison history, indexing, retention, cleanup schedules, or implicit - discovery of prior bundles; or -- changes to ordinary report content or normal generation behavior. - -Any later automated evaluation, replay, sampling, or publication feature -requires a separate accepted roadmap. - -## Completion Criteria - -The feature is complete when a maintainer can select several Promptkit -profiles, have them execute concurrently against one exact prepared report -package, and receive a safe, flat, deterministic comparison bundle whose -manifest accurately describes every success and failure. Configured and -explicit destinations must follow the accepted output policy, replacement must -never mix or silently destroy unrelated contents, cancellation and partial -failure must be race-safe, ordinary notification must remain disabled, and all -affected canonical documentation must describe the implemented behavior. - -## Open Questions - -None. The scope, prerequisites, user intent, and target behavior are defined -above.