Complete the Stage 18 comparison execution audit

docs/roadmap/audit.md

@@ -1,6 +1,6 @@
 # Repository Audit Ledger
 
-Status: In progress; Stages 1-17 complete.
+Status: In progress; Stages 1-18 complete.
 
 This temporary roadmap document is the evidence ledger for the staged audit
 defined by [the audit plan](audit-plan.md). It records audit evidence and
@@ -135,6 +135,15 @@ new parent directories. Flat versioned bundles, deterministic encoding and
 hashing, exact file-set recognition, broad-target and final-symlink rejection,
 commit-time destination reauthorization, rollback after install failure, and
 committed-new-bundle truth otherwise match their contracts.
+Stage 18 found one medium-severity comparison-execution defect: cancellation
+observed after profile goroutines join rewrites every nonsuccessful outcome as
+canceled, including independent profile failures that completed before the
+cancellation. Sequential preflight, collect/prepare once, immutable per-profile
+inputs, concurrent execution through one backend-governed executor, joined
+goroutines, deterministic ordering and debug identities, coherent partial
+publication, committed-path projection, never-notify policy, content-safe CLI
+summaries, and result-bearing failure exit behavior otherwise match their
+contracts.
 Subsystem conclusions and final disposition remain pending the later stages.
 
 ## Baseline Metadata
@@ -279,7 +288,7 @@ inventory commands, graph index refresh, and graph architecture inspection.
 | 15 | Audit single-report generation and atomic output | Complete |
 | 16 | Audit batch orchestration and Distributor notification | Complete |
 | 17 | Audit comparison contracts and transactional publication | Complete |
-| 18 | Audit comparison execution and CLI integration | Pending |
+| 18 | Audit comparison execution and CLI integration | Complete |
 | 19 | Audit test hermeticity and execution hygiene | Pending |
 | 20 | Audit test risk coverage and ownership | Pending |
 | 21 | Audit test durability, duplication, and maintenance cost | Pending |
@@ -313,7 +322,7 @@ with evidence about meaningful risks, test ownership, gaps, and duplication.
 | Single-report publication and preservation | `internal/app`, `internal/fileutil` | 15 | Partly insufficient. Destination precedence, regular-file publication, same-directory temporary replacement, prior-output preservation, temporary cleanup, restrictive modes, truthful committed paths, and notification-after-publication have focused owners. `AUD-045` through `AUD-047` record destructive non-regular target handling, late failure for valid long basenames, and cancellation not checked at the rename boundary. |
 | Batch partial success and notification | `internal/app`, `internal/adapters/distributor` | 16 | Partly insufficient. Collect-once planning, complete output preflight, deterministic sequential execution, independent partial success, report-only counters, published-source selection, and one all-success batch notification are coherent. `AUD-048` through `AUD-052` record late Distributor endpoint rejection, unsafe and unbounded response diagnostics, cancellation flattened into report failure, and the missing production HTTP adapter test boundary. |
 | Comparison identity and transactional publication | `internal/comparison` | 17 | Partly insufficient. Versioned flat manifests, ordered identity helpers, canonical encoding, data-package hashing, exact file sets/types, read-only destination classification, commit-time reauthorization, sibling staging, replacement rollback, restrictive modes, and committed cleanup state have focused package owners. `AUD-053` through `AUD-057` record ambiguous manifest fields, unenforced report filenames, a replacement cancellation gap, partially deleted retained backups, and late long-destination staging failure. |
-| Comparison concurrency and CLI behavior | `internal/app`, `internal/cli` | 18 | Pending |
+| Comparison concurrency and CLI behavior | `internal/app`, `internal/cli` | 18 | Partly insufficient. Explicit profiles are preflighted sequentially, one immutable prepared package feeds concurrent profile-local executions, goroutines join, slice positions lock durable/CLI order, partial bundles publish coherently, and safe summaries preserve committed paths and fail the action on any error. `AUD-058` records a missing mixed failure/cancellation case and the resulting overwrite of an already completed profile failure. |
 | Hermeticity, execution hygiene, portfolio coverage, and durability | Repository-wide suite | 19-21 | Pending |
 | Cross-cutting efficiency and maintainability | Graph metrics plus focused tests | 22-23 | Pending |
 | Documentation and executable-contract coherence | Canonical documents, code, schemas, templates, examples | 24 | Pending |
@@ -2496,6 +2505,51 @@ with evidence about meaningful risks, test ownership, gaps, and duplication.
 - Related findings: `AUD-046`
 - Remediation reference: pending
 
+### AUD-058: Late cancellation overwrites completed profile failures
+
+- Stage: 18
+- Status: candidate
+- Severity: medium
+- Confidence: high
+- Category: correctness
+- Area: `internal/app.executeComparisonProfiles` and comparison cancellation
+  result construction
+- Evidence: Every profile goroutine writes its own ordered outcome and joins
+  before aggregation. After the join, however, `executeComparisonProfiles`
+  checks `ctx.Err()` and calls `markCanceledComparisonOutcome` for every
+  outcome whose status is not `succeeded`. That sweep cannot distinguish an
+  unfinished cancellation from an independent failure that completed before a
+  peer caused cancellation; it replaces the earlier safe category, message,
+  and validation status with canceled/skipped. A temporary deterministic
+  two-profile probe released one profile into an ordinary generation failure,
+  waited until only its peer remained in flight, then canceled the shared
+  context. The completed outcome changed from the expected `generation`
+  category to `canceled`. Existing tests cover all-canceled execution and
+  independent failure without cancellation, but not their overlap.
+- Contract at risk: Profile-local failures and workflow cancellation must
+  remain distinct and truthfully represented after all goroutines join; shared
+  cancellation may classify only work that did not independently complete.
+- Impact: Operators and machine-readable CLI consumers lose the actual failure
+  class for an already completed provider or validation failure whenever a
+  concurrent peer later observes cancellation. This can hide the cause that
+  needs remediation and makes the ordered partial result historically false,
+  although cancellation correctly prevents bundle publication.
+- Recommendation: Track whether each execution completed and preserve its
+  resulting outcome. Mark only unstarted or genuinely cancellation-terminated
+  work as canceled; use an explicit per-position execution state rather than
+  inferring completion from success alone.
+- Test implications: Add a deterministic mixed-outcome test with one completed
+  generation or validation failure, one still-running profile, and subsequent
+  cancellation. Assert the first safe error and validation state survive, the
+  second is canceled, every goroutine is joined, and the application publishes
+  no bundle.
+- Validation: For every completion ordering, independent failures retain their
+  original safe categories while only unfinished/canceled outcomes use the
+  cancellation category; result order and no-publication behavior remain
+  unchanged under the race detector.
+- Related findings: `AUD-044`, `AUD-051`
+- Remediation reference: pending
+
 ## Retained Decisions
 
 ### RET-001: Keep the application package as the explicit composition owner
@@ -2957,6 +3011,40 @@ application distinguish pre-commit failure from a valid committed bundle with
 remaining cleanup work. Retain this transaction shape while adding the final
 cancellation gate in `AUD-055` and truthful partial-cleanup state in `AUD-056`.
 
+### RET-039: Keep ordered prepare-once comparison execution
+
+Comparison validates the explicit profile list, inspects the prompt and each
+profile sequentially in caller order, collects weather once, and prepares one
+immutable logical report before starting model work. Each profile execution
+receives a private copy of the same serialized YAML and writes only its own
+preallocated result position. The application constructs and shares one
+executor, leaves provider-specific capacity enforcement to its backend, and
+waits for every started goroutine before returning. Retain this boundary: it
+keeps compared inputs identical and makes completion timing irrelevant to
+durable and CLI order while avoiding duplicate collection and derivation.
+
+### RET-040: Keep comparison failures profile-local until one bundle decision
+
+An individual generation, validation, render, or debug failure produces a
+bounded failure entry without stopping peers. After every profile completes,
+the application builds and validates one logical bundle containing the shared
+data package and exactly the successful Markdown reports, publishes it once,
+and then returns an aggregate action failure when any profile failed. Shared
+cancellation instead prevents publication, and comparison has no notifier
+path. Retain this single-publication and never-notify policy while correcting
+the mixed cancellation accounting in `AUD-058`.
+
+### RET-041: Keep comparison CLI summaries as safe result projections
+
+The CLI preserves application result order, exposes report and bundle paths
+only after they are authoritative, retains committed paths when old-backup
+cleanup fails, and replaces internal errors with bounded stable categories and
+messages. Quiet mode suppresses output without suppressing the returned error;
+normal mode emits one JSON summary for result-bearing outcomes, and any
+profile, publication, cleanup, cancellation, or aggregate error yields failed
+status and a nonzero command result. Retain this projection rather than
+reconstructing comparison state or provider diagnostics in the CLI.
+
 ## Open Questions
 
 No Stage 1 open questions or unexplained baseline failures remain.
@@ -3256,6 +3344,24 @@ Stage 17 routed these investigation leads to their assigned later stages:
   “atomic” wording scoped to visible namespace state rather than durable-media
   acknowledgment.
 
+Stage 18 routed these investigation leads to their assigned later stages:
+
+- Comparison starts one goroutine per selected profile and relies on the
+  Promptkit backend's configured capacity rather than adding an application
+  semaphore. This is the documented ownership split and current profile sets
+  are configuration-sized; Stage 22 should assess extreme profile-count
+  goroutine and retained-result cost with evidence rather than Stage 18
+  inventing a second concurrency limit.
+- The project-owned executor interface is used concurrently by comparison, but
+  its type comment does not state that implementations must support concurrent
+  calls. The production adapter and race-tested application fake do so. Stage
+  24 should make that substitutability requirement explicit when reconciling
+  executable and written contracts.
+- Comparison publication inherits the post-backup cancellation and cleanup
+  state defects in `AUD-055` and `AUD-056`. Stage 18 verified their committed
+  paths and safe CLI projection without duplicating package-owned findings;
+  Stages 25-26 should validate the repaired end-to-end states.
+
 ## Stage Log
 
 ### Stage 1: Establish The Baseline And Audit Ledger
@@ -4181,3 +4287,63 @@ Stage 17 routed these investigation leads to their assigned later stages:
 - Retained decisions: `RET-036` through `RET-038`.
 - Open questions: the three leads recorded above are routed to their assigned
   later stages.
+
+### Stage 18: Audit Comparison Execution And CLI Integration
+
+- Status: Complete.
+- Scope reviewed: the complete comparison orchestration and result projection
+  in `internal/app/comparison.go`, concurrent profile execution in
+  `comparison_execution.go`, prompt/profile inspection and prepared-report
+  seams needed to establish preflight and input identity, all focused
+  comparison application tests, comparison-specific CLI parsing, wiring,
+  summaries, output, and tests, and the canonical comparison-execution,
+  application-orchestration, CLI, operations, and bundle contracts.
+- Exclusions: Repository-wide hermeticity, portfolio coverage, maintenance,
+  efficiency, and refactoring remain Stages 19-23; documentation-wide
+  reconciliation remains Stage 24; adversarial runtime robustness remains
+  Stage 25; and finding triage/remediation remain Stages 26-27. Stage 17 owns
+  bundle-model and filesystem-transaction defects; Stage 14 owns prepared
+  execution provenance.
+
+#### Comparison Execution And Result Accounting
+
+| Contract or risk | Execution, publication, CLI, and test evidence | Disposition |
+| --- | --- | --- |
+| Ordered explicit selection and sequential preflight | CLI parsing preserves repeated profile order and validates exact distinct IDs before config/executor work. Application inspection resolves the prompt once, then profiles sequentially in supplied order, stopping at the first failure before collection. Resolved prompt identity survives later preflight failures in the partial result. | Coherent. Generated-text catalog timing and provenance reconciliation remain `AUD-043` and `AUD-044`. |
+| Collect/prepare once and identical immutable input | After destination, debug, prompt, profile, and credential preflight, comparison calls weather collection and `prepareReport` once. Every execution receives an independent byte copy of the same serialized YAML plus cloned prepared values, so provider mutation cannot affect peers. Focused concurrent tests compare the captured package bytes. | Coherent and retained as `RET-039`. |
+| One executor, concurrency, and backend capacity | CLI constructs one Promptkit adapter and passes it into one comparison request. Application starts one goroutine per selected profile; the backend owns configured provider capacity rather than an application semaphore. Production execution creates a private prepared handle per call. | Ownership matches the contract. Extreme profile-count cost is routed to Stage 22, and the interface comment gap to Stage 24. |
+| Goroutine lifecycle and shared state | Each goroutine captures its own loop index/profile, writes only that preallocated slice element, and decrements one wait group. The caller waits for every started goroutine before reading outcomes or returning. Immutable prepared state and concurrency-safe executor/debug boundaries are shared; the scoped race suites pass. | Joined and race-clean for covered production boundaries; retained as `RET-039`. |
+| Profile isolation and cancellation | Generation, validation, render, and debug failures stay in their own safe bounded outcome while peers continue. Context reaches every execution, prevents later starts when observed, joins started work, and prevents publication. The post-join sweep nevertheless overwrites an independent completed failure when any peer makes cancellation observable. | Mixed failure identity is incorrect in `AUD-058`; all-canceled joining and failure-without-cancellation have focused tests. |
+| Deterministic result and debug identity | Results are initialized and stored by caller position, copied to the application result in that order, encoded into the manifest in that order, and projected by CLI without sorting. Filenames use position/count/profile identity; debug run IDs include comparison identity, ordinal width, and profile slug. Inverse completion and distinct-debug tests cover both properties. | Completion order cannot change durable or CLI order; retained as `RET-039`. Canonical filename validation remains `AUD-054`. |
+| Partial and all-failed publication | After all noncanceled work joins, one logical bundle contains the exact shared package and only successful Markdown reports. Independent failures do not suppress publication; both partial-success and all-failed manifests commit, then the application returns a comparison aggregate error. Paths become visible only after commit. | Coherent and retained as `RET-040`. Execution provenance remains `AUD-044`. |
+| Publication and post-commit cleanup failures | Pre-commit destination/publication errors leave manifest, data-package, and report paths empty. A committed bundle followed by backup-cleanup failure retains authoritative target paths and profile report paths while returning a typed failure. CLI maps destination and cleanup errors to stable content-safe categories. | End-to-end projection is truthful for the currently reported package state. Underlying cancellation/partial-cleanup defects remain `AUD-055` and `AUD-056`. |
+| Never-notify policy | `ComparisonRequest`, comparison orchestration, and CLI comparison wiring expose no notifier. The complete call path ends at bundle publication and aggregate result construction, never Distributor delivery. | Matches policy and retained in `RET-040`. |
+| Safe summaries and exit behavior | Result-bearing outcomes emit one ordered JSON summary unless quiet. Per-profile errors are already bounded `SafeError` values; top-level aggregate, prompt, destination, cleanup, cancellation, and deadline errors map to fixed safe categories/messages. Success requires all profiles plus committed manifest/data paths; partial, all-failed, publication, cleanup, and cancellation paths return an error and failed status. | Coherent and retained as `RET-041`. Pre-result input/config failures deliberately remain ordinary CLI diagnostics. |
+| Test ownership and cost | Application tests own order under concurrent completion, equal input bytes, capacity observation, failure isolation, cancellation joining, debug identities, collect/prepare/publish ordering, partial/all-failed bundles, destination changes, preservation, and cleanup paths. CLI tests own parsing/wiring, one executor, safe structured outcomes, quiet mode, committed cleanup, and status. | Focused ownership is strong except for the mixed cancellation overlap in `AUD-058`; broader suite and cost review remain Stages 19-22. |
+
+#### Commands And Evidence
+
+- Used codebase graph symbol and augmented-code search, exact snippets, and
+  caller/data-flow inspection for comparison orchestration, profile execution,
+  cancellation marking, bundle/result copying, CLI request construction, safe
+  summaries, and comparison tests. Bounded text inspection covered the scoped
+  production and test paths, exact Stage 18 boundary, task-specific canonical
+  documents, and all repository policies.
+- Traced pre-result validation, destination/debug/prompt/profile preflight,
+  collection/preparation, concurrent success/failure/cancellation, logical
+  bundle validation, pre-commit publication failure, committed cleanup failure,
+  aggregate failure, quiet output, and CLI result/exit projection. Cross-checked
+  ordered identities and artifact paths at every handoff.
+- Ran a temporary deterministic two-profile probe in which one provider failure
+  completed before its still-running peer was canceled. It proved the joined
+  cancellation sweep changed the completed outcome from `generation` to
+  `canceled`; the probe was removed without retaining source or test changes.
+- Ran `go test -race` with focused coverage for `./internal/app` and
+  `./internal/cli`; both passed with 76.0% and 75.8% statement coverage.
+  Coverage guided missing-overlap inspection but was not itself treated as a
+  finding. Ran `go test ./...`, `go vet ./...`,
+  `go run ./cmd/weatherreporter --help`, and `git diff --check`; all passed.
+- Findings: `AUD-058`.
+- Retained decisions: `RET-039` through `RET-041`.
+- Open questions: the three leads recorded above are routed to their assigned
+  later stages.