Audit state persistence and file safety

docs/roadmap/audit.md

@@ -19,10 +19,10 @@ change only roadmap audit documents do not change that production target.
 ## Executive Summary
 
 Pending final synthesis. The initial baseline is healthy. The architecture,
-configuration/CLI, pipeline composition, reference/handoff, and runtime reviews
+configuration/CLI, pipeline composition, reference/handoff, runtime, and state
-have found two High findings, three Medium findings, and eight Low findings,
+reviews have found two High findings, four Medium findings, and ten Low
-with no production dependency inversion, unbounded framework worker pool, or
+findings, with no production dependency inversion, unbounded framework worker
-completion-order-dependent result assembly.
+pool, completion-order-dependent result assembly, or debug-to-cache coupling.
 
 ## Finding Index
 
@@ -42,6 +42,9 @@ Final cross-area ordering is pending synthesis.
 | RUN-002 | High | Correctness | Isolate typed validator values from stage output |
 | RUN-003 | Medium | Correctness | Preserve warnings from the terminal rejected attempt |
 | RUN-004 | Low | Documentation/Comments | Document the lane collector's liveness invariant |
+| STATE-001 | Medium | Correctness | Preserve distinct identities in state paths |
+| STATE-002 | Low | Efficiency | Remove redundant post-decode clones from canonical codecs |
+| STATE-003 | Low | Duplication | Publish output through the confined file writer |
 
 ## Findings
 
@@ -455,6 +458,114 @@ Final cross-area ordering is pending synthesis.
   continuation-overlap, worker-bound, race, and shuffled tests.
 - **Grouping:** Independent.
 
+### State, Checkpoints, Debugging, And File Safety
+
+### STATE-001 — Preserve distinct identities in state paths
+
+- **Severity:** Medium
+- **Category:** Correctness
+- **Evidence:** The resolver accepts any non-empty trimmed ordered-step and
+  lane IDs (`internal/framework/pipeline/profile.go:341`–`351` and
+  1382–1392). Checkpoint path construction then sends both IDs through
+  `checkpointPathComponent` (`internal/framework/checkpoint/recorder.go:476`–
+  `508`), which maps `.`, `..`, and every value containing `..` to the single
+  component `_`. The trace side duplicates that encoder in
+  `debugPathComponent` (`internal/framework/pipeline/debug.go:37`–`62`), and
+  `cleanDebugPath` at lines 364–374 additionally applies `path.Clean` before
+  encoding. Thus valid distinct identities such as `.` and `_`, or `a..b` and
+  `_`, select the same checkpoint component and can also collapse to the same
+  debug path. Existing filesystem tests cover traversal and symlink rejection,
+  but no focused test asserts that accepted identities map injectively.
+- **Impact:** Two valid lanes or steps in one resolved pipeline can overwrite
+  each other's checkpoint manifests and payloads. Exact manifest validation
+  prevents a mismatched artifact from being silently reused, but ordinary
+  resume repeatedly invalidates and re-executes the colliding work, and a
+  required selective-recompute predecessor can become unavailable after its
+  sibling overwrites it. Requested debug records can likewise overwrite or be
+  attributed to the wrong identity. Atomic rename does not protect against a
+  logical-name collision.
+- **Recommendation:** Introduce one shared, injective path-component encoding
+  for accepted application identities. Preserve already-safe components, but
+  encode reserved dot components and every unsafe rune rather than replacing a
+  whole value with `_`; do not run identity-bearing debug paths through a
+  lossy clean operation first. Keep path joining and confinement separate from
+  identity encoding.
+- **Preserve:** Retain trimmed non-empty identity validation, human-readable
+  ordinary IDs, exact step/lane checks inside checkpoint manifests, narrow
+  relative state paths, symlink rejection, and recoverable cold execution for
+  incompatible reconstructible cache entries.
+- **Validation:** Add table and uniqueness cases covering `.`, `..`, `_`,
+  embedded `..`, separators, tildes, and Unicode, then run a two-identity
+  filesystem checkpoint/resume and debug trace case that proves distinct files
+  and successful accepted-normalize hydration. Include selective recomputation
+  so a required predecessor remains reusable.
+- **Grouping:** Independent.
+
+### STATE-002 — Remove redundant post-decode clones from canonical codecs
+
+- **Severity:** Low
+- **Category:** Efficiency
+- **Evidence:** `chunkmap.Codec.Decode` schema-validates and decodes JSON into a
+  fresh value, canonicalizes that value, then deep-clones it again before
+  returning (`internal/framework/chunkmap/codec.go:149`–`171`).
+  `evidencecontext.Codec.Decode` follows the same sequence
+  (`internal/framework/evidencecontext/codec.go:79`–`101`), while its
+  `canonicalize` already deep-clones the decoded document at lines 169–203;
+  the final `clone` at lines 299–320 therefore makes a second complete copy of
+  context/reference slices, source-unit structs, and nested metadata. Unlike an
+  encode caller, the JSON decoder retains no caller-owned object graph that
+  must be protected from canonicalization.
+- **Impact:** Hydrating or consuming canonical artifacts performs avoidable
+  slice and nested-metadata allocations. The evidence-context path can clone a
+  large source-derived object graph twice after JSON decoding, increasing peak
+  allocation and latency without adding an ownership boundary.
+- **Recommendation:** Separate canonicalization of an already-owned decoded
+  value from the defensive-copy entry point used by encoders and external
+  callers. Return the canonical decoded value directly once validation and
+  normalization succeed; retain exactly one clone wherever caller-owned input
+  could otherwise be mutated.
+- **Preserve:** Keep schema validation, unknown-field and trailing-value
+  rejection, digest reconstruction, annotation/context normalization,
+  encode-time caller isolation, and a fully independently owned decode result.
+- **Validation:** Retain and extend ownership tests that mutate both the input
+  value after encode and the returned value after decode, then use focused
+  allocation assertions or benchmarks for a nested chunk map and evidence
+  context to confirm that decode no longer performs the redundant deep copy.
+- **Grouping:** Independent.
+
+### STATE-003 — Publish output through the confined file writer
+
+- **Severity:** Low
+- **Category:** Duplication
+- **Evidence:** The CLI's private `writeFileAtomic`
+  (`internal/cli/run.go:826`–`856`) independently implements the same
+  create-temp, write, chmod, close, rename, and cleanup sequence as
+  `internal/core/fileio.writeAtomic` (`internal/core/fileio/fileio.go:105`–
+  `133`). `writeOutputFiles` calls the CLI copy after its own path checks and
+  directory creation (`internal/cli/run.go:754`–`787`), while
+  `fileio.WriteBytes` additionally centralizes relative-path confinement and
+  symlink-component rejection. Debug summaries, traces, and checkpoints
+  already use that core primitive; durable output is the divergent state
+  writer.
+- **Impact:** Atomic-publication fixes must be made and tested in two places,
+  and the copies have already drifted in their confinement checks. The fresh,
+  exclusively created run directory limits current output exposure, so this is
+  a maintenance and defense-in-depth issue rather than an observed escape.
+- **Recommendation:** After prevalidating all logical output names and
+  exclusively creating the run directory, publish each file through
+  `fileio.WriteBytes` (or a narrowly exported equivalent) using the run
+  directory as root. Remove the CLI atomic-write copy rather than introducing
+  another generic filesystem abstraction.
+- **Preserve:** Validate every logical name before allocating state, refuse an
+  existing run directory unchanged, retain a newly created partial bundle on a
+  later write failure, keep output directory/file modes at 0755/0644, and
+  preserve logical-name context in errors.
+- **Validation:** Retain the existing output ordering, collision, permission,
+  and partial-bundle cases; add a symlink-component case at the shared writer
+  boundary and verify no temporary file remains after write, chmod, close, or
+  rename failure.
+- **Grouping:** Independent.
+
 <!--
 Finding template for later audit stages:
 
@@ -567,6 +678,26 @@ Finding template for later audit stages:
   invokes only its wrapped collaborator, so there is no framework lock-order
   cycle; removing them would push concurrency requirements into filesystem and
   test implementations.
+- Filesystem checkpoint recorder and loader methods intentionally repeat the
+  stage-shaped interface surface. Each small adapter fixes a stage, manifest
+  status, dependency set, and payload codec before delegating to shared
+  manifest validation or publication. Collapsing them into a reflection- or
+  string-driven state engine would weaken typed call sites and contextual
+  validation; STATE-001 concerns only their shared path-component encoding.
+- The chunk-plan store intentionally does not use the general `fileio` writer.
+  Its source-addressed directory is an independently replaceable cache entry,
+  and its `os.Root`-relative implementation rejects non-directory digest
+  entries, non-regular or symlinked plans, named pipes, and replacement races,
+  while syncing a private temporary file before rename. Those stronger
+  cache-entry mechanics should remain local rather than being generalized into
+  output/debug/checkpoint publication.
+- Source metadata, accepted chunk maps, and evidence contexts retain separate
+  clone helpers because their ownership graphs and canonical invariants differ:
+  source metadata handles nested dynamic values and cycles, chunk maps own
+  annotations and chunk slices, and evidence contexts own source units and
+  metadata. STATE-002 removes only copies of an already-owned decoded graph; it
+  should not replace these helpers with reflection or remove encode-time
+  isolation.
 
 ## Areas Reviewed Without Findings
 
@@ -855,6 +986,89 @@ Finding template for later audit stages:
   branches. The consequential missing runtime cases are included in RUN-001
   through RUN-003 rather than duplicated as test-only findings.
 
+### State, Checkpoints, Debugging, And File Safety
+
+- **Independent state lifecycles:** `runPipelineCommand` selects output,
+  chunk-plan, checkpoint, and debug roots only in the CLI. Bypass returns before
+  resolving or constructing a chunk-plan store; disabled checkpoints construct
+  only no-op collaborators and reject resume; enabled recording always creates
+  a recorder but creates a loader only for resume; and debug allocation occurs
+  only after an explicit debug request. None of these roots enters a module or
+  another state family's identity, and debug records are never read by reuse
+  code.
+- **Chunk-plan cache:** The store accepts only a canonical lowercase SHA-256
+  source digest as its key, opens the exact digest directory through `os.Root`,
+  rejects symlink/non-directory digest entries and symlink/non-regular plan
+  files, strictly decodes one schema-versioned JSON value, revalidates plan
+  provenance and digest, and returns typed missing/invalid/hit decisions. A
+  reused plan is cloned, materialized against the current document, and passed
+  through the whole-plan validators. Only a newly accepted plan is published;
+  random 0600 temporary files are synced and renamed within the confined 0700
+  entry. Missing, invalid, or deleted entries are reconstructible.
+- **Checkpoint identity and ordinary resume:** Identity hashes the complete
+  resolved pipeline, raw input, selected lanes, runtime overrides, external and
+  generated reference provenance, observed profile/runtime provenance, and
+  prepared component fingerprints. The directory uses readable/prefix
+  components, while every manifest retains and validates the complete identity
+  digest and exact stage/step/lane/module/dependencies. Payload publication
+  precedes a succeeded manifest, and the manifest retains payload digests, so
+  interruption or a changed dependency produces execution/invalidation rather
+  than stale reuse. STATE-001 records the remaining non-injective step/lane
+  component mapping.
+- **Selective recomputation:** The CLI computes a forward forced closure and a
+  backward set of required reusable producers. The runner applies
+  `forced_recompute` before decoding an otherwise reusable artifact. Required
+  producers use only an accepted workspace-v3 normalize manifest with exact
+  invocation, producer provenance, artifact digest, active codec/schema/media,
+  and canonical bytes; they deliberately do not require extract/merge state or
+  current consumer dependencies. A failed required load records its typed
+  decision and stops before producer or consumer execution.
+- **Decisions and diagnostics:** Checkpoint readers assign category and reason
+  code at the validation branch for missing, path, read, decode, schema,
+  identity, stage, dependency, payload, digest, and reuse outcomes. Runner
+  hydration adds codec and canonicality codes, forced policy adds the recompute
+  code, and required-predecessor errors name only stable step/lane identity and
+  code. Event detail is selected from code-owned text, UTF-8 normalized, and
+  bounded rather than copied from a cache error or caller payload.
+- **File safety and recovery:** Output names are all validated before exclusive
+  run-directory creation; existing output/debug run directories are refused;
+  later output failure intentionally retains the new partial bundle. General
+  state writes use same-directory temporary files and atomic rename after
+  narrow relative-path checks, with 0700/0600 cache/debug modes and 0755/0644
+  output modes. General file I/O rejects existing symlink components; the
+  chunk-plan store additionally uses root-relative no-follow opens and
+  race-oriented entry rechecks. No production path automatically moves,
+  deletes, or cleans output, cache, or completed debug state; debug allocation
+  removes only its just-created partial leaf on setup failure.
+- **Debug and terminalization:** A bundle owns distinct summary and trace
+  directories. Summary records use redacted invocation/configuration and
+  bounded checkpoint decisions; trace requests receive cloned application
+  payloads and no environment enumeration. Debug write failures fail the
+  requested run but cannot affect checkpoint identity or reuse. Guarded
+  terminalization writes one report, preserves an existing primary error over
+  report failure, attempts one error log, and joins persistence failures only
+  as secondary diagnostics. STATE-001 records trace-name collisions, not a
+  cache dependency or redaction leak.
+- **Canonical ownership:** Source digesting uses canonical document/plan
+  serialization and cycle-aware metadata cloning. Chunk-map and evidence-
+  context codecs strictly validate schema, reject unknown fields and trailing
+  values, canonicalize nested identities/annotations/context, and return owned
+  graphs. Their encode-time clones are required ownership boundaries;
+  STATE-002 records only the redundant copies after decoding already-owned
+  JSON. STATE-003 records the one extractable duplicate writer; checkpoint
+  stage adapters, cache-specific `os.Root` mechanics, and type-specific clone
+  helpers remain intentional.
+- **Focused test review:** File-I/O and debug-bundle tests cover confinement,
+  atomic replacement, symlinks, modes, refusal, cleanup, and redaction.
+  Checkpoint tests cover identity, manifests, interrupted publication,
+  categories/reason bounds, dependency invalidation, accepted normalize, and
+  codec canonicality. Chunk-plan tests cover corrupt entries, races, named
+  pipes, strict decode, materialization, and permissions; chunk-map and
+  evidence-context tests cover canonical round trips and ownership. CLI cache,
+  recomputation, state-hardening, run-contract, and terminal tests cover the
+  composed lifecycles and primary-error precedence. The uncovered identity,
+  copy, and writer issues are recorded as STATE-001 through STATE-003.
+
 ## Validation Record
 
 | Date | Scope | Command or check | Result |
@@ -884,6 +1098,10 @@ Finding template for later audit stages:
 | 2026-08-08 | Pipeline race tests | `go test -race ./internal/framework/pipeline` | Pass |
 | 2026-08-08 | Fresh pipeline tests | `go test -count=1 ./internal/framework/pipeline` | Pass |
 | 2026-08-08 | Shuffled pipeline tests | `go test -shuffle=on ./internal/framework/pipeline` | Pass |
+| 2026-08-08 | Audit target integrity before state review | `git diff --quiet 92e89076a268089e703978fb9d7176200e93344c..HEAD -- . ':(exclude)docs/roadmap/**'` | Pass; production target unchanged |
+| 2026-08-08 | State graph and lifecycle review | Exact symbol reads, complexity and call traces across CLI root construction, core file I/O/debug allocation, chunk-plan storage, checkpoint identity/loading/recording, runner reuse decisions, trace/summary writers, terminalization, and canonical codecs | STATE-001 through STATE-003 recorded; independent lifecycle, typed decisions, accepted hydration, debug isolation, and primary-error precedence otherwise confirmed |
+| 2026-08-08 | State collaborator race tests | `go test -race ./internal/core/fileio ./internal/core/debugbundle ./internal/framework/checkpoint ./internal/framework/chunkplan ./internal/framework/chunkmap ./internal/framework/debug ./internal/framework/evidencecontext` | Pass |
+| 2026-08-08 | CLI state and terminal tests | `go test ./internal/cli` | Pass |
 
 ## Coverage Matrix
 
@@ -894,7 +1112,7 @@ Finding template for later audit stages:
 | Pipeline resolution, preparation, and typed registries | Reviewed | Internal pipeline/module docs; `internal/framework/contracts/`; pipeline profile, options, module, construction, preparation, fingerprint, stage/validator/chain/codec/evidence registry implementations and focused tests | Target-integrity check, graph architecture/complexity/call traces, focused tests and vet | PIPE-001, PIPE-002 |
 | References and ordered handoffs | Reviewed | Reference and ordered-step sections of configuration, internal pipeline, and state docs; pipeline reference resolution/materialization, preparation ownership, generated handoff, consumer fingerprint, checkpoint hydration, and runner barriers; CLI selector/recomputation owners; focused profile, reference, handoff, checkpoint, recomputation, and assembled integration tests | Target-integrity check, graph architecture/complexity/call traces, focused pipeline/CLI tests, assembled integration tests | REF-001, REF-002, REF-003 |
 | Execution, validation, retry, and concurrency | Reviewed | Internal pipeline runtime contract; runner, chunk planning/validation, concurrent lane engine, typed execution/validation, retry/normalize, synchronization, output suppression, and focused concurrency, retry, rejection, session, typed-checkpoint, debug, manifest, candidate-encoding, and handoff tests | Target-integrity check, graph state-machine/call/complexity review, race tests, fresh tests, shuffled tests | RUN-001, RUN-002, RUN-003, RUN-004 |
-| State, checkpoints, debugging, and file safety | Pending | — | — | — |
+| State, checkpoints, debugging, and file safety | Reviewed | State and operations docs plus architecture state/security policy; CLI output, cache-root, checkpoint identity, recomputation, debug allocation, and terminal owners; core file I/O, debug bundle, and source digest/clone helpers; framework checkpoint, chunk-plan, chunk-map, debug, evidence-context implementations and focused tests | Target-integrity check, graph architecture/complexity/call traces, state collaborator race tests, CLI tests | STATE-001, STATE-002, STATE-003 |
 | LLM runtime, prompt filesystems, and assets | Pending | — | — | — |
 | Generic and Seriatim modules | Pending | — | — | — |
 | Shared D&D types, codecs, and family mechanics | Pending | — | — | — |