Files
narratio/docs/roadmap/implementation.md

56 KiB
Raw Blame History

Audit Remediation Implementation Plan

Purpose and status

This document is the executable roadmap for remediating the confirmed findings in audit-findings.md. It is written for a gpt-5.6-terra coding agent that will implement exactly one stage per prompt, in order.

The audit is an immutable requirements and evidence ledger. Do not edit audit-findings.md, reinterpret accepted findings, or copy its detailed analysis into this document. The line ranges below are stable references to the evidence and acceptance criteria that the implementing agent must read for each stage.

This plan owns remediation progress; a separate audit.md tracker is not needed. All stages are pending when this plan is created.

Stage Summary Primary findings Status
1 Align data classification and group workspace modes RSK-004 Completed
2 Enforce safe identifiers and fuzz path/source contracts COR-002, TST-013 Completed
3 Consolidate crash-durable atomic file replacement RSK-002, DUP-001, DUP-005 Completed
4 Add confined destination and download/install capabilities COR-003, DUP-003, TST-003 Completed
5 Confine recursive cleanup and replace sentinel locks RSK-003 Completed
6 Harden API-key file acquisition RSK-010 Completed
7 Bound and verify external result acquisition RSK-013, TST-007 Completed
8 Terminate owned subprocess trees RSK-011 Completed
9 Redact and cap subprocess diagnostics RSK-012 Completed
10 Confine publish archive reads COR-005 Completed
11 Make manifest and run identity singular COR-001, TST-006 Completed
12 Centralize handled terminal-failure persistence RSK-001, TST-002, SIM-001, COM-001 Completed
13 Introduce the immutable remote-commit model and legacy boundary ARC-003 Completed
14 Publish through immutable commits and canonical mappings COR-004, COR-011, DUP-002, TST-004 Completed
15 Make remote locks generation-safe and harden pagination RSK-005, RSK-014 Completed
16 Persist retryable post-commit cleanup state COR-006, COR-007 Completed
17 Bind restore/status to a committed snapshot and reject conflicts COR-008, COR-009, TST-005 Completed
18 Serialize restore transitions and make restored paths portable RSK-006, RSK-008 Completed
19 Bind audio cache reuse to remote object identity RSK-007 Completed
20 Unify previous-source readiness and eliminate duplicate transfers COR-010, EFF-001 Completed
21 Tighten configuration parsing, values, and expectations COR-012COR-015, TST-011, TST-014 Pending
22 Make product configuration truthful and own remote temp files COR-024, RSK-009, ARC-004 Pending
23 Stream WhisperX uploads and make the adapter race-safe COR-016, EFF-002, TST-001 Pending
24 Correct prepare/transcribe transition semantics COR-017COR-019, TST-008 Pending
25 Enforce output-path authority and shared singleton resolution ARC-006, DUP-006 Pending
26 Centralize typed extraction-bundle evidence DUP-007 Pending
27 Bind extraction reuse to direct transcript identity COR-020, TST-009 Pending
28 Establish one effective artifact set and catalog bootstrap COR-022, ARC-007, DUP-008 Pending
29 Make analyze resolution typed, optional, actionable, and deterministic COR-021, COR-023, RSK-015, SIM-003, TST-010 Pending
30 Remove misleading contracts and align Audita ownership ARC-001, ARC-005, SIM-004, COM-003, COM-006 Pending
31 Enforce CI validation and streamline the assembled test suite TST-012, TST-015 Pending
32 Reconcile lifecycle/analyze documentation and close the remediation COM-002, COM-005 Pending

Governing decisions

The following decisions are settled requirements, not questions for the implementing agent:

  1. Ordinary Narratio and Notarius campaign/session data is deliberately shareable, not private or sensitive. Group-owned, group-writable workspaces are an operational requirement. API keys are the only sensitive data handled by these applications and must remain private.
  2. On POSIX systems, Narratio-managed ordinary workspace directories and files should converge on setgid 02775 and 0664 respectively, inheriting the workspace's existing group. API-key directories and files must be 0700 and 0600. Do not add ownership-changing behavior or assume Narratio may chown. Windows behavior must preserve the same collaboration/security intent using the platform's available guarantees and explicit operational guidance.
  3. Remote current-state publication uses an immutable run-scoped commit manifest selected by one final pointer update. Readers may accept coherent legacy state, but all compatibility code must be isolated behind a clearly named boundary, covered by dedicated tests, and carry an in-code removal comment. New writers must never produce the legacy format.
  4. Explicit notification no-op is supported. Configured notification backends or recipients are rejected until a real provider exists.
  5. External-output limits are runaway safeguards, not normal operating limits. Each limit must be a named, centrally discoverable package constant and at least an order of magnitude above expected ordinary output. Every limit error must identify the enforcing adapter/contract, the configured byte limit, and what exceeded it. Narratio-owned stdout/stderr capture is streaming and capped; reaching its cap terminates the owned process tree. Hard whole-disk protection is an operations concern implemented with filesystem/service/container/volume quotas, not output-root polling.
  6. Retain every accepted-risk disposition in the audit. In particular, do not turn abrupt process-death detection into a distributed liveness system, do not add rollback of arbitrary external side effects, and do not add generic speculative abstractions rejected by the audit.
  7. CI must run ordinary tests, vet, build, and example checks on every change and release path; the complete race suite runs on every change once TST-001 is repaired; shuffled tests run on a schedule. Add native macOS/Windows jobs only when authoritative Woodpecker runner labels and usable runners are available. Do not invent labels. Cross-compilation is useful but does not count as native filesystem/process evidence.

Instructions for every stage

For each implementation prompt, the coding agent must:

  1. Read docs/development.md, then the exact audit ranges and canonical documents named by that stage. Follow the task-specific reading guide in docs/development.md. Do not read all 4,000+ audit lines unless a stage explicitly requires them.
  2. Inspect the current tree before editing. Earlier stages may have changed names and ownership boundaries. Use the repository knowledge graph first for code discovery and call tracing, then use text search for documentation, string literals, configuration, and evidence the graph cannot supply.
  3. Confirm the worktree state and preserve unrelated user changes. Do not rewrite or amend earlier commits unless explicitly instructed.
  4. Implement the stage completely, including production code, focused regression tests, platform-specific implementations where applicable, current-behavior documentation, and removal of code made obsolete by the stage. Do not leave a second competing path or defer required tests to a later stage unless this plan explicitly says so.
  5. Prefer typed, owner-specific contracts and existing package ownership. Do not create a generic abstraction merely because two call sites look similar. Keep policy at the owner and mechanism in internal/fileops or the relevant adapter/package.
  6. Treat path/race/security checks as production behavior. Validation followed by an unconfined pathname operation is not a completed fix. Unsupported safe behavior must fail closed with an actionable error.
  7. Run focused tests while developing, then at minimum go test ./... for every stage. For production-code changes also run go vet ./..., go build ./..., and the repository's documentation/example validation command. Run targeted -race tests for concurrency changes and platform build/test checks named by the stage. Do not require live cloud or third-party services.
  8. Compare the final diff with the stage's finding list and exit criteria. Update only that stage's row in the table from Pending to Completed, adding the implementing commit hash if the workflow supplies one. Do not mark a stage complete while a required check is failing or a required behavior is missing.
  9. Leave docs/roadmap/audit-findings.md unchanged. If implementation evidence reveals that a confirmed finding is factually impossible to satisfy as written, stop and report the contradiction instead of silently changing its meaning.

Stage 1 — Align data classification and group workspace modes

Read first: audit-findings.md lines 26602693 (RSK-004), 41714224 (resolved decisions and accepted risks); docs/policy/architecture.md; docs/operations.md; and the workspace/artifact/file-operation package docs named by docs/development.md.

Goal: Make the architecture and implementation consistently describe and support shareable, group-owned workspaces while keeping API keys outside that ordinary-data policy.

Work:

  • Revise the architecture policy's current-behavior contract: campaign/session, manifests, transcripts, generated configs, logs, and Notarius artifacts are intentionally shareable; API-key material is sensitive.
  • Identify every Narratio-owned ordinary workspace creation/promotion path. Give the owning packages named, discoverable mode constants and converge final POSIX directories/files to setgid 02775/0664, without changing ownership.
  • Preserve the parent workspace group through nested creation and atomic replacement. Account for restrictive caller umasks explicitly; do not rely on a favorable umask to add group write or setgid bits.
  • Keep temporary-file exposure no broader than the intended final object and do not weaken the separate API-key rules implemented in Stage 6.
  • Document Windows ACL/ownership expectations and any mode-bit limitations; do not claim POSIX mode semantics on Windows.

Tests and exit criteria: Add focused tests under restrictive umasks for nested directories, final files, promoted Notarius trees, and repeated replacement. Verify that group-write/setgid behavior is correct on POSIX and platform behavior is explicit elsewhere. The architecture, operations, and security statements must agree. No test should assert that ordinary campaign output is private.

Stage 2 — Enforce safe identifiers and fuzz path/source contracts

Read first: audit-findings.md lines 17701810 (COR-002) and 38373852 (TST-013); configuration, manifest, workspace, artifact, and source-grammar docs.

Depends on: Stage 1.

Goal: Prevent externally influenced identity tokens from becoming absolute, traversing, platform-ambiguous, or control-bearing filesystem segments.

Work:

  • Define one strict portable segment grammar for campaign IDs, session IDs, run IDs, artifact identifiers, and other opaque filesystem identity tokens: ASCII letters, digits, ., _, and - only; reject empty, ./.., separators, drive/volume syntax, absolute forms, control characters, and all other bytes.
  • Enforce the contract at configuration/model construction and again at the artifact/path boundary. Return actionable errors; never sanitize or silently rewrite identity.
  • Audit manifest/source parsing for equivalent tokens. Unsafe legacy values must fail with migration guidance, not be accepted because an older file contains them.
  • Preserve intentional multi-segment paths only in typed path contracts that are separately confined; do not force path-valued fields through the opaque-ID API.

Tests and exit criteria: Add table and fuzz tests for Unix/Windows separators, drive forms, Unicode lookalikes, controls, empty/dot segments, and valid boundary values. Include source-token parsers as required by TST-013. Prove all path builders reject unsafe identifiers before filesystem access.

Stage 3 — Consolidate crash-durable atomic file replacement

Read first: audit-findings.md lines 25902624 (RSK-002), 34073427 (DUP-001), and 34933514 (DUP-005); internal/fileops and manifest persistence docs.

Depends on: Stages 12.

Goal: Provide one owner-neutral byte-file installation mechanism with explicit durability semantics, while leaving serialization and policy with callers.

Work:

  • Implement or complete a narrowly scoped internal/fileops byte-write/replace primitive: sibling temporary file, complete write, file sync, atomic rename or documented platform replacement, then parent-directory sync where supported.
  • Accept the caller-owned final mode; make cleanup and error composition explicit. Preserve cancellation checkpoints in callers before durable commit.
  • Migrate duplicate local manifest, artifact, and adapter byte-write helpers to the primitive. Keep manifest encoding/versioning out of fileops.
  • Define and test the platform contract for directory sync and replacement. A platform limitation must be explicit, not silently treated as equivalent.

Tests and exit criteria: Exercise short writes, write/sync/rename/directory-sync failures through injected seams, repeated replacement, mode preservation, and temporary cleanup. Existing manifest and adapter behavior must remain compatible. No duplicate sibling-temp byte writer should remain.

Stage 4 — Add confined destination and download/install capabilities

Read first: audit-findings.md lines 18111852 (COR-003), 34483471 (DUP-003), and 35963617 (TST-003); filesystem threat model and internal/fileops docs.

Depends on: Stages 23.

Goal: Make destination mutation resistant to ancestor and leaf symlink races, and share the safe remote-download installation mechanism without sharing caller policy.

Work:

  • Add root-relative, handle-confined creation, replacement, and promotion capabilities that verify/traverse every ancestor without following symlinks. Use platform-specific implementations where necessary and fail closed if the required guarantee is unavailable.
  • Replace validate-then-use pathname mutations in artifact creation, manifest installation, Notarius tree promotion, restore/download installation, and related destinations.
  • Implement one destination-confined sibling-temp download/install mechanism. Callers continue to own object identity, verification, force/conflict policy, and final mode.
  • Do not implement recursive deletion here; Stage 5 owns that distinct capability.

Tests and exit criteria: Add adversarial tests that replace intermediate directories and leaves with symlinks between logical phases, plus regular success, collision, cleanup, and platform tests. Prove no external target is modified. TST-003 is primarily owned by this stage and may receive additional cleanup cases in Stage 5.

Stage 5 — Confine recursive cleanup and replace sentinel locks

Read first: audit-findings.md lines 18111852 (COR-003), 26252659 (RSK-003), and 35963617 (TST-003); local locking and workspace lifecycle docs.

Depends on: Stage 4.

Goal: Make deletion root-confined and make local crash recovery rely on OS-released locks rather than stale file existence.

Work:

  • Add a root-relative, no-follow cleanup/delete capability and migrate recursive cleanup paths. Never pass an unchecked or re-resolved absolute path to broad recursive deletion.
  • Replace existence-based sentinel locking with a held OS lock whose ownership is released by the kernel on process death. Retain useful owner/run metadata without using metadata-file existence as the lock decision.
  • Implement explicit Unix and Windows lock backends, wait/contention behavior, cancellation, and release-error reporting. Do not auto-delete another process's active lock.
  • Ensure lock files and cleanup roots follow the group workspace policy where they are ordinary runtime data.

Tests and exit criteria: Cover contention across independent processes, process-death recovery, cancellation, release failures, symlink swaps, malicious cleanup inputs, and idempotent cleanup. Run native tests where available and at least cross-build all platform-specific files.

Stage 6 — Harden API-key file acquisition

Read first: audit-findings.md lines 28702900 (RSK-010); architecture, configuration, and integration docs governing API keys.

Depends on: Stages 1, 2, and 4.

Goal: Make the sole sensitive-data path private, bounded, regular-file-only, and resistant to path races.

Work:

  • Resolve API-key directories and files through a root-confined, no-follow read path. Reject symlinks, non-regular files, unsafe relative names, replacement races, and oversized content.
  • Define the small API-key byte limit as a named, discoverable constant in the owning package. Errors must name the API-key loader and the limit.
  • Enforce 0700 directory and 0600 file privacy for Narratio-created secret storage on POSIX, and reject unsafe existing modes where Narratio reads key files. Document Windows ownership/ACL requirements without pretending mode bits are authoritative there.
  • Preserve documented environment/file precedence without persisting or logging secret values.

Tests and exit criteria: Cover mode rejection, symlink/ancestor swaps, non-regular files, over-limit reads, valid newline trimming, precedence, and error redaction. No API-key content may appear in failures or fixtures intended for logs.

Stage 7 — Bound and verify external result acquisition

Read first: audit-findings.md lines 29633006 (RSK-013), 36913718 (TST-007), and 37443774 (TST-009); adapter contracts and integration docs.

Depends on: Stages 4 and 6.

Goal: Prevent a runaway adapter or producer from forcing unbounded memory or post-process disk consumption while leaving abundant headroom for real sessions.

Work:

  • Factor the non-secret portions of the Stage 6 read mechanism into an owner-neutral bounded, regular-file, no-follow acquisition primitive if that does not already exist. Keep each contract's limit in its owning package.
  • Inventory every externally produced file Narratio reads after a process exits: adapter JSON, Audita output, Scriptorium output, WhisperX result, Notarius lane result/catalog/bundle evidence, and any equivalent stage handoff. Migrate all of them to bounded verified acquisition.
  • Set generous documented constants at least ten times larger than credible ordinary output. Do not embed numeric byte limits in call sites or hide them in a generic helper.
  • Every over-limit error must state the responsible adapter or producer contract, the configured limit, and the file/result category. Never return partial parsed output.
  • Document that hard disk exhaustion is prevented operationally with filesystem, service, container, or volume quotas; do not add polling of output-root size.

Tests and exit criteria: Add exact-limit and limit-plus-one tests, growth/race cases, non-regular and symlink cases, malformed/truncated results, and clear error assertions for each adapter family. TST-007 is primarily owned here; later adapter stages add their specialized liveness cases.

Stage 8 — Terminate owned subprocess trees

Read first: audit-findings.md lines 29012931 (RSK-011); adapter process and platform-support docs.

Depends on: Stage 7.

Goal: Ensure cancellation, timeout, and enforced output limits stop and reap the complete subprocess tree Narratio owns.

Work:

  • Introduce a narrow platform-specific process-runner capability that starts an owned process in a Unix process group or Windows job object and can terminate and reap the tree.
  • Migrate Notarius, Audita, Scriptorium, WhisperX, and other owned external command execution to it without broadening environment inheritance.
  • Define graceful-then-forceful cancellation behavior, bounded waits, error composition, and already-exited handling. Ensure descendants cannot retain pipes indefinitely after the leader exits.

Tests and exit criteria: Use helper processes that spawn descendants and hold stdout/stderr open. Prove cancellation/timeout kills descendants and reaps the leader. Add native Unix/Windows tests where runners exist and cross-build all platform code; record no unsupported fallback as safe.

Stage 9 — Redact and cap subprocess diagnostics

Read first: audit-findings.md lines 29322962 (RSK-012) and 29633006 (RSK-013); logging, adapter, and operations docs.

Depends on: Stages 68.

Goal: Make persisted stdout/stderr and diagnostic tails bounded and unable to disclose known credentials.

Work:

  • Define an explicit set of sensitive environment-variable names and pass their values only to a streaming redaction layer; never persist the values or include them in error text.
  • Minimize inherited environments for external processes while preserving the documented variables needed for execution.
  • Stream stdout and stderr through redaction into capped diagnostic files using named, centrally discoverable limits in the owning process/adapter package. Redaction must work when a secret spans write chunks. Diagnostic tails must be derived from already-redacted content.
  • If either capture limit is reached, terminate and reap the process tree through Stage 8 and return an unmistakable error naming stdout/stderr, the responsible adapter, and the byte limit. Do not report successful partial execution.

Tests and exit criteria: Cover exact cap, cap-plus-one, split-token secrets, overlapping secret values, empty values, tails, cancellation, descendant cleanup, and on-disk inspection. Assert that neither raw nor transformed error paths expose test credentials.

Stage 10 — Confine publish archive reads

Read first: audit-findings.md lines 18911924 (COR-005); publish, artifact, manifest, and storage docs.

Depends on: Stages 2, 4, and 7.

Goal: Ensure publish uploads exactly the verified regular files inside the selected run root, despite symlink or replacement races.

Work:

  • Enumerate publish candidates through the artifact/run manifest rather than an untrusted tree walk.
  • Open each candidate root-relatively without following ancestors or the leaf; verify regular-file type and any declared identity immediately on the opened object, then stream from that verified handle.
  • Reject undeclared files, directories, devices, sockets, symlinks, and identity changes. Preserve deterministic ordering and actionable source-relative errors.
  • Keep object-store destination policy out of the filesystem primitive.

Tests and exit criteria: Add regular, leaf-symlink, ancestor-swap, post-enumeration replacement, non-regular, and undeclared-file cases. A hostile workspace must never cause bytes outside the run root to be uploaded.

Stage 11 — Make manifest and run identity singular

Read first: audit-findings.md lines 17311769 (COR-001) and 36703690 (TST-006); configuration, environment, run-state, and manifest docs.

Depends on: Stages 23.

Goal: Use one validated campaign/session/run identity per invocation and recompute all run-scoped manifest fields together.

Work:

  • At invocation start, resolve and validate one effective configuration and bind the runtime environment to it; remove later reads from stale or unvalidated configuration instances.
  • On every run, recompute campaign, session, run ID, source references, and other run-scoped manifest identity as a coherent set. Do not retain fields from the prior invocation merely because a manifest object is reused.
  • Reject disagreement among configured, CLI-selected, persisted, and manifest campaign/session identity with an actionable error before side effects.
  • Preserve intentionally session-scoped history separately from the current run-scoped projection.

Tests and exit criteria: Add consecutive-run regression tests using one process/environment, explicit mismatch tests for every authority pair, and TST-006 tests proving downstream operations receive the resolved configuration instance.

Stage 12 — Centralize handled terminal-failure persistence

Read first: audit-findings.md lines 25532589 (RSK-001), 33883406 (TST-002), 38863906 (SIM-001), and 39803997 (COM-001); run/session lifecycle docs.

Depends on: Stages 3 and 11.

Goal: Ensure every handled command failure terminalizes the authoritative session record and best-effort diagnostic run record with correct compounded errors.

Work:

  • Introduce the narrow terminal-failure operation described by SIM-001: persist the session's terminal failure first, then update the run diagnostic record. Keep the running transition separate and run-first.
  • Route every handled pipeline/stage error and cancellation path through it. Return an error that preserves the operation failure plus any session/run store disagreement; do not mask the primary failure or report success.
  • Add the short rationale comment required by COM-001 explaining the deliberately asymmetric two-ledger ordering.
  • Keep abrupt process death as the accepted residual risk. Do not add heartbeats or pretend a stale running record proves liveness.

Tests and exit criteria: Inject session-store and run-store failures at each step, including cancellation and already-terminal records. Assert call order, persisted truth, and composed errors. Remove duplicate terminalization sequences.

Stage 13 — Introduce the immutable remote-commit model and legacy boundary

Read first: audit-findings.md lines 18531890 (COR-004), 19922037 (COR-008), and 32563279 (ARC-003); remote manifest, storage, restore, and compatibility docs.

Depends on: Stages 23 and 11.

Goal: Define one versioned, immutable snapshot contract selected by a small current pointer, without mixing local metadata or temporary compatibility into the new model.

Work:

  • Add a typed, versioned run-scoped remote commit manifest whose identity and declared object set are immutable. Include campaign/session/run identity, enough version/checksum/generation metadata to bind readers to the selected snapshot, and an extensible typed source-to-destination artifact mapping.
  • Define the final current pointer as the sole mutable selector of a commit. The pointer must be small, parseable, and able to select exactly one immutable commit; it is not a second mutable manifest.
  • Separate local-only metadata from remote commit state. Remove or split current_pointer_written as required by ARC-003; derive remote commitment from the selected pointer/commit relationship rather than an independently mutable boolean.
  • Isolate coherent legacy-pair reading in a clearly named compatibility file or package boundary with dedicated tests and an in-code comment stating that it is temporary and removable after migration. Reject torn or ambiguous legacy pairs. Do not add legacy writing.

Tests and exit criteria: Unit-test model validation, versions, unknown fields per repository policy, pointer/commit mismatch, checksum/generation mismatch, legacy coherent/torn cases, and the removable boundary. This stage may introduce the model and readers without switching publish; Stage 14 switches the writer.

Stage 14 — Publish through immutable commits and canonical mappings

Read first: audit-findings.md lines 18531890 (COR-004), 20682103 (COR-011), 34283447 (DUP-002), and 36183642 (TST-004); publish, artifact, remote-manifest, and storage docs.

Depends on: Stages 10 and 13.

Goal: Make new publication an immutable-object transaction whose last remote mutation selects the complete snapshot.

Work:

  • Make the artifact package's canonical run-manifest path the only source used by publish; remove the duplicate path construction identified by DUP-002.
  • Before upload, build and validate the complete deterministic source-to-remote destination mapping using exact artifact/source identity, not basename heuristics. Reject collisions and ambiguity.
  • Upload/verify every immutable data object and the immutable commit manifest first. Update the current pointer exactly once, last. Never upload or mutate current-state data after pointer selection.
  • New publish writes only the new protocol. Preserve the isolated legacy reader from Stage 13 solely for existing remote state.
  • Make publish tests stateful enough to observe remote ordering, generations, object contents, failure injection, and retry behavior.

Tests and exit criteria: Prove failures before the pointer leave the old snapshot selected; successful publication selects only a complete new snapshot; same-basename sources remain distinct; retries are deterministic; and no post-pointer upload occurs. This is the primary TST-004 stage; Stages 1516 extend the same stateful fake.

Stage 15 — Make remote locks generation-safe and harden pagination

Read first: audit-findings.md lines 26942731 (RSK-005), 30073032 (RSK-014), 36183642 (TST-004), and 36913718 (TST-007); storage adapter and publish lock docs.

Depends on: Stage 14.

Goal: Make remote ownership effective immediately and replacement conditional, and prevent non-progressing object-store pagination loops.

Work:

  • Extend the object-store contract with the minimum conditional write/replace metadata needed for generation/ETag-safe lock acquisition, renewal, and release. Implement it in the S3 adapter and stateful fake without reducing provider preconditions to a local check-then-write.
  • Commit remote lock ownership immediately, retry conditional conflicts with bounded/cancellable policy, and never release another owner's generation.
  • Revalidate lock ownership at the protected publish commit point immediately before the current pointer changes. Loss of ownership must leave the old snapshot selected.
  • In paginated listing, reject an empty or repeated continuation token when the provider claims more results. Include adapter/bucket/prefix context without secrets.

Tests and exit criteria: Add concurrent-owner, stale-generation, renewal/release conflict, lock-loss-at-commit, cancellation, empty token, repeated token, and normal multi-page tests. Run relevant tests under -race.

Stage 16 — Persist retryable post-commit cleanup state

Read first: audit-findings.md lines 19251991 (COR-006 and COR-007); publish, remote manifest, cleanup, and run/session lifecycle docs.

Depends on: Stages 5 and 1215.

Goal: Make deletion after publication a durable, idempotent obligation that does not disappear when execution state is reset or an attempt fails.

Work:

  • Persist a typed cleanup obligation independently of the current executed-step list before destructive cleanup can begin. Bind it to the committed run and exact declared objects/paths.
  • On retry/resume, detect and continue outstanding cleanup even when the current stage list is empty or rebuilt. Keep cleanup idempotent and confined through Stage 5.
  • Persist completion evidence only after deletion has succeeded or an already-absent target has been safely verified. A failed evidence write must leave a retryable obligation rather than falsely completed state.
  • Define command outcomes when publication committed but cleanup failed: the committed snapshot remains authoritative, while the invocation clearly reports incomplete cleanup and can retry it.

Tests and exit criteria: In the stateful publish fake, inject failures before, during, and after deletion/evidence persistence; restart with empty execution state; retry multiple times; and prove no unrelated target is deleted.

Stage 17 — Bind restore/status to a committed snapshot and reject conflicts

Read first: audit-findings.md lines 19922067 (COR-008 and COR-009) and 36433669 (TST-005); restore, status, remote manifest, and workspace docs.

Depends on: Stages 4, 1316.

Goal: Restore and status must observe one pointer-selected immutable snapshot, and --force must never overwrite a directory conflict.

Work:

  • Resolve the current pointer once, load and validate the selected immutable commit, and use only its declared object set and generation/checksum metadata throughout one restore/status operation.
  • Do not reconstruct current truth from mutable prefixes or a mixture of runs. Route coherent legacy state through only the isolated Stage 13 compatibility reader.
  • Define conflict types before mutation. --force may replace eligible regular managed files but must reject directory/non-regular conflicts, leaving no completed local manifest and no false success.
  • Make the restore fake stateful and able to mutate remote objects/pointers between calls so snapshot-binding failures are observable.

Tests and exit criteria: Cover pointer changes during restore, undeclared and missing objects, version/generation mismatch, coherent legacy restore, file and directory conflicts with/without force, partial download, and manifest absence on failure. This is the primary TST-005 stage; Stages 1820 extend it.

Stage 18 — Serialize restore transitions and make restored paths portable

Read first: audit-findings.md lines 27322769 (RSK-006) and 28032838 (RSK-008), plus TST-005 at lines 36433669; restore, local lock, manifest, and runner reuse docs.

Depends on: Stages 5 and 17.

Goal: Prevent runner/restore races and ensure restored manifests cannot retain unsafe producer-machine absolute paths.

Work:

  • Acquire the session/workspace lock before executable-state classification and revalidate state under the lock. All runner reuse checks must participate in the same transition protocol.
  • Persist a durable incomplete-restore marker before the first local mutation. Keep it blocking runner reuse after a partial forced restore; clear it only after all verified installs and the completed local manifest are durable.
  • Rebase restored artifact paths from portable logical references to the selected local root. Reject absolute paths outside that root and unsafe relative paths. Retain an original absolute path, if useful, only as bounded non-authoritative provenance.
  • Do not attempt rollback of arbitrary external effects; the durable incomplete marker and safe retry are the recovery contract.

Tests and exit criteria: Add concurrent runner/restore process tests, classification races, forced partial restore/restart/retry, Unix/Windows absolute path forms, traversal, safe rebasing, and provenance bounds.

Stage 19 — Bind audio cache reuse to remote object identity

Read first: audit-findings.md lines 27702802 (RSK-007) and TST-005 at lines 36433669; previous-audio, object-store, and restore docs.

Depends on: Stages 4, 15, and 1718.

Goal: Reuse restored/downloaded audio only when it is a verified regular file for the selected remote object version.

Work:

  • Persist a sidecar or equivalent typed cache record binding local audio to remote object key plus authoritative version/generation/checksum information.
  • Verify the local candidate root-relatively as a regular no-follow file and validate size/digest as required before reuse. Do not treat an unqualified multipart ETag as a content hash.
  • Refresh or fail clearly when identity evidence is absent, stale, ambiguous, or unsupported; --force must not mean trust an unrelated local file.
  • Install refreshed content using Stage 4's confined/durable download path.

Tests and exit criteria: Cover valid reuse, changed generation with same name, missing/stale sidecar, multipart ETag, symlink/non-regular cache entries, corrupted bytes, interrupted refresh, and force behavior.

Stage 20 — Unify previous-source readiness and eliminate duplicate transfers

Read first: audit-findings.md lines 20382103 (COR-010 and COR-011) and 30643112 (EFF-001), plus TST-005 at lines 36433669; previous-cache, restore, prepare, and dry-run docs.

Depends on: Stages 14 and 1719.

Goal: Give restore, prepare, run, and dry-run one authoritative previous-source resolver and retain verified downloads for the later restore snapshot.

Work:

  • Implement one resolver that evaluates required/optional previous inputs from the pointer-selected commit's exact source-to-destination mapping. Required absence is actionable; optional absence is a typed non-error.
  • Eliminate basename fallback from the new protocol. Keep any necessary legacy fallback only inside the isolated Stage 13 compatibility boundary and reject ambiguous matches.
  • Reuse the generation-bound, verified candidate downloaded while constructing the previous-source view when the restore snapshot needs the same object. Do not reopen an unverified path or perform a second network transfer.
  • Count actual transferred bytes once and make dry-run/readiness documentation accurately describe remote checks and potential download/cache work.

Tests and exit criteria: Prove identical behavior across prepare/restore/run, required/optional cases, same basenames, legacy ambiguity, generation changes, single-transfer accounting, cancellation, and dry-run output.

Stage 21 — Tighten configuration parsing, values, and expectations

Read first: audit-findings.md lines 21042253 (COR-012 through COR-015), 38053821 (TST-011), and 38533868 (TST-014); configuration and CLI docs.

Depends on: Stages 2 and 11.

Goal: Reject ambiguous or semantically invalid configuration before side effects and give the configuration package focused, non-duplicative tests.

Work:

  • Require exactly one YAML document, including rejection of a second empty or comment-only document if the parser exposes it as another document.
  • Validate all duration values that control retries, timeouts, polling, or leases as positive and within any existing semantic bounds.
  • Select storage backends only through explicit supported configuration; remove inference from incidental S3-shaped fields and reject incomplete/unknown modes.
  • Make the previous-session flag a strict expectation: when requested, absence or mismatch is an actionable error, not a silent downgrade.
  • Centralize test environment cleanup and reorganize configuration tests into loader, defaulting, and validation ownership as required by TST-011/TST-014.

Tests and exit criteria: Cover multi-document YAML, zero/negative/overflow-like durations, backend combinations, strict previous expectations, environment isolation, defaults, unknown fields, and stable actionable errors.

Status: Completed.

Stage 22 — Make product configuration truthful and own remote temp files

Read first: audit-findings.md lines 25212552 (COR-024), 31573197 (ARC-004), and 28392869 (RSK-009); configuration schema, notifications, Scriptorium, remote config, and integration docs.

Depends on: Stages 4, 7, and 21.

Goal: Expose only implemented configuration behavior and keep remote-config temporary paths within a command-scoped lifecycle.

Work:

  • Remove inert Scriptorium artifact and path fields from schema, Go types, examples, validation, and docs. Reject obsolete fields under the repository's strict configuration policy with clear migration guidance where feasible.
  • Support notifications only as an explicit no-op mode. Reject any configured backend, recipient, or provider-shaped value until a provider exists; align CLI and docs.
  • Replace ad hoc remote-config temp paths with one owned command-scoped object or lifecycle. Clean it on success, validation failure, adapter failure, cancellation, and early returns using confined deletion.
  • Never serialize ephemeral local temp paths into manifests, run records, or diagnostic output that purports to be reusable state.

Tests and exit criteria: Cover obsolete/unknown product fields, explicit no-op, backend/recipient rejection, every remote-config exit path, cancellation, stale cleanup, and manifest inspection for ephemeral paths.

Status: Completed.

Stage 23 — Stream WhisperX uploads and make the adapter race-safe

Read first: audit-findings.md lines 22542281 (COR-016), 31133156 (EFF-002), 33633387 (TST-001), and 36913718 (TST-007); WhisperX integration docs.

Depends on: Stages 79 and 21.

Goal: Make WhisperX transport bounded, retry-correct, cancellable, and safe under the race detector.

Work:

  • Accept only absolute http or https WhisperX endpoints; reject file and other schemes before execution.
  • Stream multipart bodies through a pipe per attempt instead of buffering the complete audio request. Each retry gets a fresh reader/writer and deterministic metadata; producer failures and cancellation must reach the HTTP caller without goroutine leaks.
  • Retain bounded response/result acquisition from Stage 7 and make request/response limits and errors adapter-specific.
  • Synchronize the Whisper fake's request collection and expose snapshots through a safe accessor. Migrate tests away from direct shared-slice reads.
  • Add evidence that memory behavior is no longer proportional to complete upload size, using deterministic backpressure/streaming tests rather than fragile heap thresholds alone.

Tests and exit criteria: Cover schemes, retries, mid-stream read failure, server early close, cancellation, blocked producer, exact payloads, bounded responses, and goroutine completion. Run Whisper packages and their callers under go test -race; this stage unblocks the full race suite in Stage 31.

Status: Completed.

Stage 24 — Correct prepare/transcribe transition semantics

Read first: audit-findings.md lines 22822379 (COR-017 through COR-019) and 37193743 (TST-008); prepare, transcribe, previous-cache, artifacts, and stage lifecycle docs.

Depends on: Stages 12 and 2023.

Goal: Make prepare/transcribe state transitions exact, cancellation-safe, and independent of accidental basename behavior.

Work:

  • Clear the managed previous-session tree when the effective requirement set is empty as well as when it is populated. Publish only the current invocation's verified result set.
  • Track transcribe work as planned, dispatched, and completed. Cancellation or a missing completion may not be reported as success and may not materialize a completed artifact/manifest entry.
  • Detect duplicate cleaned-audio source identities before dispatch. Allow distinct source paths with the same basename and preserve exact identity through outputs.
  • Keep state mutations inside the established lifecycle/terminalization contract.

Tests and exit criteria: Add no-requirements cleanup, stale prior tree, cancellation at each phase, partial completion, duplicate source, same-basename distinct source, retry, and no-false-manifest tests. This is the primary TST-008 stage; Stage 25 adds output/resolver boundary cases.

Status: Completed.

Stage 25 — Enforce output-path authority and shared singleton resolution

Read first: audit-findings.md lines 33063330 (ARC-006), 35153537 (DUP-006), and 37193743 (TST-008); stage request/result, artifact resolver, and transcription docs.

Depends on: Stage 24.

Goal: Make the stage-requested destination authoritative and resolve singleton transcripts through one artifact-owned contract.

Work:

  • Require an adapter result path to be empty (meaning the requested path) or exactly equal to the normalized stage-requested path. Reject any other path before materialization.
  • Put the equality/empty contract in one shared stage/adapter boundary helper and add conformance tests for all built-in adapters.
  • Route singleton transcript resolution through the artifact package's canonical resolver. Keep plural raw-source discovery separate; do not collapse different cardinality semantics into one ambiguous helper.
  • Remove duplicate singleton-selection code and stale error wording.

Tests and exit criteria: Cover empty/exact/different/relative-equivalent paths, no output, multiple transcripts, unsafe paths, adapter conformance, and plural raw sources. No adapter-selected alternate path may become authoritative.

Status: Completed.

Stage 26 — Centralize typed extraction-bundle evidence

Read first: audit-findings.md lines 35383568 (DUP-007) and 39073932 (SIM-002, merged into DUP-007), plus extraction artifact and resume/catalog docs.

Depends on: Stages 2, 4, and 7.

Goal: Evaluate extraction-bundle evidence once while preserving the different policy decisions made by resume and catalog consumers.

Work:

  • Add an artifact-owned typed proof/result that classifies extraction evidence as valid, absent, obsolete, or unsafe and carries only verified metadata/paths.
  • Centralize structural and filesystem proof: expected files, manifest/bundle coherence, safe root-relative regular files, versions, and existing identity evidence. Do not encode resume/catalog policy into the proof.
  • Make resume and catalog map the same proof to their own documented outcomes. Remove duplicate validators and the merged SIM-002 opportunity for a broad generic resolver.

Tests and exit criteria: Use one shared evidence fixture matrix for both consumers and assert their intentionally different outcomes for every typed state. Include unsafe root, symlink, incomplete, obsolete-version, and valid cases.

Status: Completed.

Stage 27 — Bind extraction reuse to direct transcript identity

Read first: audit-findings.md lines 23802423 (COR-020) and 37443774 (TST-009); extraction fingerprint, artifact evidence, direct-input, and resume docs.

Depends on: Stages 2526.

Goal: Never reuse extraction output after the direct transcript input changes while keeping the external producer's transitive dependencies explicit.

Work:

  • Extend the extraction fingerprint with a deterministic checksum of the exact trimmed direct transcript bytes and stable source/producer identity needed by the contract.
  • Recompute that direct-input identity on every resume decision and feed it into the typed Stage 26 evidence proof. A mismatch is obsolete evidence, not success.
  • Keep Notarius/external transitive dependency freshness under the existing explicit --force contract unless stable producer evidence is available; do not invent a false dependency graph.
  • Ensure extraction failure or obsolescence does not leave newly valid-looking catalog/manifest entries.

Tests and exit criteria: Add unchanged/changed/trimmed transcript cases, producer/source changes, resume/restart, force, unsafe root, over-limit lane result, and failure-before-publication tests. This is the primary TST-009 stage.

Status: Completed.

Stage 28 — Establish one effective artifact set and catalog bootstrap

Read first: audit-findings.md lines 24562491 (COR-022), 33313362 (ARC-007), 35693595 (DUP-008), and 37753804 (TST-010); analyze, extraction catalog, artifact selection, configuration, and publish-filter docs.

Depends on: Stages 21 and 2627.

Goal: Resolve artifact selection once into a typed effective set used by every analyze precondition and input-composition path.

Work:

  • Define one typed effective artifact set. Explicit CLI/stage selection overrides configured defaults even when the selected artifact is disabled by default.
  • Use that exact set for validation, prerequisite planning, extraction catalog loading, prompt/input composition, and analyze execution. Keep publication filtering a separate typed decision.
  • Centralize only deterministic catalog definitions/bootstrap ordering in the artifact owner. Keep caller-specific missing/optional/error policy outside the shared bootstrap.
  • Remove duplicate catalog lists and boolean mixtures that can diverge.

Tests and exit criteria: Table-test configured defaults versus explicit selection, disabled overrides, empty selections, ordering, catalog versions, publication filters, and the exact composed analyze inputs. This is the primary TST-010 stage; Stage 29 adds resolution/error cases.

Status: Completed.

Stage 29 — Make analyze resolution typed, optional, actionable, and deterministic

Read first: audit-findings.md lines 24242455 (COR-021), 24922520 (COR-023), 30333063 (RSK-015), 39333961 (SIM-003), and 37753804 (TST-010); analyze, stage command, artifact dependency, and troubleshooting docs.

Depends on: Stage 28.

Goal: Give analyze one typed resolution/execution plan with correct optional semantics, deterministic diagnostics, and executable remediation guidance.

Work:

  • Represent built-in artifact resolution as typed present/absent/error results. Optional absence is omitted; required absence returns a stable actionable error.
  • Build one analyze execution context/effective plan from the Stage 28 artifact set and typed resolutions. Do not add a generic resolver unrelated to analyze.
  • Sort aggregated missing/dependency diagnostics by stable artifact/source keys. Never expose map iteration order.
  • Correct remediation text to the real command form: narratio run-stage prepare <session_id> --force, with context-specific alternatives only when they are actually valid.

Tests and exit criteria: Cover optional missing built-ins, required missing, mixed custom/built-in artifacts, repeated randomized map insertion, exact guidance, context construction, and final prompt inputs. Run deterministic tests repeatedly.

Status: Completed.

Stage 30 — Remove misleading contracts and align Audita ownership

Read first: audit-findings.md lines 31983228 (ARC-001), 32803305 (ARC-005), 39623979 (SIM-004), 40214037 (COM-003), and 40854101 (COM-006); adapter, stage API, Audita, extraction source, and developer docs.

Depends on: Stages 2329.

Goal: Remove dead or misleading surfaces after the behavioral contracts have stabilized and place static Audita policy in its constructor.

Work:

  • Remove the unused IODecl/Declares surface and update tests/docs rather than preserving a compatibility shim for an internal unused contract.
  • Move static Audita settings to constructor/configuration ownership. Per-run requests carry only run-specific paths, modules, and execution data. Update the fake and all call sites.
  • Delete dead helpers made obsolete by the remediation. Do not delete a narrow helper merely because it has one caller when it still owns a real invariant.
  • Remove the stale Audita TODO and add a rationale comment only if ownership is not self-evident. Add the extraction-source doc comment required by COM-006.

Tests and exit criteria: Compile all adapters and fakes, run their focused tests, search for removed symbols/TODOs/stale request fields, and confirm public package documentation matches the remaining contract.

Stage 31 — Enforce CI validation and streamline the assembled test suite

Read first: audit-findings.md lines 38223836 (TST-012) and 38693885 (TST-015); testing policy, development workflow, Woodpecker configuration, and release docs.

Depends on: All production stages, especially Stage 23.

Goal: Make ordinary automated validation unavoidable and consolidate broad stage matrices only after the focused regression protections exist.

Work:

  • Make every change and release path run ordinary tests, go vet, go build, and documentation/example validation. A release must consume a revision that passed the required validation, not merely run alongside an optional test pipeline.
  • Run the full go test -race ./... suite on every change now that TST-001 is fixed. Add scheduled shuffled/repeated execution for order dependence.
  • Preserve focused owner tests from prior stages, then consolidate redundant broad stage matrices as directed by TST-015. Do not reduce behavior coverage or hide platform cases in an opaque mega-test.
  • Add native macOS/Windows jobs only when authoritative runner labels and successful native runners are available. Until then, retain explicit cross-build checks and document the native evidence gap without inventing labels or marking cross-compilation as equivalent.

Tests and exit criteria: Validate Woodpecker syntax and dependency graph, exercise the local equivalents of every job, prove failure propagation to release eligibility, and inspect the final suite for duplicated matrices and lost cases.

Stage 32 — Reconcile lifecycle/analyze documentation and close the remediation

Read first: audit-findings.md lines 32293255 (ARC-002, merged into COM-002), 39984020 (COM-002), 40384055 (COM-004, merged into COM-002), and 40564084 (COM-005); all current-behavior policy and package documents modified by the preceding stages.

Depends on: Stages 131.

Goal: Make the repository's normative documentation describe the implemented architecture exactly and mechanically verify that every actionable audit finding has a completed primary stage.

Work:

  • Document the distinction between successful no-output, optional absence, and a stage's explicit self-skip. Keep the production lifecycle semantics implemented in earlier stages; do not add a new lifecycle abstraction for COM-002.
  • Reconcile extraction catalog schema, effective selection/override semantics, analyze consumption, optional absence, force/freshness rules, remote commit and compatibility behavior, restore recovery, collaborative modes, secret handling, output limits, and operations quota guidance.
  • Review examples, troubleshooting commands, package docs, diagrams, and comments against the current code. Remove superseded statements rather than layering new caveats over them.
  • Verify the traceability inventory below. Every confirmed actionable finding must have one completed primary stage; merged/rejected findings must retain their recorded disposition; accepted risks must not have been silently expanded.

Tests and exit criteria: Run the full Stage 31 validation locally, run all documentation/example checks, verify clean generated/example state, inspect the final diff and repository status, and mark this stage complete only when no required remediation remains.

Finding traceability inventory

This inventory assigns exactly one primary implementation stage to each confirmed actionable finding. A stage may read or test a finding owned by another stage when the behavior crosses a boundary, but it must not create a second competing fix.

Finding(s) Primary stage
COR-001 11
COR-002 2
COR-003 4
COR-004 14
COR-005 10
COR-006, COR-007 16
COR-008, COR-009 17
COR-010 20
COR-011 14
COR-012, COR-013, COR-014, COR-015 21
COR-016 23
COR-017, COR-018, COR-019 24
COR-020 27
COR-021, COR-023 29
COR-022 28
COR-024 22
RSK-001 12
RSK-002 3
RSK-003 5
RSK-004 1
RSK-005 15
RSK-006, RSK-008 18
RSK-007 19
RSK-009 22
RSK-010 6
RSK-011 8
RSK-012 9
RSK-013 7
RSK-014 15
RSK-015 29
EFF-001 20
EFF-002 23
ARC-001 30
ARC-003 13
ARC-004 22
ARC-005 30
ARC-006 25
ARC-007 28
TST-001 23
TST-002 12
TST-003 4
TST-004 14
TST-005 17
TST-006 11
TST-007 7
TST-008 24
TST-009 27
TST-010 28
TST-011, TST-014 21
TST-012, TST-015 31
TST-013 2
DUP-001, DUP-005 3
DUP-002 14
DUP-003 4
DUP-006 25
DUP-007 26
DUP-008 28
SIM-001 12
SIM-003 29
SIM-004 30
COM-001 12
COM-002, COM-005 32
COM-003, COM-006 30

The remaining confirmed IDs have non-independent dispositions and must not receive separate implementation work: ARC-002 and COM-004 are merged into COM-002; SIM-002 is merged into DUP-007; DUP-004 is rejected. Their audit rationale remains authoritative.

Open Questions

None. The governing decisions above make this plan decision-complete. Native macOS/Windows CI activation remains contingent on external runner availability, but the required implementation, cross-build checks, and rule against inventing runner labels are already specified and do not require an architectural decision.