# ADR-0006 Staged Implementation Plan This document is the executable implementation plan for the target state in [ADR-0006 Feature Roadmap](adr0006.md), governed by [ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is not implemented. The audience is an LLM coding agent. Implement the stages in order. Each stage is scoped to finish with a compiling, tested repository and may be assigned as one implementation prompt. ## Execution Rules Before every stage: 1. read [Development](../development.md), both documents under `docs/policy/`, ADR-0006, the feature roadmap, this plan, and every stage-specific document named below; 2. inspect `git status` and preserve user changes; 3. inspect the focused package contracts and tests before editing; and 4. confirm that all earlier stages are complete. During every stage: - implement only that stage and prerequisites discovered to be inseparable; - keep physical path selection in the CLI composition root and keep pipeline modules independent of output, cache, and debug roots; - preserve checkpoint identity, chunk-plan identity, durable output schemas, deterministic ordering, cancellation, redaction, confined paths, and atomic writes unless this plan expressly changes them; - treat debug persistence failures as command failures once debug is requested; - add or update focused tests with every behavior change; - keep unimplemented behavior under `docs/roadmap/` until the final documentation stage; and - do not begin the next stage after meeting the current exit criteria. At the end of every stage: 1. run the focused tests listed for that stage; 2. run `go test ./...`; 3. run `go vet ./...`; 4. run `go build ./cmd/notarius`; 5. run `git diff --check`; and 6. mark the stage complete in this document only after all checks pass. If repository reality conflicts with this plan, stop and update the plan in the same change before implementing a materially different design. Do not silently invent a new public contract. ## Fixed Decisions The following choices are settled for this implementation. ### Public state model - The only public filesystem surfaces are output, cache, and debug. - Output is durable user data. Cache is reconstructible. Debug is explicitly requested inspection data and is not a cache input. - `workspace` and standalone `diagnostics` disappear from configuration, CLI flags, operator messages, debug metadata, package names exposed to callers, and current-behavior documentation. - Domain uses of the word “diagnostic,” including validator diagnostic artifact paths, are unchanged when they do not describe the removed filesystem surface. ### Configuration version 3 - `SupportedFileConfigVersion` becomes `3`. Version 3 has these runtime types: ```go type OutputConfig struct { Directory string `json:"directory"` } type CacheConfig struct { ChunkPlans ChunkPlanCacheConfig `json:"chunk_plans"` Checkpoints CheckpointCacheConfig `json:"checkpoints"` } type ChunkPlanCacheConfig struct { Directory string `json:"directory,omitempty"` Mode pipeline.ChunkCacheMode `json:"mode"` } type CheckpointCacheConfig struct { Directory string `json:"directory,omitempty"` } type DebugConfig struct { Directory string `json:"directory"` } ``` `Config` contains `Output`, `Cache`, and `Debug` fields and no `Workspace` or `Diagnostics` field. File-config types mirror the YAML shape in the feature roadmap. - Defaults are `./notarius-output`, chunk-plan mode `auto`, empty cache directories meaning per-user defaults, and `./notarius-debug`. - There is no persistent checkpoint enablement, resume enablement, debug enablement, diagnostics enablement, or retention setting. - Output-directory precedence is explicit `--output-dir`, `NOTARIUS_OUTPUT_DIR`, file, then `./notarius-output`. - Chunk-plan mode precedence is explicit `--chunk_cache`, `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, file, then `auto`. - Chunk-plan root precedence is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, file, then `/notarius/chunk-plans`. - Checkpoint root precedence is `NOTARIUS_CACHE_CHECKPOINTS_DIR`, file, then `/notarius/checkpoints`. - Debug-root precedence is explicit `--debug-dir`, `NOTARIUS_DEBUG_DIR`, file, then `./notarius-debug`. A directory value never enables debug. - A configured cache directory names the exact family root. No family suffix is appended. Explicit relative paths retain the current CLI convention and are interpreted relative to the process working directory. - Empty cache directory fields in a version 3 file select their per-user defaults. A supplied environment directory override and an explicit CLI directory flag must be non-empty after trimming. Empty output or debug file values and malformed modes are errors. - Every supplied file, environment, and CLI value is validated even when a higher-precedence value wins. Apply CLI overrides only after file and environment parsing has succeeded. - Parse the YAML version header before decoding the strict version 3 schema so a version 2 file receives the intentional migration error rather than an incidental unknown-field error. Continue to use `KnownFields(true)` for the supported schema. ### Cache families - ADR-0005 behavior and the chunk-plan envelope and path layout remain byte-compatible. Move only configuration ownership and the default-root helper; do not change lookup, validation, publication, provenance, or materialization behavior. - Chunk-plan `bypass` does not resolve or create its root. `auto` and `refresh` resolve it lazily when the run constructs the store. - Checkpoint I/O exists only for an invocation with `--resume`. Without that flag, the CLI does not resolve the checkpoint root, compute a checkpoint identity, construct a checkpoint store, load a checkpoint, or record one. - With `--resume`, the CLI constructs both loader and recorder. Compatible state is reused; missing or incompatible state executes normally and writes replacement checkpoints. - Checkpoint identities, relative paths, manifests, payloads, compatibility rules, and dependency fingerprints remain unchanged. Existing files are reusable when the new root points at the former `/checkpoints` directory. - Preserve the existing checkpoint JSON field `workspace_schema_version` and values `notarius.workspace.v2` and `notarius.workspace.v1`. They are frozen wire compatibility identifiers, not public configuration or package terminology. Rename internal Go identifiers only if doing so cannot alter serialized bytes or compatibility messages. - The recommended Linux service roots are `/var/cache/notarius/chunk-plans` and `/var/cache/notarius/checkpoints`, provisioned independently for the dedicated service account. ### Debug bundle - `--debug` is the only debug enablement control. `--debug-dir ` is valid only when `--debug` is present; misuse is a CLI syntax error with exit code `2`. - One requested bundle is allocated before pipeline resolution at `//`, with `summary/` and `trace/` children. - Bundle directories are created with mode `0700` and files with mode `0600`. Do not silently change permissions on a pre-existing operator-owned debug parent; enforce restrictive permissions on directories and files created for the bundle. - `summary/` contains the current redacted diagnostics artifacts, using their current logical filenames where applicable. `trace/` contains the current deep debug paths. There is no retention mode or automatic deletion. - The summary run report uses `debug_path`, not `diagnostics_path`. Invocation and effective-config payloads use debug/state terminology and contain no obsolete public workspace settings. - The redacted summary excludes raw source, references, annotations, prompts, model responses, credentials, and malformed cache bytes. Trace data may contain application data but never credentials. - Debug collection records only explicit application-owned payloads. It does not dump the process environment, inspect unrelated files, or capture host secrets. Existing sensitive-key and credential-shaped-value redaction stays in force at every serialization boundary. - A bundle inherits the sensitivity of its captured application data. The additional operational risks are copying, aggregation, and retention, not an intrinsically higher sensitivity class. - If any requested summary or trace write fails, the command exits `1`. When a primary operation and a debug write both fail, report both errors on stderr without recursively attempting the same failed write. - On success, stdout reports the durable output directory and, when enabled, the debug bundle path. Once allocation succeeds, failure output also names the bundle path. ### Compatibility and documentation - Version 2 configuration is rejected; no legacy fields or environment aliases remain after the implementation. The error directs the operator to the version 2-to-3 migration section in `docs/config.md`. - Existing chunk-plan and checkpoint state is never moved or deleted automatically. Existing diagnostics and debug directories are likewise left untouched. - The durable output contract and chunk-plan envelope schema are not versioned because of this refactor. - `docs/config.md` owns the complete version 2-to-3 configuration and environment migration example. `docs/operations.md` owns physical state reuse and cleanup guidance. Other documents link to those owners rather than duplicating volatile details. ## Planned Internal Boundaries Use these package responsibilities unless an existing collision requires the smallest obvious naming adjustment. - `internal/core/fileio` owns generic confined relative-path construction and atomic JSON/byte writes with caller-selected directory and file modes. Its errors say “file” or “artifact,” never “workspace.” It must not replace the chunk-plan store's stronger `os.Root`-based entry protections. - `internal/framework/checkpoint` owns checkpoint identity, manifests, payload codecs, filesystem loader, and filesystem recorder. Constructors accept an exact checkpoint root and an identity; they do not accept a workspace or a general state settings object. - `internal/framework/chunkplan` continues to own durable chunk-plan storage. Per-user cache-root default helpers belong with configuration/path resolution, not in a workspace package. - `internal/core/debugbundle` owns allocation of a per-run bundle and the redacted summary writer. It exposes exact `Path`, `SummaryRoot`, and `TraceRoot` values and has no retention API. - `internal/framework/debug` remains the pipeline-facing deep-trace adapter but becomes a root-based filesystem recorder aimed at the bundle's `trace/` directory. Pipeline debug interfaces and synchronization remain unchanged. - `internal/cli` remains the only composition root. It resolves effective roots, allocates requested state, constructs collaborators, writes summary artifacts, and reports paths. After the cutover, delete `internal/core/workspace` and `internal/core/diagnostics` if no remaining code has a distinct domain-neutral reason to retain them. Do not preserve wrapper packages solely to avoid updating imports. ## Stage 1: Accept ADR-0006 and decouple cache storage internals **Status:** Not started ### Objective Accept the architectural decision and remove checkpoint and chunk-plan storage from workspace-specific constructors without changing current public configuration or CLI behavior yet. ### Read first - `internal/core/workspace/` - `internal/framework/checkpoint/` - `internal/framework/chunkplan/` - `internal/framework/pipeline/checkpoint.go` - `internal/cli/chunk_cache_test.go` - the checkpoint sections of `docs/internal/pipeline.md` and `docs/internal/diagnostics.md` ### Implement 1. Confirm ADR-0006 matches the finalized feature roadmap, then change only its status from `Proposed` to `Accepted`. 2. Add the generic confined-write primitives in `internal/core/fileio`, with caller-selected `0700`/`0600` support and focused path, atomicity, and permission tests. 3. Move checkpoint identity, manifest, path, and persistence ownership into `internal/framework/checkpoint`. Preserve every serialized field, schema string, digest input, path component, validation reason, and payload byte. 4. Replace `NewWorkspaceRecorder` and `NewWorkspaceLoader` with root-based filesystem constructors. During this preparatory stage, adapt the existing CLI to pass the checkpoint root derived from its current workspace settings so public behavior remains unchanged. 5. Move `DefaultChunkPlanRoot` out of `internal/core/workspace` into the configuration/path-resolution boundary and retain the existing injected `UserCacheDir` behavior and errors. 6. Remove checkpoint-specific and chunk-plan-specific files from `internal/core/workspace`. Leave only pieces still required by the current diagnostics/debug composition until later stages. ### Tests Prove that: - old and refactored checkpoint identities and relative paths are identical; - representative manifests marshal to identical JSON and v1/v2 compatibility behavior is unchanged; - a root-based recorder's output is reusable by the root-based loader; - confined writes reject absolute, unclean, traversal, and backslash paths; - atomic writes use the requested modes and do not leave temporary files after failure; and - chunk-plan default-root and bypass tests remain unchanged in behavior. Run at minimum: ```sh go test ./internal/core/fileio go test ./internal/framework/checkpoint ./internal/framework/chunkplan go test ./internal/cli ``` ### Exit criteria ADR-0006 is accepted; cache persistence no longer depends on workspace-owned identity or constructors; all current user-visible behavior remains unchanged; and the repository-wide checks pass. ## Stage 2: Build the unified debug-bundle collaborators **Status:** Not started ### Objective Implement and test the target debug bundle behind internal constructors before changing the public CLI enablement and configuration model. ### Read first - `internal/core/diagnostics/` - `internal/framework/debug/recorder.go` - `internal/framework/pipeline/debug.go` - `internal/framework/pipeline/runner_attempt_debug_test.go` - `internal/framework/pipeline/runner_terminal_debug_test.go` - `internal/core/config/redaction.go` - [Diagnostics Internals](../internal/diagnostics.md) ### Implement 1. Add `internal/core/debugbundle` with a collision-safe allocator that accepts the exact debug parent, uses the established `run-` ID shape, creates `/summary` and `/trace`, and returns a bundle exposing those exact paths. 2. Port the current diagnostics artifact constants and redacted writer methods into a summary writer rooted at `summary/`. Remove retention decisions from the new type. Use `0600` files and confined atomic writes. 3. Define summary-owned invocation and run-report payloads. Rename diagnostics-specific methods and fields to summary/debug terminology while retaining the useful artifact filenames listed in the feature roadmap. 4. Replace the deep trace recorder's workspace-settings constructor with a constructor accepting the exact `trace/` root. Keep the existing `pipeline.DebugRecorder` contract and synchronization wrapper. 5. Rename `RedactedDiagnosticsPayload` to a neutral redacted-summary method and update its tests. Ensure the v3 effective configuration planned for the next stage can be represented without secrets or raw application payloads. 6. Do not expose `--debug` or change current run behavior in this stage. The new bundle is an internal collaborator exercised directly by tests until the atomic CLI cutover. ### Tests Add focused tests for: - collision retry and exhausted allocation; - exact `summary/` and `trace/` layout; - `0700` bundle directories and `0600` files; - path traversal and symlink/path confinement behavior; - every summary artifact and error log; - absence of retention/deletion behavior; - credential and sensitive-key redaction; - no ambient environment or unrelated filesystem capture; and - trace-recorder write failure propagation and concurrent synchronized writes. Run at minimum: ```sh go test ./internal/core/debugbundle go test ./internal/core/config go test ./internal/framework/debug ./internal/framework/pipeline ``` ### Exit criteria The target bundle can be allocated and populated through tested summary and trace collaborators, no public invocation behavior has changed, and all repository-wide checks pass. ## Stage 3: Cut configuration and CLI behavior over to the three surfaces **Status:** Not started ### Objective Make the version 3 configuration, new flags, independent roots, resume policy, and opt-in debug bundle the complete public runtime behavior in one atomic cutover. ### Read first - all files under `internal/core/config/` - `internal/cli/run.go`, `internal/cli/run_test.go`, `internal/cli/chunk_cache_test.go`, and `internal/cli/compatibility_test.go` - the collaborators completed in Stages 1 and 2 - `examples/*.config.yml` - the Target Configuration, Target CLI, and Filesystem Layout sections of the feature roadmap ### Implement 1. Replace the runtime and file configuration models with the fixed version 3 types. Remove legacy diagnostics/workspace defaults, merge logic, validation, effective-config fields, and redaction output. 2. Implement strict two-pass version handling. Reject version 2 with an actionable migration message; reject other unsupported versions clearly; and strictly decode version 3 with unknown fields rejected. 3. Implement the five new environment variables and remove all environment aliases listed for removal in the feature roadmap. Preserve malformed-source validation before precedence selection. 4. Add `--debug` and `--debug-dir`; remove `--diagnostics-dir`; update usage, flag reordering, value validation, and exit-code behavior. Keep `--output-dir`, `--chunk_cache`, and `--resume` with their target semantics. 5. Resolve the effective output directory after configuration loading and CLI overrides. Use it for durable placement and summary metadata; do not fall back to a separate CLI-only default helper. 6. When `--debug` is present, resolve the effective debug root and allocate the bundle after configuration succeeds but before catalog lookup or pipeline resolution. Use its run ID for the entire invocation, construct the summary writer and trace recorder, and record invocation metadata immediately. Without `--debug`, use the existing injected clock to generate a run ID and construct no debug collaborator or directory. 7. Resolve the chunk-plan root only for `auto` or `refresh`, using the new configuration family. Preserve all ADR-0005 store construction behavior. 8. When `--resume` is absent, pass no-op checkpoint collaborators without resolving `UserCacheDir` for checkpoints. When it is present, resolve the configured or per-user checkpoint root, create the identity, and construct both root-based loader and recorder. 9. Replace diagnostics writes in the command with conditional summary writes. Populate invocation metadata, redacted effective configuration, resolved pipeline, resolved references, checkpoint events, chunk-plan decisions, run manifest, warnings, run report, and error text as each becomes available. 10. Replace retention-aware failure handling with one failure path that prints the primary error, attempts a single summary error record when a bundle exists, reports any secondary debug error, and includes the allocated bundle path. Never delete a requested bundle. 11. Report output and debug paths as fixed above. Keep warning and exit-code behavior otherwise unchanged. 12. Convert both maintained example configs and package testdata configs to version 3 so every executable example uses the supported schema. ### Tests Update or add CLI and config tests covering: - exact v3 defaults, file fields, strict unknown fields, JSON/redacted shape, and validation; - precedence for output, chunk mode/root, checkpoint root, and debug root; - malformed lower-precedence file or environment values despite valid CLI overrides; - actionable version 2 rejection before legacy unknown fields are decoded; - rejection of every removed environment variable and configuration surface by absence of effect, with obsolete file fields rejected as unknown; - `--debug-dir` without `--debug`, empty explicit directory values, and removed `--diagnostics-dir`; - no debug allocation for ordinary runs and allocation before resolution failures for debug runs; - no checkpoint root resolution, loading, or recording without `--resume`; - first-use resume writes checkpoints and later resume reuses them; - `bypass` does not resolve a chunk-plan root; - output directory selection and unchanged durable logical files; - success and failure messages with and without a debug path; and - requested summary or trace write failures producing exit code `1` without masking a primary error. Run at minimum: ```sh go test ./internal/core/config go test ./internal/cli go test ./internal/modules/integration ./internal/modules/seriatim/input/transcript ``` ### Exit criteria Only version 3 configuration is accepted; ordinary runs expose output and chunk-plan cache only; resume and debug are invocation-controlled; all examples and CLI tests use the new public contract; and the repository-wide checks pass. ## Stage 4: Remove legacy workspace and diagnostics implementation **Status:** Not started ### Objective Complete the internal ownership refactor and prove that no obsolete public state model remains hidden behind compatibility wrappers. ### Read first - the complete import graph reported by `rg 'core/(workspace|diagnostics)'` - `internal/core/workspace/` - `internal/core/diagnostics/` - `internal/framework/checkpoint/` - `internal/core/debugbundle/` - `internal/framework/debug/` - `internal/cli/run.go` ### Implement 1. Delete the remaining workspace settings, path, writer, and tests after moving any still-valid generic behavior to `fileio`, `checkpoint`, or `debugbundle`. 2. Delete the old diagnostics package, retention modes, run directory, and tests after confirming all desired redacted artifacts are owned and tested by `debugbundle`. 3. Remove compatibility constructors, aliases, fields, environment names, and dead CLI helpers introduced solely by the old state model. Do not retain deprecated parsing. 4. Rename internal variables, metadata fields, recorder types, and errors that still use workspace or diagnostics to describe the removed surfaces. 5. Retain and document the frozen checkpoint wire schema identifiers. Retain unrelated domain “diagnostic” terminology. 6. Confirm generic framework and pipeline packages receive only collaborator interfaces and never physical roots. ### Tests - Run `rg` checks proving there are no remaining imports of deleted packages, public configuration fields, removed environment names, removed flags, or operator-facing workspace/diagnostics messages. - Allow matches only in ADR/history, version 2 migration guidance, frozen checkpoint wire compatibility, and unrelated domain diagnostic contracts. - Run all checkpoint, debug, CLI, and integration tests after deletion. ### Exit criteria The old packages and compatibility surfaces are gone; remaining historical terms are intentional and documented; package ownership matches the Planned Internal Boundaries; and all checks pass. ## Stage 5: Harden cross-surface security and compatibility **Status:** Not started ### Objective Exercise the completed feature across failures, permissions, reuse, redaction, and surface independence before changing current-behavior documentation. ### Read first - all focused tests changed in Stages 1 through 4 - `internal/cli/compatibility_test.go` - `internal/framework/pipeline/runner_*debug*_test.go` - `internal/framework/checkpoint/*_test.go` - `internal/framework/chunkplan/store_test.go` - the Security and Lifecycle and Compatibility Policy sections of the feature roadmap ### Implement 1. Add an end-to-end state-surface matrix covering debug on/off, resume on/off, and chunk modes `auto`, `bypass`, and `refresh`. Assert exactly which roots are resolved and which files are created. 2. Add integration coverage proving chunk plans and checkpoints use independent configured roots and that debug never affects cache selection or reuse. 3. Reuse a representative pre-refactor checkpoint fixture from an explicitly selected old checkpoint root. Assert byte-compatible loading and normal replacement behavior; do not create a migration path or rewrite untouched entries. 4. Cover early failures before debug allocation, failures after allocation but before pipeline execution, pipeline failures, output-write failures, summary failures, and trace failures. Assert stable exit codes, stderr content, and retained bundle state. 5. Inspect all summary artifacts for forbidden raw content and secrets. Inspect trace artifacts to prove expected application data is present while API keys, credential-shaped values, unrelated environment values, and unrelated file contents are absent. 6. Assert created bundle and checkpoint modes on supported Unix systems and preserve portable behavior on other supported platforms. 7. Confirm deletion is never automatic for output or debug, while manually removing an exact cache entry only causes recomputation. 8. Confirm durable logical output and chunk-plan envelope bytes have not changed solely because of ADR-0006. ### Tests and validation Run focused packages while iterating, then run: ```sh go test ./... go vet ./... go build ./cmd/notarius git diff --check ``` Run race-enabled tests for the concurrent debug and checkpoint collaborators if supported by the environment: ```sh go test -race ./internal/framework/debug ./internal/framework/checkpoint ./internal/cli ``` ### Exit criteria All three surfaces are independent under success and failure; compatibility and permissions are proven; debug captures only intended data; durable contracts are unchanged; and all repository-wide checks pass. ## Stage 6: Update canonical documentation and close the roadmap **Status:** Not started ### Objective Make current-behavior documentation and maintained examples describe the implemented version 3 state model, then remove completed planning documents. ### Read first - [Documentation Policy](../policy/documentation.md) - [Architecture Policy](../policy/architecture.md) - [CLI Reference](../cli.md) - [Configuration](../config.md) - [Operations](../operations.md) - [Internal Overview](../internal/overview.md) - [Diagnostics Internals](../internal/diagnostics.md) - [Development](../development.md) - `README.md`, `examples/`, and the final code and tests ### Implement 1. Update `docs/config.md` for version 3 fields, defaults, validation, precedence, and the five supported state environment variables. Add one complete version 2-to-3 before/after migration example and list every removed setting and environment variable from the feature roadmap. 2. Update `docs/cli.md` for `--debug`, `--debug-dir`, target `--resume` semantics, removed `--diagnostics-dir`, reporting, and failure behavior. 3. Rewrite the relevant `docs/operations.md` sections around output, chunk-plan cache, checkpoint cache, and debug bundles. Document exact layouts, permissions, inherited sensitivity, aggregation/copying/retention risk, cleanup, compatible old-root reuse, per-user defaults, and the two `/var/cache/notarius/...` Linux service recommendations. 4. Update `docs/policy/architecture.md` to describe the implemented three surfaces and remove the obsolete five-surface model. Keep normative safety rules concise and link operational details to Operations. 5. Replace `docs/internal/diagnostics.md` with a correctly named internal state document, or distribute its content among focused internal documents if that better matches the final packages. Update `docs/internal/overview.md`, `docs/internal/pipeline.md`, `docs/development.md`, and all inbound links. 6. Update README orientation and examples only where they currently mention the old schema or invocation. Do not duplicate the complete CLI, configuration, or operations contracts. 7. Search the repository for stale flags, fields, environment names, paths, package names, and workspace/diagnostics terminology. Preserve only ADR history, explicit version 2 migration text, frozen checkpoint wire fields, and unrelated domain diagnostics. 8. After implementation and documentation validation are complete, remove `docs/roadmap/adr0006.md` and this implementation plan. They are completed planning artifacts; the accepted ADR and canonical current-behavior documents become authoritative. ### Tests and validation - Validate every command, field, default, environment variable, path, and example against code and focused tests. - Validate all changed links and the task-specific reading map. - Run any repository documentation or link checker if present. - Run the complete repository-wide validation in the Execution Rules. ### Exit criteria Canonical documentation describes only implemented behavior, migration and operations each have one owner, all maintained examples are valid version 3, no stale public state terminology remains, completed roadmap documents are removed, and all checks pass.