From 286fb9dce7f31c115f660ec14afd29434e388092 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 17 Jul 2026 23:48:14 -0500 Subject: [PATCH] Add ADR, roadmap, and staged implementation plan for filesystem surface refactoring --- ...6-separate-output-cache-and-debug-state.md | 114 +++ docs/roadmap/adr0006.md | 334 +++++++++ docs/roadmap/implementation.md | 657 ++++++++++++++++++ 3 files changed, 1105 insertions(+) create mode 100644 docs/adr/0006-separate-output-cache-and-debug-state.md create mode 100644 docs/roadmap/adr0006.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/adr/0006-separate-output-cache-and-debug-state.md b/docs/adr/0006-separate-output-cache-and-debug-state.md new file mode 100644 index 0000000..a49afef --- /dev/null +++ b/docs/adr/0006-separate-output-cache-and-debug-state.md @@ -0,0 +1,114 @@ +# ADR-0006: Separate output, cache, and debug state + +**Status:** Proposed +**Date:** 2026-07-17 + +## Context + +Notarius currently exposes a workspace as a shared parent for checkpoints, +debug artifacts, and preferred diagnostics settings. Diagnostics are a second +inspection surface with their own enablement, directory, retention, and legacy +configuration. Durable output uses a separate CLI-selected root, while the +canonical chunk-plan cache introduced by ADR-0005 correctly uses an independent +cache root. + +These concepts reflect implementation history more than operator intent. A user +must understand differences among workspace state, diagnostics, debug artifacts, +checkpoints, and chunk plans before deciding where Notarius may write. Some of +those distinctions are important internally: a redacted run summary has a +different sensitivity from a trace containing source material, prompts, and +model responses. They do not require separate public filesystem categories. + +Notarius needs a smaller state model that communicates why data exists, how it +may be treated, and whether it is reconstructible. + +## Decision + +Notarius exposes three filesystem surfaces: output, cache, and debug. The +public workspace concept and diagnostics as a separate output surface are +removed. + +### Output + +Output is the durable result of a run and the only surface intended for normal +consumption. It contains the logical files produced by the output stage, +including the maintained result, manifest, warning, and rejection contracts. +Output is not cache or inspection state. + +### Cache + +Cache contains reconstructible state used to avoid repeated work or resume an +interrupted workflow. Canonical chunk plans and invocation checkpoints are +distinct cache families with independent identities, compatibility rules, +enablement policies, locations, and cleanup lifecycles. + +ADR-0005 continues to govern canonical chunk-plan selection and reuse. Grouping +chunk plans and checkpoints under the public cache category does not permit a +checkpoint to compete with canonical plan reuse or couple their storage roots. + +Checkpointing is an invocation policy rather than a prerequisite hidden in +persistent workspace configuration. An explicit resume invocation may read +compatible checkpoints and record replacement checkpoint state for work it +executes. Runs that do not request resume perform no checkpoint I/O. + +### Debug + +Debug is an explicitly requested per-run inspection bundle intended for +developers and troubleshooting. It is off by default. When enabled, one bundle +contains both redacted run summaries and detailed stage and LLM traces. The +internal distinction between a safe summary and a sensitive trace remains, but +there is one public enablement and location model. + +Debug data is never a cache input and has no automatic retention policy. +Notarius does not create a debug directory unless debug is requested, and it +does not automatically delete a requested bundle. Credentials remain redacted +at every level, while the bundle as a whole is treated as potentially sensitive +because traces may contain source, reference, prompt, model-response, and +intermediate artifact content. + +Concise progress, warnings, and failures continue to use stdout or stderr. A +run without debug may fail without producing a filesystem inspection record. + +Exact configuration fields, CLI flags, default paths, layouts, compatibility +handling, and migration mechanics are configuration and operational contracts +rather than part of this decision. + +## Alternatives considered + +- Keep workspace, diagnostics, checkpoints, debug, and chunk-plan cache as + separate public concepts. This preserves compatibility and the current safe + default-on failure records, but retains overlapping configuration and asks + operators to reason about implementation-specific categories. +- Keep diagnostics as an always-available redacted operational surface and use + debug only for sensitive traces. This distinction is useful for a daemon or + managed service with an operational logging contract, but the current CLI can + report concise failures on stderr and provide inspection data when explicitly + requested. +- Put all non-output state beneath one physical root. This minimizes path + configuration, but couples reconstructible caches to per-run inspection data + and couples cache families whose identity, sensitivity, and cleanup policies + differ. +- Treat checkpoints as durable run state rather than cache. This emphasizes + resumability, but checkpoints are derived, compatibility-checked data that may + be deleted and recomputed. Cache more accurately describes their lifecycle. + +## Consequences + +The operator model becomes smaller: normal runs produce output and may use +cache; developers explicitly request debug. Public configuration no longer +exposes a workspace or overlapping diagnostics and debug systems. + +The implementation retains separate collaborators and serializers where their +security or lifecycle boundaries differ. Redacted summaries remain useful as +the index to a debug bundle, and chunk plans and checkpoints retain separate +stores even though both are cache. + +Existing configuration, environment variables, flags, examples, and +documentation require a deliberate compatibility transition. Default-on +diagnostic directories disappear, so failures without debug are inspectable +only through stderr and any durable output completed before the failure. + +Debug becomes easier to request and substantially more complete, but enabling +it creates sensitive files that the operator must protect and remove. Cache +cleanup is recoverable but may repeat expensive work, while deleting output is +data loss from the user's perspective. diff --git a/docs/roadmap/adr0006.md b/docs/roadmap/adr0006.md new file mode 100644 index 0000000..ef567f5 --- /dev/null +++ b/docs/roadmap/adr0006.md @@ -0,0 +1,334 @@ +# ADR-0006 Feature Roadmap + +This roadmap defines the intended end state for +[ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is +not implemented. + +## Intent + +Notarius should communicate filesystem behavior in terms of why data exists: +durable output, reconstructible cache, or explicitly requested debug material. +The public configuration and CLI should not expose internal distinctions such +as workspace state or a second diagnostics product. + +The implementation may retain focused writers, serializers, and security +boundaries. This roadmap changes their public composition and lifecycle rather +than requiring every kind of data to share one package or physical root. + +## Target Configuration + +The configuration schema advances to version 3 and uses these top-level +surfaces: + +```yaml +version: 3 + +output: + directory: ./notarius-output + +cache: + chunk_plans: + directory: "" + mode: auto + checkpoints: + directory: "" + +debug: + directory: ./notarius-debug +``` + +An empty cache directory selects that cache family's platform-appropriate +per-user default. A configured directory is the exact root; Notarius does not +append the cache-family name to it. + +The `workspace` and `diagnostics` top-level configuration surfaces do not exist +in version 3. In particular, version 3 has no persistent debug enablement, +diagnostics enablement or retention, checkpoint enablement, or shared workspace +directory. + +### Output configuration + +`output.directory` selects the durable output root. Precedence is the explicit +`--output-dir` flag, `NOTARIUS_OUTPUT_DIR`, the file value, then +`./notarius-output`. + +The pipeline's existing `pipelines..output` module binding remains separate: +it selects the output encoder, while top-level `output.directory` selects where +the CLI places that encoder's logical files. + +### Chunk-plan cache configuration + +`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh` with the +ADR-0005 semantics. Precedence is the explicit `--chunk_cache` flag, +`NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then `auto`. + +`cache.chunk_plans.directory` selects the exact canonical plan root. Precedence +is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, the file value, then +`/notarius/chunk-plans`. On Linux, the default therefore uses a +valid absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache` when that +variable is unset. A relative `XDG_CACHE_HOME` remains an error. + +The recommended system-service value remains +`/var/cache/notarius/chunk-plans`. ADR-0005 plan identities, envelopes, +permissions, atomic publication, validation, provenance, and cleanup semantics +do not change. + +### Checkpoint cache configuration + +`cache.checkpoints.directory` selects the exact checkpoint root. Precedence is +`NOTARIUS_CACHE_CHECKPOINTS_DIR`, the file value, then +`/notarius/checkpoints`. + +For a system-wide Linux deployment under a dedicated service account, the +recommended configured root is `/var/cache/notarius/checkpoints`, provisioned +with restrictive service-account ownership independently from the chunk-plan +root. + +Checkpoint directories and files use restrictive permissions because they may +contain source text and intermediate artifacts. They inherit the sensitivity of +the data captured in them. Their existing compatibility validation, pipeline +identity, source and lane payloads, and downstream dependency fingerprints +remain intact. Checkpoints do not store or select chunk results. + +There is no `cache.checkpoints.enabled` setting. Checkpoint I/O is controlled by +the invocation's `--resume` flag. + +### Debug configuration + +`debug.directory` selects the debug root but does not enable debug output. +Precedence is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then +`./notarius-debug`. + +There is no environment or file setting that enables debug. This prevents an +ambient production configuration from silently recording potentially sensitive +trace data and creating additional retained copies of application data. + +## Target CLI + +The run command retains `--output-dir`, `--chunk_cache`, and `--resume`, and +adds: + +- `--debug`: enable the complete debug bundle for this invocation; +- `--debug-dir `: override its root; valid only with `--debug`. + +`--diagnostics-dir` is removed. There is no `--workspace`, `--cache-dir`, or +separate diagnostics flag. + +`--resume` means “reuse compatible checkpoints when present and record +checkpoint state for work executed by this invocation.” On a first invocation +with no compatible state, it executes normally and creates checkpoints. A run +without `--resume` neither loads nor records checkpoints and does not resolve or +create the checkpoint root. + +The CLI reports the durable output directory on success. When debug is enabled, +it also reports the allocated debug bundle path on success and includes that +path in failure reporting once allocation has succeeded. + +Malformed values from every supplied configuration source remain errors even +when a higher-precedence source would otherwise override them. + +## Filesystem Layout + +### Output + +Durable logical files retain the current per-run layout: + +```text +//... +``` + +Output creation remains atomic per file. The maintained logical JSON output +contract, including manifests, warnings, rejections, and normalized artifacts, +does not change solely because of this state-model refactor. + +### Chunk plans + +The existing ADR-0005 layout remains: + +```text +//plan.json +``` + +Chunk-plan state remains independent of checkpoint and debug roots. + +### Checkpoints + +The current identity hierarchy moves beneath the independently selected +checkpoint root: + +```text +//-///... +``` + +The configured root replaces the former implicit +`/checkpoints` prefix. Existing manifests and payload +schemas remain compatible when pointed at their prior physical checkpoint root. + +### Debug bundle + +One explicitly requested bundle is allocated before pipeline resolution at: + +```text +// + summary/ + trace/ +``` + +`summary/` contains the redacted artifacts currently associated with +diagnostics: invocation metadata, redacted effective configuration, resolved +pipeline and reference provenance, checkpoint decisions, chunk-plan decisions, +run manifest, warnings, run report, and error text when available. + +`trace/` contains the existing deep debug material: source and stage inputs and +outputs, plans and materialized chunks, annotations, validator attempts, +prompts, model responses, retry timing, and intermediate serialized artifacts. + +Summary data never contains raw source, references, annotations, prompts, model +responses, credentials, or malformed cache bytes. Trace data may contain all +of those except credentials, which remain redacted. A debug bundle inherits the +sensitivity of the most sensitive data captured in it; it does not constitute a +separate, intrinsically higher sensitivity class. Because it aggregates and +retains an additional copy of application data, its root and files use +restrictive permissions by default. + +Trace data may preserve source or reference content that the durable output +summarized, rejected, or intentionally omitted. This can make the bundle more +sensitive than the output alone, but not more sensitive than the complete set +of application data supplied to or produced by the run. Restrictive defaults +are therefore precautionary handling for data of unknown sensitivity, not a +classification of debug material as uniquely sensitive. + +Debug persistence failures are command failures because the operator explicitly +requested the bundle. No debug directory is created without `--debug`. A +requested bundle is never automatically deleted based on success, warnings, or +failure; cleanup is explicit. + +## Runtime Ownership + +The CLI remains the composition root for physical paths and filesystem-backed +collaborators. Pipeline modules continue to return logical data and never +receive output, cache, or debug roots. + +The framework may retain separate interfaces for checkpoint recording and +loading, chunk-plan storage, redacted summary recording, and deep trace +recording. The single debug facility may compose multiple internal recorders; +unification does not require weakening redaction or passing raw trace data +through summary models. + +The public `workspace` type and terminology disappear from configuration, +invocation messages, debug metadata, and operator documentation. The standalone +diagnostics directory and retention surface also disappear. Domain contracts +that use “diagnostic” in another sense, such as a validator-provided diagnostic +artifact path, are not renamed solely by this decision. Generic confined-path +and atomic-write helpers may remain in an internal package but must not impose a +workspace abstraction on callers. + +## Security and Lifecycle + +- Output is durable user data. Notarius never automatically deletes it. +- Chunk plans and checkpoints are reconstructible cache. Exact entries or roots + may be removed, with the documented cost of recomputation. +- Debug is explicit inspection data. It is off by default, retained when + requested, and removed only by an explicit operator action. +- Checkpoints and debug bundles inherit the sensitivity of the application data + they capture. Their security concern is the additional retained copy and, for + debug bundles, aggregation of that data—not an intrinsically higher + sensitivity classification. +- Operators are responsible for choosing debug locations, access controls, and + retention appropriate to the data processed by the run. Documentation must + explain that a bundle can retain material omitted from the durable output. +- Cache and debug roots are separate trust boundaries and must not be shared + among mutually untrusted users. +- Credentials are excluded from output, cache, summaries, traces, logs, errors, + examples, and redacted configuration. +- Debug collection is allowlisted to application-owned data. It must not capture + unrelated process environment values, host secrets, or arbitrary filesystem + content merely because they are available to the process. +- Default runs may create output and use the default chunk-plan cache, but do + not create checkpoint or debug state unless their invocation flags request + those surfaces. + +## Compatibility Policy + +Version 3 is a deliberate configuration break. Version 2 files are rejected +with an actionable message directing operators to the migration documentation; +Notarius does not retain parallel legacy field parsing indefinitely. + +The version 2 migration is: + +- `workspace.chunk_cache.mode` -> `cache.chunk_plans.mode`; +- `workspace.chunk_cache.directory` -> `cache.chunk_plans.directory`; +- `/checkpoints` -> `cache.checkpoints.directory`, when + existing checkpoint reuse is desired; +- `--diagnostics-dir`, diagnostics fields, workspace diagnostics fields, and + their environment variables -> removal or the new debug controls; +- `workspace.debug.enabled` -> removal; invoke with `--debug`; +- `/debug` -> `debug.directory`, if the same parent is + desired; and +- `workspace.resume.enabled` -> removal; invoke with `--resume`. + +Environment migration is: + +- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE` -> + `NOTARIUS_CACHE_CHUNK_PLANS_MODE`; +- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` -> + `NOTARIUS_CACHE_CHUNK_PLANS_DIR`; +- `NOTARIUS_WORKSPACE_DIR` -> no direct replacement; configure the applicable + output, checkpoint-cache, and debug roots separately; +- `NOTARIUS_WORKSPACE_RESUME_ENABLED` and + `NOTARIUS_WORKSPACE_DEBUG_ENABLED` -> removal in favor of invocation flags; + and +- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`, + `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`, `NOTARIUS_WORK_DIR`, and + `NOTARIUS_DIAGNOSTICS_RETENTION` -> removal with the standalone diagnostics + surface. + +Existing chunk-plan files are reused without migration when the new directory +resolves to the same root. Existing checkpoint files are reusable when +`cache.checkpoints.directory` names the former checkpoint root and their normal +identity and schema checks pass. Existing diagnostics and debug directories are +not moved, merged, or deleted automatically. + +The durable output schema and ADR-0005 cache envelope schema are not versioned +solely because configuration advances to version 3. + +## Documentation Outcomes + +When implemented: + +- the CLI reference owns all flags and invocation semantics; +- configuration owns version 3 fields, environment variables, defaults, and + precedence; +- operations owns physical layouts, permissions, cleanup, failure inspection, + and per-user and system-service cache guidance; +- the integration contract continues to own durable logical output; +- internal pipeline and component documents explain collaborator composition + without reintroducing public workspace terminology; and +- migration documentation provides one complete version 2 to version 3 example. + +Current-behavior documentation must not describe this target before the +corresponding behavior is implemented. + +## Non-Goals + +This feature does not introduce: + +- remote output, cache, or debug storage; +- cache garbage collection, quotas, archival, history, or rollback; +- automatic debug retention or upload; +- a daemon logging, metrics, or telemetry system; +- a change to pipeline topology, artifact schemas, chunk-plan identity, or + module contracts; or +- a requirement that output, different cache families, and debug share one + physical parent directory. + +## Completion Outcomes + +The feature is complete when ordinary runs expose only durable output and +canonical chunk-plan cache behavior; checkpoint I/O occurs only under +`--resume`; one `--debug` invocation produces a complete retained summary and +trace bundle; no public workspace or diagnostics configuration remains; version +2 migration failures are actionable; existing compatible chunk plans and +checkpoints remain reusable at explicitly selected roots; and all configuration, +CLI, filesystem, security, compatibility, integration, and repository-wide +tests pass. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..eebf665 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,657 @@ +# ADR-0006 Staged Implementation Plan + +This document is the executable implementation plan for the target state in +[ADR-0006 Feature Roadmap](adr0006.md), governed by +[ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is +not implemented. + +The audience is an LLM coding agent. Implement the stages in order. Each stage +is scoped to finish with a compiling, tested repository and may be assigned as +one implementation prompt. + +## Execution Rules + +Before every stage: + +1. read [Development](../development.md), both documents under `docs/policy/`, + ADR-0006, the feature roadmap, this plan, and every stage-specific document + named below; +2. inspect `git status` and preserve user changes; +3. inspect the focused package contracts and tests before editing; and +4. confirm that all earlier stages are complete. + +During every stage: + +- implement only that stage and prerequisites discovered to be inseparable; +- keep physical path selection in the CLI composition root and keep pipeline + modules independent of output, cache, and debug roots; +- preserve checkpoint identity, chunk-plan identity, durable output schemas, + deterministic ordering, cancellation, redaction, confined paths, and atomic + writes unless this plan expressly changes them; +- treat debug persistence failures as command failures once debug is requested; +- add or update focused tests with every behavior change; +- keep unimplemented behavior under `docs/roadmap/` until the final + documentation stage; and +- do not begin the next stage after meeting the current exit criteria. + +At the end of every stage: + +1. run the focused tests listed for that stage; +2. run `go test ./...`; +3. run `go vet ./...`; +4. run `go build ./cmd/notarius`; +5. run `git diff --check`; and +6. mark the stage complete in this document only after all checks pass. + +If repository reality conflicts with this plan, stop and update the plan in the +same change before implementing a materially different design. Do not silently +invent a new public contract. + +## Fixed Decisions + +The following choices are settled for this implementation. + +### Public state model + +- The only public filesystem surfaces are output, cache, and debug. +- Output is durable user data. Cache is reconstructible. Debug is explicitly + requested inspection data and is not a cache input. +- `workspace` and standalone `diagnostics` disappear from configuration, CLI + flags, operator messages, debug metadata, package names exposed to callers, + and current-behavior documentation. +- Domain uses of the word “diagnostic,” including validator diagnostic artifact + paths, are unchanged when they do not describe the removed filesystem + surface. + +### Configuration version 3 + +- `SupportedFileConfigVersion` becomes `3`. Version 3 has these runtime types: + + ```go + type OutputConfig struct { + Directory string `json:"directory"` + } + + type CacheConfig struct { + ChunkPlans ChunkPlanCacheConfig `json:"chunk_plans"` + Checkpoints CheckpointCacheConfig `json:"checkpoints"` + } + + type ChunkPlanCacheConfig struct { + Directory string `json:"directory,omitempty"` + Mode pipeline.ChunkCacheMode `json:"mode"` + } + + type CheckpointCacheConfig struct { + Directory string `json:"directory,omitempty"` + } + + type DebugConfig struct { + Directory string `json:"directory"` + } + ``` + + `Config` contains `Output`, `Cache`, and `Debug` fields and no `Workspace` or + `Diagnostics` field. File-config types mirror the YAML shape in the feature + roadmap. +- Defaults are `./notarius-output`, chunk-plan mode `auto`, empty cache + directories meaning per-user defaults, and `./notarius-debug`. +- There is no persistent checkpoint enablement, resume enablement, debug + enablement, diagnostics enablement, or retention setting. +- Output-directory precedence is explicit `--output-dir`, + `NOTARIUS_OUTPUT_DIR`, file, then `./notarius-output`. +- Chunk-plan mode precedence is explicit `--chunk_cache`, + `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, file, then `auto`. +- Chunk-plan root precedence is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, file, then + `/notarius/chunk-plans`. +- Checkpoint root precedence is `NOTARIUS_CACHE_CHECKPOINTS_DIR`, file, then + `/notarius/checkpoints`. +- Debug-root precedence is explicit `--debug-dir`, `NOTARIUS_DEBUG_DIR`, file, + then `./notarius-debug`. A directory value never enables debug. +- A configured cache directory names the exact family root. No family suffix is + appended. Explicit relative paths retain the current CLI convention and are + interpreted relative to the process working directory. +- Empty cache directory fields in a version 3 file select their per-user + defaults. A supplied environment directory override and an explicit CLI + directory flag must be non-empty after trimming. Empty output or debug file + values and malformed modes are errors. +- Every supplied file, environment, and CLI value is validated even when a + higher-precedence value wins. Apply CLI overrides only after file and + environment parsing has succeeded. +- Parse the YAML version header before decoding the strict version 3 schema so + a version 2 file receives the intentional migration error rather than an + incidental unknown-field error. Continue to use `KnownFields(true)` for the + supported schema. + +### Cache families + +- ADR-0005 behavior and the chunk-plan envelope and path layout remain + byte-compatible. Move only configuration ownership and the default-root + helper; do not change lookup, validation, publication, provenance, or + materialization behavior. +- Chunk-plan `bypass` does not resolve or create its root. `auto` and `refresh` + resolve it lazily when the run constructs the store. +- Checkpoint I/O exists only for an invocation with `--resume`. Without that + flag, the CLI does not resolve the checkpoint root, compute a checkpoint + identity, construct a checkpoint store, load a checkpoint, or record one. +- With `--resume`, the CLI constructs both loader and recorder. Compatible + state is reused; missing or incompatible state executes normally and writes + replacement checkpoints. +- Checkpoint identities, relative paths, manifests, payloads, compatibility + rules, and dependency fingerprints remain unchanged. Existing files are + reusable when the new root points at the former + `/checkpoints` directory. +- Preserve the existing checkpoint JSON field + `workspace_schema_version` and values `notarius.workspace.v2` and + `notarius.workspace.v1`. They are frozen wire compatibility identifiers, not + public configuration or package terminology. Rename internal Go identifiers + only if doing so cannot alter serialized bytes or compatibility messages. +- The recommended Linux service roots are + `/var/cache/notarius/chunk-plans` and `/var/cache/notarius/checkpoints`, + provisioned independently for the dedicated service account. + +### Debug bundle + +- `--debug` is the only debug enablement control. `--debug-dir ` is valid + only when `--debug` is present; misuse is a CLI syntax error with exit code + `2`. +- One requested bundle is allocated before pipeline resolution at + `//`, with `summary/` and `trace/` children. +- Bundle directories are created with mode `0700` and files with mode `0600`. + Do not silently change permissions on a pre-existing operator-owned debug + parent; enforce restrictive permissions on directories and files created for + the bundle. +- `summary/` contains the current redacted diagnostics artifacts, using their + current logical filenames where applicable. `trace/` contains the current + deep debug paths. There is no retention mode or automatic deletion. +- The summary run report uses `debug_path`, not `diagnostics_path`. Invocation + and effective-config payloads use debug/state terminology and contain no + obsolete public workspace settings. +- The redacted summary excludes raw source, references, annotations, prompts, + model responses, credentials, and malformed cache bytes. Trace data may + contain application data but never credentials. +- Debug collection records only explicit application-owned payloads. It does + not dump the process environment, inspect unrelated files, or capture host + secrets. Existing sensitive-key and credential-shaped-value redaction stays + in force at every serialization boundary. +- A bundle inherits the sensitivity of its captured application data. The + additional operational risks are copying, aggregation, and retention, not an + intrinsically higher sensitivity class. +- If any requested summary or trace write fails, the command exits `1`. When a + primary operation and a debug write both fail, report both errors on stderr + without recursively attempting the same failed write. +- On success, stdout reports the durable output directory and, when enabled, + the debug bundle path. Once allocation succeeds, failure output also names + the bundle path. + +### Compatibility and documentation + +- Version 2 configuration is rejected; no legacy fields or environment aliases + remain after the implementation. The error directs the operator to the + version 2-to-3 migration section in `docs/config.md`. +- Existing chunk-plan and checkpoint state is never moved or deleted + automatically. Existing diagnostics and debug directories are likewise left + untouched. +- The durable output contract and chunk-plan envelope schema are not versioned + because of this refactor. +- `docs/config.md` owns the complete version 2-to-3 configuration and + environment migration example. `docs/operations.md` owns physical state reuse + and cleanup guidance. Other documents link to those owners rather than + duplicating volatile details. + +## Planned Internal Boundaries + +Use these package responsibilities unless an existing collision requires the +smallest obvious naming adjustment. + +- `internal/core/fileio` owns generic confined relative-path construction and + atomic JSON/byte writes with caller-selected directory and file modes. Its + errors say “file” or “artifact,” never “workspace.” It must not replace the + chunk-plan store's stronger `os.Root`-based entry protections. +- `internal/framework/checkpoint` owns checkpoint identity, manifests, payload + codecs, filesystem loader, and filesystem recorder. Constructors accept an + exact checkpoint root and an identity; they do not accept a workspace or a + general state settings object. +- `internal/framework/chunkplan` continues to own durable chunk-plan storage. + Per-user cache-root default helpers belong with configuration/path resolution, + not in a workspace package. +- `internal/core/debugbundle` owns allocation of a per-run bundle and the + redacted summary writer. It exposes exact `Path`, `SummaryRoot`, and + `TraceRoot` values and has no retention API. +- `internal/framework/debug` remains the pipeline-facing deep-trace adapter but + becomes a root-based filesystem recorder aimed at the bundle's `trace/` + directory. Pipeline debug interfaces and synchronization remain unchanged. +- `internal/cli` remains the only composition root. It resolves effective + roots, allocates requested state, constructs collaborators, writes summary + artifacts, and reports paths. + +After the cutover, delete `internal/core/workspace` and +`internal/core/diagnostics` if no remaining code has a distinct domain-neutral +reason to retain them. Do not preserve wrapper packages solely to avoid +updating imports. + +## Stage 1: Accept ADR-0006 and decouple cache storage internals + +**Status:** Not started + +### Objective + +Accept the architectural decision and remove checkpoint and chunk-plan storage +from workspace-specific constructors without changing current public +configuration or CLI behavior yet. + +### Read first + +- `internal/core/workspace/` +- `internal/framework/checkpoint/` +- `internal/framework/chunkplan/` +- `internal/framework/pipeline/checkpoint.go` +- `internal/cli/chunk_cache_test.go` +- the checkpoint sections of `docs/internal/pipeline.md` and + `docs/internal/diagnostics.md` + +### Implement + +1. Confirm ADR-0006 matches the finalized feature roadmap, then change only its + status from `Proposed` to `Accepted`. +2. Add the generic confined-write primitives in `internal/core/fileio`, with + caller-selected `0700`/`0600` support and focused path, atomicity, and + permission tests. +3. Move checkpoint identity, manifest, path, and persistence ownership into + `internal/framework/checkpoint`. Preserve every serialized field, schema + string, digest input, path component, validation reason, and payload byte. +4. Replace `NewWorkspaceRecorder` and `NewWorkspaceLoader` with root-based + filesystem constructors. During this preparatory stage, adapt the existing + CLI to pass the checkpoint root derived from its current workspace settings + so public behavior remains unchanged. +5. Move `DefaultChunkPlanRoot` out of `internal/core/workspace` into the + configuration/path-resolution boundary and retain the existing injected + `UserCacheDir` behavior and errors. +6. Remove checkpoint-specific and chunk-plan-specific files from + `internal/core/workspace`. Leave only pieces still required by the current + diagnostics/debug composition until later stages. + +### Tests + +Prove that: + +- old and refactored checkpoint identities and relative paths are identical; +- representative manifests marshal to identical JSON and v1/v2 compatibility + behavior is unchanged; +- a root-based recorder's output is reusable by the root-based loader; +- confined writes reject absolute, unclean, traversal, and backslash paths; +- atomic writes use the requested modes and do not leave temporary files after + failure; and +- chunk-plan default-root and bypass tests remain unchanged in behavior. + +Run at minimum: + +```sh +go test ./internal/core/fileio +go test ./internal/framework/checkpoint ./internal/framework/chunkplan +go test ./internal/cli +``` + +### Exit criteria + +ADR-0006 is accepted; cache persistence no longer depends on workspace-owned +identity or constructors; all current user-visible behavior remains unchanged; +and the repository-wide checks pass. + +## Stage 2: Build the unified debug-bundle collaborators + +**Status:** Not started + +### Objective + +Implement and test the target debug bundle behind internal constructors before +changing the public CLI enablement and configuration model. + +### Read first + +- `internal/core/diagnostics/` +- `internal/framework/debug/recorder.go` +- `internal/framework/pipeline/debug.go` +- `internal/framework/pipeline/runner_attempt_debug_test.go` +- `internal/framework/pipeline/runner_terminal_debug_test.go` +- `internal/core/config/redaction.go` +- [Diagnostics Internals](../internal/diagnostics.md) + +### Implement + +1. Add `internal/core/debugbundle` with a collision-safe allocator that accepts + the exact debug parent, uses the established `run-` ID + shape, creates `/summary` and `/trace`, and returns a bundle + exposing those exact paths. +2. Port the current diagnostics artifact constants and redacted writer methods + into a summary writer rooted at `summary/`. Remove retention decisions from + the new type. Use `0600` files and confined atomic writes. +3. Define summary-owned invocation and run-report payloads. Rename + diagnostics-specific methods and fields to summary/debug terminology while + retaining the useful artifact filenames listed in the feature roadmap. +4. Replace the deep trace recorder's workspace-settings constructor with a + constructor accepting the exact `trace/` root. Keep the existing + `pipeline.DebugRecorder` contract and synchronization wrapper. +5. Rename `RedactedDiagnosticsPayload` to a neutral redacted-summary method and + update its tests. Ensure the v3 effective configuration planned for the next + stage can be represented without secrets or raw application payloads. +6. Do not expose `--debug` or change current run behavior in this stage. The new + bundle is an internal collaborator exercised directly by tests until the + atomic CLI cutover. + +### Tests + +Add focused tests for: + +- collision retry and exhausted allocation; +- exact `summary/` and `trace/` layout; +- `0700` bundle directories and `0600` files; +- path traversal and symlink/path confinement behavior; +- every summary artifact and error log; +- absence of retention/deletion behavior; +- credential and sensitive-key redaction; +- no ambient environment or unrelated filesystem capture; and +- trace-recorder write failure propagation and concurrent synchronized writes. + +Run at minimum: + +```sh +go test ./internal/core/debugbundle +go test ./internal/core/config +go test ./internal/framework/debug ./internal/framework/pipeline +``` + +### Exit criteria + +The target bundle can be allocated and populated through tested summary and +trace collaborators, no public invocation behavior has changed, and all +repository-wide checks pass. + +## Stage 3: Cut configuration and CLI behavior over to the three surfaces + +**Status:** Not started + +### Objective + +Make the version 3 configuration, new flags, independent roots, resume policy, +and opt-in debug bundle the complete public runtime behavior in one atomic +cutover. + +### Read first + +- all files under `internal/core/config/` +- `internal/cli/run.go`, `internal/cli/run_test.go`, + `internal/cli/chunk_cache_test.go`, and `internal/cli/compatibility_test.go` +- the collaborators completed in Stages 1 and 2 +- `examples/*.config.yml` +- the Target Configuration, Target CLI, and Filesystem Layout sections of the + feature roadmap + +### Implement + +1. Replace the runtime and file configuration models with the fixed version 3 + types. Remove legacy diagnostics/workspace defaults, merge logic, + validation, effective-config fields, and redaction output. +2. Implement strict two-pass version handling. Reject version 2 with an + actionable migration message; reject other unsupported versions clearly; + and strictly decode version 3 with unknown fields rejected. +3. Implement the five new environment variables and remove all environment + aliases listed for removal in the feature roadmap. Preserve malformed-source + validation before precedence selection. +4. Add `--debug` and `--debug-dir`; remove `--diagnostics-dir`; update usage, + flag reordering, value validation, and exit-code behavior. Keep + `--output-dir`, `--chunk_cache`, and `--resume` with their target semantics. +5. Resolve the effective output directory after configuration loading and CLI + overrides. Use it for durable placement and summary metadata; do not fall + back to a separate CLI-only default helper. +6. When `--debug` is present, resolve the effective debug root and allocate the + bundle after configuration succeeds but before catalog lookup or pipeline + resolution. Use its run ID for the entire invocation, construct the summary + writer and trace recorder, and record invocation metadata immediately. + Without `--debug`, use the existing injected clock to generate a run ID and + construct no debug collaborator or directory. +7. Resolve the chunk-plan root only for `auto` or `refresh`, using the new + configuration family. Preserve all ADR-0005 store construction behavior. +8. When `--resume` is absent, pass no-op checkpoint collaborators without + resolving `UserCacheDir` for checkpoints. When it is present, resolve the + configured or per-user checkpoint root, create the identity, and construct + both root-based loader and recorder. +9. Replace diagnostics writes in the command with conditional summary writes. + Populate invocation metadata, redacted effective configuration, resolved + pipeline, resolved references, checkpoint events, chunk-plan decisions, run + manifest, warnings, run report, and error text as each becomes available. +10. Replace retention-aware failure handling with one failure path that prints + the primary error, attempts a single summary error record when a bundle + exists, reports any secondary debug error, and includes the allocated bundle + path. Never delete a requested bundle. +11. Report output and debug paths as fixed above. Keep warning and exit-code + behavior otherwise unchanged. +12. Convert both maintained example configs and package testdata configs to + version 3 so every executable example uses the supported schema. + +### Tests + +Update or add CLI and config tests covering: + +- exact v3 defaults, file fields, strict unknown fields, JSON/redacted shape, + and validation; +- precedence for output, chunk mode/root, checkpoint root, and debug root; +- malformed lower-precedence file or environment values despite valid CLI + overrides; +- actionable version 2 rejection before legacy unknown fields are decoded; +- rejection of every removed environment variable and configuration surface by + absence of effect, with obsolete file fields rejected as unknown; +- `--debug-dir` without `--debug`, empty explicit directory values, and removed + `--diagnostics-dir`; +- no debug allocation for ordinary runs and allocation before resolution + failures for debug runs; +- no checkpoint root resolution, loading, or recording without `--resume`; +- first-use resume writes checkpoints and later resume reuses them; +- `bypass` does not resolve a chunk-plan root; +- output directory selection and unchanged durable logical files; +- success and failure messages with and without a debug path; and +- requested summary or trace write failures producing exit code `1` without + masking a primary error. + +Run at minimum: + +```sh +go test ./internal/core/config +go test ./internal/cli +go test ./internal/modules/integration ./internal/modules/seriatim/input/transcript +``` + +### Exit criteria + +Only version 3 configuration is accepted; ordinary runs expose output and +chunk-plan cache only; resume and debug are invocation-controlled; all examples +and CLI tests use the new public contract; and the repository-wide checks pass. + +## Stage 4: Remove legacy workspace and diagnostics implementation + +**Status:** Not started + +### Objective + +Complete the internal ownership refactor and prove that no obsolete public state +model remains hidden behind compatibility wrappers. + +### Read first + +- the complete import graph reported by `rg 'core/(workspace|diagnostics)'` +- `internal/core/workspace/` +- `internal/core/diagnostics/` +- `internal/framework/checkpoint/` +- `internal/core/debugbundle/` +- `internal/framework/debug/` +- `internal/cli/run.go` + +### Implement + +1. Delete the remaining workspace settings, path, writer, and tests after moving + any still-valid generic behavior to `fileio`, `checkpoint`, or + `debugbundle`. +2. Delete the old diagnostics package, retention modes, run directory, and + tests after confirming all desired redacted artifacts are owned and tested + by `debugbundle`. +3. Remove compatibility constructors, aliases, fields, environment names, and + dead CLI helpers introduced solely by the old state model. Do not retain + deprecated parsing. +4. Rename internal variables, metadata fields, recorder types, and errors that + still use workspace or diagnostics to describe the removed surfaces. +5. Retain and document the frozen checkpoint wire schema identifiers. Retain + unrelated domain “diagnostic” terminology. +6. Confirm generic framework and pipeline packages receive only collaborator + interfaces and never physical roots. + +### Tests + +- Run `rg` checks proving there are no remaining imports of deleted packages, + public configuration fields, removed environment names, removed flags, or + operator-facing workspace/diagnostics messages. +- Allow matches only in ADR/history, version 2 migration guidance, frozen + checkpoint wire compatibility, and unrelated domain diagnostic contracts. +- Run all checkpoint, debug, CLI, and integration tests after deletion. + +### Exit criteria + +The old packages and compatibility surfaces are gone; remaining historical +terms are intentional and documented; package ownership matches the Planned +Internal Boundaries; and all checks pass. + +## Stage 5: Harden cross-surface security and compatibility + +**Status:** Not started + +### Objective + +Exercise the completed feature across failures, permissions, reuse, redaction, +and surface independence before changing current-behavior documentation. + +### Read first + +- all focused tests changed in Stages 1 through 4 +- `internal/cli/compatibility_test.go` +- `internal/framework/pipeline/runner_*debug*_test.go` +- `internal/framework/checkpoint/*_test.go` +- `internal/framework/chunkplan/store_test.go` +- the Security and Lifecycle and Compatibility Policy sections of the feature + roadmap + +### Implement + +1. Add an end-to-end state-surface matrix covering debug on/off, resume on/off, + and chunk modes `auto`, `bypass`, and `refresh`. Assert exactly which roots + are resolved and which files are created. +2. Add integration coverage proving chunk plans and checkpoints use independent + configured roots and that debug never affects cache selection or reuse. +3. Reuse a representative pre-refactor checkpoint fixture from an explicitly + selected old checkpoint root. Assert byte-compatible loading and normal + replacement behavior; do not create a migration path or rewrite untouched + entries. +4. Cover early failures before debug allocation, failures after allocation but + before pipeline execution, pipeline failures, output-write failures, summary + failures, and trace failures. Assert stable exit codes, stderr content, and + retained bundle state. +5. Inspect all summary artifacts for forbidden raw content and secrets. Inspect + trace artifacts to prove expected application data is present while API keys, + credential-shaped values, unrelated environment values, and unrelated file + contents are absent. +6. Assert created bundle and checkpoint modes on supported Unix systems and + preserve portable behavior on other supported platforms. +7. Confirm deletion is never automatic for output or debug, while manually + removing an exact cache entry only causes recomputation. +8. Confirm durable logical output and chunk-plan envelope bytes have not changed + solely because of ADR-0006. + +### Tests and validation + +Run focused packages while iterating, then run: + +```sh +go test ./... +go vet ./... +go build ./cmd/notarius +git diff --check +``` + +Run race-enabled tests for the concurrent debug and checkpoint collaborators if +supported by the environment: + +```sh +go test -race ./internal/framework/debug ./internal/framework/checkpoint ./internal/cli +``` + +### Exit criteria + +All three surfaces are independent under success and failure; compatibility and +permissions are proven; debug captures only intended data; durable contracts +are unchanged; and all repository-wide checks pass. + +## Stage 6: Update canonical documentation and close the roadmap + +**Status:** Not started + +### Objective + +Make current-behavior documentation and maintained examples describe the +implemented version 3 state model, then remove completed planning documents. + +### Read first + +- [Documentation Policy](../policy/documentation.md) +- [Architecture Policy](../policy/architecture.md) +- [CLI Reference](../cli.md) +- [Configuration](../config.md) +- [Operations](../operations.md) +- [Internal Overview](../internal/overview.md) +- [Diagnostics Internals](../internal/diagnostics.md) +- [Development](../development.md) +- `README.md`, `examples/`, and the final code and tests + +### Implement + +1. Update `docs/config.md` for version 3 fields, defaults, validation, + precedence, and the five supported state environment variables. Add one + complete version 2-to-3 before/after migration example and list every removed + setting and environment variable from the feature roadmap. +2. Update `docs/cli.md` for `--debug`, `--debug-dir`, target `--resume` + semantics, removed `--diagnostics-dir`, reporting, and failure behavior. +3. Rewrite the relevant `docs/operations.md` sections around output, chunk-plan + cache, checkpoint cache, and debug bundles. Document exact layouts, + permissions, inherited sensitivity, aggregation/copying/retention risk, + cleanup, compatible old-root reuse, per-user defaults, and the two + `/var/cache/notarius/...` Linux service recommendations. +4. Update `docs/policy/architecture.md` to describe the implemented three + surfaces and remove the obsolete five-surface model. Keep normative safety + rules concise and link operational details to Operations. +5. Replace `docs/internal/diagnostics.md` with a correctly named internal state + document, or distribute its content among focused internal documents if that + better matches the final packages. Update `docs/internal/overview.md`, + `docs/internal/pipeline.md`, `docs/development.md`, and all inbound links. +6. Update README orientation and examples only where they currently mention the + old schema or invocation. Do not duplicate the complete CLI, configuration, + or operations contracts. +7. Search the repository for stale flags, fields, environment names, paths, + package names, and workspace/diagnostics terminology. Preserve only ADR + history, explicit version 2 migration text, frozen checkpoint wire fields, + and unrelated domain diagnostics. +8. After implementation and documentation validation are complete, remove + `docs/roadmap/adr0006.md` and this implementation plan. They are completed + planning artifacts; the accepted ADR and canonical current-behavior + documents become authoritative. + +### Tests and validation + +- Validate every command, field, default, environment variable, path, and + example against code and focused tests. +- Validate all changed links and the task-specific reading map. +- Run any repository documentation or link checker if present. +- Run the complete repository-wide validation in the Execution Rules. + +### Exit criteria + +Canonical documentation describes only implemented behavior, migration and +operations each have one owner, all maintained examples are valid version 3, +no stale public state terminology remains, completed roadmap documents are +removed, and all checks pass.