notarius/docs/roadmap/implementation.md

ADR-0006 Staged Implementation Plan

This document is the executable implementation plan for the target state in ADR-0006 Feature Roadmap, governed by ADR-0006. The feature is not implemented.

The audience is an LLM coding agent. Implement the stages in order. Each stage is scoped to finish with a compiling, tested repository and may be assigned as one implementation prompt.

Execution Rules

Before every stage:

1. read Development, both documents under docs/policy/, ADR-0006, the feature roadmap, this plan, and every stage-specific document named below;
2. inspect git status and preserve user changes;
3. inspect the focused package contracts and tests before editing; and
4. confirm that all earlier stages are complete.

During every stage:

• implement only that stage and prerequisites discovered to be inseparable;
• keep physical path selection in the CLI composition root and keep pipeline modules independent of output, cache, and debug roots;
• preserve checkpoint identity, chunk-plan identity, durable output schemas, deterministic ordering, cancellation, redaction, confined paths, and atomic writes unless this plan expressly changes them;
• treat debug persistence failures as command failures once debug is requested;
• add or update focused tests with every behavior change;
• keep unimplemented behavior under docs/roadmap/ until the final documentation stage; and
• do not begin the next stage after meeting the current exit criteria.

At the end of every stage:

1. run the focused tests listed for that stage;
2. run go test ./...;
3. run go vet ./...;
4. run go build ./cmd/notarius;
5. run git diff --check; and
6. mark the stage complete in this document only after all checks pass.

If repository reality conflicts with this plan, stop and update the plan in the same change before implementing a materially different design. Do not silently invent a new public contract.

Fixed Decisions

The following choices are settled for this implementation.

Public state model

• The only public filesystem surfaces are output, cache, and debug.
• Output is durable user data. Cache is reconstructible. Debug is explicitly requested inspection data and is not a cache input.
• workspace and standalone diagnostics disappear from configuration, CLI flags, operator messages, debug metadata, package names exposed to callers, and current-behavior documentation.
• Domain uses of the word “diagnostic,” including validator diagnostic artifact paths, are unchanged when they do not describe the removed filesystem surface.

Configuration version 3

• SupportedFileConfigVersion becomes 3. Version 3 has these runtime types:

  type OutputConfig struct {
      Directory string `json:"directory"`
  }
  
  type CacheConfig struct {
      ChunkPlans ChunkPlanCacheConfig  `json:"chunk_plans"`
      Checkpoints CheckpointCacheConfig `json:"checkpoints"`
  }
  
  type ChunkPlanCacheConfig struct {
      Directory string                  `json:"directory,omitempty"`
      Mode      pipeline.ChunkCacheMode `json:"mode"`
  }
  
  type CheckpointCacheConfig struct {
      Directory string `json:"directory,omitempty"`
  }
  
  type DebugConfig struct {
      Directory string `json:"directory"`
  }
  

  Config contains Output, Cache, and Debug fields and no Workspace or Diagnostics field. File-config types mirror the YAML shape in the feature roadmap.

• Defaults are ./notarius-output, chunk-plan mode auto, empty cache directories meaning per-user defaults, and ./notarius-debug.

• There is no persistent checkpoint enablement, resume enablement, debug enablement, diagnostics enablement, or retention setting.

• Output-directory precedence is explicit --output-dir, NOTARIUS_OUTPUT_DIR, file, then ./notarius-output.

• Chunk-plan mode precedence is explicit --chunk_cache, NOTARIUS_CACHE_CHUNK_PLANS_MODE, file, then auto.

• Chunk-plan root precedence is NOTARIUS_CACHE_CHUNK_PLANS_DIR, file, then <os.UserCacheDir>/notarius/chunk-plans.

• Checkpoint root precedence is NOTARIUS_CACHE_CHECKPOINTS_DIR, file, then <os.UserCacheDir>/notarius/checkpoints.

• Debug-root precedence is explicit --debug-dir, NOTARIUS_DEBUG_DIR, file, then ./notarius-debug. A directory value never enables debug.

• A configured cache directory names the exact family root. No family suffix is appended. Explicit relative paths retain the current CLI convention and are interpreted relative to the process working directory.

• Empty cache directory fields in a version 3 file select their per-user defaults. A supplied environment directory override and an explicit CLI directory flag must be non-empty after trimming. Empty output or debug file values and malformed modes are errors.

• Every supplied file, environment, and CLI value is validated even when a higher-precedence value wins. Apply CLI overrides only after file and environment parsing has succeeded.

• Parse the YAML version header before decoding the strict version 3 schema so a version 2 file receives the intentional migration error rather than an incidental unknown-field error. Continue to use KnownFields(true) for the supported schema.

Cache families

• ADR-0005 behavior and the chunk-plan envelope and path layout remain byte-compatible. Move only configuration ownership and the default-root helper; do not change lookup, validation, publication, provenance, or materialization behavior.
• Chunk-plan bypass does not resolve or create its root. auto and refresh resolve it lazily when the run constructs the store.
• Checkpoint I/O exists only for an invocation with --resume. Without that flag, the CLI does not resolve the checkpoint root, compute a checkpoint identity, construct a checkpoint store, load a checkpoint, or record one.
• With --resume, the CLI constructs both loader and recorder. Compatible state is reused; missing or incompatible state executes normally and writes replacement checkpoints.
• Checkpoint identities, relative paths, manifests, payloads, compatibility rules, and dependency fingerprints remain unchanged. Existing files are reusable when the new root points at the former <workspace.directory>/checkpoints directory.
• Preserve the existing checkpoint JSON field workspace_schema_version and values notarius.workspace.v2 and notarius.workspace.v1. They are frozen wire compatibility identifiers, not public configuration or package terminology. Rename internal Go identifiers only if doing so cannot alter serialized bytes or compatibility messages.
• The recommended Linux service roots are /var/cache/notarius/chunk-plans and /var/cache/notarius/checkpoints, provisioned independently for the dedicated service account.

Debug bundle

• --debug is the only debug enablement control. --debug-dir <path> is valid only when --debug is present; misuse is a CLI syntax error with exit code 2.
• One requested bundle is allocated before pipeline resolution at <debug-root>/<run-id>/, with summary/ and trace/ children.
• Bundle directories are created with mode 0700 and files with mode 0600. Do not silently change permissions on a pre-existing operator-owned debug parent; enforce restrictive permissions on directories and files created for the bundle.
• summary/ contains the current redacted diagnostics artifacts, using their current logical filenames where applicable. trace/ contains the current deep debug paths. There is no retention mode or automatic deletion.
• The summary run report uses debug_path, not diagnostics_path. Invocation and effective-config payloads use debug/state terminology and contain no obsolete public workspace settings.
• The redacted summary excludes raw source, references, annotations, prompts, model responses, credentials, and malformed cache bytes. Trace data may contain application data but never credentials.
• Debug collection records only explicit application-owned payloads. It does not dump the process environment, inspect unrelated files, or capture host secrets. Existing sensitive-key and credential-shaped-value redaction stays in force at every serialization boundary.
• A bundle inherits the sensitivity of its captured application data. The additional operational risks are copying, aggregation, and retention, not an intrinsically higher sensitivity class.
• If any requested summary or trace write fails, the command exits 1. When a primary operation and a debug write both fail, report both errors on stderr without recursively attempting the same failed write.
• On success, stdout reports the durable output directory and, when enabled, the debug bundle path. Once allocation succeeds, failure output also names the bundle path.

Compatibility and documentation

• Version 2 configuration is rejected; no legacy fields or environment aliases remain after the implementation. The error directs the operator to the version 2-to-3 migration section in docs/config.md.
• Existing chunk-plan and checkpoint state is never moved or deleted automatically. Existing diagnostics and debug directories are likewise left untouched.
• The durable output contract and chunk-plan envelope schema are not versioned because of this refactor.
• docs/config.md owns the complete version 2-to-3 configuration and environment migration example. docs/operations.md owns physical state reuse and cleanup guidance. Other documents link to those owners rather than duplicating volatile details.

Planned Internal Boundaries

Use these package responsibilities unless an existing collision requires the smallest obvious naming adjustment.

• internal/core/fileio owns generic confined relative-path construction and atomic JSON/byte writes with caller-selected directory and file modes. Its errors say “file” or “artifact,” never “workspace.” It must not replace the chunk-plan store's stronger os.Root-based entry protections.
• internal/framework/checkpoint owns checkpoint identity, manifests, payload codecs, filesystem loader, and filesystem recorder. Constructors accept an exact checkpoint root and an identity; they do not accept a workspace or a general state settings object.
• internal/framework/chunkplan continues to own durable chunk-plan storage. Per-user cache-root default helpers belong with configuration/path resolution, not in a workspace package.
• internal/core/debugbundle owns allocation of a per-run bundle and the redacted summary writer. It exposes exact Path, SummaryRoot, and TraceRoot values and has no retention API.
• internal/framework/debug remains the pipeline-facing deep-trace adapter but becomes a root-based filesystem recorder aimed at the bundle's trace/ directory. Pipeline debug interfaces and synchronization remain unchanged.
• internal/cli remains the only composition root. It resolves effective roots, allocates requested state, constructs collaborators, writes summary artifacts, and reports paths.

After the cutover, delete internal/core/workspace and internal/core/diagnostics if no remaining code has a distinct domain-neutral reason to retain them. Do not preserve wrapper packages solely to avoid updating imports.

Stage 1: Accept ADR-0006 and decouple cache storage internals

Status: Complete

Objective

Accept the architectural decision and remove checkpoint and chunk-plan storage from workspace-specific constructors without changing current public configuration or CLI behavior yet.

Read first

• internal/core/workspace/
• internal/framework/checkpoint/
• internal/framework/chunkplan/
• internal/framework/pipeline/checkpoint.go
• internal/cli/chunk_cache_test.go
• the checkpoint sections of docs/internal/pipeline.md and docs/internal/diagnostics.md

Implement

1. Confirm ADR-0006 matches the finalized feature roadmap, then change only its status from Proposed to Accepted.
2. Add the generic confined-write primitives in internal/core/fileio, with caller-selected 0700/0600 support and focused path, atomicity, and permission tests.
3. Move checkpoint identity, manifest, path, and persistence ownership into internal/framework/checkpoint. Preserve every serialized field, schema string, digest input, path component, validation reason, and payload byte.
4. Replace NewWorkspaceRecorder and NewWorkspaceLoader with root-based filesystem constructors. During this preparatory stage, adapt the existing CLI to pass the checkpoint root derived from its current workspace settings so public behavior remains unchanged.
5. Move DefaultChunkPlanRoot out of internal/core/workspace into the configuration/path-resolution boundary and retain the existing injected UserCacheDir behavior and errors.
6. Remove checkpoint-specific and chunk-plan-specific files from internal/core/workspace. Leave only pieces still required by the current diagnostics/debug composition until later stages.

Tests

Prove that:

• old and refactored checkpoint identities and relative paths are identical;
• representative manifests marshal to identical JSON and v1/v2 compatibility behavior is unchanged;
• a root-based recorder's output is reusable by the root-based loader;
• confined writes reject absolute, unclean, traversal, and backslash paths;
• atomic writes use the requested modes and do not leave temporary files after failure; and
• chunk-plan default-root and bypass tests remain unchanged in behavior.

Run at minimum:

go test ./internal/core/fileio
go test ./internal/framework/checkpoint ./internal/framework/chunkplan
go test ./internal/cli

Exit criteria

ADR-0006 is accepted; cache persistence no longer depends on workspace-owned identity or constructors; all current user-visible behavior remains unchanged; and the repository-wide checks pass.

Stage 2: Build the unified debug-bundle collaborators

Status: Not started

Objective

Implement and test the target debug bundle behind internal constructors before changing the public CLI enablement and configuration model.

Read first

• internal/core/diagnostics/
• internal/framework/debug/recorder.go
• internal/framework/pipeline/debug.go
• internal/framework/pipeline/runner_attempt_debug_test.go
• internal/framework/pipeline/runner_terminal_debug_test.go
• internal/core/config/redaction.go
• Diagnostics Internals

Implement

1. Add internal/core/debugbundle with a collision-safe allocator that accepts the exact debug parent, uses the established run-<unix-nanoseconds> ID shape, creates <run-id>/summary and <run-id>/trace, and returns a bundle exposing those exact paths.
2. Port the current diagnostics artifact constants and redacted writer methods into a summary writer rooted at summary/. Remove retention decisions from the new type. Use 0600 files and confined atomic writes.
3. Define summary-owned invocation and run-report payloads. Rename diagnostics-specific methods and fields to summary/debug terminology while retaining the useful artifact filenames listed in the feature roadmap.
4. Replace the deep trace recorder's workspace-settings constructor with a constructor accepting the exact trace/ root. Keep the existing pipeline.DebugRecorder contract and synchronization wrapper.
5. Rename RedactedDiagnosticsPayload to a neutral redacted-summary method and update its tests. Ensure the v3 effective configuration planned for the next stage can be represented without secrets or raw application payloads.
6. Do not expose --debug or change current run behavior in this stage. The new bundle is an internal collaborator exercised directly by tests until the atomic CLI cutover.

Tests

Add focused tests for:

• collision retry and exhausted allocation;
• exact summary/ and trace/ layout;
• 0700 bundle directories and 0600 files;
• path traversal and symlink/path confinement behavior;
• every summary artifact and error log;
• absence of retention/deletion behavior;
• credential and sensitive-key redaction;
• no ambient environment or unrelated filesystem capture; and
• trace-recorder write failure propagation and concurrent synchronized writes.

Run at minimum:

go test ./internal/core/debugbundle
go test ./internal/core/config
go test ./internal/framework/debug ./internal/framework/pipeline

Exit criteria

The target bundle can be allocated and populated through tested summary and trace collaborators, no public invocation behavior has changed, and all repository-wide checks pass.

Stage 3: Cut configuration and CLI behavior over to the three surfaces

Status: Not started

Objective

Make the version 3 configuration, new flags, independent roots, resume policy, and opt-in debug bundle the complete public runtime behavior in one atomic cutover.

Read first

• all files under internal/core/config/
• internal/cli/run.go, internal/cli/run_test.go, internal/cli/chunk_cache_test.go, and internal/cli/compatibility_test.go
• the collaborators completed in Stages 1 and 2
• examples/*.config.yml
• the Target Configuration, Target CLI, and Filesystem Layout sections of the feature roadmap

Implement

1. Replace the runtime and file configuration models with the fixed version 3 types. Remove legacy diagnostics/workspace defaults, merge logic, validation, effective-config fields, and redaction output.
2. Implement strict two-pass version handling. Reject version 2 with an actionable migration message; reject other unsupported versions clearly; and strictly decode version 3 with unknown fields rejected.
3. Implement the five new environment variables and remove all environment aliases listed for removal in the feature roadmap. Preserve malformed-source validation before precedence selection.
4. Add --debug and --debug-dir; remove --diagnostics-dir; update usage, flag reordering, value validation, and exit-code behavior. Keep --output-dir, --chunk_cache, and --resume with their target semantics.
5. Resolve the effective output directory after configuration loading and CLI overrides. Use it for durable placement and summary metadata; do not fall back to a separate CLI-only default helper.
6. When --debug is present, resolve the effective debug root and allocate the bundle after configuration succeeds but before catalog lookup or pipeline resolution. Use its run ID for the entire invocation, construct the summary writer and trace recorder, and record invocation metadata immediately. Without --debug, use the existing injected clock to generate a run ID and construct no debug collaborator or directory.
7. Resolve the chunk-plan root only for auto or refresh, using the new configuration family. Preserve all ADR-0005 store construction behavior.
8. When --resume is absent, pass no-op checkpoint collaborators without resolving UserCacheDir for checkpoints. When it is present, resolve the configured or per-user checkpoint root, create the identity, and construct both root-based loader and recorder.
9. Replace diagnostics writes in the command with conditional summary writes. Populate invocation metadata, redacted effective configuration, resolved pipeline, resolved references, checkpoint events, chunk-plan decisions, run manifest, warnings, run report, and error text as each becomes available.
10. Replace retention-aware failure handling with one failure path that prints the primary error, attempts a single summary error record when a bundle exists, reports any secondary debug error, and includes the allocated bundle path. Never delete a requested bundle.
11. Report output and debug paths as fixed above. Keep warning and exit-code behavior otherwise unchanged.
12. Convert both maintained example configs and package testdata configs to version 3 so every executable example uses the supported schema.

Tests

Update or add CLI and config tests covering:

• exact v3 defaults, file fields, strict unknown fields, JSON/redacted shape, and validation;
• precedence for output, chunk mode/root, checkpoint root, and debug root;
• malformed lower-precedence file or environment values despite valid CLI overrides;
• actionable version 2 rejection before legacy unknown fields are decoded;
• rejection of every removed environment variable and configuration surface by absence of effect, with obsolete file fields rejected as unknown;
• --debug-dir without --debug, empty explicit directory values, and removed --diagnostics-dir;
• no debug allocation for ordinary runs and allocation before resolution failures for debug runs;
• no checkpoint root resolution, loading, or recording without --resume;
• first-use resume writes checkpoints and later resume reuses them;
• bypass does not resolve a chunk-plan root;
• output directory selection and unchanged durable logical files;
• success and failure messages with and without a debug path; and
• requested summary or trace write failures producing exit code 1 without masking a primary error.

Run at minimum:

go test ./internal/core/config
go test ./internal/cli
go test ./internal/modules/integration ./internal/modules/seriatim/input/transcript

Exit criteria

Only version 3 configuration is accepted; ordinary runs expose output and chunk-plan cache only; resume and debug are invocation-controlled; all examples and CLI tests use the new public contract; and the repository-wide checks pass.

Stage 4: Remove legacy workspace and diagnostics implementation

Status: Not started

Objective

Complete the internal ownership refactor and prove that no obsolete public state model remains hidden behind compatibility wrappers.

Read first

• the complete import graph reported by rg 'core/(workspace|diagnostics)'
• internal/core/workspace/
• internal/core/diagnostics/
• internal/framework/checkpoint/
• internal/core/debugbundle/
• internal/framework/debug/
• internal/cli/run.go

Implement

1. Delete the remaining workspace settings, path, writer, and tests after moving any still-valid generic behavior to fileio, checkpoint, or debugbundle.
2. Delete the old diagnostics package, retention modes, run directory, and tests after confirming all desired redacted artifacts are owned and tested by debugbundle.
3. Remove compatibility constructors, aliases, fields, environment names, and dead CLI helpers introduced solely by the old state model. Do not retain deprecated parsing.
4. Rename internal variables, metadata fields, recorder types, and errors that still use workspace or diagnostics to describe the removed surfaces.
5. Retain and document the frozen checkpoint wire schema identifiers. Retain unrelated domain “diagnostic” terminology.
6. Confirm generic framework and pipeline packages receive only collaborator interfaces and never physical roots.

Tests

• Run rg checks proving there are no remaining imports of deleted packages, public configuration fields, removed environment names, removed flags, or operator-facing workspace/diagnostics messages.
• Allow matches only in ADR/history, version 2 migration guidance, frozen checkpoint wire compatibility, and unrelated domain diagnostic contracts.
• Run all checkpoint, debug, CLI, and integration tests after deletion.

Exit criteria

The old packages and compatibility surfaces are gone; remaining historical terms are intentional and documented; package ownership matches the Planned Internal Boundaries; and all checks pass.

Stage 5: Harden cross-surface security and compatibility

Status: Not started

Objective

Exercise the completed feature across failures, permissions, reuse, redaction, and surface independence before changing current-behavior documentation.

Read first

• all focused tests changed in Stages 1 through 4
• internal/cli/compatibility_test.go
• internal/framework/pipeline/runner_*debug*_test.go
• internal/framework/checkpoint/*_test.go
• internal/framework/chunkplan/store_test.go
• the Security and Lifecycle and Compatibility Policy sections of the feature roadmap

Implement

1. Add an end-to-end state-surface matrix covering debug on/off, resume on/off, and chunk modes auto, bypass, and refresh. Assert exactly which roots are resolved and which files are created.
2. Add integration coverage proving chunk plans and checkpoints use independent configured roots and that debug never affects cache selection or reuse.
3. Reuse a representative pre-refactor checkpoint fixture from an explicitly selected old checkpoint root. Assert byte-compatible loading and normal replacement behavior; do not create a migration path or rewrite untouched entries.
4. Cover early failures before debug allocation, failures after allocation but before pipeline execution, pipeline failures, output-write failures, summary failures, and trace failures. Assert stable exit codes, stderr content, and retained bundle state.
5. Inspect all summary artifacts for forbidden raw content and secrets. Inspect trace artifacts to prove expected application data is present while API keys, credential-shaped values, unrelated environment values, and unrelated file contents are absent.
6. Assert created bundle and checkpoint modes on supported Unix systems and preserve portable behavior on other supported platforms.
7. Confirm deletion is never automatic for output or debug, while manually removing an exact cache entry only causes recomputation.
8. Confirm durable logical output and chunk-plan envelope bytes have not changed solely because of ADR-0006.

Tests and validation

Run focused packages while iterating, then run:

go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check

Run race-enabled tests for the concurrent debug and checkpoint collaborators if supported by the environment:

go test -race ./internal/framework/debug ./internal/framework/checkpoint ./internal/cli

Exit criteria

All three surfaces are independent under success and failure; compatibility and permissions are proven; debug captures only intended data; durable contracts are unchanged; and all repository-wide checks pass.

Stage 6: Update canonical documentation and close the roadmap

Status: Not started

Objective

Make current-behavior documentation and maintained examples describe the implemented version 3 state model, then remove completed planning documents.

Read first

• Documentation Policy
• Architecture Policy
• CLI Reference
• Configuration
• Operations
• Internal Overview
• Diagnostics Internals
• Development
• README.md, examples/, and the final code and tests

Implement

1. Update docs/config.md for version 3 fields, defaults, validation, precedence, and the five supported state environment variables. Add one complete version 2-to-3 before/after migration example and list every removed setting and environment variable from the feature roadmap.
2. Update docs/cli.md for --debug, --debug-dir, target --resume semantics, removed --diagnostics-dir, reporting, and failure behavior.
3. Rewrite the relevant docs/operations.md sections around output, chunk-plan cache, checkpoint cache, and debug bundles. Document exact layouts, permissions, inherited sensitivity, aggregation/copying/retention risk, cleanup, compatible old-root reuse, per-user defaults, and the two /var/cache/notarius/... Linux service recommendations.
4. Update docs/policy/architecture.md to describe the implemented three surfaces and remove the obsolete five-surface model. Keep normative safety rules concise and link operational details to Operations.
5. Replace docs/internal/diagnostics.md with a correctly named internal state document, or distribute its content among focused internal documents if that better matches the final packages. Update docs/internal/overview.md, docs/internal/pipeline.md, docs/development.md, and all inbound links.
6. Update README orientation and examples only where they currently mention the old schema or invocation. Do not duplicate the complete CLI, configuration, or operations contracts.
7. Search the repository for stale flags, fields, environment names, paths, package names, and workspace/diagnostics terminology. Preserve only ADR history, explicit version 2 migration text, frozen checkpoint wire fields, and unrelated domain diagnostics.
8. After implementation and documentation validation are complete, remove docs/roadmap/adr0006.md and this implementation plan. They are completed planning artifacts; the accepted ADR and canonical current-behavior documents become authoritative.

Tests and validation

• Validate every command, field, default, environment variable, path, and example against code and focused tests.
• Validate all changed links and the task-specific reading map.
• Run any repository documentation or link checker if present.
• Run the complete repository-wide validation in the Execution Rules.

Exit criteria

Canonical documentation describes only implemented behavior, migration and operations each have one owner, all maintained examples are valid version 3, no stale public state terminology remains, completed roadmap documents are removed, and all checks pass.