eric/notarius
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Add ADR, roadmap, and staged implementation plan for filesystem surface refactoring

Browse Source
This commit is contained in:
Eric Rakestraw
2026-07-17 23:48:14 -05:00
parent 9fa9154dda
commit 286fb9dce7
3 changed files with 1105 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

114
docs/adr/0006-separate-output-cache-and-debug-state.md Normal file
View File

@@ -0,0 +1,114 @@
# ADR-0006: Separate output, cache, and debug state
**Status:** Proposed
**Date:** 2026-07-17
## Context
Notarius currently exposes a workspace as a shared parent for checkpoints,
debug artifacts, and preferred diagnostics settings. Diagnostics are a second
inspection surface with their own enablement, directory, retention, and legacy
configuration. Durable output uses a separate CLI-selected root, while the
canonical chunk-plan cache introduced by ADR-0005 correctly uses an independent
cache root.
These concepts reflect implementation history more than operator intent. A user
must understand differences among workspace state, diagnostics, debug artifacts,
checkpoints, and chunk plans before deciding where Notarius may write. Some of
those distinctions are important internally: a redacted run summary has a
different sensitivity from a trace containing source material, prompts, and
model responses. They do not require separate public filesystem categories.
Notarius needs a smaller state model that communicates why data exists, how it
may be treated, and whether it is reconstructible.
## Decision
Notarius exposes three filesystem surfaces: output, cache, and debug. The
public workspace concept and diagnostics as a separate output surface are
removed.
### Output
Output is the durable result of a run and the only surface intended for normal
consumption. It contains the logical files produced by the output stage,
including the maintained result, manifest, warning, and rejection contracts.
Output is not cache or inspection state.
### Cache
Cache contains reconstructible state used to avoid repeated work or resume an
interrupted workflow. Canonical chunk plans and invocation checkpoints are
distinct cache families with independent identities, compatibility rules,
enablement policies, locations, and cleanup lifecycles.
ADR-0005 continues to govern canonical chunk-plan selection and reuse. Grouping
chunk plans and checkpoints under the public cache category does not permit a
checkpoint to compete with canonical plan reuse or couple their storage roots.
Checkpointing is an invocation policy rather than a prerequisite hidden in
persistent workspace configuration. An explicit resume invocation may read
compatible checkpoints and record replacement checkpoint state for work it
executes. Runs that do not request resume perform no checkpoint I/O.
### Debug
Debug is an explicitly requested per-run inspection bundle intended for
developers and troubleshooting. It is off by default. When enabled, one bundle
contains both redacted run summaries and detailed stage and LLM traces. The
internal distinction between a safe summary and a sensitive trace remains, but
there is one public enablement and location model.
Debug data is never a cache input and has no automatic retention policy.
Notarius does not create a debug directory unless debug is requested, and it
does not automatically delete a requested bundle. Credentials remain redacted
at every level, while the bundle as a whole is treated as potentially sensitive
because traces may contain source, reference, prompt, model-response, and
intermediate artifact content.
Concise progress, warnings, and failures continue to use stdout or stderr. A
run without debug may fail without producing a filesystem inspection record.
Exact configuration fields, CLI flags, default paths, layouts, compatibility
handling, and migration mechanics are configuration and operational contracts
rather than part of this decision.
## Alternatives considered
- Keep workspace, diagnostics, checkpoints, debug, and chunk-plan cache as
separate public concepts. This preserves compatibility and the current safe
default-on failure records, but retains overlapping configuration and asks
operators to reason about implementation-specific categories.
- Keep diagnostics as an always-available redacted operational surface and use
debug only for sensitive traces. This distinction is useful for a daemon or
managed service with an operational logging contract, but the current CLI can
report concise failures on stderr and provide inspection data when explicitly
requested.
- Put all non-output state beneath one physical root. This minimizes path
configuration, but couples reconstructible caches to per-run inspection data
and couples cache families whose identity, sensitivity, and cleanup policies
differ.
- Treat checkpoints as durable run state rather than cache. This emphasizes
resumability, but checkpoints are derived, compatibility-checked data that may
be deleted and recomputed. Cache more accurately describes their lifecycle.
## Consequences
The operator model becomes smaller: normal runs produce output and may use
cache; developers explicitly request debug. Public configuration no longer
exposes a workspace or overlapping diagnostics and debug systems.
The implementation retains separate collaborators and serializers where their
security or lifecycle boundaries differ. Redacted summaries remain useful as
the index to a debug bundle, and chunk plans and checkpoints retain separate
stores even though both are cache.
Existing configuration, environment variables, flags, examples, and
documentation require a deliberate compatibility transition. Default-on
diagnostic directories disappear, so failures without debug are inspectable
only through stderr and any durable output completed before the failure.
Debug becomes easier to request and substantially more complete, but enabling
it creates sensitive files that the operator must protect and remove. Cache
cleanup is recoverable but may repeat expensive work, while deleting output is
data loss from the user's perspective.

334
docs/roadmap/adr0006.md Normal file
View File

@@ -0,0 +1,334 @@
# ADR-0006 Feature Roadmap
This roadmap defines the intended end state for
[ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is
not implemented.
## Intent
Notarius should communicate filesystem behavior in terms of why data exists:
durable output, reconstructible cache, or explicitly requested debug material.
The public configuration and CLI should not expose internal distinctions such
as workspace state or a second diagnostics product.
The implementation may retain focused writers, serializers, and security
boundaries. This roadmap changes their public composition and lifecycle rather
than requiring every kind of data to share one package or physical root.
## Target Configuration
The configuration schema advances to version 3 and uses these top-level
surfaces:
```yaml
version: 3
output:
directory: ./notarius-output
cache:
chunk_plans:
directory: ""
mode: auto
checkpoints:
directory: ""
debug:
directory: ./notarius-debug
```
An empty cache directory selects that cache family's platform-appropriate
per-user default. A configured directory is the exact root; Notarius does not
append the cache-family name to it.
The `workspace` and `diagnostics` top-level configuration surfaces do not exist
in version 3. In particular, version 3 has no persistent debug enablement,
diagnostics enablement or retention, checkpoint enablement, or shared workspace
directory.
### Output configuration
`output.directory` selects the durable output root. Precedence is the explicit
`--output-dir` flag, `NOTARIUS_OUTPUT_DIR`, the file value, then
`./notarius-output`.
The pipeline's existing `pipelines.<id>.output` module binding remains separate:
it selects the output encoder, while top-level `output.directory` selects where
the CLI places that encoder's logical files.
### Chunk-plan cache configuration
`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh` with the
ADR-0005 semantics. Precedence is the explicit `--chunk_cache` flag,
`NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then `auto`.
`cache.chunk_plans.directory` selects the exact canonical plan root. Precedence
is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, the file value, then
`<os.UserCacheDir>/notarius/chunk-plans`. On Linux, the default therefore uses a
valid absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache` when that
variable is unset. A relative `XDG_CACHE_HOME` remains an error.
The recommended system-service value remains
`/var/cache/notarius/chunk-plans`. ADR-0005 plan identities, envelopes,
permissions, atomic publication, validation, provenance, and cleanup semantics
do not change.
### Checkpoint cache configuration
`cache.checkpoints.directory` selects the exact checkpoint root. Precedence is
`NOTARIUS_CACHE_CHECKPOINTS_DIR`, the file value, then
`<os.UserCacheDir>/notarius/checkpoints`.
For a system-wide Linux deployment under a dedicated service account, the
recommended configured root is `/var/cache/notarius/checkpoints`, provisioned
with restrictive service-account ownership independently from the chunk-plan
root.
Checkpoint directories and files use restrictive permissions because they may
contain source text and intermediate artifacts. They inherit the sensitivity of
the data captured in them. Their existing compatibility validation, pipeline
identity, source and lane payloads, and downstream dependency fingerprints
remain intact. Checkpoints do not store or select chunk results.
There is no `cache.checkpoints.enabled` setting. Checkpoint I/O is controlled by
the invocation's `--resume` flag.
### Debug configuration
`debug.directory` selects the debug root but does not enable debug output.
Precedence is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then
`./notarius-debug`.
There is no environment or file setting that enables debug. This prevents an
ambient production configuration from silently recording potentially sensitive
trace data and creating additional retained copies of application data.
## Target CLI
The run command retains `--output-dir`, `--chunk_cache`, and `--resume`, and
adds:
- `--debug`: enable the complete debug bundle for this invocation;
- `--debug-dir <path>`: override its root; valid only with `--debug`.
`--diagnostics-dir` is removed. There is no `--workspace`, `--cache-dir`, or
separate diagnostics flag.
`--resume` means “reuse compatible checkpoints when present and record
checkpoint state for work executed by this invocation.” On a first invocation
with no compatible state, it executes normally and creates checkpoints. A run
without `--resume` neither loads nor records checkpoints and does not resolve or
create the checkpoint root.
The CLI reports the durable output directory on success. When debug is enabled,
it also reports the allocated debug bundle path on success and includes that
path in failure reporting once allocation has succeeded.
Malformed values from every supplied configuration source remain errors even
when a higher-precedence source would otherwise override them.
## Filesystem Layout
### Output
Durable logical files retain the current per-run layout:
```text
<output-root>/<run-id>/...
```
Output creation remains atomic per file. The maintained logical JSON output
contract, including manifests, warnings, rejections, and normalized artifacts,
does not change solely because of this state-model refactor.
### Chunk plans
The existing ADR-0005 layout remains:
```text
<chunk-plan-root>/<source-sha256-hex>/plan.json
```
Chunk-plan state remains independent of checkpoint and debug roots.
### Checkpoints
The current identity hierarchy moves beneath the independently selected
checkpoint root:
```text
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/...
```
The configured root replaces the former implicit
`<workspace.directory>/checkpoints` prefix. Existing manifests and payload
schemas remain compatible when pointed at their prior physical checkpoint root.
### Debug bundle
One explicitly requested bundle is allocated before pipeline resolution at:
```text
<debug-root>/<run-id>/
summary/
trace/
```
`summary/` contains the redacted artifacts currently associated with
diagnostics: invocation metadata, redacted effective configuration, resolved
pipeline and reference provenance, checkpoint decisions, chunk-plan decisions,
run manifest, warnings, run report, and error text when available.
`trace/` contains the existing deep debug material: source and stage inputs and
outputs, plans and materialized chunks, annotations, validator attempts,
prompts, model responses, retry timing, and intermediate serialized artifacts.
Summary data never contains raw source, references, annotations, prompts, model
responses, credentials, or malformed cache bytes. Trace data may contain all
of those except credentials, which remain redacted. A debug bundle inherits the
sensitivity of the most sensitive data captured in it; it does not constitute a
separate, intrinsically higher sensitivity class. Because it aggregates and
retains an additional copy of application data, its root and files use
restrictive permissions by default.
Trace data may preserve source or reference content that the durable output
summarized, rejected, or intentionally omitted. This can make the bundle more
sensitive than the output alone, but not more sensitive than the complete set
of application data supplied to or produced by the run. Restrictive defaults
are therefore precautionary handling for data of unknown sensitivity, not a
classification of debug material as uniquely sensitive.
Debug persistence failures are command failures because the operator explicitly
requested the bundle. No debug directory is created without `--debug`. A
requested bundle is never automatically deleted based on success, warnings, or
failure; cleanup is explicit.
## Runtime Ownership
The CLI remains the composition root for physical paths and filesystem-backed
collaborators. Pipeline modules continue to return logical data and never
receive output, cache, or debug roots.
The framework may retain separate interfaces for checkpoint recording and
loading, chunk-plan storage, redacted summary recording, and deep trace
recording. The single debug facility may compose multiple internal recorders;
unification does not require weakening redaction or passing raw trace data
through summary models.
The public `workspace` type and terminology disappear from configuration,
invocation messages, debug metadata, and operator documentation. The standalone
diagnostics directory and retention surface also disappear. Domain contracts
that use “diagnostic” in another sense, such as a validator-provided diagnostic
artifact path, are not renamed solely by this decision. Generic confined-path
and atomic-write helpers may remain in an internal package but must not impose a
workspace abstraction on callers.
## Security and Lifecycle
- Output is durable user data. Notarius never automatically deletes it.
- Chunk plans and checkpoints are reconstructible cache. Exact entries or roots
may be removed, with the documented cost of recomputation.
- Debug is explicit inspection data. It is off by default, retained when
requested, and removed only by an explicit operator action.
- Checkpoints and debug bundles inherit the sensitivity of the application data
they capture. Their security concern is the additional retained copy and, for
debug bundles, aggregation of that data—not an intrinsically higher
sensitivity classification.
- Operators are responsible for choosing debug locations, access controls, and
retention appropriate to the data processed by the run. Documentation must
explain that a bundle can retain material omitted from the durable output.
- Cache and debug roots are separate trust boundaries and must not be shared
among mutually untrusted users.
- Credentials are excluded from output, cache, summaries, traces, logs, errors,
examples, and redacted configuration.
- Debug collection is allowlisted to application-owned data. It must not capture
unrelated process environment values, host secrets, or arbitrary filesystem
content merely because they are available to the process.
- Default runs may create output and use the default chunk-plan cache, but do
not create checkpoint or debug state unless their invocation flags request
those surfaces.
## Compatibility Policy
Version 3 is a deliberate configuration break. Version 2 files are rejected
with an actionable message directing operators to the migration documentation;
Notarius does not retain parallel legacy field parsing indefinitely.
The version 2 migration is:
- `workspace.chunk_cache.mode` -> `cache.chunk_plans.mode`;
- `workspace.chunk_cache.directory` -> `cache.chunk_plans.directory`;
- `<workspace.directory>/checkpoints` -> `cache.checkpoints.directory`, when
existing checkpoint reuse is desired;
- `--diagnostics-dir`, diagnostics fields, workspace diagnostics fields, and
their environment variables -> removal or the new debug controls;
- `workspace.debug.enabled` -> removal; invoke with `--debug`;
- `<workspace.directory>/debug` -> `debug.directory`, if the same parent is
desired; and
- `workspace.resume.enabled` -> removal; invoke with `--resume`.
Environment migration is:
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE` ->
`NOTARIUS_CACHE_CHUNK_PLANS_MODE`;
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` ->
`NOTARIUS_CACHE_CHUNK_PLANS_DIR`;
- `NOTARIUS_WORKSPACE_DIR` -> no direct replacement; configure the applicable
output, checkpoint-cache, and debug roots separately;
- `NOTARIUS_WORKSPACE_RESUME_ENABLED` and
`NOTARIUS_WORKSPACE_DEBUG_ENABLED` -> removal in favor of invocation flags;
and
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`,
`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`, `NOTARIUS_WORK_DIR`, and
`NOTARIUS_DIAGNOSTICS_RETENTION` -> removal with the standalone diagnostics
surface.
Existing chunk-plan files are reused without migration when the new directory
resolves to the same root. Existing checkpoint files are reusable when
`cache.checkpoints.directory` names the former checkpoint root and their normal
identity and schema checks pass. Existing diagnostics and debug directories are
not moved, merged, or deleted automatically.
The durable output schema and ADR-0005 cache envelope schema are not versioned
solely because configuration advances to version 3.
## Documentation Outcomes
When implemented:
- the CLI reference owns all flags and invocation semantics;
- configuration owns version 3 fields, environment variables, defaults, and
precedence;
- operations owns physical layouts, permissions, cleanup, failure inspection,
and per-user and system-service cache guidance;
- the integration contract continues to own durable logical output;
- internal pipeline and component documents explain collaborator composition
without reintroducing public workspace terminology; and
- migration documentation provides one complete version 2 to version 3 example.
Current-behavior documentation must not describe this target before the
corresponding behavior is implemented.
## Non-Goals
This feature does not introduce:
- remote output, cache, or debug storage;
- cache garbage collection, quotas, archival, history, or rollback;
- automatic debug retention or upload;
- a daemon logging, metrics, or telemetry system;
- a change to pipeline topology, artifact schemas, chunk-plan identity, or
module contracts; or
- a requirement that output, different cache families, and debug share one
physical parent directory.
## Completion Outcomes
The feature is complete when ordinary runs expose only durable output and
canonical chunk-plan cache behavior; checkpoint I/O occurs only under
`--resume`; one `--debug` invocation produces a complete retained summary and
trace bundle; no public workspace or diagnostics configuration remains; version
2 migration failures are actionable; existing compatible chunk plans and
checkpoints remain reusable at explicitly selected roots; and all configuration,
CLI, filesystem, security, compatibility, integration, and repository-wide
tests pass.

657
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,657 @@
# ADR-0006 Staged Implementation Plan
This document is the executable implementation plan for the target state in
[ADR-0006 Feature Roadmap](adr0006.md), governed by
[ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is
not implemented.
The audience is an LLM coding agent. Implement the stages in order. Each stage
is scoped to finish with a compiling, tested repository and may be assigned as
one implementation prompt.
## Execution Rules
Before every stage:
1. read [Development](../development.md), both documents under `docs/policy/`,
ADR-0006, the feature roadmap, this plan, and every stage-specific document
named below;
2. inspect `git status` and preserve user changes;
3. inspect the focused package contracts and tests before editing; and
4. confirm that all earlier stages are complete.
During every stage:
- implement only that stage and prerequisites discovered to be inseparable;
- keep physical path selection in the CLI composition root and keep pipeline
modules independent of output, cache, and debug roots;
- preserve checkpoint identity, chunk-plan identity, durable output schemas,
deterministic ordering, cancellation, redaction, confined paths, and atomic
writes unless this plan expressly changes them;
- treat debug persistence failures as command failures once debug is requested;
- add or update focused tests with every behavior change;
- keep unimplemented behavior under `docs/roadmap/` until the final
documentation stage; and
- do not begin the next stage after meeting the current exit criteria.
At the end of every stage:
1. run the focused tests listed for that stage;
2. run `go test ./...`;
3. run `go vet ./...`;
4. run `go build ./cmd/notarius`;
5. run `git diff --check`; and
6. mark the stage complete in this document only after all checks pass.
If repository reality conflicts with this plan, stop and update the plan in the
same change before implementing a materially different design. Do not silently
invent a new public contract.
## Fixed Decisions
The following choices are settled for this implementation.
### Public state model
- The only public filesystem surfaces are output, cache, and debug.
- Output is durable user data. Cache is reconstructible. Debug is explicitly
requested inspection data and is not a cache input.
- `workspace` and standalone `diagnostics` disappear from configuration, CLI
flags, operator messages, debug metadata, package names exposed to callers,
and current-behavior documentation.
- Domain uses of the word “diagnostic,” including validator diagnostic artifact
paths, are unchanged when they do not describe the removed filesystem
surface.
### Configuration version 3
- `SupportedFileConfigVersion` becomes `3`. Version 3 has these runtime types:
```go
type OutputConfig struct {
Directory string `json:"directory"`
}
type CacheConfig struct {
ChunkPlans ChunkPlanCacheConfig `json:"chunk_plans"`
Checkpoints CheckpointCacheConfig `json:"checkpoints"`
}
type ChunkPlanCacheConfig struct {
Directory string `json:"directory,omitempty"`
Mode pipeline.ChunkCacheMode `json:"mode"`
}
type CheckpointCacheConfig struct {
Directory string `json:"directory,omitempty"`
}
type DebugConfig struct {
Directory string `json:"directory"`
}
```
`Config` contains `Output`, `Cache`, and `Debug` fields and no `Workspace` or
`Diagnostics` field. File-config types mirror the YAML shape in the feature
roadmap.
- Defaults are `./notarius-output`, chunk-plan mode `auto`, empty cache
directories meaning per-user defaults, and `./notarius-debug`.
- There is no persistent checkpoint enablement, resume enablement, debug
enablement, diagnostics enablement, or retention setting.
- Output-directory precedence is explicit `--output-dir`,
`NOTARIUS_OUTPUT_DIR`, file, then `./notarius-output`.
- Chunk-plan mode precedence is explicit `--chunk_cache`,
`NOTARIUS_CACHE_CHUNK_PLANS_MODE`, file, then `auto`.
- Chunk-plan root precedence is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, file, then
`<os.UserCacheDir>/notarius/chunk-plans`.
- Checkpoint root precedence is `NOTARIUS_CACHE_CHECKPOINTS_DIR`, file, then
`<os.UserCacheDir>/notarius/checkpoints`.
- Debug-root precedence is explicit `--debug-dir`, `NOTARIUS_DEBUG_DIR`, file,
then `./notarius-debug`. A directory value never enables debug.
- A configured cache directory names the exact family root. No family suffix is
appended. Explicit relative paths retain the current CLI convention and are
interpreted relative to the process working directory.
- Empty cache directory fields in a version 3 file select their per-user
defaults. A supplied environment directory override and an explicit CLI
directory flag must be non-empty after trimming. Empty output or debug file
values and malformed modes are errors.
- Every supplied file, environment, and CLI value is validated even when a
higher-precedence value wins. Apply CLI overrides only after file and
environment parsing has succeeded.
- Parse the YAML version header before decoding the strict version 3 schema so
a version 2 file receives the intentional migration error rather than an
incidental unknown-field error. Continue to use `KnownFields(true)` for the
supported schema.
### Cache families
- ADR-0005 behavior and the chunk-plan envelope and path layout remain
byte-compatible. Move only configuration ownership and the default-root
helper; do not change lookup, validation, publication, provenance, or
materialization behavior.
- Chunk-plan `bypass` does not resolve or create its root. `auto` and `refresh`
resolve it lazily when the run constructs the store.
- Checkpoint I/O exists only for an invocation with `--resume`. Without that
flag, the CLI does not resolve the checkpoint root, compute a checkpoint
identity, construct a checkpoint store, load a checkpoint, or record one.
- With `--resume`, the CLI constructs both loader and recorder. Compatible
state is reused; missing or incompatible state executes normally and writes
replacement checkpoints.
- Checkpoint identities, relative paths, manifests, payloads, compatibility
rules, and dependency fingerprints remain unchanged. Existing files are
reusable when the new root points at the former
`<workspace.directory>/checkpoints` directory.
- Preserve the existing checkpoint JSON field
`workspace_schema_version` and values `notarius.workspace.v2` and
`notarius.workspace.v1`. They are frozen wire compatibility identifiers, not
public configuration or package terminology. Rename internal Go identifiers
only if doing so cannot alter serialized bytes or compatibility messages.
- The recommended Linux service roots are
`/var/cache/notarius/chunk-plans` and `/var/cache/notarius/checkpoints`,
provisioned independently for the dedicated service account.
### Debug bundle
- `--debug` is the only debug enablement control. `--debug-dir <path>` is valid
only when `--debug` is present; misuse is a CLI syntax error with exit code
`2`.
- One requested bundle is allocated before pipeline resolution at
`<debug-root>/<run-id>/`, with `summary/` and `trace/` children.
- Bundle directories are created with mode `0700` and files with mode `0600`.
Do not silently change permissions on a pre-existing operator-owned debug
parent; enforce restrictive permissions on directories and files created for
the bundle.
- `summary/` contains the current redacted diagnostics artifacts, using their
current logical filenames where applicable. `trace/` contains the current
deep debug paths. There is no retention mode or automatic deletion.
- The summary run report uses `debug_path`, not `diagnostics_path`. Invocation
and effective-config payloads use debug/state terminology and contain no
obsolete public workspace settings.
- The redacted summary excludes raw source, references, annotations, prompts,
model responses, credentials, and malformed cache bytes. Trace data may
contain application data but never credentials.
- Debug collection records only explicit application-owned payloads. It does
not dump the process environment, inspect unrelated files, or capture host
secrets. Existing sensitive-key and credential-shaped-value redaction stays
in force at every serialization boundary.
- A bundle inherits the sensitivity of its captured application data. The
additional operational risks are copying, aggregation, and retention, not an
intrinsically higher sensitivity class.
- If any requested summary or trace write fails, the command exits `1`. When a
primary operation and a debug write both fail, report both errors on stderr
without recursively attempting the same failed write.
- On success, stdout reports the durable output directory and, when enabled,
the debug bundle path. Once allocation succeeds, failure output also names
the bundle path.
### Compatibility and documentation
- Version 2 configuration is rejected; no legacy fields or environment aliases
remain after the implementation. The error directs the operator to the
version 2-to-3 migration section in `docs/config.md`.
- Existing chunk-plan and checkpoint state is never moved or deleted
automatically. Existing diagnostics and debug directories are likewise left
untouched.
- The durable output contract and chunk-plan envelope schema are not versioned
because of this refactor.
- `docs/config.md` owns the complete version 2-to-3 configuration and
environment migration example. `docs/operations.md` owns physical state reuse
and cleanup guidance. Other documents link to those owners rather than
duplicating volatile details.
## Planned Internal Boundaries
Use these package responsibilities unless an existing collision requires the
smallest obvious naming adjustment.
- `internal/core/fileio` owns generic confined relative-path construction and
atomic JSON/byte writes with caller-selected directory and file modes. Its
errors say “file” or “artifact,” never “workspace.” It must not replace the
chunk-plan store's stronger `os.Root`-based entry protections.
- `internal/framework/checkpoint` owns checkpoint identity, manifests, payload
codecs, filesystem loader, and filesystem recorder. Constructors accept an
exact checkpoint root and an identity; they do not accept a workspace or a
general state settings object.
- `internal/framework/chunkplan` continues to own durable chunk-plan storage.
Per-user cache-root default helpers belong with configuration/path resolution,
not in a workspace package.
- `internal/core/debugbundle` owns allocation of a per-run bundle and the
redacted summary writer. It exposes exact `Path`, `SummaryRoot`, and
`TraceRoot` values and has no retention API.
- `internal/framework/debug` remains the pipeline-facing deep-trace adapter but
becomes a root-based filesystem recorder aimed at the bundle's `trace/`
directory. Pipeline debug interfaces and synchronization remain unchanged.
- `internal/cli` remains the only composition root. It resolves effective
roots, allocates requested state, constructs collaborators, writes summary
artifacts, and reports paths.
After the cutover, delete `internal/core/workspace` and
`internal/core/diagnostics` if no remaining code has a distinct domain-neutral
reason to retain them. Do not preserve wrapper packages solely to avoid
updating imports.
## Stage 1: Accept ADR-0006 and decouple cache storage internals
**Status:** Not started
### Objective
Accept the architectural decision and remove checkpoint and chunk-plan storage
from workspace-specific constructors without changing current public
configuration or CLI behavior yet.
### Read first
- `internal/core/workspace/`
- `internal/framework/checkpoint/`
- `internal/framework/chunkplan/`
- `internal/framework/pipeline/checkpoint.go`
- `internal/cli/chunk_cache_test.go`
- the checkpoint sections of `docs/internal/pipeline.md` and
`docs/internal/diagnostics.md`
### Implement
1. Confirm ADR-0006 matches the finalized feature roadmap, then change only its
status from `Proposed` to `Accepted`.
2. Add the generic confined-write primitives in `internal/core/fileio`, with
caller-selected `0700`/`0600` support and focused path, atomicity, and
permission tests.
3. Move checkpoint identity, manifest, path, and persistence ownership into
`internal/framework/checkpoint`. Preserve every serialized field, schema
string, digest input, path component, validation reason, and payload byte.
4. Replace `NewWorkspaceRecorder` and `NewWorkspaceLoader` with root-based
filesystem constructors. During this preparatory stage, adapt the existing
CLI to pass the checkpoint root derived from its current workspace settings
so public behavior remains unchanged.
5. Move `DefaultChunkPlanRoot` out of `internal/core/workspace` into the
configuration/path-resolution boundary and retain the existing injected
`UserCacheDir` behavior and errors.
6. Remove checkpoint-specific and chunk-plan-specific files from
`internal/core/workspace`. Leave only pieces still required by the current
diagnostics/debug composition until later stages.
### Tests
Prove that:
- old and refactored checkpoint identities and relative paths are identical;
- representative manifests marshal to identical JSON and v1/v2 compatibility
behavior is unchanged;
- a root-based recorder's output is reusable by the root-based loader;
- confined writes reject absolute, unclean, traversal, and backslash paths;
- atomic writes use the requested modes and do not leave temporary files after
failure; and
- chunk-plan default-root and bypass tests remain unchanged in behavior.
Run at minimum:
```sh
go test ./internal/core/fileio
go test ./internal/framework/checkpoint ./internal/framework/chunkplan
go test ./internal/cli
```
### Exit criteria
ADR-0006 is accepted; cache persistence no longer depends on workspace-owned
identity or constructors; all current user-visible behavior remains unchanged;
and the repository-wide checks pass.
## Stage 2: Build the unified debug-bundle collaborators
**Status:** Not started
### Objective
Implement and test the target debug bundle behind internal constructors before
changing the public CLI enablement and configuration model.
### Read first
- `internal/core/diagnostics/`
- `internal/framework/debug/recorder.go`
- `internal/framework/pipeline/debug.go`
- `internal/framework/pipeline/runner_attempt_debug_test.go`
- `internal/framework/pipeline/runner_terminal_debug_test.go`
- `internal/core/config/redaction.go`
- [Diagnostics Internals](../internal/diagnostics.md)
### Implement
1. Add `internal/core/debugbundle` with a collision-safe allocator that accepts
the exact debug parent, uses the established `run-<unix-nanoseconds>` ID
shape, creates `<run-id>/summary` and `<run-id>/trace`, and returns a bundle
exposing those exact paths.
2. Port the current diagnostics artifact constants and redacted writer methods
into a summary writer rooted at `summary/`. Remove retention decisions from
the new type. Use `0600` files and confined atomic writes.
3. Define summary-owned invocation and run-report payloads. Rename
diagnostics-specific methods and fields to summary/debug terminology while
retaining the useful artifact filenames listed in the feature roadmap.
4. Replace the deep trace recorder's workspace-settings constructor with a
constructor accepting the exact `trace/` root. Keep the existing
`pipeline.DebugRecorder` contract and synchronization wrapper.
5. Rename `RedactedDiagnosticsPayload` to a neutral redacted-summary method and
update its tests. Ensure the v3 effective configuration planned for the next
stage can be represented without secrets or raw application payloads.
6. Do not expose `--debug` or change current run behavior in this stage. The new
bundle is an internal collaborator exercised directly by tests until the
atomic CLI cutover.
### Tests
Add focused tests for:
- collision retry and exhausted allocation;
- exact `summary/` and `trace/` layout;
- `0700` bundle directories and `0600` files;
- path traversal and symlink/path confinement behavior;
- every summary artifact and error log;
- absence of retention/deletion behavior;
- credential and sensitive-key redaction;
- no ambient environment or unrelated filesystem capture; and
- trace-recorder write failure propagation and concurrent synchronized writes.
Run at minimum:
```sh
go test ./internal/core/debugbundle
go test ./internal/core/config
go test ./internal/framework/debug ./internal/framework/pipeline
```
### Exit criteria
The target bundle can be allocated and populated through tested summary and
trace collaborators, no public invocation behavior has changed, and all
repository-wide checks pass.
## Stage 3: Cut configuration and CLI behavior over to the three surfaces
**Status:** Not started
### Objective
Make the version 3 configuration, new flags, independent roots, resume policy,
and opt-in debug bundle the complete public runtime behavior in one atomic
cutover.
### Read first
- all files under `internal/core/config/`
- `internal/cli/run.go`, `internal/cli/run_test.go`,
`internal/cli/chunk_cache_test.go`, and `internal/cli/compatibility_test.go`
- the collaborators completed in Stages 1 and 2
- `examples/*.config.yml`
- the Target Configuration, Target CLI, and Filesystem Layout sections of the
feature roadmap
### Implement
1. Replace the runtime and file configuration models with the fixed version 3
types. Remove legacy diagnostics/workspace defaults, merge logic,
validation, effective-config fields, and redaction output.
2. Implement strict two-pass version handling. Reject version 2 with an
actionable migration message; reject other unsupported versions clearly;
and strictly decode version 3 with unknown fields rejected.
3. Implement the five new environment variables and remove all environment
aliases listed for removal in the feature roadmap. Preserve malformed-source
validation before precedence selection.
4. Add `--debug` and `--debug-dir`; remove `--diagnostics-dir`; update usage,
flag reordering, value validation, and exit-code behavior. Keep
`--output-dir`, `--chunk_cache`, and `--resume` with their target semantics.
5. Resolve the effective output directory after configuration loading and CLI
overrides. Use it for durable placement and summary metadata; do not fall
back to a separate CLI-only default helper.
6. When `--debug` is present, resolve the effective debug root and allocate the
bundle after configuration succeeds but before catalog lookup or pipeline
resolution. Use its run ID for the entire invocation, construct the summary
writer and trace recorder, and record invocation metadata immediately.
Without `--debug`, use the existing injected clock to generate a run ID and
construct no debug collaborator or directory.
7. Resolve the chunk-plan root only for `auto` or `refresh`, using the new
configuration family. Preserve all ADR-0005 store construction behavior.
8. When `--resume` is absent, pass no-op checkpoint collaborators without
resolving `UserCacheDir` for checkpoints. When it is present, resolve the
configured or per-user checkpoint root, create the identity, and construct
both root-based loader and recorder.
9. Replace diagnostics writes in the command with conditional summary writes.
Populate invocation metadata, redacted effective configuration, resolved
pipeline, resolved references, checkpoint events, chunk-plan decisions, run
manifest, warnings, run report, and error text as each becomes available.
10. Replace retention-aware failure handling with one failure path that prints
the primary error, attempts a single summary error record when a bundle
exists, reports any secondary debug error, and includes the allocated bundle
path. Never delete a requested bundle.
11. Report output and debug paths as fixed above. Keep warning and exit-code
behavior otherwise unchanged.
12. Convert both maintained example configs and package testdata configs to
version 3 so every executable example uses the supported schema.
### Tests
Update or add CLI and config tests covering:
- exact v3 defaults, file fields, strict unknown fields, JSON/redacted shape,
and validation;
- precedence for output, chunk mode/root, checkpoint root, and debug root;
- malformed lower-precedence file or environment values despite valid CLI
overrides;
- actionable version 2 rejection before legacy unknown fields are decoded;
- rejection of every removed environment variable and configuration surface by
absence of effect, with obsolete file fields rejected as unknown;
- `--debug-dir` without `--debug`, empty explicit directory values, and removed
`--diagnostics-dir`;
- no debug allocation for ordinary runs and allocation before resolution
failures for debug runs;
- no checkpoint root resolution, loading, or recording without `--resume`;
- first-use resume writes checkpoints and later resume reuses them;
- `bypass` does not resolve a chunk-plan root;
- output directory selection and unchanged durable logical files;
- success and failure messages with and without a debug path; and
- requested summary or trace write failures producing exit code `1` without
masking a primary error.
Run at minimum:
```sh
go test ./internal/core/config
go test ./internal/cli
go test ./internal/modules/integration ./internal/modules/seriatim/input/transcript
```
### Exit criteria
Only version 3 configuration is accepted; ordinary runs expose output and
chunk-plan cache only; resume and debug are invocation-controlled; all examples
and CLI tests use the new public contract; and the repository-wide checks pass.
## Stage 4: Remove legacy workspace and diagnostics implementation
**Status:** Not started
### Objective
Complete the internal ownership refactor and prove that no obsolete public state
model remains hidden behind compatibility wrappers.
### Read first
- the complete import graph reported by `rg 'core/(workspace|diagnostics)'`
- `internal/core/workspace/`
- `internal/core/diagnostics/`
- `internal/framework/checkpoint/`
- `internal/core/debugbundle/`
- `internal/framework/debug/`
- `internal/cli/run.go`
### Implement
1. Delete the remaining workspace settings, path, writer, and tests after moving
any still-valid generic behavior to `fileio`, `checkpoint`, or
`debugbundle`.
2. Delete the old diagnostics package, retention modes, run directory, and
tests after confirming all desired redacted artifacts are owned and tested
by `debugbundle`.
3. Remove compatibility constructors, aliases, fields, environment names, and
dead CLI helpers introduced solely by the old state model. Do not retain
deprecated parsing.
4. Rename internal variables, metadata fields, recorder types, and errors that
still use workspace or diagnostics to describe the removed surfaces.
5. Retain and document the frozen checkpoint wire schema identifiers. Retain
unrelated domain “diagnostic” terminology.
6. Confirm generic framework and pipeline packages receive only collaborator
interfaces and never physical roots.
### Tests
- Run `rg` checks proving there are no remaining imports of deleted packages,
public configuration fields, removed environment names, removed flags, or
operator-facing workspace/diagnostics messages.
- Allow matches only in ADR/history, version 2 migration guidance, frozen
checkpoint wire compatibility, and unrelated domain diagnostic contracts.
- Run all checkpoint, debug, CLI, and integration tests after deletion.
### Exit criteria
The old packages and compatibility surfaces are gone; remaining historical
terms are intentional and documented; package ownership matches the Planned
Internal Boundaries; and all checks pass.
## Stage 5: Harden cross-surface security and compatibility
**Status:** Not started
### Objective
Exercise the completed feature across failures, permissions, reuse, redaction,
and surface independence before changing current-behavior documentation.
### Read first
- all focused tests changed in Stages 1 through 4
- `internal/cli/compatibility_test.go`
- `internal/framework/pipeline/runner_*debug*_test.go`
- `internal/framework/checkpoint/*_test.go`
- `internal/framework/chunkplan/store_test.go`
- the Security and Lifecycle and Compatibility Policy sections of the feature
roadmap
### Implement
1. Add an end-to-end state-surface matrix covering debug on/off, resume on/off,
and chunk modes `auto`, `bypass`, and `refresh`. Assert exactly which roots
are resolved and which files are created.
2. Add integration coverage proving chunk plans and checkpoints use independent
configured roots and that debug never affects cache selection or reuse.
3. Reuse a representative pre-refactor checkpoint fixture from an explicitly
selected old checkpoint root. Assert byte-compatible loading and normal
replacement behavior; do not create a migration path or rewrite untouched
entries.
4. Cover early failures before debug allocation, failures after allocation but
before pipeline execution, pipeline failures, output-write failures, summary
failures, and trace failures. Assert stable exit codes, stderr content, and
retained bundle state.
5. Inspect all summary artifacts for forbidden raw content and secrets. Inspect
trace artifacts to prove expected application data is present while API keys,
credential-shaped values, unrelated environment values, and unrelated file
contents are absent.
6. Assert created bundle and checkpoint modes on supported Unix systems and
preserve portable behavior on other supported platforms.
7. Confirm deletion is never automatic for output or debug, while manually
removing an exact cache entry only causes recomputation.
8. Confirm durable logical output and chunk-plan envelope bytes have not changed
solely because of ADR-0006.
### Tests and validation
Run focused packages while iterating, then run:
```sh
go test ./...
go vet ./...
go build ./cmd/notarius
git diff --check
```
Run race-enabled tests for the concurrent debug and checkpoint collaborators if
supported by the environment:
```sh
go test -race ./internal/framework/debug ./internal/framework/checkpoint ./internal/cli
```
### Exit criteria
All three surfaces are independent under success and failure; compatibility and
permissions are proven; debug captures only intended data; durable contracts
are unchanged; and all repository-wide checks pass.
## Stage 6: Update canonical documentation and close the roadmap
**Status:** Not started
### Objective
Make current-behavior documentation and maintained examples describe the
implemented version 3 state model, then remove completed planning documents.
### Read first
- [Documentation Policy](../policy/documentation.md)
- [Architecture Policy](../policy/architecture.md)
- [CLI Reference](../cli.md)
- [Configuration](../config.md)
- [Operations](../operations.md)
- [Internal Overview](../internal/overview.md)
- [Diagnostics Internals](../internal/diagnostics.md)
- [Development](../development.md)
- `README.md`, `examples/`, and the final code and tests
### Implement
1. Update `docs/config.md` for version 3 fields, defaults, validation,
precedence, and the five supported state environment variables. Add one
complete version 2-to-3 before/after migration example and list every removed
setting and environment variable from the feature roadmap.
2. Update `docs/cli.md` for `--debug`, `--debug-dir`, target `--resume`
semantics, removed `--diagnostics-dir`, reporting, and failure behavior.
3. Rewrite the relevant `docs/operations.md` sections around output, chunk-plan
cache, checkpoint cache, and debug bundles. Document exact layouts,
permissions, inherited sensitivity, aggregation/copying/retention risk,
cleanup, compatible old-root reuse, per-user defaults, and the two
`/var/cache/notarius/...` Linux service recommendations.
4. Update `docs/policy/architecture.md` to describe the implemented three
surfaces and remove the obsolete five-surface model. Keep normative safety
rules concise and link operational details to Operations.
5. Replace `docs/internal/diagnostics.md` with a correctly named internal state
document, or distribute its content among focused internal documents if that
better matches the final packages. Update `docs/internal/overview.md`,
`docs/internal/pipeline.md`, `docs/development.md`, and all inbound links.
6. Update README orientation and examples only where they currently mention the
old schema or invocation. Do not duplicate the complete CLI, configuration,
or operations contracts.
7. Search the repository for stale flags, fields, environment names, paths,
package names, and workspace/diagnostics terminology. Preserve only ADR
history, explicit version 2 migration text, frozen checkpoint wire fields,
and unrelated domain diagnostics.
8. After implementation and documentation validation are complete, remove
`docs/roadmap/adr0006.md` and this implementation plan. They are completed
planning artifacts; the accepted ADR and canonical current-behavior
documents become authoritative.
### Tests and validation
- Validate every command, field, default, environment variable, path, and
example against code and focused tests.
- Validate all changed links and the task-specific reading map.
- Run any repository documentation or link checker if present.
- Run the complete repository-wide validation in the Execution Rules.
### Exit criteria
Canonical documentation describes only implemented behavior, migration and
operations each have one owner, all maintained examples are valid version 3,
no stale public state terminology remains, completed roadmap documents are
removed, and all checks pass.