diff --git a/README.md b/README.md index 89b0650..91b50c2 100644 --- a/README.md +++ b/README.md @@ -5,7 +5,8 @@ with explicit, configurable pipeline modules. The current implementation reads Seriatim transcript JSON, chunks the source units, extracts D&D spell-cast artifacts with a Scriptorium-backed LLM runtime, -and writes JSON output plus diagnostics when enabled. +and writes JSON output. Add `--debug` when a per-run inspection bundle is +needed. ```sh OPENROUTER_API_KEY=... \ diff --git a/docs/cli.md b/docs/cli.md index 1fb66c8..1e22527 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -9,7 +9,7 @@ For the minimal end-to-end invocation, see the [README](../README.md). ```text notarius help -notarius run --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--diagnostics-dir path] [--llm-profile id] [--resume] [--session-id id] [--reference selector=path] [--without-reference selector] +notarius run --input path/to/source.json [--config path/to/config.yml] [--only lane-a,lane-b] [--chunk_cache auto|bypass|refresh] [--output-dir path] [--resume] [--debug [--debug-dir path]] [--llm-profile id] [--session-id id] [--reference selector=path] [--without-reference selector] notarius config validate [--config path/to/config.yml] [--pipeline pipeline-id] [--only lane-a,lane-b] notarius pipelines list [--config path/to/config.yml] [--json] ``` @@ -35,11 +35,11 @@ Flags: invocation. `auto` reuses a valid plan by canonical source digest, `bypass` performs no plan-cache I/O, and `refresh` regenerates and replaces a valid plan only after chunk validation succeeds. See - [Configuration](config.md#workspace) for the persistent setting, precedence, + [Configuration](config.md#state-surfaces) for the persistent setting, precedence, and cache-root selection. - `--output-dir path`: output root. Defaults to `./notarius-output`. -- `--diagnostics-dir path`: diagnostics work directory override for this - invocation. It does not change the workspace directory. +- `--debug`: allocate and retain one debug bundle for this invocation. +- `--debug-dir path`: debug-bundle root override. This flag requires `--debug`. - `--llm-profile id`: override every effective LLM-capable pipeline module binding with one Scriptorium profile ID. Validator-specific profiles are not overridden. @@ -52,8 +52,9 @@ Flags: `=path`. On success, the command prints the completed pipeline ID, normalized output and -rejected output counts, and the output directory. If the run completes with warnings, -the warning count is printed to stderr. +rejected output counts, and the output directory. A debug-enabled run also +prints `debug=`. If the run completes with warnings, the warning +count is printed to stderr. Reference flags are resolved against selected chunk, extractor, merger, and normalizer targets before the run starts. Flat slot names are accepted only @@ -122,7 +123,9 @@ go run ./cmd/notarius run dnd-session \ --session-id campaign-17-session-04 ``` -The resume flag can be added to an otherwise identical run invocation: +The resume flag can be added to an otherwise identical run invocation. It both +loads compatible checkpoints and records replacements for work executed by that +invocation; without it, the checkpoint root is not used: ```sh go run ./cmd/notarius run dnd-session \ @@ -131,6 +134,18 @@ go run ./cmd/notarius run dnd-session \ --resume ``` +Use `--debug` to retain the redacted summary and trace bundle for one run. The +bundle is allocated before pipeline resolution; once allocated, its path is +also printed to stderr if the command fails. Debug-write failures cause exit +code `1`. + +```sh +go run ./cmd/notarius run dnd-session \ + --config examples/dnd-spells.config.yml \ + --input examples/seriatim-minimal-transcript.json \ + --debug --debug-dir ./notarius-debug +``` + Use `refresh` when intentionally replacing the cached plan for the same source: ```sh @@ -149,8 +164,8 @@ go run ./cmd/notarius run dnd-session \ --chunk_cache bypass ``` -For checkpoint behavior, durable output, diagnostics, retention, and failure -inspection, see [Operations](operations.md). +`--diagnostics-dir` has been removed. For checkpoint behavior, durable output, +debug-bundle lifecycle, and failure inspection, see [Operations](operations.md). ## `config validate` @@ -202,8 +217,8 @@ go run ./cmd/notarius pipelines list \ - `0`: command succeeded. - `1`: command syntax was valid, but loading config, resolving modules, running - the pipeline, calling the provider, writing output, or writing diagnostics - failed. + the pipeline, calling the provider, writing output, or writing a requested + debug bundle failed. - `2`: command syntax was invalid, a command was unknown, a required argument was missing, or a flag value was malformed. diff --git a/docs/config.md b/docs/config.md index 818461b..e288438 100644 --- a/docs/config.md +++ b/docs/config.md @@ -2,8 +2,9 @@ This is the canonical reference for implemented Notarius configuration. -Notarius reads YAML config files with `version: 2`. File config is applied over -built-in defaults, then environment overrides are applied. +Notarius reads YAML config files with `version: 3`. File configuration is +applied over built-in defaults, then environment overrides are applied. Explicit +CLI overrides are applied last where the command supports them. ## Discovery @@ -21,17 +22,18 @@ The explicit-path option is defined in the [CLI reference](cli.md). - [Minimal D&D spell configuration](../examples/dnd-spells.config.yml) - [Production-oriented D&D spell configuration](../examples/dnd-spells-production.config.yml) -Both complete files are validated by the CLI test suite. The fragments below -illustrate individual fields and are not alternate complete configurations. +Both are complete version 3 files. The fragments below illustrate individual +fields and are not alternate complete configurations. ## Top-Level Fields -- `version`: required. The only supported value is `2`. +- `version`: required. The only supported value is `3`. - `scriptorium`: optional Scriptorium profile source settings. - `pipelines`: optional map of pipeline IDs to pipeline definitions. - `concurrency`: optional global concurrency settings. -- `workspace`: optional workspace settings for Notarius-owned local state. -- `diagnostics`: optional diagnostics settings. +- `output`: optional durable output placement. +- `cache`: optional chunk-plan and checkpoint cache placement. +- `debug`: optional debug-bundle placement. It does not enable debug capture. Unknown YAML fields are rejected. The removed top-level `llm_profiles` field is rejected; execution profiles now come from Scriptorium. @@ -42,14 +44,13 @@ Built-in defaults: - `concurrency.total_llm`: `1` - `concurrency.stage_workers.extract`: effective `concurrency.total_llm` -- `diagnostics.work_dir`: `/tmp/notarius` -- `diagnostics.retention`: `auto` -- `workspace.directory`: unset -- `workspace.diagnostics.enabled`: `true` -- `workspace.resume.enabled`: `false` -- `workspace.debug.enabled`: `false` -- `workspace.chunk_cache.mode`: `auto` -- `workspace.chunk_cache.directory`: unset +- `output.directory`: `./notarius-output` +- `cache.chunk_plans.mode`: `auto` +- `cache.chunk_plans.directory`: unset, selecting + `/notarius/chunk-plans` +- `cache.checkpoints.directory`: unset, selecting + `/notarius/checkpoints` +- `debug.directory`: `./notarius-debug` No pipelines are built in. A run requires a configured pipeline. @@ -92,22 +93,15 @@ These environment variables are applied after the config file: - `NOTARIUS_CONFIG`: config discovery path. - `NOTARIUS_TOTAL_LLM_CONCURRENCY`: integer global LLM concurrency. - `NOTARIUS_STAGE_WORKERS_EXTRACT`: integer extract worker limit. -- `NOTARIUS_WORKSPACE_DIR`: workspace directory. -- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`: boolean diagnostics enablement. -- `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`: workspace diagnostics retention - mode. -- `NOTARIUS_WORKSPACE_RESUME_ENABLED`: boolean resume checkpointing - enablement. -- `NOTARIUS_WORKSPACE_DEBUG_ENABLED`: boolean debug artifact enablement. -- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`: chunk-plan cache mode. -- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`: chunk-plan cache root. -- `NOTARIUS_WORK_DIR`: deprecated diagnostics work directory compatibility - override. -- `NOTARIUS_DIAGNOSTICS_RETENTION`: deprecated diagnostics retention - compatibility override. +- `NOTARIUS_OUTPUT_DIR`: durable output root. +- `NOTARIUS_CACHE_CHUNK_PLANS_MODE`: chunk-plan cache mode. +- `NOTARIUS_CACHE_CHUNK_PLANS_DIR`: chunk-plan cache root. +- `NOTARIUS_CACHE_CHECKPOINTS_DIR`: checkpoint cache root. +- `NOTARIUS_DEBUG_DIR`: debug-bundle root. -Integer environment values must parse as base-10 integers. Boolean environment -values must parse as Go booleans such as `true`, `false`, `1`, or `0`. +Integer environment values must parse as base-10 integers. Directory overrides +must be non-empty after trimming. Cache-directory fields in a file may be +empty, which deliberately selects the corresponding per-user default. The removed `NOTARIUS_LLM_DEFAULT_*` variables are not read. Configure provider endpoint, model, and credential environment variable names through Scriptorium @@ -341,71 +335,111 @@ Both modules accept UTF-8 plain text, Markdown, YAML, or JSON reference files. The extractor uses references only as supporting disambiguation material; spell casts still must be present in the source transcript. -## Workspace +## State Surfaces -`workspace` fields: +The `output`, `cache`, and `debug` top-level fields select independent physical +roots. Their layout, permissions, lifecycle, and sensitive-data handling are +defined in [Operations](operations.md). -- `directory`: optional workspace root for Notarius-owned local state. -- `resume.enabled`: boolean resume checkpointing setting. -- `debug.enabled`: boolean debug artifact setting. -- `chunk_cache.mode`: persistent chunk-plan cache mode: `auto`, `bypass`, or - `refresh`. The default is `auto`. -- `chunk_cache.directory`: optional chunk-plan cache root. This value is the - root itself; Notarius does not append `chunk-plans` to it. -- `diagnostics`: optional diagnostics settings defined below. +```yaml +output: + directory: ./notarius-output +cache: + chunk_plans: + directory: "" + mode: auto + checkpoints: + directory: "" +debug: + directory: ./notarius-debug +``` -`workspace.resume.enabled`, `workspace.debug.enabled`, and -`workspace.chunk_cache` are independent. `workspace.directory` does not affect -chunk-plan placement. For directory layout, state lifecycle, permissions, and -sensitive content, see [Operations](operations.md). +`output.directory` is the durable output root. Its precedence is +`--output-dir`, `NOTARIUS_OUTPUT_DIR`, the file value, then the default. -`chunk_cache.mode` accepts only `auto`, `bypass`, and `refresh`. In `auto`, a -valid source-addressed plan is reused and a missing or invalid record is -regenerated and published after chunk validation. `bypass` neither reads nor -writes plan-cache state. `refresh` always generates a new plan and publishes it -only after validation succeeds. +`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh`. Its precedence +is `--chunk_cache`, `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then +`auto`. `auto` reuses a valid source-addressed plan and regenerates missing or +invalid records; `bypass` performs no plan-cache I/O; `refresh` regenerates and +publishes a plan after chunk validation. -Configuration values are applied in file then environment order; an explicit -`--chunk_cache` CLI value has highest precedence for the mode. The cache root -is selected from environment, file, then the per-user default; there is no CLI -root override. Every supplied value is parsed strictly even when a higher -precedence value wins, so malformed configuration is still an error. +`cache.chunk_plans.directory` and `cache.checkpoints.directory` each name an +exact cache-family root. Their precedence is the corresponding environment +variable, the file value, then the family-specific per-user default. There is +no CLI cache-root override. The defaults are +`/notarius/chunk-plans` and +`/notarius/checkpoints`; on Unix, `os.UserCacheDir` ordinarily +uses an absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache`. A relative +`XDG_CACHE_HOME` is an error. -When `chunk_cache.directory` is unset, the root is -`/notarius/chunk-plans`. On Unix this is ordinarily -`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is an absolute -path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative -`XDG_CACHE_HOME` is rejected by `os.UserCacheDir`; Notarius reports that as a -configuration error and does not fall back to another directory. +Checkpoint I/O occurs only for `notarius run --resume`. That invocation loads +compatible checkpoints and records work it executes. Without `--resume`, +Notarius does not resolve, create, load, or record the checkpoint root. -## Diagnostics +`debug.directory` chooses a root but never enables debug capture. Its precedence +is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then the default. +Only `--debug` requests a bundle; `--debug-dir` is valid only with `--debug`. -Preferred workspace diagnostics fields: +Every supplied file, environment, and CLI value is validated even when a +higher-precedence value wins. -- `workspace.diagnostics.enabled`: set to `false` to skip creating diagnostics - run directories and diagnostics artifacts. -- `workspace.diagnostics.retention`: `auto`, `always`, or `never`. +## Version 2 To Version 3 Migration -Defaults for workspace and diagnostics fields are listed in -[Defaults](#defaults). +Version 2 files are rejected. Move each setting to the surface it controls and +remove obsolete enablement and retention controls. This complete before/after +example preserves an existing chunk-plan cache and checkpoint directory while +choosing an output and debug root explicitly. -`workspace.diagnostics.retention` overrides legacy diagnostics retention when -set. +```yaml +# Version 2 (no longer accepted) +version: 2 +workspace: + directory: /srv/notarius/state + resume: + enabled: true + debug: + enabled: true + chunk_cache: + directory: /srv/notarius/chunk-plans + mode: auto + diagnostics: + retention: always +diagnostics: + work_dir: /srv/notarius/inspection +``` -`diagnostics` fields: +```yaml +# Version 3 +version: 3 +output: + directory: /srv/notarius/output +cache: + chunk_plans: + directory: /srv/notarius/chunk-plans + mode: auto + checkpoints: + directory: /srv/notarius/state/checkpoints +debug: + directory: /srv/notarius/debug +``` -- `work_dir`: deprecated compatibility directory for per-run diagnostics. -- `retention`: deprecated compatibility retention mode. `auto`, `always`, or - `never`. +Run the migrated configuration with `--resume` when checkpoint reuse or +recording is wanted, and with `--debug` when a debug bundle is wanted. -Existing `diagnostics.work_dir`, `diagnostics.retention`, `NOTARIUS_WORK_DIR`, -and `NOTARIUS_DIAGNOSTICS_RETENTION` inputs remain supported for compatibility. -New configuration should use `workspace.directory` and -`workspace.diagnostics.retention` instead. - -For retention behavior and the physical diagnostics layout, see -[Operations](operations.md#retention). For the invocation-specific diagnostics -override, see [CLI Reference](cli.md#run). +The removed fields are `workspace.directory`, `workspace.resume.enabled`, +`workspace.debug.enabled`, `workspace.chunk_cache.mode`, +`workspace.chunk_cache.directory`, `workspace.diagnostics.enabled`, +`workspace.diagnostics.retention`, `diagnostics.work_dir`, and +`diagnostics.retention`. The removed environment variables are `NOTARIUS_WORKSPACE_DIR`, +`NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`, +`NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`, +`NOTARIUS_WORKSPACE_RESUME_ENABLED`, `NOTARIUS_WORKSPACE_DEBUG_ENABLED`, +`NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`, +`NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`, `NOTARIUS_WORK_DIR`, and +`NOTARIUS_DIAGNOSTICS_RETENTION`. The chunk-cache variables are replaced by +`NOTARIUS_CACHE_CHUNK_PLANS_MODE` and +`NOTARIUS_CACHE_CHUNK_PLANS_DIR`; the former shared directory has no direct +replacement. ## Validation @@ -417,8 +451,8 @@ Configuration validation checks: - positive global LLM concurrency; - supported stage-worker keys and an effective extract worker count in the inclusive range `1..concurrency.total_llm`; -- supported diagnostics retention and non-empty work directory; -- a supported chunk-cache mode and a chunk-cache directory without NUL bytes; +- non-empty output and debug directories; +- a supported chunk-cache mode and state-surface directories without NUL bytes; - stale removed fields such as `llm_profiles`. Pipeline resolution additionally checks: diff --git a/docs/development.md b/docs/development.md index 04b68b9..713b552 100644 --- a/docs/development.md +++ b/docs/development.md @@ -19,7 +19,7 @@ implemented component map. | Pipeline resolution or execution | [Pipeline Internals](internal/pipeline.md) | It documents profiles, references, validation, retries, checkpoints, and runner behavior. | | Production modules or validators | [Module Internals](internal/modules.md) | It documents implemented module contracts, capabilities, assets, and registration. | | LLM clients, prompts, schemas, profiles, or scheduling | [LLM Runtime](internal/llm.md) | It documents the transport boundary and Scriptorium integration. | -| Diagnostics, workspace state, resume, or debug artifacts | [Diagnostics Internals](internal/diagnostics.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. | +| Output, cache, resume, or debug artifacts | [Run State Internals](internal/state.md), [Operations](operations.md), and [Configuration](config.md) | These separate implementation details, operator behavior, and configuration contracts. | | CLI or user-visible configuration behavior | [CLI Reference](cli.md) and [Configuration](config.md) | These are the canonical user and operator references. | | External input formats, artifact schemas, or durable output files | [Integration Contracts](integrations/) | Integration documents define external and durable data contracts. | | Proposed or unimplemented behavior | [Roadmap](roadmap/) | Future work belongs only in roadmap documentation until implemented. | diff --git a/docs/internal/diagnostics.md b/docs/internal/diagnostics.md deleted file mode 100644 index 3eb27a1..0000000 --- a/docs/internal/diagnostics.md +++ /dev/null @@ -1,105 +0,0 @@ -# Diagnostics Internals - -`internal/core/diagnostics` provides the scoped writer and retention decision -used by `internal/cli`. The physical layout, artifact inventory, retention -semantics, failure inspection, and cleanup procedures are canonical in -[Operations](../operations.md#diagnostics-directory). Configuration fields and -defaults are canonical in [Configuration](../config.md#diagnostics). - -## Run Directory - -`NewRunDirectory` normalizes empty constructor inputs, creates the effective -diagnostics root when needed, and allocates a unique timestamp-based child -directory. It retries a bounded number of collisions before failing. The -resulting `RunDirectory` retains its creation time and retention mode for later -metadata and cleanup decisions. - -The package does not resolve workspace configuration. `internal/cli` derives -effective workspace settings first and passes the diagnostics root into the -constructor. - -## Scoped Writers - -Typed methods on `RunDirectory` write invocation metadata, redacted effective -configuration, resolved pipeline/reference data, checkpoint events, source data -when explicitly requested, manifests, reports, warnings, redacted chunk-plan -summaries, and error text. The -current filenames and their operator-facing contents are listed in -[Operations](../operations.md#diagnostics-directory). - -The chunk-plan summary records the effective mode, source and candidate -digests, requested module, lookup decision, materialization action, validation -decision, and publication decision. Its closed decision values make failures -and recoverable invalid records inspectable without serializing plan ranges, -annotations, source content, reference content, prompts, model responses, or -raw invalid-file bytes. Lookup reasons are derived only from the lookup status: -`stored chunk plan is valid`, `chunk plan not found`, `stored chunk plan is -invalid`, or `chunk plan lookup skipped`. Store-provided reasons and malformed -record details never enter this artifact. - -JSON methods indent their payload and append a newline. All artifact writes use -a temporary file in the target directory, apply the requested permissions, and -rename it into place. Artifact resolution accepts only a single relative base -name; absolute paths, separators, and paths escaping the run directory fail -before writing. - -## Redacted Configuration - -`WriteRedactedEffectiveConfig` accepts a `RedactedDiagnosticsPayload` provider -rather than a raw config value. `internal/core/config` implements that contract -by cloning effective config data and removing secret-shaped values before JSON -encoding. The diagnostics package therefore never needs configuration-specific -field knowledge. - -## Retention Coordination - -`ShouldRetainRunDirectory` is a pure decision over the effective retention mode, -run success, and warning presence. `ApplyRetention` uses that result to remove -only its own run directory. Unsupported modes retain data as a fail-safe, though -normal CLI execution rejects them during config validation. - -The meaning of each supported mode belongs in -[Operations](../operations.md#retention); this package implements that contract -without loading config or inspecting run artifacts. - -## CLI State Flow - -When diagnostics are enabled, `internal/cli` creates the run directory after -configuration loading and before pipeline resolution. It then writes artifacts -as state becomes available: invocation data, effective resolution data, -pipeline results, and the final report. This ordering permits later failures to -retain the context already established. - -Failures before construction have no `RunDirectory`. Later failures write an -error log, preserve any available partial manifest and chunk-plan summary, and -apply a failed-run -retention decision. A diagnostics write failure is itself a command failure so -the CLI does not report success after losing requested inspection data. - -When diagnostics are disabled, the CLI carries a nil run directory and the -shared `writeDiagnostics` helper turns writes into no-ops. User-facing errors -still go to stderr; that invocation behavior is documented in -[Operations](../operations.md#failures). - -## Package Guarantees - -- A `RunDirectory` writes and removes only within its allocated directory. -- JSON and error artifacts use atomic replacement. -- Nil receivers and invalid typed payloads return errors rather than panicking. -- Retention never removes a failed run and never targets the diagnostics root. -- Diagnostics models contain inspection metadata, not the durable output - contract. -- Checkpoint and debug serializers remain separate framework components. -- Secret-handling follows the invariant in - [Architecture](../policy/architecture.md#state-output-and-safety). - -## Tests To Inspect - -- `internal/core/diagnostics/run_dir_test.go`: allocation, artifact confinement, - atomic writes, retention, and failure behavior. -- `internal/core/diagnostics/artifacts_test.go`: stable artifact identifiers. -- `internal/core/config/redaction_test.go`: clone-and-redact payload behavior. -- `internal/core/workspace/settings_test.go`: effective diagnostics-root and - enablement handoff. -- `internal/cli/run_test.go`: creation timing, artifact sequencing, disabled - diagnostics, overrides, failures, and retention integration. diff --git a/docs/internal/llm.md b/docs/internal/llm.md index 93efef2..e9b8234 100644 --- a/docs/internal/llm.md +++ b/docs/internal/llm.md @@ -106,14 +106,13 @@ schemas remain package-owned. ## Debug And Redaction Boundaries The pipeline may wrap the client with a debug recorder that captures prepared -prompt/response material for an explicitly enabled debug run. Default -diagnostics and manifests receive identities, hashes, usage, and selected -profile summaries rather than prompt, source, reference, schema, or response -content. +prompt/response material for an explicitly requested debug run. Debug summaries +and manifests receive identities, hashes, usage, and selected profile summaries +rather than prompt, source, reference, schema, or response content. The Scriptorium error wrapper removes bearer credential values from surfaced provider errors; `RedactSecrets` and `ErrorWithSecretsRedacted` support known -secret values elsewhere in the runtime. Config diagnostics use a separate +secret values elsewhere in the runtime. Config summaries use a separate clone-and-redact path in `internal/core/config`. These mechanisms implement the security invariant in [Architecture](../policy/architecture.md#state-output-and-safety); operator diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 9b8651b..d3f34db 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -11,8 +11,8 @@ boundaries and dependency direction belong in `cmd/notarius` delegates to `internal/cli`, the production composition root. The CLI loads configuration, builds the production catalogs and runtime collaborators, invokes `internal/framework/pipeline`, and places the logical -output files returned by the runner. Diagnostics, checkpoints, and debug -recorders are optional side-channel collaborators supplied at this boundary. +output files returned by the runner. Cache and debug collaborators are supplied +at this boundary. Resolution produces a fixed ordered workflow and a sorted set of artifact lanes. Preparation constructs the complete module and validator set before the @@ -25,7 +25,7 @@ normalize continuations that may overlap across lanes. | Package | Implemented responsibility | | --- | --- | | `cmd/notarius` | Executable entry point and process exit delegation. | -| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, workspace collaborator setup, durable writes, and user-facing results. | +| `internal/cli` | Command parsing, config discovery, package-family registrar invocation, LLM client construction, reference materialization, state collaborator setup, durable writes, and user-facing results. | ## Core Packages @@ -33,9 +33,9 @@ normalize continuations that may overlap across lanes. | --- | --- | | `internal/core/artifacts` | Run-manifest and provenance models. | | `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. | -| `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. | +| `internal/core/debugbundle` | Explicit per-run debug-bundle allocation and redacted summary writing. | +| `internal/core/fileio` | Generic confined atomic file and JSON writes with caller-selected permissions. | | `internal/core/source` | Generic source documents, units, chunks, canonical references, validation, deterministic source digests, and independent metadata materialization. | -| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. | ## Framework Packages @@ -46,9 +46,9 @@ normalize continuations that may overlap across lanes. | `internal/framework/validate` | Shared validator decision and cardinality helpers. | | `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. | | `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. | -| `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. | +| `internal/framework/checkpoint` | Root-based checkpoint loading, recording, identity, and payload serialization. | | `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. | -| `internal/framework/debug` | Workspace-backed framework and LLM debug recording. | +| `internal/framework/debug` | Root-based framework and LLM debug recording. | Framework contracts provide typed artifact, provenance-wrapper, chunk-validator, serialized-validator, and @@ -122,14 +122,13 @@ Implementation details for all production extensions are in | Surface | Implemented owners | Internal purpose | | --- | --- | --- | | Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. | -| Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. | -| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable extract, merge, and normalize outcomes. | +| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes. | | Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. | -| Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. | +| Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. | -Physical layout, retention, recovery, and sensitive-data handling are defined +Physical layout, cleanup, recovery, and sensitive-data handling are defined in [Operations](../operations.md). Concrete stage modules receive recorder -interfaces and request data, not workspace paths. +interfaces and request data, not physical state roots. ## Focused Documentation @@ -139,5 +138,5 @@ interfaces and request data, not workspace paths. registration, and the contributor recipe for adding an extension. - [LLM Runtime](llm.md): structured completion contracts, Scriptorium adapter, assets, scheduling, profile recording, and redaction. -- [Diagnostics Internals](diagnostics.md): scoped writers, retention - coordination, CLI failure flow, and path safety. +- [Run State Internals](state.md): output, cache, debug collaborator + composition, and path safety. diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index 4e62f26..dcbb5b4 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -143,8 +143,8 @@ reference spanning the first selected unit through the last. `pipeline.RunOutput` carries the run manifest, accepted normalized serialized artifacts with lane and normalizer provenance, rejected results, warnings, checkpoint events, and logical files returned by the -output encoder. The CLI owns diagnostics and durable filesystem writes after the -runner returns. +output encoder. The CLI owns debug-summary and durable filesystem writes after +the runner returns. ## Execution Flow @@ -199,7 +199,7 @@ configuration, requested chunker, options, references, lanes, validators, or LLM profile do not prevent a source-digest hit. The manifest records both the currently requested chunker and the effective plan producer. Cache state and paths are configured and operated outside the runner; see -[Configuration](../config.md#workspace) and [Operations](../operations.md). +[Configuration](../config.md#state-surfaces) and [Operations](../operations.md). The extract job channel has the same capacity as the effective extract worker count, so dispatch applies backpressure. A fixed continuation executor prevents diff --git a/docs/internal/state.md b/docs/internal/state.md new file mode 100644 index 0000000..290af89 --- /dev/null +++ b/docs/internal/state.md @@ -0,0 +1,56 @@ +# Run State Internals + +This document describes the implementation collaborators behind output, cache, +and debug state. User-visible fields belong in [Configuration](../config.md), +and layouts and lifecycle belong in [Operations](../operations.md). + +## Composition + +`internal/cli` is the only physical-path composition root. It resolves the +effective configuration, selects exact roots, allocates requested debug bundles, +constructs cache collaborators, writes logical output files, and reports paths. +Pipeline modules receive interfaces and request data, never output, cache, or +debug roots. + +## Output And Cache + +The pipeline runner returns logical output files. The CLI places them beneath +the selected output root with confined, atomic writes. + +`internal/framework/chunkplan` owns source-addressed plan storage, validation, +and atomic publication. Its store is constructed only when the selected mode is +not `bypass`. + +`internal/framework/checkpoint` owns checkpoint identity, manifests, payload +codecs, loader, and recorder. The CLI constructs both loader and recorder only +for a `--resume` invocation. The serialized +`workspace_schema_version` identifiers are frozen wire-compatibility fields; +they do not describe a current public state surface. + +`internal/core/fileio` provides confined atomic file writes used by state +collaborators. The chunk-plan store retains its stronger entry validation. + +## Debug Bundles + +`internal/core/debugbundle` allocates an explicitly requested per-run bundle +with `summary/` and `trace/` roots. `SummaryWriter` persists redacted command, +resolution, run, warning, and failure artifacts. `internal/framework/debug` +implements the pipeline-facing trace recorder under the trace root. + +The CLI allocates a bundle before pipeline resolution and treats requested +summary or trace persistence failures as command failures. The pipeline's debug +boundaries redact sensitive metadata and credential-shaped bytes while allowing +application-owned trace material. Debug data is never a checkpoint source or +cache input. + +## Tests To Inspect + +- `internal/cli/state_surfaces_test.go`: debug allocation and configuration + boundaries. +- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures, + permissions, cleanup, and redaction. +- `internal/core/debugbundle/*_test.go`: bundle allocation and summary writes. +- `internal/framework/checkpoint/*_test.go`: checkpoint serialization and + reuse. +- `internal/framework/chunkplan/store_test.go`: plan envelope, confinement, + publication, and permissions. diff --git a/docs/operations.md b/docs/operations.md index 6509bc3..a219337 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -1,303 +1,170 @@ # Operations -This is the canonical reference for operating implemented Notarius runs. +This is the canonical guide to operating Notarius filesystem state. Command +syntax is in the [CLI reference](cli.md); field definitions and precedence are +in [Configuration](config.md). -## Normal Run +## State Model -A run reads one source file, resolves one configured pipeline, executes its -modules, writes durable output, and writes diagnostics when enabled. Start with -the [README quickstart](../README.md), then use the [CLI reference](cli.md) for -invocation options. +Notarius uses three independent filesystem surfaces: -For production, configure an application-owned workspace such as -`/var/lib/notarius` and ensure the Notarius process can create files below it. -For local development, prefer an ignored project-local workspace such as -`./.notarius/workspace`. See [Configuration](config.md#workspace) for workspace -fields. +- output is durable user data; +- cache is reconstructible chunk-plan and checkpoint state; and +- debug is explicitly requested inspection data. -## Output Directory +Choose separate roots and access controls for each surface. A normal run writes +durable output and may use the chunk-plan cache. It does not create checkpoint +or debug state unless its invocation includes `--resume` or `--debug`. -Durable output is written to: +## Output + +Durable logical files are written under: ```text // ``` -The output root and its invocation-specific override are defined in the -[CLI reference](cli.md#run). Output writes are atomic per file. The -[JSON output contract](integrations/json-output.md) defines the logical files, -paths, schemas, and media types inside each run directory. +Each output file is written atomically. Notarius never automatically removes +output. The [JSON output contract](integrations/json-output.md) owns the +logical file names, schemas, and media types inside a run directory. -## Diagnostics Directory - -Diagnostics are written under: - -```text -// -``` - -When a workspace directory is configured, diagnostics are written under -`/diagnostics//`. An invocation-specific override -changes only the diagnostics root, not the workspace root. Configuration and -environment controls are defined in [Configuration](config.md); the override -flag is defined in the [CLI reference](cli.md#run). - -Diagnostics can be disabled through configuration. When disabled, Notarius -does not create a diagnostics run directory or write diagnostics artifacts; -concise failures are still printed to stderr. - -Implemented diagnostics artifacts: - -- `invocation.json`: command metadata such as operation, config path, input - path, selected lanes, run ID, and pipeline digest when available. -- `effective-config.json`: resolved config without raw API keys. -- `resolved-pipeline.json`: resolved module bindings and pipeline digest. -- `resolved-references.json`: resolved reference provenance, including target - stage, lane ID when present, origin, digest, media type, byte size, and - binding source, without reference content. -- `checkpoint-events.json`: checkpoint steps that were reused or executed - during an explicit resume invocation. -- `chunk-plan.json`: redacted plan-cache lookup, validation, and publication - summary. It contains identifiers and decisions, never source units, plan - annotations, reference content, prompts, model responses, or invalid-file - bytes. -- `run-manifest.json`: the same run manifest written to durable output when it - is available, including top-level module metadata when present. -- `warnings.json`: warning list. -- `run-report.json`: counts, status, output path, diagnostics path, and run ID. -- `error.log`: failure message, written after diagnostics directory creation - when a run fails. +Remove an output run directory only after its consumer data is no longer +needed. This is data deletion, not cache cleanup. ## Chunk-Plan Cache -The chunk-plan cache is independent of the workspace and checkpoints. Its -configuration and selection precedence are defined in -[Configuration](config.md#workspace); the invocation override is documented in -the [CLI reference](cli.md#run). - -When no root is configured, a normal Linux user uses -`$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid absolute -path, or `$HOME/.cache/notarius/chunk-plans` when it is unset. A relative -`XDG_CACHE_HOME` is a configuration error. A configured -`workspace.chunk_cache.directory` is the root itself, not a parent to which -Notarius adds a suffix. - -Each source digest has one file: +Chunk plans are stored at: ```text //plan.json ``` -Directories are created with `0700` permissions and plan files with `0600`. -`auto` reuses a complete valid plan or regenerates an absent or invalid one; -`refresh` deliberately regenerates; `bypass` performs no cache I/O. A stored -plan is still validated and materialized against the current source before use, -and the current run's chunk validators always run. Invalid state is recoverable: -an `auto` run regenerates and atomically replaces it only after validation -succeeds. Delete an exact cache root or digest directory only when regeneration -cost is acceptable. +`auto` reuses a complete valid plan or regenerates missing or invalid state. +`refresh` regenerates and atomically replaces a plan after chunk validation. +`bypass` performs no plan-cache I/O and does not resolve or create the root. +Plan selection is source-addressed and independent of checkpoint and debug +roots. -The configured root is the cache trust boundary. An operator-supplied root path -may itself resolve through a symlink, but cache-owned digest directories and -plan files must be real directory and regular-file entries. Links or other -unexpected entry types are rejected rather than followed. +When its directory is empty in configuration, the root is +`/notarius/chunk-plans`. A configured directory is the exact +root; no suffix is appended. Directories and files created by the store use +`0700` and `0600` permissions on supported Unix systems. The configured root +is a trust boundary: do not share it among mutually untrusted users. -Publication uses atomic replacement. Concurrent readers observe a complete old -or new plan, and concurrent writers leave one complete valid winner; there is -no history, lock protocol, or rollback facility. Do not share a cache root -between mutually untrusted users because plans can contain source-derived -structure and annotations. +Remove an exact digest directory or the configured root only when accepting the +cost of recomputing plans and any chunk-stage work. Cache publication is atomic; +there is no history, locking, garbage collection, or rollback facility. -For a system-wide Linux deployment under a dedicated service account, configure -and provision a separate restrictive root such as: +For a Linux service account, provision a dedicated restrictive root such as: ```yaml -workspace: - chunk_cache: +cache: + chunk_plans: directory: /var/cache/notarius/chunk-plans ``` -`/var/cache/notarius/chunk-plans` is a recommended configured service root, not -the unprivileged default. The operator or package installer must create it with -restrictive service-account ownership and permissions before use. +## Checkpoint Cache -## Checkpoints +Checkpoint state is used only by an invocation with `--resume`. That invocation +loads compatible completed work and records checkpoints for work it executes. +Without `--resume`, Notarius neither resolves nor creates the checkpoint root, +and neither loads nor records checkpoints. -When checkpoint writing is enabled for a configured workspace, runs write -checkpoints under: +Checkpoints use the selected root and the existing identity hierarchy: ```text -/checkpoints//-/// +//-///... ``` -Each workflow step owns its own manifest and payload files. There is no -root-level checkpoint summary. Ordinary invocations execute the pipeline -normally and refresh checkpoints. An explicit resume invocation reuses valid -checkpoints and executes any missing, invalid, or incompatible step normally. -Configuration controls checkpoint writing, while the explicit resume option is -defined in the [Configuration](config.md#workspace) and -[CLI](cli.md#run) references. +An empty configured directory selects +`/notarius/checkpoints`. The root is exact when configured. +Created directories and files use `0700` and `0600` permissions on supported +Unix systems. -Checkpoints do not include raw prompts, raw reference contents, raw LLM request -payloads, or debug traces. They can still contain source text, intermediate -extracted content, rejected outputs, metadata, warnings, and content digests. -Treat checkpoint directories as sensitive local state. +Checkpoint payloads can contain source text, intermediate artifacts, metadata, +warnings, and content digests. Treat them as sensitive derived application +data. Compatible files from a former checkpoint root remain reusable when +`cache.checkpoints.directory` names that exact existing root. They are not +moved, migrated, or deleted automatically. The frozen serialized identifier +`workspace_schema_version` remains part of checkpoint compatibility; it is not +a configuration setting. -A checkpoint is reused only when its stored status, dependencies, payloads, and -digests match the current invocation. Changes to input bytes, the resolved -pipeline, selected lanes, the runtime LLM profile override, or bound reference -content invalidate reuse. The resolved pipeline identity includes effective -default and explicitly overridden validator chains, so adding, removing, -reordering, or reconfiguring a validator invalidates checkpoints even when the -pipeline profile itself is unchanged. +For a Linux service account, independently provision: -Typed artifact checkpoints additionally record codec-owned bytes, artifact -kind, schema ID and version, exact schema digest, and media type. A missing or -mismatched codec identity, or bytes the current codec cannot decode, is reported -as a checkpoint reuse miss. The affected operation executes normally and, when -checkpoint writing is enabled, replaces the incompatible checkpoint. +```yaml +cache: + checkpoints: + directory: /var/cache/notarius/checkpoints +``` -Current checkpoint manifests use workspace schema `notarius.workspace.v2`. -Manifests written with `notarius.workspace.v1` are incompatible because their -chunk provenance has an older shape. On the first explicit resume after an -upgrade, each affected checkpoint is treated as a reuse miss and its workflow -step executes normally. The compatibility check does not migrate or delete the -v1 files; when checkpoint writing is enabled, normal execution refreshes the -affected checkpoint files in the current schema. +Remove an exact checkpoint identity directory or the configured root only when +recomputation is acceptable. -Runs do not reuse checkpoints unless explicitly requested. Without reuse, the -workflow executes normally and refreshes checkpoint files when checkpointing is -enabled. +## Debug Bundles -## Debug - -When debug recording is enabled for a configured workspace, runs write debug -artifacts under: +Only `notarius run --debug` enables debug collection. The selected root contains +one retained bundle per invocation: ```text -/debug// +// + summary/ + trace/ ``` -Debug output is per invocation. It is independent of checkpointing and is not -used for resume. Enabling debug does not write checkpoints, and enabling resume -checkpointing does not write debug output. +`summary/` contains redacted invocation, effective-configuration, resolved +pipeline and reference provenance, checkpoint and chunk-plan decisions, run +manifest, warnings, report, and any available error text. It excludes raw +source, references, annotations, prompts, model responses, credentials, and +malformed cache bytes. -Debug artifacts include inputs and outputs for source, chunk, extract, merge, -normalize, and output work, structured LLM request and response data, validator -requests and results, timing, and retry attempt metadata. LLM calls made inside -a module retry write `prompt-000N.json`, `response-000N.json`, and -`response-content-000N.*` files under that attempt directory and are linked -from its `llm_calls` array. Validator calls use separate attempt scopes under -`validate/` and are not duplicated into the module attempt. Prompt content is -written inline in the prompt artifact. The response metadata and body use the -paired files described above; the body is pretty-printed JSON when possible -and raw text otherwise. Retrying stages use these stable module-attempt paths: +`trace/` contains application-owned execution detail, including source and +stage material, plans, chunks, validator attempts, prompts, model responses, +timing, and serialized artifacts. It may retain application data omitted from +output. Credentials, credential-shaped values, sensitive metadata, unrelated +environment values, and unrelated filesystem content are not captured. -```text -chunk/attempt-.json -extract//chunk-/attempt-.json -merge//attempt-.json -merge//attempt-/prompt-.json -merge//attempt-/response-.json -merge//attempt-/response-content-. +Bundles inherit the sensitivity of the application data they capture. Their +additional risk comes from copying and aggregating that data, so restrict +access, avoid shared roots between untrusted users, and define retention outside +Notarius. Created bundle directories use `0700` and files use `0600` on +supported Unix systems. -normalize//attempt-.json -normalize//attempt-/prompt-.json -normalize//attempt-/response-.json -normalize//attempt-/response-content-. -``` +Notarius never automatically deletes a requested bundle. If allocation +succeeds, its path is reported on success and failure. A requested summary or +trace write failure makes the command fail, preserving whatever bundle data was +already written for inspection. -Every executed chunk, extract, merge, and normalize attempt has one terminal -envelope recording acceptance, validator rejection, or a module, validator, -candidate-serialization, or final-serialization error as applicable. It -includes attempt-local warnings and any available candidate or rejection. A -failure before a candidate exists has no candidate payload. If the envelope -cannot be persisted, the run does not retry that module attempt and reports the -debug failure together with any primary attempt error. +## Failures And Warnings -Chunk-plan candidates, materialized chunks, annotations, and chunk-attempt -details appear only in these opt-in debug artifacts. They are intentionally not -included in normal manifests or the `chunk-plan.json` diagnostics summary. - -Checkpoint-reused extract, merge, and normalize work retains the stage-level -input and output artifacts but has no retry-attempt artifacts -because no module attempt executed. Debug artifacts may contain source -material, reference material, prompt inputs, model outputs, and other sensitive -data. Typed artifact -envelopes include domain-neutral codec identity, redacted metadata and content, -and digests of the stable codec bytes. API keys are not written, and obvious -credential-shaped values and sensitive map keys are redacted, but debug -directories should still be protected as sensitive local state. - -## Retention - -Diagnostics retention uses the effective mode selected through configuration; -see [Configuration](config.md#diagnostics) for the fields, environment -overrides, precedence, and default. - -- `auto`: keep failed runs and successful runs with warnings; remove successful - warning-free runs. -- `always`: keep every diagnostics run directory. -- `never`: remove successful run directories; failed runs are still retained. - -## Failures - -Failures before diagnostics directory creation, such as a missing config file or -an unusable diagnostics work directory, are printed to stderr and may not have a -diagnostics run directory. - -Failures after diagnostics directory creation are printed to stderr and written -to `error.log`. Depending on where the failure occurred, the directory may also -contain artifacts written before the failure. - -If durable output writing fails after the pipeline completes, diagnostics are -retained for inspection. - -## Warnings - -A successful run with warnings exits with code `0`, prints a warning count to -stderr, and writes warnings to durable output and diagnostics when retained. - -The [JSON output contract](integrations/json-output.md) defines durable warning -and validation-status fields. +Failures before debug allocation are reported on stderr without a bundle. +Failures after allocation report the bundle path on stderr and write `error.log` +when that summary write succeeds. An output-write failure leaves the allocated +bundle in place. A successful run with warnings exits `0`, reports a warning +count on stderr, and records warnings in durable output and any requested debug +summary. ## Cleanup -It is safe to remove specific old run directories after their output and -diagnostics are no longer needed: +Use exact paths for manual cleanup. Examples: ```sh -rm -rf /tmp/notarius/run-1234567890 rm -rf ./notarius-output/run-1234567890 +rm -rf /var/cache/notarius/chunk-plans/0123abcd +rm -rf /var/cache/notarius/checkpoints/pipeline/input-0123/pipeline-4567/identity-89ab +rm -rf ./notarius-debug/run-1234567890 ``` -Workspace checkpoint and debug directories can also be removed when no longer -needed. Remove exact identity or run directories, for example: - -```sh -rm -rf /var/lib/notarius/checkpoints/dnd-session/seriatim-abcdef123456/7890abcd1234/identityabcd1234 -rm -rf /var/lib/notarius/debug/run-1234567890 -``` - -Chunk-plan cache entries can likewise be removed by exact digest directory or -configured root. Removal is recoverable, but the next non-bypass run may need -to regenerate plans and repeat any chunk-stage LLM work. - -Use exact run-directory paths. Avoid broad cleanup commands against parent -directories unless they are part of your own operational policy. +Avoid broad recursive cleanup against a parent root unless it is an explicit +operator policy. Output deletion is permanent user-data loss. Cache deletion is +recoverable but can repeat expensive work. Debug deletion removes troubleshooting +evidence and any retained application-data copy. ## Operational Limits Provider retries and timeouts are handled by Scriptorium according to the selected execution profile. Pipeline module retry settings are defined in -[Configuration](config.md#module-bindings). There is no separate CLI retry -command. - -Extract worker concurrency and actual provider-call concurrency are separate -limits. Their configuration, defaults, and validation are defined in -[Configuration](config.md#concurrency). Cancellation stops undispatched extract -work; already started work is allowed to finish or observe cancellation before -the run reports failure. - -Notarius writes local files only. Remote storage and archive management are not -part of the implemented CLI. +[Configuration](config.md#module-bindings). Extract worker concurrency and +actual provider-call concurrency are separate limits; their fields and +validation are defined in [Configuration](config.md#concurrency). Notarius +writes local files only; remote storage and archive management are outside the +implemented CLI. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index 656e616..a5a2753 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -114,8 +114,8 @@ plan, while the current run still applies its configured chunk validators to the materialized chunks. The framework owns orchestration and handoff provenance. Modules return logical -results and warnings; they do not own CLI reporting, workspace paths, durable -file placement, checkpoints, or diagnostics. +results and warnings; they do not own CLI reporting, physical output, cache, or +debug roots, durable file placement, or checkpoint and debug lifecycle. After pipeline-wide chunking, extraction uses bounded framework concurrency. One run-wide worker pool receives chunk-scoped lane jobs in deterministic @@ -188,31 +188,32 @@ record identities and summaries rather than secret or large payload content. ## State, Output, And Safety -Durable output, diagnostics, checkpoints, and debug artifacts are separate -surfaces with separate ownership: +Notarius exposes three filesystem surfaces with independent roots and +lifecycle: -- output modules define logical durable output; the application boundary owns - filesystem placement; -- diagnostics provide redacted run inspection and are not the durable output - contract; -- checkpoints support validated stage reuse and are not diagnostics; -- debug artifacts are opt-in inspection data and may contain sensitive source, - prompt, reference, and model-output content. +- output is durable user data; output modules define logical files and the CLI + owns their placement; +- cache is reconstructible state, with separate chunk-plan and checkpoint + families; and +- debug is explicitly requested inspection data, combining a redacted summary + with a detailed trace. -Chunk-plan cache state is an additional independent surface. It is keyed only -by canonical source digest, is not rooted under `workspace.directory`, and is -not a checkpoint or a diagnostic. A cache record is atomically replaced as one -complete plan envelope; it has no history, locking, or rollback interface. -Invalid records are recoverable cache misses rather than pipeline state. +Chunk plans are keyed only by canonical source digest. Checkpoints are used only +for an invocation that explicitly requests resume. Debug is never a cache input +and is never created without an explicit request. Pipeline modules receive +collaborator interfaces and never physical roots. -Writes of durable state are atomic where practical. Paths for writes, moves, -overwrites, and deletion must be narrow and explicit. Cleanup that can lose data -is opt-in. +Writes are atomic where practical. Paths for writes, moves, overwrites, and +deletion must be narrow and explicit. Notarius never automatically deletes +output or requested debug bundles; cache cleanup is explicit and recoverable. -Secrets must not appear in errors, logs, diagnostics, manifests, -documentation, examples, or redacted configuration. Default logs and -diagnostics must not include large source, prompt, reference, or artifact -payloads. +Secrets must not appear in errors, logs, output, cache, debug summaries, +traces, manifests, documentation, examples, or redacted configuration. Debug +collection is allowlisted to application-owned payloads and must not capture +unrelated process environment values or filesystem content. Trace data may +contain application data and therefore inherits its sensitivity; operators own +access controls and retention. Physical layout and operation are defined in +[Operations](../operations.md). ## Architectural Non-Goals diff --git a/docs/policy/documentation.md b/docs/policy/documentation.md index 467871c..ad9818a 100644 --- a/docs/policy/documentation.md +++ b/docs/policy/documentation.md @@ -63,7 +63,7 @@ secret values. | Documentation organization | `docs/policy/documentation.md` | Documentation ownership, audience boundaries, maintenance rules, and ADR/document lifecycle. | Application architecture or product behavior. | | CLI contract | `docs/cli.md` | Commands, arguments, flags, invocation semantics, and exit codes. | End-to-end operating procedures, configuration field definitions, runtime filesystem layout, module implementation details. | | Configuration contract | `docs/config.md` | Discovery and precedence, file schema, fields, defaults, environment overrides, validation rules, and user-selectable module or validator keys. | Complete example files, CLI syntax, runtime state lifecycle, module implementation details. | -| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, diagnostics use, retention, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. | +| Operations | `docs/operations.md` | Runtime workflows, physical filesystem and state layout, output, cache, and debug handling, resume, cleanup, permissions, recovery, and operational limits. | CLI flag syntax, configuration field definitions, logical output schemas, implementation mechanics. | | Public HTTP contract, if introduced | `docs/api.md` | Routes, authentication, media types, request and response schemas, status codes, pagination, caching, idempotency, rate limits, and HTTP retry semantics. | Client walkthroughs, upstream or downstream integration internals, implementation detail. | | Consumer guidance, if a public package or API is introduced | `docs/consumers/` | Task-oriented use of the public interface, minimal client examples, and consumer responsibilities. | HTTP wire semantics, external protocol contracts, internal implementation detail. | | External and durable integration contracts | `docs/integrations/` | External file formats and protocols, upstream and downstream contracts, logical output bundle paths and schemas, media types, and compatibility behavior. | Physical runtime placement and lifecycle, internal transformations, CLI syntax, configuration defaults. | diff --git a/docs/roadmap/adr0006.md b/docs/roadmap/adr0006.md deleted file mode 100644 index ef567f5..0000000 --- a/docs/roadmap/adr0006.md +++ /dev/null @@ -1,334 +0,0 @@ -# ADR-0006 Feature Roadmap - -This roadmap defines the intended end state for -[ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is -not implemented. - -## Intent - -Notarius should communicate filesystem behavior in terms of why data exists: -durable output, reconstructible cache, or explicitly requested debug material. -The public configuration and CLI should not expose internal distinctions such -as workspace state or a second diagnostics product. - -The implementation may retain focused writers, serializers, and security -boundaries. This roadmap changes their public composition and lifecycle rather -than requiring every kind of data to share one package or physical root. - -## Target Configuration - -The configuration schema advances to version 3 and uses these top-level -surfaces: - -```yaml -version: 3 - -output: - directory: ./notarius-output - -cache: - chunk_plans: - directory: "" - mode: auto - checkpoints: - directory: "" - -debug: - directory: ./notarius-debug -``` - -An empty cache directory selects that cache family's platform-appropriate -per-user default. A configured directory is the exact root; Notarius does not -append the cache-family name to it. - -The `workspace` and `diagnostics` top-level configuration surfaces do not exist -in version 3. In particular, version 3 has no persistent debug enablement, -diagnostics enablement or retention, checkpoint enablement, or shared workspace -directory. - -### Output configuration - -`output.directory` selects the durable output root. Precedence is the explicit -`--output-dir` flag, `NOTARIUS_OUTPUT_DIR`, the file value, then -`./notarius-output`. - -The pipeline's existing `pipelines..output` module binding remains separate: -it selects the output encoder, while top-level `output.directory` selects where -the CLI places that encoder's logical files. - -### Chunk-plan cache configuration - -`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh` with the -ADR-0005 semantics. Precedence is the explicit `--chunk_cache` flag, -`NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then `auto`. - -`cache.chunk_plans.directory` selects the exact canonical plan root. Precedence -is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, the file value, then -`/notarius/chunk-plans`. On Linux, the default therefore uses a -valid absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache` when that -variable is unset. A relative `XDG_CACHE_HOME` remains an error. - -The recommended system-service value remains -`/var/cache/notarius/chunk-plans`. ADR-0005 plan identities, envelopes, -permissions, atomic publication, validation, provenance, and cleanup semantics -do not change. - -### Checkpoint cache configuration - -`cache.checkpoints.directory` selects the exact checkpoint root. Precedence is -`NOTARIUS_CACHE_CHECKPOINTS_DIR`, the file value, then -`/notarius/checkpoints`. - -For a system-wide Linux deployment under a dedicated service account, the -recommended configured root is `/var/cache/notarius/checkpoints`, provisioned -with restrictive service-account ownership independently from the chunk-plan -root. - -Checkpoint directories and files use restrictive permissions because they may -contain source text and intermediate artifacts. They inherit the sensitivity of -the data captured in them. Their existing compatibility validation, pipeline -identity, source and lane payloads, and downstream dependency fingerprints -remain intact. Checkpoints do not store or select chunk results. - -There is no `cache.checkpoints.enabled` setting. Checkpoint I/O is controlled by -the invocation's `--resume` flag. - -### Debug configuration - -`debug.directory` selects the debug root but does not enable debug output. -Precedence is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then -`./notarius-debug`. - -There is no environment or file setting that enables debug. This prevents an -ambient production configuration from silently recording potentially sensitive -trace data and creating additional retained copies of application data. - -## Target CLI - -The run command retains `--output-dir`, `--chunk_cache`, and `--resume`, and -adds: - -- `--debug`: enable the complete debug bundle for this invocation; -- `--debug-dir `: override its root; valid only with `--debug`. - -`--diagnostics-dir` is removed. There is no `--workspace`, `--cache-dir`, or -separate diagnostics flag. - -`--resume` means “reuse compatible checkpoints when present and record -checkpoint state for work executed by this invocation.” On a first invocation -with no compatible state, it executes normally and creates checkpoints. A run -without `--resume` neither loads nor records checkpoints and does not resolve or -create the checkpoint root. - -The CLI reports the durable output directory on success. When debug is enabled, -it also reports the allocated debug bundle path on success and includes that -path in failure reporting once allocation has succeeded. - -Malformed values from every supplied configuration source remain errors even -when a higher-precedence source would otherwise override them. - -## Filesystem Layout - -### Output - -Durable logical files retain the current per-run layout: - -```text -//... -``` - -Output creation remains atomic per file. The maintained logical JSON output -contract, including manifests, warnings, rejections, and normalized artifacts, -does not change solely because of this state-model refactor. - -### Chunk plans - -The existing ADR-0005 layout remains: - -```text -//plan.json -``` - -Chunk-plan state remains independent of checkpoint and debug roots. - -### Checkpoints - -The current identity hierarchy moves beneath the independently selected -checkpoint root: - -```text -//-///... -``` - -The configured root replaces the former implicit -`/checkpoints` prefix. Existing manifests and payload -schemas remain compatible when pointed at their prior physical checkpoint root. - -### Debug bundle - -One explicitly requested bundle is allocated before pipeline resolution at: - -```text -// - summary/ - trace/ -``` - -`summary/` contains the redacted artifacts currently associated with -diagnostics: invocation metadata, redacted effective configuration, resolved -pipeline and reference provenance, checkpoint decisions, chunk-plan decisions, -run manifest, warnings, run report, and error text when available. - -`trace/` contains the existing deep debug material: source and stage inputs and -outputs, plans and materialized chunks, annotations, validator attempts, -prompts, model responses, retry timing, and intermediate serialized artifacts. - -Summary data never contains raw source, references, annotations, prompts, model -responses, credentials, or malformed cache bytes. Trace data may contain all -of those except credentials, which remain redacted. A debug bundle inherits the -sensitivity of the most sensitive data captured in it; it does not constitute a -separate, intrinsically higher sensitivity class. Because it aggregates and -retains an additional copy of application data, its root and files use -restrictive permissions by default. - -Trace data may preserve source or reference content that the durable output -summarized, rejected, or intentionally omitted. This can make the bundle more -sensitive than the output alone, but not more sensitive than the complete set -of application data supplied to or produced by the run. Restrictive defaults -are therefore precautionary handling for data of unknown sensitivity, not a -classification of debug material as uniquely sensitive. - -Debug persistence failures are command failures because the operator explicitly -requested the bundle. No debug directory is created without `--debug`. A -requested bundle is never automatically deleted based on success, warnings, or -failure; cleanup is explicit. - -## Runtime Ownership - -The CLI remains the composition root for physical paths and filesystem-backed -collaborators. Pipeline modules continue to return logical data and never -receive output, cache, or debug roots. - -The framework may retain separate interfaces for checkpoint recording and -loading, chunk-plan storage, redacted summary recording, and deep trace -recording. The single debug facility may compose multiple internal recorders; -unification does not require weakening redaction or passing raw trace data -through summary models. - -The public `workspace` type and terminology disappear from configuration, -invocation messages, debug metadata, and operator documentation. The standalone -diagnostics directory and retention surface also disappear. Domain contracts -that use “diagnostic” in another sense, such as a validator-provided diagnostic -artifact path, are not renamed solely by this decision. Generic confined-path -and atomic-write helpers may remain in an internal package but must not impose a -workspace abstraction on callers. - -## Security and Lifecycle - -- Output is durable user data. Notarius never automatically deletes it. -- Chunk plans and checkpoints are reconstructible cache. Exact entries or roots - may be removed, with the documented cost of recomputation. -- Debug is explicit inspection data. It is off by default, retained when - requested, and removed only by an explicit operator action. -- Checkpoints and debug bundles inherit the sensitivity of the application data - they capture. Their security concern is the additional retained copy and, for - debug bundles, aggregation of that data—not an intrinsically higher - sensitivity classification. -- Operators are responsible for choosing debug locations, access controls, and - retention appropriate to the data processed by the run. Documentation must - explain that a bundle can retain material omitted from the durable output. -- Cache and debug roots are separate trust boundaries and must not be shared - among mutually untrusted users. -- Credentials are excluded from output, cache, summaries, traces, logs, errors, - examples, and redacted configuration. -- Debug collection is allowlisted to application-owned data. It must not capture - unrelated process environment values, host secrets, or arbitrary filesystem - content merely because they are available to the process. -- Default runs may create output and use the default chunk-plan cache, but do - not create checkpoint or debug state unless their invocation flags request - those surfaces. - -## Compatibility Policy - -Version 3 is a deliberate configuration break. Version 2 files are rejected -with an actionable message directing operators to the migration documentation; -Notarius does not retain parallel legacy field parsing indefinitely. - -The version 2 migration is: - -- `workspace.chunk_cache.mode` -> `cache.chunk_plans.mode`; -- `workspace.chunk_cache.directory` -> `cache.chunk_plans.directory`; -- `/checkpoints` -> `cache.checkpoints.directory`, when - existing checkpoint reuse is desired; -- `--diagnostics-dir`, diagnostics fields, workspace diagnostics fields, and - their environment variables -> removal or the new debug controls; -- `workspace.debug.enabled` -> removal; invoke with `--debug`; -- `/debug` -> `debug.directory`, if the same parent is - desired; and -- `workspace.resume.enabled` -> removal; invoke with `--resume`. - -Environment migration is: - -- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE` -> - `NOTARIUS_CACHE_CHUNK_PLANS_MODE`; -- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` -> - `NOTARIUS_CACHE_CHUNK_PLANS_DIR`; -- `NOTARIUS_WORKSPACE_DIR` -> no direct replacement; configure the applicable - output, checkpoint-cache, and debug roots separately; -- `NOTARIUS_WORKSPACE_RESUME_ENABLED` and - `NOTARIUS_WORKSPACE_DEBUG_ENABLED` -> removal in favor of invocation flags; - and -- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`, - `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`, `NOTARIUS_WORK_DIR`, and - `NOTARIUS_DIAGNOSTICS_RETENTION` -> removal with the standalone diagnostics - surface. - -Existing chunk-plan files are reused without migration when the new directory -resolves to the same root. Existing checkpoint files are reusable when -`cache.checkpoints.directory` names the former checkpoint root and their normal -identity and schema checks pass. Existing diagnostics and debug directories are -not moved, merged, or deleted automatically. - -The durable output schema and ADR-0005 cache envelope schema are not versioned -solely because configuration advances to version 3. - -## Documentation Outcomes - -When implemented: - -- the CLI reference owns all flags and invocation semantics; -- configuration owns version 3 fields, environment variables, defaults, and - precedence; -- operations owns physical layouts, permissions, cleanup, failure inspection, - and per-user and system-service cache guidance; -- the integration contract continues to own durable logical output; -- internal pipeline and component documents explain collaborator composition - without reintroducing public workspace terminology; and -- migration documentation provides one complete version 2 to version 3 example. - -Current-behavior documentation must not describe this target before the -corresponding behavior is implemented. - -## Non-Goals - -This feature does not introduce: - -- remote output, cache, or debug storage; -- cache garbage collection, quotas, archival, history, or rollback; -- automatic debug retention or upload; -- a daemon logging, metrics, or telemetry system; -- a change to pipeline topology, artifact schemas, chunk-plan identity, or - module contracts; or -- a requirement that output, different cache families, and debug share one - physical parent directory. - -## Completion Outcomes - -The feature is complete when ordinary runs expose only durable output and -canonical chunk-plan cache behavior; checkpoint I/O occurs only under -`--resume`; one `--debug` invocation produces a complete retained summary and -trace bundle; no public workspace or diagnostics configuration remains; version -2 migration failures are actionable; existing compatible chunk plans and -checkpoints remain reusable at explicitly selected roots; and all configuration, -CLI, filesystem, security, compatibility, integration, and repository-wide -tests pass. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 752234a..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,657 +0,0 @@ -# ADR-0006 Staged Implementation Plan - -This document is the executable implementation plan for the target state in -[ADR-0006 Feature Roadmap](adr0006.md), governed by -[ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is -not implemented. - -The audience is an LLM coding agent. Implement the stages in order. Each stage -is scoped to finish with a compiling, tested repository and may be assigned as -one implementation prompt. - -## Execution Rules - -Before every stage: - -1. read [Development](../development.md), both documents under `docs/policy/`, - ADR-0006, the feature roadmap, this plan, and every stage-specific document - named below; -2. inspect `git status` and preserve user changes; -3. inspect the focused package contracts and tests before editing; and -4. confirm that all earlier stages are complete. - -During every stage: - -- implement only that stage and prerequisites discovered to be inseparable; -- keep physical path selection in the CLI composition root and keep pipeline - modules independent of output, cache, and debug roots; -- preserve checkpoint identity, chunk-plan identity, durable output schemas, - deterministic ordering, cancellation, redaction, confined paths, and atomic - writes unless this plan expressly changes them; -- treat debug persistence failures as command failures once debug is requested; -- add or update focused tests with every behavior change; -- keep unimplemented behavior under `docs/roadmap/` until the final - documentation stage; and -- do not begin the next stage after meeting the current exit criteria. - -At the end of every stage: - -1. run the focused tests listed for that stage; -2. run `go test ./...`; -3. run `go vet ./...`; -4. run `go build ./cmd/notarius`; -5. run `git diff --check`; and -6. mark the stage complete in this document only after all checks pass. - -If repository reality conflicts with this plan, stop and update the plan in the -same change before implementing a materially different design. Do not silently -invent a new public contract. - -## Fixed Decisions - -The following choices are settled for this implementation. - -### Public state model - -- The only public filesystem surfaces are output, cache, and debug. -- Output is durable user data. Cache is reconstructible. Debug is explicitly - requested inspection data and is not a cache input. -- `workspace` and standalone `diagnostics` disappear from configuration, CLI - flags, operator messages, debug metadata, package names exposed to callers, - and current-behavior documentation. -- Domain uses of the word “diagnostic,” including validator diagnostic artifact - paths, are unchanged when they do not describe the removed filesystem - surface. - -### Configuration version 3 - -- `SupportedFileConfigVersion` becomes `3`. Version 3 has these runtime types: - - ```go - type OutputConfig struct { - Directory string `json:"directory"` - } - - type CacheConfig struct { - ChunkPlans ChunkPlanCacheConfig `json:"chunk_plans"` - Checkpoints CheckpointCacheConfig `json:"checkpoints"` - } - - type ChunkPlanCacheConfig struct { - Directory string `json:"directory,omitempty"` - Mode pipeline.ChunkCacheMode `json:"mode"` - } - - type CheckpointCacheConfig struct { - Directory string `json:"directory,omitempty"` - } - - type DebugConfig struct { - Directory string `json:"directory"` - } - ``` - - `Config` contains `Output`, `Cache`, and `Debug` fields and no `Workspace` or - `Diagnostics` field. File-config types mirror the YAML shape in the feature - roadmap. -- Defaults are `./notarius-output`, chunk-plan mode `auto`, empty cache - directories meaning per-user defaults, and `./notarius-debug`. -- There is no persistent checkpoint enablement, resume enablement, debug - enablement, diagnostics enablement, or retention setting. -- Output-directory precedence is explicit `--output-dir`, - `NOTARIUS_OUTPUT_DIR`, file, then `./notarius-output`. -- Chunk-plan mode precedence is explicit `--chunk_cache`, - `NOTARIUS_CACHE_CHUNK_PLANS_MODE`, file, then `auto`. -- Chunk-plan root precedence is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, file, then - `/notarius/chunk-plans`. -- Checkpoint root precedence is `NOTARIUS_CACHE_CHECKPOINTS_DIR`, file, then - `/notarius/checkpoints`. -- Debug-root precedence is explicit `--debug-dir`, `NOTARIUS_DEBUG_DIR`, file, - then `./notarius-debug`. A directory value never enables debug. -- A configured cache directory names the exact family root. No family suffix is - appended. Explicit relative paths retain the current CLI convention and are - interpreted relative to the process working directory. -- Empty cache directory fields in a version 3 file select their per-user - defaults. A supplied environment directory override and an explicit CLI - directory flag must be non-empty after trimming. Empty output or debug file - values and malformed modes are errors. -- Every supplied file, environment, and CLI value is validated even when a - higher-precedence value wins. Apply CLI overrides only after file and - environment parsing has succeeded. -- Parse the YAML version header before decoding the strict version 3 schema so - a version 2 file receives the intentional migration error rather than an - incidental unknown-field error. Continue to use `KnownFields(true)` for the - supported schema. - -### Cache families - -- ADR-0005 behavior and the chunk-plan envelope and path layout remain - byte-compatible. Move only configuration ownership and the default-root - helper; do not change lookup, validation, publication, provenance, or - materialization behavior. -- Chunk-plan `bypass` does not resolve or create its root. `auto` and `refresh` - resolve it lazily when the run constructs the store. -- Checkpoint I/O exists only for an invocation with `--resume`. Without that - flag, the CLI does not resolve the checkpoint root, compute a checkpoint - identity, construct a checkpoint store, load a checkpoint, or record one. -- With `--resume`, the CLI constructs both loader and recorder. Compatible - state is reused; missing or incompatible state executes normally and writes - replacement checkpoints. -- Checkpoint identities, relative paths, manifests, payloads, compatibility - rules, and dependency fingerprints remain unchanged. Existing files are - reusable when the new root points at the former - `/checkpoints` directory. -- Preserve the existing checkpoint JSON field - `workspace_schema_version` and values `notarius.workspace.v2` and - `notarius.workspace.v1`. They are frozen wire compatibility identifiers, not - public configuration or package terminology. Rename internal Go identifiers - only if doing so cannot alter serialized bytes or compatibility messages. -- The recommended Linux service roots are - `/var/cache/notarius/chunk-plans` and `/var/cache/notarius/checkpoints`, - provisioned independently for the dedicated service account. - -### Debug bundle - -- `--debug` is the only debug enablement control. `--debug-dir ` is valid - only when `--debug` is present; misuse is a CLI syntax error with exit code - `2`. -- One requested bundle is allocated before pipeline resolution at - `//`, with `summary/` and `trace/` children. -- Bundle directories are created with mode `0700` and files with mode `0600`. - Do not silently change permissions on a pre-existing operator-owned debug - parent; enforce restrictive permissions on directories and files created for - the bundle. -- `summary/` contains the current redacted diagnostics artifacts, using their - current logical filenames where applicable. `trace/` contains the current - deep debug paths. There is no retention mode or automatic deletion. -- The summary run report uses `debug_path`, not `diagnostics_path`. Invocation - and effective-config payloads use debug/state terminology and contain no - obsolete public workspace settings. -- The redacted summary excludes raw source, references, annotations, prompts, - model responses, credentials, and malformed cache bytes. Trace data may - contain application data but never credentials. -- Debug collection records only explicit application-owned payloads. It does - not dump the process environment, inspect unrelated files, or capture host - secrets. Existing sensitive-key and credential-shaped-value redaction stays - in force at every serialization boundary. -- A bundle inherits the sensitivity of its captured application data. The - additional operational risks are copying, aggregation, and retention, not an - intrinsically higher sensitivity class. -- If any requested summary or trace write fails, the command exits `1`. When a - primary operation and a debug write both fail, report both errors on stderr - without recursively attempting the same failed write. -- On success, stdout reports the durable output directory and, when enabled, - the debug bundle path. Once allocation succeeds, failure output also names - the bundle path. - -### Compatibility and documentation - -- Version 2 configuration is rejected; no legacy fields or environment aliases - remain after the implementation. The error directs the operator to the - version 2-to-3 migration section in `docs/config.md`. -- Existing chunk-plan and checkpoint state is never moved or deleted - automatically. Existing diagnostics and debug directories are likewise left - untouched. -- The durable output contract and chunk-plan envelope schema are not versioned - because of this refactor. -- `docs/config.md` owns the complete version 2-to-3 configuration and - environment migration example. `docs/operations.md` owns physical state reuse - and cleanup guidance. Other documents link to those owners rather than - duplicating volatile details. - -## Planned Internal Boundaries - -Use these package responsibilities unless an existing collision requires the -smallest obvious naming adjustment. - -- `internal/core/fileio` owns generic confined relative-path construction and - atomic JSON/byte writes with caller-selected directory and file modes. Its - errors say “file” or “artifact,” never “workspace.” It must not replace the - chunk-plan store's stronger `os.Root`-based entry protections. -- `internal/framework/checkpoint` owns checkpoint identity, manifests, payload - codecs, filesystem loader, and filesystem recorder. Constructors accept an - exact checkpoint root and an identity; they do not accept a workspace or a - general state settings object. -- `internal/framework/chunkplan` continues to own durable chunk-plan storage. - Per-user cache-root default helpers belong with configuration/path resolution, - not in a workspace package. -- `internal/core/debugbundle` owns allocation of a per-run bundle and the - redacted summary writer. It exposes exact `Path`, `SummaryRoot`, and - `TraceRoot` values and has no retention API. -- `internal/framework/debug` remains the pipeline-facing deep-trace adapter but - becomes a root-based filesystem recorder aimed at the bundle's `trace/` - directory. Pipeline debug interfaces and synchronization remain unchanged. -- `internal/cli` remains the only composition root. It resolves effective - roots, allocates requested state, constructs collaborators, writes summary - artifacts, and reports paths. - -After the cutover, delete `internal/core/workspace` and -`internal/core/diagnostics` if no remaining code has a distinct domain-neutral -reason to retain them. Do not preserve wrapper packages solely to avoid -updating imports. - -## Stage 1: Accept ADR-0006 and decouple cache storage internals - -**Status:** Complete - -### Objective - -Accept the architectural decision and remove checkpoint and chunk-plan storage -from workspace-specific constructors without changing current public -configuration or CLI behavior yet. - -### Read first - -- `internal/core/workspace/` -- `internal/framework/checkpoint/` -- `internal/framework/chunkplan/` -- `internal/framework/pipeline/checkpoint.go` -- `internal/cli/chunk_cache_test.go` -- the checkpoint sections of `docs/internal/pipeline.md` and - `docs/internal/diagnostics.md` - -### Implement - -1. Confirm ADR-0006 matches the finalized feature roadmap, then change only its - status from `Proposed` to `Accepted`. -2. Add the generic confined-write primitives in `internal/core/fileio`, with - caller-selected `0700`/`0600` support and focused path, atomicity, and - permission tests. -3. Move checkpoint identity, manifest, path, and persistence ownership into - `internal/framework/checkpoint`. Preserve every serialized field, schema - string, digest input, path component, validation reason, and payload byte. -4. Replace `NewWorkspaceRecorder` and `NewWorkspaceLoader` with root-based - filesystem constructors. During this preparatory stage, adapt the existing - CLI to pass the checkpoint root derived from its current workspace settings - so public behavior remains unchanged. -5. Move `DefaultChunkPlanRoot` out of `internal/core/workspace` into the - configuration/path-resolution boundary and retain the existing injected - `UserCacheDir` behavior and errors. -6. Remove checkpoint-specific and chunk-plan-specific files from - `internal/core/workspace`. Leave only pieces still required by the current - diagnostics/debug composition until later stages. - -### Tests - -Prove that: - -- old and refactored checkpoint identities and relative paths are identical; -- representative manifests marshal to identical JSON and v1/v2 compatibility - behavior is unchanged; -- a root-based recorder's output is reusable by the root-based loader; -- confined writes reject absolute, unclean, traversal, and backslash paths; -- atomic writes use the requested modes and do not leave temporary files after - failure; and -- chunk-plan default-root and bypass tests remain unchanged in behavior. - -Run at minimum: - -```sh -go test ./internal/core/fileio -go test ./internal/framework/checkpoint ./internal/framework/chunkplan -go test ./internal/cli -``` - -### Exit criteria - -ADR-0006 is accepted; cache persistence no longer depends on workspace-owned -identity or constructors; all current user-visible behavior remains unchanged; -and the repository-wide checks pass. - -## Stage 2: Build the unified debug-bundle collaborators - -**Status:** Complete - -### Objective - -Implement and test the target debug bundle behind internal constructors before -changing the public CLI enablement and configuration model. - -### Read first - -- `internal/core/diagnostics/` -- `internal/framework/debug/recorder.go` -- `internal/framework/pipeline/debug.go` -- `internal/framework/pipeline/runner_attempt_debug_test.go` -- `internal/framework/pipeline/runner_terminal_debug_test.go` -- `internal/core/config/redaction.go` -- [Diagnostics Internals](../internal/diagnostics.md) - -### Implement - -1. Add `internal/core/debugbundle` with a collision-safe allocator that accepts - the exact debug parent, uses the established `run-` ID - shape, creates `/summary` and `/trace`, and returns a bundle - exposing those exact paths. -2. Port the current diagnostics artifact constants and redacted writer methods - into a summary writer rooted at `summary/`. Remove retention decisions from - the new type. Use `0600` files and confined atomic writes. -3. Define summary-owned invocation and run-report payloads. Rename - diagnostics-specific methods and fields to summary/debug terminology while - retaining the useful artifact filenames listed in the feature roadmap. -4. Replace the deep trace recorder's workspace-settings constructor with a - constructor accepting the exact `trace/` root. Keep the existing - `pipeline.DebugRecorder` contract and synchronization wrapper. -5. Rename `RedactedDiagnosticsPayload` to a neutral redacted-summary method and - update its tests. Ensure the v3 effective configuration planned for the next - stage can be represented without secrets or raw application payloads. -6. Do not expose `--debug` or change current run behavior in this stage. The new - bundle is an internal collaborator exercised directly by tests until the - atomic CLI cutover. - -### Tests - -Add focused tests for: - -- collision retry and exhausted allocation; -- exact `summary/` and `trace/` layout; -- `0700` bundle directories and `0600` files; -- path traversal and symlink/path confinement behavior; -- every summary artifact and error log; -- absence of retention/deletion behavior; -- credential and sensitive-key redaction; -- no ambient environment or unrelated filesystem capture; and -- trace-recorder write failure propagation and concurrent synchronized writes. - -Run at minimum: - -```sh -go test ./internal/core/debugbundle -go test ./internal/core/config -go test ./internal/framework/debug ./internal/framework/pipeline -``` - -### Exit criteria - -The target bundle can be allocated and populated through tested summary and -trace collaborators, no public invocation behavior has changed, and all -repository-wide checks pass. - -## Stage 3: Cut configuration and CLI behavior over to the three surfaces - -**Status:** Complete - -### Objective - -Make the version 3 configuration, new flags, independent roots, resume policy, -and opt-in debug bundle the complete public runtime behavior in one atomic -cutover. - -### Read first - -- all files under `internal/core/config/` -- `internal/cli/run.go`, `internal/cli/run_test.go`, - `internal/cli/chunk_cache_test.go`, and `internal/cli/compatibility_test.go` -- the collaborators completed in Stages 1 and 2 -- `examples/*.config.yml` -- the Target Configuration, Target CLI, and Filesystem Layout sections of the - feature roadmap - -### Implement - -1. Replace the runtime and file configuration models with the fixed version 3 - types. Remove legacy diagnostics/workspace defaults, merge logic, - validation, effective-config fields, and redaction output. -2. Implement strict two-pass version handling. Reject version 2 with an - actionable migration message; reject other unsupported versions clearly; - and strictly decode version 3 with unknown fields rejected. -3. Implement the five new environment variables and remove all environment - aliases listed for removal in the feature roadmap. Preserve malformed-source - validation before precedence selection. -4. Add `--debug` and `--debug-dir`; remove `--diagnostics-dir`; update usage, - flag reordering, value validation, and exit-code behavior. Keep - `--output-dir`, `--chunk_cache`, and `--resume` with their target semantics. -5. Resolve the effective output directory after configuration loading and CLI - overrides. Use it for durable placement and summary metadata; do not fall - back to a separate CLI-only default helper. -6. When `--debug` is present, resolve the effective debug root and allocate the - bundle after configuration succeeds but before catalog lookup or pipeline - resolution. Use its run ID for the entire invocation, construct the summary - writer and trace recorder, and record invocation metadata immediately. - Without `--debug`, use the existing injected clock to generate a run ID and - construct no debug collaborator or directory. -7. Resolve the chunk-plan root only for `auto` or `refresh`, using the new - configuration family. Preserve all ADR-0005 store construction behavior. -8. When `--resume` is absent, pass no-op checkpoint collaborators without - resolving `UserCacheDir` for checkpoints. When it is present, resolve the - configured or per-user checkpoint root, create the identity, and construct - both root-based loader and recorder. -9. Replace diagnostics writes in the command with conditional summary writes. - Populate invocation metadata, redacted effective configuration, resolved - pipeline, resolved references, checkpoint events, chunk-plan decisions, run - manifest, warnings, run report, and error text as each becomes available. -10. Replace retention-aware failure handling with one failure path that prints - the primary error, attempts a single summary error record when a bundle - exists, reports any secondary debug error, and includes the allocated bundle - path. Never delete a requested bundle. -11. Report output and debug paths as fixed above. Keep warning and exit-code - behavior otherwise unchanged. -12. Convert both maintained example configs and package testdata configs to - version 3 so every executable example uses the supported schema. - -### Tests - -Update or add CLI and config tests covering: - -- exact v3 defaults, file fields, strict unknown fields, JSON/redacted shape, - and validation; -- precedence for output, chunk mode/root, checkpoint root, and debug root; -- malformed lower-precedence file or environment values despite valid CLI - overrides; -- actionable version 2 rejection before legacy unknown fields are decoded; -- rejection of every removed environment variable and configuration surface by - absence of effect, with obsolete file fields rejected as unknown; -- `--debug-dir` without `--debug`, empty explicit directory values, and removed - `--diagnostics-dir`; -- no debug allocation for ordinary runs and allocation before resolution - failures for debug runs; -- no checkpoint root resolution, loading, or recording without `--resume`; -- first-use resume writes checkpoints and later resume reuses them; -- `bypass` does not resolve a chunk-plan root; -- output directory selection and unchanged durable logical files; -- success and failure messages with and without a debug path; and -- requested summary or trace write failures producing exit code `1` without - masking a primary error. - -Run at minimum: - -```sh -go test ./internal/core/config -go test ./internal/cli -go test ./internal/modules/integration ./internal/modules/seriatim/input/transcript -``` - -### Exit criteria - -Only version 3 configuration is accepted; ordinary runs expose output and -chunk-plan cache only; resume and debug are invocation-controlled; all examples -and CLI tests use the new public contract; and the repository-wide checks pass. - -## Stage 4: Remove legacy workspace and diagnostics implementation - -**Status:** Complete - -### Objective - -Complete the internal ownership refactor and prove that no obsolete public state -model remains hidden behind compatibility wrappers. - -### Read first - -- the complete import graph reported by `rg 'core/(workspace|diagnostics)'` -- `internal/core/workspace/` -- `internal/core/diagnostics/` -- `internal/framework/checkpoint/` -- `internal/core/debugbundle/` -- `internal/framework/debug/` -- `internal/cli/run.go` - -### Implement - -1. Delete the remaining workspace settings, path, writer, and tests after moving - any still-valid generic behavior to `fileio`, `checkpoint`, or - `debugbundle`. -2. Delete the old diagnostics package, retention modes, run directory, and - tests after confirming all desired redacted artifacts are owned and tested - by `debugbundle`. -3. Remove compatibility constructors, aliases, fields, environment names, and - dead CLI helpers introduced solely by the old state model. Do not retain - deprecated parsing. -4. Rename internal variables, metadata fields, recorder types, and errors that - still use workspace or diagnostics to describe the removed surfaces. -5. Retain and document the frozen checkpoint wire schema identifiers. Retain - unrelated domain “diagnostic” terminology. -6. Confirm generic framework and pipeline packages receive only collaborator - interfaces and never physical roots. - -### Tests - -- Run `rg` checks proving there are no remaining imports of deleted packages, - public configuration fields, removed environment names, removed flags, or - operator-facing workspace/diagnostics messages. -- Allow matches only in ADR/history, version 2 migration guidance, frozen - checkpoint wire compatibility, and unrelated domain diagnostic contracts. -- Run all checkpoint, debug, CLI, and integration tests after deletion. - -### Exit criteria - -The old packages and compatibility surfaces are gone; remaining historical -terms are intentional and documented; package ownership matches the Planned -Internal Boundaries; and all checks pass. - -## Stage 5: Harden cross-surface security and compatibility - -**Status:** Complete - -### Objective - -Exercise the completed feature across failures, permissions, reuse, redaction, -and surface independence before changing current-behavior documentation. - -### Read first - -- all focused tests changed in Stages 1 through 4 -- `internal/cli/compatibility_test.go` -- `internal/framework/pipeline/runner_*debug*_test.go` -- `internal/framework/checkpoint/*_test.go` -- `internal/framework/chunkplan/store_test.go` -- the Security and Lifecycle and Compatibility Policy sections of the feature - roadmap - -### Implement - -1. Add an end-to-end state-surface matrix covering debug on/off, resume on/off, - and chunk modes `auto`, `bypass`, and `refresh`. Assert exactly which roots - are resolved and which files are created. -2. Add integration coverage proving chunk plans and checkpoints use independent - configured roots and that debug never affects cache selection or reuse. -3. Reuse a representative pre-refactor checkpoint fixture from an explicitly - selected old checkpoint root. Assert byte-compatible loading and normal - replacement behavior; do not create a migration path or rewrite untouched - entries. -4. Cover early failures before debug allocation, failures after allocation but - before pipeline execution, pipeline failures, output-write failures, summary - failures, and trace failures. Assert stable exit codes, stderr content, and - retained bundle state. -5. Inspect all summary artifacts for forbidden raw content and secrets. Inspect - trace artifacts to prove expected application data is present while API keys, - credential-shaped values, unrelated environment values, and unrelated file - contents are absent. -6. Assert created bundle and checkpoint modes on supported Unix systems and - preserve portable behavior on other supported platforms. -7. Confirm deletion is never automatic for output or debug, while manually - removing an exact cache entry only causes recomputation. -8. Confirm durable logical output and chunk-plan envelope bytes have not changed - solely because of ADR-0006. - -### Tests and validation - -Run focused packages while iterating, then run: - -```sh -go test ./... -go vet ./... -go build ./cmd/notarius -git diff --check -``` - -Run race-enabled tests for the concurrent debug and checkpoint collaborators if -supported by the environment: - -```sh -go test -race ./internal/framework/debug ./internal/framework/checkpoint ./internal/cli -``` - -### Exit criteria - -All three surfaces are independent under success and failure; compatibility and -permissions are proven; debug captures only intended data; durable contracts -are unchanged; and all repository-wide checks pass. - -## Stage 6: Update canonical documentation and close the roadmap - -**Status:** Not started - -### Objective - -Make current-behavior documentation and maintained examples describe the -implemented version 3 state model, then remove completed planning documents. - -### Read first - -- [Documentation Policy](../policy/documentation.md) -- [Architecture Policy](../policy/architecture.md) -- [CLI Reference](../cli.md) -- [Configuration](../config.md) -- [Operations](../operations.md) -- [Internal Overview](../internal/overview.md) -- [Diagnostics Internals](../internal/diagnostics.md) -- [Development](../development.md) -- `README.md`, `examples/`, and the final code and tests - -### Implement - -1. Update `docs/config.md` for version 3 fields, defaults, validation, - precedence, and the five supported state environment variables. Add one - complete version 2-to-3 before/after migration example and list every removed - setting and environment variable from the feature roadmap. -2. Update `docs/cli.md` for `--debug`, `--debug-dir`, target `--resume` - semantics, removed `--diagnostics-dir`, reporting, and failure behavior. -3. Rewrite the relevant `docs/operations.md` sections around output, chunk-plan - cache, checkpoint cache, and debug bundles. Document exact layouts, - permissions, inherited sensitivity, aggregation/copying/retention risk, - cleanup, compatible old-root reuse, per-user defaults, and the two - `/var/cache/notarius/...` Linux service recommendations. -4. Update `docs/policy/architecture.md` to describe the implemented three - surfaces and remove the obsolete five-surface model. Keep normative safety - rules concise and link operational details to Operations. -5. Replace `docs/internal/diagnostics.md` with a correctly named internal state - document, or distribute its content among focused internal documents if that - better matches the final packages. Update `docs/internal/overview.md`, - `docs/internal/pipeline.md`, `docs/development.md`, and all inbound links. -6. Update README orientation and examples only where they currently mention the - old schema or invocation. Do not duplicate the complete CLI, configuration, - or operations contracts. -7. Search the repository for stale flags, fields, environment names, paths, - package names, and workspace/diagnostics terminology. Preserve only ADR - history, explicit version 2 migration text, frozen checkpoint wire fields, - and unrelated domain diagnostics. -8. After implementation and documentation validation are complete, remove - `docs/roadmap/adr0006.md` and this implementation plan. They are completed - planning artifacts; the accepted ADR and canonical current-behavior - documents become authoritative. - -### Tests and validation - -- Validate every command, field, default, environment variable, path, and - example against code and focused tests. -- Validate all changed links and the task-specific reading map. -- Run any repository documentation or link checker if present. -- Run the complete repository-wide validation in the Execution Rules. - -### Exit criteria - -Canonical documentation describes only implemented behavior, migration and -operations each have one owner, all maintained examples are valid version 3, -no stale public state terminology remains, completed roadmap documents are -removed, and all checks pass.