notarius/docs/roadmap/adr0006.md

ADR-0006 Feature Roadmap

This roadmap defines the intended end state for ADR-0006. The feature is not implemented.

Intent

Notarius should communicate filesystem behavior in terms of why data exists: durable output, reconstructible cache, or explicitly requested debug material. The public configuration and CLI should not expose internal distinctions such as workspace state or a second diagnostics product.

The implementation may retain focused writers, serializers, and security boundaries. This roadmap changes their public composition and lifecycle rather than requiring every kind of data to share one package or physical root.

Target Configuration

The configuration schema advances to version 3 and uses these top-level surfaces:

version: 3

output:
  directory: ./notarius-output

cache:
  chunk_plans:
    directory: ""
    mode: auto
  checkpoints:
    directory: ""

debug:
  directory: ./notarius-debug

An empty cache directory selects that cache family's platform-appropriate per-user default. A configured directory is the exact root; Notarius does not append the cache-family name to it.

The workspace and diagnostics top-level configuration surfaces do not exist in version 3. In particular, version 3 has no persistent debug enablement, diagnostics enablement or retention, checkpoint enablement, or shared workspace directory.

Output configuration

output.directory selects the durable output root. Precedence is the explicit --output-dir flag, NOTARIUS_OUTPUT_DIR, the file value, then ./notarius-output.

The pipeline's existing pipelines.<id>.output module binding remains separate: it selects the output encoder, while top-level output.directory selects where the CLI places that encoder's logical files.

Chunk-plan cache configuration

cache.chunk_plans.mode accepts auto, bypass, or refresh with the ADR-0005 semantics. Precedence is the explicit --chunk_cache flag, NOTARIUS_CACHE_CHUNK_PLANS_MODE, the file value, then auto.

cache.chunk_plans.directory selects the exact canonical plan root. Precedence is NOTARIUS_CACHE_CHUNK_PLANS_DIR, the file value, then <os.UserCacheDir>/notarius/chunk-plans. On Linux, the default therefore uses a valid absolute $XDG_CACHE_HOME or falls back to $HOME/.cache when that variable is unset. A relative XDG_CACHE_HOME remains an error.

The recommended system-service value remains /var/cache/notarius/chunk-plans. ADR-0005 plan identities, envelopes, permissions, atomic publication, validation, provenance, and cleanup semantics do not change.

Checkpoint cache configuration

cache.checkpoints.directory selects the exact checkpoint root. Precedence is NOTARIUS_CACHE_CHECKPOINTS_DIR, the file value, then <os.UserCacheDir>/notarius/checkpoints.

For a system-wide Linux deployment under a dedicated service account, the recommended configured root is /var/cache/notarius/checkpoints, provisioned with restrictive service-account ownership independently from the chunk-plan root.

Checkpoint directories and files use restrictive permissions because they may contain source text and intermediate artifacts. They inherit the sensitivity of the data captured in them. Their existing compatibility validation, pipeline identity, source and lane payloads, and downstream dependency fingerprints remain intact. Checkpoints do not store or select chunk results.

There is no cache.checkpoints.enabled setting. Checkpoint I/O is controlled by the invocation's --resume flag.

Debug configuration

debug.directory selects the debug root but does not enable debug output. Precedence is --debug-dir, NOTARIUS_DEBUG_DIR, the file value, then ./notarius-debug.

There is no environment or file setting that enables debug. This prevents an ambient production configuration from silently recording potentially sensitive trace data and creating additional retained copies of application data.

Target CLI

The run command retains --output-dir, --chunk_cache, and --resume, and adds:

• --debug: enable the complete debug bundle for this invocation;
• --debug-dir <path>: override its root; valid only with --debug.

--diagnostics-dir is removed. There is no --workspace, --cache-dir, or separate diagnostics flag.

--resume means “reuse compatible checkpoints when present and record checkpoint state for work executed by this invocation.” On a first invocation with no compatible state, it executes normally and creates checkpoints. A run without --resume neither loads nor records checkpoints and does not resolve or create the checkpoint root.

The CLI reports the durable output directory on success. When debug is enabled, it also reports the allocated debug bundle path on success and includes that path in failure reporting once allocation has succeeded.

Malformed values from every supplied configuration source remain errors even when a higher-precedence source would otherwise override them.

Filesystem Layout

Output

Durable logical files retain the current per-run layout:

<output-root>/<run-id>/...

Output creation remains atomic per file. The maintained logical JSON output contract, including manifests, warnings, rejections, and normalized artifacts, does not change solely because of this state-model refactor.

Chunk plans

The existing ADR-0005 layout remains:

<chunk-plan-root>/<source-sha256-hex>/plan.json

Chunk-plan state remains independent of checkpoint and debug roots.

Checkpoints

The current identity hierarchy moves beneath the independently selected checkpoint root:

<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/...

The configured root replaces the former implicit <workspace.directory>/checkpoints prefix. Existing manifests and payload schemas remain compatible when pointed at their prior physical checkpoint root.

Debug bundle

One explicitly requested bundle is allocated before pipeline resolution at:

<debug-root>/<run-id>/
  summary/
  trace/

summary/ contains the redacted artifacts currently associated with diagnostics: invocation metadata, redacted effective configuration, resolved pipeline and reference provenance, checkpoint decisions, chunk-plan decisions, run manifest, warnings, run report, and error text when available.

trace/ contains the existing deep debug material: source and stage inputs and outputs, plans and materialized chunks, annotations, validator attempts, prompts, model responses, retry timing, and intermediate serialized artifacts.

Summary data never contains raw source, references, annotations, prompts, model responses, credentials, or malformed cache bytes. Trace data may contain all of those except credentials, which remain redacted. A debug bundle inherits the sensitivity of the most sensitive data captured in it; it does not constitute a separate, intrinsically higher sensitivity class. Because it aggregates and retains an additional copy of application data, its root and files use restrictive permissions by default.

Trace data may preserve source or reference content that the durable output summarized, rejected, or intentionally omitted. This can make the bundle more sensitive than the output alone, but not more sensitive than the complete set of application data supplied to or produced by the run. Restrictive defaults are therefore precautionary handling for data of unknown sensitivity, not a classification of debug material as uniquely sensitive.

Debug persistence failures are command failures because the operator explicitly requested the bundle. No debug directory is created without --debug. A requested bundle is never automatically deleted based on success, warnings, or failure; cleanup is explicit.

Runtime Ownership

The CLI remains the composition root for physical paths and filesystem-backed collaborators. Pipeline modules continue to return logical data and never receive output, cache, or debug roots.

The framework may retain separate interfaces for checkpoint recording and loading, chunk-plan storage, redacted summary recording, and deep trace recording. The single debug facility may compose multiple internal recorders; unification does not require weakening redaction or passing raw trace data through summary models.

The public workspace type and terminology disappear from configuration, invocation messages, debug metadata, and operator documentation. The standalone diagnostics directory and retention surface also disappear. Domain contracts that use “diagnostic” in another sense, such as a validator-provided diagnostic artifact path, are not renamed solely by this decision. Generic confined-path and atomic-write helpers may remain in an internal package but must not impose a workspace abstraction on callers.

Security and Lifecycle

• Output is durable user data. Notarius never automatically deletes it.
• Chunk plans and checkpoints are reconstructible cache. Exact entries or roots may be removed, with the documented cost of recomputation.
• Debug is explicit inspection data. It is off by default, retained when requested, and removed only by an explicit operator action.
• Checkpoints and debug bundles inherit the sensitivity of the application data they capture. Their security concern is the additional retained copy and, for debug bundles, aggregation of that data—not an intrinsically higher sensitivity classification.
• Operators are responsible for choosing debug locations, access controls, and retention appropriate to the data processed by the run. Documentation must explain that a bundle can retain material omitted from the durable output.
• Cache and debug roots are separate trust boundaries and must not be shared among mutually untrusted users.
• Credentials are excluded from output, cache, summaries, traces, logs, errors, examples, and redacted configuration.
• Debug collection is allowlisted to application-owned data. It must not capture unrelated process environment values, host secrets, or arbitrary filesystem content merely because they are available to the process.
• Default runs may create output and use the default chunk-plan cache, but do not create checkpoint or debug state unless their invocation flags request those surfaces.

Compatibility Policy

Version 3 is a deliberate configuration break. Version 2 files are rejected with an actionable message directing operators to the migration documentation; Notarius does not retain parallel legacy field parsing indefinitely.

The version 2 migration is:

• workspace.chunk_cache.mode -> cache.chunk_plans.mode;
• workspace.chunk_cache.directory -> cache.chunk_plans.directory;
• <workspace.directory>/checkpoints -> cache.checkpoints.directory, when existing checkpoint reuse is desired;
• --diagnostics-dir, diagnostics fields, workspace diagnostics fields, and their environment variables -> removal or the new debug controls;
• workspace.debug.enabled -> removal; invoke with --debug;
• <workspace.directory>/debug -> debug.directory, if the same parent is desired; and
• workspace.resume.enabled -> removal; invoke with --resume.

Environment migration is:

• NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE -> NOTARIUS_CACHE_CHUNK_PLANS_MODE;
• NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR -> NOTARIUS_CACHE_CHUNK_PLANS_DIR;
• NOTARIUS_WORKSPACE_DIR -> no direct replacement; configure the applicable output, checkpoint-cache, and debug roots separately;
• NOTARIUS_WORKSPACE_RESUME_ENABLED and NOTARIUS_WORKSPACE_DEBUG_ENABLED -> removal in favor of invocation flags; and
• NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED, NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION, NOTARIUS_WORK_DIR, and NOTARIUS_DIAGNOSTICS_RETENTION -> removal with the standalone diagnostics surface.

Existing chunk-plan files are reused without migration when the new directory resolves to the same root. Existing checkpoint files are reusable when cache.checkpoints.directory names the former checkpoint root and their normal identity and schema checks pass. Existing diagnostics and debug directories are not moved, merged, or deleted automatically.

The durable output schema and ADR-0005 cache envelope schema are not versioned solely because configuration advances to version 3.

Documentation Outcomes

When implemented:

• the CLI reference owns all flags and invocation semantics;
• configuration owns version 3 fields, environment variables, defaults, and precedence;
• operations owns physical layouts, permissions, cleanup, failure inspection, and per-user and system-service cache guidance;
• the integration contract continues to own durable logical output;
• internal pipeline and component documents explain collaborator composition without reintroducing public workspace terminology; and
• migration documentation provides one complete version 2 to version 3 example.

Current-behavior documentation must not describe this target before the corresponding behavior is implemented.

Non-Goals

This feature does not introduce:

• remote output, cache, or debug storage;
• cache garbage collection, quotas, archival, history, or rollback;
• automatic debug retention or upload;
• a daemon logging, metrics, or telemetry system;
• a change to pipeline topology, artifact schemas, chunk-plan identity, or module contracts; or
• a requirement that output, different cache families, and debug share one physical parent directory.

Completion Outcomes

The feature is complete when ordinary runs expose only durable output and canonical chunk-plan cache behavior; checkpoint I/O occurs only under --resume; one --debug invocation produces a complete retained summary and trace bundle; no public workspace or diagnostics configuration remains; version 2 migration failures are actionable; existing compatible chunk plans and checkpoints remain reusable at explicitly selected roots; and all configuration, CLI, filesystem, security, compatibility, integration, and repository-wide tests pass.