notarius/docs/roadmap/adr0006.md

# ADR-0006 Feature Roadmap

This roadmap defines the intended end state for
[ADR-0006](../adr/0006-separate-output-cache-and-debug-state.md). The feature is
not implemented.

## Intent

Notarius should communicate filesystem behavior in terms of why data exists:
durable output, reconstructible cache, or explicitly requested debug material.
The public configuration and CLI should not expose internal distinctions such
as workspace state or a second diagnostics product.

The implementation may retain focused writers, serializers, and security
boundaries. This roadmap changes their public composition and lifecycle rather
than requiring every kind of data to share one package or physical root.

## Target Configuration

The configuration schema advances to version 3 and uses these top-level
surfaces:

```yaml
version: 3

output:
  directory: ./notarius-output

cache:
  chunk_plans:
    directory: ""
    mode: auto
  checkpoints:
    directory: ""

debug:
  directory: ./notarius-debug
```

An empty cache directory selects that cache family's platform-appropriate
per-user default. A configured directory is the exact root; Notarius does not
append the cache-family name to it.

The `workspace` and `diagnostics` top-level configuration surfaces do not exist
in version 3. In particular, version 3 has no persistent debug enablement,
diagnostics enablement or retention, checkpoint enablement, or shared workspace
directory.

### Output configuration

`output.directory` selects the durable output root. Precedence is the explicit
`--output-dir` flag, `NOTARIUS_OUTPUT_DIR`, the file value, then
`./notarius-output`.

The pipeline's existing `pipelines.<id>.output` module binding remains separate:
it selects the output encoder, while top-level `output.directory` selects where
the CLI places that encoder's logical files.

### Chunk-plan cache configuration

`cache.chunk_plans.mode` accepts `auto`, `bypass`, or `refresh` with the
ADR-0005 semantics. Precedence is the explicit `--chunk_cache` flag,
`NOTARIUS_CACHE_CHUNK_PLANS_MODE`, the file value, then `auto`.

`cache.chunk_plans.directory` selects the exact canonical plan root. Precedence
is `NOTARIUS_CACHE_CHUNK_PLANS_DIR`, the file value, then
`<os.UserCacheDir>/notarius/chunk-plans`. On Linux, the default therefore uses a
valid absolute `$XDG_CACHE_HOME` or falls back to `$HOME/.cache` when that
variable is unset. A relative `XDG_CACHE_HOME` remains an error.

The recommended system-service value remains
`/var/cache/notarius/chunk-plans`. ADR-0005 plan identities, envelopes,
permissions, atomic publication, validation, provenance, and cleanup semantics
do not change.

### Checkpoint cache configuration

`cache.checkpoints.directory` selects the exact checkpoint root. Precedence is
`NOTARIUS_CACHE_CHECKPOINTS_DIR`, the file value, then
`<os.UserCacheDir>/notarius/checkpoints`.

For a system-wide Linux deployment under a dedicated service account, the
recommended configured root is `/var/cache/notarius/checkpoints`, provisioned
with restrictive service-account ownership independently from the chunk-plan
root.

Checkpoint directories and files use restrictive permissions because they may
contain source text and intermediate artifacts. They inherit the sensitivity of
the data captured in them. Their existing compatibility validation, pipeline
identity, source and lane payloads, and downstream dependency fingerprints
remain intact. Checkpoints do not store or select chunk results.

There is no `cache.checkpoints.enabled` setting. Checkpoint I/O is controlled by
the invocation's `--resume` flag.

### Debug configuration

`debug.directory` selects the debug root but does not enable debug output.
Precedence is `--debug-dir`, `NOTARIUS_DEBUG_DIR`, the file value, then
`./notarius-debug`.

There is no environment or file setting that enables debug. This prevents an
ambient production configuration from silently recording potentially sensitive
trace data and creating additional retained copies of application data.

## Target CLI

The run command retains `--output-dir`, `--chunk_cache`, and `--resume`, and
adds:

- `--debug`: enable the complete debug bundle for this invocation;
- `--debug-dir <path>`: override its root; valid only with `--debug`.

`--diagnostics-dir` is removed. There is no `--workspace`, `--cache-dir`, or
separate diagnostics flag.

`--resume` means “reuse compatible checkpoints when present and record
checkpoint state for work executed by this invocation.” On a first invocation
with no compatible state, it executes normally and creates checkpoints. A run
without `--resume` neither loads nor records checkpoints and does not resolve or
create the checkpoint root.

The CLI reports the durable output directory on success. When debug is enabled,
it also reports the allocated debug bundle path on success and includes that
path in failure reporting once allocation has succeeded.

Malformed values from every supplied configuration source remain errors even
when a higher-precedence source would otherwise override them.

## Filesystem Layout

### Output

Durable logical files retain the current per-run layout:

```text
<output-root>/<run-id>/...
```

Output creation remains atomic per file. The maintained logical JSON output
contract, including manifests, warnings, rejections, and normalized artifacts,
does not change solely because of this state-model refactor.

### Chunk plans

The existing ADR-0005 layout remains:

```text
<chunk-plan-root>/<source-sha256-hex>/plan.json
```

Chunk-plan state remains independent of checkpoint and debug roots.

### Checkpoints

The current identity hierarchy moves beneath the independently selected
checkpoint root:

```text
<checkpoint-root>/<pipeline-id>/<input-key>-<source-or-input-digest>/<pipeline-digest>/<identity-digest>/...
```

The configured root replaces the former implicit
`<workspace.directory>/checkpoints` prefix. Existing manifests and payload
schemas remain compatible when pointed at their prior physical checkpoint root.

### Debug bundle

One explicitly requested bundle is allocated before pipeline resolution at:

```text
<debug-root>/<run-id>/
  summary/
  trace/
```

`summary/` contains the redacted artifacts currently associated with
diagnostics: invocation metadata, redacted effective configuration, resolved
pipeline and reference provenance, checkpoint decisions, chunk-plan decisions,
run manifest, warnings, run report, and error text when available.

`trace/` contains the existing deep debug material: source and stage inputs and
outputs, plans and materialized chunks, annotations, validator attempts,
prompts, model responses, retry timing, and intermediate serialized artifacts.

Summary data never contains raw source, references, annotations, prompts, model
responses, credentials, or malformed cache bytes. Trace data may contain all
of those except credentials, which remain redacted. A debug bundle inherits the
sensitivity of the most sensitive data captured in it; it does not constitute a
separate, intrinsically higher sensitivity class. Because it aggregates and
retains an additional copy of application data, its root and files use
restrictive permissions by default.

Trace data may preserve source or reference content that the durable output
summarized, rejected, or intentionally omitted. This can make the bundle more
sensitive than the output alone, but not more sensitive than the complete set
of application data supplied to or produced by the run. Restrictive defaults
are therefore precautionary handling for data of unknown sensitivity, not a
classification of debug material as uniquely sensitive.

Debug persistence failures are command failures because the operator explicitly
requested the bundle. No debug directory is created without `--debug`. A
requested bundle is never automatically deleted based on success, warnings, or
failure; cleanup is explicit.

## Runtime Ownership

The CLI remains the composition root for physical paths and filesystem-backed
collaborators. Pipeline modules continue to return logical data and never
receive output, cache, or debug roots.

The framework may retain separate interfaces for checkpoint recording and
loading, chunk-plan storage, redacted summary recording, and deep trace
recording. The single debug facility may compose multiple internal recorders;
unification does not require weakening redaction or passing raw trace data
through summary models.

The public `workspace` type and terminology disappear from configuration,
invocation messages, debug metadata, and operator documentation. The standalone
diagnostics directory and retention surface also disappear. Domain contracts
that use “diagnostic” in another sense, such as a validator-provided diagnostic
artifact path, are not renamed solely by this decision. Generic confined-path
and atomic-write helpers may remain in an internal package but must not impose a
workspace abstraction on callers.

## Security and Lifecycle

- Output is durable user data. Notarius never automatically deletes it.
- Chunk plans and checkpoints are reconstructible cache. Exact entries or roots
  may be removed, with the documented cost of recomputation.
- Debug is explicit inspection data. It is off by default, retained when
  requested, and removed only by an explicit operator action.
- Checkpoints and debug bundles inherit the sensitivity of the application data
  they capture. Their security concern is the additional retained copy and, for
  debug bundles, aggregation of that data—not an intrinsically higher
  sensitivity classification.
- Operators are responsible for choosing debug locations, access controls, and
  retention appropriate to the data processed by the run. Documentation must
  explain that a bundle can retain material omitted from the durable output.
- Cache and debug roots are separate trust boundaries and must not be shared
  among mutually untrusted users.
- Credentials are excluded from output, cache, summaries, traces, logs, errors,
  examples, and redacted configuration.
- Debug collection is allowlisted to application-owned data. It must not capture
  unrelated process environment values, host secrets, or arbitrary filesystem
  content merely because they are available to the process.
- Default runs may create output and use the default chunk-plan cache, but do
  not create checkpoint or debug state unless their invocation flags request
  those surfaces.

## Compatibility Policy

Version 3 is a deliberate configuration break. Version 2 files are rejected
with an actionable message directing operators to the migration documentation;
Notarius does not retain parallel legacy field parsing indefinitely.

The version 2 migration is:

- `workspace.chunk_cache.mode` -> `cache.chunk_plans.mode`;
- `workspace.chunk_cache.directory` -> `cache.chunk_plans.directory`;
- `<workspace.directory>/checkpoints` -> `cache.checkpoints.directory`, when
  existing checkpoint reuse is desired;
- `--diagnostics-dir`, diagnostics fields, workspace diagnostics fields, and
  their environment variables -> removal or the new debug controls;
- `workspace.debug.enabled` -> removal; invoke with `--debug`;
- `<workspace.directory>/debug` -> `debug.directory`, if the same parent is
  desired; and
- `workspace.resume.enabled` -> removal; invoke with `--resume`.

Environment migration is:

- `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE` ->
  `NOTARIUS_CACHE_CHUNK_PLANS_MODE`;
- `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` ->
  `NOTARIUS_CACHE_CHUNK_PLANS_DIR`;
- `NOTARIUS_WORKSPACE_DIR` -> no direct replacement; configure the applicable
  output, checkpoint-cache, and debug roots separately;
- `NOTARIUS_WORKSPACE_RESUME_ENABLED` and
  `NOTARIUS_WORKSPACE_DEBUG_ENABLED` -> removal in favor of invocation flags;
  and
- `NOTARIUS_WORKSPACE_DIAGNOSTICS_ENABLED`,
  `NOTARIUS_WORKSPACE_DIAGNOSTICS_RETENTION`, `NOTARIUS_WORK_DIR`, and
  `NOTARIUS_DIAGNOSTICS_RETENTION` -> removal with the standalone diagnostics
  surface.

Existing chunk-plan files are reused without migration when the new directory
resolves to the same root. Existing checkpoint files are reusable when
`cache.checkpoints.directory` names the former checkpoint root and their normal
identity and schema checks pass. Existing diagnostics and debug directories are
not moved, merged, or deleted automatically.

The durable output schema and ADR-0005 cache envelope schema are not versioned
solely because configuration advances to version 3.

## Documentation Outcomes

When implemented:

- the CLI reference owns all flags and invocation semantics;
- configuration owns version 3 fields, environment variables, defaults, and
  precedence;
- operations owns physical layouts, permissions, cleanup, failure inspection,
  and per-user and system-service cache guidance;
- the integration contract continues to own durable logical output;
- internal pipeline and component documents explain collaborator composition
  without reintroducing public workspace terminology; and
- migration documentation provides one complete version 2 to version 3 example.

Current-behavior documentation must not describe this target before the
corresponding behavior is implemented.

## Non-Goals

This feature does not introduce:

- remote output, cache, or debug storage;
- cache garbage collection, quotas, archival, history, or rollback;
- automatic debug retention or upload;
- a daemon logging, metrics, or telemetry system;
- a change to pipeline topology, artifact schemas, chunk-plan identity, or
  module contracts; or
- a requirement that output, different cache families, and debug share one
  physical parent directory.

## Completion Outcomes

The feature is complete when ordinary runs expose only durable output and
canonical chunk-plan cache behavior; checkpoint I/O occurs only under
`--resume`; one `--debug` invocation produces a complete retained summary and
trace bundle; no public workspace or diagnostics configuration remains; version
2 migration failures are actionable; existing compatible chunk plans and
checkpoints remain reusable at explicitly selected roots; and all configuration,
CLI, filesystem, security, compatibility, integration, and repository-wide
tests pass.