notarius/docs/roadmap/adr0005.md

# ADR-0005 Feature Roadmap

This roadmap records the implemented target state for
[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The
[implementation record](implementation.md) preserves the completed work.

## Intent

Notarius may run multiple extraction passes over the same source. Chunking,
especially LLM-backed chunking over a large transcript, can be substantially
more expensive than later per-chunk operations. Stable chunk material also
improves the opportunity for provider-side prompt-cache reads across passes.

Notarius therefore prioritizes stable, aggressively reused source partitions
over automatically applying later chunk-module, reference, prompt, model, or
pipeline changes. An operator must explicitly request repartitioning.

## Target State

### Canonical source plan

Each validated generic source document has at most one stored chunk plan. The
source document's canonical digest selects that plan. Pipeline identity,
selected lanes, configured chunk module and options, references, validators,
prompts, schemas, and LLM settings do not participate in lookup.

In the default mode, the first accepted plan for a source is reused by later
pipelines and invocations. A configured chunk module runs only when no valid
stored plan exists, when reuse is bypassed for one invocation, or when the
operator requests refresh.

### Plan and annotation contract

A plan contains an ordered, non-empty collection of inclusive source-unit
ranges. The framework validates these boundaries and deterministically
materializes runtime chunks from the current source document.

Chunk modules may attach optional domain annotations at plan or range scope.
Annotations are namespaced, validated JSON values. They are preserved through
storage and materialization but are never a hard cross-domain capability.
Downstream modules may rely on the generic chunk contract only.

The store contains boundaries, annotations, warnings, and provenance. It does
not contain fully materialized chunks or duplicated source-unit content.

### Persistence

Chunk plans use a dedicated cache root that is independent of
`workspace.directory` and the locations of checkpoints, diagnostics, debug
artifacts, and durable output. The default is the platform-appropriate per-user
cache directory: on Linux, `$XDG_CACHE_HOME/notarius/chunk-plans` when
`XDG_CACHE_HOME` is set to a valid absolute path, otherwise
`$HOME/.cache/notarius/chunk-plans` when it is unset, through the platform
cache-directory resolver. An invalid relative `XDG_CACHE_HOME` is an error, not
a fallback.

Operators may set `workspace.chunk_cache.directory` to replace that root. The
recommended system-wide setting for a dedicated Linux service account is
`/var/cache/notarius/chunk-plans`. This is an operational recommendation, not
the application default; the service account must own the directory and it must
not be shared across mutually untrusted users.

One mutable, versioned plan file is stored beneath the chunk-plan root under the
full canonical source digest. Normal publication and refresh replace that file
atomically; history, rollback, multiple variants, and content deduplication are
outside this feature.

Plan state is potentially sensitive. Paths are confined, directories and files
use restrictive permissions, default diagnostics omit annotation and source
payloads, and debug output remains explicitly opt-in.

### Runtime policy

The effective chunk-cache mode is one of:

- `auto`: load a valid stored plan; otherwise generate, validate, and publish
  one;
- `bypass`: do not read or write plan state for this invocation; or
- `refresh`: generate and validate a plan, then atomically replace stored state.

`auto` is the default. Invalid or incompatible stored state is a reported miss
in `auto`; it is replaced only after a newly generated plan is accepted. A
failed generation never overwrites prior state. Concurrent writes must never
expose partial data and use last-successful-atomic-write semantics.

The public CLI override is
`--chunk_cache <auto|bypass|refresh>`. Configuration and environment values use
the same three modes, with CLI taking highest precedence.

### Validation and execution

Framework plan validation and deterministic materialization run for generated
and reused plans. The current pipeline's configured chunk-validator chain then
validates the materialized chunks.

On a cache hit, the configured chunk module is constructed during normal
pipeline preparation but its operation is not invoked and it makes no LLM call.
A validator rejection of a structurally valid reused plan is a run outcome; it
does not implicitly authorize rechunking.

Only the generic `chunks` capability is hard. Domain annotations such as D&D
scene information are opportunistic and cannot be required solely because the
current pipeline selected the module that normally produces them.

Canonical plans are the sole owner of chunk reuse. Invocation-scoped
checkpoints no longer load or record chunk outputs. Downstream checkpoint
identity continues to depend on the digest of the effective materialized
chunks, so refresh invalidates affected downstream work.

### Provenance

Every run distinguishes the chunk module requested by the resolved pipeline
from the producer of the effective stored plan. Durable provenance records the
source and plan digests, plan schema version, effective cache mode and action,
producer module, producer references and LLM profile where applicable, and
non-sensitive producer metadata.

Default diagnostics record lookup, validation, generation, and publication
decisions without source or annotation payloads. Opt-in debug artifacts may
contain complete plan and annotation material and are treated as sensitive.

### Deployment documentation

The implemented configuration reference identifies the dedicated directory
field, its environment override, and the per-user default. The operations guide
documents the resulting filesystem layout and permissions, and recommends
`/var/cache/notarius/chunk-plans` for a system-wide Linux deployment running as
a dedicated service account. It also explains that cache deletion is
recoverable but may repeat expensive chunk generation. Neither document
presents the system-wide path as the default for an ordinary unprivileged
invocation.

## Compatibility Policy

Existing fully materialized chunk checkpoint files are not migrated or promoted
to canonical plans. They are ignored for chunk reuse after this feature lands.
The first `auto` run generates the source's plan through the configured chunk
module.

The existing source digest is the canonical lookup identity. Any source change
that alters that digest creates a separate plan. Earlier plan schema versions,
malformed files, and digest mismatches are incompatible and follow the invalid
`auto`-miss behavior.

## Non-Goals

This feature does not provide:

- plan editing, comparison, history, rollback, or garbage collection;
- remote or shared plan storage;
- multiple active or automatically selected plan variants for one source;
- mandatory domain annotation contracts;
- automatic rechunking because configuration or model inputs changed; or
- a guarantee that an LLM provider will report cache hits.

## Completion Outcomes

**Achieved.** Independent runs over the same canonical source
reuse byte-stable materialized chunks across pipeline, lane, reference,
chunk-module, and LLM configuration changes; bypass and refresh obey their
documented state semantics; provenance identifies the effective producer;
legacy chunk checkpoints cannot compete with plan reuse; and all focused,
integration, compatibility, CLI, and repository-wide validation pass. The
configuration and operations references must also document both the per-user
default and the recommended system-wide Linux setting.