notarius/docs/roadmap/adr0005.md

ADR-0005 Feature Roadmap

This roadmap defines the intended end state for ADR-0005. The feature is not implemented. The ordered coding work belongs in the implementation plan.

Intent

Notarius may run multiple extraction passes over the same source. Chunking, especially LLM-backed chunking over a large transcript, can be substantially more expensive than later per-chunk operations. Stable chunk material also improves the opportunity for provider-side prompt-cache reads across passes.

Notarius therefore prioritizes stable, aggressively reused source partitions over automatically applying later chunk-module, reference, prompt, model, or pipeline changes. An operator must explicitly request repartitioning.

Target State

Canonical source plan

Each validated generic source document has at most one stored chunk plan. The source document's canonical digest selects that plan. Pipeline identity, selected lanes, configured chunk module and options, references, validators, prompts, schemas, and LLM settings do not participate in lookup.

In the default mode, the first accepted plan for a source is reused by later pipelines and invocations. A configured chunk module runs only when no valid stored plan exists, when reuse is bypassed for one invocation, or when the operator requests refresh.

Plan and annotation contract

A plan contains an ordered, non-empty collection of inclusive source-unit ranges. The framework validates these boundaries and deterministically materializes runtime chunks from the current source document.

Chunk modules may attach optional domain annotations at plan or range scope. Annotations are namespaced, validated JSON values. They are preserved through storage and materialization but are never a hard cross-domain capability. Downstream modules may rely on the generic chunk contract only.

The store contains boundaries, annotations, warnings, and provenance. It does not contain fully materialized chunks or duplicated source-unit content.

Persistence

Chunk plans use a dedicated cache root that is independent of workspace.directory and the locations of checkpoints, diagnostics, debug artifacts, and durable output. The default is the platform-appropriate per-user cache directory: on Linux, $XDG_CACHE_HOME/notarius/chunk-plans when XDG_CACHE_HOME is set to a valid absolute path, otherwise $HOME/.cache/notarius/chunk-plans when it is unset, through the platform cache-directory resolver. An invalid relative XDG_CACHE_HOME is an error, not a fallback.

Operators may set workspace.chunk_cache.directory to replace that root. The recommended system-wide setting for a dedicated Linux service account is /var/cache/notarius/chunk-plans. This is an operational recommendation, not the application default; the service account must own the directory and it must not be shared across mutually untrusted users.

One mutable, versioned plan file is stored beneath the chunk-plan root under the full canonical source digest. Normal publication and refresh replace that file atomically; history, rollback, multiple variants, and content deduplication are outside this feature.

Plan state is potentially sensitive. Paths are confined, directories and files use restrictive permissions, default diagnostics omit annotation and source payloads, and debug output remains explicitly opt-in.

Runtime policy

The effective chunk-cache mode is one of:

• auto: load a valid stored plan; otherwise generate, validate, and publish one;
• bypass: do not read or write plan state for this invocation; or
• refresh: generate and validate a plan, then atomically replace stored state.

auto is the default. Invalid or incompatible stored state is a reported miss in auto; it is replaced only after a newly generated plan is accepted. A failed generation never overwrites prior state. Concurrent writes must never expose partial data and use last-successful-atomic-write semantics.

The public CLI override is --chunk_cache <auto|bypass|refresh>. Configuration and environment values use the same three modes, with CLI taking highest precedence.

Validation and execution

Framework plan validation and deterministic materialization run for generated and reused plans. The current pipeline's configured chunk-validator chain then validates the materialized chunks.

On a cache hit, the configured chunk module is constructed during normal pipeline preparation but its operation is not invoked and it makes no LLM call. A validator rejection of a structurally valid reused plan is a run outcome; it does not implicitly authorize rechunking.

Only the generic chunks capability is hard. Domain annotations such as D&D scene information are opportunistic and cannot be required solely because the current pipeline selected the module that normally produces them.

Canonical plans are the sole owner of chunk reuse. Invocation-scoped checkpoints no longer load or record chunk outputs. Downstream checkpoint identity continues to depend on the digest of the effective materialized chunks, so refresh invalidates affected downstream work.

Provenance

Every run distinguishes the chunk module requested by the resolved pipeline from the producer of the effective stored plan. Durable provenance records the source and plan digests, plan schema version, effective cache mode and action, producer module, producer references and LLM profile where applicable, and non-sensitive producer metadata.

Default diagnostics record lookup, validation, generation, and publication decisions without source or annotation payloads. Opt-in debug artifacts may contain complete plan and annotation material and are treated as sensitive.

Deployment documentation

The implemented configuration reference identifies the dedicated directory field, its environment override, and the per-user default. The operations guide documents the resulting filesystem layout and permissions, and recommends /var/cache/notarius/chunk-plans for a system-wide Linux deployment running as a dedicated service account. It also explains that cache deletion is recoverable but may repeat expensive chunk generation. Neither document presents the system-wide path as the default for an ordinary unprivileged invocation.

Compatibility Policy

Existing fully materialized chunk checkpoint files are not migrated or promoted to canonical plans. They are ignored for chunk reuse after this feature lands. The first auto run generates the source's plan through the configured chunk module.

The existing source digest is the canonical lookup identity. Any source change that alters that digest creates a separate plan. Earlier plan schema versions, malformed files, and digest mismatches are incompatible and follow the invalid auto-miss behavior.

Non-Goals

This feature does not provide:

• plan editing, comparison, history, rollback, or garbage collection;
• remote or shared plan storage;
• multiple active or automatically selected plan variants for one source;
• mandatory domain annotation contracts;
• automatic rechunking because configuration or model inputs changed; or
• a guarantee that an LLM provider will report cache hits.

Completion Outcomes

The feature is complete when independent runs over the same canonical source reuse byte-stable materialized chunks across pipeline, lane, reference, chunk-module, and LLM configuration changes; bypass and refresh obey their documented state semantics; provenance identifies the effective producer; legacy chunk checkpoints cannot compete with plan reuse; and all focused, integration, compatibility, CLI, and repository-wide validation passes. The configuration and operations references must also document both the per-user default and the recommended system-wide Linux setting.