Add ADR, roadmap, and staged implementation plan for input chunk caching

docs/roadmap/adr0005.md

@@ -0,0 +1,166 @@
+# ADR-0005 Feature Roadmap
+
+This roadmap defines the intended end state for
+[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature
+is not implemented. The ordered coding work belongs in the
+[implementation plan](implementation.md).
+
+## Intent
+
+Notarius may run multiple extraction passes over the same source. Chunking,
+especially LLM-backed chunking over a large transcript, can be substantially
+more expensive than later per-chunk operations. Stable chunk material also
+improves the opportunity for provider-side prompt-cache reads across passes.
+
+Notarius therefore prioritizes stable, aggressively reused source partitions
+over automatically applying later chunk-module, reference, prompt, model, or
+pipeline changes. An operator must explicitly request repartitioning.
+
+## Target State
+
+### Canonical source plan
+
+Each validated generic source document has at most one stored chunk plan. The
+source document's canonical digest selects that plan. Pipeline identity,
+selected lanes, configured chunk module and options, references, validators,
+prompts, schemas, and LLM settings do not participate in lookup.
+
+In the default mode, the first accepted plan for a source is reused by later
+pipelines and invocations. A configured chunk module runs only when no valid
+stored plan exists, when reuse is bypassed for one invocation, or when the
+operator requests refresh.
+
+### Plan and annotation contract
+
+A plan contains an ordered, non-empty collection of inclusive source-unit
+ranges. The framework validates these boundaries and deterministically
+materializes runtime chunks from the current source document.
+
+Chunk modules may attach optional domain annotations at plan or range scope.
+Annotations are namespaced, validated JSON values. They are preserved through
+storage and materialization but are never a hard cross-domain capability.
+Downstream modules may rely on the generic chunk contract only.
+
+The store contains boundaries, annotations, warnings, and provenance. It does
+not contain fully materialized chunks or duplicated source-unit content.
+
+### Persistence
+
+Chunk plans use a dedicated cache root that is independent of
+`workspace.directory` and the locations of checkpoints, diagnostics, debug
+artifacts, and durable output. The default is the platform-appropriate per-user
+cache directory: on Linux, `$XDG_CACHE_HOME/notarius/chunk-plans` when
+`XDG_CACHE_HOME` is set to a valid absolute path, otherwise
+`$HOME/.cache/notarius/chunk-plans` when it is unset, through the platform
+cache-directory resolver. An invalid relative `XDG_CACHE_HOME` is an error, not
+a fallback.
+
+Operators may set `workspace.chunk_cache.directory` to replace that root. The
+recommended system-wide setting for a dedicated Linux service account is
+`/var/cache/notarius/chunk-plans`. This is an operational recommendation, not
+the application default; the service account must own the directory and it must
+not be shared across mutually untrusted users.
+
+One mutable, versioned plan file is stored beneath the chunk-plan root under the
+full canonical source digest. Normal publication and refresh replace that file
+atomically; history, rollback, multiple variants, and content deduplication are
+outside this feature.
+
+Plan state is potentially sensitive. Paths are confined, directories and files
+use restrictive permissions, default diagnostics omit annotation and source
+payloads, and debug output remains explicitly opt-in.
+
+### Runtime policy
+
+The effective chunk-cache mode is one of:
+
+- `auto`: load a valid stored plan; otherwise generate, validate, and publish
+  one;
+- `bypass`: do not read or write plan state for this invocation; or
+- `refresh`: generate and validate a plan, then atomically replace stored state.
+
+`auto` is the default. Invalid or incompatible stored state is a reported miss
+in `auto`; it is replaced only after a newly generated plan is accepted. A
+failed generation never overwrites prior state. Concurrent writes must never
+expose partial data and use last-successful-atomic-write semantics.
+
+The public CLI override is
+`--chunk_cache <auto|bypass|refresh>`. Configuration and environment values use
+the same three modes, with CLI taking highest precedence.
+
+### Validation and execution
+
+Framework plan validation and deterministic materialization run for generated
+and reused plans. The current pipeline's configured chunk-validator chain then
+validates the materialized chunks.
+
+On a cache hit, the configured chunk module is constructed during normal
+pipeline preparation but its operation is not invoked and it makes no LLM call.
+A validator rejection of a structurally valid reused plan is a run outcome; it
+does not implicitly authorize rechunking.
+
+Only the generic `chunks` capability is hard. Domain annotations such as D&D
+scene information are opportunistic and cannot be required solely because the
+current pipeline selected the module that normally produces them.
+
+Canonical plans are the sole owner of chunk reuse. Invocation-scoped
+checkpoints no longer load or record chunk outputs. Downstream checkpoint
+identity continues to depend on the digest of the effective materialized
+chunks, so refresh invalidates affected downstream work.
+
+### Provenance
+
+Every run distinguishes the chunk module requested by the resolved pipeline
+from the producer of the effective stored plan. Durable provenance records the
+source and plan digests, plan schema version, effective cache mode and action,
+producer module, producer references and LLM profile where applicable, and
+non-sensitive producer metadata.
+
+Default diagnostics record lookup, validation, generation, and publication
+decisions without source or annotation payloads. Opt-in debug artifacts may
+contain complete plan and annotation material and are treated as sensitive.
+
+### Deployment documentation
+
+The implemented configuration reference identifies the dedicated directory
+field, its environment override, and the per-user default. The operations guide
+documents the resulting filesystem layout and permissions, and recommends
+`/var/cache/notarius/chunk-plans` for a system-wide Linux deployment running as
+a dedicated service account. It also explains that cache deletion is
+recoverable but may repeat expensive chunk generation. Neither document
+presents the system-wide path as the default for an ordinary unprivileged
+invocation.
+
+## Compatibility Policy
+
+Existing fully materialized chunk checkpoint files are not migrated or promoted
+to canonical plans. They are ignored for chunk reuse after this feature lands.
+The first `auto` run generates the source's plan through the configured chunk
+module.
+
+The existing source digest is the canonical lookup identity. Any source change
+that alters that digest creates a separate plan. Earlier plan schema versions,
+malformed files, and digest mismatches are incompatible and follow the invalid
+`auto`-miss behavior.
+
+## Non-Goals
+
+This feature does not provide:
+
+- plan editing, comparison, history, rollback, or garbage collection;
+- remote or shared plan storage;
+- multiple active or automatically selected plan variants for one source;
+- mandatory domain annotation contracts;
+- automatic rechunking because configuration or model inputs changed; or
+- a guarantee that an LLM provider will report cache hits.
+
+## Completion Outcomes
+
+The feature is complete when independent runs over the same canonical source
+reuse byte-stable materialized chunks across pipeline, lane, reference,
+chunk-module, and LLM configuration changes; bypass and refresh obey their
+documented state semantics; provenance identifies the effective producer;
+legacy chunk checkpoints cannot compete with plan reuse; and all focused,
+integration, compatibility, CLI, and repository-wide validation passes. The
+configuration and operations references must also document both the per-user
+default and the recommended system-wide Linux setting.