From 1c13e1d64a3042230679b1c927567af8248eeb55 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 17 Jul 2026 18:38:08 -0500 Subject: [PATCH] Add ADR, roadmap, and staged implementation plan for input chunk caching --- ...5-cache-canonical-chunk-plans-by-source.md | 138 +++ docs/roadmap/adr0005.md | 166 +++ docs/roadmap/implementation.md | 965 ++++++++++++++++++ 3 files changed, 1269 insertions(+) create mode 100644 docs/adr/0005-cache-canonical-chunk-plans-by-source.md create mode 100644 docs/roadmap/adr0005.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/adr/0005-cache-canonical-chunk-plans-by-source.md b/docs/adr/0005-cache-canonical-chunk-plans-by-source.md new file mode 100644 index 0000000..3426325 --- /dev/null +++ b/docs/adr/0005-cache-canonical-chunk-plans-by-source.md @@ -0,0 +1,138 @@ +# ADR-0005: Cache one canonical chunk plan per source + +**Status:** Proposed +**Date:** 2026-07-17 + +## Context + +Notarius may run several extraction passes over the same source. A D&D +transcript, for example, may first produce NPC artifacts and later produce +spell or combat artifacts, with output from an earlier pass supplied as a +reference to a later pass. + +An LLM-backed chunker may process an entire, potentially large source in one +expensive request. Recomputing boundaries for every pipeline or pass repeats +that cost and can make otherwise comparable extraction runs use different +source partitions. Stable chunk material also gives later extraction requests +a better opportunity to benefit from provider-side prompt caching. + +Chunk boundaries can affect extraction quality. Evidence may span a boundary, +overlap may produce duplicates, and different partitions may change the context +available to a model. Merge and normalization should remove structural signs +of chunking from durable output, but they cannot guarantee recovery of evidence +that an extractor did not receive. + +Notarius therefore needs an explicit policy for choosing between automatically +applying the latest chunking configuration and preserving one stable partition +for repeated work on the same source. + +## Decision + +Notarius assigns one active canonical chunk plan to a source and reuses that +plan by default across pipelines and invocations. + +The canonical source identity is derived from the validated generic source +document and covers the source-unit identity, order, and content needed to +interpret plan boundaries. Input-adapter and chunk-producer identities are +recorded as provenance, but the active-plan lookup does not vary with: + +- pipeline identity or selected artifact lanes; +- the configured chunk module or its options; +- references; +- LLM provider, model, profile, prompt, or response schema; or +- configuration for later pipeline stages. + +When an active plan exists, Notarius uses it even if the current pipeline +configures a different chunk module or different chunk-module settings. The +configured chunk module generates a plan only when none exists or when the +operator explicitly requests recomputation. + +The framework-owned minimum plan contract is an ordered, non-empty set of +source-unit ranges. Each range identifies the inclusive start and end unit for +one chunk. A chunk module may also provide namespaced, domain-specific +annotations at plan or range scope. Those annotations are stored with the plan +and passed through the pipeline when present, but they remain optional. +Downstream stages must not assume that annotations associated with the +currently configured chunk module are present on a reused plan produced by a +different module. + +The cache stores the plan rather than fully materialized chunks. The framework +validates a reused plan against the current source and deterministically +materializes its ranges into chunks. The same source and plan must produce +byte-stable chunk input for later stages. + +Canonical plan storage is a distinct cache surface with an independently +configurable location. It is not coupled to the roots or lifecycles of +invocation checkpoints, diagnostics, debug artifacts, or durable output. This +allows per-user and system-service deployments to apply cache-specific +ownership, permissions, placement, and cleanup policy without relocating other +Notarius state. + +One mutable active plan is stored under the canonical source identity and +retains provenance for the module and relevant runtime inputs that produced it. +The effective plan producer is reported separately from the chunk module +requested by the current pipeline; reuse must not attribute cached boundaries +or annotations to a module that did not produce them. + +Reuse is enabled by default. Operators can explicitly: + +- bypass cached plans for an invocation without changing the active plan; or +- recompute a plan with the configured chunk module and make it active for + later work. + +Exact storage layout, configuration fields, CLI syntax, publication mechanics, +recovery behavior, and diagnostics are implementation and operational +contracts rather than part of this decision. + +## Alternatives considered + +- Recompute chunks on every invocation. This always applies the current + chunking configuration, but repeats the most expensive stage and weakens + provider-side caching and cross-pass comparability. +- Cache every distinct chunking request by including module options, + references, prompts, profiles, and other runtime inputs in its identity. This + closely associates a cached result with its producing request, but reduces + reuse and permits boundary drift across operationally different passes. +- Key plans by source plus chunk module and options. This shares plans across + pipelines using the same strategy, but changing the configured strategy + silently selects a different partition rather than preserving one canonical + partition for the source. +- Require operators to name or supply a plan for every run. Explicit selection + is reproducible and may be useful as an advanced operation, but adds friction + to the default workflow and does not provide automatic reuse. +- Store fully materialized chunks. This simplifies loading, but duplicates + source content and couples durable state to the current chunk representation + rather than the stable boundary decision. +- Store canonical plans beneath the general workspace root. This would reuse an + existing location setting, but it couples a reusable application cache to + checkpoint, diagnostic, and debug state that have different ownership, + sensitivity, retention, and deployment requirements. + +## Consequences + +Independent pipelines and passes over the same source use stable boundaries by +default. This reduces repeated LLM work, improves cross-pass comparability, and +increases the opportunity for cached provider reads. + +The configured chunk module may not execute, and its settings may have no +effect, when an active plan already exists. Domain-specific annotations reflect +the plan's original producer and may be absent or differ from those the current +module would produce. User-visible provenance must make the effective plan +clear. + +A poor or outdated partition remains active until an operator replaces it. +This can preserve suboptimal context boundaries and affect extraction recall or +duplication even when merge and normalization hide the partition structure in +durable output. Stable reuse is an intentional priority over automatically +incorporating later chunk-strategy changes. + +The framework gains a durable minimal chunk-plan contract and deterministic +materialization responsibility. Chunk modules must separate required boundary +output from optional annotations, and downstream modules may rely only on the +minimal boundary contract unless a future decision introduces an explicit plan +compatibility mechanism. + +Operators must configure and secure canonical plan storage independently from +other workspace state when the per-user default is not appropriate. Removing +that cache remains recoverable because Notarius can regenerate it from the +source, but doing so may repeat an expensive LLM operation. diff --git a/docs/roadmap/adr0005.md b/docs/roadmap/adr0005.md new file mode 100644 index 0000000..2e17f80 --- /dev/null +++ b/docs/roadmap/adr0005.md @@ -0,0 +1,166 @@ +# ADR-0005 Feature Roadmap + +This roadmap defines the intended end state for +[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature +is not implemented. The ordered coding work belongs in the +[implementation plan](implementation.md). + +## Intent + +Notarius may run multiple extraction passes over the same source. Chunking, +especially LLM-backed chunking over a large transcript, can be substantially +more expensive than later per-chunk operations. Stable chunk material also +improves the opportunity for provider-side prompt-cache reads across passes. + +Notarius therefore prioritizes stable, aggressively reused source partitions +over automatically applying later chunk-module, reference, prompt, model, or +pipeline changes. An operator must explicitly request repartitioning. + +## Target State + +### Canonical source plan + +Each validated generic source document has at most one stored chunk plan. The +source document's canonical digest selects that plan. Pipeline identity, +selected lanes, configured chunk module and options, references, validators, +prompts, schemas, and LLM settings do not participate in lookup. + +In the default mode, the first accepted plan for a source is reused by later +pipelines and invocations. A configured chunk module runs only when no valid +stored plan exists, when reuse is bypassed for one invocation, or when the +operator requests refresh. + +### Plan and annotation contract + +A plan contains an ordered, non-empty collection of inclusive source-unit +ranges. The framework validates these boundaries and deterministically +materializes runtime chunks from the current source document. + +Chunk modules may attach optional domain annotations at plan or range scope. +Annotations are namespaced, validated JSON values. They are preserved through +storage and materialization but are never a hard cross-domain capability. +Downstream modules may rely on the generic chunk contract only. + +The store contains boundaries, annotations, warnings, and provenance. It does +not contain fully materialized chunks or duplicated source-unit content. + +### Persistence + +Chunk plans use a dedicated cache root that is independent of +`workspace.directory` and the locations of checkpoints, diagnostics, debug +artifacts, and durable output. The default is the platform-appropriate per-user +cache directory: on Linux, `$XDG_CACHE_HOME/notarius/chunk-plans` when +`XDG_CACHE_HOME` is set to a valid absolute path, otherwise +`$HOME/.cache/notarius/chunk-plans` when it is unset, through the platform +cache-directory resolver. An invalid relative `XDG_CACHE_HOME` is an error, not +a fallback. + +Operators may set `workspace.chunk_cache.directory` to replace that root. The +recommended system-wide setting for a dedicated Linux service account is +`/var/cache/notarius/chunk-plans`. This is an operational recommendation, not +the application default; the service account must own the directory and it must +not be shared across mutually untrusted users. + +One mutable, versioned plan file is stored beneath the chunk-plan root under the +full canonical source digest. Normal publication and refresh replace that file +atomically; history, rollback, multiple variants, and content deduplication are +outside this feature. + +Plan state is potentially sensitive. Paths are confined, directories and files +use restrictive permissions, default diagnostics omit annotation and source +payloads, and debug output remains explicitly opt-in. + +### Runtime policy + +The effective chunk-cache mode is one of: + +- `auto`: load a valid stored plan; otherwise generate, validate, and publish + one; +- `bypass`: do not read or write plan state for this invocation; or +- `refresh`: generate and validate a plan, then atomically replace stored state. + +`auto` is the default. Invalid or incompatible stored state is a reported miss +in `auto`; it is replaced only after a newly generated plan is accepted. A +failed generation never overwrites prior state. Concurrent writes must never +expose partial data and use last-successful-atomic-write semantics. + +The public CLI override is +`--chunk_cache `. Configuration and environment values use +the same three modes, with CLI taking highest precedence. + +### Validation and execution + +Framework plan validation and deterministic materialization run for generated +and reused plans. The current pipeline's configured chunk-validator chain then +validates the materialized chunks. + +On a cache hit, the configured chunk module is constructed during normal +pipeline preparation but its operation is not invoked and it makes no LLM call. +A validator rejection of a structurally valid reused plan is a run outcome; it +does not implicitly authorize rechunking. + +Only the generic `chunks` capability is hard. Domain annotations such as D&D +scene information are opportunistic and cannot be required solely because the +current pipeline selected the module that normally produces them. + +Canonical plans are the sole owner of chunk reuse. Invocation-scoped +checkpoints no longer load or record chunk outputs. Downstream checkpoint +identity continues to depend on the digest of the effective materialized +chunks, so refresh invalidates affected downstream work. + +### Provenance + +Every run distinguishes the chunk module requested by the resolved pipeline +from the producer of the effective stored plan. Durable provenance records the +source and plan digests, plan schema version, effective cache mode and action, +producer module, producer references and LLM profile where applicable, and +non-sensitive producer metadata. + +Default diagnostics record lookup, validation, generation, and publication +decisions without source or annotation payloads. Opt-in debug artifacts may +contain complete plan and annotation material and are treated as sensitive. + +### Deployment documentation + +The implemented configuration reference identifies the dedicated directory +field, its environment override, and the per-user default. The operations guide +documents the resulting filesystem layout and permissions, and recommends +`/var/cache/notarius/chunk-plans` for a system-wide Linux deployment running as +a dedicated service account. It also explains that cache deletion is +recoverable but may repeat expensive chunk generation. Neither document +presents the system-wide path as the default for an ordinary unprivileged +invocation. + +## Compatibility Policy + +Existing fully materialized chunk checkpoint files are not migrated or promoted +to canonical plans. They are ignored for chunk reuse after this feature lands. +The first `auto` run generates the source's plan through the configured chunk +module. + +The existing source digest is the canonical lookup identity. Any source change +that alters that digest creates a separate plan. Earlier plan schema versions, +malformed files, and digest mismatches are incompatible and follow the invalid +`auto`-miss behavior. + +## Non-Goals + +This feature does not provide: + +- plan editing, comparison, history, rollback, or garbage collection; +- remote or shared plan storage; +- multiple active or automatically selected plan variants for one source; +- mandatory domain annotation contracts; +- automatic rechunking because configuration or model inputs changed; or +- a guarantee that an LLM provider will report cache hits. + +## Completion Outcomes + +The feature is complete when independent runs over the same canonical source +reuse byte-stable materialized chunks across pipeline, lane, reference, +chunk-module, and LLM configuration changes; bypass and refresh obey their +documented state semantics; provenance identifies the effective producer; +legacy chunk checkpoints cannot compete with plan reuse; and all focused, +integration, compatibility, CLI, and repository-wide validation passes. The +configuration and operations references must also document both the per-user +default and the recommended system-wide Linux setting. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..0158b7e --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,965 @@ +# ADR-0005 Staged Implementation Plan + +This document is the executable implementation plan for the target state in +[ADR-0005 Feature Roadmap](adr0005.md), governed by +[ADR-0005](../adr/0005-cache-canonical-chunk-plans-by-source.md). The feature is +not implemented. + +The audience is an LLM coding agent. Implement the stages in order. Each stage +is deliberately scoped to finish with a compiling, tested repository and may be +assigned as one implementation prompt. + +## Execution Rules + +Before every stage: + +1. read [Development](../development.md), both documents under `docs/policy/`, + ADR-0005, this plan, and every stage-specific document named below; +2. inspect `git status` and preserve user changes; +3. inspect the focused package contracts and tests before editing; and +4. confirm that all earlier stages are complete. + +During every stage: + +- implement only that stage and prerequisites discovered to be inseparable; +- use existing package boundaries and helpers before introducing abstractions; +- keep framework and core packages independent of concrete D&D modules; +- preserve cancellation, deterministic ordering, redaction, path confinement, + and atomic-write invariants; +- add or update focused tests with every behavior change; +- do not update current-behavior documentation before the final documentation + stage, except for ADR acceptance in Stage 1; and +- do not begin the next stage after meeting the current exit criteria. + +At the end of every stage: + +1. run the focused tests listed for that stage; +2. run `go test ./...`; +3. run `go vet ./...`; +4. run `go build ./cmd/notarius`; +5. run `git diff --check`; and +6. mark the stage complete in this document only after all checks pass. + +If repository reality conflicts with this plan, stop and update the plan in the +same change before implementing a materially different design. Do not silently +invent a new contract. + +## Fixed Decisions + +These choices are settled for this implementation: + +- `source.SourceDocument.Digest` is the cache lookup identity. +- The chunk-plan root is independent of `workspace.directory` and every other + state root. +- Its default joins the directory returned by `os.UserCacheDir` with + `notarius/chunk-plans`. +- `workspace.chunk_cache.directory` overrides that default and names the + chunk-plan root itself. The recommended value for a system-wide Linux service + is `/var/cache/notarius/chunk-plans`. +- The plan file is + `/<64-character-lowercase-source-sha256-hex>/plan.json`. The + validated `sha256:` prefix remains part of the logical digest but is removed + from the path segment. +- The plan envelope schema is `notarius.chunk-plan.v1`. +- There is one mutable plan file per source, atomically replaced. There are no + immutable plan objects, pointers, history, locks, or rollback. +- Concurrent writers use last-successful-atomic-write semantics; they must + never expose a partial file. +- Chunk modules retain the `contracts.Chunker` name but replace `Chunk` with a + `Plan` operation. No legacy interface remains after Stage 2. +- Plans and ranges use `map[string]json.RawMessage` annotations. Annotation + values are canonical valid JSON and namespaces are non-empty trimmed strings. +- Materialized chunks expose range annotations through + `source.Chunk.Annotations` and plan annotations through + `source.Chunk.PlanAnnotations`, never through `Metadata`. +- Framework materialization owns chunk IDs, indexes, references, content, + media type, units, and generic metadata. +- Only the generic `chunks` capability is hard; `chunks.scenes` is removed. +- Cached-plan hits still construct the configured chunker during preparation + but never call its operation or the chunk-stage LLM. +- The current pipeline's configured chunk validators run against materialized + chunks on every path. +- Canonical plan storage is the only chunk-reuse mechanism; chunk checkpoint + APIs and payloads are removed. +- The mode values are `auto`, `bypass`, and `refresh`. They are valid in config, + environment, and CLI, including persistent `refresh` configuration. +- The YAML field is `workspace.chunk_cache.mode`. +- The YAML directory field is `workspace.chunk_cache.directory`. +- The environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`. +- The root environment variable is `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR`. +- The CLI flag is `--chunk_cache `. +- Precedence is CLI, environment, file, then default `auto`. +- Chunk-plan-root precedence is environment, file, then the per-user default; + there is no CLI root override. +- Every supplied file, environment, and CLI value is parsed strictly even when + a higher-precedence source would override it; precedence selects among valid + values and does not mask malformed configuration. +- Existing chunk checkpoint files are ignored and never migrated. +- A structurally invalid stored plan is an `auto` miss. A configured-validator + rejection of a structurally valid hit does not trigger regeneration. + +## Planned Contracts + +Use these names unless an existing collision requires the smallest obvious +adjustment. + +### Source-zone types + +Add to `internal/core/source`: + +```go +type ChunkAnnotations map[string]json.RawMessage + +type ChunkPlan struct { + SourceDigest string `json:"source_digest"` + Ranges []ChunkRange `json:"ranges"` + Annotations ChunkAnnotations `json:"annotations,omitempty"` +} + +type ChunkRange struct { + StartUnitID int `json:"start_unit_id"` + EndUnitID int `json:"end_unit_id"` + Annotations ChunkAnnotations `json:"annotations,omitempty"` +} +``` + +Add `Annotations ChunkAnnotations` and +`PlanAnnotations ChunkAnnotations` to `source.Chunk`, with JSON names +`annotations` and `plan_annotations` and `omitempty` on both. + +The plan digest covers `SourceDigest`, ordered ranges, and canonical annotation +bytes. It excludes producer provenance, warnings, timestamps, and storage +schema. Materialization produces: + +- IDs `chunk-%06d`, numbered from one; +- zero-based indexes in plan order; +- references spanning the first and last units in each range; +- cloned contiguous source units; +- media type `application/json`; +- canonical JSON content shaped as `{"units":[...]}`; +- generic metadata containing `start_unit_id`, `end_unit_id`, and `unit_count`; +- cloned range annotations in `Chunk.Annotations`; and +- cloned plan annotations in `Chunk.PlanAnnotations`. + +The plan and range scopes remain distinct even when both use the same namespace. +Copying plan annotations into each runtime chunk is intentional: it makes them +available to downstream chunk-scoped operations without duplicating source +content in durable storage. + +Framework validation permits gaps and overlap between ranges, but requires +strictly increasing start positions and rejects duplicate or backward ranges. +Producing modules may impose stricter policy before returning a plan. + +### Module contract + +Replace the current chunk result and operation with: + +```go +type ChunkPlanResult struct { + Plan source.ChunkPlan + Warnings []Warning +} + +type Chunker interface { + Key() string + ReferenceSlots() []ReferenceSlot + Plan(context.Context, ChunkRequest) (ChunkPlanResult, error) +} +``` + +Keep `ChunkRequest`, chunk module keys, stage name `chunk`, configuration +bindings, registries, and preparation ownership unchanged. + +### Durable plan record + +Define framework-facing record and store contracts beside other pipeline +collaborator contracts: + +```go +type ChunkPlanProducer struct { + InputModule string + ChunkModule string + LLMProfile string + References []artifacts.ReferenceProvenance + Metadata map[string]any +} + +type ChunkPlanRecord struct { + SchemaVersion string + SourceDigest string + PlanDigest string + Plan source.ChunkPlan + Producer ChunkPlanProducer + Warnings []contracts.Warning + CreatedAt time.Time +} + +type ChunkPlanStore interface { + Load(sourceDigest string) (ChunkPlanRecord, ChunkPlanDecision, error) + Save(ChunkPlanRecord) error +} + +type ChunkPlanStoreFactory func(root string) (ChunkPlanStore, error) + +type ChunkPlanDecision struct { + Status ChunkPlanStatus + Reason string +} + +type ChunkPlanStatus string + +const ( + ChunkPlanHit ChunkPlanStatus = "hit" + ChunkPlanMissing ChunkPlanStatus = "missing" + ChunkPlanInvalid ChunkPlanStatus = "invalid" +) +``` + +`ChunkPlanDecision` reports `hit`, `missing`, or `invalid` plus a redacted +reason. A miss or invalid result returns a zero record and nil error. Missing +and structurally invalid records are recoverable in `auto`. Filesystem access +failures other than absence return an error and are framework failures. + +The concrete constructor is: + +```go +// Package chunkplan +func NewFilesystemStore(root string) (pipeline.ChunkPlanStore, error) +``` + +It treats `root` as the exact chunk-plan root. CLI composition injects this +constructor through `cli.Options.ChunkPlanStoreFactory`; tests may inject a +recording factory. `cli.Options.UserCacheDir` has signature +`func() (string, error)` and defaults to `os.UserCacheDir`. + +The JSON envelope uses snake-case equivalents of the record fields. It stores +references as provenance only, never reference content. Before returning a hit, +the loader validates schema version, source digest, plan digest, plan structure, +annotation JSON, and materializability against the current source in the runner. + +### Cache policy and provenance + +Add a `ChunkCacheMode` enum with `auto`, `bypass`, and `refresh`, plus strict +parsing and validation. + +Add to `RunInput`: + +```go +ChunkCacheMode ChunkCacheMode +ChunkPlans ChunkPlanStore +``` + +An empty mode used by direct framework callers means `bypass`; the production +CLI always supplies the effective non-empty mode. + +Add a run-manifest summary: + +```go +type ChunkPlanManifest struct { + Mode string `json:"mode"` + Action string `json:"action,omitempty"` + SourceDigest string `json:"source_digest,omitempty"` + PlanDigest string `json:"plan_digest,omitempty"` + PlanSchemaVersion string `json:"plan_schema_version,omitempty"` + RequestedModule string `json:"requested_module"` + ProducerInputModule string `json:"producer_input_module,omitempty"` + ProducerModule string `json:"producer_module,omitempty"` + ProducerLLMProfile string `json:"producer_llm_profile,omitempty"` + ProducerReferences []ReferenceProvenance `json:"producer_references,omitempty"` + ProducerMetadata map[string]any `json:"producer_metadata,omitempty"` + CreatedAt *time.Time `json:"created_at,omitempty"` +} +``` + +`Action` is one of `reused`, `generated`, `refreshed`, or `bypassed`. Retain +the existing top-level `input_module` and `chunker` fields as the requested +pipeline modules. Add this field to `artifacts.RunManifest`: + +```go +ChunkPlan *ChunkPlanManifest `json:"chunk_plan,omitempty"` +``` + +The runner initializes it with the effective mode and requested chunk module, +then fills candidate and producer fields when a plan reaches materialization. + +## Stage 1: Finalize ADR and add the source-zone plan model + +**Status:** Not started + +### Objective + +Accept the architectural decision and introduce domain-neutral plan, +annotation, validation, digest, clone, and materialization primitives without +changing chunk-module execution yet. + +### Read first + +- `internal/core/source/source.go` +- `internal/core/source/digest.go` +- `internal/core/source/validation.go` +- `internal/core/source/source_test.go` +- every current `source.Chunk` clone or serialization helper found with `rg` + +### Implement + +1. Revise ADR-0005 before acceptance: + - confirm it retains one source-addressed mutable plan and atomic refresh; + - preserve the independent cache-surface decision; + - keep filesystem and CLI details out of the ADR; and + - change status from `Proposed` to `Accepted`. +2. Add `ChunkAnnotations`, `ChunkPlan`, and `ChunkRange` in + `internal/core/source`. +3. Add canonical annotation validation and cloning. Canonicalize each raw JSON + value by decoding with `json.Decoder.UseNumber` and re-encoding; reject + blank namespaces, invalid JSON, trailing JSON values, and non-finite or + unsupported values. +4. Add plan validation against a `SourceDocument` using the rules in Planned + Contracts. +5. Add `DigestChunkPlan` and deterministic `MaterializeChunkPlan`. +6. Add `Annotations` and `PlanAnnotations` to `source.Chunk`, `DigestChunk`, + every clone helper, debug envelope, and serialized chunk-validation + representation. Ensure all ownership boundaries clone raw bytes and retain + the two scopes separately. +7. Do not change the chunker interface or runner flow in this stage. + +### Tests + +Add focused tests for: + +- annotation canonicalization, invalid JSON, blank namespaces, and mutation + safety; +- plan source mismatch, missing ranges, missing units, backward ranges, + duplicate starts, gaps, and overlap; +- stable plan digests and digest changes for boundary or annotation changes; +- exact materialized IDs, references, content bytes, metadata, annotations, + gaps, and overlap; +- repeated byte-identical materialization; and +- plan- and range-annotation preservation through existing chunk clones, debug + conversion, serialized validation, and downstream extract requests. + +Run at minimum: + +```sh +go test ./internal/core/source +go test ./internal/framework/contracts ./internal/framework/pipeline +``` + +### Exit criteria + +The repository is green; ADR-0005 is accepted; plan primitives are complete and +tested; current modules and the runner still behave as before. + +## Stage 2: Replace the chunk operation with plan generation + +**Status:** Not started + +### Objective + +Make all chunk modules produce plans and make the runner materialize those plans +without adding durable reuse yet. + +### Read first + +- `docs/internal/modules.md` +- `docs/internal/pipeline.md` +- `internal/framework/contracts/contracts.go` +- `internal/framework/pipeline/chunker_registry.go` +- `internal/framework/pipeline/runner.go` +- `internal/framework/pipeline/chunk_validation.go` +- both production chunk-module packages and all chunk fakes in tests + +### Implement + +1. Replace `ChunkResult` with `ChunkPlanResult` and `Chunker.Chunk` with + `Chunker.Plan`; keep the request, registry, module keys, and stage name. +2. Update all fakes, registry tests, preparation tests, integration helpers, and + compile-time interface assertions in the same stage. Do not retain adapters + or a legacy chunk-returning interface. +3. Convert `generic/chunk/units` to emit ranges only. Preserve `max_units` and + `overlap_units` semantics. +4. Convert `dnd/chunk/scenes` to emit namespace `dnd/scenes` at both scopes: + - each range value is an object with exact keys `short_title`, + `primary_mode`, `main_participants`, `summary`, `boundary_note`, and + `boundary_confidence`, preserving the current normalized scene values; + and + - the plan value is an object with exact key `boundary_caveats`, containing + the normalized caveat array. +5. Preserve accepted scene boundary caveats as `contracts.Warning` values as + well as in the plan annotation so warning behavior remains visible. +6. Remove `chunks.scenes` from the D&D scene module's provided capabilities. + Keep only `chunks`. +7. Change the runner’s non-cached chunk path to: + - call `Plan` under the existing retry and debug attempt boundary; + - validate the plan structurally; + - materialize chunks; + - run the configured chunk-validator chain on materialized chunks; + - preserve retry and rejection semantics; and + - pass materialized chunks to lanes exactly as before. +8. Update chunk debug payloads to distinguish the generated plan from + materialized chunks. Plan annotations may appear only in opt-in debug data. +9. Leave current chunk checkpoint loading and recording temporarily in place, + storing materialized chunks, so existing resume tests remain green until + Stage 4 removes that path. + +### Tests + +Update focused module and framework tests to cover: + +- exact generic ranges and overlap; +- exact D&D scene annotations and warning conversion; +- annotations absent from generic plans; +- framework materialization rather than module materialization; +- chunk validators receiving materialized annotated chunks; +- retries around generation plus validation; +- stable chunk IDs independent of producing module; and +- no `chunks.scenes` resolution dependency. + +Run at minimum: + +```sh +go test ./internal/modules/generic/chunk/units +go test ./internal/modules/dnd/chunk/scenes +go test ./internal/framework/contracts ./internal/framework/pipeline +go test ./internal/modules/integration ./internal/cli +``` + +### Exit criteria + +Every chunk module uses the single plan contract, the runner always +materializes, production behavior remains functionally equivalent apart from +framework-owned stable chunk IDs and namespaced annotations, and the repository +is green. + +## Stage 3: Add dedicated cache configuration and the plan store + +**Status:** Not started + +### Objective + +Implement the durable source-addressed store and its independent cache-root +configuration without changing the runner to consume the store. + +### Read first + +- `docs/config.md` +- `docs/operations.md` +- `internal/core/config/*` +- `internal/core/workspace/*` +- `internal/framework/checkpoint/` as the local pattern for a workspace-backed + implementation +- `internal/cli/run.go` configuration loading, without wiring the store yet + +### Implement + +1. Add `ChunkCacheMode` and strict parsing in the pipeline package. +2. Add `WorkspaceChunkCacheConfig` with `Mode` and `Directory` to config and + file config. The YAML fields are `workspace.chunk_cache.mode` and + `workspace.chunk_cache.directory`. +3. Set default mode to `auto`; support + `NOTARIUS_WORKSPACE_CHUNK_CACHE_MODE`; preserve precedence file then + environment. CLI precedence arrives in Stage 5. +4. Support `NOTARIUS_WORKSPACE_CHUNK_CACHE_DIR` with environment-over-file + precedence. Trim the value; an unset or empty effective value selects the + per-user default. Do not add a CLI directory override. +5. Add `workspace.DefaultChunkPlanRoot`, accepting an injectable + user-cache-directory resolver and returning + `filepath.Join(userCacheDir, "notarius", "chunk-plans")`. Production later + supplies `os.UserCacheDir`; tests supply a stub. Reject resolver errors and + empty returned directories. Use this exact signature: + + ```go + func DefaultChunkPlanRoot(userCacheDir func() (string, error)) (string, error) + ``` +6. Treat a non-empty `workspace.chunk_cache.directory` as the plan root itself, + cleaned with the same path rules as other workspace roots. It does not gain + an implicit `chunk-plans` suffix. `workspace.directory` must not affect this + root, and the chunk-cache directory must not affect checkpoint, diagnostics, + debug, or output roots. Do not change `workspace.FromConfig` or + `workspace.Settings` for this feature. +7. Add the plan record, producer, decision, and store interfaces described + above in the pipeline package. +8. Add `internal/framework/chunkplan` with: + - a filesystem implementation; + - a constructor that accepts the already resolved chunk-plan root; + - path `/<64-character-source-sha256-hex>/plan.json`, after + strictly validating and stripping the logical `sha256:` prefix; + - schema `notarius.chunk-plan.v1`; + - directories mode `0700` and file mode `0600`; + - atomic same-directory temp-file rename; + - strict envelope decoding and unknown-field rejection; + - complete digest and annotation validation; and + - no raw reference content. +9. `Load` distinguishes missing, invalid, and operational error. `Save` always + atomically replaces and implements last-successful-write semantics. +10. Keep store fakes local to their tests; do not add a production in-memory or + no-op store because `bypass` and a nil collaborator already express + disabled persistence. +11. Do not wire the store into `Runner` or CLI execution in this stage. + +### Tests + +Cover: + +- config defaults, YAML decoding, environment override, invalid values, + cloning, redaction, and validation; +- directory environment-over-file precedence, empty-value fallback, cloning, + redaction, cleaning, and validation; +- exact injected per-user default and resolver failure or empty-result errors; +- independence in both directions between `workspace.directory` and + `workspace.chunk_cache.directory`; +- exact acceptance of `/var/cache/notarius/chunk-plans` as a configured root + without writing to that real path; +- safe full-digest paths and rejection of malformed digests; +- `0700`/`0600` permissions where supported; +- envelope round trip, strict decoding, source and plan digest mismatch, + invalid annotations, schema mismatch, missing files, operational read errors, + atomic replacement, and concurrent writers; and +- absence of raw reference content in persisted JSON. + +Run at minimum: + +```sh +go test ./internal/core/config ./internal/core/workspace +go test ./internal/framework/chunkplan ./internal/framework/pipeline +go test ./internal/cli +``` + +### Exit criteria + +Configuration and storage are complete and independently tested, but normal +runs still use Stage 2’s generate-and-materialize behavior. + +## Stage 4: Integrate cache policy and remove chunk checkpoints + +**Status:** Not started + +### Objective + +Make the runner’s chunk path use the canonical plan store under all three modes +and establish it as the only chunk-reuse mechanism. + +### Read first + +- `internal/framework/pipeline/runner.go` +- `internal/framework/pipeline/checkpoint.go` +- `internal/framework/pipeline/runner_concurrent.go` +- `internal/framework/checkpoint/loader.go` +- `internal/framework/checkpoint/recorder.go` +- focused runner retry, rejection, debug, and checkpoint tests + +### Implement + +1. Add `ChunkCacheMode` and `ChunkPlans` to `RunInput`. Empty mode means + `bypass` for direct framework callers. +2. Add synchronized store wrapping only if runner access can be concurrent; + otherwise document and test the intentionally serial chunk-plan access. +3. Implement mode behavior: + - `auto` hit: load, structurally validate against the current source, + materialize, then run configured chunk validators once; + - `auto` miss or invalid: generate through existing retries, validate, + materialize, validate chunks, then save; + - `bypass`: generate and validate normally without calling `Load` or `Save`; + - `refresh`: generate and validate normally, then replace through `Save`. +4. A valid-hit validator rejection is recorded without retry, store mutation, + or implicit regeneration. A store I/O error is a framework error. +5. A failed or rejected generated plan is never saved. An invalid old file is + overwritten only after a replacement passes module execution, structural + validation, materialization, and configured validators. +6. Build producer provenance from the prepared input module key, prepared chunk + module key, resolved chunk-stage LLM profile, chunk-target auxiliary + reference provenance, and chunk-module manifest metadata. Clone and redact + according to existing boundaries. +7. Replay stored producer warnings on a hit, then append current validation + warnings. Do not duplicate warnings from discarded generation attempts. +8. Remove all chunk methods and types from `CheckpointRecorder`, + `CheckpointLoader`, synchronized wrappers, workspace recorder/loader, and + checkpoint manifests and envelopes. +9. Remove runner chunk-checkpoint decisions and events. Retain source, extract, + merge, and normalize checkpoints. Keep downstream fingerprints based on the + digest of materialized chunks. +10. Ignore existing on-disk `chunk/` checkpoint files; do not read, delete, or + migrate them. +11. Update debug behavior: + - hit records lookup and materialization but no chunk module attempt or LLM + call; + - generated, bypassed, and refreshed paths retain attempt debug; and + - default debug envelopes remain redacted. + +### Tests + +Add table-driven runner tests for every mode and outcome: + +- hit, missing, invalid, and I/O error; +- generation success, error, rejection, retry, and cancellation; +- validator warning, rejection, and error on hits and generated plans; +- bypass performs zero store calls; +- refresh performs no load and exactly one successful save; +- invalid record remains untouched when replacement fails; +- stored warnings replay once; +- no module or LLM call on hit; +- downstream chunk digest changes on refresh; +- legacy chunk checkpoint files are ignored; and +- source and lane checkpoint behavior remains intact. + +Run at minimum: + +```sh +go test ./internal/framework/pipeline +go test ./internal/framework/checkpoint +go test ./internal/modules/integration +``` + +### Exit criteria + +The runner implements complete cache semantics, chunk checkpoints no longer +exist as a code path, all direct framework tests are green, and production CLI +runs are not yet wired to persistent plan storage. + +## Stage 5: Wire persistent policy into the CLI + +**Status:** Not started + +### Objective + +Make normal CLI runs resolve the effective mode and dedicated chunk-plan root, +construct the durable plan store when appropriate, and pass it into the runner. + +### Read first + +- `docs/cli.md` +- `docs/config.md` +- `cmd/notarius/main.go` and `internal/cli/run.go` +- CLI flag, environment, invocation-metadata, and exit-code tests +- the workspace and configuration code changed in Stage 3 + +### Implement + +1. Add the exact flag `--chunk_cache `. Represent an + omitted CLI value separately from an explicit value so precedence is not + inferred from the default printed by the flag package. +2. Resolve the effective mode in this order: explicit CLI flag, environment, + configuration file, default `auto`. An invalid CLI flag is a usage error + with exit code 2; invalid file or environment configuration follows the + existing configuration-load failure path and exit code 1. +3. Include an explicitly requested CLI value in invocation metadata and the + resolved mode in effective-configuration metadata without adding field-level + origin tracking or plan and annotation payloads. Name the optional + invocation JSON field `chunk_cache_override`; the resolved mode already + appears at `workspace.chunk_cache.mode` in the redacted effective config. +4. Add `UserCacheDir func() (string, error)` and + `ChunkPlanStoreFactory pipeline.ChunkPlanStoreFactory` to `cli.Options`, + defaulting them to `os.UserCacheDir` and + `chunkplan.NewFilesystemStore`. For `auto` and `refresh`, choose the non-empty + effective `workspace.chunk_cache.directory`; otherwise resolve the per-user + default with `workspace.DefaultChunkPlanRoot`. Invoke the store factory with + that exact root and pass the result and effective mode through `RunInput`. + Never append `chunk-plans` to an explicit directory. The injectable factory + lets CLI tests assert root selection without writing to production paths. +5. For `bypass`, do not resolve, inspect, or create the chunk-plan root and pass + no store. Existing general-workspace setup remains unchanged and independent: + `workspace.directory`, resume, diagnostics, and debug continue to use their + current paths and semantics. +6. Keep resume policy independent of plan reuse. Enabling, disabling, or + selecting a resume run must not change the chunk-cache mode or path. +7. Configuration validation and pipeline-listing commands validate the mode but + do not create workspace directories or plan files. +8. Update CLI usage text only as needed to expose the new flag. Defer all + narrative documentation to Stage 8. + +### Tests + +Add CLI and integration tests for: + +- a default `auto` run writing a plan and a second independent invocation + reusing it without a chunk operation or chunk LLM call; +- file, environment, and CLI precedence, including persistent `refresh` in + file and environment configuration; +- `bypass` performing no plan-store access and resolving or creating no default + chunk-plan root; +- invalid flag, environment, and file values with their established error + classifications; +- an explicit chunk-plan root from file and environment, including exact use of + `/var/cache/notarius/chunk-plans` without an appended suffix, using a + recording store factory and never writing to that real path; +- an explicit root succeeding without calling a failing per-user cache resolver; +- an injected per-user default resolving exactly to + `/notarius/chunk-plans`; +- `workspace.directory` changes having no effect on chunk-plan placement and a + chunk-cache directory having no effect on other workspace state; +- per-user cache-root discovery failure under `auto`, `refresh`, and `bypass`, + including the allowed no-state `bypass` case, plus unwritable-store failures; +- resume plus each cache mode, showing independent behavior; +- validation and listing commands having no filesystem side effects; and +- reuse across changed pipeline, chunk module/options, references, validators, + and LLM settings when the source digest is unchanged. + +Run at minimum: + +```sh +go test ./internal/cli +go test ./internal/framework/pipeline ./internal/framework/chunkplan +go test ./internal/modules/integration +``` + +### Exit criteria + +The production CLI implements the complete mode and precedence contract, +independent invocations reuse source-addressed plans by default, bypass and +refresh have exact state semantics, and the repository is green. + +## Stage 6: Expose producer provenance and safe diagnostics + +**Status:** Not started + +### Objective + +Make every output and diagnostic record distinguish the requested chunk module +from the producer of the effective plan without exposing sensitive payloads by +default. + +### Read first + +- `docs/integrations/json-output.md` +- `docs/internal/diagnostics.md` +- manifest assembly in `internal/framework/pipeline/runner.go` +- `internal/core/artifacts/artifacts.go` +- current diagnostics and debug-envelope implementations and tests + +### Implement + +1. Add `ChunkPlanManifest` to the run manifest and all required clone, + conversion, and serialization paths. +2. Populate `mode` and exactly one action after a candidate plan has passed + structural validation and materialization: + - `reused` for an `auto` hit; + - `generated` for an `auto` miss or invalid record; + - `refreshed` for generation under `refresh`; and + - `bypassed` for generation under `bypass`. + Retain that action if the current chunk-validator chain rejects the + candidate; publication still follows Stage 4's accept-only rule. If no + candidate reaches materialization, leave action and candidate plan fields + empty in the partial failure manifest and report the failure in diagnostics. +3. Preserve the existing top-level `chunker` and its module metadata as the + requested module. Never overwrite those fields with cached producer data. +4. Populate producer module, LLM profile, references, metadata, creation time, + source digest, plan digest, and schema from the effective stored or newly + generated record. A reused record reports its original producer, even when + it differs from the requested module. +5. Define producer LLM profile as the resolved chunk-stage profile used during + generation. Leave it empty for producers that did not use an LLM; do not + substitute the current run's requested profile on reuse. +6. Keep producer references as existing redacted provenance records. Deep-clone + metadata, references, warnings, annotations, and raw JSON at every ownership + boundary. +7. Add a redacted `chunk-plan.json` diagnostic artifact, backed by a summary on + `RunOutput` so the CLI can write it on success, rejection, or framework + failure. It contains mode, source and candidate plan digests, requested + module, lookup status and redacted reason, action, validation status, and + publication status. Use these closed values: + - lookup: `hit`, `missing`, `invalid`, or `skipped`; + - validation: `not_run`, `approved`, `approved_with_warnings`, `rejected`, + or `error`; and + - publication: `not_requested`, `published`, or `not_published`. + It contains no source units, materialized chunk content, annotation bytes, + reference content, prompts, model responses, or raw invalid-file content. +8. Keep complete plan, annotations, attempts, and materialized chunks confined + to existing opt-in debug artifacts and apply their normal restrictive file + handling. +9. Preserve JSON-output compatibility by adding fields rather than renaming + unrelated existing fields. Omit producer-only optional values when absent in + the same style as the surrounding manifest schema. + +### Tests + +Cover: + +- all four actions and their exact manifest fields; +- requested and producer modules differing on a hit; +- producer metadata remaining stable when current configuration changes; +- LLM-backed and non-LLM producers; +- deep-clone and mutation safety for raw annotations and metadata; +- default diagnostic redaction, including invalid stored files; +- complete data appearing only under explicit debug configuration; and +- JSON serialization and compatibility fixtures. + +Run at minimum: + +```sh +go test ./internal/core/artifacts +go test ./internal/framework/pipeline +go test ./internal/cli +``` + +### Exit criteria + +Manifests and diagnostics explain which plan was requested, selected, and +produced; default output remains redacted; debug behavior is explicit; and all +compatibility tests pass. + +## Stage 7: Add cross-run, corruption, and concurrency hardening + +**Status:** Not started + +### Objective + +Exercise the completed design across package boundaries and adversarial state, +then close any correctness gaps without changing the settled contract. + +### Read first + +- all tests added in Stages 1–6 +- end-to-end CLI and module integration test harnesses +- atomic-write, cancellation, retry, and race-sensitive framework code +- repository import-boundary tests + +### Implement + +1. Add black-box tests using separate CLI application instances and a shared + temporary chunk-plan root. Prove reuse across different pipelines, + configured chunk modules, options, references, lanes, validators, and LLM + profiles. +2. Prove that a changed canonical source digest selects a different path, while + byte-identical canonical sources reuse the same plan. +3. Exercise generic plans through D&D pipelines and D&D-annotated plans through + generic pipelines. Optional annotations must survive, while no downstream + component may require `dnd/scenes` merely because the requested chunker is + the D&D scene module. +4. Add corruption cases for truncated JSON, unknown fields, schema mismatch, + source mismatch, plan-digest mismatch, bad boundaries, invalid annotations, + and materialization failure. Verify `auto` regeneration and non-destructive + failure behavior exactly. +5. Add concurrent reader/writer and writer/writer tests. Readers observe either + the old complete envelope or the new complete envelope; after successful + writers, the final file is one complete valid record. Do not assert which + writer wins. +6. Exercise interruption before save, during temp-file creation, and around + rename where the implementation permits deterministic fault injection. + Prior valid state must remain readable unless a complete replacement was + published. +7. Verify cancellation, retry limits, warning order, error classification, and + debug ordering on misses and hits. No cache hit may consume chunk-stage retry + budget or invoke the module. +8. Verify legacy chunk checkpoint files cannot influence reuse and that lane + resume still keys from effective materialized chunk digests after refresh. +9. Extend import-boundary tests so `internal/core/source` owns no framework or + module dependencies and the generic store owns no concrete module imports. +10. Ensure examples and all tests isolate workspace state in temporary + directories and isolate chunk-plan state with a temporary configured root + or explicitly select `bypass`; tests must never write to the real per-user + default or `/var/cache/notarius/chunk-plans`. +11. Run the race detector over the packages that access the plan store and + runner concurrently, and fix any race within feature scope. + +### Tests + +In addition to the new focused tests, run: + +```sh +go test ./... +go test -race ./internal/framework/chunkplan ./internal/framework/pipeline +go vet ./... +go build ./cmd/notarius +``` + +### Exit criteria + +The feature is demonstrated across independent runs, corruption and concurrent +publication cannot expose partial state, cross-domain annotations remain +optional, cancellation and retry behavior is deterministic, and all repository +and race checks pass. + +## Stage 8: Publish current-behavior documentation + +**Status:** Not started + +### Objective + +After implementation is complete, make the supported behavior discoverable +without duplicating canonical ownership across documents. + +### Read first + +- `docs/policy/architecture.md` +- `docs/policy/documentation.md` +- every current-behavior document linked below +- the final code, flags, schemas, examples, and tests from Stages 1–7 + +### Implement + +1. Update `docs/policy/architecture.md` with the + plan-generation/materialization + boundary, source-addressed reuse, and the independent cache-state surface at + the level appropriate for the canonical system overview. +2. Update `docs/internal/pipeline.md` with mode behavior, hit/miss control flow, + validator placement, retry behavior, and removal of chunk checkpoints. +3. Update `docs/internal/modules.md` with the `Chunker.Plan` contract, generic + boundaries, optional annotation namespaces, and framework materialization. +4. Update `docs/internal/overview.md` with the new plan-store package and + current state ownership, and remove chunk checkpoint ownership from its + checkpoint description. Keep detailed checkpoint and plan execution flow in + `docs/internal/pipeline.md` rather than creating a parallel internal state + reference. +5. Update `docs/config.md` with `workspace.chunk_cache.mode`, + `workspace.chunk_cache.directory`, both environment variables, strict + values, precedence, and the per-user cache-root default. State explicitly + that `workspace.directory` does not affect plan placement and that the + configured directory is the root itself. Keep field-level details owned + here. Document that Unix `os.UserCacheDir` rejects a relative + `XDG_CACHE_HOME` and that this is a configuration error rather than a fallback. +6. Update `docs/cli.md` with `--chunk_cache` semantics and examples. Link to + configuration rather than restating its entire precedence model. +7. Update `docs/operations.md` with plan location, permissions, invalid-state + recovery, refresh and bypass operations, concurrency semantics, sensitive + debug data, deletion as a recoverable but potentially expensive + regeneration event, and the no-history/no-rollback limitation. Document both: + - the ordinary Linux per-user default, + `$XDG_CACHE_HOME/notarius/chunk-plans` when `XDG_CACHE_HOME` is a valid + absolute path, or `$HOME/.cache/notarius/chunk-plans` when it is unset; and + - `/var/cache/notarius/chunk-plans` as the recommended configured root for a + system-wide Linux deployment running under a dedicated service account. + Explain that the system path is not the unprivileged default, show the + following minimal configuration, require the operator or package installer + to provision it with restrictive service-account ownership and permissions, + and warn against sharing the cache across mutually untrusted users: + + ```yaml + workspace: + chunk_cache: + directory: /var/cache/notarius/chunk-plans + ``` +8. Update `docs/integrations/json-output.md` and + `docs/internal/diagnostics.md` with requested versus producer provenance, + actions, warnings, and redaction boundaries. +9. Keep package-level plan/range/annotation and materialization invariants in + `docs/internal/pipeline.md`; no separate source document is needed unless + the implemented subsystem becomes too large for that canonical owner. +10. Update examples only when they materially clarify operation; examples must + use temporary or explicitly configured workspace and chunk-cache paths. +11. Remove future-tense disclaimers from the feature roadmap, mark its + completion outcomes as achieved, and mark all completed stages in this + document. Keep the roadmap as historical target-state context rather than + copying implementation details into it. +12. Validate every changed link, heading, code symbol, flag, environment name, + path, and JSON field against the implementation. Search for stale + `Chunker.Chunk`, `ChunkResult`, `chunks.scenes`, and chunk-checkpoint claims. + +### Tests and validation + +Run: + +```sh +go test ./... +go vet ./... +go build ./cmd/notarius +git diff --check +``` + +Also run repository link or documentation checks if present, and manually +inspect the reading map in `docs/development.md` and component links in +`docs/internal/overview.md` for canonical ownership and navigation consistency. + +### Exit criteria + +All canonical documents describe implemented behavior, operational guidance is +complete and non-duplicative, no stale chunk API or checkpoint documentation +remains, roadmap status is accurate, and the repository is green.