Document chunk plan caching and provenance

docs/internal/diagnostics.md

@@ -22,10 +22,18 @@ constructor.
 
 Typed methods on `RunDirectory` write invocation metadata, redacted effective
 configuration, resolved pipeline/reference data, checkpoint events, source data
-when explicitly requested, manifests, reports, warnings, and error text. The
+when explicitly requested, manifests, reports, warnings, redacted chunk-plan
+summaries, and error text. The
 current filenames and their operator-facing contents are listed in
 [Operations](../operations.md#diagnostics-directory).
 
+The chunk-plan summary records the effective mode, source and candidate
+digests, requested module, lookup decision, materialization action, validation
+decision, and publication decision. Its closed decision values make failures
+and recoverable invalid records inspectable without serializing plan ranges,
+annotations, source content, reference content, prompts, model responses, or
+raw invalid-file bytes.
+
 JSON methods indent their payload and append a newline. All artifact writes use
 a temporary file in the target directory, apply the requested permissions, and
 rename it into place. Artifact resolution accepts only a single relative base
@@ -60,7 +68,8 @@ pipeline results, and the final report. This ordering permits later failures to
 retain the context already established.
 
 Failures before construction have no `RunDirectory`. Later failures write an
-error log, preserve any available partial manifest, and apply a failed-run
+error log, preserve any available partial manifest and chunk-plan summary, and
+apply a failed-run
 retention decision. A diagnostics write failure is itself a command failure so
 the CLI does not report success after losing requested inspection data.
 

docs/internal/modules.md

@@ -67,34 +67,41 @@ rules are defined in the
 
 ## Chunkers
 
+Chunkers implement `contracts.Chunker.Plan`. A plan identifies ordered source
+unit ranges and may carry optional namespaced JSON annotations; it does not
+contain materialized chunk content. The framework canonicalizes annotations,
+validates ranges against the current source, and materializes chunk IDs,
+indexes, references, content, units, and generic metadata. Annotation
+namespaces remain optional data: generic framework code and downstream modules
+must not require D&D scene annotations or import `dnd/scenes`.
+
 ### `internal/modules/generic/chunk/units`
 
-The generic chunker validates the source document, walks units in configured
-windows, clones each selected unit, and emits deterministic ordered chunk IDs.
-Overlap changes the next window start but never reorders units. It records the
-first and last unit and unit count in chunk metadata, and derives the chunk's
-canonical source reference from those unit references.
+The generic chunker validates the source document and returns ranges over units
+in configured windows. Overlap changes the next window start but never reorders
+units. Framework materialization derives the resulting chunk identity and
+generic metadata from those ranges.
 
 The accepted options and defaults are defined in
 [Configuration](../config.md#implemented-production-modules). Generic
 framework validation canonicalizes the returned unit slices before extraction.
 The chunker decodes its options during construction and retains only the typed
-window settings used by `Chunk`.
+window settings used by `Plan`.
 
 ### `internal/modules/dnd/chunk/scenes`
 
 The scene chunker prepares a structured Scriptorium request from the full
 transcript, session, and optional D&D reference inputs. It validates the model's
 scene boundaries against source-unit IDs and converts them into deterministic
-chunks with canonical source references spanning each scene's units.
-Preparation injects the shared structured LLM client into the chunker; `Chunk`
+plan ranges with optional scene annotations. Preparation injects the shared
+structured LLM client into the chunker; `Plan`
 supplies only the run-specific profile, session, source, references, and
 metadata.
 
 Scene validation requires sequential, contiguous, non-overlapping coverage from
-the first source unit through the last. Each chunk contains JSON scene content
-and module-owned metadata for the scene description, boundaries, confidence,
-participants, and unit count. Boundary caveats become warnings. Malformed
+the first source unit through the last. Scene descriptions, boundaries,
+confidence, and participants are module-owned annotations. Boundary caveats
+become warnings. Malformed
 structured output is returned as an error; there is no fallback chunker.
 
 The package embeds its prompt and response schema and reports their non-secret

docs/internal/overview.md

@@ -35,7 +35,7 @@ normalize continuations that may overlap across lanes.
 | `internal/core/config` | Defaults, YAML parsing, environment overrides, validation, redaction, and effective pipeline resolution. |
 | `internal/core/diagnostics` | Scoped run directories, diagnostics writers, atomic writes, and retention decisions. |
 | `internal/core/source` | Generic source documents, units, chunks, canonical references, lookup, validation, and deterministic source digests. |
-| `internal/core/workspace` | Effective workspace settings, confined paths and writes, checkpoint identity, and checkpoint manifest models. |
+| `internal/core/workspace` | Effective workspace settings, confined paths and writes, and checkpoint identity and manifest models. |
 
 ## Framework Packages
 
@@ -47,6 +47,7 @@ normalize continuations that may overlap across lanes.
 | `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
 | `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
 | `internal/framework/checkpoint` | Workspace-backed checkpoint loading, recording, and payload serialization. |
+| `internal/framework/chunkplan` | Source-addressed chunk-plan filesystem storage, envelope validation, and atomic publication. |
 | `internal/framework/debug` | Workspace-backed framework and LLM debug recording. |
 
 Framework contracts provide typed artifact, provenance-wrapper, chunk-validator,
@@ -122,7 +123,8 @@ Implementation details for all production extensions are in
 | --- | --- | --- |
 | Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
 | Diagnostics | `internal/core/diagnostics` and `internal/cli` | Record redacted invocation, resolution, result, and failure inspection data. |
-| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable stage outcomes. |
+| Checkpoints | `internal/framework/checkpoint` and `internal/core/workspace` | Validate and serialize reusable extract, merge, and normalize outcomes. |
+| Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
 | Debug artifacts | `internal/framework/debug` and pipeline instrumentation | Capture sensitive framework-boundary and LLM-call material. |
 
 Physical layout, retention, recovery, and sensitive-data handling are defined

docs/internal/pipeline.md

@@ -8,7 +8,8 @@ defaults, and selectable keys are defined in
 
 Resolution fixes the selected lanes and all stage bindings; preparation
 constructs every selected implementation before the runner begins source work.
-After serial input parsing and chunking, the runner dispatches extract work to
+After serial input parsing and plan selection or generation, the runner
+materializes chunks and dispatches extract work to
 one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
 merge and normalize operations remain serial and may overlap other lanes once
 all extracts for that lane are terminal.
@@ -118,7 +119,8 @@ validator context as applicable. It never invokes an operation method.
 `PreparedPipeline` keeps private constructed executors and exposes cloned
 resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
 that prepared pipeline, raw source input, run identity and timing, optional
-session and profile metadata, and checkpoint/debug collaborators. The runner
+session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
+collaborators. The runner
 parses source bytes through the already constructed input adapter. Later stage
 requests receive the generic source model; extract requests receive
 chunk-scoped input material, while chunk, merge, and normalize requests retain
@@ -151,8 +153,9 @@ The runner:
 1. validates its prepared input;
 2. parses the raw input with the prepared adapter and validates the generic
    source document;
-3. obtains or executes the chunk result;
-4. validates and canonicalizes chunks;
+3. selects a stored plan or executes the configured chunker's `Plan` operation;
+4. canonicalizes and materializes the plan, then validates the resulting
+   chunks;
 5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
    bounded lane continuation when all extracts for that lane are terminal;
 6. invokes the prepared output encoder and validates its logical file results;
@@ -173,6 +176,31 @@ and validators while performing these transitions:
 Module-provided warnings and payload warnings are promoted only from attempts
 whose results are accepted and used.
 
+## Chunk Plans And Reuse
+
+`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
+ordered unit-ID ranges, and optional plan or range annotations. The framework
+owns plan canonicalization and materialization. It creates the generic chunks
+and therefore owns their IDs, indexes, source references, JSON content, units,
+media type, and generic metadata. Plan and range annotations are independently
+owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
+
+In `auto`, the runner looks up the source digest before invoking the chunker. A
+valid hit is materialized and sent through the current run's configured chunk
+validators; it does not invoke the chunk module, consume its retry budget, or
+make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
+generates a candidate. `refresh` generates without lookup; `bypass` generates
+without cache access. Generated plans are published only after the full chunk
+validator chain approves them. A validator rejection is a regular rejected
+pipeline outcome and never replaces a cached plan.
+
+The store is source-addressed, not pipeline-addressed. Changes to pipeline
+configuration, requested chunker, options, references, lanes, validators, or
+LLM profile do not prevent a source-digest hit. The manifest records both the
+currently requested chunker and the effective plan producer. Cache state and
+paths are configured and operated outside the runner; see
+[Configuration](../config.md#workspace) and [Operations](../operations.md).
+
 The extract job channel has the same capacity as the effective extract worker
 count, so dispatch applies backpressure. A fixed continuation executor prevents
 ready or checkpoint-reused lanes from creating one goroutine each. Workers and
@@ -180,19 +208,19 @@ continuations publish lane-local results; the coordinator is the only writer of
 aggregate output and merges those results in resolved lane and source-chunk
 order.
 
-## Chunk Canonicalization
+## Plan Canonicalization And Chunk Materialization
 
-Before lane execution, generic validation requires unique chunk IDs, matching
-source identity, indexes matching returned order, a valid canonical reference,
-non-empty content and media type, and at least one valid source unit per chunk.
-Units may not repeat inside a chunk and must form a contiguous range in
-source-document order. The chunk reference must exactly match the source and
-the first and last unit references.
+Plan canonicalization requires canonical JSON annotations, a matching source
+digest, at least one range, existing ordered boundaries, and increasing range
+starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
+Materialization deterministically reconstructs each range from the current
+source document and copies annotations without interpreting their namespaces.
 
-The runner then rebuilds each chunk's unit slice from the source document by
-unit ID. It preserves the canonical reference, content, media type, and cloned
-metadata. The framework permits gaps and overlap between separate
-chunks; stricter coverage policy belongs to the chunk implementation.
+Before lane execution, generic chunk validation checks the materialized chunks'
+identities, order, source references, content, media type, units, and metadata.
+No chunk checkpoint participates in plan selection: plan storage is the only
+chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
+use materialized chunk digests as their dependencies.
 
 ## Validation And Retries