Document chunk plan caching and provenance

docs/internal/pipeline.md

@@ -8,7 +8,8 @@ defaults, and selectable keys are defined in
 
 Resolution fixes the selected lanes and all stage bindings; preparation
 constructs every selected implementation before the runner begins source work.
-After serial input parsing and chunking, the runner dispatches extract work to
+After serial input parsing and plan selection or generation, the runner
+materializes chunks and dispatches extract work to
 one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
 merge and normalize operations remain serial and may overlap other lanes once
 all extracts for that lane are terminal.
@@ -118,7 +119,8 @@ validator context as applicable. It never invokes an operation method.
 `PreparedPipeline` keeps private constructed executors and exposes cloned
 resolved input, chunk, lane, and output identities. `pipeline.RunInput` carries
 that prepared pipeline, raw source input, run identity and timing, optional
-session and profile metadata, and checkpoint/debug collaborators. The runner
+session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
+collaborators. The runner
 parses source bytes through the already constructed input adapter. Later stage
 requests receive the generic source model; extract requests receive
 chunk-scoped input material, while chunk, merge, and normalize requests retain
@@ -151,8 +153,9 @@ The runner:
 1. validates its prepared input;
 2. parses the raw input with the prepared adapter and validates the generic
    source document;
-3. obtains or executes the chunk result;
-4. validates and canonicalizes chunks;
+3. selects a stored plan or executes the configured chunker's `Plan` operation;
+4. canonicalizes and materializes the plan, then validates the resulting
+   chunks;
 5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
    bounded lane continuation when all extracts for that lane are terminal;
 6. invokes the prepared output encoder and validates its logical file results;
@@ -173,6 +176,31 @@ and validators while performing these transitions:
 Module-provided warnings and payload warnings are promoted only from attempts
 whose results are accepted and used.
 
+## Chunk Plans And Reuse
+
+`Chunker.Plan` returns a `source.ChunkPlan`: the canonical source digest,
+ordered unit-ID ranges, and optional plan or range annotations. The framework
+owns plan canonicalization and materialization. It creates the generic chunks
+and therefore owns their IDs, indexes, source references, JSON content, units,
+media type, and generic metadata. Plan and range annotations are independently
+owned raw JSON and become `Chunk.PlanAnnotations` and `Chunk.Annotations`.
+
+In `auto`, the runner looks up the source digest before invoking the chunker. A
+valid hit is materialized and sent through the current run's configured chunk
+validators; it does not invoke the chunk module, consume its retry budget, or
+make a chunk-stage LLM call. A missing, invalid, or unmaterializable record
+generates a candidate. `refresh` generates without lookup; `bypass` generates
+without cache access. Generated plans are published only after the full chunk
+validator chain approves them. A validator rejection is a regular rejected
+pipeline outcome and never replaces a cached plan.
+
+The store is source-addressed, not pipeline-addressed. Changes to pipeline
+configuration, requested chunker, options, references, lanes, validators, or
+LLM profile do not prevent a source-digest hit. The manifest records both the
+currently requested chunker and the effective plan producer. Cache state and
+paths are configured and operated outside the runner; see
+[Configuration](../config.md#workspace) and [Operations](../operations.md).
+
 The extract job channel has the same capacity as the effective extract worker
 count, so dispatch applies backpressure. A fixed continuation executor prevents
 ready or checkpoint-reused lanes from creating one goroutine each. Workers and
@@ -180,19 +208,19 @@ continuations publish lane-local results; the coordinator is the only writer of
 aggregate output and merges those results in resolved lane and source-chunk
 order.
 
-## Chunk Canonicalization
+## Plan Canonicalization And Chunk Materialization
 
-Before lane execution, generic validation requires unique chunk IDs, matching
-source identity, indexes matching returned order, a valid canonical reference,
-non-empty content and media type, and at least one valid source unit per chunk.
-Units may not repeat inside a chunk and must form a contiguous range in
-source-document order. The chunk reference must exactly match the source and
-the first and last unit references.
+Plan canonicalization requires canonical JSON annotations, a matching source
+digest, at least one range, existing ordered boundaries, and increasing range
+starts. Ranges may overlap or leave gaps; a chunker may impose stricter policy.
+Materialization deterministically reconstructs each range from the current
+source document and copies annotations without interpreting their namespaces.
 
-The runner then rebuilds each chunk's unit slice from the source document by
-unit ID. It preserves the canonical reference, content, media type, and cloned
-metadata. The framework permits gaps and overlap between separate
-chunks; stricter coverage policy belongs to the chunk implementation.
+Before lane execution, generic chunk validation checks the materialized chunks'
+identities, order, source references, content, media type, units, and metadata.
+No chunk checkpoint participates in plan selection: plan storage is the only
+chunk-reuse mechanism. Extract, merge, and normalize checkpoints continue to
+use materialized chunk digests as their dependencies.
 
 ## Validation And Retries