Document ordered pipeline operations and retire sequential examples

docs/internal/pipeline.md

@@ -6,13 +6,15 @@ Their fixed workflow and ownership boundaries are defined by
 defaults, and selectable keys are defined in
 [Configuration](../config.md#pipelines).
 
-Resolution fixes the selected lanes and all stage bindings; preparation
-constructs every selected implementation before the runner begins source work.
-After serial input parsing and plan selection or generation, the runner
-materializes chunks and dispatches extract work to
-one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
-merge and normalize operations remain serial and may overlap other lanes once
-all extracts for that lane are terminal.
+Resolution fixes the ordered steps, selected lanes, and all stage bindings;
+preparation constructs every selected implementation before the runner begins
+source work. After serial input parsing and plan selection or generation, the
+runner materializes chunks and executes one step at a time. Within a step,
+extract work uses one bounded run-wide worker pool in chunk-first, lane-second
+order. Each lane's merge and normalize operations remain serial, and lanes in
+the same step may overlap once their extracts are terminal. A later step cannot
+start across its barrier until every earlier lane is terminal and each required
+generated artifact has been accepted and handed off.
 
 ## Resolution
 
@@ -22,38 +24,48 @@ calls `pipeline.ResolvePipeline`.
 
 `ResolvePipeline`:
 
-1. selects and sorts artifact lanes;
-2. completes omitted bindings using the documented configuration defaults;
-3. looks up each module and validator spec without constructing it;
-4. for a typed extractor, derives its artifact kind, requires the codec, and
+1. selects the explicit ordered steps, or creates the implicit `default` step
+   from the legacy top-level `artifacts` map;
+2. selects and sorts artifact lanes within each step while enforcing global lane
+   identity;
+3. completes omitted bindings using the documented configuration defaults;
+4. looks up each module and validator spec without constructing it;
+5. for a typed extractor, derives its artifact kind, requires the codec, and
    selects exact-type merger, normalizer, and validator variants;
-5. checks required and provided capabilities in workflow order;
-6. resolves target-aware reference bindings and validator chains;
-7. validates each selected module and validator option set through its registry
+6. checks required and provided capabilities in workflow order;
+7. resolves external and generated target-aware reference bindings and
+   validates producer order, consumer slot declarations, and artifact-kind
+   compatibility;
+8. validates each selected module and validator option set through its registry
    entry; and
-8. calculates a digest over the resolved structure, including typed artifact
-   kind and schema identity and the effective validator policy in its resolved
-   execution order.
+9. calculates a digest over the resolved structure, including step order, step
+   IDs, lane membership, generated topology, producer and consumer identities,
+   typed artifact kind and schema identity, and the effective validator policy
+   in its resolved execution order.
 
-Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
-bindings, validator chains, reference targets, and the digest. It does not read
-reference bytes or construct runtime modules. CLI lane and reference selector
-syntax is defined in the [CLI reference](../cli.md#run).
+Resolution returns a `ResolvedPipeline` containing ordered steps, lanes,
+concrete bindings, validator chains, reference targets, and the digest. It does
+not read external reference bytes or construct runtime modules. CLI lane and
+reference selector syntax is defined in the [CLI reference](../cli.md#run).
 
-The digest includes each resolved validator chain's stage, lane, owning module,
-ordered validator bindings, execution classes, targets, and artifact kinds.
-Changing a default chain or an explicit override therefore changes pipeline
-identity whenever it changes the effective validator policy.
+The digest includes each resolved step's ID and lane membership, generated
+producer/consumer topology, and each validator chain's stage, lane, owning
+module, ordered validator bindings, execution classes, targets, and artifact
+kinds. Changing step order, a dependency, a default chain, or an explicit
+override therefore changes pipeline identity whenever it changes effective
+execution policy.
 
 ## Reference Materialization
 
 The CLI calls `MaterializeReferences` after resolution and before constructing
-the LLM client or running the pipeline. The materializer checks each binding
-against its resolved target declaration, reads and validates the file, and
-builds both a `contracts.ReferenceSet` and provenance-only metadata on the
-corresponding `ResolvedReferenceTarget`.
+the LLM client or running the pipeline. For external bindings, the materializer
+checks each binding against its resolved target declaration, reads and validates
+the file, and builds both a `contracts.ReferenceSet` and provenance-only
+metadata on the corresponding `ResolvedReferenceTarget`. A structured
+generated binding is declaration-only at this point: its producer bytes do not
+exist until the producer lane reaches an accepted normalized result.
 
-Preparation delivers the materialized set for each target through
+Preparation delivers the materialized external set for each target through
 `pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target;
 extractors and extract validators receive the lane extract target; mergers and
 merge validators receive the lane merge target; and normalizers and normalize
@@ -62,10 +74,20 @@ an empty set because those stages cannot declare references. Every builder gets
 an isolated deep clone of its target set, so construction-time mutation cannot
 change another builder, the resolved pipeline, or later runtime requests.
 
+Prepared consumers do not need to be reconstructed when generated content is
+available. At the step boundary, the runner encodes the accepted producer value
+through its registered canonical codec, validates the generated bytes against
+each target slot's kind, schema, media type, and size, and clones one immutable
+reference item into the operation request. The item includes canonical digest,
+size, and bounded producer provenance but no filesystem URI. A handoff failure
+is a framework dependency error and prevents every consumer in that step from
+starting.
+
 The runner continues to clone the resulting set into the chunk, extract, merge,
 or normalize request that owns the target. LLM-backed extensions may convert
 those items into named prompt inputs. Reference content remains separate from
-source evidence and source digests.
+source evidence and source digests, whether the item came from a file or a
+generated handoff.
 
 Binding precedence, path resolution, accepted content, and media-type behavior
 are configuration contracts; see [Configuration](../config.md#pipelines).
@@ -120,10 +142,12 @@ The current production catalog and default chain are listed only in
 ## Preparation And Runner Boundary
 
 `pipeline.Prepare` receives a resolved pipeline, the registries, and shared
-module dependencies. It constructs input; chunk and its validators; each lane's
-extract, merge, and normalize modules and validator chains in resolved order;
-then output. It stops at the first error with pipeline, stage, lane, module, and
-validator context as applicable. It never invokes an operation method.
+module dependencies. It constructs input; chunk and its validators; every
+step's lane extract, merge, and normalize modules and validator chains in
+resolved order; then output. It stops at the first error with pipeline, step,
+stage, lane, module, and validator context as applicable. It never invokes an
+operation method. Generated references are not available during preparation;
+the operation request is the handoff boundary.
 
 `PreparedPipeline` keeps private constructed executors and exposes cloned
 resolved input, chunk, lane, and output identities. Prepared components may
@@ -136,8 +160,8 @@ stable and must not contain source content, credentials, local paths,
 timestamps, or other invocation-specific values.
 
 `pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional
-session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
-collaborators. The runner
+session and profile metadata, a chunk-plan store and mode, a checkpoint
+execution policy, and checkpoint/debug collaborators. The runner
 parses source bytes through the already constructed input adapter. Later stage
 requests receive the generic source model; extract requests receive
 chunk-scoped input material, while chunk, merge, and normalize requests retain
@@ -145,7 +169,9 @@ access to the original source material. Input, chunk, and output operation
 requests do not carry raw module options. The chunk request also does not carry
 an LLM client; an LLM-backed chunker receives the shared client during
 preparation. Their operation requests retain run-specific source, reference,
-profile, session, and metadata context as applicable.
+profile, session, metadata, and step-handoff context as applicable. A generated
+reference is cloned into each compatible consumer request and is never exposed
+as a path.
 
 Prepared lanes retain exact-type-checked erased operation closures. The runner
 uses those closures to keep each value typed through extraction, validation,
@@ -173,10 +199,15 @@ The runner:
 3. selects a stored plan or executes the configured chunker's `Plan` operation;
 4. canonicalizes and materializes the plan, then validates the resulting
    chunks;
-5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
-   bounded lane continuation when all extracts for that lane are terminal;
-6. invokes the prepared output encoder and validates its logical file results;
-7. returns the assembled manifest, outcomes, warnings, and files.
+5. executes each resolved step in configuration order. For one step, it
+   dispatches extract jobs in source-chunk then resolved-lane order, starts a
+   bounded lane continuation when all extracts for that lane are terminal, and
+   waits for every lane to become terminal;
+6. encodes and validates each accepted normalized producer artifact, then
+   builds the immutable generated reference sets for the next step;
+7. invokes the prepared output encoder only after every step succeeds and
+   validates its logical file results;
+8. returns the assembled manifest, outcomes, warnings, and files.
 
 Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
 and validators while performing these transitions:
@@ -190,6 +221,13 @@ and validators while performing these transitions:
 6. normalize the accepted merge result;
 7. validate and append the accepted normalized result.
 
+At a step barrier, a lane with no accepted normalized output is still a regular
+rejection unless a later generated binding names that lane as a required
+producer. In that case the runner raises a deterministic dependency error and
+does not start the consumer step. One accepted typed artifact may fan out to
+multiple compatible target slots. Consumers in the same step may run
+concurrently after the handoff; no work crosses the barrier early.
+
 Module-provided warnings and payload warnings are promoted only from attempts
 whose results are accepted and used.
 
@@ -271,10 +309,19 @@ entering the normal handoff path. Typed extract, merge, and normalize
 checkpoints store codec bytes with artifact kind, schema ID and version, exact
 schema digest, and media type. Reuse compares that identity with the prepared
 codec and decodes through the codec; missing identity, mismatches, corrupt
-bytes, and decode failures become explicit reuse misses and execute the step
+bytes, and decode failures become explicit reuse misses and execute the lane
 normally. Dependency fingerprints and debug content digests use the same stable
 codec bytes that cross those boundaries.
 
+Generated references add downstream dependencies containing the producer's
+artifact kind, complete schema identity, media type, canonical content digest,
+and size. Compatible producer checkpoints may therefore feed a later step
+without re-executing the producer. A missing, rejected, corrupt, incompatible,
+or changed producer invalidates every transitive dependent lane while leaving
+independent work eligible for reuse. The runner records bounded decision
+categories: `reused`, `executed`, `forced_recompute`, and
+`dependency_invalidated`.
+
 The CLI includes prepared-component fingerprints in the run-wide checkpoint
 identity alongside resolved configuration, raw input, reference provenance,
 runtime overrides, and LLM-profile fingerprints. Module metadata is not used
@@ -283,6 +330,11 @@ values that can change accepted output. Adding or changing a component
 fingerprint intentionally produces a cold cache miss. Existing checkpoint
 schemas and paths remain unchanged.
 
+The CLI's `--recompute-step` policy forces the selected step and all transitive
+dependents, but requires reusable checkpoints for all selected producers that
+precede it. It changes loader decisions only; it does not alter persistent
+checkpoint identity.
+
 Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
 boundaries. Every executed chunk, extract, merge, and normalize attempt writes
 one terminal envelope for acceptance, validator rejection, module or validator