Document ordered pipeline operations and retire sequential examples

docs/internal/modules.md

@@ -104,11 +104,12 @@ The NPC identity package owns Unicode comparison keys, deterministic
 The registry package resolves one optional normalized artifact through the
 strict codec, validates whole-registry identity, canonicalizes its JSON, and
 provides immutable records, prompt input, semantic digest, count, and exact
-canonical-name/alias lookup. It owns the `npcs` slot and its bounded,
-content-safe preparation failures. NPC source references are durable
-provenance and are not treated as evidence for a consuming pipeline. The NPC
-codec owns the strict durable `dnd/npc-list` JSON boundary and exposes
-candidate versus approved encode/decode operations.
+canonical-name/alias lookup. External files cross this boundary during
+preparation; generated artifacts cross it at the ordered step handoff. It owns
+the `npcs` slot and its bounded, content-safe validation failures. NPC source
+references are durable provenance and are not treated as evidence for a
+consuming pipeline. The NPC codec owns the strict durable `dnd/npc-list` JSON
+boundary and exposes candidate versus approved encode/decode operations.
 
 The `internal/modules/dnd/codec/combatturns` package owns the durable
 `dnd/combat-turn-list` schema and candidate versus approved JSON boundary. It
@@ -220,9 +221,12 @@ Shared D&D helpers keep prompt input
 names and source-unit reference conversion consistent with the scene chunker.
 
 The extractor also declares the optional `npcs` registry slot and consumes the
-prepared immutable registry boundary from `internal/modules/dnd/npcs/registry`.
-A bound registry adds only `npc_registry_digest` and `npc_count` to manifest
-metadata and an `npc_registry` checkpoint fingerprint. The unbound prompt
+immutable registry boundary from `internal/modules/dnd/npcs/registry`. An
+external registry is prepared before execution; a generated registry is
+validated and supplied at operation time. External bindings may add only
+`npc_registry_digest` and `npc_count` to module metadata and an
+`npc_registry` checkpoint fingerprint. Generated bindings are represented by
+framework handoff provenance and dependency fingerprints. The unbound prompt
 input is exactly `{"npcs":[]}` and has no registry provenance or fingerprint.
 The shared NPC grounding fragment is placed immediately after the common
 campaign reference message and is included in the spell prompt fingerprint.
@@ -310,16 +314,18 @@ independently for extraction and normalization.
 
 ### `internal/modules/dnd/normalize/combatturns`
 
-The combat normalizer prepares the optional NPC registry once and uses the
-immutable prepared view during runtime. It display-normalizes combat fields,
+The combat normalizer prepares an external NPC registry before execution or
+receives a generated registry at the ordered step handoff, then uses the
+immutable view during runtime. It display-normalizes combat fields,
 rewrites exact canonical-name or alias matches for actors and targets, orders
 and deduplicates source references, stable-sorts records by source-document
 position, and collapses only exact duplicate identities with fully valid
 evidence. It deep-clones output storage and emits bounded warnings scoped to
 merged input indexes. Its metadata and fingerprints identify the normalization
-and NPC identity policies, with registry digest/count only when bound. The
-normalizer is included in the production D&D registrar with the default combat
-normalization chain.
+and NPC identity policies. External bindings may contribute registry
+digest/count metadata; generated identity is retained in framework provenance
+and dependency fingerprints. The normalizer is included in the production D&D
+registrar with the default combat normalization chain.
 
 ## Output Encoder
 

docs/internal/overview.md

@@ -14,11 +14,13 @@ collaborators, invokes `internal/framework/pipeline`, and places the logical
 output files returned by the runner. Cache and debug collaborators are supplied
 at this boundary.
 
-Resolution produces a fixed ordered workflow and a sorted set of artifact
-lanes. Preparation constructs the complete module and validator set before the
-runner receives source bytes. Source parsing and chunking are serial; extraction
-uses a bounded run-wide worker pool, followed by serial per-lane merge and
-normalize continuations that may overlap across lanes.
+Resolution produces a fixed ordered workflow of steps and globally unique,
+sorted artifact lanes. Preparation constructs the complete module and validator
+set before the runner receives source bytes. Source parsing and chunking are
+serial. Each step then uses a bounded run-wide extraction pool followed by
+serial per-lane merge and normalize continuations. A step barrier prevents
+later consumers from starting until all earlier lanes are terminal and their
+required normalized artifacts have crossed the typed handoff.
 
 ## Application Boundary
 
@@ -42,7 +44,7 @@ normalize continuations that may overlap across lanes.
 | Package | Implemented responsibility |
 | --- | --- |
 | `internal/framework/contracts` | Source-stage contracts plus artifact identity, schema, serialized representation, codec, validator, reference, output, and structured-completion interfaces and data types. |
-| `internal/framework/pipeline` | Module and artifact-codec registries, option validation, profile resolution, capability checks, reference materialization, complete pipeline preparation, retries, orchestration, warnings, and manifest population. |
+| `internal/framework/pipeline` | Module and artifact-codec registries, ordered-step and generated-reference resolution, option validation, profile resolution, capability checks, external reference materialization, complete pipeline preparation, typed handoff, retries, orchestration, warnings, checkpoint decisions, and manifest population. |
 | `internal/framework/validate` | Shared validator decision and cardinality helpers. |
 | `internal/framework/llm` | Scriptorium-backed structured completions, prompt/schema registration, scheduling, profile recording, and secret redaction. |
 | `internal/framework/promptfs` | Builds module prompt filesystems from module-owned and caller-provided shared prompt assets. |
@@ -111,12 +113,15 @@ this package. Domain-neutral prompt filesystem composition lives in
 `internal/framework/promptfs`.
 
 The `dnd/npcs/registry` package owns the optional `npcs` registry boundary.
-Preparation strictly decodes and identity-validates one normalized JSON
-artifact, emits canonical registry JSON to the spell prompt, and records only
-its semantic digest and count in prepared metadata. The raw reference remains
-independently tracked by pipeline provenance. An absent registry is represented
-only by the empty prompt value `{"npcs":[]}`. Spell extraction consumes this
-shared registry boundary without changing its public module contract.
+External references are strictly decoded and identity-validated during
+preparation; generated references are decoded and identity-validated at the
+ordered step handoff. Both paths emit canonical registry JSON to operation-time
+spell and combat prompt or normalization requests. The framework records
+generated identity and bounded producer provenance, while the raw external
+reference remains independently tracked by pipeline provenance. An absent
+registry is represented only by the empty prompt value `{"npcs":[]}`. Spell
+and combat consumers use this shared boundary without changing their public
+module contracts.
 
 Generic validators under `internal/modules/generic/validate` provide
 unconditional test decisions, JSON syntax validation, and JSON Schema
@@ -144,12 +149,12 @@ Implementation details for all production extensions are in
 | Surface | Implemented owners | Internal purpose |
 | --- | --- | --- |
 | Durable output | Output module, pipeline runner, and CLI writer | Return logical consumer files and place them for a run. |
-| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes. |
+| Cache checkpoints | `internal/framework/checkpoint` and `internal/cli` | Validate and serialize reusable extract, merge, and normalize outcomes, including ordered-step scope and generated-artifact dependency decisions. |
 | Chunk-plan cache | `internal/framework/chunkplan` and `internal/cli` | Persist and select source-addressed plans before framework materialization. |
 | Debug bundles | `internal/core/debugbundle`, `internal/framework/debug`, and pipeline instrumentation | Persist redacted summaries and application-owned traces. |
 
 Physical layout, cleanup, recovery, and sensitive-data handling are defined
-in [Operations](../operations.md). Concrete stage modules receive recorder
+in [Operations](../operations.md). Concrete modules receive recorder
 interfaces and request data, not physical state roots.
 
 ## Focused Documentation

docs/internal/pipeline.md

@@ -6,13 +6,15 @@ Their fixed workflow and ownership boundaries are defined by
 defaults, and selectable keys are defined in
 [Configuration](../config.md#pipelines).
 
-Resolution fixes the selected lanes and all stage bindings; preparation
-constructs every selected implementation before the runner begins source work.
-After serial input parsing and plan selection or generation, the runner
-materializes chunks and dispatches extract work to
-one bounded run-wide worker pool in chunk-first, lane-second order. Each lane's
-merge and normalize operations remain serial and may overlap other lanes once
-all extracts for that lane are terminal.
+Resolution fixes the ordered steps, selected lanes, and all stage bindings;
+preparation constructs every selected implementation before the runner begins
+source work. After serial input parsing and plan selection or generation, the
+runner materializes chunks and executes one step at a time. Within a step,
+extract work uses one bounded run-wide worker pool in chunk-first, lane-second
+order. Each lane's merge and normalize operations remain serial, and lanes in
+the same step may overlap once their extracts are terminal. A later step cannot
+start across its barrier until every earlier lane is terminal and each required
+generated artifact has been accepted and handed off.
 
 ## Resolution
 
@@ -22,38 +24,48 @@ calls `pipeline.ResolvePipeline`.
 
 `ResolvePipeline`:
 
-1. selects and sorts artifact lanes;
-2. completes omitted bindings using the documented configuration defaults;
-3. looks up each module and validator spec without constructing it;
-4. for a typed extractor, derives its artifact kind, requires the codec, and
+1. selects the explicit ordered steps, or creates the implicit `default` step
+   from the legacy top-level `artifacts` map;
+2. selects and sorts artifact lanes within each step while enforcing global lane
+   identity;
+3. completes omitted bindings using the documented configuration defaults;
+4. looks up each module and validator spec without constructing it;
+5. for a typed extractor, derives its artifact kind, requires the codec, and
    selects exact-type merger, normalizer, and validator variants;
-5. checks required and provided capabilities in workflow order;
-6. resolves target-aware reference bindings and validator chains;
-7. validates each selected module and validator option set through its registry
+6. checks required and provided capabilities in workflow order;
+7. resolves external and generated target-aware reference bindings and
+   validates producer order, consumer slot declarations, and artifact-kind
+   compatibility;
+8. validates each selected module and validator option set through its registry
    entry; and
-8. calculates a digest over the resolved structure, including typed artifact
-   kind and schema identity and the effective validator policy in its resolved
-   execution order.
+9. calculates a digest over the resolved structure, including step order, step
+   IDs, lane membership, generated topology, producer and consumer identities,
+   typed artifact kind and schema identity, and the effective validator policy
+   in its resolved execution order.
 
-Resolution returns a `ResolvedPipeline` containing ordered lanes, concrete
-bindings, validator chains, reference targets, and the digest. It does not read
-reference bytes or construct runtime modules. CLI lane and reference selector
-syntax is defined in the [CLI reference](../cli.md#run).
+Resolution returns a `ResolvedPipeline` containing ordered steps, lanes,
+concrete bindings, validator chains, reference targets, and the digest. It does
+not read external reference bytes or construct runtime modules. CLI lane and
+reference selector syntax is defined in the [CLI reference](../cli.md#run).
 
-The digest includes each resolved validator chain's stage, lane, owning module,
-ordered validator bindings, execution classes, targets, and artifact kinds.
-Changing a default chain or an explicit override therefore changes pipeline
-identity whenever it changes the effective validator policy.
+The digest includes each resolved step's ID and lane membership, generated
+producer/consumer topology, and each validator chain's stage, lane, owning
+module, ordered validator bindings, execution classes, targets, and artifact
+kinds. Changing step order, a dependency, a default chain, or an explicit
+override therefore changes pipeline identity whenever it changes effective
+execution policy.
 
 ## Reference Materialization
 
 The CLI calls `MaterializeReferences` after resolution and before constructing
-the LLM client or running the pipeline. The materializer checks each binding
-against its resolved target declaration, reads and validates the file, and
-builds both a `contracts.ReferenceSet` and provenance-only metadata on the
-corresponding `ResolvedReferenceTarget`.
+the LLM client or running the pipeline. For external bindings, the materializer
+checks each binding against its resolved target declaration, reads and validates
+the file, and builds both a `contracts.ReferenceSet` and provenance-only
+metadata on the corresponding `ResolvedReferenceTarget`. A structured
+generated binding is declaration-only at this point: its producer bytes do not
+exist until the producer lane reaches an accepted normalized result.
 
-Preparation delivers the materialized set for each target through
+Preparation delivers the materialized external set for each target through
 `pipeline.BuildRequest`: chunkers and chunk validators receive the chunk target;
 extractors and extract validators receive the lane extract target; mergers and
 merge validators receive the lane merge target; and normalizers and normalize
@@ -62,10 +74,20 @@ an empty set because those stages cannot declare references. Every builder gets
 an isolated deep clone of its target set, so construction-time mutation cannot
 change another builder, the resolved pipeline, or later runtime requests.
 
+Prepared consumers do not need to be reconstructed when generated content is
+available. At the step boundary, the runner encodes the accepted producer value
+through its registered canonical codec, validates the generated bytes against
+each target slot's kind, schema, media type, and size, and clones one immutable
+reference item into the operation request. The item includes canonical digest,
+size, and bounded producer provenance but no filesystem URI. A handoff failure
+is a framework dependency error and prevents every consumer in that step from
+starting.
+
 The runner continues to clone the resulting set into the chunk, extract, merge,
 or normalize request that owns the target. LLM-backed extensions may convert
 those items into named prompt inputs. Reference content remains separate from
-source evidence and source digests.
+source evidence and source digests, whether the item came from a file or a
+generated handoff.
 
 Binding precedence, path resolution, accepted content, and media-type behavior
 are configuration contracts; see [Configuration](../config.md#pipelines).
@@ -120,10 +142,12 @@ The current production catalog and default chain are listed only in
 ## Preparation And Runner Boundary
 
 `pipeline.Prepare` receives a resolved pipeline, the registries, and shared
-module dependencies. It constructs input; chunk and its validators; each lane's
-extract, merge, and normalize modules and validator chains in resolved order;
-then output. It stops at the first error with pipeline, stage, lane, module, and
-validator context as applicable. It never invokes an operation method.
+module dependencies. It constructs input; chunk and its validators; every
+step's lane extract, merge, and normalize modules and validator chains in
+resolved order; then output. It stops at the first error with pipeline, step,
+stage, lane, module, and validator context as applicable. It never invokes an
+operation method. Generated references are not available during preparation;
+the operation request is the handoff boundary.
 
 `PreparedPipeline` keeps private constructed executors and exposes cloned
 resolved input, chunk, lane, and output identities. Prepared components may
@@ -136,8 +160,8 @@ stable and must not contain source content, credentials, local paths,
 timestamps, or other invocation-specific values.
 
 `pipeline.RunInput` carries that prepared pipeline, raw source input, run identity and timing, optional
-session and profile metadata, a chunk-plan store and mode, and checkpoint/debug
-collaborators. The runner
+session and profile metadata, a chunk-plan store and mode, a checkpoint
+execution policy, and checkpoint/debug collaborators. The runner
 parses source bytes through the already constructed input adapter. Later stage
 requests receive the generic source model; extract requests receive
 chunk-scoped input material, while chunk, merge, and normalize requests retain
@@ -145,7 +169,9 @@ access to the original source material. Input, chunk, and output operation
 requests do not carry raw module options. The chunk request also does not carry
 an LLM client; an LLM-backed chunker receives the shared client during
 preparation. Their operation requests retain run-specific source, reference,
-profile, session, and metadata context as applicable.
+profile, session, metadata, and step-handoff context as applicable. A generated
+reference is cloned into each compatible consumer request and is never exposed
+as a path.
 
 Prepared lanes retain exact-type-checked erased operation closures. The runner
 uses those closures to keep each value typed through extraction, validation,
@@ -173,10 +199,15 @@ The runner:
 3. selects a stored plan or executes the configured chunker's `Plan` operation;
 4. canonicalizes and materializes the plan, then validates the resulting
    chunks;
-5. dispatches extract jobs in source-chunk then resolved-lane order, starting a
-   bounded lane continuation when all extracts for that lane are terminal;
-6. invokes the prepared output encoder and validates its logical file results;
-7. returns the assembled manifest, outcomes, warnings, and files.
+5. executes each resolved step in configuration order. For one step, it
+   dispatches extract jobs in source-chunk then resolved-lane order, starts a
+   bounded lane continuation when all extracts for that lane are terminal, and
+   waits for every lane to become terminal;
+6. encodes and validates each accepted normalized producer artifact, then
+   builds the immutable generated reference sets for the next step;
+7. invokes the prepared output encoder only after every step succeeds and
+   validates its logical file results;
+8. returns the assembled manifest, outcomes, warnings, and files.
 
 Within each artifact lane, it reuses the prepared extractor, merger, normalizer,
 and validators while performing these transitions:
@@ -190,6 +221,13 @@ and validators while performing these transitions:
 6. normalize the accepted merge result;
 7. validate and append the accepted normalized result.
 
+At a step barrier, a lane with no accepted normalized output is still a regular
+rejection unless a later generated binding names that lane as a required
+producer. In that case the runner raises a deterministic dependency error and
+does not start the consumer step. One accepted typed artifact may fan out to
+multiple compatible target slots. Consumers in the same step may run
+concurrently after the handoff; no work crosses the barrier early.
+
 Module-provided warnings and payload warnings are promoted only from attempts
 whose results are accepted and used.
 
@@ -271,10 +309,19 @@ entering the normal handoff path. Typed extract, merge, and normalize
 checkpoints store codec bytes with artifact kind, schema ID and version, exact
 schema digest, and media type. Reuse compares that identity with the prepared
 codec and decodes through the codec; missing identity, mismatches, corrupt
-bytes, and decode failures become explicit reuse misses and execute the step
+bytes, and decode failures become explicit reuse misses and execute the lane
 normally. Dependency fingerprints and debug content digests use the same stable
 codec bytes that cross those boundaries.
 
+Generated references add downstream dependencies containing the producer's
+artifact kind, complete schema identity, media type, canonical content digest,
+and size. Compatible producer checkpoints may therefore feed a later step
+without re-executing the producer. A missing, rejected, corrupt, incompatible,
+or changed producer invalidates every transitive dependent lane while leaving
+independent work eligible for reuse. The runner records bounded decision
+categories: `reused`, `executed`, `forced_recompute`, and
+`dependency_invalidated`.
+
 The CLI includes prepared-component fingerprints in the run-wide checkpoint
 identity alongside resolved configuration, raw input, reference provenance,
 runtime overrides, and LLM-profile fingerprints. Module metadata is not used
@@ -283,6 +330,11 @@ values that can change accepted output. Adding or changing a component
 fingerprint intentionally produces a cold cache miss. Existing checkpoint
 schemas and paths remain unchanged.
 
+The CLI's `--recompute-step` policy forces the selected step and all transitive
+dependents, but requires reusable checkpoints for all selected producers that
+precede it. It changes loader decisions only; it does not alter persistent
+checkpoint identity.
+
 Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
 boundaries. Every executed chunk, extract, merge, and normalize attempt writes
 one terminal envelope for acceptance, validator rejection, module or validator

docs/internal/state.md

@@ -32,6 +32,17 @@ The serialized
 `workspace_schema_version` identifiers are frozen wire-compatibility fields;
 they do not describe a current public state surface.
 
+Ordered-step lane checkpoints include the step identity in their storage scope.
+When a later lane consumes a generated artifact, its dependency fingerprints
+include the producer's artifact kind, complete schema identity, media type,
+canonical content digest, and size. A producer checkpoint may be decoded through
+the registered codec and handed off without rerunning it. Missing, rejected,
+corrupt, incompatible, or changed producer state produces a bounded
+`dependency_invalidated` decision for every transitive dependent lane; it does
+not permit stale downstream reuse. The CLI's selective recomputation policy
+records `forced_recompute` for the selected step and its dependents while
+requiring compatible predecessor checkpoints.
+
 `internal/core/fileio` provides confined atomic file writes used by state
 collaborators. The chunk-plan store retains its stronger entry validation.
 
@@ -48,6 +59,11 @@ boundaries redact sensitive metadata and credential-shaped bytes while allowing
 application-owned trace material. Debug data is never a checkpoint source or
 cache input.
 
+Generated reference bytes exist only in cloned operation requests and are not
+written as paths into checkpoints, manifests, or debug summaries. Those state
+surfaces retain canonical identities and bounded producer provenance so that a
+resume decision can be explained without copying generated campaign content.
+
 After allocation, one CLI-owned state value accumulates the known report paths,
 pipeline outcome counts, and validation status. A single guarded terminalization
 operation writes the success report, or makes one attempt each to write the