Plan the ordered pipeline steps implementation

docs/roadmap/dnd.md

@@ -1,35 +0,0 @@
-# D&D Module Harmonization And Prompt Reuse
-
-This roadmap tracked harmonization of the spell, NPC, and combat-turn lanes.
-The implemented behavior is now owned by [Module Internals](../internal/modules.md)
-and [LLM Runtime Internals](../internal/llm.md); this document records status
-and the one remaining external prerequisite rather than duplicating those
-current-behavior references.
-
-## Implementation Status
-
-The offline/runtime work is complete:
-
-- shared prompt assets have explicit per-prompt manifests that drive both
-  mounting and prompt fingerprints;
-- shared prompt inputs, deterministic chunk material preparation, and cited
-  source traversal are centralized;
-- prompt ordering and cache-control policy are aligned across the D&D
-  extraction prompts, while scene chunking retains its distinct prompt shape;
-- validator prerequisite, bounded-diagnostic, and checkpoint policies are
-  aligned without changing compatibility-sensitive identifiers;
-- production registration remains explicit and typed, with grouped composition,
-  central default-chain ownership, and artifact-specific merge behavior;
-- private LLM response schemas remain package-owned and separate from durable
-  codec schemas; and
-- focused tests and repository-wide test, vet, build, and diff checks pass.
-
-## Deferred evaluation
-
-An optional live before/after provider comparison remains deferred because it
-requires credentials and a maintained human-reviewed transcript or fixture
-set. It is not part of the default test suite or merge gate. If those
-prerequisites become available, record only aggregate extraction-review
-results, prompt token counts, cache-hit/cache-write metrics, and non-secret
-prompt hashes. Never commit transcript content, rendered prompts, credentials,
-endpoints, or private reference material.

docs/roadmap/future.md

@@ -5,121 +5,17 @@ configuration, operations, internal, and integration docs. This roadmap records
 future work only. Items are ordered roughly by current value and specificity,
 not as committed release dates.
 
-## Near-Term: Ordered Pipeline Steps
+## Proposed Next Scope: Ordered Pipeline Steps
 
-Allow one configured pipeline to contain multiple ordered execution steps so
-accepted artifacts from an earlier step can become generated references for
-later steps in the same run. This is a bounded extension of the fixed pipeline
-model, not an arbitrary DAG or general workflow language.
+Add ordered groups of artifact lanes, canonical generated-artifact references,
+and dependency-aware checkpoint reuse without introducing a general DAG. The
+first production workflow runs normalized D&D NPC extraction before spell and
+combat-turn extraction and supplies that NPC artifact to their declared
+reference slots.
 
-Input parsing and chunk planning remain pipeline-wide. Each step selects one or
-more artifact lanes; every selected lane completes extraction, validation,
-merge, normalization, and validation before dependent later steps begin. Lanes
-within the same step remain independent and may execute concurrently. The
-runner exposes only accepted normalized artifacts across a step boundary; raw
-extracts, rejected outputs, and intermediate merge results cannot become
-downstream references.
-
-Generated-reference bindings must be explicit in resolved configuration. A
-binding identifies an earlier producing lane and one declared reference slot on
-a later consuming lane. Resolution must reject missing producers, references to
-the same or a later step, incompatible artifact kinds or media types, undeclared
-consumer slots, cycles, and ambiguous bindings. A configured external reference
-and a generated reference cannot bind the same effective target slot; reject
-that pipeline or runtime override instead of applying a precedence rule.
-Each effective target slot accepts at most one producer. One generated artifact
-may fan out to multiple compatible target slots in a later step; aggregation
-from multiple producers into one slot is deferred until a concrete use case
-defines deterministic semantics.
-
-Step-scoped reference defaults mirror existing pipeline-level reference
-defaults. A step binds an earlier artifact once, and the binding automatically
-applies to every target in that step that declares the named slot. Target-local
-bindings remain available when only one module should consume the artifact; do
-not combine a target-local and step-scoped binding for the same effective slot.
-A generated source uses a structured, unambiguous form rather than encoding a
-producer into a path-like string. The target configuration shape is:
-
-```yaml
-steps:
-  - id: identify-npcs
-    artifacts:
-      npcs:
-        extract: dnd/npcs
-        normalize: dnd/npcs
-  - id: grounded-events
-    references:
-      npcs:
-        artifact:
-          step: identify-npcs
-          lane: npcs
-    artifacts:
-      spells:
-        extract: dnd/spells
-        normalize: dnd/spells
-      combat:
-        extract: dnd/combat-turns
-        normalize: dnd/combat-turns
-```
-
-This snippet shows the step and reference portion of a pipeline; pipeline-wide
-input, chunk, output, and other references are omitted. Existing scalar
-reference values continue to mean external file paths; the structured
-`artifact` form means an accepted normalized artifact from the named earlier
-step and lane. Do not infer generated bindings from module keys, matching lane
-names, or D&D-specific knowledge in the framework.
-
-The handoff should use the producer artifact's canonical codec representation
-and retain its artifact kind, schema identity, media type, content digest, and
-producer provenance. Generated references provide context or disambiguation,
-not source evidence. They use the existing module-facing reference contract
-where possible; typed or domain-specific adapters may validate and prepare a
-reference without moving domain concepts into the pipeline framework.
-
-Pipeline identity, manifests, checkpoint dependencies, debug records, and
-errors must include step identity and generated-reference provenance. A
-downstream checkpoint is reusable only when the upstream artifact identity and
-content digest match. Resume should reconstruct an accepted upstream artifact
-through its registered codec instead of requiring the producing lane to run
-again when its checkpoint is reusable.
-
-### Dependency-aware resume and recomputation
-
-Treat generated-reference bindings as checkpoint dependencies. Reusing an
-earlier step is safe only when its existing checkpoint and codec identity are
-valid. Reusing a dependent step additionally requires an exact match for every
-upstream artifact kind, schema identity, media type, and canonical content
-digest it consumed. A changed, missing, rejected, corrupt, or incompatible
-producer artifact invalidates all transitive dependent checkpoints; the runner
-must never combine a newly produced upstream artifact with stale downstream
-output.
-
-Support selectively recomputing one configured step and all of its transitive
-dependents while retaining reusable independent and predecessor work. The
-operator-facing selection mechanism should identify a stable configured step,
-not individual internal stage operations. Resolution must reject a selection
-that would omit a required predecessor without a reusable accepted artifact.
-
-Manifests, checkpoint events, and diagnostics should distinguish work that was
-executed, reused, or invalidated and record a bounded non-secret reason for
-dependency-driven invalidation. Completion order must not affect invalidation,
-public artifact ordering, or the set of dependent steps selected for rerun.
-
-If a required producer finishes without an accepted normalized artifact, fail
-the entire run with a deterministic dependency error. Do not start any
-dependent step. Preserve the upstream rejection or empty-result outcome and
-step provenance in the failed run manifest so the cause remains auditable.
-
-The first production workflow is D&D NPC grounding:
-
-1. the first step runs the NPC lane through accepted normalized output; and
-2. the second step runs spell and combat-turn lanes, binding that NPC artifact
-   to the spell extractor and to the combat extractor and normalizer through
-   their existing `npcs` reference slots.
-
-Spell and combat-turn extraction may run concurrently after the NPC handoff is
-available. The NPC artifact may disambiguate participant identity, but it does
-not prove that a spell cast or combat turn occurred.
+The bounded target, compatibility and architecture decisions, exclusions, and
+acceptance criteria are defined in
+[Proposed Scope: Ordered Pipeline Steps](ordered-pipeline-steps.md).
 
 ## Near-Term D&D Pipeline
 

docs/roadmap/ordered-pipeline-steps.md

@@ -0,0 +1,340 @@
+# Proposed Scope: Ordered Pipeline Steps
+
+## Status
+
+Proposed as the next implementation scope. This document refines the
+near-term priority in [Future Work](future.md) into a bounded feature target. It
+defines desired behavior and scope, not a file-by-file implementation plan.
+
+## Recommendation
+
+Implement ordered pipeline steps, generated artifact references, and
+dependency-aware checkpoint reuse as one coherent platform capability. Prove
+the capability with the D&D workflow in which accepted normalized NPC output
+is produced first and then supplied to spell extraction, combat-turn
+extraction, and combat-turn normalization.
+
+This should precede item extraction. It establishes the artifact-handoff and
+dependency semantics that the planned item lane will also consume, without
+combining two substantial features in one implementation effort.
+
+## Intended Outcome
+
+A configured pipeline may contain multiple ordered steps while retaining one
+pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run
+manifest, and failure boundary. Every artifact lane still follows the fixed
+extract, validate, merge, validate, normalize, and validate lifecycle. Steps
+add explicit barriers between groups of lanes; they do not create arbitrary
+stage graphs.
+
+An accepted normalized artifact from an earlier step may be bound explicitly
+to declared reference slots in a later step. The framework remains
+domain-neutral, and generated references remain contextual material rather than
+source evidence.
+
+## Fixed Product And Architecture Decisions
+
+### Pipeline shape
+
+- Input parsing and chunk planning remain pipeline-wide and execute once.
+- A step contains one or more artifact lanes. Step order is configuration order.
+- Lanes within a step remain independent and may use the existing bounded
+  concurrency model.
+- A later step cannot begin until every lane in the current step is terminal
+  and every generated artifact it requires is accepted and available.
+- Public artifact and failure ordering is step order followed by deterministic
+  lane and source-chunk order, never completion order.
+- Output encoding occurs once, after every step succeeds.
+- This is not an arbitrary DAG, a general workflow language, concurrent
+  cross-lane reconciliation, or permission for modules to invoke other modules.
+
+### Configuration model
+
+Existing single-step pipelines remain valid. A top-level `artifacts` map is
+treated as an implicit step with stable ID `default`. A pipeline may configure
+either `artifacts` or `steps`, but not both. Explicit steps must be non-empty
+and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique
+across the entire pipeline so output paths, selectors, manifests, errors, and
+checkpoint scopes remain unambiguous.
+
+The target configuration shape is:
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    chunk: generic
+    steps:
+      - id: identify-npcs
+        artifacts:
+          npcs:
+            extract: dnd/npcs
+            normalize: dnd/npcs
+      - id: grounded-events
+        references:
+          npcs:
+            artifact:
+              step: identify-npcs
+              lane: npcs
+        artifacts:
+          spells:
+            extract: dnd/spells
+            normalize: dnd/spells
+          combat:
+            extract: dnd/combat-turns
+            normalize: dnd/combat-turns
+    output: json
+```
+
+Existing scalar reference values continue to represent external file paths.
+The structured `artifact` form identifies accepted normalized output from one
+earlier step and lane. Generated artifact bindings are allowed at step scope or
+at an individual module target; they are not inferred from module keys, lane
+names, slot names, or domain knowledge.
+
+A step-scoped reference applies automatically to every selected target in that
+step that declares the slot. In the example, one `npcs` binding reaches the
+spell extractor plus the combat extractor and normalizer. A target-local
+binding is used when only one module should consume the artifact.
+
+Pipeline-level external references remain defaults. Step-local external
+references override pipeline-level external defaults, and target-local
+external references retain their existing precedence. A generated reference
+and an external reference may not resolve to the same effective target slot;
+configuration or a runtime override that creates that conflict is invalid.
+Likewise, a step-scoped and target-local generated binding cannot both target
+the same effective slot.
+
+Each effective target slot accepts at most one producer. One producer may fan
+out to multiple compatible slots in a later step. Aggregating several producer
+artifacts into one slot is outside this scope.
+
+Reference-slot specs gain optional generated-artifact compatibility metadata.
+A generated binding is allowed only when the consumer slot declares the
+producer's artifact kind; the producer's registered codec supplies the exact
+schema identity and media type used for the handoff. The D&D `npcs` consumer
+slots declare the normalized NPC-list artifact kind. Existing external-file
+slots and bindings retain their current behavior and do not acquire an artifact
+kind merely because their bytes happen to decode as one.
+
+### Resolution and preparation
+
+Resolution validates the complete ordered structure before source processing.
+It must reject duplicate identities, missing producers, same-step or forward
+references, undeclared slots, reference conflicts, and incompatible artifact
+kind, schema, media type, or cardinality constraints that are statically
+discoverable. Size is checked when canonical producer bytes exist at handoff.
+Ordered steps make cycles structurally impossible; resolution must not
+introduce a general graph scheduler to rediscover their order.
+
+The resolved pipeline and its digest include step order, step IDs, lane
+membership, generated-reference topology, producer identity, consumer targets,
+and existing module and validator policy. Cloning, redaction, canonical JSON,
+debug summaries, and manifests preserve the same structure without reference
+content or secrets.
+
+All modules and validators are still selected, option-validated, and
+constructed before source parsing. Generated content cannot be supplied during
+construction because it does not exist yet. The framework therefore augments
+the existing operation-request `References` at the step boundary. Consumers
+that currently assume an NPC registry is construction-only must accept the
+generated registry from their operation request without deferring general
+module construction until after upstream work.
+
+Only validation that inherently depends on generated bytes may occur at the
+handoff. A handoff validation failure is a contextual framework error and fails
+the run before any consumer in that step begins.
+
+### Generated artifact handoff
+
+Only accepted normalized output may cross a step boundary. Raw extraction
+responses, rejected artifacts, merge intermediates, and validator diagnostics
+cannot be bound as references.
+
+The framework serializes the producer through its registered canonical artifact
+codec and constructs one immutable reference item containing:
+
+- the declared target slot;
+- canonical artifact bytes and media type;
+- artifact kind and schema ID, name, version, and schema digest;
+- canonical content digest and size; and
+- producer pipeline, step, lane, and module provenance.
+
+A generated binding requires exactly one accepted normalized artifact from its
+producer lane. No artifact is a missing dependency, while more than one is a
+cardinality error; a typed collection such as an NPC list remains one artifact.
+Combining several normalized outputs into one reference is aggregation and is
+outside this scope.
+
+The existing slot contract remains authoritative for accepted media types,
+maximum size, and cardinality. Generated content is cloned at ownership
+boundaries and never exposed through a filesystem path. Manifests and debug
+summaries record identities and bounded provenance, not artifact content.
+
+Configuring a generated binding makes that dependency required even when the
+consumer module declares the underlying slot optional. An accepted artifact
+whose domain collection is empty is still a valid artifact and may be handed
+off. If the producer has no accepted normalized artifact, the entire run fails
+with a deterministic dependency error and no later step begins.
+
+### Checkpoint reuse and selective recomputation
+
+Generated references participate in downstream checkpoint dependencies by
+artifact kind, complete schema identity, media type, and canonical content
+digest. The pipeline digest protects topology; stage dependency fingerprints
+protect the exact upstream artifact consumed. The runner must never combine a
+new or changed producer with stale dependent output.
+
+Compatible producer checkpoints may be decoded through the registered codec
+and handed to later steps without rerunning the producer. A missing, rejected,
+corrupt, incompatible, or changed producer invalidates every transitive
+dependent checkpoint. Independent work remains reusable.
+
+Add one operator control, `--recompute-step <step-id>`, with these semantics:
+
+- it requires checkpoint recording and `--resume`;
+- the selected step and all transitive dependents execute rather than reuse
+  their checkpoints;
+- valid required predecessors and unrelated work remain reusable;
+- the recompute selection affects loader decisions, not the persistent
+  checkpoint identity of otherwise identical work; and
+- the command fails before dependent execution if a required predecessor has
+  no reusable accepted artifact.
+
+Existing `--only` behavior remains unchanged for implicit single-step
+pipelines. Combining `--only` with explicit multi-step pipelines is outside
+this scope and should be rejected with actionable guidance rather than given
+implicit dependency-expansion semantics.
+
+Checkpoint events, manifests, and diagnostics distinguish executed, reused,
+forced-recomputed, and dependency-invalidated work. Invalidation reasons are
+bounded, deterministic, and free of reference content, local paths, or secrets.
+Old checkpoint state need not be migrated; it must produce a safe, explicit
+cold miss rather than an error or unsafe reuse.
+
+### Failure, cancellation, and concurrency
+
+The existing run-wide worker and provider-call limits apply across every step.
+Workers may be reused between steps, but concurrency cannot cross a step
+barrier. A framework error cancels started work using the existing bounded
+drain behavior and prevents later steps and output encoding. Rejections remain
+recorded outcomes, but failure to produce a normalized artifact required by a
+generated binding escalates to the run-level dependency error described above.
+
+The failed manifest retains completed upstream outcomes, step and lane
+provenance, rejections, checkpoint events, and the dependency failure without
+embedding generated artifact content.
+
+## D&D Proving Workflow
+
+The production acceptance workflow has two explicit steps:
+
+1. `identify-npcs` runs the NPC lane through normalization and its complete
+   validator policy.
+2. `grounded-events` receives the canonical NPC artifact in its step-scoped
+   `npcs` reference and runs spell and combat-turn lanes. The binding reaches
+   spell extraction, combat-turn extraction, and combat-turn normalization.
+
+Spell and combat-turn lanes may execute concurrently after the handoff. NPC
+content may ground names and identities but cannot establish a spell cast or
+combat event; source units remain the only event evidence.
+
+The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be
+replaced or supplemented by one ordered-pipeline example. Existing module keys,
+artifact contracts, reference slot names, prompt IDs, and D&D evidence policy
+remain unchanged.
+
+## Included Work
+
+- Configuration parsing, validation, cloning, defaults, redaction, and
+  documentation for explicit steps and structured generated references.
+- Domain-neutral resolved step, dependency, producer, and consumer identities.
+- Generated-artifact compatibility metadata on reference-slot contracts,
+  including D&D NPC-list declarations for every `npcs` consumer.
+- Step-aware preparation metadata and runner orchestration.
+- Canonical codec handoff into existing reference request contracts.
+- Required-dependency failure and bounded provenance behavior.
+- Dependency-aware checkpoint reuse, invalidation, events, and selective step
+  recomputation.
+- D&D NPC-first production composition for spell and combat-turn consumers.
+- Refactoring the affected D&D consumers so generated NPC references are
+  available at operation time while retaining early static construction.
+- Maintained examples and updates to current architecture, configuration, CLI,
+  operations, internal, integration, and testing documentation when behavior
+  lands.
+- An ADR recording the bounded ordered-step extension to the fixed pipeline
+  architecture and its explicit rejection of a general DAG.
+
+## Explicitly Out Of Scope
+
+- D&D item extraction or any other new artifact lane.
+- Cross-artifact NPC ID fields or artifact-schema migration machinery.
+- Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or
+  module-controlled scheduling.
+- Multiple source inputs, per-step input adapters, per-step chunk plans, or
+  per-step output encoders.
+- Aggregating multiple generated artifacts into one reference slot.
+- Optional or best-effort generated dependencies; a configured dependency is
+  required in this scope.
+- Prior-run or cross-pipeline generated references.
+- `--only` dependency closure for explicit multi-step pipelines.
+- Cross-lane reconciliation or domain concepts in the generic framework.
+- Live-provider tests or model-quality changes to D&D prompts.
+
+## Acceptance Criteria
+
+The scope is complete when:
+
+- all existing single-step configurations retain their current behavior;
+- explicit step order and dependency topology resolve deterministically and
+  affect pipeline identity;
+- invalid producer, consumer, conflict, ordering, type, schema, media, and
+  cardinality configurations fail before source processing when statically
+  discoverable, while content-size violations fail at handoff;
+- no consumer step begins before all required generated artifacts are accepted,
+  canonicalized, and validated for its target slots;
+- one producer artifact fans out safely to every compatible target selected by
+  a step-scoped binding;
+- missing required producer output fails the complete run before dependent work;
+- changing NPC output invalidates spell and combat-turn checkpoints while
+  leaving compatible independent work reusable;
+- selective step recomputation executes exactly the selected dependency closure
+  and reports why work was executed, reused, or invalidated;
+- the D&D ordered workflow supplies NPC content to spell extraction, combat-turn
+  extraction, and combat-turn normalization without treating it as evidence;
+- completion timing cannot change public ordering, failure selection, or
+  dependency behavior;
+- output, manifests, checkpoints, and debug artifacts contain the required
+  identities and provenance without leaking generated reference content; and
+- repository-wide tests, vet, build, maintained-example checks, and
+  documentation validation pass.
+
+## Testing Strategy
+
+Tests should protect behavior and invariants rather than the implementation's
+internal scheduler shape.
+
+- Configuration contract tests own legacy shorthand, explicit step parsing,
+  source-form discrimination, conflicts, and redaction.
+- Resolution tests own ordering, global lane uniqueness, dependency validation,
+  slot compatibility, fan-out, cloning, canonical JSON, and digest changes.
+- Runner tests own step barriers, within-step bounded concurrency, stable
+  ordering, cancellation, required-producer failure, and immutable handoff.
+- Checkpoint tests own producer decoding, exact dependency matching, transitive
+  invalidation, forced recomputation, cold misses, and bounded decisions.
+- One CLI contract test should cover the recompute control and its invalid
+  combinations.
+- One D&D integration test with offline fake LLM responses should prove the
+  complete NPC-to-spell-and-combat handoff, including combat normalization.
+- Maintained configuration examples should be parsed and resolved through the
+  production catalog.
+
+Do not add scheduler choreography tests, exact goroutine-count assertions,
+complete manifest snapshots, exact diagnostic strings, or duplicated tests for
+every invalid configuration at every layer. No test may require credentials or
+a live model provider.
+
+## Open Questions
+
+None required to define this scope. Exact internal type names and implementation
+decomposition are intentionally not feature-policy decisions.