Plan the ordered pipeline steps implementation

docs/roadmap/ordered-pipeline-steps.md

@@ -0,0 +1,340 @@
+# Proposed Scope: Ordered Pipeline Steps
+
+## Status
+
+Proposed as the next implementation scope. This document refines the
+near-term priority in [Future Work](future.md) into a bounded feature target. It
+defines desired behavior and scope, not a file-by-file implementation plan.
+
+## Recommendation
+
+Implement ordered pipeline steps, generated artifact references, and
+dependency-aware checkpoint reuse as one coherent platform capability. Prove
+the capability with the D&D workflow in which accepted normalized NPC output
+is produced first and then supplied to spell extraction, combat-turn
+extraction, and combat-turn normalization.
+
+This should precede item extraction. It establishes the artifact-handoff and
+dependency semantics that the planned item lane will also consume, without
+combining two substantial features in one implementation effort.
+
+## Intended Outcome
+
+A configured pipeline may contain multiple ordered steps while retaining one
+pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run
+manifest, and failure boundary. Every artifact lane still follows the fixed
+extract, validate, merge, validate, normalize, and validate lifecycle. Steps
+add explicit barriers between groups of lanes; they do not create arbitrary
+stage graphs.
+
+An accepted normalized artifact from an earlier step may be bound explicitly
+to declared reference slots in a later step. The framework remains
+domain-neutral, and generated references remain contextual material rather than
+source evidence.
+
+## Fixed Product And Architecture Decisions
+
+### Pipeline shape
+
+- Input parsing and chunk planning remain pipeline-wide and execute once.
+- A step contains one or more artifact lanes. Step order is configuration order.
+- Lanes within a step remain independent and may use the existing bounded
+  concurrency model.
+- A later step cannot begin until every lane in the current step is terminal
+  and every generated artifact it requires is accepted and available.
+- Public artifact and failure ordering is step order followed by deterministic
+  lane and source-chunk order, never completion order.
+- Output encoding occurs once, after every step succeeds.
+- This is not an arbitrary DAG, a general workflow language, concurrent
+  cross-lane reconciliation, or permission for modules to invoke other modules.
+
+### Configuration model
+
+Existing single-step pipelines remain valid. A top-level `artifacts` map is
+treated as an implicit step with stable ID `default`. A pipeline may configure
+either `artifacts` or `steps`, but not both. Explicit steps must be non-empty
+and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique
+across the entire pipeline so output paths, selectors, manifests, errors, and
+checkpoint scopes remain unambiguous.
+
+The target configuration shape is:
+
+```yaml
+pipelines:
+  dnd-session:
+    input: seriatim
+    chunk: generic
+    steps:
+      - id: identify-npcs
+        artifacts:
+          npcs:
+            extract: dnd/npcs
+            normalize: dnd/npcs
+      - id: grounded-events
+        references:
+          npcs:
+            artifact:
+              step: identify-npcs
+              lane: npcs
+        artifacts:
+          spells:
+            extract: dnd/spells
+            normalize: dnd/spells
+          combat:
+            extract: dnd/combat-turns
+            normalize: dnd/combat-turns
+    output: json
+```
+
+Existing scalar reference values continue to represent external file paths.
+The structured `artifact` form identifies accepted normalized output from one
+earlier step and lane. Generated artifact bindings are allowed at step scope or
+at an individual module target; they are not inferred from module keys, lane
+names, slot names, or domain knowledge.
+
+A step-scoped reference applies automatically to every selected target in that
+step that declares the slot. In the example, one `npcs` binding reaches the
+spell extractor plus the combat extractor and normalizer. A target-local
+binding is used when only one module should consume the artifact.
+
+Pipeline-level external references remain defaults. Step-local external
+references override pipeline-level external defaults, and target-local
+external references retain their existing precedence. A generated reference
+and an external reference may not resolve to the same effective target slot;
+configuration or a runtime override that creates that conflict is invalid.
+Likewise, a step-scoped and target-local generated binding cannot both target
+the same effective slot.
+
+Each effective target slot accepts at most one producer. One producer may fan
+out to multiple compatible slots in a later step. Aggregating several producer
+artifacts into one slot is outside this scope.
+
+Reference-slot specs gain optional generated-artifact compatibility metadata.
+A generated binding is allowed only when the consumer slot declares the
+producer's artifact kind; the producer's registered codec supplies the exact
+schema identity and media type used for the handoff. The D&D `npcs` consumer
+slots declare the normalized NPC-list artifact kind. Existing external-file
+slots and bindings retain their current behavior and do not acquire an artifact
+kind merely because their bytes happen to decode as one.
+
+### Resolution and preparation
+
+Resolution validates the complete ordered structure before source processing.
+It must reject duplicate identities, missing producers, same-step or forward
+references, undeclared slots, reference conflicts, and incompatible artifact
+kind, schema, media type, or cardinality constraints that are statically
+discoverable. Size is checked when canonical producer bytes exist at handoff.
+Ordered steps make cycles structurally impossible; resolution must not
+introduce a general graph scheduler to rediscover their order.
+
+The resolved pipeline and its digest include step order, step IDs, lane
+membership, generated-reference topology, producer identity, consumer targets,
+and existing module and validator policy. Cloning, redaction, canonical JSON,
+debug summaries, and manifests preserve the same structure without reference
+content or secrets.
+
+All modules and validators are still selected, option-validated, and
+constructed before source parsing. Generated content cannot be supplied during
+construction because it does not exist yet. The framework therefore augments
+the existing operation-request `References` at the step boundary. Consumers
+that currently assume an NPC registry is construction-only must accept the
+generated registry from their operation request without deferring general
+module construction until after upstream work.
+
+Only validation that inherently depends on generated bytes may occur at the
+handoff. A handoff validation failure is a contextual framework error and fails
+the run before any consumer in that step begins.
+
+### Generated artifact handoff
+
+Only accepted normalized output may cross a step boundary. Raw extraction
+responses, rejected artifacts, merge intermediates, and validator diagnostics
+cannot be bound as references.
+
+The framework serializes the producer through its registered canonical artifact
+codec and constructs one immutable reference item containing:
+
+- the declared target slot;
+- canonical artifact bytes and media type;
+- artifact kind and schema ID, name, version, and schema digest;
+- canonical content digest and size; and
+- producer pipeline, step, lane, and module provenance.
+
+A generated binding requires exactly one accepted normalized artifact from its
+producer lane. No artifact is a missing dependency, while more than one is a
+cardinality error; a typed collection such as an NPC list remains one artifact.
+Combining several normalized outputs into one reference is aggregation and is
+outside this scope.
+
+The existing slot contract remains authoritative for accepted media types,
+maximum size, and cardinality. Generated content is cloned at ownership
+boundaries and never exposed through a filesystem path. Manifests and debug
+summaries record identities and bounded provenance, not artifact content.
+
+Configuring a generated binding makes that dependency required even when the
+consumer module declares the underlying slot optional. An accepted artifact
+whose domain collection is empty is still a valid artifact and may be handed
+off. If the producer has no accepted normalized artifact, the entire run fails
+with a deterministic dependency error and no later step begins.
+
+### Checkpoint reuse and selective recomputation
+
+Generated references participate in downstream checkpoint dependencies by
+artifact kind, complete schema identity, media type, and canonical content
+digest. The pipeline digest protects topology; stage dependency fingerprints
+protect the exact upstream artifact consumed. The runner must never combine a
+new or changed producer with stale dependent output.
+
+Compatible producer checkpoints may be decoded through the registered codec
+and handed to later steps without rerunning the producer. A missing, rejected,
+corrupt, incompatible, or changed producer invalidates every transitive
+dependent checkpoint. Independent work remains reusable.
+
+Add one operator control, `--recompute-step <step-id>`, with these semantics:
+
+- it requires checkpoint recording and `--resume`;
+- the selected step and all transitive dependents execute rather than reuse
+  their checkpoints;
+- valid required predecessors and unrelated work remain reusable;
+- the recompute selection affects loader decisions, not the persistent
+  checkpoint identity of otherwise identical work; and
+- the command fails before dependent execution if a required predecessor has
+  no reusable accepted artifact.
+
+Existing `--only` behavior remains unchanged for implicit single-step
+pipelines. Combining `--only` with explicit multi-step pipelines is outside
+this scope and should be rejected with actionable guidance rather than given
+implicit dependency-expansion semantics.
+
+Checkpoint events, manifests, and diagnostics distinguish executed, reused,
+forced-recomputed, and dependency-invalidated work. Invalidation reasons are
+bounded, deterministic, and free of reference content, local paths, or secrets.
+Old checkpoint state need not be migrated; it must produce a safe, explicit
+cold miss rather than an error or unsafe reuse.
+
+### Failure, cancellation, and concurrency
+
+The existing run-wide worker and provider-call limits apply across every step.
+Workers may be reused between steps, but concurrency cannot cross a step
+barrier. A framework error cancels started work using the existing bounded
+drain behavior and prevents later steps and output encoding. Rejections remain
+recorded outcomes, but failure to produce a normalized artifact required by a
+generated binding escalates to the run-level dependency error described above.
+
+The failed manifest retains completed upstream outcomes, step and lane
+provenance, rejections, checkpoint events, and the dependency failure without
+embedding generated artifact content.
+
+## D&D Proving Workflow
+
+The production acceptance workflow has two explicit steps:
+
+1. `identify-npcs` runs the NPC lane through normalization and its complete
+   validator policy.
+2. `grounded-events` receives the canonical NPC artifact in its step-scoped
+   `npcs` reference and runs spell and combat-turn lanes. The binding reaches
+   spell extraction, combat-turn extraction, and combat-turn normalization.
+
+Spell and combat-turn lanes may execute concurrently after the handoff. NPC
+content may ground names and identities but cannot establish a spell cast or
+combat event; source units remain the only event evidence.
+
+The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be
+replaced or supplemented by one ordered-pipeline example. Existing module keys,
+artifact contracts, reference slot names, prompt IDs, and D&D evidence policy
+remain unchanged.
+
+## Included Work
+
+- Configuration parsing, validation, cloning, defaults, redaction, and
+  documentation for explicit steps and structured generated references.
+- Domain-neutral resolved step, dependency, producer, and consumer identities.
+- Generated-artifact compatibility metadata on reference-slot contracts,
+  including D&D NPC-list declarations for every `npcs` consumer.
+- Step-aware preparation metadata and runner orchestration.
+- Canonical codec handoff into existing reference request contracts.
+- Required-dependency failure and bounded provenance behavior.
+- Dependency-aware checkpoint reuse, invalidation, events, and selective step
+  recomputation.
+- D&D NPC-first production composition for spell and combat-turn consumers.
+- Refactoring the affected D&D consumers so generated NPC references are
+  available at operation time while retaining early static construction.
+- Maintained examples and updates to current architecture, configuration, CLI,
+  operations, internal, integration, and testing documentation when behavior
+  lands.
+- An ADR recording the bounded ordered-step extension to the fixed pipeline
+  architecture and its explicit rejection of a general DAG.
+
+## Explicitly Out Of Scope
+
+- D&D item extraction or any other new artifact lane.
+- Cross-artifact NPC ID fields or artifact-schema migration machinery.
+- Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or
+  module-controlled scheduling.
+- Multiple source inputs, per-step input adapters, per-step chunk plans, or
+  per-step output encoders.
+- Aggregating multiple generated artifacts into one reference slot.
+- Optional or best-effort generated dependencies; a configured dependency is
+  required in this scope.
+- Prior-run or cross-pipeline generated references.
+- `--only` dependency closure for explicit multi-step pipelines.
+- Cross-lane reconciliation or domain concepts in the generic framework.
+- Live-provider tests or model-quality changes to D&D prompts.
+
+## Acceptance Criteria
+
+The scope is complete when:
+
+- all existing single-step configurations retain their current behavior;
+- explicit step order and dependency topology resolve deterministically and
+  affect pipeline identity;
+- invalid producer, consumer, conflict, ordering, type, schema, media, and
+  cardinality configurations fail before source processing when statically
+  discoverable, while content-size violations fail at handoff;
+- no consumer step begins before all required generated artifacts are accepted,
+  canonicalized, and validated for its target slots;
+- one producer artifact fans out safely to every compatible target selected by
+  a step-scoped binding;
+- missing required producer output fails the complete run before dependent work;
+- changing NPC output invalidates spell and combat-turn checkpoints while
+  leaving compatible independent work reusable;
+- selective step recomputation executes exactly the selected dependency closure
+  and reports why work was executed, reused, or invalidated;
+- the D&D ordered workflow supplies NPC content to spell extraction, combat-turn
+  extraction, and combat-turn normalization without treating it as evidence;
+- completion timing cannot change public ordering, failure selection, or
+  dependency behavior;
+- output, manifests, checkpoints, and debug artifacts contain the required
+  identities and provenance without leaking generated reference content; and
+- repository-wide tests, vet, build, maintained-example checks, and
+  documentation validation pass.
+
+## Testing Strategy
+
+Tests should protect behavior and invariants rather than the implementation's
+internal scheduler shape.
+
+- Configuration contract tests own legacy shorthand, explicit step parsing,
+  source-form discrimination, conflicts, and redaction.
+- Resolution tests own ordering, global lane uniqueness, dependency validation,
+  slot compatibility, fan-out, cloning, canonical JSON, and digest changes.
+- Runner tests own step barriers, within-step bounded concurrency, stable
+  ordering, cancellation, required-producer failure, and immutable handoff.
+- Checkpoint tests own producer decoding, exact dependency matching, transitive
+  invalidation, forced recomputation, cold misses, and bounded decisions.
+- One CLI contract test should cover the recompute control and its invalid
+  combinations.
+- One D&D integration test with offline fake LLM responses should prove the
+  complete NPC-to-spell-and-combat handoff, including combat normalization.
+- Maintained configuration examples should be parsed and resolved through the
+  production catalog.
+
+Do not add scheduler choreography tests, exact goroutine-count assertions,
+complete manifest snapshots, exact diagnostic strings, or duplicated tests for
+every invalid configuration at every layer. No test may require credentials or
+a live model provider.
+
+## Open Questions
+
+None required to define this scope. Exact internal type names and implementation
+decomposition are intentionally not feature-policy decisions.