notarius/docs/roadmap/ordered-pipeline-steps.md

Proposed Scope: Ordered Pipeline Steps

Status

Proposed as the next implementation scope. This document refines the near-term priority in Future Work into a bounded feature target. It defines desired behavior and scope, not a file-by-file implementation plan.

Recommendation

Implement ordered pipeline steps, generated artifact references, and dependency-aware checkpoint reuse as one coherent platform capability. Prove the capability with the D&D workflow in which accepted normalized NPC output is produced first and then supplied to spell extraction, combat-turn extraction, and combat-turn normalization.

This should precede item extraction. It establishes the artifact-handoff and dependency semantics that the planned item lane will also consume, without combining two substantial features in one implementation effort.

Intended Outcome

A configured pipeline may contain multiple ordered steps while retaining one pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run manifest, and failure boundary. Every artifact lane still follows the fixed extract, validate, merge, validate, normalize, and validate lifecycle. Steps add explicit barriers between groups of lanes; they do not create arbitrary stage graphs.

An accepted normalized artifact from an earlier step may be bound explicitly to declared reference slots in a later step. The framework remains domain-neutral, and generated references remain contextual material rather than source evidence.

Fixed Product And Architecture Decisions

Pipeline shape

• Input parsing and chunk planning remain pipeline-wide and execute once.
• A step contains one or more artifact lanes. Step order is configuration order.
• Lanes within a step remain independent and may use the existing bounded concurrency model.
• A later step cannot begin until every lane in the current step is terminal and every generated artifact it requires is accepted and available.
• Public artifact and failure ordering is step order followed by deterministic lane and source-chunk order, never completion order.
• Output encoding occurs once, after every step succeeds.
• This is not an arbitrary DAG, a general workflow language, concurrent cross-lane reconciliation, or permission for modules to invoke other modules.

Configuration model

Existing single-step pipelines remain valid. A top-level artifacts map is treated as an implicit step with stable ID default. A pipeline may configure either artifacts or steps, but not both. Explicit steps must be non-empty and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique across the entire pipeline so output paths, selectors, manifests, errors, and checkpoint scopes remain unambiguous.

The target configuration shape is:

pipelines:
  dnd-session:
    input: seriatim
    chunk: generic
    steps:
      - id: identify-npcs
        artifacts:
          npcs:
            extract: dnd/npcs
            normalize: dnd/npcs
      - id: grounded-events
        references:
          npcs:
            artifact:
              step: identify-npcs
              lane: npcs
        artifacts:
          spells:
            extract: dnd/spells
            normalize: dnd/spells
          combat:
            extract: dnd/combat-turns
            normalize: dnd/combat-turns
    output: json

Existing scalar reference values continue to represent external file paths. The structured artifact form identifies accepted normalized output from one earlier step and lane. Generated artifact bindings are allowed at step scope or at an individual module target; they are not inferred from module keys, lane names, slot names, or domain knowledge.

A step-scoped reference applies automatically to every selected target in that step that declares the slot. In the example, one npcs binding reaches the spell extractor plus the combat extractor and normalizer. A target-local binding is used when only one module should consume the artifact.

Pipeline-level external references remain defaults. Step-local external references override pipeline-level external defaults, and target-local external references retain their existing precedence. A generated reference and an external reference may not resolve to the same effective target slot; configuration or a runtime override that creates that conflict is invalid. Likewise, a step-scoped and target-local generated binding cannot both target the same effective slot.

Each effective target slot accepts at most one producer. One producer may fan out to multiple compatible slots in a later step. Aggregating several producer artifacts into one slot is outside this scope.

Reference-slot specs gain optional generated-artifact compatibility metadata. A generated binding is allowed only when the consumer slot declares the producer's artifact kind; the producer's registered codec supplies the exact schema identity and media type used for the handoff. The D&D npcs consumer slots declare the normalized NPC-list artifact kind. Existing external-file slots and bindings retain their current behavior and do not acquire an artifact kind merely because their bytes happen to decode as one.

Resolution and preparation

Resolution validates the complete ordered structure before source processing. It must reject duplicate identities, missing producers, same-step or forward references, undeclared slots, reference conflicts, and incompatible artifact kind, schema, media type, or cardinality constraints that are statically discoverable. Size is checked when canonical producer bytes exist at handoff. Ordered steps make cycles structurally impossible; resolution must not introduce a general graph scheduler to rediscover their order.

The resolved pipeline and its digest include step order, step IDs, lane membership, generated-reference topology, producer identity, consumer targets, and existing module and validator policy. Cloning, redaction, canonical JSON, debug summaries, and manifests preserve the same structure without reference content or secrets.

All modules and validators are still selected, option-validated, and constructed before source parsing. Generated content cannot be supplied during construction because it does not exist yet. The framework therefore augments the existing operation-request References at the step boundary. Consumers that currently assume an NPC registry is construction-only must accept the generated registry from their operation request without deferring general module construction until after upstream work.

Only validation that inherently depends on generated bytes may occur at the handoff. A handoff validation failure is a contextual framework error and fails the run before any consumer in that step begins.

Generated artifact handoff

Only accepted normalized output may cross a step boundary. Raw extraction responses, rejected artifacts, merge intermediates, and validator diagnostics cannot be bound as references.

The framework serializes the producer through its registered canonical artifact codec and constructs one immutable reference item containing:

• the declared target slot;
• canonical artifact bytes and media type;
• artifact kind and schema ID, name, version, and schema digest;
• canonical content digest and size; and
• producer pipeline, step, lane, and module provenance.

A generated binding requires exactly one accepted normalized artifact from its producer lane. No artifact is a missing dependency, while more than one is a cardinality error; a typed collection such as an NPC list remains one artifact. Combining several normalized outputs into one reference is aggregation and is outside this scope.

The existing slot contract remains authoritative for accepted media types, maximum size, and cardinality. Generated content is cloned at ownership boundaries and never exposed through a filesystem path. Manifests and debug summaries record identities and bounded provenance, not artifact content.

Configuring a generated binding makes that dependency required even when the consumer module declares the underlying slot optional. An accepted artifact whose domain collection is empty is still a valid artifact and may be handed off. If the producer has no accepted normalized artifact, the entire run fails with a deterministic dependency error and no later step begins.

Checkpoint reuse and selective recomputation

Generated references participate in downstream checkpoint dependencies by artifact kind, complete schema identity, media type, and canonical content digest. The pipeline digest protects topology; stage dependency fingerprints protect the exact upstream artifact consumed. The runner must never combine a new or changed producer with stale dependent output.

Compatible producer checkpoints may be decoded through the registered codec and handed to later steps without rerunning the producer. A missing, rejected, corrupt, incompatible, or changed producer invalidates every transitive dependent checkpoint. Independent work remains reusable.

Add one operator control, --recompute-step <step-id>, with these semantics:

• it requires checkpoint recording and --resume;
• the selected step and all transitive dependents execute rather than reuse their checkpoints;
• valid required predecessors and unrelated work remain reusable;
• the recompute selection affects loader decisions, not the persistent checkpoint identity of otherwise identical work; and
• the command fails before dependent execution if a required predecessor has no reusable accepted artifact.

Existing --only behavior remains unchanged for implicit single-step pipelines. Combining --only with explicit multi-step pipelines is outside this scope and should be rejected with actionable guidance rather than given implicit dependency-expansion semantics.

Checkpoint events, manifests, and diagnostics distinguish executed, reused, forced-recomputed, and dependency-invalidated work. Invalidation reasons are bounded, deterministic, and free of reference content, local paths, or secrets. Old checkpoint state need not be migrated; it must produce a safe, explicit cold miss rather than an error or unsafe reuse.

Failure, cancellation, and concurrency

The existing run-wide worker and provider-call limits apply across every step. Workers may be reused between steps, but concurrency cannot cross a step barrier. A framework error cancels started work using the existing bounded drain behavior and prevents later steps and output encoding. Rejections remain recorded outcomes, but failure to produce a normalized artifact required by a generated binding escalates to the run-level dependency error described above.

The failed manifest retains completed upstream outcomes, step and lane provenance, rejections, checkpoint events, and the dependency failure without embedding generated artifact content.

D&D Proving Workflow

The production acceptance workflow has two explicit steps:

1. identify-npcs runs the NPC lane through normalization and its complete validator policy.
2. grounded-events receives the canonical NPC artifact in its step-scoped npcs reference and runs spell and combat-turn lanes. The binding reaches spell extraction, combat-turn extraction, and combat-turn normalization.

Spell and combat-turn lanes may execute concurrently after the handoff. NPC content may ground names and identities but cannot establish a spell cast or combat event; source units remain the only event evidence.

The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be replaced or supplemented by one ordered-pipeline example. Existing module keys, artifact contracts, reference slot names, prompt IDs, and D&D evidence policy remain unchanged.

Included Work

• Configuration parsing, validation, cloning, defaults, redaction, and documentation for explicit steps and structured generated references.
• Domain-neutral resolved step, dependency, producer, and consumer identities.
• Generated-artifact compatibility metadata on reference-slot contracts, including D&D NPC-list declarations for every npcs consumer.
• Step-aware preparation metadata and runner orchestration.
• Canonical codec handoff into existing reference request contracts.
• Required-dependency failure and bounded provenance behavior.
• Dependency-aware checkpoint reuse, invalidation, events, and selective step recomputation.
• D&D NPC-first production composition for spell and combat-turn consumers.
• Refactoring the affected D&D consumers so generated NPC references are available at operation time while retaining early static construction.
• Maintained examples and updates to current architecture, configuration, CLI, operations, internal, integration, and testing documentation when behavior lands.
• An ADR recording the bounded ordered-step extension to the fixed pipeline architecture and its explicit rejection of a general DAG.

Explicitly Out Of Scope

• D&D item extraction or any other new artifact lane.
• Cross-artifact NPC ID fields or artifact-schema migration machinery.
• Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or module-controlled scheduling.
• Multiple source inputs, per-step input adapters, per-step chunk plans, or per-step output encoders.
• Aggregating multiple generated artifacts into one reference slot.
• Optional or best-effort generated dependencies; a configured dependency is required in this scope.
• Prior-run or cross-pipeline generated references.
• --only dependency closure for explicit multi-step pipelines.
• Cross-lane reconciliation or domain concepts in the generic framework.
• Live-provider tests or model-quality changes to D&D prompts.

Acceptance Criteria

The scope is complete when:

• all existing single-step configurations retain their current behavior;
• explicit step order and dependency topology resolve deterministically and affect pipeline identity;
• invalid producer, consumer, conflict, ordering, type, schema, media, and cardinality configurations fail before source processing when statically discoverable, while content-size violations fail at handoff;
• no consumer step begins before all required generated artifacts are accepted, canonicalized, and validated for its target slots;
• one producer artifact fans out safely to every compatible target selected by a step-scoped binding;
• missing required producer output fails the complete run before dependent work;
• changing NPC output invalidates spell and combat-turn checkpoints while leaving compatible independent work reusable;
• selective step recomputation executes exactly the selected dependency closure and reports why work was executed, reused, or invalidated;
• the D&D ordered workflow supplies NPC content to spell extraction, combat-turn extraction, and combat-turn normalization without treating it as evidence;
• completion timing cannot change public ordering, failure selection, or dependency behavior;
• output, manifests, checkpoints, and debug artifacts contain the required identities and provenance without leaking generated reference content; and
• repository-wide tests, vet, build, maintained-example checks, and documentation validation pass.

Testing Strategy

Tests should protect behavior and invariants rather than the implementation's internal scheduler shape.

• Configuration contract tests own legacy shorthand, explicit step parsing, source-form discrimination, conflicts, and redaction.
• Resolution tests own ordering, global lane uniqueness, dependency validation, slot compatibility, fan-out, cloning, canonical JSON, and digest changes.
• Runner tests own step barriers, within-step bounded concurrency, stable ordering, cancellation, required-producer failure, and immutable handoff.
• Checkpoint tests own producer decoding, exact dependency matching, transitive invalidation, forced recomputation, cold misses, and bounded decisions.
• One CLI contract test should cover the recompute control and its invalid combinations.
• One D&D integration test with offline fake LLM responses should prove the complete NPC-to-spell-and-combat handoff, including combat normalization.
• Maintained configuration examples should be parsed and resolved through the production catalog.

Do not add scheduler choreography tests, exact goroutine-count assertions, complete manifest snapshots, exact diagnostic strings, or duplicated tests for every invalid configuration at every layer. No test may require credentials or a live model provider.

Open Questions

None required to define this scope. Exact internal type names and implementation decomposition are intentionally not feature-policy decisions.