notarius/docs/roadmap/ordered-pipeline-steps.md

# Proposed Scope: Ordered Pipeline Steps

## Status

Proposed as the next implementation scope. This document refines the
near-term priority in [Future Work](future.md) into a bounded feature target. It
defines desired behavior and scope, not a file-by-file implementation plan.

## Recommendation

Implement ordered pipeline steps, generated artifact references, and
dependency-aware checkpoint reuse as one coherent platform capability. Prove
the capability with the D&D workflow in which accepted normalized NPC output
is produced first and then supplied to spell extraction, combat-turn
extraction, and combat-turn normalization.

This should precede item extraction. It establishes the artifact-handoff and
dependency semantics that the planned item lane will also consume, without
combining two substantial features in one implementation effort.

## Intended Outcome

A configured pipeline may contain multiple ordered steps while retaining one
pipeline-wide input, chunk plan, output, worker budget, LLM scheduler, run
manifest, and failure boundary. Every artifact lane still follows the fixed
extract, validate, merge, validate, normalize, and validate lifecycle. Steps
add explicit barriers between groups of lanes; they do not create arbitrary
stage graphs.

An accepted normalized artifact from an earlier step may be bound explicitly
to declared reference slots in a later step. The framework remains
domain-neutral, and generated references remain contextual material rather than
source evidence.

## Fixed Product And Architecture Decisions

### Pipeline shape

- Input parsing and chunk planning remain pipeline-wide and execute once.
- A step contains one or more artifact lanes. Step order is configuration order.
- Lanes within a step remain independent and may use the existing bounded
  concurrency model.
- A later step cannot begin until every lane in the current step is terminal
  and every generated artifact it requires is accepted and available.
- Public artifact and failure ordering is step order followed by deterministic
  lane and source-chunk order, never completion order.
- Output encoding occurs once, after every step succeeds.
- This is not an arbitrary DAG, a general workflow language, concurrent
  cross-lane reconciliation, or permission for modules to invoke other modules.

### Configuration model

Existing single-step pipelines remain valid. A top-level `artifacts` map is
treated as an implicit step with stable ID `default`. A pipeline may configure
either `artifacts` or `steps`, but not both. Explicit steps must be non-empty
and have unique, trimmed, non-empty IDs. Artifact lane IDs must remain unique
across the entire pipeline so output paths, selectors, manifests, errors, and
checkpoint scopes remain unambiguous.

The target configuration shape is:

```yaml
pipelines:
  dnd-session:
    input: seriatim
    chunk: generic
    steps:
      - id: identify-npcs
        artifacts:
          npcs:
            extract: dnd/npcs
            normalize: dnd/npcs
      - id: grounded-events
        references:
          npcs:
            artifact:
              step: identify-npcs
              lane: npcs
        artifacts:
          spells:
            extract: dnd/spells
            normalize: dnd/spells
          combat:
            extract: dnd/combat-turns
            normalize: dnd/combat-turns
    output: json
```

Existing scalar reference values continue to represent external file paths.
The structured `artifact` form identifies accepted normalized output from one
earlier step and lane. Generated artifact bindings are allowed at step scope or
at an individual module target; they are not inferred from module keys, lane
names, slot names, or domain knowledge.

A step-scoped reference applies automatically to every selected target in that
step that declares the slot. In the example, one `npcs` binding reaches the
spell extractor plus the combat extractor and normalizer. A target-local
binding is used when only one module should consume the artifact.

Pipeline-level external references remain defaults. Step-local external
references override pipeline-level external defaults, and target-local
external references retain their existing precedence. A generated reference
and an external reference may not resolve to the same effective target slot;
configuration or a runtime override that creates that conflict is invalid.
Likewise, a step-scoped and target-local generated binding cannot both target
the same effective slot.

Each effective target slot accepts at most one producer. One producer may fan
out to multiple compatible slots in a later step. Aggregating several producer
artifacts into one slot is outside this scope.

Reference-slot specs gain optional generated-artifact compatibility metadata.
A generated binding is allowed only when the consumer slot declares the
producer's artifact kind; the producer's registered codec supplies the exact
schema identity and media type used for the handoff. The D&D `npcs` consumer
slots declare the normalized NPC-list artifact kind. Existing external-file
slots and bindings retain their current behavior and do not acquire an artifact
kind merely because their bytes happen to decode as one.

### Resolution and preparation

Resolution validates the complete ordered structure before source processing.
It must reject duplicate identities, missing producers, same-step or forward
references, undeclared slots, reference conflicts, and incompatible artifact
kind, schema, media type, or cardinality constraints that are statically
discoverable. Size is checked when canonical producer bytes exist at handoff.
Ordered steps make cycles structurally impossible; resolution must not
introduce a general graph scheduler to rediscover their order.

The resolved pipeline and its digest include step order, step IDs, lane
membership, generated-reference topology, producer identity, consumer targets,
and existing module and validator policy. Cloning, redaction, canonical JSON,
debug summaries, and manifests preserve the same structure without reference
content or secrets.

All modules and validators are still selected, option-validated, and
constructed before source parsing. Generated content cannot be supplied during
construction because it does not exist yet. The framework therefore augments
the existing operation-request `References` at the step boundary. Consumers
that currently assume an NPC registry is construction-only must accept the
generated registry from their operation request without deferring general
module construction until after upstream work.

Only validation that inherently depends on generated bytes may occur at the
handoff. A handoff validation failure is a contextual framework error and fails
the run before any consumer in that step begins.

### Generated artifact handoff

Only accepted normalized output may cross a step boundary. Raw extraction
responses, rejected artifacts, merge intermediates, and validator diagnostics
cannot be bound as references.

The framework serializes the producer through its registered canonical artifact
codec and constructs one immutable reference item containing:

- the declared target slot;
- canonical artifact bytes and media type;
- artifact kind and schema ID, name, version, and schema digest;
- canonical content digest and size; and
- producer pipeline, step, lane, and module provenance.

A generated binding requires exactly one accepted normalized artifact from its
producer lane. No artifact is a missing dependency, while more than one is a
cardinality error; a typed collection such as an NPC list remains one artifact.
Combining several normalized outputs into one reference is aggregation and is
outside this scope.

The existing slot contract remains authoritative for accepted media types,
maximum size, and cardinality. Generated content is cloned at ownership
boundaries and never exposed through a filesystem path. Manifests and debug
summaries record identities and bounded provenance, not artifact content.

Configuring a generated binding makes that dependency required even when the
consumer module declares the underlying slot optional. An accepted artifact
whose domain collection is empty is still a valid artifact and may be handed
off. If the producer has no accepted normalized artifact, the entire run fails
with a deterministic dependency error and no later step begins.

### Checkpoint reuse and selective recomputation

Generated references participate in downstream checkpoint dependencies by
artifact kind, complete schema identity, media type, and canonical content
digest. The pipeline digest protects topology; stage dependency fingerprints
protect the exact upstream artifact consumed. The runner must never combine a
new or changed producer with stale dependent output.

Compatible producer checkpoints may be decoded through the registered codec
and handed to later steps without rerunning the producer. A missing, rejected,
corrupt, incompatible, or changed producer invalidates every transitive
dependent checkpoint. Independent work remains reusable.

Add one operator control, `--recompute-step <step-id>`, with these semantics:

- it requires checkpoint recording and `--resume`;
- the selected step and all transitive dependents execute rather than reuse
  their checkpoints;
- valid required predecessors and unrelated work remain reusable;
- the recompute selection affects loader decisions, not the persistent
  checkpoint identity of otherwise identical work; and
- the command fails before dependent execution if a required predecessor has
  no reusable accepted artifact.

Existing `--only` behavior remains unchanged for implicit single-step
pipelines. Combining `--only` with explicit multi-step pipelines is outside
this scope and should be rejected with actionable guidance rather than given
implicit dependency-expansion semantics.

Checkpoint events, manifests, and diagnostics distinguish executed, reused,
forced-recomputed, and dependency-invalidated work. Invalidation reasons are
bounded, deterministic, and free of reference content, local paths, or secrets.
Old checkpoint state need not be migrated; it must produce a safe, explicit
cold miss rather than an error or unsafe reuse.

### Failure, cancellation, and concurrency

The existing run-wide worker and provider-call limits apply across every step.
Workers may be reused between steps, but concurrency cannot cross a step
barrier. A framework error cancels started work using the existing bounded
drain behavior and prevents later steps and output encoding. Rejections remain
recorded outcomes, but failure to produce a normalized artifact required by a
generated binding escalates to the run-level dependency error described above.

The failed manifest retains completed upstream outcomes, step and lane
provenance, rejections, checkpoint events, and the dependency failure without
embedding generated artifact content.

## D&D Proving Workflow

The production acceptance workflow has two explicit steps:

1. `identify-npcs` runs the NPC lane through normalization and its complete
   validator policy.
2. `grounded-events` receives the canonical NPC artifact in its step-scoped
   `npcs` reference and runs spell and combat-turn lanes. The binding reaches
   spell extraction, combat-turn extraction, and combat-turn normalization.

Spell and combat-turn lanes may execute concurrently after the handoff. NPC
content may ground names and identities but cannot establish a spell cast or
combat event; source units remain the only event evidence.

The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be
replaced or supplemented by one ordered-pipeline example. Existing module keys,
artifact contracts, reference slot names, prompt IDs, and D&D evidence policy
remain unchanged.

## Included Work

- Configuration parsing, validation, cloning, defaults, redaction, and
  documentation for explicit steps and structured generated references.
- Domain-neutral resolved step, dependency, producer, and consumer identities.
- Generated-artifact compatibility metadata on reference-slot contracts,
  including D&D NPC-list declarations for every `npcs` consumer.
- Step-aware preparation metadata and runner orchestration.
- Canonical codec handoff into existing reference request contracts.
- Required-dependency failure and bounded provenance behavior.
- Dependency-aware checkpoint reuse, invalidation, events, and selective step
  recomputation.
- D&D NPC-first production composition for spell and combat-turn consumers.
- Refactoring the affected D&D consumers so generated NPC references are
  available at operation time while retaining early static construction.
- Maintained examples and updates to current architecture, configuration, CLI,
  operations, internal, integration, and testing documentation when behavior
  lands.
- An ADR recording the bounded ordered-step extension to the fixed pipeline
  architecture and its explicit rejection of a general DAG.

## Explicitly Out Of Scope

- D&D item extraction or any other new artifact lane.
- Cross-artifact NPC ID fields or artifact-schema migration machinery.
- Arbitrary DAGs, conditional branches, loops, joins, dynamic step creation, or
  module-controlled scheduling.
- Multiple source inputs, per-step input adapters, per-step chunk plans, or
  per-step output encoders.
- Aggregating multiple generated artifacts into one reference slot.
- Optional or best-effort generated dependencies; a configured dependency is
  required in this scope.
- Prior-run or cross-pipeline generated references.
- `--only` dependency closure for explicit multi-step pipelines.
- Cross-lane reconciliation or domain concepts in the generic framework.
- Live-provider tests or model-quality changes to D&D prompts.

## Acceptance Criteria

The scope is complete when:

- all existing single-step configurations retain their current behavior;
- explicit step order and dependency topology resolve deterministically and
  affect pipeline identity;
- invalid producer, consumer, conflict, ordering, type, schema, media, and
  cardinality configurations fail before source processing when statically
  discoverable, while content-size violations fail at handoff;
- no consumer step begins before all required generated artifacts are accepted,
  canonicalized, and validated for its target slots;
- one producer artifact fans out safely to every compatible target selected by
  a step-scoped binding;
- missing required producer output fails the complete run before dependent work;
- changing NPC output invalidates spell and combat-turn checkpoints while
  leaving compatible independent work reusable;
- selective step recomputation executes exactly the selected dependency closure
  and reports why work was executed, reused, or invalidated;
- the D&D ordered workflow supplies NPC content to spell extraction, combat-turn
  extraction, and combat-turn normalization without treating it as evidence;
- completion timing cannot change public ordering, failure selection, or
  dependency behavior;
- output, manifests, checkpoints, and debug artifacts contain the required
  identities and provenance without leaking generated reference content; and
- repository-wide tests, vet, build, maintained-example checks, and
  documentation validation pass.

## Testing Strategy

Tests should protect behavior and invariants rather than the implementation's
internal scheduler shape.

- Configuration contract tests own legacy shorthand, explicit step parsing,
  source-form discrimination, conflicts, and redaction.
- Resolution tests own ordering, global lane uniqueness, dependency validation,
  slot compatibility, fan-out, cloning, canonical JSON, and digest changes.
- Runner tests own step barriers, within-step bounded concurrency, stable
  ordering, cancellation, required-producer failure, and immutable handoff.
- Checkpoint tests own producer decoding, exact dependency matching, transitive
  invalidation, forced recomputation, cold misses, and bounded decisions.
- One CLI contract test should cover the recompute control and its invalid
  combinations.
- One D&D integration test with offline fake LLM responses should prove the
  complete NPC-to-spell-and-combat handoff, including combat normalization.
- Maintained configuration examples should be parsed and resolved through the
  production catalog.

Do not add scheduler choreography tests, exact goroutine-count assertions,
complete manifest snapshots, exact diagnostic strings, or duplicated tests for
every invalid configuration at every layer. No test may require credentials or
a live model provider.

## Open Questions

None required to define this scope. Exact internal type names and implementation
decomposition are intentionally not feature-policy decisions.