notarius/docs/roadmap/implementation.md

# Implementation Plan: Ordered Pipeline Steps

## Purpose

Implement the target state in [Ordered Pipeline Steps](ordered-pipeline-steps.md).
That feature roadmap is the authority for product intent, policy choices,
acceptance criteria, and exclusions. This document turns it into an ordered,
decision-complete implementation sequence for a coding agent.

This is a platform refactor followed by one D&D proving workflow. It does not
include item extraction, a general DAG scheduler, schema migrations for D&D
artifacts, or any other work excluded by the feature roadmap.

## Instructions For Every Stage

Before changing code, read `docs/development.md` and all documents under
`docs/policy/`. Preserve the fixed pipeline lifecycle and the two-zone data
model described there. Use the repository's code knowledge graph for code
discovery before falling back to text search.

Implement the stages in order. Each stage must leave the repository formatted,
building, and passing the focused tests it changes. Do not maintain two
competing internal representations merely to reduce a refactor: legacy
top-level `artifacts` is a configuration compatibility form, while ordered
steps are the single resolved, prepared, and runtime representation.

Follow these cross-stage rules:

- Keep input parsing, chunk planning, output encoding, run identity, worker
  budgets, provider scheduling, and the failure boundary pipeline-wide.
- Preserve the lane lifecycle and existing typed module interfaces. Steps
  schedule lanes; they do not permit modules to call other modules.
- Construct every selected module and validator before source parsing. Supply
  generated bytes only through cloned operation-request `References`.
- Resolve and report deterministic order as step order, then lane ID, then
  source/chunk order. Never expose completion order.
- Treat generated references as context, not source evidence. Never put their
  content in manifests, checkpoint decisions, logs, or debug summaries.
- Add narrow, behavior-oriented, offline tests at the stable owner of each new
  invariant. Do not add scheduler choreography, exact goroutine counts, full
  manifest snapshots, exact error-string snapshots, or live-provider tests.
- Keep configuration file version 3. The new fields are a backward-compatible
  extension; checkpoint wire state receives its own explicit compatibility
  change in Stage 4.
- Update current-behavior documentation only in Stage 6, when the complete
  behavior exists. The ADR may be added in Stage 1 because it records the
  architecture decision rather than claiming released behavior.

## Canonical Internal Model

Use the following model throughout the implementation. Exact unexported helper
names may follow local conventions, but their responsibilities and invariants
are fixed here.

### Configuration and profile types

- Add `PipelineStepProfile` with `ID`, `Artifacts`, and `References` fields.
- Add `Steps []PipelineStepProfile` to `PipelineProfile`; retain top-level
  `Artifacts` only as the legacy input form.
- Replace reference-map values in profile and file-config types with a
  discriminated `ReferenceSource`. It has exactly one of:
  - an external path, represented by the existing scalar YAML form; or
  - `ArtifactReference{Step, Lane}`, represented by
    `artifact: {step: ..., lane: ...}`.
- Keep CLI `--reference` values external-path overrides. They do not create or
  replace generated bindings. `--reference-unbind` may remove an effective
  external binding, but must not silently remove a generated dependency.
- Step-level `References` use the same source type as pipeline and target-local
  references. Pipeline-level generated references are invalid because there is
  no well-defined consumer step and they could imply a forward dependency.

### Resolved and prepared types

- Make `ResolvedPipeline.Steps []ResolvedPipelineStep` the sole lane container.
  Each resolved step has a stable `ID`, ordered `ArtifactLanes`, and expanded
  generated consumer bindings. Remove the flat `ArtifactLanes` field after all
  callers migrate; provide read-only iteration/lookup helpers where callers
  need all lanes.
- Resolve legacy top-level `artifacts` as one step named `default`. Do not add a
  synthetic step back into effective user configuration output.
- Keep lane IDs globally unique. A producer and consumer are identified by
  pipeline ID, step ID, and lane ID; module targets additionally include stage,
  module key, and slot name.
- Represent external and generated bindings as distinct resolved variants.
  Never encode a producer selector into a path string.
- Mirror resolved steps in `PreparedPipeline`. A prepared step contains its
  already-constructed lanes and static external reference sets. Generated
  content is held only in run-local state and overlaid onto request references
  before a consumer step begins.

### Generated compatibility and handoff

- Extend `contracts.ReferenceSlot` with an optional cloned list of accepted
  generated `ArtifactKind` values. An empty list means external references only.
  Do not infer compatibility by decoding external bytes.
- A generated producer supplies exactly one accepted normalized artifact per
  binding in this feature. Zero artifacts is a missing dependency; more than
  one is a cardinality error. A collection such as an NPC list is one typed
  artifact, not multiple reference items. Supporting aggregation or several
  generated items in one slot remains out of scope.
- Use the producer kind's registered `ArtifactCodecSpec` as the canonical schema
  identity and media contract. Kind compatibility plus the single registered
  codec makes schema compatibility exact; also validate the consumer's media,
  size, and cardinality constraints.
- Extend `contracts.ReferenceItem` with optional generated-artifact identity and
  producer provenance rather than overloading file origin fields. Include kind,
  schema ID/name/version/digest, producer pipeline/step/lane/module, content
  digest, media type, and size. Clone byte slices and nested metadata at every
  ownership boundary.
- The runtime dependency fingerprint is the canonical tuple of producer
  identity, artifact kind, complete schema identity, media type, and content
  digest. It must be added to every extract, merge, normalize, and validator
  checkpoint dependency that receives the generated slot.

## Stage 1: Configuration, Contracts, And Resolution

### Objective

Accept and fully validate ordered steps and structured artifact reference
sources while preserving legacy single-step configuration. No runner behavior
changes belong in this stage.

### Changes

1. Add an ADR under `docs/adr/` following the existing numbering and template.
   Record the bounded ordered-step extension, the single pipeline-wide
   input/chunk/output boundary, explicit generated bindings, barriers between
   steps, and the rejection of a general DAG.
2. In `internal/core/config/file_config.go` and its validation/application
   helpers:
   - add file forms for ordered steps and discriminated reference sources;
   - use strict YAML decoding for the mapping form and reject unknown fields,
     empty selectors, ambiguous scalar-plus-artifact values, and non-string
     scalar paths;
   - reject pipelines containing both `artifacts` and `steps`, empty explicit
     steps, duplicate trimmed step IDs, and duplicate lane IDs across steps;
   - preserve list order for steps and validator chains and deterministic key
     ordering for lane maps; and
   - update cloning, defaults application, effective-config rendering, and
     redaction so selectors remain visible but external paths follow the current
     path redaction policy.
3. In `internal/framework/contracts/contracts.go`, add generated artifact-kind
   compatibility to `ReferenceSlot` and update `CloneReferenceSlots` to deep
   copy it. Keep existing external-slot behavior unchanged.
4. In `internal/framework/pipeline/profile.go` and reference-resolution helpers:
   - introduce the canonical step and reference-source types described above;
   - normalize legacy `artifacts` to resolved step `default`;
   - resolve all step lanes and module/validator contracts before resolving
     generated bindings, so producer codec metadata and consumer slots are
     available for compatibility checks;
   - expand a step-scoped binding to every selected target in that step that
     declares the slot, for both external and generated sources; do not treat
     targets without that slot as errors;
   - retain current pipeline, target-local, and CLI precedence for external
     bindings, inserting step-local external bindings between pipeline defaults
     and target-local bindings;
   - reject every generated/external collision on an effective target slot and
     every step-scoped/target-local generated collision;
   - reject missing producers, same-step or forward producers, undeclared
     target-local slots, non-normalized producer lanes, incompatible kinds,
     schemas or media types, and selectors made ambiguous by duplicate IDs; and
   - include step IDs/order, lane membership, producer selectors, expanded
     consumers, and module/validator policy in canonical cloning and the
     pipeline digest.
5. Migrate catalog, config validation, `--only` lane selection, checkpoint
   identity construction, reference provenance discovery, and debug/effective
   configuration code to iterate the resolved step model. Reject `--only` when
   the source profile has explicit steps; retain its current behavior for the
   implicit `default` step.

### Tests

- Extend `internal/core/config/*_test.go` with table-driven contracts for legacy
  shorthand, explicit step order, strict source-form parsing, mutual exclusion,
  identity collisions, cloning, effective output, and redaction.
- Extend `internal/framework/pipeline/profile_test.go`,
  `typed_resolution_test.go`, and reference tests for deterministic expansion,
  precedence/conflicts, backward/forward rules, global lane uniqueness,
  generated-kind/media compatibility, canonical cloning, and digest changes.
- Add a contract test showing a legacy profile resolves to `default` with an
  otherwise equivalent lane lifecycle and ordering.
- Prefer one representative case per validation owner; do not duplicate every
  parser failure at the resolver layer.

### Completion Gate

Run focused config, contracts, and pipeline resolution tests, then `go test
./internal/core/config ./internal/framework/contracts ./internal/framework/pipeline`.
The runner may still reject or lack execution support for more than one
resolved step, but all configurations and topology must resolve without a flat
parallel lane model.

## Stage 2: Step-Aware Preparation And Execution

### Objective

Execute independent ordered steps with hard barriers, even when no generated
references are configured. Preserve the existing typed lane lifecycle and one
bounded run-wide concurrency budget.

### Changes

1. Refactor `internal/framework/pipeline/prepare.go` so `Prepare` walks resolved
   steps in order and constructs every lane module and validator before it
   returns. A failure in any later step must occur before source parsing.
   Materialize only external reference sources during this phase; generated
   selectors carry no bytes yet.
2. Refactor `runner.go` and `runner_concurrent.go` into a small ordered-step
   coordinator plus the existing bounded lane engine:
   - parse input and compute/reuse the chunk plan once;
   - invoke the lane engine once per step, passing only that step's prepared
     lanes and immutable request-reference view;
   - wait for every lane in the step to become terminal before advancing;
   - reuse the same worker/provider limits without permitting tasks from
     adjacent steps to overlap; and
   - encode output once after all steps succeed.
3. Accumulate accepted, rejected, debug, and checkpoint outcomes across steps.
   Sort public results by step index, lane ID, source ID, and chunk index/ref as
   applicable. Add step identity to internal errors and debug events where lane
   identity alone no longer explains execution context.
4. Preserve current fail-fast cancellation and bounded drain behavior. A
   framework error in a step cancels its started work, prevents all later steps,
   and prevents output encoding while retaining completed upstream outcomes.
5. Until Stage 3 lands, reject execution of any pipeline with a generated
   binding before source parsing. This temporary guard prevents an accepted
   configuration from running a consumer without its declared dependency.

### Tests

- Add runner tests with deterministic fake modules proving strict barriers,
  concurrency within one step, reuse of the same global budget, stable output
  and failure order despite inverted completion timing, and no output encoding
  after failure.
- Add a preparation test proving every module and validator in every step is
  constructed before the input adapter is invoked.
- Retain existing single-step concurrency and cancellation tests unchanged
  where possible; migrate fixtures to the implicit `default` resolved step.

### Completion Gate

Run `go test ./internal/framework/pipeline ./internal/cli` plus the repository's
race-test target for the framework if one is defined in `docs/development.md`.
Both legacy pipelines and explicit pipelines without generated references must
run successfully.

## Stage 3: Canonical Generated-Artifact Handoff And Provenance

### Objective

Make accepted normalized output from an earlier lane available as an immutable
operation-time reference to later consumers, with safe checkpoint dependencies
and bounded provenance.

### Changes

1. Add a domain-neutral handoff component in `internal/framework/pipeline` that:
   - locates the producer's accepted normalized output after its step barrier;
   - requires exactly one output and rejects missing, rejected-only, or multiple
     outputs deterministically;
   - canonicalizes it through `ArtifactCodecRegistry.Encode` (decoding a reused
     serialized checkpoint through the registered codec first when necessary);
   - verifies kind, exact codec schema identity, accepted media type, maximum
     size, and one-item cardinality for every expanded consumer slot; and
   - creates independently cloned `ReferenceItem` values for fan-out targets.
2. Build all generated bindings required by a consumer step before starting any
   lane in that step. If any handoff fails, return one contextual dependency
   error and start none of the step's consumers. An accepted typed empty list is
   valid; absence of an accepted artifact is not.
3. Merge generated items with each target's already-materialized external
   `ReferenceSet` only after resolution has proven there is no collision. Pass
   the resulting cloned set through existing extraction, merge, normalize, and
   validation request structs. Never mutate prepared static reference sets.
4. Add the canonical generated dependency fingerprint to checkpoint loader and
   recorder inputs for every receiving operation immediately in this stage.
   This is required before generated pipelines can safely use resume; do not
   defer it to selective recomputation work.
5. Extend artifacts and runtime provenance:
   - add step ID to artifact-lane, normalized-output, rejected-output, and
     reference provenance records where needed for unambiguous context;
   - record external and generated origins distinctly;
   - for generated references record producer identities, codec schema
     identity, media type, digest, and size, but never content or a fabricated
     filesystem URI; and
   - update generic JSON output and debug summaries to serialize these bounded
     fields deterministically.
6. Ensure checkpoint, debug, and output clones own their byte slices and maps.
   A consumer or test mutation must not affect another fan-out consumer or the
   producer artifact.

### Tests

- Add handoff tests for one producer fan-out, empty typed collections, missing
  output, rejected-only output, multiple outputs, type/schema/media/size
  mismatch, target immutability, and failure before consumer start.
- Add a resume-oriented runner test proving changed canonical producer content
  changes the consumer dependency fingerprint and prevents stale reuse.
- Extend artifacts/JSON/debug contract tests with field assertions for bounded
  generated provenance and explicit assertions that content and local paths are
  absent. Avoid whole-document snapshots.

### Completion Gate

Run focused pipeline, artifacts, JSON output, and debug tests, followed by `go
test ./internal/framework/... ./internal/core/artifacts
./internal/modules/generic/output/json`. A generic fake-codec pipeline must
complete a two-step handoff both from fresh execution and a compatible reused
producer checkpoint.

## Stage 4: Dependency-Aware Checkpoints And Selective Recompute

### Objective

Complete safe reuse, transitive invalidation, observable decisions, and the
`--recompute-step` operator control.

### Changes

1. Update checkpoint identity and manifests in
   `internal/framework/checkpoint`:
   - introduce workspace schema `notarius.workspace.v3` and retain explicit
     recognition of v1/v2 as incompatible cold misses;
   - include step ID in lane-stage manifests and lookup/recording context;
   - include ordered topology in the persistent pipeline identity; and
   - preserve the Stage 3 generated fingerprint tuple exactly, without content
     or secrets in decision reasons.
2. When loading a normalized producer checkpoint, validate its manifest,
   deserialize through the registered codec, re-encode canonically, and compare
   schema/media/content fingerprints before exposing it to handoff. Corrupt,
   missing, rejected, or incompatible state is a cold miss for ordinary resume
   and must never reach a consumer.
3. Build a lane-level dependency index from resolved generated bindings. Use it
   to invalidate only transitive consumer lanes when producer identity or bytes
   change; unrelated lanes, including unrelated lanes in a later step, remain
   eligible for reuse. The step barrier still applies when reused and executed
   lanes coexist.
4. Add a single-value `--recompute-step <step-id>` flag in
   `internal/cli/run.go`. Reject repeated occurrences, unknown steps, use
   without checkpoint recording, use without `--resume`, and combination with
   `--only`. Accept the stable implicit step ID `default`; selecting it forces
   every lane in that single step. One selected step is sufficient for this
   scope; do not add multi-selection semantics.
5. Convert the selected step to a force-execution set containing every lane in
   that step plus the lane-level transitive dependency closure. Loader policy,
   not persistent checkpoint identity, applies the force set. Required
   predecessors and unrelated lanes remain reusable. If a required predecessor
   was not selected and has no reusable accepted artifact, fail before starting
   a dependent lane rather than implicitly recomputing it.
6. Replace the boolean-only checkpoint reporting model as needed with a bounded
   decision category: `executed`, `reused`, `forced_recompute`, or
   `dependency_invalidated`, plus a stable reason code and optional safe detail.
   Propagate it to checkpoint events, manifests, debug summaries, and CLI
   diagnostics. Do not make exact prose part of a test contract.

### Tests

- In checkpoint loader/recorder tests, cover v1/v2 cold misses, v3 step
  identity, compatible producer decode, corruption, exact dependency match,
  content/schema changes, and bounded reason fields.
- In pipeline tests, cover transitive invalidation and reuse of unrelated work.
- Add one CLI contract table for valid recomputation and every invalid flag
  combination, plus one execution test proving the selected closure is forced
  while a predecessor and unrelated lane are reused.
- Do not assert internal loader call counts when observable decision records
  and outputs establish the behavior.

### Completion Gate

Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline
./internal/cli`, then checkpoint/debug state tests under the race detector as
directed by `docs/development.md`. Manually inspect one test fixture's decision
records to confirm they contain no reference bytes, secret values, or local
paths.

## Stage 5: D&D NPC-First Production Adoption

### Objective

Adopt the platform in the production D&D composition: NPC normalization runs
first, and its canonical artifact grounds spell extraction, combat-turn
extraction, and combat-turn normalization at operation time.

### Changes

1. Declare the normalized NPC-list artifact kind on every `npcs` consumer slot:
   - `internal/modules/dnd/extract/spells`;
   - `internal/modules/dnd/extract/combatturns`; and
   - `internal/modules/dnd/normalize/combatturns`.
   Keep the slot optional for external standalone use; a configured generated
   binding becomes required through the framework dependency.
2. Refactor `internal/modules/dnd/npcs/registry` to provide a package-owned,
   concurrency-safe operation-time resolver/cache:
   - validate and seed construction-time external references so bad static
     configuration still fails before source parsing;
   - resolve the effective `npcs` item from each operation request;
   - reuse the seeded immutable registry when its digest matches and cache a
     generated registry by canonical digest so concurrent chunk operations do
     not repeatedly decode it; and
   - return cloned or immutable views and never retain caller-owned content.
3. Change the spell extractor, combat-turn extractor, and combat-turn
   normalizer to obtain the NPC registry from `req.References` for each
   operation through that resolver. Do not reconstruct modules at a step
   boundary. Retain current behavior when the request has no NPC item.
4. Keep static prompt/schema metadata and construction-time external reference
   fingerprints intact. Record generated NPC identity through framework
   reference provenance and dependency fingerprints; do not place
   operation-varying generated digests into singleton module metadata.
5. Preserve prompt ordering and evidence policy. The generated NPC listing is
   stable contextual input before the variable transcript, but it cannot prove
   that a spell or combat event occurred. Source-unit citations remain the only
   event evidence.
6. Add a maintained explicit two-step D&D configuration under `examples/` and a
   matching production-catalog fixture under
   `internal/modules/integration/testdata/`. Replace or supplement the two
   manual NPC-to-spell and NPC-to-combat configurations with this single
   NPC-first workflow; retain standalone examples that demonstrate external
   NPC references where they remain useful.
7. Add one offline integration test using scripted/fake LLM responses. Assert
   NPC normalization completes first, the exact canonical NPC digest reaches
   all three consumers, spell and combat lanes may both succeed, combat
   normalization uses the registry, and generated NPC context is not accepted
   as source evidence by itself.

### Tests

- Add focused registry and module tests for static fallback, generated override,
  concurrent cache safety, malformed runtime content, and ownership isolation.
- Extend prompt-input tests only for semantic placement and content. Do not add
  exact prompt-length or message-count change detectors.
- Run the single end-to-end offline D&D handoff test rather than duplicating it
  separately for every consumer.

### Completion Gate

Run all D&D package tests and integration tests, including the race detector for
the registry cache, then parse and resolve every maintained example through the
production catalog. No test may require credentials or a live LLM provider.

## Stage 6: Current Documentation And Release Verification

### Objective

Document the now-implemented behavior in its canonical current owners, retire
obsolete manual workflow guidance, and perform repository-wide verification.

### Changes

1. Update `docs/config.md` with version-3 `steps`, legacy `artifacts`
   compatibility, reference source forms, precedence/conflict rules, global
   identity rules, and the exact D&D two-step example.
2. Update `docs/cli.md` with `--recompute-step`, prerequisites, closure
   semantics, invalid combinations, and checkpoint decision categories.
3. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and
   `docs/internal/overview.md` with the fixed ordered-step model, preparation
   timing, handoff boundary, codec use, provenance, and checkpoint dependency
   behavior. Keep the architecture description linear and explicitly state that
   this is not a general DAG.
4. Update `docs/operations.md` with resume/recompute procedures and safe failure
   diagnosis. Update the D&D integration documents for generated NPC context,
   evidence limitations, and the maintained workflow. Update
   `docs/policy/testing.md` only if the implementation exposes a genuinely new
   durable testing policy; do not restate feature-specific tests there.
5. Update maintained example checks and any README/index links that point to
   replaced sequential examples. Keep the feature roadmap as historical target
   state until the project applies its normal roadmap-completion process; do not
   turn it into a duplicate current-behavior manual.
6. Review changed exported identifiers and package comments, run formatting and
   static analysis, and remove obsolete flat-lane compatibility helpers,
   temporary test adapters, dead sequential-workflow code, and TODOs introduced
   during earlier stages.

### Verification

Run the repository-prescribed commands from `docs/development.md`, including:

1. all unit and integration tests;
2. race tests for concurrency-sensitive framework, checkpoint, and D&D registry
   packages;
3. `go vet` and the normal build;
4. maintained-example/config validation; and
5. documentation checks.

Perform one final acceptance review against every bullet in
[Ordered Pipeline Steps](ordered-pipeline-steps.md). Inspect a successful fresh
run, a resumed run, a forced-recompute run, and a missing-producer failure for
deterministic ordering, correct dependency decisions, bounded provenance, and
absence of generated content or secrets in manifests/debug state.

### Completion Gate

The feature is complete only when every acceptance criterion in the feature
roadmap is demonstrated by a stable test or maintained example and the full
repository verification suite passes.

## Open Questions

None. The feature roadmap and this plan fix all product and architecture choices
required for implementation. If an implementation detail conflicts with a
policy document, the policy document takes precedence; if it would change the
product semantics above, stop and amend the roadmap rather than deciding it in
code.