notarius/docs/roadmap/implementation.md

Implementation Plan: Ordered Pipeline Steps

Purpose

Implement the target state in Ordered Pipeline Steps. That feature roadmap is the authority for product intent, policy choices, acceptance criteria, and exclusions. This document turns it into an ordered, decision-complete implementation sequence for a coding agent.

This is a platform refactor followed by one D&D proving workflow. It does not include item extraction, a general DAG scheduler, schema migrations for D&D artifacts, or any other work excluded by the feature roadmap.

Instructions For Every Stage

Before changing code, read docs/development.md and all documents under docs/policy/. Preserve the fixed pipeline lifecycle and the two-zone data model described there. Use the repository's code knowledge graph for code discovery before falling back to text search.

Implement the stages in order. Each stage must leave the repository formatted, building, and passing the focused tests it changes. Do not maintain two competing internal representations merely to reduce a refactor: legacy top-level artifacts is a configuration compatibility form, while ordered steps are the single resolved, prepared, and runtime representation.

Follow these cross-stage rules:

• Keep input parsing, chunk planning, output encoding, run identity, worker budgets, provider scheduling, and the failure boundary pipeline-wide.
• Preserve the lane lifecycle and existing typed module interfaces. Steps schedule lanes; they do not permit modules to call other modules.
• Construct every selected module and validator before source parsing. Supply generated bytes only through cloned operation-request References.
• Resolve and report deterministic order as step order, then lane ID, then source/chunk order. Never expose completion order.
• Treat generated references as context, not source evidence. Never put their content in manifests, checkpoint decisions, logs, or debug summaries.
• Add narrow, behavior-oriented, offline tests at the stable owner of each new invariant. Do not add scheduler choreography, exact goroutine counts, full manifest snapshots, exact error-string snapshots, or live-provider tests.
• Keep configuration file version 3. The new fields are a backward-compatible extension; checkpoint wire state receives its own explicit compatibility change in Stage 4.
• Update current-behavior documentation only in Stage 6, when the complete behavior exists. The ADR may be added in Stage 1 because it records the architecture decision rather than claiming released behavior.

Canonical Internal Model

Use the following model throughout the implementation. Exact unexported helper names may follow local conventions, but their responsibilities and invariants are fixed here.

Configuration and profile types

• Add PipelineStepProfile with ID, Artifacts, and References fields.
• Add Steps []PipelineStepProfile to PipelineProfile; retain top-level Artifacts only as the legacy input form.
• Replace reference-map values in profile and file-config types with a discriminated ReferenceSource. It has exactly one of:
  • an external path, represented by the existing scalar YAML form; or
  • ArtifactReference{Step, Lane}, represented by artifact: {step: ..., lane: ...}.
• Keep CLI --reference values external-path overrides. They do not create or replace generated bindings. --reference-unbind may remove an effective external binding, but must not silently remove a generated dependency.
• Step-level References use the same source type as pipeline and target-local references. Pipeline-level generated references are invalid because there is no well-defined consumer step and they could imply a forward dependency.

Resolved and prepared types

• Make ResolvedPipeline.Steps []ResolvedPipelineStep the sole lane container. Each resolved step has a stable ID, ordered ArtifactLanes, and expanded generated consumer bindings. Remove the flat ArtifactLanes field after all callers migrate; provide read-only iteration/lookup helpers where callers need all lanes.
• Resolve legacy top-level artifacts as one step named default. Do not add a synthetic step back into effective user configuration output.
• Keep lane IDs globally unique. A producer and consumer are identified by pipeline ID, step ID, and lane ID; module targets additionally include stage, module key, and slot name.
• Represent external and generated bindings as distinct resolved variants. Never encode a producer selector into a path string.
• Mirror resolved steps in PreparedPipeline. A prepared step contains its already-constructed lanes and static external reference sets. Generated content is held only in run-local state and overlaid onto request references before a consumer step begins.

Generated compatibility and handoff

• Extend contracts.ReferenceSlot with an optional cloned list of accepted generated ArtifactKind values. An empty list means external references only. Do not infer compatibility by decoding external bytes.
• A generated producer supplies exactly one accepted normalized artifact per binding in this feature. Zero artifacts is a missing dependency; more than one is a cardinality error. A collection such as an NPC list is one typed artifact, not multiple reference items. Supporting aggregation or several generated items in one slot remains out of scope.
• Use the producer kind's registered ArtifactCodecSpec as the canonical schema identity and media contract. Kind compatibility plus the single registered codec makes schema compatibility exact; also validate the consumer's media, size, and cardinality constraints.
• Extend contracts.ReferenceItem with optional generated-artifact identity and producer provenance rather than overloading file origin fields. Include kind, schema ID/name/version/digest, producer pipeline/step/lane/module, content digest, media type, and size. Clone byte slices and nested metadata at every ownership boundary.
• The runtime dependency fingerprint is the canonical tuple of producer identity, artifact kind, complete schema identity, media type, and content digest. It must be added to every extract, merge, normalize, and validator checkpoint dependency that receives the generated slot.

Stage 1: Configuration, Contracts, And Resolution

Objective

Accept and fully validate ordered steps and structured artifact reference sources while preserving legacy single-step configuration. No runner behavior changes belong in this stage.

Changes

1. Add an ADR under docs/adr/ following the existing numbering and template. Record the bounded ordered-step extension, the single pipeline-wide input/chunk/output boundary, explicit generated bindings, barriers between steps, and the rejection of a general DAG.
2. In internal/core/config/file_config.go and its validation/application helpers:
   • add file forms for ordered steps and discriminated reference sources;
   • use strict YAML decoding for the mapping form and reject unknown fields, empty selectors, ambiguous scalar-plus-artifact values, and non-string scalar paths;
   • reject pipelines containing both artifacts and steps, empty explicit steps, duplicate trimmed step IDs, and duplicate lane IDs across steps;
   • preserve list order for steps and validator chains and deterministic key ordering for lane maps; and
   • update cloning, defaults application, effective-config rendering, and redaction so selectors remain visible but external paths follow the current path redaction policy.
3. In internal/framework/contracts/contracts.go, add generated artifact-kind compatibility to ReferenceSlot and update CloneReferenceSlots to deep copy it. Keep existing external-slot behavior unchanged.
4. In internal/framework/pipeline/profile.go and reference-resolution helpers:
   • introduce the canonical step and reference-source types described above;
   • normalize legacy artifacts to resolved step default;
   • resolve all step lanes and module/validator contracts before resolving generated bindings, so producer codec metadata and consumer slots are available for compatibility checks;
   • expand a step-scoped binding to every selected target in that step that declares the slot, for both external and generated sources; do not treat targets without that slot as errors;
   • retain current pipeline, target-local, and CLI precedence for external bindings, inserting step-local external bindings between pipeline defaults and target-local bindings;
   • reject every generated/external collision on an effective target slot and every step-scoped/target-local generated collision;
   • reject missing producers, same-step or forward producers, undeclared target-local slots, non-normalized producer lanes, incompatible kinds, schemas or media types, and selectors made ambiguous by duplicate IDs; and
   • include step IDs/order, lane membership, producer selectors, expanded consumers, and module/validator policy in canonical cloning and the pipeline digest.
5. Migrate catalog, config validation, --only lane selection, checkpoint identity construction, reference provenance discovery, and debug/effective configuration code to iterate the resolved step model. Reject --only when the source profile has explicit steps; retain its current behavior for the implicit default step.

Tests

• Extend internal/core/config/*_test.go with table-driven contracts for legacy shorthand, explicit step order, strict source-form parsing, mutual exclusion, identity collisions, cloning, effective output, and redaction.
• Extend internal/framework/pipeline/profile_test.go, typed_resolution_test.go, and reference tests for deterministic expansion, precedence/conflicts, backward/forward rules, global lane uniqueness, generated-kind/media compatibility, canonical cloning, and digest changes.
• Add a contract test showing a legacy profile resolves to default with an otherwise equivalent lane lifecycle and ordering.
• Prefer one representative case per validation owner; do not duplicate every parser failure at the resolver layer.

Completion Gate

Run focused config, contracts, and pipeline resolution tests, then go test ./internal/core/config ./internal/framework/contracts ./internal/framework/pipeline. The runner may still reject or lack execution support for more than one resolved step, but all configurations and topology must resolve without a flat parallel lane model.

Stage 2: Step-Aware Preparation And Execution

Objective

Execute independent ordered steps with hard barriers, even when no generated references are configured. Preserve the existing typed lane lifecycle and one bounded run-wide concurrency budget.

Changes

1. Refactor internal/framework/pipeline/prepare.go so Prepare walks resolved steps in order and constructs every lane module and validator before it returns. A failure in any later step must occur before source parsing. Materialize only external reference sources during this phase; generated selectors carry no bytes yet.
2. Refactor runner.go and runner_concurrent.go into a small ordered-step coordinator plus the existing bounded lane engine:
   • parse input and compute/reuse the chunk plan once;
   • invoke the lane engine once per step, passing only that step's prepared lanes and immutable request-reference view;
   • wait for every lane in the step to become terminal before advancing;
   • reuse the same worker/provider limits without permitting tasks from adjacent steps to overlap; and
   • encode output once after all steps succeed.
3. Accumulate accepted, rejected, debug, and checkpoint outcomes across steps. Sort public results by step index, lane ID, source ID, and chunk index/ref as applicable. Add step identity to internal errors and debug events where lane identity alone no longer explains execution context.
4. Preserve current fail-fast cancellation and bounded drain behavior. A framework error in a step cancels its started work, prevents all later steps, and prevents output encoding while retaining completed upstream outcomes.
5. Until Stage 3 lands, reject execution of any pipeline with a generated binding before source parsing. This temporary guard prevents an accepted configuration from running a consumer without its declared dependency.

Tests

• Add runner tests with deterministic fake modules proving strict barriers, concurrency within one step, reuse of the same global budget, stable output and failure order despite inverted completion timing, and no output encoding after failure.
• Add a preparation test proving every module and validator in every step is constructed before the input adapter is invoked.
• Retain existing single-step concurrency and cancellation tests unchanged where possible; migrate fixtures to the implicit default resolved step.

Completion Gate

Run go test ./internal/framework/pipeline ./internal/cli plus the repository's race-test target for the framework if one is defined in docs/development.md. Both legacy pipelines and explicit pipelines without generated references must run successfully.

Stage 3: Canonical Generated-Artifact Handoff And Provenance

Objective

Make accepted normalized output from an earlier lane available as an immutable operation-time reference to later consumers, with safe checkpoint dependencies and bounded provenance.

Changes

1. Add a domain-neutral handoff component in internal/framework/pipeline that:
   • locates the producer's accepted normalized output after its step barrier;
   • requires exactly one output and rejects missing, rejected-only, or multiple outputs deterministically;
   • canonicalizes it through ArtifactCodecRegistry.Encode (decoding a reused serialized checkpoint through the registered codec first when necessary);
   • verifies kind, exact codec schema identity, accepted media type, maximum size, and one-item cardinality for every expanded consumer slot; and
   • creates independently cloned ReferenceItem values for fan-out targets.
2. Build all generated bindings required by a consumer step before starting any lane in that step. If any handoff fails, return one contextual dependency error and start none of the step's consumers. An accepted typed empty list is valid; absence of an accepted artifact is not.
3. Merge generated items with each target's already-materialized external ReferenceSet only after resolution has proven there is no collision. Pass the resulting cloned set through existing extraction, merge, normalize, and validation request structs. Never mutate prepared static reference sets.
4. Add the canonical generated dependency fingerprint to checkpoint loader and recorder inputs for every receiving operation immediately in this stage. This is required before generated pipelines can safely use resume; do not defer it to selective recomputation work.
5. Extend artifacts and runtime provenance:
   • add step ID to artifact-lane, normalized-output, rejected-output, and reference provenance records where needed for unambiguous context;
   • record external and generated origins distinctly;
   • for generated references record producer identities, codec schema identity, media type, digest, and size, but never content or a fabricated filesystem URI; and
   • update generic JSON output and debug summaries to serialize these bounded fields deterministically.
6. Ensure checkpoint, debug, and output clones own their byte slices and maps. A consumer or test mutation must not affect another fan-out consumer or the producer artifact.

Tests

• Add handoff tests for one producer fan-out, empty typed collections, missing output, rejected-only output, multiple outputs, type/schema/media/size mismatch, target immutability, and failure before consumer start.
• Add a resume-oriented runner test proving changed canonical producer content changes the consumer dependency fingerprint and prevents stale reuse.
• Extend artifacts/JSON/debug contract tests with field assertions for bounded generated provenance and explicit assertions that content and local paths are absent. Avoid whole-document snapshots.

Completion Gate

Run focused pipeline, artifacts, JSON output, and debug tests, followed by go test ./internal/framework/... ./internal/core/artifacts ./internal/modules/generic/output/json. A generic fake-codec pipeline must complete a two-step handoff both from fresh execution and a compatible reused producer checkpoint.

Stage 4: Dependency-Aware Checkpoints And Selective Recompute

Objective

Complete safe reuse, transitive invalidation, observable decisions, and the --recompute-step operator control.

Changes

1. Update checkpoint identity and manifests in internal/framework/checkpoint:
   • introduce workspace schema notarius.workspace.v3 and retain explicit recognition of v1/v2 as incompatible cold misses;
   • include step ID in lane-stage manifests and lookup/recording context;
   • include ordered topology in the persistent pipeline identity; and
   • preserve the Stage 3 generated fingerprint tuple exactly, without content or secrets in decision reasons.
2. When loading a normalized producer checkpoint, validate its manifest, deserialize through the registered codec, re-encode canonically, and compare schema/media/content fingerprints before exposing it to handoff. Corrupt, missing, rejected, or incompatible state is a cold miss for ordinary resume and must never reach a consumer.
3. Build a lane-level dependency index from resolved generated bindings. Use it to invalidate only transitive consumer lanes when producer identity or bytes change; unrelated lanes, including unrelated lanes in a later step, remain eligible for reuse. The step barrier still applies when reused and executed lanes coexist.
4. Add a single-value --recompute-step <step-id> flag in internal/cli/run.go. Reject repeated occurrences, unknown steps, use without checkpoint recording, use without --resume, and combination with --only. Accept the stable implicit step ID default; selecting it forces every lane in that single step. One selected step is sufficient for this scope; do not add multi-selection semantics.
5. Convert the selected step to a force-execution set containing every lane in that step plus the lane-level transitive dependency closure. Loader policy, not persistent checkpoint identity, applies the force set. Required predecessors and unrelated lanes remain reusable. If a required predecessor was not selected and has no reusable accepted artifact, fail before starting a dependent lane rather than implicitly recomputing it.
6. Replace the boolean-only checkpoint reporting model as needed with a bounded decision category: executed, reused, forced_recompute, or dependency_invalidated, plus a stable reason code and optional safe detail. Propagate it to checkpoint events, manifests, debug summaries, and CLI diagnostics. Do not make exact prose part of a test contract.

Tests

• In checkpoint loader/recorder tests, cover v1/v2 cold misses, v3 step identity, compatible producer decode, corruption, exact dependency match, content/schema changes, and bounded reason fields.
• In pipeline tests, cover transitive invalidation and reuse of unrelated work.
• Add one CLI contract table for valid recomputation and every invalid flag combination, plus one execution test proving the selected closure is forced while a predecessor and unrelated lane are reused.
• Do not assert internal loader call counts when observable decision records and outputs establish the behavior.

Completion Gate

Run go test ./internal/framework/checkpoint ./internal/framework/pipeline ./internal/cli, then checkpoint/debug state tests under the race detector as directed by docs/development.md. Manually inspect one test fixture's decision records to confirm they contain no reference bytes, secret values, or local paths.

Stage 5: D&D NPC-First Production Adoption

Objective

Adopt the platform in the production D&D composition: NPC normalization runs first, and its canonical artifact grounds spell extraction, combat-turn extraction, and combat-turn normalization at operation time.

Changes

1. Declare the normalized NPC-list artifact kind on every npcs consumer slot:
   • internal/modules/dnd/extract/spells;
   • internal/modules/dnd/extract/combatturns; and
   • internal/modules/dnd/normalize/combatturns. Keep the slot optional for external standalone use; a configured generated binding becomes required through the framework dependency.
2. Refactor internal/modules/dnd/npcs/registry to provide a package-owned, concurrency-safe operation-time resolver/cache:
   • validate and seed construction-time external references so bad static configuration still fails before source parsing;
   • resolve the effective npcs item from each operation request;
   • reuse the seeded immutable registry when its digest matches and cache a generated registry by canonical digest so concurrent chunk operations do not repeatedly decode it; and
   • return cloned or immutable views and never retain caller-owned content.
3. Change the spell extractor, combat-turn extractor, and combat-turn normalizer to obtain the NPC registry from req.References for each operation through that resolver. Do not reconstruct modules at a step boundary. Retain current behavior when the request has no NPC item.
4. Keep static prompt/schema metadata and construction-time external reference fingerprints intact. Record generated NPC identity through framework reference provenance and dependency fingerprints; do not place operation-varying generated digests into singleton module metadata.
5. Preserve prompt ordering and evidence policy. The generated NPC listing is stable contextual input before the variable transcript, but it cannot prove that a spell or combat event occurred. Source-unit citations remain the only event evidence.
6. Add a maintained explicit two-step D&D configuration under examples/ and a matching production-catalog fixture under internal/modules/integration/testdata/. Replace or supplement the two manual NPC-to-spell and NPC-to-combat configurations with this single NPC-first workflow; retain standalone examples that demonstrate external NPC references where they remain useful.
7. Add one offline integration test using scripted/fake LLM responses. Assert NPC normalization completes first, the exact canonical NPC digest reaches all three consumers, spell and combat lanes may both succeed, combat normalization uses the registry, and generated NPC context is not accepted as source evidence by itself.

Tests

• Add focused registry and module tests for static fallback, generated override, concurrent cache safety, malformed runtime content, and ownership isolation.
• Extend prompt-input tests only for semantic placement and content. Do not add exact prompt-length or message-count change detectors.
• Run the single end-to-end offline D&D handoff test rather than duplicating it separately for every consumer.

Completion Gate

Run all D&D package tests and integration tests, including the race detector for the registry cache, then parse and resolve every maintained example through the production catalog. No test may require credentials or a live LLM provider.

Stage 6: Current Documentation And Release Verification

Objective

Document the now-implemented behavior in its canonical current owners, retire obsolete manual workflow guidance, and perform repository-wide verification.

Changes

1. Update docs/config.md with version-3 steps, legacy artifacts compatibility, reference source forms, precedence/conflict rules, global identity rules, and the exact D&D two-step example.
2. Update docs/cli.md with --recompute-step, prerequisites, closure semantics, invalid combinations, and checkpoint decision categories.
3. Update docs/internal/pipeline.md, docs/internal/state.md, and docs/internal/overview.md with the fixed ordered-step model, preparation timing, handoff boundary, codec use, provenance, and checkpoint dependency behavior. Keep the architecture description linear and explicitly state that this is not a general DAG.
4. Update docs/operations.md with resume/recompute procedures and safe failure diagnosis. Update the D&D integration documents for generated NPC context, evidence limitations, and the maintained workflow. Update docs/policy/testing.md only if the implementation exposes a genuinely new durable testing policy; do not restate feature-specific tests there.
5. Update maintained example checks and any README/index links that point to replaced sequential examples. Keep the feature roadmap as historical target state until the project applies its normal roadmap-completion process; do not turn it into a duplicate current-behavior manual.
6. Review changed exported identifiers and package comments, run formatting and static analysis, and remove obsolete flat-lane compatibility helpers, temporary test adapters, dead sequential-workflow code, and TODOs introduced during earlier stages.

Verification

Run the repository-prescribed commands from docs/development.md, including:

1. all unit and integration tests;
2. race tests for concurrency-sensitive framework, checkpoint, and D&D registry packages;
3. go vet and the normal build;
4. maintained-example/config validation; and
5. documentation checks.

Perform one final acceptance review against every bullet in Ordered Pipeline Steps. Inspect a successful fresh run, a resumed run, a forced-recompute run, and a missing-producer failure for deterministic ordering, correct dependency decisions, bounded provenance, and absence of generated content or secrets in manifests/debug state.

Completion Gate

The feature is complete only when every acceptance criterion in the feature roadmap is demonstrated by a stable test or maintained example and the full repository verification suite passes.

Open Questions

None. The feature roadmap and this plan fix all product and architecture choices required for implementation. If an implementation detail conflicts with a policy document, the policy document takes precedence; if it would change the product semantics above, stop and amend the roadmap rather than deciding it in code.