notarius/docs/roadmap/implementation.md

Implementation Plan: Ordered Pipeline Follow-Up

Purpose

Close the remaining correctness, observability, test, and maintainability gaps in the implemented Ordered Pipeline Steps feature. That feature roadmap remains the authority for product intent, policy choices, acceptance criteria, and exclusions. This document is the ordered, decision-complete implementation sequence for the follow-up work.

Background

The original implementation is complete in broad architecture. Pipelines now resolve and prepare one canonical ordered-step model, execute hard barriers between steps, hand accepted normalized artifacts to later operations as canonical generated references, include generated identity in checkpoint and output provenance, support --recompute-step, and use generated NPC output at operation time in the D&D spell and combat consumers. Current documentation and maintained examples describe that model.

The remaining work is narrower:

• selective recomputation currently treats an unselected predecessor as reusable only when all of its extract, merge, and normalize stage checkpoints are reusable, although the architectural dependency is its accepted normalized artifact;
• the failed stage decision is not always recorded before a required-predecessor error returns;
• checkpoint reason codes are inferred from human-readable prose instead of being assigned explicitly;
• the runner's lane and checkpoint orchestration has become too concentrated in large functions; and
• the roadmap-required CLI and resumed-producer acceptance coverage is absent.

No new product feature is introduced by this plan. Preserve configuration file version 3 and checkpoint workspace schema notarius.workspace.v3; the fixes do not require a new persistent format.

Instructions For Every Stage

Before changing code, read docs/development.md, all documents under docs/policy/, and the task-specific internal documents named there. Use the repository's code knowledge graph for code discovery before falling back to text search.

Implement the stages in order. Each stage must leave the repository formatted, building, and passing its focused tests. Preserve these invariants throughout:

• The pipeline retains one input, chunk plan, output encoder, run identity, checkpoint identity, failure boundary, worker budget, and provider scheduler.
• Every selected module and validator is constructed before source parsing.
• Steps schedule fixed extract, merge, and normalize lanes; this work must not introduce module-to-module calls or a general DAG scheduler.
• Generated artifacts remain cloned operation-time context, never source evidence. Never emit their content, source material, credentials, or local paths in decisions, manifests, logs, or summaries.
• Public ordering remains step order, lane ID, and source/chunk order as applicable. Refactoring must not expose completion order.
• Tests must follow docs/policy/testing.md: protect observable recovery, orchestration, and CLI contracts without asserting private helper calls, exact prose, goroutine choreography, or full-document snapshots.
• Update current-behavior documentation in the same stage that changes the corresponding behavior. Keep detailed implementation sequencing only here.

Target Checkpoint Semantics

Use these definitions consistently in all stages:

• A stage checkpoint is internal resumable state for extract, merge, or normalize. Ordinary resume may continue to reuse this state progressively.
• An accepted lane artifact is the one successful normalized artifact for a (step ID, lane ID, normalizer module) under the current checkpoint identity. It is the dependency exposed to a later step.
• An unselected required predecessor satisfies selective recomputation when its accepted lane artifact can be validated and hydrated. Its extract and merge stage checkpoints are not prerequisites for that handoff.
• A selected lane and its transitive dependents execute. They must never use accepted-artifact hydration to bypass forced execution.
• If a required predecessor's accepted lane artifact is missing, rejected, corrupt, non-canonical, or incompatible, fail the run before any dependent lane starts. Do not implicitly rerun that predecessor.
• Ordinary resume without --recompute-step keeps its existing progressive stage-reuse and cold-miss behavior.

Stage 1: Decompose Runner And Checkpoint Orchestration

Objective

Create explicit, testable ownership seams for ordered-step coordination, per-lane execution, and per-stage checkpoint handling without changing observable behavior.

Changes

1. Keep Runner.Run as the pipeline-wide coordinator. Extract a small step-coordination helper responsible only for iterating prepared steps, building generated reference sets at each barrier, invoking the lane engine, and merging deterministic outcomes.
2. Split runLanes in internal/framework/pipeline/runner_concurrent.go into helpers with these responsibilities:
   • initialize lane state and resolve extract checkpoint state in lane order;
   • run the bounded extract worker/continuation engine;
   • collect terminal lane results and choose failures deterministically; and
   • merge lane-local output into step output. Preserve the existing bounded channels, cancellation, drain behavior, chunk-first dispatch, lane ordering, and step barrier.
3. Split continueTypedLane in internal/framework/pipeline/runner_typed.go into stage-specific merge and normalize helpers. Each helper should own dependency construction, checkpoint loading and canonical validation, execution/retry/validation when needed, checkpoint recording, debug envelopes, and its typed result. Use small result structs rather than long parallel return lists.
4. Centralize repeated checkpoint-decision flow in one pipeline helper that can apply forced-execution policy, validate canonical stored artifacts, record the final observable decision, and return a contextual error. Do not yet change categories, reason codes, or required-predecessor semantics; Stages 2 and 3 will change those deliberately.
5. Keep stage-specific code where the data shapes genuinely differ. Do not introduce reflection, a generic stage state machine, or callbacks that hide the fixed extract/merge/normalize lifecycle.

Tests

• Existing runner, checkpoint, barrier, ordering, cancellation, retry, debug, and D&D integration tests must pass without expectation changes except moves required by renamed private test fixtures.
• Add no tests for helper boundaries or collaborator call counts. Add a narrow regression assertion only if the refactor exposes an observable behavior not already protected.

Completion Gate

Run go test ./internal/framework/pipeline ./internal/framework/checkpoint ./internal/modules/integration and the pipeline race tests. Review the diff to confirm this stage changes structure only: serialized output, checkpoint state, decision values, failure selection, and module invocation behavior must remain unchanged.

Stage 2: Make Checkpoint Decisions Typed And Observable

Objective

Assign stable decision categories and reason codes explicitly, and retain the decision that causes a required-dependency failure.

Changes

1. In internal/framework/pipeline/checkpoint.go, introduce string-backed internal types and constants for checkpoint decision categories and reason codes. Keep the current JSON strings and public artifact fields compatible. Categories remain exactly executed, reused, forced_recompute, and dependency_invalidated.
2. Replace prose inspection in internal/framework/checkpoint/loader.go with an explicit decision constructor accepting category, reason code, and optional detail. Remove every strings.Contains classification branch. Assign a code at the validation site using this bounded vocabulary:
   • loading_disabled, checkpoint_missing, checkpoint_path_invalid, checkpoint_read_failed, and checkpoint_decode_failed;
   • workspace_schema_incompatible and identity_mismatch;
   • stage_mismatch, step_mismatch, lane_mismatch, module_mismatch, and status_not_reusable;
   • dependency_mismatch for the dependency_invalidated category;
   • artifact_payload_invalid, artifact_digest_mismatch, artifact_codec_incompatible, and artifact_not_canonical; and
   • checkpoint_reused and accepted_artifact_reused for successful reuse, and recompute_step for forced execution. Retain an existing code not listed here only when a current external or documented contract already depends on it.
3. Keep detail human-readable, UTF-8, bounded, and sanitized through one helper. Detail may identify stage, step, lane, module, expected status, or schema version, but must not include artifact/reference content, source content, credentials, environment values, or local paths. Tests must assert codes and safety properties, not exact detail prose.
4. Change the centralized runner decision flow from Stage 1 so the final loader or canonical-validation decision is recorded before returning a required-predecessor error. The contextual error must identify the step and lane and include the stable reason code; it must not interpolate unsafe loader detail.
5. Propagate typed values without lossy conversion through checkpoint events, run manifests, debug summaries, and CLI diagnostics. Convert to strings only at existing serialized boundaries unless changing an internal field type is simpler and wire-compatible.
6. Update the canonical current-behavior owners in the same change: docs/internal/state.md owns the internal decision flow, while docs/operations.md owns operator diagnosis and any operator-visible reason code table. Link rather than duplicate the table elsewhere.

Tests

• Add a table in internal/framework/checkpoint that exercises one representative input per reason-code family and asserts category, code, bounded valid UTF-8 detail, and absence of supplied secret/path sentinels.
• Add pipeline behavior tests proving a required-predecessor failure records the underlying missing, corrupt, incompatible, or dependency-invalidated decision before returning.
• Retain focused manifest/debug serialization assertions for category and code; do not add exact-detail or full-manifest snapshots.

Completion Gate

Run go test ./internal/framework/checkpoint ./internal/framework/pipeline ./internal/core/artifacts ./internal/core/debugbundle ./internal/cli. Search the production checkpoint package to confirm no decision category or reason code is derived from diagnostic prose.

Stage 3: Hydrate Required Predecessors From Accepted Normalized Artifacts

Objective

Make selective recomputation enforce the lane-level accepted-artifact contract: a valid normalized producer artifact is sufficient even when its extract or merge stage cache is unavailable.

Changes

1. Extend the checkpoint loader contract with a dedicated accepted-normalized- artifact lookup. Implement it in the filesystem loader and every no-op or test implementation. The lookup receives step ID, lane ID, and normalizer module key and returns the normalized artifact, its warnings, and an explicit checkpoint decision. It must not require caller-supplied extract or merge dependency fingerprints.
2. The filesystem lookup may reuse the existing normalize manifest and payload; do not add a second persistent copy. It is reusable only when all of the following hold:
   • workspace schema is v3 and the non-empty stored checkpoint identity matches the current invocation identity;
   • stage, step, lane, normalizer module, and successful status match;
   • the manifest output digest matches the payload; and
   • the payload can subsequently be validated through the registered artifact codec. Skipping caller-supplied merge dependencies is safe only because the matched non-empty checkpoint identity already binds the current input, resolved topology and configuration, external references, runtime overrides, LLM profiles, and component semantic fingerprints. Do not weaken or omit that identity check. A loader lacking a verifiable current identity must return an unavailable decision rather than perform accepted-artifact reuse.
3. Add a pipeline hydration helper that decodes the returned serialized artifact through the producer's registered codec, re-encodes it canonically, and requires exact artifact kind, schema ID/name/version/digest, media type, content bytes, and content digest. Return a runner-owned clone with producer step, lane, module, and source identity. Any mismatch is an explicit bounded decision and the bytes never reach a consumer.
4. Before normal execution of a lane marked in RequireReusableLanes, use the accepted-artifact lookup:
   • on success, mark the lane terminal without invoking extract, merge, normalize, or their validators;
   • append the accepted normalized output in the normal deterministic location and restore only the warnings stored with that normalized checkpoint;
   • record one reused normalize decision with a stable accepted-output reuse reason code; do not synthesize extract/merge decisions, warnings, or rejections that were not loaded; and
   • allow the ordinary step barrier and generated handoff code to consume that output exactly as it consumes a freshly executed output.
5. On missing, rejected, corrupt, non-canonical, or incompatible accepted state, record the decision and fail the run before the dependent step begins. Do not fall back to stage execution. Forced lanes must bypass this hydration path and execute normally.
6. Leave ordinary resume unchanged when no lane is marked RequireReusableLanes: it may reuse or recompute extract, merge, and normalize progressively under the existing cold-miss rules.
7. Keep generated-reference fingerprints and provenance unchanged. Hydrating a byte-identical producer must yield the same canonical handoff digest and downstream dependency fingerprint as fresh execution.
8. Update docs/internal/pipeline.md, docs/internal/state.md, and docs/operations.md in this stage to distinguish progressive stage reuse from accepted normalized-artifact hydration and to document the fail-rather- than-rerun rule for invalid required predecessors.

Tests

• At the pipeline/checkpoint boundary, create a valid producer normalize checkpoint while omitting or corrupting its extract and merge checkpoints. Select a later step for recomputation and prove the producer hydrates, no producer operation or validator runs, and the dependent receives the exact canonical artifact.
• Cover missing, rejected-status, corrupt, non-canonical, wrong-codec-identity, and wrong-content-digest normalize state. Assert failure and the recorded stable decision before any consumer invocation.
• Prove forced producers execute instead of hydrating, while a reusable predecessor and an unrelated lane remain reusable.
• Prove fresh and hydrated producer outputs create identical generated provenance and downstream checkpoint fingerprints without exposing content or paths.

Completion Gate

Run go test ./internal/framework/checkpoint ./internal/framework/pipeline ./internal/modules/integration and the corresponding race tests. Manually inspect one failed test fixture to confirm the accepted artifact remains on disk but its bytes do not appear in the manifest, decision detail, debug summary, or error.

Stage 4: Complete Recompute CLI And Recovery Acceptance Coverage

Objective

Exercise the operator-visible recomputation contract through stable boundaries and close the acceptance-test omissions from the original plan.

Changes

1. Add one table-driven CLI contract test for --recompute-step covering:
   • a valid explicit step and the implicit default step;
   • repeated flags and an empty or unknown step ID;
   • use without --resume;
   • use when checkpoint recording is disabled; and
   • combination with --only. Assert exit classification and stable identifying fragments or reason codes, not complete prose.
2. Add one filesystem-backed CLI execution test using deterministic fake modules and a three-step generated dependency chain plus one unrelated lane:
   • perform a fresh checkpointed run;
   • remove or corrupt only the required producer's extract and merge state inside t.TempDir(), leaving its normalize artifact valid;
   • resume with the middle step selected;
   • assert the selected lane and transitive dependents execute, the predecessor hydrates without module calls, the unrelated lane reuses, and output and decision ordering are deterministic; and
   • invalidate the producer normalize state in a subcase and assert the command fails before dependent execution with the bounded decision preserved.
3. Add or extend one generic two-step pipeline test proving handoff succeeds both from fresh producer execution and from accepted normalized-artifact hydration. Keep this at the pipeline boundary if the CLI execution test already proves flag wiring; do not duplicate every CLI case end to end.
4. Review existing recomputation policy tests. Retain the pure closure test because it protects transitive selection, but remove or consolidate any new test that merely repeats the CLI or pipeline behavior above.
5. Make only production changes revealed as necessary by these contract tests; do not add new flag semantics or broaden the feature roadmap.

Completion Gate

Run go test ./internal/cli ./internal/framework/pipeline ./internal/framework/checkpoint, then run the CLI, pipeline, and checkpoint packages under the race detector. Confirm no test asserts internal loader call counts, exact decision detail, filesystem layout outside a temp workspace, or the exact length of any prompt or prefix.

Stage 5: Current Documentation And Release Verification

Objective

Reconcile all canonical documentation after the staged changes and verify the complete feature against policy and roadmap.

Changes

1. Re-read docs/internal/pipeline.md, docs/internal/state.md, docs/operations.md, and docs/cli.md against the final code. Correct any stale statements left by Stages 1 through 4 without duplicating their canonical contracts.
2. Ensure internal documentation describes the coordinator, lane engine, stage checkpoint flow, and accepted-artifact validation at the responsibility level without listing volatile private helper names.
3. Confirm the operator-visible reason-code table has one canonical owner and other documents link to it rather than maintaining parallel copies.
4. Remove stale implementation claims exposed by this work and ensure the feature roadmap continues to describe target policy rather than task sequencing.

Verification

Run the repository-prescribed commands from docs/development.md:

go test ./...
go vet ./...
go build ./cmd/notarius

Also run race tests for the pipeline, checkpoint, CLI state, D&D NPC registry, and D&D integration packages. Validate maintained examples/configurations through the production catalog. Review one fresh run, ordinary resumed run, forced-recompute run, hydrated-predecessor run, and invalid-predecessor failure for deterministic ordering, correct decisions, bounded provenance, and absence of generated content, secrets, or local paths in manifests and diagnostics.

Completion Gate

The follow-up is complete only when every finding summarized above is protected by a stable behavioral test, the current documentation matches the corrected implementation, the full verification suite passes, and the worktree contains no temporary adapters or TODOs introduced by these stages.

Open Questions

None. The feature roadmap and this plan fix the required product and architecture choices. If implementation reveals that accepted normalized artifacts cannot be validated safely without a persistent format change, stop and amend this plan rather than weakening identity, codec, or content validation.