diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index e2f1c9c..34c35c2 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,500 +1,392 @@ -# Implementation Plan: Ordered Pipeline Steps +# Implementation Plan: Ordered Pipeline Follow-Up ## Purpose -Implement the target state in [Ordered Pipeline Steps](ordered-pipeline-steps.md). -That feature roadmap is the authority for product intent, policy choices, -acceptance criteria, and exclusions. This document turns it into an ordered, -decision-complete implementation sequence for a coding agent. +Close the remaining correctness, observability, test, and maintainability gaps +in the implemented [Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. +That feature roadmap remains the authority for product intent, policy choices, +acceptance criteria, and exclusions. This document is the ordered, +decision-complete implementation sequence for the follow-up work. -This is a platform refactor followed by one D&D proving workflow. It does not -include item extraction, a general DAG scheduler, schema migrations for D&D -artifacts, or any other work excluded by the feature roadmap. +## Background + +The original implementation is complete in broad architecture. Pipelines now +resolve and prepare one canonical ordered-step model, execute hard barriers +between steps, hand accepted normalized artifacts to later operations as +canonical generated references, include generated identity in checkpoint and +output provenance, support `--recompute-step`, and use generated NPC output at +operation time in the D&D spell and combat consumers. Current documentation and +maintained examples describe that model. + +The remaining work is narrower: + +- selective recomputation currently treats an unselected predecessor as + reusable only when all of its extract, merge, and normalize stage checkpoints + are reusable, although the architectural dependency is its accepted + normalized artifact; +- the failed stage decision is not always recorded before a required-predecessor + error returns; +- checkpoint reason codes are inferred from human-readable prose instead of + being assigned explicitly; +- the runner's lane and checkpoint orchestration has become too concentrated in + large functions; and +- the roadmap-required CLI and resumed-producer acceptance coverage is absent. + +No new product feature is introduced by this plan. Preserve configuration file +version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do +not require a new persistent format. ## Instructions For Every Stage -Before changing code, read `docs/development.md` and all documents under -`docs/policy/`. Preserve the fixed pipeline lifecycle and the two-zone data -model described there. Use the repository's code knowledge graph for code -discovery before falling back to text search. +Before changing code, read `docs/development.md`, all documents under +`docs/policy/`, and the task-specific internal documents named there. Use the +repository's code knowledge graph for code discovery before falling back to +text search. Implement the stages in order. Each stage must leave the repository formatted, -building, and passing the focused tests it changes. Do not maintain two -competing internal representations merely to reduce a refactor: legacy -top-level `artifacts` is a configuration compatibility form, while ordered -steps are the single resolved, prepared, and runtime representation. +building, and passing its focused tests. Preserve these invariants throughout: -Follow these cross-stage rules: +- The pipeline retains one input, chunk plan, output encoder, run identity, + checkpoint identity, failure boundary, worker budget, and provider scheduler. +- Every selected module and validator is constructed before source parsing. +- Steps schedule fixed extract, merge, and normalize lanes; this work must not + introduce module-to-module calls or a general DAG scheduler. +- Generated artifacts remain cloned operation-time context, never source + evidence. Never emit their content, source material, credentials, or local + paths in decisions, manifests, logs, or summaries. +- Public ordering remains step order, lane ID, and source/chunk order as + applicable. Refactoring must not expose completion order. +- Tests must follow `docs/policy/testing.md`: protect observable recovery, + orchestration, and CLI contracts without asserting private helper calls, + exact prose, goroutine choreography, or full-document snapshots. +- Update current-behavior documentation in the same stage that changes the + corresponding behavior. Keep detailed implementation sequencing only here. -- Keep input parsing, chunk planning, output encoding, run identity, worker - budgets, provider scheduling, and the failure boundary pipeline-wide. -- Preserve the lane lifecycle and existing typed module interfaces. Steps - schedule lanes; they do not permit modules to call other modules. -- Construct every selected module and validator before source parsing. Supply - generated bytes only through cloned operation-request `References`. -- Resolve and report deterministic order as step order, then lane ID, then - source/chunk order. Never expose completion order. -- Treat generated references as context, not source evidence. Never put their - content in manifests, checkpoint decisions, logs, or debug summaries. -- Add narrow, behavior-oriented, offline tests at the stable owner of each new - invariant. Do not add scheduler choreography, exact goroutine counts, full - manifest snapshots, exact error-string snapshots, or live-provider tests. -- Keep configuration file version 3. The new fields are a backward-compatible - extension; checkpoint wire state receives its own explicit compatibility - change in Stage 4. -- Update current-behavior documentation only in Stage 6, when the complete - behavior exists. The ADR may be added in Stage 1 because it records the - architecture decision rather than claiming released behavior. +## Target Checkpoint Semantics -## Canonical Internal Model +Use these definitions consistently in all stages: -Use the following model throughout the implementation. Exact unexported helper -names may follow local conventions, but their responsibilities and invariants -are fixed here. +- A **stage checkpoint** is internal resumable state for extract, merge, or + normalize. Ordinary resume may continue to reuse this state progressively. +- An **accepted lane artifact** is the one successful normalized artifact for a + `(step ID, lane ID, normalizer module)` under the current checkpoint identity. + It is the dependency exposed to a later step. +- An unselected required predecessor satisfies selective recomputation when its + accepted lane artifact can be validated and hydrated. Its extract and merge + stage checkpoints are not prerequisites for that handoff. +- A selected lane and its transitive dependents execute. They must never use + accepted-artifact hydration to bypass forced execution. +- If a required predecessor's accepted lane artifact is missing, rejected, + corrupt, non-canonical, or incompatible, fail the run before any dependent + lane starts. Do not implicitly rerun that predecessor. +- Ordinary resume without `--recompute-step` keeps its existing progressive + stage-reuse and cold-miss behavior. -### Configuration and profile types - -- Add `PipelineStepProfile` with `ID`, `Artifacts`, and `References` fields. -- Add `Steps []PipelineStepProfile` to `PipelineProfile`; retain top-level - `Artifacts` only as the legacy input form. -- Replace reference-map values in profile and file-config types with a - discriminated `ReferenceSource`. It has exactly one of: - - an external path, represented by the existing scalar YAML form; or - - `ArtifactReference{Step, Lane}`, represented by - `artifact: {step: ..., lane: ...}`. -- Keep CLI `--reference` values external-path overrides. They do not create or - replace generated bindings. `--reference-unbind` may remove an effective - external binding, but must not silently remove a generated dependency. -- Step-level `References` use the same source type as pipeline and target-local - references. Pipeline-level generated references are invalid because there is - no well-defined consumer step and they could imply a forward dependency. - -### Resolved and prepared types - -- Make `ResolvedPipeline.Steps []ResolvedPipelineStep` the sole lane container. - Each resolved step has a stable `ID`, ordered `ArtifactLanes`, and expanded - generated consumer bindings. Remove the flat `ArtifactLanes` field after all - callers migrate; provide read-only iteration/lookup helpers where callers - need all lanes. -- Resolve legacy top-level `artifacts` as one step named `default`. Do not add a - synthetic step back into effective user configuration output. -- Keep lane IDs globally unique. A producer and consumer are identified by - pipeline ID, step ID, and lane ID; module targets additionally include stage, - module key, and slot name. -- Represent external and generated bindings as distinct resolved variants. - Never encode a producer selector into a path string. -- Mirror resolved steps in `PreparedPipeline`. A prepared step contains its - already-constructed lanes and static external reference sets. Generated - content is held only in run-local state and overlaid onto request references - before a consumer step begins. - -### Generated compatibility and handoff - -- Extend `contracts.ReferenceSlot` with an optional cloned list of accepted - generated `ArtifactKind` values. An empty list means external references only. - Do not infer compatibility by decoding external bytes. -- A generated producer supplies exactly one accepted normalized artifact per - binding in this feature. Zero artifacts is a missing dependency; more than - one is a cardinality error. A collection such as an NPC list is one typed - artifact, not multiple reference items. Supporting aggregation or several - generated items in one slot remains out of scope. -- Use the producer kind's registered `ArtifactCodecSpec` as the canonical schema - identity and media contract. Kind compatibility plus the single registered - codec makes schema compatibility exact; also validate the consumer's media, - size, and cardinality constraints. -- Extend `contracts.ReferenceItem` with optional generated-artifact identity and - producer provenance rather than overloading file origin fields. Include kind, - schema ID/name/version/digest, producer pipeline/step/lane/module, content - digest, media type, and size. Clone byte slices and nested metadata at every - ownership boundary. -- The runtime dependency fingerprint is the canonical tuple of producer - identity, artifact kind, complete schema identity, media type, and content - digest. It must be added to every extract, merge, normalize, and validator - checkpoint dependency that receives the generated slot. - -## Stage 1: Configuration, Contracts, And Resolution +## Stage 1: Decompose Runner And Checkpoint Orchestration ### Objective -Accept and fully validate ordered steps and structured artifact reference -sources while preserving legacy single-step configuration. No runner behavior -changes belong in this stage. +Create explicit, testable ownership seams for ordered-step coordination, +per-lane execution, and per-stage checkpoint handling without changing +observable behavior. ### Changes -1. Add an ADR under `docs/adr/` following the existing numbering and template. - Record the bounded ordered-step extension, the single pipeline-wide - input/chunk/output boundary, explicit generated bindings, barriers between - steps, and the rejection of a general DAG. -2. In `internal/core/config/file_config.go` and its validation/application - helpers: - - add file forms for ordered steps and discriminated reference sources; - - use strict YAML decoding for the mapping form and reject unknown fields, - empty selectors, ambiguous scalar-plus-artifact values, and non-string - scalar paths; - - reject pipelines containing both `artifacts` and `steps`, empty explicit - steps, duplicate trimmed step IDs, and duplicate lane IDs across steps; - - preserve list order for steps and validator chains and deterministic key - ordering for lane maps; and - - update cloning, defaults application, effective-config rendering, and - redaction so selectors remain visible but external paths follow the current - path redaction policy. -3. In `internal/framework/contracts/contracts.go`, add generated artifact-kind - compatibility to `ReferenceSlot` and update `CloneReferenceSlots` to deep - copy it. Keep existing external-slot behavior unchanged. -4. In `internal/framework/pipeline/profile.go` and reference-resolution helpers: - - introduce the canonical step and reference-source types described above; - - normalize legacy `artifacts` to resolved step `default`; - - resolve all step lanes and module/validator contracts before resolving - generated bindings, so producer codec metadata and consumer slots are - available for compatibility checks; - - expand a step-scoped binding to every selected target in that step that - declares the slot, for both external and generated sources; do not treat - targets without that slot as errors; - - retain current pipeline, target-local, and CLI precedence for external - bindings, inserting step-local external bindings between pipeline defaults - and target-local bindings; - - reject every generated/external collision on an effective target slot and - every step-scoped/target-local generated collision; - - reject missing producers, same-step or forward producers, undeclared - target-local slots, non-normalized producer lanes, incompatible kinds, - schemas or media types, and selectors made ambiguous by duplicate IDs; and - - include step IDs/order, lane membership, producer selectors, expanded - consumers, and module/validator policy in canonical cloning and the - pipeline digest. -5. Migrate catalog, config validation, `--only` lane selection, checkpoint - identity construction, reference provenance discovery, and debug/effective - configuration code to iterate the resolved step model. Reject `--only` when - the source profile has explicit steps; retain its current behavior for the - implicit `default` step. +1. Keep `Runner.Run` as the pipeline-wide coordinator. Extract a small + step-coordination helper responsible only for iterating prepared steps, + building generated reference sets at each barrier, invoking the lane engine, + and merging deterministic outcomes. +2. Split `runLanes` in `internal/framework/pipeline/runner_concurrent.go` into + helpers with these responsibilities: + - initialize lane state and resolve extract checkpoint state in lane order; + - run the bounded extract worker/continuation engine; + - collect terminal lane results and choose failures deterministically; and + - merge lane-local output into step output. + Preserve the existing bounded channels, cancellation, drain behavior, + chunk-first dispatch, lane ordering, and step barrier. +3. Split `continueTypedLane` in + `internal/framework/pipeline/runner_typed.go` into stage-specific merge and + normalize helpers. Each helper should own dependency construction, checkpoint + loading and canonical validation, execution/retry/validation when needed, + checkpoint recording, debug envelopes, and its typed result. Use small + result structs rather than long parallel return lists. +4. Centralize repeated checkpoint-decision flow in one pipeline helper that can + apply forced-execution policy, validate canonical stored artifacts, record + the final observable decision, and return a contextual error. Do not yet + change categories, reason codes, or required-predecessor semantics; Stages 2 + and 3 will change those deliberately. +5. Keep stage-specific code where the data shapes genuinely differ. Do not + introduce reflection, a generic stage state machine, or callbacks that hide + the fixed extract/merge/normalize lifecycle. ### Tests -- Extend `internal/core/config/*_test.go` with table-driven contracts for legacy - shorthand, explicit step order, strict source-form parsing, mutual exclusion, - identity collisions, cloning, effective output, and redaction. -- Extend `internal/framework/pipeline/profile_test.go`, - `typed_resolution_test.go`, and reference tests for deterministic expansion, - precedence/conflicts, backward/forward rules, global lane uniqueness, - generated-kind/media compatibility, canonical cloning, and digest changes. -- Add a contract test showing a legacy profile resolves to `default` with an - otherwise equivalent lane lifecycle and ordering. -- Prefer one representative case per validation owner; do not duplicate every - parser failure at the resolver layer. +- Existing runner, checkpoint, barrier, ordering, cancellation, retry, debug, + and D&D integration tests must pass without expectation changes except moves + required by renamed private test fixtures. +- Add no tests for helper boundaries or collaborator call counts. Add a narrow + regression assertion only if the refactor exposes an observable behavior not + already protected. ### Completion Gate -Run focused config, contracts, and pipeline resolution tests, then `go test -./internal/core/config ./internal/framework/contracts ./internal/framework/pipeline`. -The runner may still reject or lack execution support for more than one -resolved step, but all configurations and topology must resolve without a flat -parallel lane model. +Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint +./internal/modules/integration` and the pipeline race tests. Review the diff to +confirm this stage changes structure only: serialized output, checkpoint state, +decision values, failure selection, and module invocation behavior must remain +unchanged. -## Stage 2: Step-Aware Preparation And Execution +## Stage 2: Make Checkpoint Decisions Typed And Observable ### Objective -Execute independent ordered steps with hard barriers, even when no generated -references are configured. Preserve the existing typed lane lifecycle and one -bounded run-wide concurrency budget. +Assign stable decision categories and reason codes explicitly, and retain the +decision that causes a required-dependency failure. ### Changes -1. Refactor `internal/framework/pipeline/prepare.go` so `Prepare` walks resolved - steps in order and constructs every lane module and validator before it - returns. A failure in any later step must occur before source parsing. - Materialize only external reference sources during this phase; generated - selectors carry no bytes yet. -2. Refactor `runner.go` and `runner_concurrent.go` into a small ordered-step - coordinator plus the existing bounded lane engine: - - parse input and compute/reuse the chunk plan once; - - invoke the lane engine once per step, passing only that step's prepared - lanes and immutable request-reference view; - - wait for every lane in the step to become terminal before advancing; - - reuse the same worker/provider limits without permitting tasks from - adjacent steps to overlap; and - - encode output once after all steps succeed. -3. Accumulate accepted, rejected, debug, and checkpoint outcomes across steps. - Sort public results by step index, lane ID, source ID, and chunk index/ref as - applicable. Add step identity to internal errors and debug events where lane - identity alone no longer explains execution context. -4. Preserve current fail-fast cancellation and bounded drain behavior. A - framework error in a step cancels its started work, prevents all later steps, - and prevents output encoding while retaining completed upstream outcomes. -5. Until Stage 3 lands, reject execution of any pipeline with a generated - binding before source parsing. This temporary guard prevents an accepted - configuration from running a consumer without its declared dependency. +1. In `internal/framework/pipeline/checkpoint.go`, introduce string-backed + internal types and constants for checkpoint decision categories and reason + codes. Keep the current JSON strings and public artifact fields compatible. + Categories remain exactly `executed`, `reused`, `forced_recompute`, and + `dependency_invalidated`. +2. Replace prose inspection in `internal/framework/checkpoint/loader.go` with an + explicit decision constructor accepting category, reason code, and optional + detail. Remove every `strings.Contains` classification branch. Assign a code + at the validation site using this bounded vocabulary: + - `loading_disabled`, `checkpoint_missing`, `checkpoint_path_invalid`, + `checkpoint_read_failed`, and `checkpoint_decode_failed`; + - `workspace_schema_incompatible` and `identity_mismatch`; + - `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch`, and + `status_not_reusable`; + - `dependency_mismatch` for the `dependency_invalidated` category; + - `artifact_payload_invalid`, `artifact_digest_mismatch`, + `artifact_codec_incompatible`, and `artifact_not_canonical`; and + - `checkpoint_reused` and `accepted_artifact_reused` for successful reuse, + and `recompute_step` for forced execution. + Retain an existing code not listed here only when a current external or + documented contract already depends on it. +3. Keep detail human-readable, UTF-8, bounded, and sanitized through one helper. + Detail may identify stage, step, lane, module, expected status, or schema + version, but must not include artifact/reference content, source content, + credentials, environment values, or local paths. Tests must assert codes and + safety properties, not exact detail prose. +4. Change the centralized runner decision flow from Stage 1 so the final loader + or canonical-validation decision is recorded before returning a + required-predecessor error. The contextual error must identify the step and + lane and include the stable reason code; it must not interpolate unsafe + loader detail. +5. Propagate typed values without lossy conversion through checkpoint events, + run manifests, debug summaries, and CLI diagnostics. Convert to strings only + at existing serialized boundaries unless changing an internal field type is + simpler and wire-compatible. +6. Update the canonical current-behavior owners in the same change: + `docs/internal/state.md` owns the internal decision flow, while + `docs/operations.md` owns operator diagnosis and any operator-visible reason + code table. Link rather than duplicate the table elsewhere. ### Tests -- Add runner tests with deterministic fake modules proving strict barriers, - concurrency within one step, reuse of the same global budget, stable output - and failure order despite inverted completion timing, and no output encoding - after failure. -- Add a preparation test proving every module and validator in every step is - constructed before the input adapter is invoked. -- Retain existing single-step concurrency and cancellation tests unchanged - where possible; migrate fixtures to the implicit `default` resolved step. - -### Completion Gate - -Run `go test ./internal/framework/pipeline ./internal/cli` plus the repository's -race-test target for the framework if one is defined in `docs/development.md`. -Both legacy pipelines and explicit pipelines without generated references must -run successfully. - -## Stage 3: Canonical Generated-Artifact Handoff And Provenance - -### Objective - -Make accepted normalized output from an earlier lane available as an immutable -operation-time reference to later consumers, with safe checkpoint dependencies -and bounded provenance. - -### Changes - -1. Add a domain-neutral handoff component in `internal/framework/pipeline` that: - - locates the producer's accepted normalized output after its step barrier; - - requires exactly one output and rejects missing, rejected-only, or multiple - outputs deterministically; - - canonicalizes it through `ArtifactCodecRegistry.Encode` (decoding a reused - serialized checkpoint through the registered codec first when necessary); - - verifies kind, exact codec schema identity, accepted media type, maximum - size, and one-item cardinality for every expanded consumer slot; and - - creates independently cloned `ReferenceItem` values for fan-out targets. -2. Build all generated bindings required by a consumer step before starting any - lane in that step. If any handoff fails, return one contextual dependency - error and start none of the step's consumers. An accepted typed empty list is - valid; absence of an accepted artifact is not. -3. Merge generated items with each target's already-materialized external - `ReferenceSet` only after resolution has proven there is no collision. Pass - the resulting cloned set through existing extraction, merge, normalize, and - validation request structs. Never mutate prepared static reference sets. -4. Add the canonical generated dependency fingerprint to checkpoint loader and - recorder inputs for every receiving operation immediately in this stage. - This is required before generated pipelines can safely use resume; do not - defer it to selective recomputation work. -5. Extend artifacts and runtime provenance: - - add step ID to artifact-lane, normalized-output, rejected-output, and - reference provenance records where needed for unambiguous context; - - record external and generated origins distinctly; - - for generated references record producer identities, codec schema - identity, media type, digest, and size, but never content or a fabricated - filesystem URI; and - - update generic JSON output and debug summaries to serialize these bounded - fields deterministically. -6. Ensure checkpoint, debug, and output clones own their byte slices and maps. - A consumer or test mutation must not affect another fan-out consumer or the - producer artifact. - -### Tests - -- Add handoff tests for one producer fan-out, empty typed collections, missing - output, rejected-only output, multiple outputs, type/schema/media/size - mismatch, target immutability, and failure before consumer start. -- Add a resume-oriented runner test proving changed canonical producer content - changes the consumer dependency fingerprint and prevents stale reuse. -- Extend artifacts/JSON/debug contract tests with field assertions for bounded - generated provenance and explicit assertions that content and local paths are - absent. Avoid whole-document snapshots. - -### Completion Gate - -Run focused pipeline, artifacts, JSON output, and debug tests, followed by `go -test ./internal/framework/... ./internal/core/artifacts -./internal/modules/generic/output/json`. A generic fake-codec pipeline must -complete a two-step handoff both from fresh execution and a compatible reused -producer checkpoint. - -## Stage 4: Dependency-Aware Checkpoints And Selective Recompute - -### Objective - -Complete safe reuse, transitive invalidation, observable decisions, and the -`--recompute-step` operator control. - -### Changes - -1. Update checkpoint identity and manifests in - `internal/framework/checkpoint`: - - introduce workspace schema `notarius.workspace.v3` and retain explicit - recognition of v1/v2 as incompatible cold misses; - - include step ID in lane-stage manifests and lookup/recording context; - - include ordered topology in the persistent pipeline identity; and - - preserve the Stage 3 generated fingerprint tuple exactly, without content - or secrets in decision reasons. -2. When loading a normalized producer checkpoint, validate its manifest, - deserialize through the registered codec, re-encode canonically, and compare - schema/media/content fingerprints before exposing it to handoff. Corrupt, - missing, rejected, or incompatible state is a cold miss for ordinary resume - and must never reach a consumer. -3. Build a lane-level dependency index from resolved generated bindings. Use it - to invalidate only transitive consumer lanes when producer identity or bytes - change; unrelated lanes, including unrelated lanes in a later step, remain - eligible for reuse. The step barrier still applies when reused and executed - lanes coexist. -4. Add a single-value `--recompute-step ` flag in - `internal/cli/run.go`. Reject repeated occurrences, unknown steps, use - without checkpoint recording, use without `--resume`, and combination with - `--only`. Accept the stable implicit step ID `default`; selecting it forces - every lane in that single step. One selected step is sufficient for this - scope; do not add multi-selection semantics. -5. Convert the selected step to a force-execution set containing every lane in - that step plus the lane-level transitive dependency closure. Loader policy, - not persistent checkpoint identity, applies the force set. Required - predecessors and unrelated lanes remain reusable. If a required predecessor - was not selected and has no reusable accepted artifact, fail before starting - a dependent lane rather than implicitly recomputing it. -6. Replace the boolean-only checkpoint reporting model as needed with a bounded - decision category: `executed`, `reused`, `forced_recompute`, or - `dependency_invalidated`, plus a stable reason code and optional safe detail. - Propagate it to checkpoint events, manifests, debug summaries, and CLI - diagnostics. Do not make exact prose part of a test contract. - -### Tests - -- In checkpoint loader/recorder tests, cover v1/v2 cold misses, v3 step - identity, compatible producer decode, corruption, exact dependency match, - content/schema changes, and bounded reason fields. -- In pipeline tests, cover transitive invalidation and reuse of unrelated work. -- Add one CLI contract table for valid recomputation and every invalid flag - combination, plus one execution test proving the selected closure is forced - while a predecessor and unrelated lane are reused. -- Do not assert internal loader call counts when observable decision records - and outputs establish the behavior. +- Add a table in `internal/framework/checkpoint` that exercises one + representative input per reason-code family and asserts category, code, + bounded valid UTF-8 detail, and absence of supplied secret/path sentinels. +- Add pipeline behavior tests proving a required-predecessor failure records the + underlying missing, corrupt, incompatible, or dependency-invalidated + decision before returning. +- Retain focused manifest/debug serialization assertions for category and code; + do not add exact-detail or full-manifest snapshots. ### Completion Gate Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline -./internal/cli`, then checkpoint/debug state tests under the race detector as -directed by `docs/development.md`. Manually inspect one test fixture's decision -records to confirm they contain no reference bytes, secret values, or local -paths. +./internal/core/artifacts ./internal/core/debugbundle ./internal/cli`. Search the +production checkpoint package to confirm no decision category or reason code is +derived from diagnostic prose. -## Stage 5: D&D NPC-First Production Adoption +## Stage 3: Hydrate Required Predecessors From Accepted Normalized Artifacts ### Objective -Adopt the platform in the production D&D composition: NPC normalization runs -first, and its canonical artifact grounds spell extraction, combat-turn -extraction, and combat-turn normalization at operation time. +Make selective recomputation enforce the lane-level accepted-artifact contract: +a valid normalized producer artifact is sufficient even when its extract or +merge stage cache is unavailable. ### Changes -1. Declare the normalized NPC-list artifact kind on every `npcs` consumer slot: - - `internal/modules/dnd/extract/spells`; - - `internal/modules/dnd/extract/combatturns`; and - - `internal/modules/dnd/normalize/combatturns`. - Keep the slot optional for external standalone use; a configured generated - binding becomes required through the framework dependency. -2. Refactor `internal/modules/dnd/npcs/registry` to provide a package-owned, - concurrency-safe operation-time resolver/cache: - - validate and seed construction-time external references so bad static - configuration still fails before source parsing; - - resolve the effective `npcs` item from each operation request; - - reuse the seeded immutable registry when its digest matches and cache a - generated registry by canonical digest so concurrent chunk operations do - not repeatedly decode it; and - - return cloned or immutable views and never retain caller-owned content. -3. Change the spell extractor, combat-turn extractor, and combat-turn - normalizer to obtain the NPC registry from `req.References` for each - operation through that resolver. Do not reconstruct modules at a step - boundary. Retain current behavior when the request has no NPC item. -4. Keep static prompt/schema metadata and construction-time external reference - fingerprints intact. Record generated NPC identity through framework - reference provenance and dependency fingerprints; do not place - operation-varying generated digests into singleton module metadata. -5. Preserve prompt ordering and evidence policy. The generated NPC listing is - stable contextual input before the variable transcript, but it cannot prove - that a spell or combat event occurred. Source-unit citations remain the only - event evidence. -6. Add a maintained explicit two-step D&D configuration under `examples/` and a - matching production-catalog fixture under - `internal/modules/integration/testdata/`. Replace or supplement the two - manual NPC-to-spell and NPC-to-combat configurations with this single - NPC-first workflow; retain standalone examples that demonstrate external - NPC references where they remain useful. -7. Add one offline integration test using scripted/fake LLM responses. Assert - NPC normalization completes first, the exact canonical NPC digest reaches - all three consumers, spell and combat lanes may both succeed, combat - normalization uses the registry, and generated NPC context is not accepted - as source evidence by itself. +1. Extend the checkpoint loader contract with a dedicated accepted-normalized- + artifact lookup. Implement it in the filesystem loader and every no-op or + test implementation. The lookup receives step ID, lane ID, and normalizer + module key and returns the normalized artifact, its warnings, and an explicit + checkpoint decision. It must not require caller-supplied extract or merge + dependency fingerprints. +2. The filesystem lookup may reuse the existing normalize manifest and payload; + do not add a second persistent copy. It is reusable only when all of the + following hold: + - workspace schema is v3 and the non-empty stored checkpoint identity matches + the current invocation identity; + - stage, step, lane, normalizer module, and successful status match; + - the manifest output digest matches the payload; and + - the payload can subsequently be validated through the registered artifact + codec. + Skipping caller-supplied merge dependencies is safe only because the matched + non-empty checkpoint identity already binds the current input, resolved + topology and configuration, external references, runtime overrides, LLM + profiles, and component semantic fingerprints. Do not weaken or omit that + identity check. + A loader lacking a verifiable current identity must return an unavailable + decision rather than perform accepted-artifact reuse. +3. Add a pipeline hydration helper that decodes the returned serialized + artifact through the producer's registered codec, re-encodes it canonically, + and requires exact artifact kind, schema ID/name/version/digest, media type, + content bytes, and content digest. Return a runner-owned clone with producer + step, lane, module, and source identity. Any mismatch is an explicit bounded + decision and the bytes never reach a consumer. +4. Before normal execution of a lane marked in `RequireReusableLanes`, use the + accepted-artifact lookup: + - on success, mark the lane terminal without invoking extract, merge, + normalize, or their validators; + - append the accepted normalized output in the normal deterministic location + and restore only the warnings stored with that normalized checkpoint; + - record one `reused` normalize decision with a stable accepted-output reuse + reason code; do not synthesize extract/merge decisions, warnings, or + rejections that were not loaded; and + - allow the ordinary step barrier and generated handoff code to consume that + output exactly as it consumes a freshly executed output. +5. On missing, rejected, corrupt, non-canonical, or incompatible accepted state, + record the decision and fail the run before the dependent step begins. Do + not fall back to stage execution. Forced lanes must bypass this hydration + path and execute normally. +6. Leave ordinary resume unchanged when no lane is marked + `RequireReusableLanes`: it may reuse or recompute extract, merge, and + normalize progressively under the existing cold-miss rules. +7. Keep generated-reference fingerprints and provenance unchanged. Hydrating a + byte-identical producer must yield the same canonical handoff digest and + downstream dependency fingerprint as fresh execution. +8. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and + `docs/operations.md` in this stage to distinguish progressive stage reuse + from accepted normalized-artifact hydration and to document the fail-rather- + than-rerun rule for invalid required predecessors. ### Tests -- Add focused registry and module tests for static fallback, generated override, - concurrent cache safety, malformed runtime content, and ownership isolation. -- Extend prompt-input tests only for semantic placement and content. Do not add - exact prompt-length or message-count change detectors. -- Run the single end-to-end offline D&D handoff test rather than duplicating it - separately for every consumer. +- At the pipeline/checkpoint boundary, create a valid producer normalize + checkpoint while omitting or corrupting its extract and merge checkpoints. + Select a later step for recomputation and prove the producer hydrates, no + producer operation or validator runs, and the dependent receives the exact + canonical artifact. +- Cover missing, rejected-status, corrupt, non-canonical, wrong-codec-identity, + and wrong-content-digest normalize state. Assert failure and the recorded + stable decision before any consumer invocation. +- Prove forced producers execute instead of hydrating, while a reusable + predecessor and an unrelated lane remain reusable. +- Prove fresh and hydrated producer outputs create identical generated + provenance and downstream checkpoint fingerprints without exposing content + or paths. ### Completion Gate -Run all D&D package tests and integration tests, including the race detector for -the registry cache, then parse and resolve every maintained example through the -production catalog. No test may require credentials or a live LLM provider. +Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline +./internal/modules/integration` and the corresponding race tests. Manually +inspect one failed test fixture to confirm the accepted artifact remains on +disk but its bytes do not appear in the manifest, decision detail, debug +summary, or error. -## Stage 6: Current Documentation And Release Verification +## Stage 4: Complete Recompute CLI And Recovery Acceptance Coverage ### Objective -Document the now-implemented behavior in its canonical current owners, retire -obsolete manual workflow guidance, and perform repository-wide verification. +Exercise the operator-visible recomputation contract through stable boundaries +and close the acceptance-test omissions from the original plan. ### Changes -1. Update `docs/config.md` with version-3 `steps`, legacy `artifacts` - compatibility, reference source forms, precedence/conflict rules, global - identity rules, and the exact D&D two-step example. -2. Update `docs/cli.md` with `--recompute-step`, prerequisites, closure - semantics, invalid combinations, and checkpoint decision categories. -3. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and - `docs/internal/overview.md` with the fixed ordered-step model, preparation - timing, handoff boundary, codec use, provenance, and checkpoint dependency - behavior. Keep the architecture description linear and explicitly state that - this is not a general DAG. -4. Update `docs/operations.md` with resume/recompute procedures and safe failure - diagnosis. Update the D&D integration documents for generated NPC context, - evidence limitations, and the maintained workflow. Update - `docs/policy/testing.md` only if the implementation exposes a genuinely new - durable testing policy; do not restate feature-specific tests there. -5. Update maintained example checks and any README/index links that point to - replaced sequential examples. Keep the feature roadmap as historical target - state until the project applies its normal roadmap-completion process; do not - turn it into a duplicate current-behavior manual. -6. Review changed exported identifiers and package comments, run formatting and - static analysis, and remove obsolete flat-lane compatibility helpers, - temporary test adapters, dead sequential-workflow code, and TODOs introduced - during earlier stages. - -### Verification - -Run the repository-prescribed commands from `docs/development.md`, including: - -1. all unit and integration tests; -2. race tests for concurrency-sensitive framework, checkpoint, and D&D registry - packages; -3. `go vet` and the normal build; -4. maintained-example/config validation; and -5. documentation checks. - -Perform one final acceptance review against every bullet in -[Ordered Pipeline Steps](ordered-pipeline-steps.md). Inspect a successful fresh -run, a resumed run, a forced-recompute run, and a missing-producer failure for -deterministic ordering, correct dependency decisions, bounded provenance, and -absence of generated content or secrets in manifests/debug state. +1. Add one table-driven CLI contract test for `--recompute-step` covering: + - a valid explicit step and the implicit `default` step; + - repeated flags and an empty or unknown step ID; + - use without `--resume`; + - use when checkpoint recording is disabled; and + - combination with `--only`. + Assert exit classification and stable identifying fragments or reason codes, + not complete prose. +2. Add one filesystem-backed CLI execution test using deterministic fake + modules and a three-step generated dependency chain plus one unrelated lane: + - perform a fresh checkpointed run; + - remove or corrupt only the required producer's extract and merge state + inside `t.TempDir()`, leaving its normalize artifact valid; + - resume with the middle step selected; + - assert the selected lane and transitive dependents execute, the predecessor + hydrates without module calls, the unrelated lane reuses, and output and + decision ordering are deterministic; and + - invalidate the producer normalize state in a subcase and assert the command + fails before dependent execution with the bounded decision preserved. +3. Add or extend one generic two-step pipeline test proving handoff succeeds + both from fresh producer execution and from accepted normalized-artifact + hydration. Keep this at the pipeline boundary if the CLI execution test + already proves flag wiring; do not duplicate every CLI case end to end. +4. Review existing recomputation policy tests. Retain the pure closure test + because it protects transitive selection, but remove or consolidate any new + test that merely repeats the CLI or pipeline behavior above. +5. Make only production changes revealed as necessary by these contract tests; + do not add new flag semantics or broaden the feature roadmap. ### Completion Gate -The feature is complete only when every acceptance criterion in the feature -roadmap is demonstrated by a stable test or maintained example and the full -repository verification suite passes. +Run `go test ./internal/cli ./internal/framework/pipeline +./internal/framework/checkpoint`, then run the CLI, pipeline, and checkpoint +packages under the race detector. Confirm no test asserts internal loader call +counts, exact decision detail, filesystem layout outside a temp workspace, or +the exact length of any prompt or prefix. + +## Stage 5: Current Documentation And Release Verification + +### Objective + +Reconcile all canonical documentation after the staged changes and verify the +complete feature against policy and roadmap. + +### Changes + +1. Re-read `docs/internal/pipeline.md`, `docs/internal/state.md`, + `docs/operations.md`, and `docs/cli.md` against the final code. Correct any + stale statements left by Stages 1 through 4 without duplicating their + canonical contracts. +2. Ensure internal documentation describes the coordinator, lane engine, stage + checkpoint flow, and accepted-artifact validation at the responsibility + level without listing volatile private helper names. +3. Confirm the operator-visible reason-code table has one canonical owner and + other documents link to it rather than maintaining parallel copies. +4. Remove stale implementation claims exposed by this work and ensure the + feature roadmap continues to describe target policy rather than task + sequencing. + +### Verification + +Run the repository-prescribed commands from `docs/development.md`: + +```sh +go test ./... +go vet ./... +go build ./cmd/notarius +``` + +Also run race tests for the pipeline, checkpoint, CLI state, D&D NPC registry, +and D&D integration packages. Validate maintained examples/configurations +through the production catalog. Review one fresh run, ordinary resumed run, +forced-recompute run, hydrated-predecessor run, and invalid-predecessor failure +for deterministic ordering, correct decisions, bounded provenance, and absence +of generated content, secrets, or local paths in manifests and diagnostics. + +### Completion Gate + +The follow-up is complete only when every finding summarized above is protected +by a stable behavioral test, the current documentation matches the corrected +implementation, the full verification suite passes, and the worktree contains +no temporary adapters or TODOs introduced by these stages. ## Open Questions -None. The feature roadmap and this plan fix all product and architecture choices -required for implementation. If an implementation detail conflicts with a -policy document, the policy document takes precedence; if it would change the -product semantics above, stop and amend the roadmap rather than deciding it in -code. +None. The feature roadmap and this plan fix the required product and +architecture choices. If implementation reveals that accepted normalized +artifacts cannot be validated safely without a persistent format change, stop +and amend this plan rather than weakening identity, codec, or content +validation.