diff --git a/docs/cli.md b/docs/cli.md index 32e360a..7510a99 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -38,7 +38,8 @@ Flags: - `--recompute-step step-id`: with `--resume` and checkpoint recording enabled, force the named ordered step and every transitive dependent lane to execute. Compatible required predecessors and unrelated lanes remain reusable. The - value must identify an explicit step; it cannot be combined with `--only`. + value may identify an explicit step or the implicit single-step ID `default`; + it cannot be combined with `--only`. - `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this invocation. `auto` reuses a valid plan by canonical source digest, `bypass` performs no plan-cache I/O, and `refresh` regenerates and replaces a valid @@ -176,6 +177,8 @@ Checkpoint decisions use these categories: `reused`, `executed`, detail identify the decision without exposing reference content, local paths, or secrets. `--recompute-step` requires checkpoint recording and `--resume`; unknown step IDs, empty values, and combinations with `--only` are rejected. +The operator meanings of checkpoint reason codes are maintained in +[Operations](operations.md#resume-and-selective-recompute). Use `--debug` to retain the redacted summary and trace bundle for one run. The bundle is allocated before pipeline resolution; once allocated, its path is diff --git a/docs/internal/pipeline.md b/docs/internal/pipeline.md index b6344c7..495db01 100644 --- a/docs/internal/pipeline.md +++ b/docs/internal/pipeline.md @@ -191,6 +191,13 @@ the runner returns. ## Execution Flow +The pipeline-wide coordinator owns the ordered step loop, generated-reference +sets at each barrier, and deterministic merging of step outcomes. For one step, +the lane engine initializes checkpoint state in lane order, dispatches bounded +extract work, advances terminal lanes through serial merge and normalize work, +selects failures by stable pipeline scope, and merges lane-local outcomes back +in resolved order. Completion timing never becomes public ordering. + The runner: 1. validates its prepared input; @@ -306,8 +313,8 @@ implementations when collaborators are absent. Each checkpointed workflow boundary records a running, succeeded, or failed transition. Reuse decisions are consulted in workflow order and accepted payloads are cloned before entering the normal handoff path. Typed extract, merge, and normalize -checkpoints store codec bytes with artifact kind, schema ID and version, exact -schema digest, and media type. Reuse compares that identity with the prepared +checkpoints store codec bytes with artifact kind, schema ID, name, version and +exact digest, and media type. Reuse compares that identity with the prepared codec and decodes through the codec; missing identity, mismatches, corrupt bytes, and decode failures become explicit reuse misses and execute the lane normally. Dependency fingerprints and debug content digests use the same stable @@ -339,7 +346,8 @@ hydration and execute normally. A missing, rejected, corrupt, incompatible, or changed producer blocks its dependent while leaving independent work eligible for reuse. The runner records bounded decision categories: `reused`, `executed`, `forced_recompute`, and -`dependency_invalidated`. +`dependency_invalidated`. Operator meanings for the stable reason codes belong +to [Operations](../operations.md#resume-and-selective-recompute). The CLI includes prepared-component fingerprints in the run-wide checkpoint identity alongside resolved configuration, raw input, reference provenance, @@ -350,9 +358,9 @@ fingerprint intentionally produces a cold cache miss. Existing checkpoint schemas and paths remain unchanged. The CLI's `--recompute-step` policy forces the selected step and all transitive -dependents, but requires reusable checkpoints for all selected producers that -precede it. It changes loader decisions only; it does not alter persistent -checkpoint identity. +dependents, but requires accepted normalized artifacts for every unselected +producer on which that closure depends. It changes execution policy only; it +does not alter persistent checkpoint identity. Debug instrumentation wraps run, stage, attempt, validator, and structured LLM boundaries. Every executed chunk, extract, merge, and normalize attempt writes @@ -418,7 +426,9 @@ stage, resolved lane, and source chunk rather than completion time. - `internal/framework/pipeline/references_test.go`: target resolution and materialization. - `internal/cli/run_contract_test.go`: production run transitions, retries, - rejections, warnings, debug hooks, and manifests. + rejections, warnings, CLI recomputation controls, debug hooks, and manifests. +- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed + selective recomputation and accepted-producer recovery. - `internal/cli/production_contract_test.go`: production composition and configuration-resolution smoke coverage. - `internal/cli/example_contract_test.go`: maintained example resolution and diff --git a/docs/internal/state.md b/docs/internal/state.md index 4153fb7..9faac72 100644 --- a/docs/internal/state.md +++ b/docs/internal/state.md @@ -35,13 +35,13 @@ they do not describe a current public state surface. Ordered-step lane checkpoints include the step identity in their storage scope. When a later lane consumes a generated artifact, its dependency fingerprints include the producer's artifact kind, complete schema identity, media type, -canonical content digest, and size. A producer checkpoint may be decoded through -the registered codec and handed off without rerunning it. Missing, rejected, -corrupt, incompatible, or changed producer state produces a bounded -`dependency_invalidated` decision for every transitive dependent lane; it does -not permit stale downstream reuse. The CLI's selective recomputation policy -records `forced_recompute` for the selected step and its dependents while -requiring compatible predecessor checkpoints. +canonical content digest, and size. Ordinary resume compares those fingerprints +when progressively loading consumer stage checkpoints, so changed producer +content produces `dependency_invalidated` rather than stale downstream reuse. +Selective recomputation instead requires each unselected producer's accepted +normalized artifact; invalid accepted state records its specific bounded reason +and stops before the dependent. The selected step and its transitive dependents +record `forced_recompute`. Ordinary resume loads extract, merge, and normalize checkpoints progressively and may execute later lane stages after an earlier cache miss. Selective @@ -63,7 +63,8 @@ detail passes through one UTF-8-safe bounded sanitizer and contains only allowlisted diagnostic context, never payloads, references, credentials, environment values, or physical paths. Typed categories and codes remain intact through pipeline events and become strings only in manifest and debug-summary -JSON. +JSON. [Operations](../operations.md#resume-and-selective-recompute) is the +canonical operator-facing reason-code reference. `internal/core/fileio` provides confined atomic file writes used by state collaborators. The chunk-plan store retains its stronger entry validation. @@ -98,6 +99,8 @@ separately and never replace the command's primary error. terminalization, and output/report boundaries. - `internal/cli/state_hardening_test.go`: independent roots, reuse, failures, permissions, cleanup, and redaction. +- `internal/cli/recompute_execution_contract_test.go`: selective recomputation, + filesystem recovery, deterministic decisions, and failed predecessor state. - `internal/cli/production_contract_test.go`: production composition and configuration validation at the CLI boundary. - `internal/cli/example_contract_test.go`: maintained example ownership. diff --git a/docs/operations.md b/docs/operations.md index a5c3271..0225c6d 100644 --- a/docs/operations.md +++ b/docs/operations.md @@ -161,10 +161,11 @@ for reuse. `--recompute-step ` requires both `--resume` and `cache.checkpoints.enabled: true`. It forces the named step and all transitive dependents to execute, while compatible predecessors and unrelated lanes remain -reusable. It cannot be combined with `--only`, and it does not change the -persistent identity of otherwise identical checkpoints. Decisions are bounded -and categorized as `reused`, `executed`, `forced_recompute`, or -`dependency_invalidated`. +reusable. The ID may be an explicit configured step or `default` for an +implicit single-step pipeline. It cannot be combined with `--only`, and it does +not change the persistent identity of otherwise identical checkpoints. +Decisions are bounded and categorized as `reused`, `executed`, +`forced_recompute`, or `dependency_invalidated`. For an unselected producer required by a recomputed step, Notarius loads the accepted normalized artifact directly. Valid normalize state is sufficient even @@ -182,6 +183,7 @@ generated reference content. For diagnosis, first check the producer step and lane in the manifest, then inspect checkpoint decision categories and reason codes. Rerun the producer explicitly rather than copying an artifact into the checkpoint root. + The decision that caused a required-predecessor failure is retained before the run returns, and the CLI error identifies its step, lane, and reason code. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index adad441..411af44 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -5,18 +5,6 @@ configuration, operations, internal, and integration docs. This roadmap records future work only. Items are ordered roughly by current value and specificity, not as committed release dates. -## Proposed Next Scope: Ordered Pipeline Steps - -Add ordered groups of artifact lanes, canonical generated-artifact references, -and dependency-aware checkpoint reuse without introducing a general DAG. The -first production workflow runs normalized D&D NPC extraction before spell and -combat-turn extraction and supplies that NPC artifact to their declared -reference slots. - -The bounded target, compatibility and architecture decisions, exclusions, and -acceptance criteria are defined in -[Proposed Scope: Ordered Pipeline Steps](ordered-pipeline-steps.md). - ## Near-Term D&D Pipeline ### Evaluate Spell Extraction And Normalization diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 34c35c2..8559cb3 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,11 +1,19 @@ # Implementation Plan: Ordered Pipeline Follow-Up +## Status + +Completed on 2026-07-22. This document retains the implementation sequence for +historical context; current contracts are maintained in the canonical CLI, +configuration, operations, integration, and internal documentation linked from +the [development guide](../development.md). + ## Purpose -Close the remaining correctness, observability, test, and maintainability gaps -in the implemented [Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. -That feature roadmap remains the authority for product intent, policy choices, -acceptance criteria, and exclusions. This document is the ordered, +Record the work that closed the correctness, observability, test, and +maintainability gaps in the implemented +[Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. That feature +roadmap remains the authority for product intent, policy choices, acceptance +criteria, and exclusions. This document preserves the ordered, decision-complete implementation sequence for the follow-up work. ## Background @@ -18,19 +26,18 @@ output provenance, support `--recompute-step`, and use generated NPC output at operation time in the D&D spell and combat consumers. Current documentation and maintained examples describe that model. -The remaining work is narrower: +The completed follow-up addressed these narrower gaps: -- selective recomputation currently treats an unselected predecessor as - reusable only when all of its extract, merge, and normalize stage checkpoints - are reusable, although the architectural dependency is its accepted - normalized artifact; -- the failed stage decision is not always recorded before a required-predecessor +- selective recomputation now hydrates an unselected predecessor from its + accepted normalized artifact without requiring its extract and merge state; +- the failed checkpoint decision is recorded before a required-predecessor error returns; -- checkpoint reason codes are inferred from human-readable prose instead of - being assigned explicitly; -- the runner's lane and checkpoint orchestration has become too concentrated in - large functions; and -- the roadmap-required CLI and resumed-producer acceptance coverage is absent. +- checkpoint reason codes are assigned explicitly rather than inferred from + human-readable prose; +- runner lane and checkpoint orchestration has explicit responsibility seams; + and +- CLI and resumed-producer acceptance coverage exercises the complete recovery + contract. No new product feature is introduced by this plan. Preserve configuration file version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do diff --git a/docs/roadmap/ordered-pipeline-steps.md b/docs/roadmap/ordered-pipeline-steps.md index 90f43b4..dfb3462 100644 --- a/docs/roadmap/ordered-pipeline-steps.md +++ b/docs/roadmap/ordered-pipeline-steps.md @@ -1,22 +1,19 @@ -# Proposed Scope: Ordered Pipeline Steps +# Scope: Ordered Pipeline Steps ## Status -Proposed as the next implementation scope. This document refines the -near-term priority in [Future Work](future.md) into a bounded feature target. It -defines desired behavior and scope, not a file-by-file implementation plan. +Implemented. This document preserves the bounded feature policy, architecture +choices, acceptance criteria, and exclusions. Current behavior belongs in the +canonical [CLI](../cli.md), [Configuration](../config.md), +[Operations](../operations.md), and [internal pipeline](../internal/pipeline.md) +documentation rather than in this roadmap. -## Recommendation +## Policy Recommendation -Implement ordered pipeline steps, generated artifact references, and -dependency-aware checkpoint reuse as one coherent platform capability. Prove -the capability with the D&D workflow in which accepted normalized NPC output -is produced first and then supplied to spell extraction, combat-turn -extraction, and combat-turn normalization. - -This should precede item extraction. It establishes the artifact-handoff and -dependency semantics that the planned item lane will also consume, without -combining two substantial features in one implementation effort. +Treat ordered pipeline steps, generated artifact references, and +dependency-aware checkpoint reuse as one coherent platform capability. The D&D +proving workflow produces accepted normalized NPC output first and then supplies +it to spell extraction, combat-turn extraction, and combat-turn normalization. ## Intended Outcome @@ -185,12 +182,15 @@ digest. The pipeline digest protects topology; stage dependency fingerprints protect the exact upstream artifact consumed. The runner must never combine a new or changed producer with stale dependent output. -Compatible producer checkpoints may be decoded through the registered codec -and handed to later steps without rerunning the producer. A missing, rejected, -corrupt, incompatible, or changed producer invalidates every transitive -dependent checkpoint. Independent work remains reusable. +Ordinary resume may progressively decode compatible producer and consumer stage +checkpoints through the registered codec. Selective recomputation may hydrate a +required unselected producer directly from its accepted normalized artifact; +its extract and merge state are not prerequisites. Missing, rejected, corrupt, +incompatible, or changed accepted state stops the run before dependent +execution rather than implicitly rerunning the producer. Independent work +remains reusable. -Add one operator control, `--recompute-step `, with these semantics: +The operator control `--recompute-step ` has these semantics: - it requires checkpoint recording and `--resume`; - the selected step and all transitive dependents execute rather than reuse @@ -239,10 +239,10 @@ Spell and combat-turn lanes may execute concurrently after the handoff. NPC content may ground names and identities but cannot establish a spell cast or combat event; source units remain the only event evidence. -The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be -replaced or supplemented by one ordered-pipeline example. Existing module keys, -artifact contracts, reference slot names, prompt IDs, and D&D evidence policy -remain unchanged. +The maintained workflow uses one ordered-pipeline example instead of a manual +two-run NPC-to-spell or NPC-to-combat handoff. Existing module keys, artifact +contracts, reference slot names, prompt IDs, and D&D evidence policy remain +unchanged. ## Included Work @@ -259,9 +259,8 @@ remain unchanged. - D&D NPC-first production composition for spell and combat-turn consumers. - Refactoring the affected D&D consumers so generated NPC references are available at operation time while retaining early static construction. -- Maintained examples and updates to current architecture, configuration, CLI, - operations, internal, integration, and testing documentation when behavior - lands. +- Maintained examples and current architecture, configuration, CLI, operations, + internal, integration, and testing documentation. - An ADR recording the bounded ordered-step extension to the fixed pipeline architecture and its explicit rejection of a general DAG.