Reconcile ordered pipeline documentation

This commit is contained in:
2026-07-22 02:40:36 +00:00
parent f15fd4f9c1
commit 906d97b391
7 changed files with 85 additions and 73 deletions

View File

@@ -38,7 +38,8 @@ Flags:
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled,
force the named ordered step and every transitive dependent lane to execute.
Compatible required predecessors and unrelated lanes remain reusable. The
value must identify an explicit step; it cannot be combined with `--only`.
value may identify an explicit step or the implicit single-step ID `default`;
it cannot be combined with `--only`.
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
@@ -176,6 +177,8 @@ Checkpoint decisions use these categories: `reused`, `executed`,
detail identify the decision without exposing reference content, local paths,
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
unknown step IDs, empty values, and combinations with `--only` are rejected.
The operator meanings of checkpoint reason codes are maintained in
[Operations](operations.md#resume-and-selective-recompute).
Use `--debug` to retain the redacted summary and trace bundle for one run. The
bundle is allocated before pipeline resolution; once allocated, its path is

View File

@@ -191,6 +191,13 @@ the runner returns.
## Execution Flow
The pipeline-wide coordinator owns the ordered step loop, generated-reference
sets at each barrier, and deterministic merging of step outcomes. For one step,
the lane engine initializes checkpoint state in lane order, dispatches bounded
extract work, advances terminal lanes through serial merge and normalize work,
selects failures by stable pipeline scope, and merges lane-local outcomes back
in resolved order. Completion timing never becomes public ordering.
The runner:
1. validates its prepared input;
@@ -306,8 +313,8 @@ implementations when collaborators are absent. Each checkpointed workflow
boundary records a running, succeeded, or failed transition. Reuse decisions
are consulted in workflow order and accepted payloads are cloned before
entering the normal handoff path. Typed extract, merge, and normalize
checkpoints store codec bytes with artifact kind, schema ID and version, exact
schema digest, and media type. Reuse compares that identity with the prepared
checkpoints store codec bytes with artifact kind, schema ID, name, version and
exact digest, and media type. Reuse compares that identity with the prepared
codec and decodes through the codec; missing identity, mismatches, corrupt
bytes, and decode failures become explicit reuse misses and execute the lane
normally. Dependency fingerprints and debug content digests use the same stable
@@ -339,7 +346,8 @@ hydration and execute normally. A missing, rejected, corrupt, incompatible, or
changed producer blocks its dependent while leaving independent work eligible
for reuse. The runner records bounded decision
categories: `reused`, `executed`, `forced_recompute`, and
`dependency_invalidated`.
`dependency_invalidated`. Operator meanings for the stable reason codes belong
to [Operations](../operations.md#resume-and-selective-recompute).
The CLI includes prepared-component fingerprints in the run-wide checkpoint
identity alongside resolved configuration, raw input, reference provenance,
@@ -350,9 +358,9 @@ fingerprint intentionally produces a cold cache miss. Existing checkpoint
schemas and paths remain unchanged.
The CLI's `--recompute-step` policy forces the selected step and all transitive
dependents, but requires reusable checkpoints for all selected producers that
precede it. It changes loader decisions only; it does not alter persistent
checkpoint identity.
dependents, but requires accepted normalized artifacts for every unselected
producer on which that closure depends. It changes execution policy only; it
does not alter persistent checkpoint identity.
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
@@ -418,7 +426,9 @@ stage, resolved lane, and source chunk rather than completion time.
- `internal/framework/pipeline/references_test.go`: target resolution and
materialization.
- `internal/cli/run_contract_test.go`: production run transitions, retries,
rejections, warnings, debug hooks, and manifests.
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
selective recomputation and accepted-producer recovery.
- `internal/cli/production_contract_test.go`: production composition and
configuration-resolution smoke coverage.
- `internal/cli/example_contract_test.go`: maintained example resolution and

View File

@@ -35,13 +35,13 @@ they do not describe a current public state surface.
Ordered-step lane checkpoints include the step identity in their storage scope.
When a later lane consumes a generated artifact, its dependency fingerprints
include the producer's artifact kind, complete schema identity, media type,
canonical content digest, and size. A producer checkpoint may be decoded through
the registered codec and handed off without rerunning it. Missing, rejected,
corrupt, incompatible, or changed producer state produces a bounded
`dependency_invalidated` decision for every transitive dependent lane; it does
not permit stale downstream reuse. The CLI's selective recomputation policy
records `forced_recompute` for the selected step and its dependents while
requiring compatible predecessor checkpoints.
canonical content digest, and size. Ordinary resume compares those fingerprints
when progressively loading consumer stage checkpoints, so changed producer
content produces `dependency_invalidated` rather than stale downstream reuse.
Selective recomputation instead requires each unselected producer's accepted
normalized artifact; invalid accepted state records its specific bounded reason
and stops before the dependent. The selected step and its transitive dependents
record `forced_recompute`.
Ordinary resume loads extract, merge, and normalize checkpoints progressively
and may execute later lane stages after an earlier cache miss. Selective
@@ -63,7 +63,8 @@ detail passes through one UTF-8-safe bounded sanitizer and contains only
allowlisted diagnostic context, never payloads, references, credentials,
environment values, or physical paths. Typed categories and codes remain intact
through pipeline events and become strings only in manifest and debug-summary
JSON.
JSON. [Operations](../operations.md#resume-and-selective-recompute) is the
canonical operator-facing reason-code reference.
`internal/core/fileio` provides confined atomic file writes used by state
collaborators. The chunk-plan store retains its stronger entry validation.
@@ -98,6 +99,8 @@ separately and never replace the command's primary error.
terminalization, and output/report boundaries.
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
permissions, cleanup, and redaction.
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
filesystem recovery, deterministic decisions, and failed predecessor state.
- `internal/cli/production_contract_test.go`: production composition and
configuration validation at the CLI boundary.
- `internal/cli/example_contract_test.go`: maintained example ownership.

View File

@@ -161,10 +161,11 @@ for reuse.
`--recompute-step <step-id>` requires both `--resume` and
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
dependents to execute, while compatible predecessors and unrelated lanes remain
reusable. It cannot be combined with `--only`, and it does not change the
persistent identity of otherwise identical checkpoints. Decisions are bounded
and categorized as `reused`, `executed`, `forced_recompute`, or
`dependency_invalidated`.
reusable. The ID may be an explicit configured step or `default` for an
implicit single-step pipeline. It cannot be combined with `--only`, and it does
not change the persistent identity of otherwise identical checkpoints.
Decisions are bounded and categorized as `reused`, `executed`,
`forced_recompute`, or `dependency_invalidated`.
For an unselected producer required by a recomputed step, Notarius loads the
accepted normalized artifact directly. Valid normalize state is sufficient even
@@ -182,6 +183,7 @@ generated reference content. For diagnosis, first check the producer step and
lane in the manifest, then inspect checkpoint decision categories and reason
codes. Rerun the producer explicitly rather than copying an artifact into the
checkpoint root.
The decision that caused a required-predecessor failure is retained before the
run returns, and the CLI error identifies its step, lane, and reason code.

View File

@@ -5,18 +5,6 @@ configuration, operations, internal, and integration docs. This roadmap records
future work only. Items are ordered roughly by current value and specificity,
not as committed release dates.
## Proposed Next Scope: Ordered Pipeline Steps
Add ordered groups of artifact lanes, canonical generated-artifact references,
and dependency-aware checkpoint reuse without introducing a general DAG. The
first production workflow runs normalized D&D NPC extraction before spell and
combat-turn extraction and supplies that NPC artifact to their declared
reference slots.
The bounded target, compatibility and architecture decisions, exclusions, and
acceptance criteria are defined in
[Proposed Scope: Ordered Pipeline Steps](ordered-pipeline-steps.md).
## Near-Term D&D Pipeline
### Evaluate Spell Extraction And Normalization

View File

@@ -1,11 +1,19 @@
# Implementation Plan: Ordered Pipeline Follow-Up
## Status
Completed on 2026-07-22. This document retains the implementation sequence for
historical context; current contracts are maintained in the canonical CLI,
configuration, operations, integration, and internal documentation linked from
the [development guide](../development.md).
## Purpose
Close the remaining correctness, observability, test, and maintainability gaps
in the implemented [Ordered Pipeline Steps](ordered-pipeline-steps.md) feature.
That feature roadmap remains the authority for product intent, policy choices,
acceptance criteria, and exclusions. This document is the ordered,
Record the work that closed the correctness, observability, test, and
maintainability gaps in the implemented
[Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. That feature
roadmap remains the authority for product intent, policy choices, acceptance
criteria, and exclusions. This document preserves the ordered,
decision-complete implementation sequence for the follow-up work.
## Background
@@ -18,19 +26,18 @@ output provenance, support `--recompute-step`, and use generated NPC output at
operation time in the D&D spell and combat consumers. Current documentation and
maintained examples describe that model.
The remaining work is narrower:
The completed follow-up addressed these narrower gaps:
- selective recomputation currently treats an unselected predecessor as
reusable only when all of its extract, merge, and normalize stage checkpoints
are reusable, although the architectural dependency is its accepted
normalized artifact;
- the failed stage decision is not always recorded before a required-predecessor
- selective recomputation now hydrates an unselected predecessor from its
accepted normalized artifact without requiring its extract and merge state;
- the failed checkpoint decision is recorded before a required-predecessor
error returns;
- checkpoint reason codes are inferred from human-readable prose instead of
being assigned explicitly;
- the runner's lane and checkpoint orchestration has become too concentrated in
large functions; and
- the roadmap-required CLI and resumed-producer acceptance coverage is absent.
- checkpoint reason codes are assigned explicitly rather than inferred from
human-readable prose;
- runner lane and checkpoint orchestration has explicit responsibility seams;
and
- CLI and resumed-producer acceptance coverage exercises the complete recovery
contract.
No new product feature is introduced by this plan. Preserve configuration file
version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do

View File

@@ -1,22 +1,19 @@
# Proposed Scope: Ordered Pipeline Steps
# Scope: Ordered Pipeline Steps
## Status
Proposed as the next implementation scope. This document refines the
near-term priority in [Future Work](future.md) into a bounded feature target. It
defines desired behavior and scope, not a file-by-file implementation plan.
Implemented. This document preserves the bounded feature policy, architecture
choices, acceptance criteria, and exclusions. Current behavior belongs in the
canonical [CLI](../cli.md), [Configuration](../config.md),
[Operations](../operations.md), and [internal pipeline](../internal/pipeline.md)
documentation rather than in this roadmap.
## Recommendation
## Policy Recommendation
Implement ordered pipeline steps, generated artifact references, and
dependency-aware checkpoint reuse as one coherent platform capability. Prove
the capability with the D&D workflow in which accepted normalized NPC output
is produced first and then supplied to spell extraction, combat-turn
extraction, and combat-turn normalization.
This should precede item extraction. It establishes the artifact-handoff and
dependency semantics that the planned item lane will also consume, without
combining two substantial features in one implementation effort.
Treat ordered pipeline steps, generated artifact references, and
dependency-aware checkpoint reuse as one coherent platform capability. The D&D
proving workflow produces accepted normalized NPC output first and then supplies
it to spell extraction, combat-turn extraction, and combat-turn normalization.
## Intended Outcome
@@ -185,12 +182,15 @@ digest. The pipeline digest protects topology; stage dependency fingerprints
protect the exact upstream artifact consumed. The runner must never combine a
new or changed producer with stale dependent output.
Compatible producer checkpoints may be decoded through the registered codec
and handed to later steps without rerunning the producer. A missing, rejected,
corrupt, incompatible, or changed producer invalidates every transitive
dependent checkpoint. Independent work remains reusable.
Ordinary resume may progressively decode compatible producer and consumer stage
checkpoints through the registered codec. Selective recomputation may hydrate a
required unselected producer directly from its accepted normalized artifact;
its extract and merge state are not prerequisites. Missing, rejected, corrupt,
incompatible, or changed accepted state stops the run before dependent
execution rather than implicitly rerunning the producer. Independent work
remains reusable.
Add one operator control, `--recompute-step <step-id>`, with these semantics:
The operator control `--recompute-step <step-id>` has these semantics:
- it requires checkpoint recording and `--resume`;
- the selected step and all transitive dependents execute rather than reuse
@@ -239,10 +239,10 @@ Spell and combat-turn lanes may execute concurrently after the handoff. NPC
content may ground names and identities but cannot establish a spell cast or
combat event; source units remain the only event evidence.
The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be
replaced or supplemented by one ordered-pipeline example. Existing module keys,
artifact contracts, reference slot names, prompt IDs, and D&D evidence policy
remain unchanged.
The maintained workflow uses one ordered-pipeline example instead of a manual
two-run NPC-to-spell or NPC-to-combat handoff. Existing module keys, artifact
contracts, reference slot names, prompt IDs, and D&D evidence policy remain
unchanged.
## Included Work
@@ -259,9 +259,8 @@ remain unchanged.
- D&D NPC-first production composition for spell and combat-turn consumers.
- Refactoring the affected D&D consumers so generated NPC references are
available at operation time while retaining early static construction.
- Maintained examples and updates to current architecture, configuration, CLI,
operations, internal, integration, and testing documentation when behavior
lands.
- Maintained examples and current architecture, configuration, CLI, operations,
internal, integration, and testing documentation.
- An ADR recording the bounded ordered-step extension to the fixed pipeline
architecture and its explicit rejection of a general DAG.