Reconcile ordered pipeline documentation
This commit is contained in:
@@ -38,7 +38,8 @@ Flags:
|
||||
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled,
|
||||
force the named ordered step and every transitive dependent lane to execute.
|
||||
Compatible required predecessors and unrelated lanes remain reusable. The
|
||||
value must identify an explicit step; it cannot be combined with `--only`.
|
||||
value may identify an explicit step or the implicit single-step ID `default`;
|
||||
it cannot be combined with `--only`.
|
||||
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
|
||||
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
|
||||
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
|
||||
@@ -176,6 +177,8 @@ Checkpoint decisions use these categories: `reused`, `executed`,
|
||||
detail identify the decision without exposing reference content, local paths,
|
||||
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
|
||||
unknown step IDs, empty values, and combinations with `--only` are rejected.
|
||||
The operator meanings of checkpoint reason codes are maintained in
|
||||
[Operations](operations.md#resume-and-selective-recompute).
|
||||
|
||||
Use `--debug` to retain the redacted summary and trace bundle for one run. The
|
||||
bundle is allocated before pipeline resolution; once allocated, its path is
|
||||
|
||||
@@ -191,6 +191,13 @@ the runner returns.
|
||||
|
||||
## Execution Flow
|
||||
|
||||
The pipeline-wide coordinator owns the ordered step loop, generated-reference
|
||||
sets at each barrier, and deterministic merging of step outcomes. For one step,
|
||||
the lane engine initializes checkpoint state in lane order, dispatches bounded
|
||||
extract work, advances terminal lanes through serial merge and normalize work,
|
||||
selects failures by stable pipeline scope, and merges lane-local outcomes back
|
||||
in resolved order. Completion timing never becomes public ordering.
|
||||
|
||||
The runner:
|
||||
|
||||
1. validates its prepared input;
|
||||
@@ -306,8 +313,8 @@ implementations when collaborators are absent. Each checkpointed workflow
|
||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
||||
are consulted in workflow order and accepted payloads are cloned before
|
||||
entering the normal handoff path. Typed extract, merge, and normalize
|
||||
checkpoints store codec bytes with artifact kind, schema ID and version, exact
|
||||
schema digest, and media type. Reuse compares that identity with the prepared
|
||||
checkpoints store codec bytes with artifact kind, schema ID, name, version and
|
||||
exact digest, and media type. Reuse compares that identity with the prepared
|
||||
codec and decodes through the codec; missing identity, mismatches, corrupt
|
||||
bytes, and decode failures become explicit reuse misses and execute the lane
|
||||
normally. Dependency fingerprints and debug content digests use the same stable
|
||||
@@ -339,7 +346,8 @@ hydration and execute normally. A missing, rejected, corrupt, incompatible, or
|
||||
changed producer blocks its dependent while leaving independent work eligible
|
||||
for reuse. The runner records bounded decision
|
||||
categories: `reused`, `executed`, `forced_recompute`, and
|
||||
`dependency_invalidated`.
|
||||
`dependency_invalidated`. Operator meanings for the stable reason codes belong
|
||||
to [Operations](../operations.md#resume-and-selective-recompute).
|
||||
|
||||
The CLI includes prepared-component fingerprints in the run-wide checkpoint
|
||||
identity alongside resolved configuration, raw input, reference provenance,
|
||||
@@ -350,9 +358,9 @@ fingerprint intentionally produces a cold cache miss. Existing checkpoint
|
||||
schemas and paths remain unchanged.
|
||||
|
||||
The CLI's `--recompute-step` policy forces the selected step and all transitive
|
||||
dependents, but requires reusable checkpoints for all selected producers that
|
||||
precede it. It changes loader decisions only; it does not alter persistent
|
||||
checkpoint identity.
|
||||
dependents, but requires accepted normalized artifacts for every unselected
|
||||
producer on which that closure depends. It changes execution policy only; it
|
||||
does not alter persistent checkpoint identity.
|
||||
|
||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
||||
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
|
||||
@@ -418,7 +426,9 @@ stage, resolved lane, and source chunk rather than completion time.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
materialization.
|
||||
- `internal/cli/run_contract_test.go`: production run transitions, retries,
|
||||
rejections, warnings, debug hooks, and manifests.
|
||||
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
|
||||
selective recomputation and accepted-producer recovery.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration-resolution smoke coverage.
|
||||
- `internal/cli/example_contract_test.go`: maintained example resolution and
|
||||
|
||||
@@ -35,13 +35,13 @@ they do not describe a current public state surface.
|
||||
Ordered-step lane checkpoints include the step identity in their storage scope.
|
||||
When a later lane consumes a generated artifact, its dependency fingerprints
|
||||
include the producer's artifact kind, complete schema identity, media type,
|
||||
canonical content digest, and size. A producer checkpoint may be decoded through
|
||||
the registered codec and handed off without rerunning it. Missing, rejected,
|
||||
corrupt, incompatible, or changed producer state produces a bounded
|
||||
`dependency_invalidated` decision for every transitive dependent lane; it does
|
||||
not permit stale downstream reuse. The CLI's selective recomputation policy
|
||||
records `forced_recompute` for the selected step and its dependents while
|
||||
requiring compatible predecessor checkpoints.
|
||||
canonical content digest, and size. Ordinary resume compares those fingerprints
|
||||
when progressively loading consumer stage checkpoints, so changed producer
|
||||
content produces `dependency_invalidated` rather than stale downstream reuse.
|
||||
Selective recomputation instead requires each unselected producer's accepted
|
||||
normalized artifact; invalid accepted state records its specific bounded reason
|
||||
and stops before the dependent. The selected step and its transitive dependents
|
||||
record `forced_recompute`.
|
||||
|
||||
Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
||||
and may execute later lane stages after an earlier cache miss. Selective
|
||||
@@ -63,7 +63,8 @@ detail passes through one UTF-8-safe bounded sanitizer and contains only
|
||||
allowlisted diagnostic context, never payloads, references, credentials,
|
||||
environment values, or physical paths. Typed categories and codes remain intact
|
||||
through pipeline events and become strings only in manifest and debug-summary
|
||||
JSON.
|
||||
JSON. [Operations](../operations.md#resume-and-selective-recompute) is the
|
||||
canonical operator-facing reason-code reference.
|
||||
|
||||
`internal/core/fileio` provides confined atomic file writes used by state
|
||||
collaborators. The chunk-plan store retains its stronger entry validation.
|
||||
@@ -98,6 +99,8 @@ separately and never replace the command's primary error.
|
||||
terminalization, and output/report boundaries.
|
||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||
permissions, cleanup, and redaction.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
||||
filesystem recovery, deterministic decisions, and failed predecessor state.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration validation at the CLI boundary.
|
||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
||||
|
||||
@@ -161,10 +161,11 @@ for reuse.
|
||||
`--recompute-step <step-id>` requires both `--resume` and
|
||||
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
|
||||
dependents to execute, while compatible predecessors and unrelated lanes remain
|
||||
reusable. It cannot be combined with `--only`, and it does not change the
|
||||
persistent identity of otherwise identical checkpoints. Decisions are bounded
|
||||
and categorized as `reused`, `executed`, `forced_recompute`, or
|
||||
`dependency_invalidated`.
|
||||
reusable. The ID may be an explicit configured step or `default` for an
|
||||
implicit single-step pipeline. It cannot be combined with `--only`, and it does
|
||||
not change the persistent identity of otherwise identical checkpoints.
|
||||
Decisions are bounded and categorized as `reused`, `executed`,
|
||||
`forced_recompute`, or `dependency_invalidated`.
|
||||
|
||||
For an unselected producer required by a recomputed step, Notarius loads the
|
||||
accepted normalized artifact directly. Valid normalize state is sufficient even
|
||||
@@ -182,6 +183,7 @@ generated reference content. For diagnosis, first check the producer step and
|
||||
lane in the manifest, then inspect checkpoint decision categories and reason
|
||||
codes. Rerun the producer explicitly rather than copying an artifact into the
|
||||
checkpoint root.
|
||||
|
||||
The decision that caused a required-predecessor failure is retained before the
|
||||
run returns, and the CLI error identifies its step, lane, and reason code.
|
||||
|
||||
|
||||
@@ -5,18 +5,6 @@ configuration, operations, internal, and integration docs. This roadmap records
|
||||
future work only. Items are ordered roughly by current value and specificity,
|
||||
not as committed release dates.
|
||||
|
||||
## Proposed Next Scope: Ordered Pipeline Steps
|
||||
|
||||
Add ordered groups of artifact lanes, canonical generated-artifact references,
|
||||
and dependency-aware checkpoint reuse without introducing a general DAG. The
|
||||
first production workflow runs normalized D&D NPC extraction before spell and
|
||||
combat-turn extraction and supplies that NPC artifact to their declared
|
||||
reference slots.
|
||||
|
||||
The bounded target, compatibility and architecture decisions, exclusions, and
|
||||
acceptance criteria are defined in
|
||||
[Proposed Scope: Ordered Pipeline Steps](ordered-pipeline-steps.md).
|
||||
|
||||
## Near-Term D&D Pipeline
|
||||
|
||||
### Evaluate Spell Extraction And Normalization
|
||||
|
||||
@@ -1,11 +1,19 @@
|
||||
# Implementation Plan: Ordered Pipeline Follow-Up
|
||||
|
||||
## Status
|
||||
|
||||
Completed on 2026-07-22. This document retains the implementation sequence for
|
||||
historical context; current contracts are maintained in the canonical CLI,
|
||||
configuration, operations, integration, and internal documentation linked from
|
||||
the [development guide](../development.md).
|
||||
|
||||
## Purpose
|
||||
|
||||
Close the remaining correctness, observability, test, and maintainability gaps
|
||||
in the implemented [Ordered Pipeline Steps](ordered-pipeline-steps.md) feature.
|
||||
That feature roadmap remains the authority for product intent, policy choices,
|
||||
acceptance criteria, and exclusions. This document is the ordered,
|
||||
Record the work that closed the correctness, observability, test, and
|
||||
maintainability gaps in the implemented
|
||||
[Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. That feature
|
||||
roadmap remains the authority for product intent, policy choices, acceptance
|
||||
criteria, and exclusions. This document preserves the ordered,
|
||||
decision-complete implementation sequence for the follow-up work.
|
||||
|
||||
## Background
|
||||
@@ -18,19 +26,18 @@ output provenance, support `--recompute-step`, and use generated NPC output at
|
||||
operation time in the D&D spell and combat consumers. Current documentation and
|
||||
maintained examples describe that model.
|
||||
|
||||
The remaining work is narrower:
|
||||
The completed follow-up addressed these narrower gaps:
|
||||
|
||||
- selective recomputation currently treats an unselected predecessor as
|
||||
reusable only when all of its extract, merge, and normalize stage checkpoints
|
||||
are reusable, although the architectural dependency is its accepted
|
||||
normalized artifact;
|
||||
- the failed stage decision is not always recorded before a required-predecessor
|
||||
- selective recomputation now hydrates an unselected predecessor from its
|
||||
accepted normalized artifact without requiring its extract and merge state;
|
||||
- the failed checkpoint decision is recorded before a required-predecessor
|
||||
error returns;
|
||||
- checkpoint reason codes are inferred from human-readable prose instead of
|
||||
being assigned explicitly;
|
||||
- the runner's lane and checkpoint orchestration has become too concentrated in
|
||||
large functions; and
|
||||
- the roadmap-required CLI and resumed-producer acceptance coverage is absent.
|
||||
- checkpoint reason codes are assigned explicitly rather than inferred from
|
||||
human-readable prose;
|
||||
- runner lane and checkpoint orchestration has explicit responsibility seams;
|
||||
and
|
||||
- CLI and resumed-producer acceptance coverage exercises the complete recovery
|
||||
contract.
|
||||
|
||||
No new product feature is introduced by this plan. Preserve configuration file
|
||||
version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do
|
||||
|
||||
@@ -1,22 +1,19 @@
|
||||
# Proposed Scope: Ordered Pipeline Steps
|
||||
# Scope: Ordered Pipeline Steps
|
||||
|
||||
## Status
|
||||
|
||||
Proposed as the next implementation scope. This document refines the
|
||||
near-term priority in [Future Work](future.md) into a bounded feature target. It
|
||||
defines desired behavior and scope, not a file-by-file implementation plan.
|
||||
Implemented. This document preserves the bounded feature policy, architecture
|
||||
choices, acceptance criteria, and exclusions. Current behavior belongs in the
|
||||
canonical [CLI](../cli.md), [Configuration](../config.md),
|
||||
[Operations](../operations.md), and [internal pipeline](../internal/pipeline.md)
|
||||
documentation rather than in this roadmap.
|
||||
|
||||
## Recommendation
|
||||
## Policy Recommendation
|
||||
|
||||
Implement ordered pipeline steps, generated artifact references, and
|
||||
dependency-aware checkpoint reuse as one coherent platform capability. Prove
|
||||
the capability with the D&D workflow in which accepted normalized NPC output
|
||||
is produced first and then supplied to spell extraction, combat-turn
|
||||
extraction, and combat-turn normalization.
|
||||
|
||||
This should precede item extraction. It establishes the artifact-handoff and
|
||||
dependency semantics that the planned item lane will also consume, without
|
||||
combining two substantial features in one implementation effort.
|
||||
Treat ordered pipeline steps, generated artifact references, and
|
||||
dependency-aware checkpoint reuse as one coherent platform capability. The D&D
|
||||
proving workflow produces accepted normalized NPC output first and then supplies
|
||||
it to spell extraction, combat-turn extraction, and combat-turn normalization.
|
||||
|
||||
## Intended Outcome
|
||||
|
||||
@@ -185,12 +182,15 @@ digest. The pipeline digest protects topology; stage dependency fingerprints
|
||||
protect the exact upstream artifact consumed. The runner must never combine a
|
||||
new or changed producer with stale dependent output.
|
||||
|
||||
Compatible producer checkpoints may be decoded through the registered codec
|
||||
and handed to later steps without rerunning the producer. A missing, rejected,
|
||||
corrupt, incompatible, or changed producer invalidates every transitive
|
||||
dependent checkpoint. Independent work remains reusable.
|
||||
Ordinary resume may progressively decode compatible producer and consumer stage
|
||||
checkpoints through the registered codec. Selective recomputation may hydrate a
|
||||
required unselected producer directly from its accepted normalized artifact;
|
||||
its extract and merge state are not prerequisites. Missing, rejected, corrupt,
|
||||
incompatible, or changed accepted state stops the run before dependent
|
||||
execution rather than implicitly rerunning the producer. Independent work
|
||||
remains reusable.
|
||||
|
||||
Add one operator control, `--recompute-step <step-id>`, with these semantics:
|
||||
The operator control `--recompute-step <step-id>` has these semantics:
|
||||
|
||||
- it requires checkpoint recording and `--resume`;
|
||||
- the selected step and all transitive dependents execute rather than reuse
|
||||
@@ -239,10 +239,10 @@ Spell and combat-turn lanes may execute concurrently after the handoff. NPC
|
||||
content may ground names and identities but cannot establish a spell cast or
|
||||
combat event; source units remain the only event evidence.
|
||||
|
||||
The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be
|
||||
replaced or supplemented by one ordered-pipeline example. Existing module keys,
|
||||
artifact contracts, reference slot names, prompt IDs, and D&D evidence policy
|
||||
remain unchanged.
|
||||
The maintained workflow uses one ordered-pipeline example instead of a manual
|
||||
two-run NPC-to-spell or NPC-to-combat handoff. Existing module keys, artifact
|
||||
contracts, reference slot names, prompt IDs, and D&D evidence policy remain
|
||||
unchanged.
|
||||
|
||||
## Included Work
|
||||
|
||||
@@ -259,9 +259,8 @@ remain unchanged.
|
||||
- D&D NPC-first production composition for spell and combat-turn consumers.
|
||||
- Refactoring the affected D&D consumers so generated NPC references are
|
||||
available at operation time while retaining early static construction.
|
||||
- Maintained examples and updates to current architecture, configuration, CLI,
|
||||
operations, internal, integration, and testing documentation when behavior
|
||||
lands.
|
||||
- Maintained examples and current architecture, configuration, CLI, operations,
|
||||
internal, integration, and testing documentation.
|
||||
- An ADR recording the bounded ordered-step extension to the fixed pipeline
|
||||
architecture and its explicit rejection of a general DAG.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user