Reconcile ordered pipeline documentation
This commit is contained in:
@@ -38,7 +38,8 @@ Flags:
|
|||||||
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled,
|
- `--recompute-step step-id`: with `--resume` and checkpoint recording enabled,
|
||||||
force the named ordered step and every transitive dependent lane to execute.
|
force the named ordered step and every transitive dependent lane to execute.
|
||||||
Compatible required predecessors and unrelated lanes remain reusable. The
|
Compatible required predecessors and unrelated lanes remain reusable. The
|
||||||
value must identify an explicit step; it cannot be combined with `--only`.
|
value may identify an explicit step or the implicit single-step ID `default`;
|
||||||
|
it cannot be combined with `--only`.
|
||||||
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
|
- `--chunk_cache auto|bypass|refresh`: select chunk-plan reuse for this
|
||||||
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
|
invocation. `auto` reuses a valid plan by canonical source digest, `bypass`
|
||||||
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
|
performs no plan-cache I/O, and `refresh` regenerates and replaces a valid
|
||||||
@@ -176,6 +177,8 @@ Checkpoint decisions use these categories: `reused`, `executed`,
|
|||||||
detail identify the decision without exposing reference content, local paths,
|
detail identify the decision without exposing reference content, local paths,
|
||||||
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
|
or secrets. `--recompute-step` requires checkpoint recording and `--resume`;
|
||||||
unknown step IDs, empty values, and combinations with `--only` are rejected.
|
unknown step IDs, empty values, and combinations with `--only` are rejected.
|
||||||
|
The operator meanings of checkpoint reason codes are maintained in
|
||||||
|
[Operations](operations.md#resume-and-selective-recompute).
|
||||||
|
|
||||||
Use `--debug` to retain the redacted summary and trace bundle for one run. The
|
Use `--debug` to retain the redacted summary and trace bundle for one run. The
|
||||||
bundle is allocated before pipeline resolution; once allocated, its path is
|
bundle is allocated before pipeline resolution; once allocated, its path is
|
||||||
|
|||||||
@@ -191,6 +191,13 @@ the runner returns.
|
|||||||
|
|
||||||
## Execution Flow
|
## Execution Flow
|
||||||
|
|
||||||
|
The pipeline-wide coordinator owns the ordered step loop, generated-reference
|
||||||
|
sets at each barrier, and deterministic merging of step outcomes. For one step,
|
||||||
|
the lane engine initializes checkpoint state in lane order, dispatches bounded
|
||||||
|
extract work, advances terminal lanes through serial merge and normalize work,
|
||||||
|
selects failures by stable pipeline scope, and merges lane-local outcomes back
|
||||||
|
in resolved order. Completion timing never becomes public ordering.
|
||||||
|
|
||||||
The runner:
|
The runner:
|
||||||
|
|
||||||
1. validates its prepared input;
|
1. validates its prepared input;
|
||||||
@@ -306,8 +313,8 @@ implementations when collaborators are absent. Each checkpointed workflow
|
|||||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
boundary records a running, succeeded, or failed transition. Reuse decisions
|
||||||
are consulted in workflow order and accepted payloads are cloned before
|
are consulted in workflow order and accepted payloads are cloned before
|
||||||
entering the normal handoff path. Typed extract, merge, and normalize
|
entering the normal handoff path. Typed extract, merge, and normalize
|
||||||
checkpoints store codec bytes with artifact kind, schema ID and version, exact
|
checkpoints store codec bytes with artifact kind, schema ID, name, version and
|
||||||
schema digest, and media type. Reuse compares that identity with the prepared
|
exact digest, and media type. Reuse compares that identity with the prepared
|
||||||
codec and decodes through the codec; missing identity, mismatches, corrupt
|
codec and decodes through the codec; missing identity, mismatches, corrupt
|
||||||
bytes, and decode failures become explicit reuse misses and execute the lane
|
bytes, and decode failures become explicit reuse misses and execute the lane
|
||||||
normally. Dependency fingerprints and debug content digests use the same stable
|
normally. Dependency fingerprints and debug content digests use the same stable
|
||||||
@@ -339,7 +346,8 @@ hydration and execute normally. A missing, rejected, corrupt, incompatible, or
|
|||||||
changed producer blocks its dependent while leaving independent work eligible
|
changed producer blocks its dependent while leaving independent work eligible
|
||||||
for reuse. The runner records bounded decision
|
for reuse. The runner records bounded decision
|
||||||
categories: `reused`, `executed`, `forced_recompute`, and
|
categories: `reused`, `executed`, `forced_recompute`, and
|
||||||
`dependency_invalidated`.
|
`dependency_invalidated`. Operator meanings for the stable reason codes belong
|
||||||
|
to [Operations](../operations.md#resume-and-selective-recompute).
|
||||||
|
|
||||||
The CLI includes prepared-component fingerprints in the run-wide checkpoint
|
The CLI includes prepared-component fingerprints in the run-wide checkpoint
|
||||||
identity alongside resolved configuration, raw input, reference provenance,
|
identity alongside resolved configuration, raw input, reference provenance,
|
||||||
@@ -350,9 +358,9 @@ fingerprint intentionally produces a cold cache miss. Existing checkpoint
|
|||||||
schemas and paths remain unchanged.
|
schemas and paths remain unchanged.
|
||||||
|
|
||||||
The CLI's `--recompute-step` policy forces the selected step and all transitive
|
The CLI's `--recompute-step` policy forces the selected step and all transitive
|
||||||
dependents, but requires reusable checkpoints for all selected producers that
|
dependents, but requires accepted normalized artifacts for every unselected
|
||||||
precede it. It changes loader decisions only; it does not alter persistent
|
producer on which that closure depends. It changes execution policy only; it
|
||||||
checkpoint identity.
|
does not alter persistent checkpoint identity.
|
||||||
|
|
||||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
||||||
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
|
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
|
||||||
@@ -418,7 +426,9 @@ stage, resolved lane, and source chunk rather than completion time.
|
|||||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||||
materialization.
|
materialization.
|
||||||
- `internal/cli/run_contract_test.go`: production run transitions, retries,
|
- `internal/cli/run_contract_test.go`: production run transitions, retries,
|
||||||
rejections, warnings, debug hooks, and manifests.
|
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
|
||||||
|
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
|
||||||
|
selective recomputation and accepted-producer recovery.
|
||||||
- `internal/cli/production_contract_test.go`: production composition and
|
- `internal/cli/production_contract_test.go`: production composition and
|
||||||
configuration-resolution smoke coverage.
|
configuration-resolution smoke coverage.
|
||||||
- `internal/cli/example_contract_test.go`: maintained example resolution and
|
- `internal/cli/example_contract_test.go`: maintained example resolution and
|
||||||
|
|||||||
@@ -35,13 +35,13 @@ they do not describe a current public state surface.
|
|||||||
Ordered-step lane checkpoints include the step identity in their storage scope.
|
Ordered-step lane checkpoints include the step identity in their storage scope.
|
||||||
When a later lane consumes a generated artifact, its dependency fingerprints
|
When a later lane consumes a generated artifact, its dependency fingerprints
|
||||||
include the producer's artifact kind, complete schema identity, media type,
|
include the producer's artifact kind, complete schema identity, media type,
|
||||||
canonical content digest, and size. A producer checkpoint may be decoded through
|
canonical content digest, and size. Ordinary resume compares those fingerprints
|
||||||
the registered codec and handed off without rerunning it. Missing, rejected,
|
when progressively loading consumer stage checkpoints, so changed producer
|
||||||
corrupt, incompatible, or changed producer state produces a bounded
|
content produces `dependency_invalidated` rather than stale downstream reuse.
|
||||||
`dependency_invalidated` decision for every transitive dependent lane; it does
|
Selective recomputation instead requires each unselected producer's accepted
|
||||||
not permit stale downstream reuse. The CLI's selective recomputation policy
|
normalized artifact; invalid accepted state records its specific bounded reason
|
||||||
records `forced_recompute` for the selected step and its dependents while
|
and stops before the dependent. The selected step and its transitive dependents
|
||||||
requiring compatible predecessor checkpoints.
|
record `forced_recompute`.
|
||||||
|
|
||||||
Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
||||||
and may execute later lane stages after an earlier cache miss. Selective
|
and may execute later lane stages after an earlier cache miss. Selective
|
||||||
@@ -63,7 +63,8 @@ detail passes through one UTF-8-safe bounded sanitizer and contains only
|
|||||||
allowlisted diagnostic context, never payloads, references, credentials,
|
allowlisted diagnostic context, never payloads, references, credentials,
|
||||||
environment values, or physical paths. Typed categories and codes remain intact
|
environment values, or physical paths. Typed categories and codes remain intact
|
||||||
through pipeline events and become strings only in manifest and debug-summary
|
through pipeline events and become strings only in manifest and debug-summary
|
||||||
JSON.
|
JSON. [Operations](../operations.md#resume-and-selective-recompute) is the
|
||||||
|
canonical operator-facing reason-code reference.
|
||||||
|
|
||||||
`internal/core/fileio` provides confined atomic file writes used by state
|
`internal/core/fileio` provides confined atomic file writes used by state
|
||||||
collaborators. The chunk-plan store retains its stronger entry validation.
|
collaborators. The chunk-plan store retains its stronger entry validation.
|
||||||
@@ -98,6 +99,8 @@ separately and never replace the command's primary error.
|
|||||||
terminalization, and output/report boundaries.
|
terminalization, and output/report boundaries.
|
||||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||||
permissions, cleanup, and redaction.
|
permissions, cleanup, and redaction.
|
||||||
|
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
||||||
|
filesystem recovery, deterministic decisions, and failed predecessor state.
|
||||||
- `internal/cli/production_contract_test.go`: production composition and
|
- `internal/cli/production_contract_test.go`: production composition and
|
||||||
configuration validation at the CLI boundary.
|
configuration validation at the CLI boundary.
|
||||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
||||||
|
|||||||
@@ -161,10 +161,11 @@ for reuse.
|
|||||||
`--recompute-step <step-id>` requires both `--resume` and
|
`--recompute-step <step-id>` requires both `--resume` and
|
||||||
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
|
`cache.checkpoints.enabled: true`. It forces the named step and all transitive
|
||||||
dependents to execute, while compatible predecessors and unrelated lanes remain
|
dependents to execute, while compatible predecessors and unrelated lanes remain
|
||||||
reusable. It cannot be combined with `--only`, and it does not change the
|
reusable. The ID may be an explicit configured step or `default` for an
|
||||||
persistent identity of otherwise identical checkpoints. Decisions are bounded
|
implicit single-step pipeline. It cannot be combined with `--only`, and it does
|
||||||
and categorized as `reused`, `executed`, `forced_recompute`, or
|
not change the persistent identity of otherwise identical checkpoints.
|
||||||
`dependency_invalidated`.
|
Decisions are bounded and categorized as `reused`, `executed`,
|
||||||
|
`forced_recompute`, or `dependency_invalidated`.
|
||||||
|
|
||||||
For an unselected producer required by a recomputed step, Notarius loads the
|
For an unselected producer required by a recomputed step, Notarius loads the
|
||||||
accepted normalized artifact directly. Valid normalize state is sufficient even
|
accepted normalized artifact directly. Valid normalize state is sufficient even
|
||||||
@@ -182,6 +183,7 @@ generated reference content. For diagnosis, first check the producer step and
|
|||||||
lane in the manifest, then inspect checkpoint decision categories and reason
|
lane in the manifest, then inspect checkpoint decision categories and reason
|
||||||
codes. Rerun the producer explicitly rather than copying an artifact into the
|
codes. Rerun the producer explicitly rather than copying an artifact into the
|
||||||
checkpoint root.
|
checkpoint root.
|
||||||
|
|
||||||
The decision that caused a required-predecessor failure is retained before the
|
The decision that caused a required-predecessor failure is retained before the
|
||||||
run returns, and the CLI error identifies its step, lane, and reason code.
|
run returns, and the CLI error identifies its step, lane, and reason code.
|
||||||
|
|
||||||
|
|||||||
@@ -5,18 +5,6 @@ configuration, operations, internal, and integration docs. This roadmap records
|
|||||||
future work only. Items are ordered roughly by current value and specificity,
|
future work only. Items are ordered roughly by current value and specificity,
|
||||||
not as committed release dates.
|
not as committed release dates.
|
||||||
|
|
||||||
## Proposed Next Scope: Ordered Pipeline Steps
|
|
||||||
|
|
||||||
Add ordered groups of artifact lanes, canonical generated-artifact references,
|
|
||||||
and dependency-aware checkpoint reuse without introducing a general DAG. The
|
|
||||||
first production workflow runs normalized D&D NPC extraction before spell and
|
|
||||||
combat-turn extraction and supplies that NPC artifact to their declared
|
|
||||||
reference slots.
|
|
||||||
|
|
||||||
The bounded target, compatibility and architecture decisions, exclusions, and
|
|
||||||
acceptance criteria are defined in
|
|
||||||
[Proposed Scope: Ordered Pipeline Steps](ordered-pipeline-steps.md).
|
|
||||||
|
|
||||||
## Near-Term D&D Pipeline
|
## Near-Term D&D Pipeline
|
||||||
|
|
||||||
### Evaluate Spell Extraction And Normalization
|
### Evaluate Spell Extraction And Normalization
|
||||||
|
|||||||
@@ -1,11 +1,19 @@
|
|||||||
# Implementation Plan: Ordered Pipeline Follow-Up
|
# Implementation Plan: Ordered Pipeline Follow-Up
|
||||||
|
|
||||||
|
## Status
|
||||||
|
|
||||||
|
Completed on 2026-07-22. This document retains the implementation sequence for
|
||||||
|
historical context; current contracts are maintained in the canonical CLI,
|
||||||
|
configuration, operations, integration, and internal documentation linked from
|
||||||
|
the [development guide](../development.md).
|
||||||
|
|
||||||
## Purpose
|
## Purpose
|
||||||
|
|
||||||
Close the remaining correctness, observability, test, and maintainability gaps
|
Record the work that closed the correctness, observability, test, and
|
||||||
in the implemented [Ordered Pipeline Steps](ordered-pipeline-steps.md) feature.
|
maintainability gaps in the implemented
|
||||||
That feature roadmap remains the authority for product intent, policy choices,
|
[Ordered Pipeline Steps](ordered-pipeline-steps.md) feature. That feature
|
||||||
acceptance criteria, and exclusions. This document is the ordered,
|
roadmap remains the authority for product intent, policy choices, acceptance
|
||||||
|
criteria, and exclusions. This document preserves the ordered,
|
||||||
decision-complete implementation sequence for the follow-up work.
|
decision-complete implementation sequence for the follow-up work.
|
||||||
|
|
||||||
## Background
|
## Background
|
||||||
@@ -18,19 +26,18 @@ output provenance, support `--recompute-step`, and use generated NPC output at
|
|||||||
operation time in the D&D spell and combat consumers. Current documentation and
|
operation time in the D&D spell and combat consumers. Current documentation and
|
||||||
maintained examples describe that model.
|
maintained examples describe that model.
|
||||||
|
|
||||||
The remaining work is narrower:
|
The completed follow-up addressed these narrower gaps:
|
||||||
|
|
||||||
- selective recomputation currently treats an unselected predecessor as
|
- selective recomputation now hydrates an unselected predecessor from its
|
||||||
reusable only when all of its extract, merge, and normalize stage checkpoints
|
accepted normalized artifact without requiring its extract and merge state;
|
||||||
are reusable, although the architectural dependency is its accepted
|
- the failed checkpoint decision is recorded before a required-predecessor
|
||||||
normalized artifact;
|
|
||||||
- the failed stage decision is not always recorded before a required-predecessor
|
|
||||||
error returns;
|
error returns;
|
||||||
- checkpoint reason codes are inferred from human-readable prose instead of
|
- checkpoint reason codes are assigned explicitly rather than inferred from
|
||||||
being assigned explicitly;
|
human-readable prose;
|
||||||
- the runner's lane and checkpoint orchestration has become too concentrated in
|
- runner lane and checkpoint orchestration has explicit responsibility seams;
|
||||||
large functions; and
|
and
|
||||||
- the roadmap-required CLI and resumed-producer acceptance coverage is absent.
|
- CLI and resumed-producer acceptance coverage exercises the complete recovery
|
||||||
|
contract.
|
||||||
|
|
||||||
No new product feature is introduced by this plan. Preserve configuration file
|
No new product feature is introduced by this plan. Preserve configuration file
|
||||||
version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do
|
version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do
|
||||||
|
|||||||
@@ -1,22 +1,19 @@
|
|||||||
# Proposed Scope: Ordered Pipeline Steps
|
# Scope: Ordered Pipeline Steps
|
||||||
|
|
||||||
## Status
|
## Status
|
||||||
|
|
||||||
Proposed as the next implementation scope. This document refines the
|
Implemented. This document preserves the bounded feature policy, architecture
|
||||||
near-term priority in [Future Work](future.md) into a bounded feature target. It
|
choices, acceptance criteria, and exclusions. Current behavior belongs in the
|
||||||
defines desired behavior and scope, not a file-by-file implementation plan.
|
canonical [CLI](../cli.md), [Configuration](../config.md),
|
||||||
|
[Operations](../operations.md), and [internal pipeline](../internal/pipeline.md)
|
||||||
|
documentation rather than in this roadmap.
|
||||||
|
|
||||||
## Recommendation
|
## Policy Recommendation
|
||||||
|
|
||||||
Implement ordered pipeline steps, generated artifact references, and
|
Treat ordered pipeline steps, generated artifact references, and
|
||||||
dependency-aware checkpoint reuse as one coherent platform capability. Prove
|
dependency-aware checkpoint reuse as one coherent platform capability. The D&D
|
||||||
the capability with the D&D workflow in which accepted normalized NPC output
|
proving workflow produces accepted normalized NPC output first and then supplies
|
||||||
is produced first and then supplied to spell extraction, combat-turn
|
it to spell extraction, combat-turn extraction, and combat-turn normalization.
|
||||||
extraction, and combat-turn normalization.
|
|
||||||
|
|
||||||
This should precede item extraction. It establishes the artifact-handoff and
|
|
||||||
dependency semantics that the planned item lane will also consume, without
|
|
||||||
combining two substantial features in one implementation effort.
|
|
||||||
|
|
||||||
## Intended Outcome
|
## Intended Outcome
|
||||||
|
|
||||||
@@ -185,12 +182,15 @@ digest. The pipeline digest protects topology; stage dependency fingerprints
|
|||||||
protect the exact upstream artifact consumed. The runner must never combine a
|
protect the exact upstream artifact consumed. The runner must never combine a
|
||||||
new or changed producer with stale dependent output.
|
new or changed producer with stale dependent output.
|
||||||
|
|
||||||
Compatible producer checkpoints may be decoded through the registered codec
|
Ordinary resume may progressively decode compatible producer and consumer stage
|
||||||
and handed to later steps without rerunning the producer. A missing, rejected,
|
checkpoints through the registered codec. Selective recomputation may hydrate a
|
||||||
corrupt, incompatible, or changed producer invalidates every transitive
|
required unselected producer directly from its accepted normalized artifact;
|
||||||
dependent checkpoint. Independent work remains reusable.
|
its extract and merge state are not prerequisites. Missing, rejected, corrupt,
|
||||||
|
incompatible, or changed accepted state stops the run before dependent
|
||||||
|
execution rather than implicitly rerunning the producer. Independent work
|
||||||
|
remains reusable.
|
||||||
|
|
||||||
Add one operator control, `--recompute-step <step-id>`, with these semantics:
|
The operator control `--recompute-step <step-id>` has these semantics:
|
||||||
|
|
||||||
- it requires checkpoint recording and `--resume`;
|
- it requires checkpoint recording and `--resume`;
|
||||||
- the selected step and all transitive dependents execute rather than reuse
|
- the selected step and all transitive dependents execute rather than reuse
|
||||||
@@ -239,10 +239,10 @@ Spell and combat-turn lanes may execute concurrently after the handoff. NPC
|
|||||||
content may ground names and identities but cannot establish a spell cast or
|
content may ground names and identities but cannot establish a spell cast or
|
||||||
combat event; source units remain the only event evidence.
|
combat event; source units remain the only event evidence.
|
||||||
|
|
||||||
The maintained manual two-run NPC-to-spell and NPC-to-combat examples should be
|
The maintained workflow uses one ordered-pipeline example instead of a manual
|
||||||
replaced or supplemented by one ordered-pipeline example. Existing module keys,
|
two-run NPC-to-spell or NPC-to-combat handoff. Existing module keys, artifact
|
||||||
artifact contracts, reference slot names, prompt IDs, and D&D evidence policy
|
contracts, reference slot names, prompt IDs, and D&D evidence policy remain
|
||||||
remain unchanged.
|
unchanged.
|
||||||
|
|
||||||
## Included Work
|
## Included Work
|
||||||
|
|
||||||
@@ -259,9 +259,8 @@ remain unchanged.
|
|||||||
- D&D NPC-first production composition for spell and combat-turn consumers.
|
- D&D NPC-first production composition for spell and combat-turn consumers.
|
||||||
- Refactoring the affected D&D consumers so generated NPC references are
|
- Refactoring the affected D&D consumers so generated NPC references are
|
||||||
available at operation time while retaining early static construction.
|
available at operation time while retaining early static construction.
|
||||||
- Maintained examples and updates to current architecture, configuration, CLI,
|
- Maintained examples and current architecture, configuration, CLI, operations,
|
||||||
operations, internal, integration, and testing documentation when behavior
|
internal, integration, and testing documentation.
|
||||||
lands.
|
|
||||||
- An ADR recording the bounded ordered-step extension to the fixed pipeline
|
- An ADR recording the bounded ordered-step extension to the fixed pipeline
|
||||||
architecture and its explicit rejection of a general DAG.
|
architecture and its explicit rejection of a general DAG.
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user