Reconcile ordered pipeline documentation
This commit is contained in:
@@ -191,6 +191,13 @@ the runner returns.
|
||||
|
||||
## Execution Flow
|
||||
|
||||
The pipeline-wide coordinator owns the ordered step loop, generated-reference
|
||||
sets at each barrier, and deterministic merging of step outcomes. For one step,
|
||||
the lane engine initializes checkpoint state in lane order, dispatches bounded
|
||||
extract work, advances terminal lanes through serial merge and normalize work,
|
||||
selects failures by stable pipeline scope, and merges lane-local outcomes back
|
||||
in resolved order. Completion timing never becomes public ordering.
|
||||
|
||||
The runner:
|
||||
|
||||
1. validates its prepared input;
|
||||
@@ -306,8 +313,8 @@ implementations when collaborators are absent. Each checkpointed workflow
|
||||
boundary records a running, succeeded, or failed transition. Reuse decisions
|
||||
are consulted in workflow order and accepted payloads are cloned before
|
||||
entering the normal handoff path. Typed extract, merge, and normalize
|
||||
checkpoints store codec bytes with artifact kind, schema ID and version, exact
|
||||
schema digest, and media type. Reuse compares that identity with the prepared
|
||||
checkpoints store codec bytes with artifact kind, schema ID, name, version and
|
||||
exact digest, and media type. Reuse compares that identity with the prepared
|
||||
codec and decodes through the codec; missing identity, mismatches, corrupt
|
||||
bytes, and decode failures become explicit reuse misses and execute the lane
|
||||
normally. Dependency fingerprints and debug content digests use the same stable
|
||||
@@ -339,7 +346,8 @@ hydration and execute normally. A missing, rejected, corrupt, incompatible, or
|
||||
changed producer blocks its dependent while leaving independent work eligible
|
||||
for reuse. The runner records bounded decision
|
||||
categories: `reused`, `executed`, `forced_recompute`, and
|
||||
`dependency_invalidated`.
|
||||
`dependency_invalidated`. Operator meanings for the stable reason codes belong
|
||||
to [Operations](../operations.md#resume-and-selective-recompute).
|
||||
|
||||
The CLI includes prepared-component fingerprints in the run-wide checkpoint
|
||||
identity alongside resolved configuration, raw input, reference provenance,
|
||||
@@ -350,9 +358,9 @@ fingerprint intentionally produces a cold cache miss. Existing checkpoint
|
||||
schemas and paths remain unchanged.
|
||||
|
||||
The CLI's `--recompute-step` policy forces the selected step and all transitive
|
||||
dependents, but requires reusable checkpoints for all selected producers that
|
||||
precede it. It changes loader decisions only; it does not alter persistent
|
||||
checkpoint identity.
|
||||
dependents, but requires accepted normalized artifacts for every unselected
|
||||
producer on which that closure depends. It changes execution policy only; it
|
||||
does not alter persistent checkpoint identity.
|
||||
|
||||
Debug instrumentation wraps run, stage, attempt, validator, and structured LLM
|
||||
boundaries. Every executed chunk, extract, merge, and normalize attempt writes
|
||||
@@ -418,7 +426,9 @@ stage, resolved lane, and source chunk rather than completion time.
|
||||
- `internal/framework/pipeline/references_test.go`: target resolution and
|
||||
materialization.
|
||||
- `internal/cli/run_contract_test.go`: production run transitions, retries,
|
||||
rejections, warnings, debug hooks, and manifests.
|
||||
rejections, warnings, CLI recomputation controls, debug hooks, and manifests.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: filesystem-backed
|
||||
selective recomputation and accepted-producer recovery.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration-resolution smoke coverage.
|
||||
- `internal/cli/example_contract_test.go`: maintained example resolution and
|
||||
|
||||
@@ -35,13 +35,13 @@ they do not describe a current public state surface.
|
||||
Ordered-step lane checkpoints include the step identity in their storage scope.
|
||||
When a later lane consumes a generated artifact, its dependency fingerprints
|
||||
include the producer's artifact kind, complete schema identity, media type,
|
||||
canonical content digest, and size. A producer checkpoint may be decoded through
|
||||
the registered codec and handed off without rerunning it. Missing, rejected,
|
||||
corrupt, incompatible, or changed producer state produces a bounded
|
||||
`dependency_invalidated` decision for every transitive dependent lane; it does
|
||||
not permit stale downstream reuse. The CLI's selective recomputation policy
|
||||
records `forced_recompute` for the selected step and its dependents while
|
||||
requiring compatible predecessor checkpoints.
|
||||
canonical content digest, and size. Ordinary resume compares those fingerprints
|
||||
when progressively loading consumer stage checkpoints, so changed producer
|
||||
content produces `dependency_invalidated` rather than stale downstream reuse.
|
||||
Selective recomputation instead requires each unselected producer's accepted
|
||||
normalized artifact; invalid accepted state records its specific bounded reason
|
||||
and stops before the dependent. The selected step and its transitive dependents
|
||||
record `forced_recompute`.
|
||||
|
||||
Ordinary resume loads extract, merge, and normalize checkpoints progressively
|
||||
and may execute later lane stages after an earlier cache miss. Selective
|
||||
@@ -63,7 +63,8 @@ detail passes through one UTF-8-safe bounded sanitizer and contains only
|
||||
allowlisted diagnostic context, never payloads, references, credentials,
|
||||
environment values, or physical paths. Typed categories and codes remain intact
|
||||
through pipeline events and become strings only in manifest and debug-summary
|
||||
JSON.
|
||||
JSON. [Operations](../operations.md#resume-and-selective-recompute) is the
|
||||
canonical operator-facing reason-code reference.
|
||||
|
||||
`internal/core/fileio` provides confined atomic file writes used by state
|
||||
collaborators. The chunk-plan store retains its stronger entry validation.
|
||||
@@ -98,6 +99,8 @@ separately and never replace the command's primary error.
|
||||
terminalization, and output/report boundaries.
|
||||
- `internal/cli/state_hardening_test.go`: independent roots, reuse, failures,
|
||||
permissions, cleanup, and redaction.
|
||||
- `internal/cli/recompute_execution_contract_test.go`: selective recomputation,
|
||||
filesystem recovery, deterministic decisions, and failed predecessor state.
|
||||
- `internal/cli/production_contract_test.go`: production composition and
|
||||
configuration validation at the CLI boundary.
|
||||
- `internal/cli/example_contract_test.go`: maintained example ownership.
|
||||
|
||||
Reference in New Issue
Block a user