Audit the implementation and add a roadmap with cleanup items and bugfixes
This commit is contained in:
@@ -1,500 +1,392 @@
|
||||
# Implementation Plan: Ordered Pipeline Steps
|
||||
# Implementation Plan: Ordered Pipeline Follow-Up
|
||||
|
||||
## Purpose
|
||||
|
||||
Implement the target state in [Ordered Pipeline Steps](ordered-pipeline-steps.md).
|
||||
That feature roadmap is the authority for product intent, policy choices,
|
||||
acceptance criteria, and exclusions. This document turns it into an ordered,
|
||||
decision-complete implementation sequence for a coding agent.
|
||||
Close the remaining correctness, observability, test, and maintainability gaps
|
||||
in the implemented [Ordered Pipeline Steps](ordered-pipeline-steps.md) feature.
|
||||
That feature roadmap remains the authority for product intent, policy choices,
|
||||
acceptance criteria, and exclusions. This document is the ordered,
|
||||
decision-complete implementation sequence for the follow-up work.
|
||||
|
||||
This is a platform refactor followed by one D&D proving workflow. It does not
|
||||
include item extraction, a general DAG scheduler, schema migrations for D&D
|
||||
artifacts, or any other work excluded by the feature roadmap.
|
||||
## Background
|
||||
|
||||
The original implementation is complete in broad architecture. Pipelines now
|
||||
resolve and prepare one canonical ordered-step model, execute hard barriers
|
||||
between steps, hand accepted normalized artifacts to later operations as
|
||||
canonical generated references, include generated identity in checkpoint and
|
||||
output provenance, support `--recompute-step`, and use generated NPC output at
|
||||
operation time in the D&D spell and combat consumers. Current documentation and
|
||||
maintained examples describe that model.
|
||||
|
||||
The remaining work is narrower:
|
||||
|
||||
- selective recomputation currently treats an unselected predecessor as
|
||||
reusable only when all of its extract, merge, and normalize stage checkpoints
|
||||
are reusable, although the architectural dependency is its accepted
|
||||
normalized artifact;
|
||||
- the failed stage decision is not always recorded before a required-predecessor
|
||||
error returns;
|
||||
- checkpoint reason codes are inferred from human-readable prose instead of
|
||||
being assigned explicitly;
|
||||
- the runner's lane and checkpoint orchestration has become too concentrated in
|
||||
large functions; and
|
||||
- the roadmap-required CLI and resumed-producer acceptance coverage is absent.
|
||||
|
||||
No new product feature is introduced by this plan. Preserve configuration file
|
||||
version 3 and checkpoint workspace schema `notarius.workspace.v3`; the fixes do
|
||||
not require a new persistent format.
|
||||
|
||||
## Instructions For Every Stage
|
||||
|
||||
Before changing code, read `docs/development.md` and all documents under
|
||||
`docs/policy/`. Preserve the fixed pipeline lifecycle and the two-zone data
|
||||
model described there. Use the repository's code knowledge graph for code
|
||||
discovery before falling back to text search.
|
||||
Before changing code, read `docs/development.md`, all documents under
|
||||
`docs/policy/`, and the task-specific internal documents named there. Use the
|
||||
repository's code knowledge graph for code discovery before falling back to
|
||||
text search.
|
||||
|
||||
Implement the stages in order. Each stage must leave the repository formatted,
|
||||
building, and passing the focused tests it changes. Do not maintain two
|
||||
competing internal representations merely to reduce a refactor: legacy
|
||||
top-level `artifacts` is a configuration compatibility form, while ordered
|
||||
steps are the single resolved, prepared, and runtime representation.
|
||||
building, and passing its focused tests. Preserve these invariants throughout:
|
||||
|
||||
Follow these cross-stage rules:
|
||||
- The pipeline retains one input, chunk plan, output encoder, run identity,
|
||||
checkpoint identity, failure boundary, worker budget, and provider scheduler.
|
||||
- Every selected module and validator is constructed before source parsing.
|
||||
- Steps schedule fixed extract, merge, and normalize lanes; this work must not
|
||||
introduce module-to-module calls or a general DAG scheduler.
|
||||
- Generated artifacts remain cloned operation-time context, never source
|
||||
evidence. Never emit their content, source material, credentials, or local
|
||||
paths in decisions, manifests, logs, or summaries.
|
||||
- Public ordering remains step order, lane ID, and source/chunk order as
|
||||
applicable. Refactoring must not expose completion order.
|
||||
- Tests must follow `docs/policy/testing.md`: protect observable recovery,
|
||||
orchestration, and CLI contracts without asserting private helper calls,
|
||||
exact prose, goroutine choreography, or full-document snapshots.
|
||||
- Update current-behavior documentation in the same stage that changes the
|
||||
corresponding behavior. Keep detailed implementation sequencing only here.
|
||||
|
||||
- Keep input parsing, chunk planning, output encoding, run identity, worker
|
||||
budgets, provider scheduling, and the failure boundary pipeline-wide.
|
||||
- Preserve the lane lifecycle and existing typed module interfaces. Steps
|
||||
schedule lanes; they do not permit modules to call other modules.
|
||||
- Construct every selected module and validator before source parsing. Supply
|
||||
generated bytes only through cloned operation-request `References`.
|
||||
- Resolve and report deterministic order as step order, then lane ID, then
|
||||
source/chunk order. Never expose completion order.
|
||||
- Treat generated references as context, not source evidence. Never put their
|
||||
content in manifests, checkpoint decisions, logs, or debug summaries.
|
||||
- Add narrow, behavior-oriented, offline tests at the stable owner of each new
|
||||
invariant. Do not add scheduler choreography, exact goroutine counts, full
|
||||
manifest snapshots, exact error-string snapshots, or live-provider tests.
|
||||
- Keep configuration file version 3. The new fields are a backward-compatible
|
||||
extension; checkpoint wire state receives its own explicit compatibility
|
||||
change in Stage 4.
|
||||
- Update current-behavior documentation only in Stage 6, when the complete
|
||||
behavior exists. The ADR may be added in Stage 1 because it records the
|
||||
architecture decision rather than claiming released behavior.
|
||||
## Target Checkpoint Semantics
|
||||
|
||||
## Canonical Internal Model
|
||||
Use these definitions consistently in all stages:
|
||||
|
||||
Use the following model throughout the implementation. Exact unexported helper
|
||||
names may follow local conventions, but their responsibilities and invariants
|
||||
are fixed here.
|
||||
- A **stage checkpoint** is internal resumable state for extract, merge, or
|
||||
normalize. Ordinary resume may continue to reuse this state progressively.
|
||||
- An **accepted lane artifact** is the one successful normalized artifact for a
|
||||
`(step ID, lane ID, normalizer module)` under the current checkpoint identity.
|
||||
It is the dependency exposed to a later step.
|
||||
- An unselected required predecessor satisfies selective recomputation when its
|
||||
accepted lane artifact can be validated and hydrated. Its extract and merge
|
||||
stage checkpoints are not prerequisites for that handoff.
|
||||
- A selected lane and its transitive dependents execute. They must never use
|
||||
accepted-artifact hydration to bypass forced execution.
|
||||
- If a required predecessor's accepted lane artifact is missing, rejected,
|
||||
corrupt, non-canonical, or incompatible, fail the run before any dependent
|
||||
lane starts. Do not implicitly rerun that predecessor.
|
||||
- Ordinary resume without `--recompute-step` keeps its existing progressive
|
||||
stage-reuse and cold-miss behavior.
|
||||
|
||||
### Configuration and profile types
|
||||
|
||||
- Add `PipelineStepProfile` with `ID`, `Artifacts`, and `References` fields.
|
||||
- Add `Steps []PipelineStepProfile` to `PipelineProfile`; retain top-level
|
||||
`Artifacts` only as the legacy input form.
|
||||
- Replace reference-map values in profile and file-config types with a
|
||||
discriminated `ReferenceSource`. It has exactly one of:
|
||||
- an external path, represented by the existing scalar YAML form; or
|
||||
- `ArtifactReference{Step, Lane}`, represented by
|
||||
`artifact: {step: ..., lane: ...}`.
|
||||
- Keep CLI `--reference` values external-path overrides. They do not create or
|
||||
replace generated bindings. `--reference-unbind` may remove an effective
|
||||
external binding, but must not silently remove a generated dependency.
|
||||
- Step-level `References` use the same source type as pipeline and target-local
|
||||
references. Pipeline-level generated references are invalid because there is
|
||||
no well-defined consumer step and they could imply a forward dependency.
|
||||
|
||||
### Resolved and prepared types
|
||||
|
||||
- Make `ResolvedPipeline.Steps []ResolvedPipelineStep` the sole lane container.
|
||||
Each resolved step has a stable `ID`, ordered `ArtifactLanes`, and expanded
|
||||
generated consumer bindings. Remove the flat `ArtifactLanes` field after all
|
||||
callers migrate; provide read-only iteration/lookup helpers where callers
|
||||
need all lanes.
|
||||
- Resolve legacy top-level `artifacts` as one step named `default`. Do not add a
|
||||
synthetic step back into effective user configuration output.
|
||||
- Keep lane IDs globally unique. A producer and consumer are identified by
|
||||
pipeline ID, step ID, and lane ID; module targets additionally include stage,
|
||||
module key, and slot name.
|
||||
- Represent external and generated bindings as distinct resolved variants.
|
||||
Never encode a producer selector into a path string.
|
||||
- Mirror resolved steps in `PreparedPipeline`. A prepared step contains its
|
||||
already-constructed lanes and static external reference sets. Generated
|
||||
content is held only in run-local state and overlaid onto request references
|
||||
before a consumer step begins.
|
||||
|
||||
### Generated compatibility and handoff
|
||||
|
||||
- Extend `contracts.ReferenceSlot` with an optional cloned list of accepted
|
||||
generated `ArtifactKind` values. An empty list means external references only.
|
||||
Do not infer compatibility by decoding external bytes.
|
||||
- A generated producer supplies exactly one accepted normalized artifact per
|
||||
binding in this feature. Zero artifacts is a missing dependency; more than
|
||||
one is a cardinality error. A collection such as an NPC list is one typed
|
||||
artifact, not multiple reference items. Supporting aggregation or several
|
||||
generated items in one slot remains out of scope.
|
||||
- Use the producer kind's registered `ArtifactCodecSpec` as the canonical schema
|
||||
identity and media contract. Kind compatibility plus the single registered
|
||||
codec makes schema compatibility exact; also validate the consumer's media,
|
||||
size, and cardinality constraints.
|
||||
- Extend `contracts.ReferenceItem` with optional generated-artifact identity and
|
||||
producer provenance rather than overloading file origin fields. Include kind,
|
||||
schema ID/name/version/digest, producer pipeline/step/lane/module, content
|
||||
digest, media type, and size. Clone byte slices and nested metadata at every
|
||||
ownership boundary.
|
||||
- The runtime dependency fingerprint is the canonical tuple of producer
|
||||
identity, artifact kind, complete schema identity, media type, and content
|
||||
digest. It must be added to every extract, merge, normalize, and validator
|
||||
checkpoint dependency that receives the generated slot.
|
||||
|
||||
## Stage 1: Configuration, Contracts, And Resolution
|
||||
## Stage 1: Decompose Runner And Checkpoint Orchestration
|
||||
|
||||
### Objective
|
||||
|
||||
Accept and fully validate ordered steps and structured artifact reference
|
||||
sources while preserving legacy single-step configuration. No runner behavior
|
||||
changes belong in this stage.
|
||||
Create explicit, testable ownership seams for ordered-step coordination,
|
||||
per-lane execution, and per-stage checkpoint handling without changing
|
||||
observable behavior.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Add an ADR under `docs/adr/` following the existing numbering and template.
|
||||
Record the bounded ordered-step extension, the single pipeline-wide
|
||||
input/chunk/output boundary, explicit generated bindings, barriers between
|
||||
steps, and the rejection of a general DAG.
|
||||
2. In `internal/core/config/file_config.go` and its validation/application
|
||||
helpers:
|
||||
- add file forms for ordered steps and discriminated reference sources;
|
||||
- use strict YAML decoding for the mapping form and reject unknown fields,
|
||||
empty selectors, ambiguous scalar-plus-artifact values, and non-string
|
||||
scalar paths;
|
||||
- reject pipelines containing both `artifacts` and `steps`, empty explicit
|
||||
steps, duplicate trimmed step IDs, and duplicate lane IDs across steps;
|
||||
- preserve list order for steps and validator chains and deterministic key
|
||||
ordering for lane maps; and
|
||||
- update cloning, defaults application, effective-config rendering, and
|
||||
redaction so selectors remain visible but external paths follow the current
|
||||
path redaction policy.
|
||||
3. In `internal/framework/contracts/contracts.go`, add generated artifact-kind
|
||||
compatibility to `ReferenceSlot` and update `CloneReferenceSlots` to deep
|
||||
copy it. Keep existing external-slot behavior unchanged.
|
||||
4. In `internal/framework/pipeline/profile.go` and reference-resolution helpers:
|
||||
- introduce the canonical step and reference-source types described above;
|
||||
- normalize legacy `artifacts` to resolved step `default`;
|
||||
- resolve all step lanes and module/validator contracts before resolving
|
||||
generated bindings, so producer codec metadata and consumer slots are
|
||||
available for compatibility checks;
|
||||
- expand a step-scoped binding to every selected target in that step that
|
||||
declares the slot, for both external and generated sources; do not treat
|
||||
targets without that slot as errors;
|
||||
- retain current pipeline, target-local, and CLI precedence for external
|
||||
bindings, inserting step-local external bindings between pipeline defaults
|
||||
and target-local bindings;
|
||||
- reject every generated/external collision on an effective target slot and
|
||||
every step-scoped/target-local generated collision;
|
||||
- reject missing producers, same-step or forward producers, undeclared
|
||||
target-local slots, non-normalized producer lanes, incompatible kinds,
|
||||
schemas or media types, and selectors made ambiguous by duplicate IDs; and
|
||||
- include step IDs/order, lane membership, producer selectors, expanded
|
||||
consumers, and module/validator policy in canonical cloning and the
|
||||
pipeline digest.
|
||||
5. Migrate catalog, config validation, `--only` lane selection, checkpoint
|
||||
identity construction, reference provenance discovery, and debug/effective
|
||||
configuration code to iterate the resolved step model. Reject `--only` when
|
||||
the source profile has explicit steps; retain its current behavior for the
|
||||
implicit `default` step.
|
||||
1. Keep `Runner.Run` as the pipeline-wide coordinator. Extract a small
|
||||
step-coordination helper responsible only for iterating prepared steps,
|
||||
building generated reference sets at each barrier, invoking the lane engine,
|
||||
and merging deterministic outcomes.
|
||||
2. Split `runLanes` in `internal/framework/pipeline/runner_concurrent.go` into
|
||||
helpers with these responsibilities:
|
||||
- initialize lane state and resolve extract checkpoint state in lane order;
|
||||
- run the bounded extract worker/continuation engine;
|
||||
- collect terminal lane results and choose failures deterministically; and
|
||||
- merge lane-local output into step output.
|
||||
Preserve the existing bounded channels, cancellation, drain behavior,
|
||||
chunk-first dispatch, lane ordering, and step barrier.
|
||||
3. Split `continueTypedLane` in
|
||||
`internal/framework/pipeline/runner_typed.go` into stage-specific merge and
|
||||
normalize helpers. Each helper should own dependency construction, checkpoint
|
||||
loading and canonical validation, execution/retry/validation when needed,
|
||||
checkpoint recording, debug envelopes, and its typed result. Use small
|
||||
result structs rather than long parallel return lists.
|
||||
4. Centralize repeated checkpoint-decision flow in one pipeline helper that can
|
||||
apply forced-execution policy, validate canonical stored artifacts, record
|
||||
the final observable decision, and return a contextual error. Do not yet
|
||||
change categories, reason codes, or required-predecessor semantics; Stages 2
|
||||
and 3 will change those deliberately.
|
||||
5. Keep stage-specific code where the data shapes genuinely differ. Do not
|
||||
introduce reflection, a generic stage state machine, or callbacks that hide
|
||||
the fixed extract/merge/normalize lifecycle.
|
||||
|
||||
### Tests
|
||||
|
||||
- Extend `internal/core/config/*_test.go` with table-driven contracts for legacy
|
||||
shorthand, explicit step order, strict source-form parsing, mutual exclusion,
|
||||
identity collisions, cloning, effective output, and redaction.
|
||||
- Extend `internal/framework/pipeline/profile_test.go`,
|
||||
`typed_resolution_test.go`, and reference tests for deterministic expansion,
|
||||
precedence/conflicts, backward/forward rules, global lane uniqueness,
|
||||
generated-kind/media compatibility, canonical cloning, and digest changes.
|
||||
- Add a contract test showing a legacy profile resolves to `default` with an
|
||||
otherwise equivalent lane lifecycle and ordering.
|
||||
- Prefer one representative case per validation owner; do not duplicate every
|
||||
parser failure at the resolver layer.
|
||||
- Existing runner, checkpoint, barrier, ordering, cancellation, retry, debug,
|
||||
and D&D integration tests must pass without expectation changes except moves
|
||||
required by renamed private test fixtures.
|
||||
- Add no tests for helper boundaries or collaborator call counts. Add a narrow
|
||||
regression assertion only if the refactor exposes an observable behavior not
|
||||
already protected.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run focused config, contracts, and pipeline resolution tests, then `go test
|
||||
./internal/core/config ./internal/framework/contracts ./internal/framework/pipeline`.
|
||||
The runner may still reject or lack execution support for more than one
|
||||
resolved step, but all configurations and topology must resolve without a flat
|
||||
parallel lane model.
|
||||
Run `go test ./internal/framework/pipeline ./internal/framework/checkpoint
|
||||
./internal/modules/integration` and the pipeline race tests. Review the diff to
|
||||
confirm this stage changes structure only: serialized output, checkpoint state,
|
||||
decision values, failure selection, and module invocation behavior must remain
|
||||
unchanged.
|
||||
|
||||
## Stage 2: Step-Aware Preparation And Execution
|
||||
## Stage 2: Make Checkpoint Decisions Typed And Observable
|
||||
|
||||
### Objective
|
||||
|
||||
Execute independent ordered steps with hard barriers, even when no generated
|
||||
references are configured. Preserve the existing typed lane lifecycle and one
|
||||
bounded run-wide concurrency budget.
|
||||
Assign stable decision categories and reason codes explicitly, and retain the
|
||||
decision that causes a required-dependency failure.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Refactor `internal/framework/pipeline/prepare.go` so `Prepare` walks resolved
|
||||
steps in order and constructs every lane module and validator before it
|
||||
returns. A failure in any later step must occur before source parsing.
|
||||
Materialize only external reference sources during this phase; generated
|
||||
selectors carry no bytes yet.
|
||||
2. Refactor `runner.go` and `runner_concurrent.go` into a small ordered-step
|
||||
coordinator plus the existing bounded lane engine:
|
||||
- parse input and compute/reuse the chunk plan once;
|
||||
- invoke the lane engine once per step, passing only that step's prepared
|
||||
lanes and immutable request-reference view;
|
||||
- wait for every lane in the step to become terminal before advancing;
|
||||
- reuse the same worker/provider limits without permitting tasks from
|
||||
adjacent steps to overlap; and
|
||||
- encode output once after all steps succeed.
|
||||
3. Accumulate accepted, rejected, debug, and checkpoint outcomes across steps.
|
||||
Sort public results by step index, lane ID, source ID, and chunk index/ref as
|
||||
applicable. Add step identity to internal errors and debug events where lane
|
||||
identity alone no longer explains execution context.
|
||||
4. Preserve current fail-fast cancellation and bounded drain behavior. A
|
||||
framework error in a step cancels its started work, prevents all later steps,
|
||||
and prevents output encoding while retaining completed upstream outcomes.
|
||||
5. Until Stage 3 lands, reject execution of any pipeline with a generated
|
||||
binding before source parsing. This temporary guard prevents an accepted
|
||||
configuration from running a consumer without its declared dependency.
|
||||
1. In `internal/framework/pipeline/checkpoint.go`, introduce string-backed
|
||||
internal types and constants for checkpoint decision categories and reason
|
||||
codes. Keep the current JSON strings and public artifact fields compatible.
|
||||
Categories remain exactly `executed`, `reused`, `forced_recompute`, and
|
||||
`dependency_invalidated`.
|
||||
2. Replace prose inspection in `internal/framework/checkpoint/loader.go` with an
|
||||
explicit decision constructor accepting category, reason code, and optional
|
||||
detail. Remove every `strings.Contains` classification branch. Assign a code
|
||||
at the validation site using this bounded vocabulary:
|
||||
- `loading_disabled`, `checkpoint_missing`, `checkpoint_path_invalid`,
|
||||
`checkpoint_read_failed`, and `checkpoint_decode_failed`;
|
||||
- `workspace_schema_incompatible` and `identity_mismatch`;
|
||||
- `stage_mismatch`, `step_mismatch`, `lane_mismatch`, `module_mismatch`, and
|
||||
`status_not_reusable`;
|
||||
- `dependency_mismatch` for the `dependency_invalidated` category;
|
||||
- `artifact_payload_invalid`, `artifact_digest_mismatch`,
|
||||
`artifact_codec_incompatible`, and `artifact_not_canonical`; and
|
||||
- `checkpoint_reused` and `accepted_artifact_reused` for successful reuse,
|
||||
and `recompute_step` for forced execution.
|
||||
Retain an existing code not listed here only when a current external or
|
||||
documented contract already depends on it.
|
||||
3. Keep detail human-readable, UTF-8, bounded, and sanitized through one helper.
|
||||
Detail may identify stage, step, lane, module, expected status, or schema
|
||||
version, but must not include artifact/reference content, source content,
|
||||
credentials, environment values, or local paths. Tests must assert codes and
|
||||
safety properties, not exact detail prose.
|
||||
4. Change the centralized runner decision flow from Stage 1 so the final loader
|
||||
or canonical-validation decision is recorded before returning a
|
||||
required-predecessor error. The contextual error must identify the step and
|
||||
lane and include the stable reason code; it must not interpolate unsafe
|
||||
loader detail.
|
||||
5. Propagate typed values without lossy conversion through checkpoint events,
|
||||
run manifests, debug summaries, and CLI diagnostics. Convert to strings only
|
||||
at existing serialized boundaries unless changing an internal field type is
|
||||
simpler and wire-compatible.
|
||||
6. Update the canonical current-behavior owners in the same change:
|
||||
`docs/internal/state.md` owns the internal decision flow, while
|
||||
`docs/operations.md` owns operator diagnosis and any operator-visible reason
|
||||
code table. Link rather than duplicate the table elsewhere.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add runner tests with deterministic fake modules proving strict barriers,
|
||||
concurrency within one step, reuse of the same global budget, stable output
|
||||
and failure order despite inverted completion timing, and no output encoding
|
||||
after failure.
|
||||
- Add a preparation test proving every module and validator in every step is
|
||||
constructed before the input adapter is invoked.
|
||||
- Retain existing single-step concurrency and cancellation tests unchanged
|
||||
where possible; migrate fixtures to the implicit `default` resolved step.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run `go test ./internal/framework/pipeline ./internal/cli` plus the repository's
|
||||
race-test target for the framework if one is defined in `docs/development.md`.
|
||||
Both legacy pipelines and explicit pipelines without generated references must
|
||||
run successfully.
|
||||
|
||||
## Stage 3: Canonical Generated-Artifact Handoff And Provenance
|
||||
|
||||
### Objective
|
||||
|
||||
Make accepted normalized output from an earlier lane available as an immutable
|
||||
operation-time reference to later consumers, with safe checkpoint dependencies
|
||||
and bounded provenance.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Add a domain-neutral handoff component in `internal/framework/pipeline` that:
|
||||
- locates the producer's accepted normalized output after its step barrier;
|
||||
- requires exactly one output and rejects missing, rejected-only, or multiple
|
||||
outputs deterministically;
|
||||
- canonicalizes it through `ArtifactCodecRegistry.Encode` (decoding a reused
|
||||
serialized checkpoint through the registered codec first when necessary);
|
||||
- verifies kind, exact codec schema identity, accepted media type, maximum
|
||||
size, and one-item cardinality for every expanded consumer slot; and
|
||||
- creates independently cloned `ReferenceItem` values for fan-out targets.
|
||||
2. Build all generated bindings required by a consumer step before starting any
|
||||
lane in that step. If any handoff fails, return one contextual dependency
|
||||
error and start none of the step's consumers. An accepted typed empty list is
|
||||
valid; absence of an accepted artifact is not.
|
||||
3. Merge generated items with each target's already-materialized external
|
||||
`ReferenceSet` only after resolution has proven there is no collision. Pass
|
||||
the resulting cloned set through existing extraction, merge, normalize, and
|
||||
validation request structs. Never mutate prepared static reference sets.
|
||||
4. Add the canonical generated dependency fingerprint to checkpoint loader and
|
||||
recorder inputs for every receiving operation immediately in this stage.
|
||||
This is required before generated pipelines can safely use resume; do not
|
||||
defer it to selective recomputation work.
|
||||
5. Extend artifacts and runtime provenance:
|
||||
- add step ID to artifact-lane, normalized-output, rejected-output, and
|
||||
reference provenance records where needed for unambiguous context;
|
||||
- record external and generated origins distinctly;
|
||||
- for generated references record producer identities, codec schema
|
||||
identity, media type, digest, and size, but never content or a fabricated
|
||||
filesystem URI; and
|
||||
- update generic JSON output and debug summaries to serialize these bounded
|
||||
fields deterministically.
|
||||
6. Ensure checkpoint, debug, and output clones own their byte slices and maps.
|
||||
A consumer or test mutation must not affect another fan-out consumer or the
|
||||
producer artifact.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add handoff tests for one producer fan-out, empty typed collections, missing
|
||||
output, rejected-only output, multiple outputs, type/schema/media/size
|
||||
mismatch, target immutability, and failure before consumer start.
|
||||
- Add a resume-oriented runner test proving changed canonical producer content
|
||||
changes the consumer dependency fingerprint and prevents stale reuse.
|
||||
- Extend artifacts/JSON/debug contract tests with field assertions for bounded
|
||||
generated provenance and explicit assertions that content and local paths are
|
||||
absent. Avoid whole-document snapshots.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run focused pipeline, artifacts, JSON output, and debug tests, followed by `go
|
||||
test ./internal/framework/... ./internal/core/artifacts
|
||||
./internal/modules/generic/output/json`. A generic fake-codec pipeline must
|
||||
complete a two-step handoff both from fresh execution and a compatible reused
|
||||
producer checkpoint.
|
||||
|
||||
## Stage 4: Dependency-Aware Checkpoints And Selective Recompute
|
||||
|
||||
### Objective
|
||||
|
||||
Complete safe reuse, transitive invalidation, observable decisions, and the
|
||||
`--recompute-step` operator control.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Update checkpoint identity and manifests in
|
||||
`internal/framework/checkpoint`:
|
||||
- introduce workspace schema `notarius.workspace.v3` and retain explicit
|
||||
recognition of v1/v2 as incompatible cold misses;
|
||||
- include step ID in lane-stage manifests and lookup/recording context;
|
||||
- include ordered topology in the persistent pipeline identity; and
|
||||
- preserve the Stage 3 generated fingerprint tuple exactly, without content
|
||||
or secrets in decision reasons.
|
||||
2. When loading a normalized producer checkpoint, validate its manifest,
|
||||
deserialize through the registered codec, re-encode canonically, and compare
|
||||
schema/media/content fingerprints before exposing it to handoff. Corrupt,
|
||||
missing, rejected, or incompatible state is a cold miss for ordinary resume
|
||||
and must never reach a consumer.
|
||||
3. Build a lane-level dependency index from resolved generated bindings. Use it
|
||||
to invalidate only transitive consumer lanes when producer identity or bytes
|
||||
change; unrelated lanes, including unrelated lanes in a later step, remain
|
||||
eligible for reuse. The step barrier still applies when reused and executed
|
||||
lanes coexist.
|
||||
4. Add a single-value `--recompute-step <step-id>` flag in
|
||||
`internal/cli/run.go`. Reject repeated occurrences, unknown steps, use
|
||||
without checkpoint recording, use without `--resume`, and combination with
|
||||
`--only`. Accept the stable implicit step ID `default`; selecting it forces
|
||||
every lane in that single step. One selected step is sufficient for this
|
||||
scope; do not add multi-selection semantics.
|
||||
5. Convert the selected step to a force-execution set containing every lane in
|
||||
that step plus the lane-level transitive dependency closure. Loader policy,
|
||||
not persistent checkpoint identity, applies the force set. Required
|
||||
predecessors and unrelated lanes remain reusable. If a required predecessor
|
||||
was not selected and has no reusable accepted artifact, fail before starting
|
||||
a dependent lane rather than implicitly recomputing it.
|
||||
6. Replace the boolean-only checkpoint reporting model as needed with a bounded
|
||||
decision category: `executed`, `reused`, `forced_recompute`, or
|
||||
`dependency_invalidated`, plus a stable reason code and optional safe detail.
|
||||
Propagate it to checkpoint events, manifests, debug summaries, and CLI
|
||||
diagnostics. Do not make exact prose part of a test contract.
|
||||
|
||||
### Tests
|
||||
|
||||
- In checkpoint loader/recorder tests, cover v1/v2 cold misses, v3 step
|
||||
identity, compatible producer decode, corruption, exact dependency match,
|
||||
content/schema changes, and bounded reason fields.
|
||||
- In pipeline tests, cover transitive invalidation and reuse of unrelated work.
|
||||
- Add one CLI contract table for valid recomputation and every invalid flag
|
||||
combination, plus one execution test proving the selected closure is forced
|
||||
while a predecessor and unrelated lane are reused.
|
||||
- Do not assert internal loader call counts when observable decision records
|
||||
and outputs establish the behavior.
|
||||
- Add a table in `internal/framework/checkpoint` that exercises one
|
||||
representative input per reason-code family and asserts category, code,
|
||||
bounded valid UTF-8 detail, and absence of supplied secret/path sentinels.
|
||||
- Add pipeline behavior tests proving a required-predecessor failure records the
|
||||
underlying missing, corrupt, incompatible, or dependency-invalidated
|
||||
decision before returning.
|
||||
- Retain focused manifest/debug serialization assertions for category and code;
|
||||
do not add exact-detail or full-manifest snapshots.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline
|
||||
./internal/cli`, then checkpoint/debug state tests under the race detector as
|
||||
directed by `docs/development.md`. Manually inspect one test fixture's decision
|
||||
records to confirm they contain no reference bytes, secret values, or local
|
||||
paths.
|
||||
./internal/core/artifacts ./internal/core/debugbundle ./internal/cli`. Search the
|
||||
production checkpoint package to confirm no decision category or reason code is
|
||||
derived from diagnostic prose.
|
||||
|
||||
## Stage 5: D&D NPC-First Production Adoption
|
||||
## Stage 3: Hydrate Required Predecessors From Accepted Normalized Artifacts
|
||||
|
||||
### Objective
|
||||
|
||||
Adopt the platform in the production D&D composition: NPC normalization runs
|
||||
first, and its canonical artifact grounds spell extraction, combat-turn
|
||||
extraction, and combat-turn normalization at operation time.
|
||||
Make selective recomputation enforce the lane-level accepted-artifact contract:
|
||||
a valid normalized producer artifact is sufficient even when its extract or
|
||||
merge stage cache is unavailable.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Declare the normalized NPC-list artifact kind on every `npcs` consumer slot:
|
||||
- `internal/modules/dnd/extract/spells`;
|
||||
- `internal/modules/dnd/extract/combatturns`; and
|
||||
- `internal/modules/dnd/normalize/combatturns`.
|
||||
Keep the slot optional for external standalone use; a configured generated
|
||||
binding becomes required through the framework dependency.
|
||||
2. Refactor `internal/modules/dnd/npcs/registry` to provide a package-owned,
|
||||
concurrency-safe operation-time resolver/cache:
|
||||
- validate and seed construction-time external references so bad static
|
||||
configuration still fails before source parsing;
|
||||
- resolve the effective `npcs` item from each operation request;
|
||||
- reuse the seeded immutable registry when its digest matches and cache a
|
||||
generated registry by canonical digest so concurrent chunk operations do
|
||||
not repeatedly decode it; and
|
||||
- return cloned or immutable views and never retain caller-owned content.
|
||||
3. Change the spell extractor, combat-turn extractor, and combat-turn
|
||||
normalizer to obtain the NPC registry from `req.References` for each
|
||||
operation through that resolver. Do not reconstruct modules at a step
|
||||
boundary. Retain current behavior when the request has no NPC item.
|
||||
4. Keep static prompt/schema metadata and construction-time external reference
|
||||
fingerprints intact. Record generated NPC identity through framework
|
||||
reference provenance and dependency fingerprints; do not place
|
||||
operation-varying generated digests into singleton module metadata.
|
||||
5. Preserve prompt ordering and evidence policy. The generated NPC listing is
|
||||
stable contextual input before the variable transcript, but it cannot prove
|
||||
that a spell or combat event occurred. Source-unit citations remain the only
|
||||
event evidence.
|
||||
6. Add a maintained explicit two-step D&D configuration under `examples/` and a
|
||||
matching production-catalog fixture under
|
||||
`internal/modules/integration/testdata/`. Replace or supplement the two
|
||||
manual NPC-to-spell and NPC-to-combat configurations with this single
|
||||
NPC-first workflow; retain standalone examples that demonstrate external
|
||||
NPC references where they remain useful.
|
||||
7. Add one offline integration test using scripted/fake LLM responses. Assert
|
||||
NPC normalization completes first, the exact canonical NPC digest reaches
|
||||
all three consumers, spell and combat lanes may both succeed, combat
|
||||
normalization uses the registry, and generated NPC context is not accepted
|
||||
as source evidence by itself.
|
||||
1. Extend the checkpoint loader contract with a dedicated accepted-normalized-
|
||||
artifact lookup. Implement it in the filesystem loader and every no-op or
|
||||
test implementation. The lookup receives step ID, lane ID, and normalizer
|
||||
module key and returns the normalized artifact, its warnings, and an explicit
|
||||
checkpoint decision. It must not require caller-supplied extract or merge
|
||||
dependency fingerprints.
|
||||
2. The filesystem lookup may reuse the existing normalize manifest and payload;
|
||||
do not add a second persistent copy. It is reusable only when all of the
|
||||
following hold:
|
||||
- workspace schema is v3 and the non-empty stored checkpoint identity matches
|
||||
the current invocation identity;
|
||||
- stage, step, lane, normalizer module, and successful status match;
|
||||
- the manifest output digest matches the payload; and
|
||||
- the payload can subsequently be validated through the registered artifact
|
||||
codec.
|
||||
Skipping caller-supplied merge dependencies is safe only because the matched
|
||||
non-empty checkpoint identity already binds the current input, resolved
|
||||
topology and configuration, external references, runtime overrides, LLM
|
||||
profiles, and component semantic fingerprints. Do not weaken or omit that
|
||||
identity check.
|
||||
A loader lacking a verifiable current identity must return an unavailable
|
||||
decision rather than perform accepted-artifact reuse.
|
||||
3. Add a pipeline hydration helper that decodes the returned serialized
|
||||
artifact through the producer's registered codec, re-encodes it canonically,
|
||||
and requires exact artifact kind, schema ID/name/version/digest, media type,
|
||||
content bytes, and content digest. Return a runner-owned clone with producer
|
||||
step, lane, module, and source identity. Any mismatch is an explicit bounded
|
||||
decision and the bytes never reach a consumer.
|
||||
4. Before normal execution of a lane marked in `RequireReusableLanes`, use the
|
||||
accepted-artifact lookup:
|
||||
- on success, mark the lane terminal without invoking extract, merge,
|
||||
normalize, or their validators;
|
||||
- append the accepted normalized output in the normal deterministic location
|
||||
and restore only the warnings stored with that normalized checkpoint;
|
||||
- record one `reused` normalize decision with a stable accepted-output reuse
|
||||
reason code; do not synthesize extract/merge decisions, warnings, or
|
||||
rejections that were not loaded; and
|
||||
- allow the ordinary step barrier and generated handoff code to consume that
|
||||
output exactly as it consumes a freshly executed output.
|
||||
5. On missing, rejected, corrupt, non-canonical, or incompatible accepted state,
|
||||
record the decision and fail the run before the dependent step begins. Do
|
||||
not fall back to stage execution. Forced lanes must bypass this hydration
|
||||
path and execute normally.
|
||||
6. Leave ordinary resume unchanged when no lane is marked
|
||||
`RequireReusableLanes`: it may reuse or recompute extract, merge, and
|
||||
normalize progressively under the existing cold-miss rules.
|
||||
7. Keep generated-reference fingerprints and provenance unchanged. Hydrating a
|
||||
byte-identical producer must yield the same canonical handoff digest and
|
||||
downstream dependency fingerprint as fresh execution.
|
||||
8. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and
|
||||
`docs/operations.md` in this stage to distinguish progressive stage reuse
|
||||
from accepted normalized-artifact hydration and to document the fail-rather-
|
||||
than-rerun rule for invalid required predecessors.
|
||||
|
||||
### Tests
|
||||
|
||||
- Add focused registry and module tests for static fallback, generated override,
|
||||
concurrent cache safety, malformed runtime content, and ownership isolation.
|
||||
- Extend prompt-input tests only for semantic placement and content. Do not add
|
||||
exact prompt-length or message-count change detectors.
|
||||
- Run the single end-to-end offline D&D handoff test rather than duplicating it
|
||||
separately for every consumer.
|
||||
- At the pipeline/checkpoint boundary, create a valid producer normalize
|
||||
checkpoint while omitting or corrupting its extract and merge checkpoints.
|
||||
Select a later step for recomputation and prove the producer hydrates, no
|
||||
producer operation or validator runs, and the dependent receives the exact
|
||||
canonical artifact.
|
||||
- Cover missing, rejected-status, corrupt, non-canonical, wrong-codec-identity,
|
||||
and wrong-content-digest normalize state. Assert failure and the recorded
|
||||
stable decision before any consumer invocation.
|
||||
- Prove forced producers execute instead of hydrating, while a reusable
|
||||
predecessor and an unrelated lane remain reusable.
|
||||
- Prove fresh and hydrated producer outputs create identical generated
|
||||
provenance and downstream checkpoint fingerprints without exposing content
|
||||
or paths.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
Run all D&D package tests and integration tests, including the race detector for
|
||||
the registry cache, then parse and resolve every maintained example through the
|
||||
production catalog. No test may require credentials or a live LLM provider.
|
||||
Run `go test ./internal/framework/checkpoint ./internal/framework/pipeline
|
||||
./internal/modules/integration` and the corresponding race tests. Manually
|
||||
inspect one failed test fixture to confirm the accepted artifact remains on
|
||||
disk but its bytes do not appear in the manifest, decision detail, debug
|
||||
summary, or error.
|
||||
|
||||
## Stage 6: Current Documentation And Release Verification
|
||||
## Stage 4: Complete Recompute CLI And Recovery Acceptance Coverage
|
||||
|
||||
### Objective
|
||||
|
||||
Document the now-implemented behavior in its canonical current owners, retire
|
||||
obsolete manual workflow guidance, and perform repository-wide verification.
|
||||
Exercise the operator-visible recomputation contract through stable boundaries
|
||||
and close the acceptance-test omissions from the original plan.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Update `docs/config.md` with version-3 `steps`, legacy `artifacts`
|
||||
compatibility, reference source forms, precedence/conflict rules, global
|
||||
identity rules, and the exact D&D two-step example.
|
||||
2. Update `docs/cli.md` with `--recompute-step`, prerequisites, closure
|
||||
semantics, invalid combinations, and checkpoint decision categories.
|
||||
3. Update `docs/internal/pipeline.md`, `docs/internal/state.md`, and
|
||||
`docs/internal/overview.md` with the fixed ordered-step model, preparation
|
||||
timing, handoff boundary, codec use, provenance, and checkpoint dependency
|
||||
behavior. Keep the architecture description linear and explicitly state that
|
||||
this is not a general DAG.
|
||||
4. Update `docs/operations.md` with resume/recompute procedures and safe failure
|
||||
diagnosis. Update the D&D integration documents for generated NPC context,
|
||||
evidence limitations, and the maintained workflow. Update
|
||||
`docs/policy/testing.md` only if the implementation exposes a genuinely new
|
||||
durable testing policy; do not restate feature-specific tests there.
|
||||
5. Update maintained example checks and any README/index links that point to
|
||||
replaced sequential examples. Keep the feature roadmap as historical target
|
||||
state until the project applies its normal roadmap-completion process; do not
|
||||
turn it into a duplicate current-behavior manual.
|
||||
6. Review changed exported identifiers and package comments, run formatting and
|
||||
static analysis, and remove obsolete flat-lane compatibility helpers,
|
||||
temporary test adapters, dead sequential-workflow code, and TODOs introduced
|
||||
during earlier stages.
|
||||
|
||||
### Verification
|
||||
|
||||
Run the repository-prescribed commands from `docs/development.md`, including:
|
||||
|
||||
1. all unit and integration tests;
|
||||
2. race tests for concurrency-sensitive framework, checkpoint, and D&D registry
|
||||
packages;
|
||||
3. `go vet` and the normal build;
|
||||
4. maintained-example/config validation; and
|
||||
5. documentation checks.
|
||||
|
||||
Perform one final acceptance review against every bullet in
|
||||
[Ordered Pipeline Steps](ordered-pipeline-steps.md). Inspect a successful fresh
|
||||
run, a resumed run, a forced-recompute run, and a missing-producer failure for
|
||||
deterministic ordering, correct dependency decisions, bounded provenance, and
|
||||
absence of generated content or secrets in manifests/debug state.
|
||||
1. Add one table-driven CLI contract test for `--recompute-step` covering:
|
||||
- a valid explicit step and the implicit `default` step;
|
||||
- repeated flags and an empty or unknown step ID;
|
||||
- use without `--resume`;
|
||||
- use when checkpoint recording is disabled; and
|
||||
- combination with `--only`.
|
||||
Assert exit classification and stable identifying fragments or reason codes,
|
||||
not complete prose.
|
||||
2. Add one filesystem-backed CLI execution test using deterministic fake
|
||||
modules and a three-step generated dependency chain plus one unrelated lane:
|
||||
- perform a fresh checkpointed run;
|
||||
- remove or corrupt only the required producer's extract and merge state
|
||||
inside `t.TempDir()`, leaving its normalize artifact valid;
|
||||
- resume with the middle step selected;
|
||||
- assert the selected lane and transitive dependents execute, the predecessor
|
||||
hydrates without module calls, the unrelated lane reuses, and output and
|
||||
decision ordering are deterministic; and
|
||||
- invalidate the producer normalize state in a subcase and assert the command
|
||||
fails before dependent execution with the bounded decision preserved.
|
||||
3. Add or extend one generic two-step pipeline test proving handoff succeeds
|
||||
both from fresh producer execution and from accepted normalized-artifact
|
||||
hydration. Keep this at the pipeline boundary if the CLI execution test
|
||||
already proves flag wiring; do not duplicate every CLI case end to end.
|
||||
4. Review existing recomputation policy tests. Retain the pure closure test
|
||||
because it protects transitive selection, but remove or consolidate any new
|
||||
test that merely repeats the CLI or pipeline behavior above.
|
||||
5. Make only production changes revealed as necessary by these contract tests;
|
||||
do not add new flag semantics or broaden the feature roadmap.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
The feature is complete only when every acceptance criterion in the feature
|
||||
roadmap is demonstrated by a stable test or maintained example and the full
|
||||
repository verification suite passes.
|
||||
Run `go test ./internal/cli ./internal/framework/pipeline
|
||||
./internal/framework/checkpoint`, then run the CLI, pipeline, and checkpoint
|
||||
packages under the race detector. Confirm no test asserts internal loader call
|
||||
counts, exact decision detail, filesystem layout outside a temp workspace, or
|
||||
the exact length of any prompt or prefix.
|
||||
|
||||
## Stage 5: Current Documentation And Release Verification
|
||||
|
||||
### Objective
|
||||
|
||||
Reconcile all canonical documentation after the staged changes and verify the
|
||||
complete feature against policy and roadmap.
|
||||
|
||||
### Changes
|
||||
|
||||
1. Re-read `docs/internal/pipeline.md`, `docs/internal/state.md`,
|
||||
`docs/operations.md`, and `docs/cli.md` against the final code. Correct any
|
||||
stale statements left by Stages 1 through 4 without duplicating their
|
||||
canonical contracts.
|
||||
2. Ensure internal documentation describes the coordinator, lane engine, stage
|
||||
checkpoint flow, and accepted-artifact validation at the responsibility
|
||||
level without listing volatile private helper names.
|
||||
3. Confirm the operator-visible reason-code table has one canonical owner and
|
||||
other documents link to it rather than maintaining parallel copies.
|
||||
4. Remove stale implementation claims exposed by this work and ensure the
|
||||
feature roadmap continues to describe target policy rather than task
|
||||
sequencing.
|
||||
|
||||
### Verification
|
||||
|
||||
Run the repository-prescribed commands from `docs/development.md`:
|
||||
|
||||
```sh
|
||||
go test ./...
|
||||
go vet ./...
|
||||
go build ./cmd/notarius
|
||||
```
|
||||
|
||||
Also run race tests for the pipeline, checkpoint, CLI state, D&D NPC registry,
|
||||
and D&D integration packages. Validate maintained examples/configurations
|
||||
through the production catalog. Review one fresh run, ordinary resumed run,
|
||||
forced-recompute run, hydrated-predecessor run, and invalid-predecessor failure
|
||||
for deterministic ordering, correct decisions, bounded provenance, and absence
|
||||
of generated content, secrets, or local paths in manifests and diagnostics.
|
||||
|
||||
### Completion Gate
|
||||
|
||||
The follow-up is complete only when every finding summarized above is protected
|
||||
by a stable behavioral test, the current documentation matches the corrected
|
||||
implementation, the full verification suite passes, and the worktree contains
|
||||
no temporary adapters or TODOs introduced by these stages.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. The feature roadmap and this plan fix all product and architecture choices
|
||||
required for implementation. If an implementation detail conflicts with a
|
||||
policy document, the policy document takes precedence; if it would change the
|
||||
product semantics above, stop and amend the roadmap rather than deciding it in
|
||||
code.
|
||||
None. The feature roadmap and this plan fix the required product and
|
||||
architecture choices. If implementation reveals that accepted normalized
|
||||
artifacts cannot be validated safely without a persistent format change, stop
|
||||
and amend this plan rather than weakening identity, codec, or content
|
||||
validation.
|
||||
|
||||
Reference in New Issue
Block a user