narratio/docs/roadmap/implementation.md

Workstream 6 Implementation Plan — Per-Artifact Analyze Checkpointing

Status

Planning document for Workstream 6, Per-artifact analyze checkpointing and recovery, in refactoring.md. Workstreams 1 through 5 are complete and are prerequisites for this plan. All stages below are pending and must be implemented in numerical order.

The purpose of this workstream is to make analyze a deterministic resumable artifact queue backed only by the session manifest. If artifacts A and B succeed, C fails, and D is not attempted, A and B must remain reusable, C must be retried, and D must follow after C succeeds. The implementation must remove the current end-of-stage analyze projection and rollback machinery rather than building checkpointing alongside it as a second durable model.

Authoritative Scope And Decisions

The authoritative product and architecture scope is Workstream 6, the dependency relationships, compatibility intent, and consolidated target state in refactoring.md. Supporting decisions are H04, H14, H15, H36, H37, C02, and C03 in audit-decisions.md, with the detailed rationale in H36 and H37 of audit.md and Finding 3 of contract-audit.md.

The following decisions are fixed for this implementation:

• Keep analyze as one fixed pipeline stage in internal/stage; per-artifact work does not become dynamic pipeline stages or a generic workflow engine.
• Keep the session manifest's analyze_artifacts collection as the only durable per-artifact authority. Do not introduce an invocation collection, journal, sidecar, database, or replacement run manifest.
• Use the existing stage.Env.ManifestStore and the canonical session-manifest path for checkpoints. Production execution already supplies that store and holds the session lock for the entire Stage.Run call. Do not add a callback framework or expose application lifecycle types to internal/stage.
• Checkpoint a copy-on-write candidate manifest. Validate and save the candidate before installing its analyze collection into the live in-memory manifest. A failed current-record checkpoint must never make newly materialized bytes authoritative through a later terminal save.
• Persist the reconciled starting collection before the first adapter call when it changes durable authority. This removes obsolete records and stales scheduled or invalid current records before replacement work begins.
• For every successful artifact: validate and materialize its canonical output, construct and validate its current record, update the runtime catalog, stale transitive dependents when output identity changed, checkpoint the complete session collection, and only then attempt the next artifact.
• For an artifact failure: retain earlier successful checkpoints, replace the active artifact with a bounded failed record, stale its current transitive dependents, and expose that collection on the live manifest. The application runner remains responsible for marking aggregate analyze failed, invalidating publish/notify, and atomically persisting the terminal manifest before the command returns the error.
• If persistence of a successful-artifact checkpoint fails, stop immediately. Keep the last durable collection, mark the active artifact failed in the live collection without promoting its incidental output, and let the runner's terminal failure save make that retry state durable when storage permits.
• Preserve the existing deterministic work planner, reconciliation, fingerprints, evidence validation, dependency order, and current-output identity comparison. File presence alone never creates authority.
• Preserve artifact selection and force behavior. With no --artifacts, the effective configured set is targeted. With --artifacts, only the named targets and their required dependency closure participate. Force reruns explicit targets; a valid current prerequisite is reused unless it is itself explicit or becomes non-current through dependency work. Unrelated valid records are retained.
• On aggregate success, return ordinary StageResult.Outputs rebuilt in deterministic normalized artifact-key order from all current session analyze records. Per-artifact records retain their own logs, generated configuration references, bounded Scriptorium provenance, and evidence; aggregate logs and metadata continue to describe the current invocation.
• On aggregate failure, aggregate outputs remain empty. Earlier current per-artifact records remain available to a later analyze run, but publish and notify are invalidated by the existing application-stage dependency relation.
• A crash or cancellation after a successful checkpoint may leave aggregate analyze running or later normalized to interrupted; the next ordinary run must reconcile the durable collection, reuse valid completed artifacts, and continue with the first failed, stale, or missing artifact.
• Keep the manifest schema and analyze-state contract version unchanged. This work changes persistence timing and removes an internal projection transport; it does not introduce a new serialized representation.
• Do not enter Workstream 7 configuration decomposition, Workstream 8 publish locks/decomposition, or Workstream 9 cleanup-stage work.

Instructions For Every Stage

Before each stage:

1. Read docs/development.md, every document under docs/policy/, this plan, and Workstream 6 in refactoring.md. Read the decision entries and audit sections named above; do not re-audit unrelated findings.
2. Read ../internal/stage-analyze.md and ../internal/manifest.md, then inspect the production files and focused tests named by the stage.
3. Prefer the codebase knowledge graph for Go symbol discovery and call tracing. Use text search for serialized fields, documentation, fixtures, and removal of projection terminology.
4. Confirm the worktree state and preserve unrelated changes. Do not begin a later workstream merely because adjacent code is visible.

During every stage:

• Keep the repository compiling and focused tests passing. A temporary private delegation or projection bridge is permitted only where the next numbered stage explicitly removes it; do not leave two durable authorities active.
• Preserve package ownership and acyclic dependency direction. Keep new types private and cohesive; do not add public extension points.
• Keep all tests deterministic, offline, and independent of real Scriptorium or paid APIs. Prefer the existing fake runner, real temporary filesystem, and a small recording/failing manifest store only where checkpoint interaction is itself the contract.
• Assert persisted manifest state, adapter calls, dependency order, and retry behavior. Do not pin private helper choreography, complete error wording, or incidental timestamps.
• Preserve bounded, non-secret errors and diagnostics. Never place prompt or transcript content, variables containing secrets, or raw credentials in new manifest fields or error assertions.
• Use gofmt on changed Go files, run the focused tests listed for the stage, and run git diff --check. Mark a stage Completed only after its exit criteria pass.

Stage 1 — Decompose The Single-Artifact Transaction

Status: Completed

Goal

Make one artifact execution understandable and independently reviewable before changing its persistence timing. This stage is behavior-preserving.

Required Work

1. Refactor executeAnalyzeArtifact in internal/stage/analyze.go into one short transaction owner plus private phases with explicit intermediate values. Keep the existing analyzeExecutionContext and artifact plan, or replace them with equally narrow private values; do not restore the former broad parameter list.
2. Separate these concerns without changing their order or contracts:
   • input identity resolution, omitted optional inputs, reused-current input reporting, Scriptorium variables, timeout, canonical/run-local paths, and request/diagnostic path preparation;
   • optional render-debug invocation and validation;
   • Scriptorium artifact invocation and adapter failure adaptation;
   • authoritative output-path selection, bounded non-empty regular-file validation, output-contract construction, canonical materialization, and installed-checksum verification; and
   • result/evidence construction, including provenance, logs, generated configuration paths, metadata, output size, and reused artifacts.
3. Place the extracted code in focused analyze-owned files when that improves navigation. Suggested responsibilities are preparation, adapter invocation, and validated materialization/evidence; exact private filenames are not a contract.
4. Keep Scriptorium request and response types inside the stage/adapter boundary. Do not move Narratio artifact, manifest, path, or metadata policy into the Scriptorium adapter.
5. Preserve the current run-local-before-canonical materialization boundary, render-debug behavior, useful error context, workspace modes, centralized size limit, and checksum comparison. Do not weaken validation or add a new write path while moving code.
6. Retain existing behavior tests at their stable boundaries. Consolidate only tests that become purely coupled to moved private helpers; do not add one test per extracted function.

Exit Criteria

• executeAnalyzeArtifact reads as a short ordered transaction over explicit phases and no longer mixes all preparation, invocation, materialization, and evidence construction in one large function.
• Existing analyze request, render, validation, materialization, provenance, and failure behavior is unchanged.
• go test ./internal/stage ./internal/adapters/scriptorium succeeds.
• git diff --check succeeds.

Stage 2 — Establish Session-Only Analyze Execution State

Status: Pending

Goal

Replace analyzeStage.Run's parallel local collections with one private session-oriented execution state that is ready to checkpoint, while retaining the existing end-of-stage persistence boundary temporarily.

Required Work

1. Introduce a private analyze execution state/accumulator that owns:
   • the shared execution context and deterministic work plan;
   • the complete reconciled session artifact collection;
   • prior current records needed for output-identity comparison;
   • the evaluated/scheduled key set used for dependent invalidation;
   • the mutable runtime artifact catalog; and
   • invocation-only aggregate logs, generated configurations, generated metadata, reused-artifact metadata, and its deduplication set.
2. Keep setup, reconciliation, work planning, adapter-availability validation, deterministic iteration, and final StageResult assembly visible in analyzeStage.Run. Move per-item state mutation and diagnostic collection behind narrow methods on the private state; do not turn the accumulator into a service locator.
3. Rename analyzeWorkPlan.ProjectedRecords, projectedAnalyzeRecords, and related private terminology to describe the reconciled starting session collection. It is the candidate durable state, not an invocation projection. Preserve the planner's read-only behavior and existing classification.
4. Make the accumulator install its complete collection into the passed session manifest on success and its failed/staled collection on error. During this stage, continue returning the existing AnalyzeStateProjection as a temporary compatibility bridge for the application runner; both views must be derived from the same collection, and no additional save is introduced.
5. Add optional ProducerRunID provenance to artifacts.Ref. Update the generic application result mapper to use that value when it is non-empty and otherwise retain the current behavior of assigning the active run ID. This is data needed by ordinary stage-result transport, not an analyze-specific branch; cover both explicit and fallback behavior in the mapper tests.
6. Add a private helper that deterministically converts all current analyze records into ordinary artifacts.Ref outputs. Preserve source ID, relative path, contract, checksum, and producer run ID. Include valid current records retained outside a partial selection; do not include stale, missing, or failed records. Continue using the projection path for aggregate outputs until Stage 3 switches ownership.
7. Rewrite stage tests to inspect the manifest collection mutated by the real analyze stage rather than manually installing returned projections. Retain a minimal assertion that the temporary projection matches that collection so the transition cannot silently diverge.

Exit Criteria

• analyzeStage.Run coordinates one private session collection rather than parallel record and diagnostic variables.
• Planning/reconciliation remains read-only, execution order and force/selection behavior are unchanged, and stage tests primarily assert manifest state.
• Ordinary StageResult.Outputs can preserve a reused artifact's original producer run identity without an analyze-specific application path.
• The existing application projection path still functions and no checkpoint timing has changed yet.
• go test ./internal/stage ./internal/app succeeds.
• git diff --check succeeds.

Stage 3 — Cut Analyze Over To Durable Per-Artifact Checkpoints

Status: Pending

Goal

Make each validated artifact completion durable before later artifact work and stop using the end-of-stage projection transport for real analyze execution.

Required Work

1. Add a private analyze checkpoint operation in internal/stage. It must:
   • require a non-nil session manifest, analyze stage record, and Env.ManifestStore when analyze has state to persist;
   • derive the canonical manifest path through artifacts helpers from the resolved workspace, campaign, and session identity;
   • validate the full candidate collection with manifest.ValidateAnalyzeArtifactCollection;
   • shallow-copy the manifest, clone the top-level stage map, copy the analyze stage record, and deep-clone the candidate analyze collection so the save cannot mutate live authority before success;
   • preserve the aggregate analyze status (normally running) and every unrelated manifest field;
   • save the candidate through Env.ManifestStore; and
   • only after success, install the candidate analyze version/collection and the store-updated manifest timestamp into the live manifest. Do not add a general manifest transaction framework or JSON round-trip clone.
2. Before invoking the first scheduled artifact, checkpoint the reconciled starting collection when it differs from durable state. This must remove records for removed configuration and stale any selected/forced/non-current authority before replacement work begins. If this save fails, invoke no adapter.
3. For each scheduled artifact, preserve this order:
   • compute its execution fingerprint;
   • execute the decomposed artifact transaction;
   • construct and validate its current manifest record;
   • prove the generated record can be added to the runtime catalog for use by later dependents;
   • stale unscheduled transitive dependents only when prior and replacement output identities differ;
   • checkpoint the complete updated session collection; and
   • only then expose the result to a later artifact.
4. If a successful artifact's checkpoint fails, stop immediately and do not call the next artifact. Build a failed record for the active artifact from the last committed collection, with no output authority; stale its current transitive dependents; and install that failure candidate in memory so the runner's terminal save can persist retry state. Return the checkpoint failure with artifact identity and manifest context.
5. Have real analyze execution return ordinary StageResult.Outputs built from every current session record plus its existing aggregate diagnostics and metadata. Stop returning AnalyzeStateProjection from analyzeStage.Run. The legacy projection type and application handlers may remain only as dead transitional code until Stage 4; add no new uses.
6. In executeSelectedStage, make a terminal save failure after any stage's apparent success mark that stage failed and invalidate succeeded transitive dependents before terminal persistence. This behavior currently occurs only on the analyze-projection branch; it must remain correct when real analyze no longer returns a projection and should be correct generically.
7. Update direct analyze fixtures so they supply a real temporary manifest.LocalStore, complete campaign/run identity, an aggregate analyze record in the state appropriate to the test, and an initially persisted manifest. Keep unit-only planner/reconciliation tests free of persistence.
8. Add focused checkpoint tests at the stage/application boundary:
   • the manifest store observes A current before B is invoked and A+B current before C is invoked;
   • a failed B checkpoint prevents C from being invoked;
   • the failed B checkpoint never persists B as current merely because its canonical file exists; and
   • final aggregate success exposes outputs for all and only current records in deterministic normalized artifact-key order, preserving each record's producer run identity.

Exit Criteria

• Every successful scheduled artifact is durable before the next adapter call.
• A checkpoint failure stops later work and cannot promote incidental output.
• Real analyze execution no longer relies on an end-of-stage state projection.
• Aggregate success/failure persistence and dependent invalidation remain application-owned and correct.
• go test ./internal/stage ./internal/app ./internal/manifest succeeds.
• git diff --check succeeds.

Stage 4 — Complete Failure Recovery And Remove Projection Machinery

Status: Pending

Goal

Complete the A/B/C/D recovery contract and delete the superseded projection, validation, application, and rollback paths.

Required Work

1. On any artifact-specific failure—including fingerprinting, preparation, render, adapter execution, result validation, materialization, evidence construction, catalog admission, or checkpoint persistence—construct one bounded failed record for the active key, clear its output authority, and stale current transitive dependents. Preserve unrelated current records and all earlier successfully checkpointed records.
2. Install the failed/staled collection on the live manifest before returning the artifact error. The runner must then use its ordinary stage-error path to mark aggregate analyze failed, clear aggregate outputs, invalidate publish and notify, and persist the terminal session manifest. Join a terminal save failure with the originating artifact/checkpoint error; never replace the useful cause.
3. Ensure unattempted artifacts behave according to existing authority: previously absent records remain absent/missing on reconciliation; unrelated valid current records remain current; and a prior current record that depends directly or transitively on the failed artifact becomes stale with no output authority.
4. Remove AnalyzeStateProjection from stage.StageResult, delete internal/app/analyze_projection.go, and remove all validation, cloning, application, capture, rollback, and special output reconstruction that exists solely for that projection. Remove fake projection stages and projection-only tests; replace their meaningful risks with real analyze checkpoint/recovery coverage.
5. Simplify executeSelectedStage to its generic result/error handling while preserving analyze's live manifest collection across aggregate lifecycle clearing. Do not add a type assertion, stage-name branch, callback, or new result union to replace the projection special case.
6. Add a durable assembled-run regression for the approved scenario:
   • configure deterministic A, B, C, and D work in dependency order;
   • make A and B succeed, C fail, and verify D is not attempted;
   • reload the manifest and verify A/B current, C failed, aggregate analyze failed, and the correct missing/stale state for D;
   • run again without force, verify only C and then D execute, and verify the final aggregate and all selected records are current; and
   • verify valid unrelated records and upstream transcript stages were never invalidated or rerun.
7. Add focused variants for failure of the first and last artifact, cancellation after at least one checkpoint, changed-output invalidation of an unselected transitive dependent, and byte/contract-identical replacement preserving an unselected dependent. Consolidate with existing incremental/failure tests so each durable risk has one clear owner.

Exit Criteria

• A/B remain durable after C fails; an ordinary later run retries C and then D without invoking A/B.
• Artifact failure and checkpoint failure leave no false current authority, and publish/notify cannot remain successfully current after aggregate failure.
• No analyze projection type, field, helper, application branch, rollback path, test fixture, or documentation claim remains.
• go test ./internal/stage ./internal/app ./internal/manifest ./internal/artifacts succeeds.
• git diff --check succeeds.

Stage 5 — Verify Force Semantics, Align Documentation, And Close Workstream 6

Status: Pending

Goal

Verify the complete operational contract, remove superseded test/documentation language, and leave Workstream 6 closed with repository-wide evidence.

Required Work

1. Review existing force/selection/reconciliation tests and retain a lean set that proves:

   • an ordinary run reuses every valid current selected artifact;
   • unqualified force reruns the full effective configured target set;
   • selected force reruns the explicit targets, includes their dependency closure, reuses valid current prerequisites, and leaves unrelated valid artifacts untouched;
   • stale, failed, missing, or dependency-invalid prerequisites execute in deterministic topological order before their dependents; and
   • an incidental canonical file without current manifest evidence is never reused.

2. Replace projection-oriented test names, helpers, and assertions with checkpoint/recovery language. Remove redundant tests whose only purpose was to validate the retired transport or rollback choreography. Do not reduce coverage of output validation, dependency invalidation, cancellation, persistence failure, or assembled resume behavior.

3. Update the canonical current-behavior owners:

   • docs/internal/stage-analyze.md for checkpoint order, resume behavior, failure semantics, and the absence of projections;
   • docs/internal/manifest.md for per-artifact checkpoint authority, aggregate status, interruption recovery, and persistence failure;
   • docs/policy/architecture.md only where its stage/persistence description currently implies a single end-of-stage analyze transaction; and
   • docs/operations.md and docs/troubleshooting.md only where operators need to understand reuse after partial analyze failure or recover from a manifest checkpoint error. Do not duplicate CLI or configuration contracts, since syntax and fields do not change.

4. Search production, tests, and current-behavior documentation for AnalyzeStateProjection, AnalyzeState, analyze projection, projected records, and end-of-stage language. Remove or revise every reference tied to the retired mechanism; preserve unrelated planning uses of the ordinary English word “projection” only when accurate.

5. Run the full repository validation from docs/development.md:

   go test ./...
   go test -race ./...
   go vet ./...
   go build ./...
   go test ./internal/doccheck
   go test ./internal/config -run '^TestExamplesLoadAndValidate$'
   

6. Run go test -race -shuffle=on -count=3 ./internal/stage ./internal/app to stress the changed recovery and persistence boundaries. Run the supported release cross-build checks if an implementation change affects build-tagged code; otherwise the repository build is sufficient for this workstream.

7. After all checks pass, mark Stages 1–5 in this document Completed and mark only Workstream 6 in refactoring.md Complete. Workstream 7 must remain the next pending workstream.

Exit Criteria

• Current documentation describes analyze as a session-manifest-backed, per-artifact checkpointed queue and contains no retired projection behavior.
• The force, partial-selection, dependency, interruption, and persistence-error cases are protected through stable behavioral boundaries.
• All repository validation and focused shuffled race checks pass.
• Workstream 6 is marked complete, Workstream 7 remains pending, and the worktree contains no accidental generated or temporary files.
• git diff --check succeeds.

Open Questions

None. The audits, approved Workstream 6 scope, existing selection contract, and the fixed decisions above resolve the product and implementation choices needed for the coding agent to execute this plan.