narratio/docs/roadmap/implementation.md

Workstream 9 Implementation Plan — Cleanup As The Final Pipeline Stage

Status

Planned implementation for Workstream 9, Cleanup as the final pipeline stage, in refactoring.md. Workstreams 1 through 8 are complete prerequisites. This plan does not independently undertake Workstream 10; its test and documentation requirements apply only where this work changes the owning behavior.

Authoritative Scope And Fixed Decisions

Workstream 9 in refactoring.md is authoritative. Its supporting records are H20 and H21 in audit.md, Findings 2 and 4 in contract-audit.md, and C02 and C04 in audit-decisions.md. Read those cited sections rather than rereading or reinterpreting the complete audits.

The following decisions are final:

• Add cleanup after notify as the twelfth and final canonical stage. publish commits remote state, notify completes delivery behavior, and only then may cleanup remove local run data.
• Retain the existing configuration names pipeline.spool.delete_audio_after_publish and pipeline.workspace.cleanup_after_publish. They select cleanup-stage targets; they no longer cause a runner-finalization side effect.
• When neither policy is enabled, cleanup returns an explicit self-skip with no outputs. Self-skips remain reconsidered on later invocations under the ordinary runner contract.
• When either policy is enabled, cleanup requires a succeeded publish stage whose metadata proves a completed pointer-selected upload: uploaded=true, skipped is not true, and canonical nonempty published_run_id, remote_commit_key, and current_commit_pointer_key values agree with the manifest's campaign/session remote prefix. Missing, malformed, skipped, or disabled publication evidence is an error and no deletion begins.
• The succeeded publish record is sufficient durable evidence. Cleanup does not reread object storage: publish success is recorded only after the pointer write, immutable run objects remain durable if a later pointer supersedes them, and adding a network dependency would make local retry less reliable.
• Compute targets from the validated published run ID, manifest campaign and session identities, and the currently resolved application-owned workspace and spool roots using canonical artifact path helpers. Never use the new invocation's RunID, LocalWorkDir, or LocalSpoolDir, and never accept a path from publish metadata as a deletion target.
• Process requested targets deterministically in spool-then-workspace order. Use fileops.RemoveAllUnderRoot; a missing root or target is successful. Preserve refusal of a root itself, traversal, and out-of-root deletion.
• Do not record per-target progress. If one deletion succeeds and a later one fails, the ordinary stage record becomes failed. A later run recomputes the same canonical targets, treats the first as already absent, and retries the remainder.
• A successful cleanup produces no canonical artifacts. Its bounded metadata may identify the published run and completed policy names, but must not become a second cleanup ledger.
• Cleanup has a semantic fingerprint over the two policy booleans and their normalized configured roots. Its resume validator revalidates publish evidence and confirms every enabled canonical target is absent. Recreated or newly enabled targets make a prior success non-resumable.
• Reject statically impossible configuration: either cleanup policy requires publish to be enabled with run upload enabled. Dynamic absence or corruption of publish evidence remains a cleanup-stage error.
• notify -> cleanup is the final direct invalidation edge and cleanup has no dependents. This makes every earlier prerequisite transitively stale a prior cleanup result while retaining the fixed relation separately from execution order.
• A default full run includes cleanup. --through notify, any earlier bounded range, narratio publish, and artifact-regeneration aliases do not perform cleanup. narratio run-stage cleanup SESSION is the explicit single-stage retry path and applies cleanup's own publication-evidence checks.
• Remove post_publish_cleanup completely. Do not migrate, reinterpret, or retain its types, validators, runner hooks, restore clearing, fixtures, or tests. Keep session manifest schema version narratio.session-manifest.v1: the retired field was optional, and strict decoding will reject manifests that actually contain it without invalidating otherwise canonical v1 state.

Instructions For Every Stage

Before each stage:

1. Read docs/development.md, all documents under docs/policy/, this plan, Workstream 9, and only its cited audit sections. Read the affected portions of ../internal/overview.md, ../internal/workspace.md, ../internal/manifest.md, ../internal/stage-publish.md, ../config.md, ../cli.md, and ../operations.md.
2. Inspect affected production code and focused tests. Prefer the codebase knowledge graph for Go discovery and call tracing; use text search for configuration, schemas, fixtures, documentation, and removal checks.
3. Confirm worktree state and preserve unrelated changes. Do not implement Workstream 10 work that is not made necessary by this cleanup change.

During each stage, use ordinary stage and manifest transitions, keep paths root-confined, preserve group-owned workspace behavior, avoid raw secrets, honor context before destructive work, and keep tests deterministic and offline. Prefer real temporary filesystems over private deletion hooks. Add tests for meaningful destructive, recovery, range, and persistence boundaries; delete tests whose only subject is the retired obligation choreography. Run gofmt on changed Go files and git diff --check. Commit only a completed stage with a short plain-English message, and update each canonical document in the same stage as the behavior it owns.

Stage 1 — Define Cleanup Evidence, Targets, And Configuration Rules

Status: Completed

Goal

Create one deterministic, side-effect-free cleanup plan and reject configurations that can never authorize it.

Required Work

1. Add package-private cleanup planning types in internal/stage. A plan contains the validated published run ID and zero, one, or two root/target/ policy entries in deterministic spool-then-workspace order. Do not register the stage or delete files yet.
2. If both cleanup flags are false, return a disabled plan without requiring publish evidence. Otherwise require and validate the succeeded publish record and its metadata exactly as fixed above. Use artifact identity/key helpers rather than string-prefix approximations; add a narrow helper only if the existing remote-key API cannot validate the commit identity.
3. Construct the spool target with artifacts.SessionSpoolAudioDir and the workspace target with artifacts.SessionRunRootForCampaign, always using the publish metadata's run ID. Validate manifest/config identities before constructing paths. Do not consume Manifest.LocalWorkDir or Manifest.LocalSpoolDir.
4. Add cross-field configuration validation so either cleanup flag requires effective publish.enabled=true and publish.upload_run=true. Preserve existing defaults and field names. Add focused tests for each impossible combination and both valid enabled policies without duplicating exact error prose outside the config owner.
5. Table-test planning for disabled cleanup, each target selection, stable ordering, prior-run publication, missing/failed/skipped publish state, false or mistyped metadata, invalid run identity, wrong commit key, wrong pointer key, and canonical paths. Prove every invalid evidence case produces no plan that can be executed.
6. Update the cleanup fields in docs/config.md in this stage to describe the new validation and their role as final-stage target selectors. Do not yet describe the unregistered stage as implemented elsewhere.

Verification And Exit Criteria

Run go test ./internal/config ./internal/stage -run 'Cleanup|Publish' and git diff --check. Cleanup authorization and targets are deterministic and cannot be derived from the current invocation or uncommitted publication.

Stage 2 — Implement Idempotent Cleanup Stage Behavior

Status: Pending

Goal

Implement cleanup as an ordinary, independently testable stage without yet changing the canonical pipeline.

Required Work

1. Add cleanupStage with name cleanup. Its Run obtains the Stage 1 plan, self-skips when disabled, checks context before each target, and removes each enabled target with the root-scoped filesystem helper.
2. Make fileops.RemoveAllUnderRoot explicitly idempotent when either the configured root or target is already absent. Do not weaken rejection of root deletion, traversal, or out-of-root targets and do not introduce a general filesystem facade.
3. Return a successful StageResult with no outputs, logs, or generated configuration. Keep metadata bounded to the published run and completed policy identifiers; do not persist target paths, completion flags, or an obligation-like structure.
4. Implement cleanup semantic fingerprinting with a new private versioned payload containing normalized roots and policy booleans. Implement ResumeValidator: disabled cleanup is reconsidered through self-skip rather than reusable success; enabled cleanup first validates the current publish evidence and then reports resumable only when all planned targets remain absent.
5. Test through the stage contract with t.TempDir: disabled self-skip; spool-only, workspace-only, and both; preservation of another run and the durable cache; already-absent root/target; cancellation before deletion; refusal of unsafe scope; first-target success followed by second-target failure and successful idempotent retry; no canonical outputs; fingerprint changes; and resume after success versus target recreation. Use real filesystem state rather than adding mutable package globals or a deletion interface solely for tests.

Verification And Exit Criteria

Run go test ./internal/fileops ./internal/stage -run Cleanup, go test -race ./internal/stage -run Cleanup, and git diff --check. The unregistered stage provides safe, idempotent behavior through existing filesystem and manifest contracts.

Stage 3 — Integrate Cleanup Into Canonical Orchestration

Status: Pending

Goal

Make cleanup the normal final stage and remove cleanup from runner finalization.

Required Work

1. Register cleanupStage{} after notify in stage.All; make it selectable by the existing planner and run-stage command without a special command or alias.
2. Extend the validated invalidation relation with notify -> cleanup and an empty cleanup classification. Update transitive expectations so every earlier stage reaches cleanup while preserving the render/extract sibling relationship.
3. Reduce finalizeInvocation to summary construction. Delete its cleanup call and cleanup-specific terminal-failure path. Remove removeScopedDir from invocationDependencies and all normalization/test plumbing made dead by that removal.
4. Delete internal/app/post_publish_cleanup.go. Retain application cleanup helpers only where the explicit clean command still uses them; remove or rename types and wrappers whose only owner was post-publish cleanup.
5. Replace post_publish_cleanup_test.go with proportionate runner-level integration coverage. Prove full order ends in cleanup, notify failure prevents cleanup, publish failure prevents cleanup, cleanup failure leaves publish succeeded and remote state untouched, a later ordinary full run skips reusable upstream work and retries cleanup, and a forced prerequisite stales cleanup.
6. Cover selection semantics: the default full plan includes cleanup; --through notify and earlier ranges do not delete; publish executes only publish; direct run-stage cleanup works with valid evidence and fails safely without it; artifact selection remains accepted only by analyze and publish. Update plan, command, assembled-workflow, bounded-prerequisite, runner, and stage-count expectations without broad snapshots.
7. Update current-behavior owners in the same stage: architecture's publish and cleanup boundary, the internal overview/stage list and lifecycle, docs/cli.md, docs/operations.md, docs/internal/workspace.md, and docs/internal/stage-publish.md. Add a focused docs/internal/stage-cleanup.md and link it from the overview. Clearly state that the standalone publish command does not invoke cleanup and normal resume retries a failed cleanup stage without republishing valid state.

Verification And Exit Criteria

Run go test ./internal/stage ./internal/app, go test -race ./internal/app -run 'Cleanup|Runner|RunStage|Bounded|Plan', go test ./internal/doccheck, and git diff --check. Execution order, range behavior, failure attribution, and retry are ordinary stage behavior, with no post-loop deletion path.

Stage 4 — Remove The Cleanup Obligation From Manifest State

Status: Pending

Goal

Delete the second cleanup state machine and leave stage status as the only durable cleanup progress model.

Required Work

1. Remove CleanupTarget, PostPublishCleanup, and Manifest.PostPublishCleanup; remove their validation, cloning, test helpers, JSON fixtures, and persistence assertions.
2. Remove restore's special clearing of post_publish_cleanup. Restore should treat cleanup exactly like other stage state and must not synthesize or reset a separate obligation.
3. Keep SessionManifestSchemaVersion at v1. Confirm strict decoding rejects a literal post_publish_cleanup field as unknown, while canonical v1 manifests without it still round-trip unchanged. Add no compatibility reader, migration code, or silent field discard.
4. Update docs/internal/manifest.md, restore/workspace internals, operations, and troubleshooting where they own the removed state or recovery advice. Migration guidance should tell operators with an outstanding old obligation to finish cleanup with the pre-upgrade binary or verify remote publication, remove/regenerate the incompatible local manifest, and then use the cleanup stage. Do not present manual JSON editing as the normal migration path.
5. Search production, tests, fixtures, examples, and non-audit documentation for PostPublishCleanup, CleanupTarget, post_publish_cleanup, runPostPublishCleanup, and post-publish cleanup obligation. Remove every live reference; historical roadmap/audit records may retain the terms.

Verification And Exit Criteria

Run go test ./internal/manifest ./internal/app ./internal/stage, go test ./internal/doccheck, and git diff --check. The session manifest contains only the ordinary cleanup stage record, and old obligation-bearing documents fail strict decoding rather than entering a compatibility path.

Stage 5 — Consolidate, Validate, And Close Workstream 9

Status: Pending

Goal

Ensure the reduced implementation, tests, and canonical documentation agree, then record the workstream complete.

Required Work

1. Review the complete diff for duplicated evidence/target logic, obsolete runner branches, redundant wrappers, stale fixtures, accidental remote cleanup reads, and any new per-target persistence. Consolidate only into the narrow owners established above.
2. Confirm the effective behavioral matrix: disabled self-skip; valid publish evidence before deletion; final-stage ordering; bounded exclusion; direct retry; idempotent absence; partial-deletion recovery; publish success surviving cleanup failure; semantic invalidation; and destructive confinement.
3. Check all current-behavior documentation, CLI help, examples, configuration diffs, stage lists, and migration guidance. Volatile details must have one canonical owner and roadmap documents must not be cited as current runtime documentation.
4. Run every repository-wide validation command from docs/development.md, plus git diff --check. Run the scheduled shuffle/repetition command on the affected internal/app, internal/stage, internal/manifest, and internal/fileops packages. Record platform-only limitations without weakening or silently skipping applicable checks.
5. Only after all checks pass, mark every stage in this plan and Workstream 9 in refactoring.md Complete. Leave Workstream 10 as the next pending scope and do not prepare or publish a release.

Verification And Exit Criteria

At minimum run:

go test ./...
go test -race ./...
go test -race -shuffle=on -count=3 ./internal/app ./internal/stage ./internal/manifest ./internal/fileops
go vet ./...
go build ./...
go test ./internal/doccheck
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
git diff --check

The worktree contains one final canonical cleanup stage, no special cleanup obligation or runner side effect, aligned documentation, and no Workstream 10 scope beyond changes required by this behavior.

Open Questions

None. The approved audits, refactoring roadmap, and fixed decisions above make this implementation plan decision-complete.