narratio/docs/roadmap/implementation.md

Runner Lifecycle Refactoring Implementation Plan

Status And Purpose

This is the decision-complete implementation plan for Workstream 5, Runner lifecycle and pre-execution decisions, in refactoring.md. Workstreams 1 through 4 are complete. Workstream 5 is the next pending workstream and is the only feature scope of this plan.

The implementation must reduce the change blast and cognitive load of internal/app.executePlan without changing the pipeline's observable behavior. It must leave one explicit lifecycle owner, one session lock, one durable session-manifest authority, a narrow shared planning/execution decision model, and invocation-scoped test seams. It must not create a workflow engine, a general dependency-injection framework, or new public extension points.

The authoritative scope and target state are Workstream 5, the dependency relationships, and the consolidated target state in refactoring.md. The supporting audit decisions are H04, H07, H14, H15, H17, H34, and H35 in audit-decisions.md, with detailed analysis in audit.md.

Scope And Fixed Decisions

• Keep the existing internal/app package boundary. Use private types, functions, and focused files; do not create a runner package or expose lifecycle internals.
• Keep the session manifest as the sole durable local progress ledger. Run IDs, run directories, diagnostics, and RunSummary remain, but no second durable invocation record may be introduced.
• Preserve the canonical flat stage order, bounded contiguous execution, dependency-aware invalidation outside the selected range, stage-owned resume validation, self-skip semantics, force semantics, and current analyze projection behavior.
• Preserve the two prerequisite checks: a read-only check before persistent mutation and a second check against the manifest while holding the session lock.
• Keep lock acquisition and the deferred release visibly paired in the top-level lifecycle function. A release failure remains the final invocation error when no earlier error exists and is joined with an earlier error otherwise.
• Preserve manifest ordering: establish identity before adapter construction; persist running before invoking a stage; persist terminal stage state before moving to the next stage; and persist terminal invocation failures where the current runner does so. Do not make diagnostics a persistence authority.
• Keep actual Stage.Run results, result validation, persistence, logging, and failure handling execution-only. Planning may model a successful or configured self-skipped outcome on a cloned manifest, but must never pretend that it observed a runtime result.
• The shared pure decision value must describe only whether to run or skip and the pre-execution stale/invalidation work implied by recorded status, force, and normalized resume validation. Small transition helpers may be shared only where planning and execution perform the same manifest mutation.
• Preserve analyze's current single end-of-stage projection/save boundary and rollback on save failure. Per-artifact checkpointing belongs to Workstream 6 and is out of scope.
• Preserve post-publish cleanup and its durable obligation behavior until Workstream 9. This work may move the call into finalization and make its destructive function invocation-scoped, but must not turn cleanup into a stage or change eligibility/retry semantics.
• Continue using stage.Env for injected stage collaborators. Add only small private dependency values for nondeterministic or external construction boundaries that cannot be represented by stage.Env, and pass those values explicitly per command or invocation.
• Remove executeStagesFn and runCommandFn. Replace their tests with private command helpers that accept a pipeline executor value for that call. Do not migrate unrelated restore/config-loader globals in this workstream; their owners should adopt the same narrow pattern when later work touches them.
• Replace the runner's mutable clock use with an invocation-scoped clock. A normal immutable nowUTC() function may remain for unrelated command owners. Route post-publish deletion through the invocation dependency value and remove removeRunScopedDirFn.
• Preserve current CLI syntax, output, errors at their stable semantic level, configuration behavior, workspace modes, secret loading/redaction, remote lock revalidation, Linux/macOS/Windows support, and offline operation.
• Do not optimize for a particular private file layout or exact helper count. The finished top-level runner must nonetheless read as a short, explicit sequence of preparation, lock ownership, locked initialization, composition, stage execution, and finalization rather than forwarding all work to another monolith.
• Mark only Workstream 5 Complete, and only after all stages and full validation succeed. Workstream 6 remains the next pending workstream.

Instructions For Every Stage

Before each stage:

1. Read docs/development.md, every document under docs/policy/, this plan, Workstream 5 plus the dependency and consolidated-target sections of refactoring.md, and audit rows H04, H07, H14, H15, H17, H34, and H35 in audit-decisions.md. Consult the corresponding detailed findings in audit.md when a boundary is unclear.
2. Inspect the named production code and focused tests. Prefer the codebase knowledge graph for Go symbol discovery and call tracing; use text search for documentation, fixtures, serialized fields, and global-variable assignments.
3. Confirm the worktree state and preserve unrelated changes. Do not enter Workstream 6 or Workstream 9 merely because analyze or cleanup code is adjacent to runner lifecycle code.

During every stage:

• Keep the repository compiling and focused tests passing. Do not leave both an old and new runner path active except for a temporary private delegation that the next numbered stage explicitly removes.
• Preserve package ownership and acyclic dependency direction. Private state should aggregate already-related values, not become a service locator.
• Preserve error wrapping and joining where errors represent distinct failures. Do not freeze incidental complete error strings in new tests.
• Keep tests deterministic, offline, behavior-focused, and lean under docs/policy/testing.md. Prefer existing real temporary filesystems and current adapter/storage fakes. Do not add mock choreography for private lifecycle helpers.
• Update the canonical current-behavior document owned by a changed behavior in the same stage. Pure file movement or private extraction requires no user documentation change.
• Run gofmt on changed Go files, focused tests for the changed boundary, and git diff --check. Mark a stage Completed only after its exit criteria pass.

Stage 1 — Unify Pre-Execution Decisions

Status: Completed

Goal

Give planning and execution one pure representation of the decision made before a stage attempt, while keeping planning predictions distinct from observed runtime outcomes.

Required Work

1. In internal/app/run_control.go or a focused private file, replace the current action-only decision with an unexported value that records: run versus skip; whether the selected successful record must first be marked stale; the stage-owned reason for that stale transition; and the application-owned invalidation reason, if any. Its constructor must be pure: it receives the prior manifest status, force, and an already normalized resume-validation result and does not mutate state, persist, log, read files, invoke a stage, or obtain the current time.
2. Encode these cases explicitly:
   • a non-forced succeeded record with no validator result or a resumable result skips;
   • a non-forced succeeded record with a non-resumable result runs after the selected record is staled with the validator's reason and succeeded dependents are invalidated with staleReasonNotResumable;
   • a forced stage runs and invalidates succeeded dependents with staleReasonForcedReplacement when the attempt begins; and
   • absent, failed, stale, or skipped aggregate records run without inventing a resume-stale transition.
3. Add a small helper that applies only the identical pre-execution manifest changes described by the decision at a caller-supplied timestamp. Retain the existing fixed invalidation relation and stable application-owned reasons. The caller remains responsible for persistence and error context.
4. Use the decision constructor and application helper from both Plan and executePlan. Each caller may perform stage-owned resume validation before constructing the decision. Preserve analyze planning's read-only artifact details when analyze is already a run candidate; those details are preview information, not an execution outcome.
5. Extract only the exact shared attempt/result mutations already identical in both paths: beginning an attempt (running plus forced invalidation), applying a configured/runtime self-skip and its conditional invalidation, and applying a predicted/actual success and its conditional invalidation. Require callers to supply the actual or modeled outcome, outputs, semantic fingerprint, prior outcome, and timestamp. Do not put Stage.Run, result validation, logging, or manifest saves in these helpers.
6. Rewrite modelPlannedStageRun around these values or delete it if the resulting focused helpers make it redundant. Keep all planning mutation on the cloned manifest.
7. Add table-driven decision tests for every status/force/resume combination and focused parity tests proving that plan and execution produce the same run/skip choice and dependent-stale effects for ordinary skip, non-resumable success, forced replacement, unchanged self-skip, changed self-skip, and a successful result replacing a non-success record. Assert durable state and decision semantics, not private helper call order or exact timestamps.

Exit Criteria

• go test ./internal/app succeeds.
• Plan and executePlan no longer independently encode the same pre-execution stale/force decision sequence.
• The decision constructor is pure, and actual stage outcomes and persistence remain outside it.
• Planning remains read-only and execution behavior is unchanged.
• git diff --check succeeds.

Stage 2 — Establish The Invocation Runtime And Lock Lifecycle

Status: Completed

Goal

Make invocation state cohesive and make the top-level transaction order obvious without hiding session-lock ownership.

Required Work

1. Introduce an unexported invocationRuntime (or equivalently narrow private type) under internal/app. It should carry only state already shared across lifecycle phases: context, resolved configuration, bounded plan/stages, normalized options, effective artifacts, run identity and canonical paths, stage.Env, manifest store, artifact store, loaded session manifest, and summary accumulators. Do not expose it or let stages depend on it.
2. Extract unlocked preparation that performs the current read-only bounded prerequisite inspection, resolves effective artifacts, generates the run ID, resolves invocation identity, initializes/clones stage.Env fields, and supplies default manifest/artifact stores and logger. Preserve the rule that injected Env values provide collaborators but never an alternate config, selected-artifact set, or effective artifact set.
3. Extract layout preparation separately. Keep EnsureLayoutFor before lock acquisition as today, verify its manifest path against the canonical resolved path, and return enough state for the locked lifecycle.
4. In the short top-level executor, acquire the session lock after unlocked preparation and visibly install one deferred release beside that acquisition. Preserve exact release-error joining behavior. Do not put lock ownership in invocationRuntime, a constructor, or a helper whose lifetime is ambiguous.
5. Extract locked initialization that checks the incomplete-restore marker, loads and validates or creates the session manifest, rechecks bounded prerequisites, applies identity and effective-configuration provenance, saves the identity transition, ensures the canonical run directory, and loads configured secrets. All steps after manifest availability that currently record a terminal failure must continue to do so through the session-only failure helper.
6. Keep pre-lock failures free of persistent mutation and keep all later manifest transitions protected by the same held lock. Do not add rollback, a second lock, or a second manifest authority.
7. Adapt existing bounded-run, restore-marker, identity, run-directory, terminal-failure, and lock-release tests to exercise the assembled lifecycle. Add coverage only if the two-check prerequisite ordering or lock-release error behavior is not already credibly protected.

Exit Criteria

• go test ./internal/app ./internal/artifacts ./internal/manifest succeeds.
• The top-level executor visibly performs unlocked preparation, acquires and defers release of exactly one session lock, initializes locked state, and proceeds to execution.
• Invalid preflight requests create no persistent invocation state; the locked prerequisite recheck still prevents races with a prior invocation.
• The private runtime has cohesive fields and no public or cross-package API.
• git diff --check succeeds.

Stage 3 — Isolate Collaborator Composition

Status: Completed

Goal

Make production collaborator construction one bounded lifecycle phase that is selective, testable per invocation, and independent of stage transitions.

Required Work

1. Define a small unexported invocation dependency value with explicit defaults for the nondeterministic and external construction boundaries used by this runner: UTC time, run-ID generation, default adapter/object-store creation, effective remote-lock loading/revalidation support, and post-publish scoped removal. Include a field only where tests need control over a real external, destructive, clock, or randomness boundary; continue injecting completed stage collaborators through stage.Env.
2. Add a private executor variant that accepts that dependency value and have the ordinary executor supply production defaults. Normalize missing fields once at the entry boundary so lifecycle methods never fall back to mutable package state. Do not export the dependency type or store it globally.
3. Route run-ID creation and every runner timestamp through the invocation dependencies. Convert nowUTC to an ordinary function for unrelated command owners if it no longer needs to be mutable; do not broaden this stage into a restore lifecycle refactor.
4. Extract collaborator composition from the runner into one operation over the invocation runtime. Preserve current selection gates exactly: WhisperX only for transcribe; Seriatim only for merge, normalize, trim, or render; Audita only for polish; Notarius only when selected extraction requires it; Scriptorium only for trim or analyze; object storage only when selected stages/effective artifacts require it; remote lock loading only when needed; and the no-op notifier only for notify.
5. Preserve injected collaborators, secret-loaded environment behavior, effective lock merging, and the publish-time revalidation closure. A composition failure after identity initialization remains a persisted terminal session failure with its existing useful context.
6. Route post-publish cleanup's scoped deletion through the invocation dependency value and delete removeRunScopedDirFn. Preserve cleanup obligation creation, per-target persistence, retry, and failure behavior.
7. Consolidate selective-composition tests into table-driven or focused behavioral cases. Test that an unselected external boundary is not constructed only where avoiding that construction is an operational requirement; do not assert private helper ordering.

Exit Criteria

• go test ./internal/app succeeds, including composition, remote-lock, cleanup, and bounded-stage tests.
• Collaborator construction is absent from the top-level runner and stage loop, but remains an application-owned production-composition concern.
• Runner time, randomness, external construction, and destructive cleanup are controllable per invocation without mutable runner globals.
• Selected-stage composition and terminal-failure behavior remain unchanged.
• git diff --check succeeds.

Stage 4 — Extract One-Stage Execution

Status: Pending

Goal

Give one private operation complete ownership of a single stage attempt while preserving every durable transition and failure boundary.

Required Work

1. Extract an operation on the invocation runtime that executes exactly one selected stage and returns a small execution-owned outcome sufficient to update the in-memory summary or stop the invocation. It may coordinate the shared decision/transition helpers but must not select the next stage or release the session lock.
2. Within that operation, preserve this order:
   • compute the stage semantic fingerprint;
   • run stage-owned resume validation when the recorded success is a skip candidate;
   • build and apply the pure pre-execution decision and persist any non-resumable stale/invalidation transition;
   • return a skip outcome without invoking the stage when still resumable;
   • capture prior aggregate/analyze state, begin the attempt, persist running, and then invoke Stage.Run;
   • validate the stage result and any analyze projection;
   • apply failure, self-skip, or success transitions and required dependent invalidation; and
   • persist the terminal state before returning.
3. Preserve failure semantics for semantic fingerprinting, resume validation, invalidation, the pre-run save, stage execution, result validation, self-skip/success saves, and terminal failure persistence. A failed stage or persistence operation stops later stages. Distinct invalidation/save errors remain joined with the originating failure.
4. Preserve current analyze behavior exactly: a valid partial failure projection may be applied before recording failure; a successful projection supplies aggregate outputs; and failure to save a successful projection restores prior analyze authority before recording aggregate failure. Introduce no callback/store dependency into internal/stage.
5. Keep manifest save calls explicit in the one-stage owner. Do not hide them in generic state-machine hooks or teach manifest.Store orchestration policy.
6. Reduce the outer stage loop to call the one-stage operation, append its executed/skipped classification, and stop on error. Preserve deterministic StageNames, Executed, and Skipped order and current logging semantics.
7. Reconcile runner tests around observable manifest state, invocation summary, stage calls, and errors. Preserve coverage for ordinary skip, force, non-resumable state, self-skip, result validation, cancellation, downstream invalidation outside a bounded range, save failures, analyze rollback, and stopping after failure. Remove redundant tests that exist only to pin the old monolithic control flow.

Exit Criteria

• go test ./internal/app ./internal/stage ./internal/manifest succeeds.
• The stage loop contains no duplicated transition policy and one-stage execution has one clear persistence owner.
• A later stage is never attempted after a stage or required manifest save fails, and no failed/interrupted attempt is presented as successful.
• Analyze still uses its pre-Workstream-6 aggregate persistence boundary.
• git diff --check succeeds.

Stage 5 — Isolate Finalization And Simplify The Lifecycle Owner

Status: Pending

Goal

Complete the runner decomposition so the lifecycle and transaction boundaries are easier to read than the former monolith.

Required Work

1. Extract finalization that performs the current post-publish cleanup only when publish actually executed, using the runtime's session manifest, manifest path, environment, executed-stage list, and invocation-scoped deletion dependency. Preserve the special cleanup-failure persistence and the fact that a committed remote publish remains authoritative.
2. Build RunSummary from the runtime's resolved identity and ordered accumulators only after all selected stages and required cleanup complete. Continue returning no success summary on invocation failure.
3. Make the top-level executor an explicit sequence: unlocked preparation; layout preparation; visible lock acquisition/deferred release; locked state initialization; collaborator composition; ordered one-stage execution; and finalization/summary. It must retain meaningful error context at each phase and must not delegate that entire sequence to a new monolithic helper.
4. Place private lifecycle concerns in focused files according to ownership—for example runtime/preparation, composition, stage execution, and finalization— while keeping exact shared transition logic in run_control.go. Remove dead helpers, stale comments, duplicate state, and imports left by extraction.
5. Review the resulting runtime for service-locator growth. Remove fields used by only one local operation unless retaining them materially clarifies the lifecycle. Keep collaborators in stage.Env and persistence/path policy in their existing owners.
6. Run focused assembled-workflow, cleanup, bounded-run, resume, manifest-save, and release-error tests. Add no tests for private file placement, helper count, or line count.

Exit Criteria

• go test ./internal/app ./internal/stage ./internal/manifest ./internal/artifacts succeeds.
• executePlan is a short, readable lifecycle owner with one visibly scoped lock and no nested stage state machine or collaborator-construction block.
• Cleanup remains a post-publish obligation with unchanged behavior, awaiting Workstream 9.
• The refactor has removed obsolete code rather than retaining parallel old and new lifecycle paths.
• git diff --check succeeds.

Stage 6 — Remove Global Runner Seams And Close Workstream 5

Status: Pending

Goal

Prove the target architecture through invocation-scoped command tests, accurate current documentation, and complete repository validation.

Required Work

1. Introduce the smallest private command-level helper needed to pass a pipeline-executor function for one call through Execute, Run, RegenerateArtifacts, and the single-stage command path. Public command functions continue to use the production executor. The alias must still be exactly equivalent to run --force --from extract --through analyze with caller options forwarded under the existing rules.
2. Delete executeStagesFn and runCommandFn. Rewrite affected CLI, profile, remote-session, restore-workflow, assembled-workflow, bounded-run, alias, and artifact-selection tests to call the private command helper with an explicit executor or to exercise the real assembled runner. Do not serialize tests around package-global save/replace/restore choreography.
3. Confirm that removeRunScopedDirFn is gone and that runner tests do not replace nowUTC. Inventory remaining mutable function variables in internal/app; leave unrelated configuration-loading and restore seams for their owning future work, but ensure no runner, run-command, alias, or post-publish-cleanup test mutates process-global behavior.
4. Review test value under docs/policy/testing.md. Consolidate duplicate command-wiring and lifecycle cases exposed by the new narrow seams while retaining consequential lock, persistence, invalidation, recovery, cancellation, external-boundary, and representative assembled-workflow coverage.
5. Update current internal documentation where ownership requires it: docs/internal/overview.md should summarize the explicit runner lifecycle; docs/internal/manifest.md should describe the shared pre-execution decision and single persistence owner without documenting private helper names; and docs/internal/adapters.md should describe selective application-owned composition. Update architecture policy only if implementation reveals a normative boundary not already stated. Do not duplicate CLI/configuration contracts or edit historical release documents and completed audit findings.
6. Search current code and documentation for stale claims about the monolithic runner, independent plan/run decisions, or mutable runner seams. Remove only inaccurate current-behavior text; roadmap and audit history remain valid.
7. Update Workstream 5 in refactoring.md from Pending to Complete, and change all six stage statuses plus this plan's status to completed only after the target state and all checks are proven. Leave Workstream 6 Pending.
8. Run the complete repository validation from docs/development.md: go test ./..., go test -race ./..., go vet ./..., go build ./..., go test ./internal/doccheck, and go test ./internal/config -run '^TestExamplesLoadAndValidate$'. Also run git diff --check and confirm the implementation commit leaves no unintended worktree changes.

Exit Criteria

• All repository-wide validation succeeds on Go 1.26.
• Planning and execution share one pure pre-execution decision model while actual results and persistence remain execution-owned.
• The runner reads as an explicit lifecycle with one session lock and one session-manifest authority.
• Runner, run-command, alias, and cleanup tests use invocation-scoped dependencies rather than mutable package-global function substitution.
• Current internal documentation accurately describes the implemented design at the appropriate level.
• Workstream 5 and all six stages are marked Complete; Workstream 6 is the next pending workstream.
• git status --short is empty after the implementation commit.

Open Questions

None. The approved audits, current policies, and Workstream 5 decisions resolve the behavior, architecture, testability, and sequencing choices required for implementation.