narratio/docs/roadmap/implementation.md

Workstream 4 Implementation Plan

Purpose And Status

This is the executable plan for refactoring.md, Workstream 4: One durable local manifest authority. Finding 2 of contract-audit.md and rows H14, H15, H17, C02, and C03 of audit-decisions.md own the approved decisions.

All five stages are pending and must be implemented in order. Each stage is bounded for one gpt-5.6-terra prompt. Workstreams 1 through 3 are complete. Workstreams 5 and later are out of scope: do not decompose the runner into a new lifecycle type, redesign pre-execution decisions, add per-artifact analyze checkpoints, split publish into separate transactions, add cleanup as a stage, or change remote lock policy while following this plan.

Settled Implementation Decisions

• The session manifest is the only durable local progress ledger. It alone records stage status, current artifacts, semantic evidence, diagnostics, failures, resume authority, restore authority, and publish authority.
• Delete the structured invocation run manifest completely: its schema and types, store interface and methods, path helper, lifecycle mutations, terminal recovery, analyze projection, publish dependency, fixtures, tests, and current documentation all leave the repository.
• Do not replace it with another JSON summary, journal, database, or event log. Historical per-invocation state is not an operational requirement.
• Preserve run IDs. Every invocation still receives one before session state is initialized; it remains the identity used by the session manifest, run-local directory, adapter provenance, logs, generated configuration, receipts, immutable publish prefix, command summary, and diagnostics.
• Explicitly create the canonical run directory once per invocation after the session lock and identity checks. Removing the initial run-manifest save must not make diagnostic writers responsible for accidentally creating the run root. Directory-creation failure remains an invocation initialization error.
• RunSummary remains the immediate in-memory command result and retains session ID, run ID, session manifest path, selected stage names, executed stages, and skipped stages. Remove only RunManifestPath; do not add a replacement persistent summary.
• A run-manifest write can no longer fail an otherwise valid pipeline action. Session-manifest transition and terminal-save failures remain correctness failures and continue to join with the originating error where applicable.
• Publish no longer opens or validates runs/<run_id>/manifest.json. When upload_run is enabled, it derives the current run root canonically and uses the session manifest's recorded output, log, and generated-configuration references as an allowlist. It includes only safe regular files confined to the current run root and ignores records from earlier runs.
• New remote commits do not upload or classify a run manifest. Keep RemoteArtifactTypeRunManifest as a read-only wire value so already committed and verified remote snapshots remain readable; it must have no local loader, path, special publish branch, or newly emitted declaration. This preserves a durable remote protocol without retaining the local dual-authority model.
• Preserve RemoteArtifactTypeRunFile for declared run-local diagnostics. Publishing no current-run diagnostics is valid and must not require a placeholder file.
• Remove the invocation subset from AnalyzeStateProjection and from analyze execution. During this workstream, analyze still returns and persists one validated reconciled session collection at the existing end-of-stage boundary. Do not implement artifact-by-artifact persistence yet.
• Preserve the current safe behavior when the end-of-stage analyze manifest save fails: incidental output bytes do not establish authority, prior in-memory analyze authority is restored, and the aggregate stage fails. Workstream 6 will replace this with the approved checkpoint model.
• Preserve current stage ordering, force/invalidation rules, self-skip behavior, aggregate transition timing, cleanup obligations, remote commit ordering, and command output except where they directly expose the removed run manifest.
• Delete tests whose only contract is historical run-manifest persistence. Rewrite mixed tests around session state, command summaries, run-local diagnostics, or remote commit contents, and consolidate redundant mechanism assertions rather than recreating a second audit record in tests.
• Historical release notes and completed audit documents remain unchanged. Current architecture, CLI, operations, troubleshooting, and internal documentation must describe session-only authority.
• Mark only Workstream 4 Complete, and only after all stages and final validation succeed.

Instructions For Every Stage

Before each stage:

1. Read docs/development.md, every file under docs/policy/, this plan, Workstream 4 plus the dependency and target sections of refactoring.md, Finding 2 and the Workstream-4-relevant boundary in Finding 3 of contract-audit.md, and rows H14, H15, H17, C02, and C03 of audit-decisions.md.
2. Inspect the named implementation and focused tests. Prefer the codebase graph for Go discovery and text search for schemas, paths, fixtures, documentation, and serialized field names.
3. Confirm the worktree state and preserve unrelated changes. Do not enter a later workstream merely because adjacent lifecycle, analyze, publish, or cleanup code would benefit from broader refactoring.

During every stage:

• Keep the repository compiling and focused tests passing. A temporary invocation projection or unused run-manifest domain type may remain only when the next numbered stage explicitly removes it; do not introduce a new compatibility facade.
• Preserve package ownership and acyclic dependency direction. Keep this a deletion-oriented refactor inside the existing app, stage, manifest, and artifacts owners.
• Preserve the session lock, session-manifest atomic replacement, group workspace modes, bounded I/O, no-follow publish sources, remote verification, secret redaction, and Linux/macOS/Windows support.
• Update the canonical current-behavior document owned by a change in the same stage. Do not edit historical release notes or completed audit findings.
• Keep tests offline, deterministic, and behavior-focused. Remove tests with the retired contract and retain consequential session-state, data-integrity, recovery, and remote-commit coverage at the narrowest stable boundary.
• Run gofmt on changed Go files, focused tests, and git diff --check. Mark a stage Completed only after its exit criteria pass.

Stage 1 — Decouple Publish From The Run Manifest

Status: Completed

Goal

Make publish consume only the session manifest and current run directory so the runner can stop producing a structured run manifest without losing useful diagnostic uploads.

Required Work

1. In internal/stage/publish.go, replace resolvePublishRunManifestPath with canonical current-run-root resolution based on resolved campaign, session ID, workspace root, and m.RunID using artifacts.SessionRunRootForCampaign. Require that exact root to be a real directory and retain existing symlink/non-directory rejection.
2. Remove opening, decoding, rewinding, and passing a run-manifest file. Change current-run collection to inspect manifest.Manifest stage records and add only declared ArchivePath, fallback LocalPath, Logs, and GeneratedConfigs that resolve within the selected current run root. Records referring to canonical session files or an older run must not be uploaded as current-run diagnostics.
3. Make an empty current-run diagnostic set valid. Refactor the source-opening helper so it opens each allowlisted current-run file directly, with the existing regular-file/no-follow and optional checksum verification. It must not require a distinguished manifest file.
4. Remove the manifest.json synthetic run-file entry and the special branch that emits RemoteArtifactTypeRunManifest. Continue emitting ordinary current-run diagnostics as RemoteArtifactTypeRunFile with deterministic run:<relative-path> source identities and destinations.
5. Preserve session-manifest commit generation, previous-cache uploads, configured published outputs, immutable upload verification, lock revalidation, commit upload, and pointer-last ordering unchanged.
6. Rewrite focused publish tests and helpers to create a run root and populate session-manifest diagnostic references instead of writing a run manifest. Cover current-run logs/generated configuration/declared archive outputs, exclusion of prior-run and session-canonical paths, empty diagnostics, symlink rejection, deterministic ordering, and absence of newly emitted run_manifest declarations. Retain existing upload integrity and pointer tests without duplicating them.
7. Update docs/internal/stage-publish.md and the run-upload portion of docs/operations.md: the session manifest is the diagnostic allowlist, no run manifest is uploaded, and upload_run may legitimately upload zero run-local diagnostic files.

Exit Criteria

• go test ./internal/stage ./internal/artifacts succeeds.
• Publish succeeds when the current run directory exists but contains no run manifest and no diagnostics.
• Only session-manifest-declared files confined to the current run root become run_file transfers; older-run and incidental files do not.
• New commits contain no run_manifest artifact, while remote decoding still accepts that historical wire value.
• git diff --check succeeds.

Stage 2 — Make Runner Persistence Session-Only

Status: Completed

Goal

Remove the second persistence transaction from invocation execution while preserving the existing session-ledger lifecycle and immediate command result.

Required Work

1. Remove RunOptions.RunManifestStore and RunSummary.RunManifestPath. Keep all remaining summary fields and run ID generation. Update direct callers and CLI rendering only if they refer to the removed path.
2. In executePlan, delete run-manifest path resolution, construction, identity/config projection, initial save, per-stage action/running/skip/ failure/success saves, final success save, and every error path created only by those operations.
3. After acquiring the session lock, validating/restoring session state, and resolving invocation identity, explicitly ensure the canonical SessionRunRootForCampaign directory with the ordinary workspace directory contract. Do not create manifest.json or another placeholder there.
4. Replace persistTerminalFailure with a narrowly named session-only helper. It must record the terminal failure in the session manifest when available, atomically attempt the session save, preserve the originating error, and join a session-save error without invoking any diagnostic store. Keep the distinct post-publish-cleanup failure behavior until its later workstream.
5. Simplify all runner failure paths to the new helper. Preserve the existing order of session identity save, resume validation, running transition, force invalidation, stage execution, success/failure/self-skip transition, session save, logging, and post-publish cleanup.
6. Make applyAnalyzeProjection session-only at its application boundary so the runner no longer needs a run-manifest value. The invocation subset may remain temporarily inside validation until Stage 3 deletes it.
7. Delete run-only helpers such as invocationIdentity.applyToRunManifest, requestedStageNames, applyStageResultToRunManifest, applyEffectiveConfigProvenanceToRun, and setRunStageSemanticConfig once unused.
8. Rewrite runner, terminal-failure, semantic-resume, assembled-workflow, and lifecycle tests that currently inject a RunStore or load RunSummary.RunManifestPath. Assert the durable outcome through the session manifest and the immediate invocation outcome through RunSummary. Preserve coverage for cancellation, collaborator/preflight failures, session-save failure joining, successful and failed stage transitions, self-skip, force invalidation, semantic evidence, distinct run IDs, and run-local diagnostic paths. Delete tests whose sole purpose was run-record creation, terminalization, or historical inspection.
9. Update docs/policy/architecture.md, docs/cli.md, and the runner portions of docs/internal/manifest.md: one atomic session-manifest update is the only durable progress transaction; a run ID and run directory remain, but there is no invocation manifest or persistent command summary.

Exit Criteria

• go test ./internal/app ./internal/stage ./internal/manifest succeeds.
• executePlan creates no runs/<run_id>/manifest.json and exposes no run manifest store seam, while run IDs, run directories, diagnostics, session transitions, and summaries remain available.
• A hypothetical diagnostic-record write cannot fail a stage because no such persistence operation exists; session-manifest failures remain visible and authoritative.
• Runner behavior tests obtain durable state only from the session manifest.
• git diff --check succeeds.

Stage 3 — Remove Invocation-Only Analyze Projection

Status: Pending

Goal

Give analyze one in-memory projection—the reconciled session collection—without prematurely implementing Workstream 6 checkpoint semantics.

Required Work

1. Remove Invocation from stage.AnalyzeStateProjection and update its comment to describe only the reconciled session authority proposed for the next session-manifest save. Retain the pointer wrapper so “no analyze update” remains distinguishable from a valid empty collection.
2. In internal/stage/analyze.go, delete the separate invocation-record map and every write, clone, or failure projection associated only with it. Successful and failed analyze results continue to return the complete reconciled session collection under the existing rules.
3. Simplify internal/app/analyze_projection.go: validate and clone only the session collection, remove invocation/session subset comparison, apply only the session collection, and derive aggregate analyze outputs only from that collection. Retain rejection of projections from non-analyze stages, contradictory ordinary outputs, malformed records, and illegal skipped or error result shapes.
4. Preserve captureAnalyzeState and restoreAnalyzeState for the current single-save boundary. Do not add manifest-store access to the analyze stage, persist inside its artifact loop, or change retry/force behavior in this stage.
5. Rewrite analyze_projection_test.go, analyze incremental tests, and mixed runner tests around one reconciled session collection. Delete assertions about per-invocation artifact subsets. Retain behavior coverage for partial failure state, unrelated current records, invalid projections, persistence failure rollback, and aggregate output derivation.
6. Update docs/internal/stage-analyze.md and the analyze-state section of docs/internal/manifest.md to describe one session projection and clearly leave per-artifact checkpointing as future roadmap work rather than current behavior.

Exit Criteria

• go test ./internal/stage ./internal/app ./internal/manifest succeeds.
• No production analyze type, result, validator, or test distinguishes session state from an invocation artifact subset.
• Analyze still performs one end-of-stage session save, retains valid unrelated records, persists its permitted failure projection, and restores prior in-memory authority if that save fails.
• No Workstream 6 checkpoint callback or persistence loop has been introduced.
• git diff --check succeeds.

Stage 4 — Delete The Run-Manifest Domain Model

Status: Pending

Goal

Remove the now-unreferenced structured schema, persistence API, canonical file path, and mechanism tests from their owning packages.

Required Work

1. Delete internal/manifest/run_manifest.go, RunManifestSchemaVersion, RunManifest, RunStageRecord, run status/action enums, constructors, transition methods, and all run-only validation/normalization helpers.
2. Remove manifest.RunStore and LocalStore.CreateRun, LoadRun, LoadRunReader, and SaveRun. Simplify imports and shared validation only where the deleted run code made them unnecessary; do not weaken session manifest validation.
3. Delete artifacts.SessionRunManifestPathForCampaign while retaining SessionRunsDirForCampaign and SessionRunRootForCampaign as canonical run-diagnostic and invocation-work paths.
4. Delete run_manifest_test.go and run-manifest-only store/path/schema tests. Remove run-specific portions of otherwise valuable manifest, artifact, semantic fingerprint, restore, operator, and publish fixtures. Do not replace serialization round-trip tests for a format that no longer exists.
5. Keep the remote run_manifest artifact enum accepted only by remote commit decoding/validation. Add a concise comment at its declaration explaining that Narratio no longer emits or interprets it and retains the value solely so immutable existing commits remain readable. No production symbol in manifest, app, or stage may refer to it.
6. Update package comments or focused internal documentation affected by the domain deletion in the same change.

Exit Criteria

• go test ./internal/manifest ./internal/artifacts ./internal/app ./internal/stage succeeds.
• Except for the explicitly retained RemoteArtifactTypeRunManifest wire value and its remote-decoder compatibility test, searches for RunManifest, RunStore, CreateRun, LoadRun, SaveRun, RunManifestPath, run_manifest.go, and the local runs/<run_id>/manifest.json contract find no current production or test implementation.
• Session manifest schema validation and round trips remain fully covered.
• Existing remote commit documents containing the historical run_manifest artifact type still decode and validate, but new publish plans cannot emit it.
• git diff --check succeeds.

Stage 5 — Reconcile Current Documentation And Close Workstream 4

Status: Pending

Goal

Prove the session-only target state across the repository and leave one accurate implementation narrative for the next workstream.

Required Work

1. Search current production code, tests, fixtures, examples, scripts, and nonhistorical documentation for run-manifest types, paths, JSON fields, stores, publish classifications, invocation projections, and dual-save or terminal-synchronization language. Remove stale surfaces or explain the retained run ID/run directory without implying a structured record.
2. Reconcile canonical owners, especially docs/policy/architecture.md, docs/cli.md, docs/operations.md, docs/troubleshooting.md, docs/internal/overview.md, docs/internal/manifest.md, docs/internal/stage-analyze.md, docs/internal/stage-extract.md, and docs/internal/stage-publish.md. Keep schema definitions in the manifest owner, filesystem/remote behavior in operations, and implementation details in internal documents rather than duplicating them.
3. Do not edit immutable historical release notes or completed audit documents. Roadmap references may retain the phrase “run manifest” when describing the problem and approved removal. Current instructions must not claim the file is produced or required.
4. Review the affected test suite under docs/policy/testing.md. Consolidate leftover tests that merely encode the deleted write choreography, while retaining meaningful session durability, transition, recovery, current-run diagnostics, remote publication, and representative assembled-workflow coverage.
5. Update Workstream 4 in refactoring.md from Pending to Complete and update every stage plus the purpose/status section of this plan to Completed only after the target state and all checks are proven. Do not change later workstream statuses.
6. Run the complete repository validation from docs/development.md: go test ./..., go test -race ./..., go vet ./..., go build ./..., go test ./internal/doccheck, and go test ./internal/config -run '^TestExamplesLoadAndValidate$'. Run git diff --check and confirm the worktree contains only intended changes.

Exit Criteria

• The session manifest is the only structured durable local progress record; no current code or documentation requires a run manifest.
• Run IDs, canonical run directories, bounded diagnostics, generated configuration, receipts, immediate summaries, immutable run prefixes, and remote publication remain functional without a second progress transaction.
• Publish uses session-manifest evidence, emits no new run-manifest artifact, and still reads verified historical remote commits without interpreting their old diagnostic record.
• All repository-wide validation succeeds on the maintained Go toolchain.
• Workstream 4 and all five stages are marked Complete; Workstream 5 remains the next pending workstream.
• git status --short is empty after the implementation commit.

Open Questions

None. The approved audits, current architecture, and decisions above are sufficient to implement this workstream without further product or policy choices.