narratio/docs/roadmap/refactoring.md

Narratio Refactoring Roadmap

Status

Consolidated roadmap for the upcoming refactoring work. This document defines the approved actionable scope and target state. It does not describe current behavior and is not a staged implementation plan.

The complete audit inventory and final dispositions are recorded in the Audit Decision Register. Supporting analysis remains in the three source audits. This roadmap includes only work that at least one audit identified and that the final decision register classifies for immediate action.

Goal

Reduce Narratio's implementation and maintenance burden while preserving the operational outcomes that matter: correct resumable progress, economical reuse of expensive generated artifacts, coherent remote publication, understandable cleanup and recovery, safe destructive scope, group-owned workspaces, and protection of API keys.

The refactor deliberately permits approved internal and deprecated-contract changes. It is not limited to producing an implementation mechanically equivalent to the current one.

Governing Boundaries

• Keep the existing package-level ownership and acyclic dependency direction. Prefer private types, functions, and focused files over new architectural layers.
• Treat campaign and session workspaces as trusted, intentionally shareable, group-owned data. API keys remain sensitive, and cleanup and restore remain confined to explicit application-owned roots.
• Keep the session manifest as the sole durable local progress authority.
• Preserve strict, deterministic configuration semantics without preserving incidental diagnostic order, complete wording, punctuation, or private validation choreography.
• Keep integration-specific requests, validation, and errors with their adapters. Shared infrastructure should expose only exact stable mechanics.
• Add dependencies selectively when they remove audited complexity and preserve the required supported-platform behavior.
• Update tests and canonical documentation with the behavior they own rather than preserving retired implementation mechanisms.

Actionable Workstreams

Each workstream uses one of three status identifiers: Pending, In Progress, or Complete. Workstreams are numbered in dependency-aware implementation order. The first workstream not marked Complete is the next workstream to address unless its status or an implementation plan explicitly records a blocker.

Workstream 1 — Go baseline, process supervision, and transcription concurrency

Status: Complete

Audit sources: H11, H19, H22, H24, H33, and L01.

Underlying issue

The maintained Go baseline prevents use of newer root and process primitives, the Windows subprocess owner reopens a PID instead of using a lifetime-stable handle, transcription manually implements bounded cancellation and first-error coordination, and the shared subprocess package mixes generic process ownership with Narratio environment, diagnostic, redaction, and file-writing policy.

Required changes

• Upgrade go.mod, generated module state, every maintained CI and release Go image, and release documentation to Go 1.26 in one bounded change. Go 1.26 provides os.Process.WithHandle; the required os.Root.RemoveAll operation was already available in Go 1.25.
• Validate the upgrade with the repository-wide tests, race checks, vet, maintained documentation and examples, and all supported release builds.
• Assign the suspended Windows child to its Job Object inside os.Process.WithHandle. Preserve suspended creation, job configuration, initial-thread resumption, fail-closed cleanup, and job disposal.
• Replace the transcription worker pool with errgroup.WithContext and SetLimit. Preserve speaker-qualified failures, parent-cancellation context, aggregation of successful results under a narrow lock, and deterministic output ordering.
• Clarify the internal subprocess boundary: the reusable mechanism owns process tree start, wait, termination, reaping, and typed outcomes; Narratio owns environment selection, secret-value selection and redaction, bounded log sinks, paths, modes, diagnostic wording, and tool-specific hints.
• Remove atomic YAML and byte-writing helpers from the subprocess package and route file replacement through the focused filesystem owner.

Target state

Narratio builds and tests on Go 1.26, uses the safer native Windows handle API, and has concise bounded transcription concurrency. Process-tree ownership is internally separable from Narratio policy without creating or depending on a new external subprocess library during this refactor.

Workstream 2 — Filesystem simplification and restore correctness

Status: Complete

Audit sources: H02, H06, H07, H18, H20, H21, H29, H45, H46, C04, L02, L05a, and L05c. Supporting filesystem rationale and policy are recorded in fileops.md; the requirements below are the consolidated action scope.

Underlying issue

The current FileOps subsystem promises adversarial race resistance, universal confinement and no-follow behavior, atomic no-replace directory promotion, and power-loss durability that Narratio's trusted, session-locked workspace does not require. The complexity also obscures one retained correctness boundary: restore classification can follow a symlink and treat it as an already matching regular file.

Required changes

• Replace ordinary workspace reads, writes, copies, and renames with direct os, io, and path/filepath operations near their domain owners.
• Retain small shared helpers only for corruption-resistant same-directory file replacement, centralized bounded I/O, workspace modes, and explicitly root-scoped cleanup or restore.
• Keep atomic visibility for manifests and other corruption-sensitive single files without file or directory syncing or a power-loss durability promise.
• Receive remote downloads into temporary files and validate size, checksum, format, and any adapter contract before installation. Remove redundant download-install surfaces.
• Have Notarius write one unique staging bundle inside the invocation workspace; validate its location and integration contract, normalize workspace modes, require an absent unique destination, and rename the directory directly.
• Delete directory tree-copy promotion, race hooks, identity rechecks, no-replace platform code, directory syncing, capability errors, and the tests that exist solely for those retired mechanisms.
• Use os.Root for destructive cleanup and restore. Reject escaped targets and classify a leaf or ancestor symlink as a non-regular restore conflict in both force and non-force planning. Hash only a regular target verified within the root.
• Centralize and preserve group-owned directory and file modes. Keep API-key values out of permissive workspace files, logs, and errors, and preserve all explicit credential protections.
• Keep size limits centralized and discoverable, deliberately above ordinary workloads, with errors that name the limit and enforcing adapter.

Target state

Ordinary filesystem work is clear standard-library code. Shared filesystem code is small and policy-focused; extract performs one validated rename rather than a second full-tree copy; restore and cleanup cannot escape their roots or follow symlinked targets; group-owned operation remains first-class; and API keys remain protected without treating ordinary D&D data as private.

Workstream 3 — Canonical contracts and compatibility purge

Status: Pending

Audit sources: H43 and C01.

Underlying issue

Deprecated configuration and state formats remain active runtime branches. They require alternate parsing, probing, ambiguity handling, fixtures, errors, and documentation, and prevent one canonical representation from being clearly authoritative.

Required changes

• Remove the legacy remote current/manifest.json and current/run_id.txt reader.
• Remove legacy previous-session artifact candidates and existence probing; resolve only artifacts declared by the current committed state.
• Remove unversioned party documents, separate players_file handling, and legacy session-level party or player overrides tied to that model.
• Remove aggregate-only analyze availability, fallback to incidental files, extraction records without current contract and provenance evidence, and manifest or artifact branches predating the supported schema.
• Delete associated fields, constants, types, helpers, schemas, tests, fixtures, examples, and documentation rather than retaining an internal compatibility facade.
• Require explicit supported versions or unambiguous current authority markers. Reject old state early with a clear unsupported-version or migration error.
• Provide migration guidance through regeneration or republication without keeping steady-state migration readers in production.

Target state

Every supported configuration, party document, local manifest, remote current snapshot, extraction record, analyze record, and previous-session input has one canonical resolution path. Deprecated representations are absent from runtime code and fail clearly instead of being inferred or normalized.

Workstream 4 — One durable local manifest authority

Status: Pending

Audit sources: H14, H15, H17, C02, and C03.

Underlying issue

The session manifest and structured run manifest currently mirror stage and artifact progress. Keeping both synchronized makes diagnostic persistence part of pipeline correctness and creates dual-save ordering, projections, rollback, terminal recovery, and cleanup conflicts.

Required changes

• Eliminate the structured run-manifest schema, store, paths, validation, before/after/skip/failure/terminal saves, invocation-only projections, publish classification, compatibility code, tests, fixtures, and documentation.
• Retain the run ID for invocation identity, run-local paths, adapter provenance, and immutable remote publication.
• Retain bounded logs, generated configuration, receipts, immediate command summaries, and other useful diagnostics without making their persistence a second progress transaction.
• Make the session manifest the sole local authority for stage status, artifact currency, resume, status, restore, and publication.

Target state

One atomic session-manifest update determines durable local progress. Diagnostic write failure cannot overturn otherwise valid pipeline work, and no invocation manifest must remain synchronized with the progress ledger.

Workstream 5 — Runner lifecycle and pre-execution decisions

Status: Pending

Audit sources: H04, H07, H14, H15, H17, H34, and H35.

Underlying issue

executePlan combines preflight, locking, invocation setup, collaborator construction, resume decisions, stage transitions, failure persistence, and cleanup in one high-change function. Planning and execution also independently sequence some stale marking and invalidation decisions, while application tests replace package-global functions repeatedly.

Required changes

• Introduce a private invocation runtime or equivalent cohesive private state carrying resolved identity, paths, stores, the session manifest, environment, collaborators, and command summary state.
• Extract lifecycle operations for preparation and lock acquisition, invocation-state setup, collaborator composition, one-stage execution, and finalization.
• Keep session-lock acquisition and release, session-manifest persistence, and the top-level transaction order visible in the lifecycle owner.
• Introduce a pure pre-execution decision value combining recorded status, force, resume validation, stale reason, and required invalidation.
• Share only manifest-transition helpers that planning and execution apply identically. Keep planned outcomes distinct from actual stage results and persistence.
• Add parity tests for run versus skip, non-resumable records, forced invalidation, self-skip, and changed successful results.
• Replace affected mutable package-global function seams with a small invocation-scoped dependency set. Do not introduce a general dependency injection framework.

Target state

The top-level runner reads as an explicit lifecycle with one visible lock and persistence authority. Planning and execution agree through one narrow pure decision model, actual stage outcomes remain execution-owned, and focused tests do not require process-global substitution.

Workstream 6 — Per-artifact analyze checkpointing and recovery

Status: Pending

Audit sources: H04, H14, H15, H36, H37, C02, and C03.

Underlying issue

Analyze already has useful per-artifact identity and dependency semantics, but successful results participate in session and invocation projections and are not expressed as the simple resumable queue operators need. The per-artifact transaction also mixes preparation, invocation, validation, installation, and evidence construction behind a broad parameter list.

Required changes

• Use only the session manifest's analyze artifact collection as durable authority. Remove invocation projections and their validation, application, and rollback machinery.
• Execute the effective artifact set in deterministic dependency order.
• After each artifact succeeds and its canonical output is validated and materialized, update its current record, stale any transitive dependents whose evidence is invalidated, atomically save the session manifest, and only then continue.
• If the checkpoint save fails, leave incidental output non-authoritative and allow safe recomputation on a later run.
• On artifact failure, persist its failed state, stale current transitive dependents, mark the aggregate analyze stage failed, and stop before later artifacts.
• On an ordinary later run, retain valid successes, retry the first failed, stale, or missing selected artifact, and continue after it succeeds.
• Preserve established force and artifact-selection semantics: unqualified force reruns the configured analyze set; selected force applies to the selected artifacts and required dependency closure without rerunning unrelated valid artifacts.
• Retain fingerprints, output evidence, deterministic dependency ordering, dependency-aware invalidation, and the rule that incidental files cannot create current authority.
• Give analyzeStage.Run a private session-only execution accumulator or equivalent focused state. Split each artifact transaction into input/request preparation, adapter invocation, validated materialization, and record/evidence construction under one wrapper and narrow context.

Target state

Analyze is a deterministic resumable artifact queue backed by one manifest. Successful expensive work survives a later failure, invalid dependents cannot remain current, and ordinary resume continues at the first work that actually needs attention.

Workstream 7 — Configuration structure and semantic diagnostics

Status: Pending

Audit sources: H04, H14, H15, H17, H38, H39, H40, H41, H42, C01, and C06.

Underlying issue

Configuration behavior is appropriately strict, but unrelated validation domains share one large file, several functions combine distinct phases, legacy fields remain active, and tests treat exact diagnostic choreography as though it were a versioned semantic contract.

Required changes

• Complete the legacy configuration purge defined by the canonical-contract workstream.
• Split validation within package config into core pipeline/session, external-adapter, publish/storage, and Scriptorium artifact/dependency domains.
• Separate Scriptorium artifact declaration and output checks, dependency/input checks, and variable checks. Traverse declarations deterministically where ordering affects reproducibility, without promising one particular independent-error discovery order.
• Separate Notarius reference-binding normalization from output-lane normalization and collision checks; retain enablement and scalar checks in the top-level validator.
• Move the cohesive Audita validator with the other adapter validators without fragmenting its straightforward guards.
• Split composition-envelope shape decoding, cross-field semantic validation, and removal of root-only metadata while retaining Narratio's product-specific composition model.
• Preserve strict YAML decoding, unknown-field rejection, additive imports with no duplicate ownership, deterministic profile selection, meaningful overlay behavior for explicit values and collections, centralized defaults, canonical party and artifact-family resolution, and secret-free effective digests.
• Preserve actionable field identification and enough source information to correct composition conflicts. Stop preserving complete prose, punctuation, formatting, validation call order, or unused per-node provenance.
• Rewrite brittle tests to assert typed categories, field paths, structured details, or the smallest stable semantic fragment. Retain exact strings only for intentionally stable CLI or machine-consumed contracts.

Target state

Configuration remains strict, deterministic, and operator-friendly, but its implementation is organized by semantic owner and can improve diagnostics without treating private ordering or wording as compatibility behavior.

Workstream 8 — Remote publication, per-artifact locks, and publish decomposition

Status: Pending

Audit sources: H04, H15, H18, H44, and C05.

Underlying issue

Publish contains several cohesive concerns in one large file, while remote locks use a shared mutable YAML control plane with static precedence, merging, conditional replacement, and retry machinery. Omitting a locked artifact from a new immutable snapshot can also make the canonical artifact disappear rather than keep it frozen.

Required changes

• Split publish source discovery and opening, output-rule resolution, transfer preparation and verification, immutable upload, and commit/pointer encoding into focused files while retaining publishStage.Run as the transaction owner.
• Replace static pipeline locks and the central remote lock manifest with one versioned JSON lock object per normalized stable source ID at the approved hashed key. Validate that each payload source hashes to its object name.
• Remove static/remote merging, precedence, configuration injection, shared document replacement retries, and the retired lock schema and tests.
• Extend the narrow object-store port and each implementation or fake with the deletion operation needed to remove one lock.
• Adapt lock list/add/remove behavior to independent objects: list validates the prefix, add fails if present unless force replaces metadata, and remove treats absence as idempotent success.
• Check each relevant lock before spending work to publish its artifact and recheck immediately before advancing the current pointer.
• Build a new commit so an unlocked artifact uses its newly uploaded declaration, while a locked artifact carries the exact prior immutable declaration when one exists. A locked artifact never previously published may remain absent without becoming a required-output error.
• If the final recheck discovers a newly added lock, do not advance the pointer; allow unselected immutable attempt objects to remain and rebuild on the next run. A lock removed after the initial read may conservatively carry the prior declaration once.
• Allow carried declarations to reference an earlier immutable run prefix in the same campaign and session, retaining source identity, originating run, destination, checksum, size, and generation. Readers must use only objects declared by the selected commit and must not scan mutable prefixes.
• Distinguish newly published, carried-forward locked, locked-but-never-published, optional missing, and unselected outcomes in publish metadata and inspection.
• Preserve the visible transaction boundary: immutable objects, immutable commit, final lock revalidation, then one mutable current-pointer write.

Target state

Publish remains one coherent immutable-snapshot transaction with navigable internal files. Each lock is an independent remote object, changes to one lock cannot overwrite another, and later snapshots retain the exact prior canonical version of a locked artifact while unlocked artifacts advance.

Workstream 9 — Cleanup as the final pipeline stage

Status: Pending

Audit sources: H20, H21, C02, and C04.

Underlying issue

Local deletion currently occurs as a post-publish transaction with a special manifest obligation. A cleanup error can therefore make a successfully committed remote publication appear to have failed, and the extra state model duplicates ordinary stage retry behavior.

Required changes

• Add cleanup as the final canonical stage, after every stage that may need run-local data or diagnostics.
• Remove post-publish cleanup from the runner and delete the post_publish_cleanup manifest model, persistence, compatibility code, tests, and documentation.
• Use existing cleanup configuration to select spool audio, run workspace data, both, or neither.
• Require evidence of a successful current remote publish before deleting data whose retention depends on publication, and bind targets to that published run rather than inferring arbitrary paths from current configuration.
• Self-skip clearly when nothing is configured, treat already-absent eligible targets as success, use the filesystem workstream's root-scoped safeguards, and produce no canonical artifact output.
• Record cleanup through ordinary session-manifest stage status. A cleanup failure must leave publish success and remote current state unchanged; an ordinary later run should retry cleanup while reusing valid upstream stages.
• Include cleanup in the fixed invalidation relation so a changed or forced prerequisite makes an earlier cleanup result stale where appropriate.
• Ensure a selected range ending before cleanup performs no deletion, while a default full run includes cleanup.

Target state

Remote publication and local deletion are distinct, understandable outcomes. Publish succeeds when its pointer becomes current; cleanup succeeds when its eligible local targets are absent. Cleanup failure is reported and retried as a cleanup-stage failure without republishing unchanged data.

Workstream 10 — Quality, testability, and documentation alignment

Status: Pending (cross-cutting)

Audit sources: H07, H08, H12, H13, H17, H18, H23, H25, C01, C02, C03, C04, C05, and C06.

Underlying issue

Most tests protect meaningful behavior, but some preserve mechanisms and contracts now approved for removal, some application tests rely on mutable global seams, and large structural failures can be difficult to diagnose. Canonical documentation also describes several filesystem mechanisms as durable policy.

Required changes

• Delete tests, fixtures, hooks, and helpers only when their underlying legacy format, run-manifest model, cleanup obligation, central lock store, or FileOps guarantee is removed.
• Add or retain focused behavior coverage for atomic visibility, bounded limits and adapter errors, group modes, validated downloads, Notarius rename, destructive scope, restore symlink conflicts, secret redaction, session-only progress, per-artifact checkpoints, planning/execution parity, lock carry-forward, and idempotent cleanup.
• Prefer real temporary filesystems, fake object stores, and controlled helper processes at their current durable boundaries. Do not recreate removed algorithms through private test hooks.
• Replace affected mutable application globals with the invocation-scoped dependency set defined by the runner workstream.
• Use cmp.Diff for selected large composed configurations, analyze state, restore plans, and publish mappings. Use direct or standard-library equality for simple values.
• Apply slices and maps only to shallow clones, membership checks, and deterministic key handling in code already being touched, with nil, empty, and ordering semantics verified.
• Update architecture, operations, troubleshooting, integration, internal, configuration, CLI, examples, and release documentation only where each is the canonical owner of changed behavior. Remove retired claims rather than maintaining compatibility narratives as current behavior.
• Keep repository-wide tests and vet mandatory throughout. Apply focused race, documentation/example, and cross-platform release-build checks in proportion to each workstream, with the complete release-candidate validation at the integrated boundary.

Target state

Tests protect retained outcomes and failure boundaries instead of obsolete mechanisms. Collaborators are injectable per invocation without a framework, large diffs are readable where useful, and canonical documentation accurately describes the simplified implemented contracts as each workstream lands.

Dependency Relationships

These relationships constrain a future implementation plan without defining its numbered stages:

• The Go 1.26 baseline is required before relying on the process-handle API; the root-removal API is already present in the current Go 1.25 baseline.
• The compatibility purge should remove legacy branches before maintainers invest in decomposing the corresponding current-only previous-cache, configuration, analyze, and publication paths.
• Removing the run manifest establishes the session-only authority needed by runner decomposition, per-artifact analyze checkpointing, and the ordinary cleanup stage.
• Runner lifecycle decomposition and its pre-execution decision model provide the clean stage boundary used by analyze recovery and cleanup retry.
• The per-artifact lock representation and carry-forward rules must be reflected together in publish construction, commit validation, current-state reading, operator commands, and tests.
• Filesystem simplification and root-scoped destructive helpers must be available to the cleanup stage, while restore correctness belongs in the same filesystem change rather than the old facade.
• Test and documentation alignment accompanies the workstream that changes the behavior; it is not deferred to a final bulk cleanup.

Compatibility and Release Intent

The compatibility purge intentionally rejects deprecated formats rather than silently migrating them. Migration guidance should direct users to regenerate, republish, or otherwise replace old state. No runtime compatibility layer should survive solely to make the internal refactor invisible.

The approved audit decision is that this combined work remains eligible for a minor release because the removed compatibility surfaces were already deprecated and explicitly removable. Preparing or publishing that release is not part of this roadmap.

Consolidated Target State

After this roadmap is implemented:

• Narratio has one canonical configuration and state model with no deprecated runtime fallback paths;
• the session manifest alone owns durable local progress;
• analyze preserves each valid successful artifact and resumes failed, stale, or missing work in dependency order;
• application orchestration has explicit private lifecycle phases and one narrow planning/execution decision model;
• ordinary filesystem work uses the standard library, while small focused helpers retain atomic visibility, bounds, modes, destructive scope, restore safety, and secret protection;
• Notarius output is installed by one validated rename without a duplicate tree;
• publish owns coherent remote publication, per-artifact locks preserve prior canonical declarations, and cleanup is a separate final stage;
• configuration remains strict and deterministic without freezing incidental diagnostic choreography;
• Windows subprocess ownership and transcription concurrency use the audited Go 1.26 primitives, and subprocess mechanism is internally distinct from Narratio policy; and
• tests and documentation protect and describe the reduced observable contract.

Open Questions

None. The audits and subsequent approved decisions resolve the product and architecture choices required to prepare a staged implementation plan.