narratio/docs/roadmap/refactoring.md

# Narratio Refactoring Roadmap

## Status

Consolidated roadmap for the upcoming refactoring work. This document defines
the approved actionable scope and target state. It does not describe current
behavior and is not a staged implementation plan.

The complete audit inventory and final dispositions are recorded in the
[Audit Decision Register](audit-decisions.md). Supporting analysis remains in
the three source audits. This roadmap includes only work that at least one audit
identified and that the final decision register classifies for immediate
action.

## Goal

Reduce Narratio's implementation and maintenance burden while preserving the
operational outcomes that matter: correct resumable progress, economical reuse
of expensive generated artifacts, coherent remote publication, understandable
cleanup and recovery, safe destructive scope, group-owned workspaces, and
protection of API keys.

The refactor deliberately permits approved internal and deprecated-contract
changes. It is not limited to producing an implementation mechanically
equivalent to the current one.

## Governing Boundaries

- Keep the existing package-level ownership and acyclic dependency direction.
  Prefer private types, functions, and focused files over new architectural
  layers.
- Treat campaign and session workspaces as trusted, intentionally shareable,
  group-owned data. API keys remain sensitive, and cleanup and restore remain
  confined to explicit application-owned roots.
- Keep the session manifest as the sole durable local progress authority.
- Preserve strict, deterministic configuration semantics without preserving
  incidental diagnostic order, complete wording, punctuation, or private
  validation choreography.
- Keep integration-specific requests, validation, and errors with their
  adapters. Shared infrastructure should expose only exact stable mechanics.
- Add dependencies selectively when they remove audited complexity and preserve
  the required supported-platform behavior.
- Update tests and canonical documentation with the behavior they own rather
  than preserving retired implementation mechanisms.

## Actionable Workstreams

Each workstream uses one of three status identifiers: `Pending`, `In Progress`,
or `Complete`. Workstreams are numbered in dependency-aware implementation
order. The first workstream not marked `Complete` is the next workstream to
address unless its status or an implementation plan explicitly records a
blocker.

### Workstream 1 — Go baseline, process supervision, and transcription concurrency

**Status:** Complete

**Audit sources:** H11, H19, H22, H24, H33, and L01.

#### Underlying issue

The maintained Go baseline prevents use of newer root and process primitives,
the Windows subprocess owner reopens a PID instead of using a lifetime-stable
handle, transcription manually implements bounded cancellation and first-error
coordination, and the shared subprocess package mixes generic process ownership
with Narratio environment, diagnostic, redaction, and file-writing policy.

#### Required changes

- Upgrade `go.mod`, generated module state, every maintained CI and release Go
  image, and release documentation to Go 1.26 in one bounded change. Go 1.26
  provides `os.Process.WithHandle`; the required `os.Root.RemoveAll` operation
  was already available in Go 1.25.
- Validate the upgrade with the repository-wide tests, race checks, vet,
  maintained documentation and examples, and all supported release builds.
- Assign the suspended Windows child to its Job Object inside
  `os.Process.WithHandle`. Preserve suspended creation, job configuration,
  initial-thread resumption, fail-closed cleanup, and job disposal.
- Replace the transcription worker pool with `errgroup.WithContext` and
  `SetLimit`. Preserve speaker-qualified failures, parent-cancellation context,
  aggregation of successful results under a narrow lock, and deterministic
  output ordering.
- Clarify the internal subprocess boundary: the reusable mechanism owns process
  tree start, wait, termination, reaping, and typed outcomes; Narratio owns
  environment selection, secret-value selection and redaction, bounded log
  sinks, paths, modes, diagnostic wording, and tool-specific hints.
- Remove atomic YAML and byte-writing helpers from the subprocess package and
  route file replacement through the focused filesystem owner.

#### Target state

Narratio builds and tests on Go 1.26, uses the safer native Windows handle API,
and has concise bounded transcription concurrency. Process-tree ownership is
internally separable from Narratio policy without creating or depending on a
new external subprocess library during this refactor.

### Workstream 2 — Filesystem simplification and restore correctness

**Status:** Complete

**Audit sources:** H02, H06, H07, H18, H20, H21, H29, H45, H46, C04,
L02, L05a, and L05c. Supporting filesystem rationale and policy are recorded
in [`fileops.md`](fileops.md); the requirements below are the consolidated
action scope.

#### Underlying issue

The current FileOps subsystem promises adversarial race resistance, universal
confinement and no-follow behavior, atomic no-replace directory promotion, and
power-loss durability that Narratio's trusted, session-locked workspace does
not require. The complexity also obscures one retained correctness boundary:
restore classification can follow a symlink and treat it as an already matching
regular file.

#### Required changes

- Replace ordinary workspace reads, writes, copies, and renames with direct
  `os`, `io`, and `path/filepath` operations near their domain owners.
- Retain small shared helpers only for corruption-resistant same-directory file
  replacement, centralized bounded I/O, workspace modes, and explicitly
  root-scoped cleanup or restore.
- Keep atomic visibility for manifests and other corruption-sensitive single
  files without file or directory syncing or a power-loss durability promise.
- Receive remote downloads into temporary files and validate size, checksum,
  format, and any adapter contract before installation. Remove redundant
  download-install surfaces.
- Have Notarius write one unique staging bundle inside the invocation workspace;
  validate its location and integration contract, normalize workspace modes,
  require an absent unique destination, and rename the directory directly.
- Delete directory tree-copy promotion, race hooks, identity rechecks,
  no-replace platform code, directory syncing, capability errors, and the
  tests that exist solely for those retired mechanisms.
- Use `os.Root` for destructive cleanup and restore. Reject escaped targets and
  classify a leaf or ancestor symlink as a non-regular restore conflict in both
  force and non-force planning. Hash only a regular target verified within the
  root.
- Centralize and preserve group-owned directory and file modes. Keep API-key
  values out of permissive workspace files, logs, and errors, and preserve all
  explicit credential protections.
- Keep size limits centralized and discoverable, deliberately above ordinary
  workloads, with errors that name the limit and enforcing adapter.

#### Target state

Ordinary filesystem work is clear standard-library code. Shared filesystem
code is small and policy-focused; extract performs one validated rename rather
than a second full-tree copy; restore and cleanup cannot escape their roots or
follow symlinked targets; group-owned operation remains first-class; and API
keys remain protected without treating ordinary D&D data as private.

### Workstream 3 — Canonical contracts and compatibility purge

**Status:** Complete

**Audit sources:** H43 and C01.

#### Underlying issue

Deprecated configuration and state formats remain active runtime branches.
They require alternate parsing, probing, ambiguity handling, fixtures, errors,
and documentation, and prevent one canonical representation from being clearly
authoritative.

#### Required changes

- Remove the legacy remote `current/manifest.json` and `current/run_id.txt`
  reader.
- Remove legacy previous-session artifact candidates and existence probing;
  resolve only artifacts declared by the current committed state.
- Remove unversioned party documents, separate `players_file` handling, and
  legacy session-level party or player overrides tied to that model.
- Remove aggregate-only analyze availability, fallback to incidental files,
  extraction records without current contract and provenance evidence, and
  manifest or artifact branches predating the supported schema.
- Delete associated fields, constants, types, helpers, schemas, tests, fixtures,
  examples, and documentation rather than retaining an internal compatibility
  facade.
- Require explicit supported versions or unambiguous current authority markers.
  Reject old state early with a clear unsupported-version or migration error.
- Provide migration guidance through regeneration or republication without
  keeping steady-state migration readers in production.

#### Target state

Every supported configuration, party document, local manifest, remote current
snapshot, extraction record, analyze record, and previous-session input has one
canonical resolution path. Deprecated representations are absent from runtime
code and fail clearly instead of being inferred or normalized.

### Workstream 4 — One durable local manifest authority

**Status:** Complete

**Audit sources:** H14, H15, H17, C02, and C03.

#### Underlying issue

The session manifest and structured run manifest currently mirror stage and
artifact progress. Keeping both synchronized makes diagnostic persistence part
of pipeline correctness and creates dual-save ordering, projections, rollback,
terminal recovery, and cleanup conflicts.

#### Required changes

- Eliminate the structured run-manifest schema, store, paths, validation,
  before/after/skip/failure/terminal saves, invocation-only projections,
  publish classification, compatibility code, tests, fixtures, and
  documentation.
- Retain the run ID for invocation identity, run-local paths, adapter provenance,
  and immutable remote publication.
- Retain bounded logs, generated configuration, receipts, immediate command
  summaries, and other useful diagnostics without making their persistence a
  second progress transaction.
- Make the session manifest the sole local authority for stage status, artifact
  currency, resume, status, restore, and publication.

#### Target state

One atomic session-manifest update determines durable local progress. Diagnostic
write failure cannot overturn otherwise valid pipeline work, and no invocation
manifest must remain synchronized with the progress ledger.

### Workstream 5 — Runner lifecycle and pre-execution decisions

**Status:** Complete

**Audit sources:** H04, H07, H14, H15, H17, H34, and H35.

#### Underlying issue

`executePlan` combines preflight, locking, invocation setup, collaborator
construction, resume decisions, stage transitions, failure persistence, and
cleanup in one high-change function. Planning and execution also independently
sequence some stale marking and invalidation decisions, while application tests
replace package-global functions repeatedly.

#### Required changes

- Introduce a private invocation runtime or equivalent cohesive private state
  carrying resolved identity, paths, stores, the session manifest, environment,
  collaborators, and command summary state.
- Extract lifecycle operations for preparation and lock acquisition,
  invocation-state setup, collaborator composition, one-stage execution, and
  finalization.
- Keep session-lock acquisition and release, session-manifest persistence, and
  the top-level transaction order visible in the lifecycle owner.
- Introduce a pure pre-execution decision value combining recorded status,
  force, resume validation, stale reason, and required invalidation.
- Share only manifest-transition helpers that planning and execution apply
  identically. Keep planned outcomes distinct from actual stage results and
  persistence.
- Add parity tests for run versus skip, non-resumable records, forced
  invalidation, self-skip, and changed successful results.
- Replace affected mutable package-global function seams with a small
  invocation-scoped dependency set. Do not introduce a general dependency
  injection framework.

#### Target state

The top-level runner reads as an explicit lifecycle with one visible lock and
persistence authority. Planning and execution agree through one narrow pure
decision model, actual stage outcomes remain execution-owned, and focused tests
do not require process-global substitution.

### Workstream 6 — Per-artifact analyze checkpointing and recovery

**Status:** Complete

**Audit sources:** H04, H14, H15, H36, H37, C02, and C03.

#### Underlying issue

Analyze already has useful per-artifact identity and dependency semantics, but
successful results participate in session and invocation projections and are
not expressed as the simple resumable queue operators need. The per-artifact
transaction also mixes preparation, invocation, validation, installation, and
evidence construction behind a broad parameter list.

#### Required changes

- Use only the session manifest's analyze artifact collection as durable
  authority. Remove invocation projections and their validation, application,
  and rollback machinery.
- Execute the effective artifact set in deterministic dependency order.
- After each artifact succeeds and its canonical output is validated and
  materialized, update its current record, stale any transitive dependents whose
  evidence is invalidated, atomically save the session manifest, and only then
  continue.
- If the checkpoint save fails, leave incidental output non-authoritative and
  allow safe recomputation on a later run.
- On artifact failure, persist its failed state, stale current transitive
  dependents, mark the aggregate analyze stage failed, and stop before later
  artifacts.
- On an ordinary later run, retain valid successes, retry the first failed,
  stale, or missing selected artifact, and continue after it succeeds.
- Preserve established force and artifact-selection semantics: unqualified
  force reruns the configured analyze set; selected force applies to the
  selected artifacts and required dependency closure without rerunning
  unrelated valid artifacts.
- Retain fingerprints, output evidence, deterministic dependency ordering,
  dependency-aware invalidation, and the rule that incidental files cannot
  create current authority.
- Give `analyzeStage.Run` a private session-only execution accumulator or
  equivalent focused state. Split each artifact transaction into input/request
  preparation, adapter invocation, validated materialization, and
  record/evidence construction under one wrapper and narrow context.

#### Target state

Analyze is a deterministic resumable artifact queue backed by one manifest.
Successful expensive work survives a later failure, invalid dependents cannot
remain current, and ordinary resume continues at the first work that actually
needs attention.

### Workstream 7 — Configuration structure and semantic diagnostics

**Status:** Complete

**Audit sources:** H04, H14, H15, H17, H38, H39, H40, H41, H42, C01, and C06.

#### Underlying issue

Configuration behavior is appropriately strict, but unrelated validation
domains share one large file, several functions combine distinct phases, legacy
fields remain active, and tests treat exact diagnostic choreography as though
it were a versioned semantic contract.

#### Required changes

- Complete the legacy configuration purge defined by the canonical-contract
  workstream.
- Split validation within package `config` into core pipeline/session,
  external-adapter, publish/storage, and Scriptorium artifact/dependency
  domains.
- Separate Scriptorium artifact declaration and output checks,
  dependency/input checks, and variable checks. Traverse declarations
  deterministically where ordering affects reproducibility, without promising
  one particular independent-error discovery order.
- Separate Notarius reference-binding normalization from output-lane
  normalization and collision checks; retain enablement and scalar checks in
  the top-level validator.
- Move the cohesive Audita validator with the other adapter validators without
  fragmenting its straightforward guards.
- Split composition-envelope shape decoding, cross-field semantic validation,
  and removal of root-only metadata while retaining Narratio's product-specific
  composition model.
- Preserve strict YAML decoding, unknown-field rejection, additive imports with
  no duplicate ownership, deterministic profile selection, meaningful overlay
  behavior for explicit values and collections, centralized defaults, canonical
  party and artifact-family resolution, and secret-free effective digests.
- Preserve actionable field identification and enough source information to
  correct composition conflicts. Stop preserving complete prose, punctuation,
  formatting, validation call order, or unused per-node provenance.
- Rewrite brittle tests to assert typed categories, field paths, structured
  details, or the smallest stable semantic fragment. Retain exact strings only
  for intentionally stable CLI or machine-consumed contracts.

#### Target state

Configuration remains strict, deterministic, and operator-friendly, but its
implementation is organized by semantic owner and can improve diagnostics
without treating private ordering or wording as compatibility behavior.

### Workstream 8 — Remote publication, per-artifact locks, and publish decomposition

**Status:** Complete

**Audit sources:** H04, H15, H18, H44, and C05.

#### Underlying issue

Publish contains several cohesive concerns in one large file, while remote
locks use a shared mutable YAML control plane with static precedence, merging,
conditional replacement, and retry machinery. Omitting a locked artifact from
a new immutable snapshot can also make the canonical artifact disappear rather
than keep it frozen.

#### Required changes

- Split publish source discovery and opening, output-rule resolution, transfer
  preparation and verification, immutable upload, and commit/pointer encoding
  into focused files while retaining `publishStage.Run` as the transaction
  owner.
- Replace static pipeline locks and the central remote lock manifest with one
  versioned JSON lock object per normalized stable source ID at the approved
  hashed key. Validate that each payload source hashes to its object name.
- Remove static/remote merging, precedence, configuration injection, shared
  document replacement retries, and the retired lock schema and tests.
- Extend the narrow object-store port and each implementation or fake with the
  deletion operation needed to remove one lock.
- Adapt lock list/add/remove behavior to independent objects: list validates the
  prefix, add fails if present unless force replaces metadata, and remove treats
  absence as idempotent success.
- Check each relevant lock before spending work to publish its artifact and
  recheck immediately before advancing the current pointer.
- Build a new commit so an unlocked artifact uses its newly uploaded
  declaration, while a locked artifact carries the exact prior immutable
  declaration when one exists. A locked artifact never previously published may
  remain absent without becoming a required-output error.
- If the final recheck discovers a newly added lock, do not advance the pointer;
  allow unselected immutable attempt objects to remain and rebuild on the next
  run. A lock removed after the initial read may conservatively carry the prior
  declaration once.
- Allow carried declarations to reference an earlier immutable run prefix in
  the same campaign and session, retaining source identity, originating run,
  destination, checksum, size, and generation. Readers must use only objects
  declared by the selected commit and must not scan mutable prefixes.
- Distinguish newly published, carried-forward locked,
  locked-but-never-published, optional missing, and unselected outcomes in
  publish metadata and inspection.
- Preserve the visible transaction boundary: immutable objects, immutable
  commit, final lock revalidation, then one mutable current-pointer write.

#### Target state

Publish remains one coherent immutable-snapshot transaction with navigable
internal files. Each lock is an independent remote object, changes to one lock
cannot overwrite another, and later snapshots retain the exact prior canonical
version of a locked artifact while unlocked artifacts advance.

### Workstream 9 — Cleanup as the final pipeline stage

**Status:** Pending

**Audit sources:** H20, H21, C02, and C04.

#### Underlying issue

Local deletion currently occurs as a post-publish transaction with a special
manifest obligation. A cleanup error can therefore make a successfully
committed remote publication appear to have failed, and the extra state model
duplicates ordinary stage retry behavior.

#### Required changes

- Add `cleanup` as the final canonical stage, after every stage that may need
  run-local data or diagnostics.
- Remove post-publish cleanup from the runner and delete the
  `post_publish_cleanup` manifest model, persistence, compatibility code,
  tests, and documentation.
- Use existing cleanup configuration to select spool audio, run workspace data,
  both, or neither.
- Require evidence of a successful current remote publish before deleting data
  whose retention depends on publication, and bind targets to that published
  run rather than inferring arbitrary paths from current configuration.
- Self-skip clearly when nothing is configured, treat already-absent eligible
  targets as success, use the filesystem workstream's root-scoped safeguards,
  and produce no canonical artifact output.
- Record cleanup through ordinary session-manifest stage status. A cleanup
  failure must leave publish success and remote current state unchanged; an
  ordinary later run should retry cleanup while reusing valid upstream stages.
- Include cleanup in the fixed invalidation relation so a changed or forced
  prerequisite makes an earlier cleanup result stale where appropriate.
- Ensure a selected range ending before cleanup performs no deletion, while a
  default full run includes cleanup.

#### Target state

Remote publication and local deletion are distinct, understandable outcomes.
Publish succeeds when its pointer becomes current; cleanup succeeds when its
eligible local targets are absent. Cleanup failure is reported and retried as a
cleanup-stage failure without republishing unchanged data.

### Workstream 10 — Quality, testability, and documentation alignment

**Status:** Pending (cross-cutting)

**Audit sources:** H07, H08, H12, H13, H17, H18, H23, H25, C01, C02, C03,
C04, C05, and C06.

#### Underlying issue

Most tests protect meaningful behavior, but some preserve mechanisms and
contracts now approved for removal, some application tests rely on mutable
global seams, and large structural failures can be difficult to diagnose.
Canonical documentation also describes several filesystem mechanisms as
durable policy.

#### Required changes

- Delete tests, fixtures, hooks, and helpers only when their underlying legacy
  format, run-manifest model, cleanup obligation, central lock store, or FileOps
  guarantee is removed.
- Add or retain focused behavior coverage for atomic visibility, bounded limits
  and adapter errors, group modes, validated downloads, Notarius rename,
  destructive scope, restore symlink conflicts, secret redaction, session-only
  progress, per-artifact checkpoints, planning/execution parity, lock
  carry-forward, and idempotent cleanup.
- Prefer real temporary filesystems, fake object stores, and controlled helper
  processes at their current durable boundaries. Do not recreate removed
  algorithms through private test hooks.
- Replace affected mutable application globals with the invocation-scoped
  dependency set defined by the runner workstream.
- Use `cmp.Diff` for selected large composed configurations, analyze state,
  restore plans, and publish mappings. Use direct or standard-library equality
  for simple values.
- Apply `slices` and `maps` only to shallow clones, membership checks, and
  deterministic key handling in code already being touched, with nil, empty,
  and ordering semantics verified.
- Update architecture, operations, troubleshooting, integration, internal,
  configuration, CLI, examples, and release documentation only where each is
  the canonical owner of changed behavior. Remove retired claims rather than
  maintaining compatibility narratives as current behavior.
- Keep repository-wide tests and vet mandatory throughout. Apply focused race,
  documentation/example, and cross-platform release-build checks in proportion
  to each workstream, with the complete release-candidate validation at the
  integrated boundary.

#### Target state

Tests protect retained outcomes and failure boundaries instead of obsolete
mechanisms. Collaborators are injectable per invocation without a framework,
large diffs are readable where useful, and canonical documentation accurately
describes the simplified implemented contracts as each workstream lands.

## Dependency Relationships

These relationships constrain a future implementation plan without defining
its numbered stages:

- The Go 1.26 baseline is required before relying on the process-handle API;
  the root-removal API is already present in the current Go 1.25 baseline.
- The compatibility purge should remove legacy branches before maintainers
  invest in decomposing the corresponding current-only previous-cache,
  configuration, analyze, and publication paths.
- Removing the run manifest establishes the session-only authority needed by
  runner decomposition, per-artifact analyze checkpointing, and the ordinary
  cleanup stage.
- Runner lifecycle decomposition and its pre-execution decision model provide
  the clean stage boundary used by analyze recovery and cleanup retry.
- The per-artifact lock representation and carry-forward rules must be reflected
  together in publish construction, commit validation, current-state reading,
  operator commands, and tests.
- Filesystem simplification and root-scoped destructive helpers must be
  available to the cleanup stage, while restore correctness belongs in the same
  filesystem change rather than the old facade.
- Test and documentation alignment accompanies the workstream that changes the
  behavior; it is not deferred to a final bulk cleanup.

## Compatibility and Release Intent

The compatibility purge intentionally rejects deprecated formats rather than
silently migrating them. Migration guidance should direct users to regenerate,
republish, or otherwise replace old state. No runtime compatibility layer should
survive solely to make the internal refactor invisible.

The approved audit decision is that this combined work remains eligible for a
minor release because the removed compatibility surfaces were already
deprecated and explicitly removable. Preparing or publishing that release is
not part of this roadmap.

## Consolidated Target State

After this roadmap is implemented:

- Narratio has one canonical configuration and state model with no deprecated
  runtime fallback paths;
- the session manifest alone owns durable local progress;
- analyze preserves each valid successful artifact and resumes failed, stale,
  or missing work in dependency order;
- application orchestration has explicit private lifecycle phases and one
  narrow planning/execution decision model;
- ordinary filesystem work uses the standard library, while small focused
  helpers retain atomic visibility, bounds, modes, destructive scope, restore
  safety, and secret protection;
- Notarius output is installed by one validated rename without a duplicate tree;
- publish owns coherent remote publication, per-artifact locks preserve prior
  canonical declarations, and cleanup is a separate final stage;
- configuration remains strict and deterministic without freezing incidental
  diagnostic choreography;
- Windows subprocess ownership and transcription concurrency use the audited
  Go 1.26 primitives, and subprocess mechanism is internally distinct from
  Narratio policy; and
- tests and documentation protect and describe the reduced observable contract.

## Open Questions

None. The audits and subsequent approved decisions resolve the product and
architecture choices required to prepare a staged implementation plan.