Plan Go and process refactoring

docs/roadmap/refactoring.md

@@ -0,0 +1,579 @@
+# Narratio Refactoring Roadmap
+
+## Status
+
+Consolidated roadmap for the upcoming refactoring work. This document defines
+the approved actionable scope and target state. It does not describe current
+behavior and is not a staged implementation plan.
+
+The complete audit inventory and final dispositions are recorded in the
+[Audit Decision Register](audit-decisions.md). Supporting analysis remains in
+the three source audits. This roadmap includes only work that at least one audit
+identified and that the final decision register classifies for immediate
+action.
+
+## Goal
+
+Reduce Narratio's implementation and maintenance burden while preserving the
+operational outcomes that matter: correct resumable progress, economical reuse
+of expensive generated artifacts, coherent remote publication, understandable
+cleanup and recovery, safe destructive scope, group-owned workspaces, and
+protection of API keys.
+
+The refactor deliberately permits approved internal and deprecated-contract
+changes. It is not limited to producing an implementation mechanically
+equivalent to the current one.
+
+## Governing Boundaries
+
+- Keep the existing package-level ownership and acyclic dependency direction.
+  Prefer private types, functions, and focused files over new architectural
+  layers.
+- Treat campaign and session workspaces as trusted, intentionally shareable,
+  group-owned data. API keys remain sensitive, and cleanup and restore remain
+  confined to explicit application-owned roots.
+- Keep the session manifest as the sole durable local progress authority.
+- Preserve strict, deterministic configuration semantics without preserving
+  incidental diagnostic order, complete wording, punctuation, or private
+  validation choreography.
+- Keep integration-specific requests, validation, and errors with their
+  adapters. Shared infrastructure should expose only exact stable mechanics.
+- Add dependencies selectively when they remove audited complexity and preserve
+  the required supported-platform behavior.
+- Update tests and canonical documentation with the behavior they own rather
+  than preserving retired implementation mechanisms.
+
+## Actionable Workstreams
+
+Each workstream uses one of three status identifiers: `Pending`, `In Progress`,
+or `Complete`. Workstreams are numbered in dependency-aware implementation
+order. The first workstream not marked `Complete` is the next workstream to
+address unless its status or an implementation plan explicitly records a
+blocker.
+
+### Workstream 1 — Go baseline, process supervision, and transcription concurrency
+
+**Status:** Pending
+
+**Audit sources:** H11, H19, H22, H24, H33, and L01.
+
+#### Underlying issue
+
+The maintained Go baseline prevents use of newer root and process primitives,
+the Windows subprocess owner reopens a PID instead of using a lifetime-stable
+handle, transcription manually implements bounded cancellation and first-error
+coordination, and the shared subprocess package mixes generic process ownership
+with Narratio environment, diagnostic, redaction, and file-writing policy.
+
+#### Required changes
+
+- Upgrade `go.mod`, generated module state, every maintained CI and release Go
+  image, and release documentation to Go 1.26 in one bounded change. Go 1.26
+  provides `os.Process.WithHandle`; the required `os.Root.RemoveAll` operation
+  was already available in Go 1.25.
+- Validate the upgrade with the repository-wide tests, race checks, vet,
+  maintained documentation and examples, and all supported release builds.
+- Assign the suspended Windows child to its Job Object inside
+  `os.Process.WithHandle`. Preserve suspended creation, job configuration,
+  initial-thread resumption, fail-closed cleanup, and job disposal.
+- Replace the transcription worker pool with `errgroup.WithContext` and
+  `SetLimit`. Preserve speaker-qualified failures, parent-cancellation context,
+  aggregation of successful results under a narrow lock, and deterministic
+  output ordering.
+- Clarify the internal subprocess boundary: the reusable mechanism owns process
+  tree start, wait, termination, reaping, and typed outcomes; Narratio owns
+  environment selection, secret-value selection and redaction, bounded log
+  sinks, paths, modes, diagnostic wording, and tool-specific hints.
+- Remove atomic YAML and byte-writing helpers from the subprocess package and
+  route file replacement through the focused filesystem owner.
+
+#### Target state
+
+Narratio builds and tests on Go 1.26, uses the safer native Windows handle API,
+and has concise bounded transcription concurrency. Process-tree ownership is
+internally separable from Narratio policy without creating or depending on a
+new external subprocess library during this refactor.
+
+### Workstream 2 — Filesystem simplification and restore correctness
+
+**Status:** Pending
+
+**Audit sources:** H02, H06, H07, H18, H20, H21, H29, H45, H46, C04,
+L02, L05a, and L05c. Supporting filesystem rationale and policy are recorded
+in [`fileops.md`](fileops.md); the requirements below are the consolidated
+action scope.
+
+#### Underlying issue
+
+The current FileOps subsystem promises adversarial race resistance, universal
+confinement and no-follow behavior, atomic no-replace directory promotion, and
+power-loss durability that Narratio's trusted, session-locked workspace does
+not require. The complexity also obscures one retained correctness boundary:
+restore classification can follow a symlink and treat it as an already matching
+regular file.
+
+#### Required changes
+
+- Replace ordinary workspace reads, writes, copies, and renames with direct
+  `os`, `io`, and `path/filepath` operations near their domain owners.
+- Retain small shared helpers only for corruption-resistant same-directory file
+  replacement, centralized bounded I/O, workspace modes, and explicitly
+  root-scoped cleanup or restore.
+- Keep atomic visibility for manifests and other corruption-sensitive single
+  files without file or directory syncing or a power-loss durability promise.
+- Receive remote downloads into temporary files and validate size, checksum,
+  format, and any adapter contract before installation. Remove redundant
+  download-install surfaces.
+- Have Notarius write one unique staging bundle inside the invocation workspace;
+  validate its location and integration contract, normalize workspace modes,
+  require an absent unique destination, and rename the directory directly.
+- Delete directory tree-copy promotion, race hooks, identity rechecks,
+  no-replace platform code, directory syncing, capability errors, and the
+  tests that exist solely for those retired mechanisms.
+- Use `os.Root` for destructive cleanup and restore. Reject escaped targets and
+  classify a leaf or ancestor symlink as a non-regular restore conflict in both
+  force and non-force planning. Hash only a regular target verified within the
+  root.
+- Centralize and preserve group-owned directory and file modes. Keep API-key
+  values out of permissive workspace files, logs, and errors, and preserve all
+  explicit credential protections.
+- Keep size limits centralized and discoverable, deliberately above ordinary
+  workloads, with errors that name the limit and enforcing adapter.
+
+#### Target state
+
+Ordinary filesystem work is clear standard-library code. Shared filesystem
+code is small and policy-focused; extract performs one validated rename rather
+than a second full-tree copy; restore and cleanup cannot escape their roots or
+follow symlinked targets; group-owned operation remains first-class; and API
+keys remain protected without treating ordinary D&D data as private.
+
+### Workstream 3 — Canonical contracts and compatibility purge
+
+**Status:** Pending
+
+**Audit sources:** H43 and C01.
+
+#### Underlying issue
+
+Deprecated configuration and state formats remain active runtime branches.
+They require alternate parsing, probing, ambiguity handling, fixtures, errors,
+and documentation, and prevent one canonical representation from being clearly
+authoritative.
+
+#### Required changes
+
+- Remove the legacy remote `current/manifest.json` and `current/run_id.txt`
+  reader.
+- Remove legacy previous-session artifact candidates and existence probing;
+  resolve only artifacts declared by the current committed state.
+- Remove unversioned party documents, separate `players_file` handling, and
+  legacy session-level party or player overrides tied to that model.
+- Remove aggregate-only analyze availability, fallback to incidental files,
+  extraction records without current contract and provenance evidence, and
+  manifest or artifact branches predating the supported schema.
+- Delete associated fields, constants, types, helpers, schemas, tests, fixtures,
+  examples, and documentation rather than retaining an internal compatibility
+  facade.
+- Require explicit supported versions or unambiguous current authority markers.
+  Reject old state early with a clear unsupported-version or migration error.
+- Provide migration guidance through regeneration or republication without
+  keeping steady-state migration readers in production.
+
+#### Target state
+
+Every supported configuration, party document, local manifest, remote current
+snapshot, extraction record, analyze record, and previous-session input has one
+canonical resolution path. Deprecated representations are absent from runtime
+code and fail clearly instead of being inferred or normalized.
+
+### Workstream 4 — One durable local manifest authority
+
+**Status:** Pending
+
+**Audit sources:** H14, H15, H17, C02, and C03.
+
+#### Underlying issue
+
+The session manifest and structured run manifest currently mirror stage and
+artifact progress. Keeping both synchronized makes diagnostic persistence part
+of pipeline correctness and creates dual-save ordering, projections, rollback,
+terminal recovery, and cleanup conflicts.
+
+#### Required changes
+
+- Eliminate the structured run-manifest schema, store, paths, validation,
+  before/after/skip/failure/terminal saves, invocation-only projections,
+  publish classification, compatibility code, tests, fixtures, and
+  documentation.
+- Retain the run ID for invocation identity, run-local paths, adapter provenance,
+  and immutable remote publication.
+- Retain bounded logs, generated configuration, receipts, immediate command
+  summaries, and other useful diagnostics without making their persistence a
+  second progress transaction.
+- Make the session manifest the sole local authority for stage status, artifact
+  currency, resume, status, restore, and publication.
+
+#### Target state
+
+One atomic session-manifest update determines durable local progress. Diagnostic
+write failure cannot overturn otherwise valid pipeline work, and no invocation
+manifest must remain synchronized with the progress ledger.
+
+### Workstream 5 — Runner lifecycle and pre-execution decisions
+
+**Status:** Pending
+
+**Audit sources:** H04, H07, H14, H15, H17, H34, and H35.
+
+#### Underlying issue
+
+`executePlan` combines preflight, locking, invocation setup, collaborator
+construction, resume decisions, stage transitions, failure persistence, and
+cleanup in one high-change function. Planning and execution also independently
+sequence some stale marking and invalidation decisions, while application tests
+replace package-global functions repeatedly.
+
+#### Required changes
+
+- Introduce a private invocation runtime or equivalent cohesive private state
+  carrying resolved identity, paths, stores, the session manifest, environment,
+  collaborators, and command summary state.
+- Extract lifecycle operations for preparation and lock acquisition,
+  invocation-state setup, collaborator composition, one-stage execution, and
+  finalization.
+- Keep session-lock acquisition and release, session-manifest persistence, and
+  the top-level transaction order visible in the lifecycle owner.
+- Introduce a pure pre-execution decision value combining recorded status,
+  force, resume validation, stale reason, and required invalidation.
+- Share only manifest-transition helpers that planning and execution apply
+  identically. Keep planned outcomes distinct from actual stage results and
+  persistence.
+- Add parity tests for run versus skip, non-resumable records, forced
+  invalidation, self-skip, and changed successful results.
+- Replace affected mutable package-global function seams with a small
+  invocation-scoped dependency set. Do not introduce a general dependency
+  injection framework.
+
+#### Target state
+
+The top-level runner reads as an explicit lifecycle with one visible lock and
+persistence authority. Planning and execution agree through one narrow pure
+decision model, actual stage outcomes remain execution-owned, and focused tests
+do not require process-global substitution.
+
+### Workstream 6 — Per-artifact analyze checkpointing and recovery
+
+**Status:** Pending
+
+**Audit sources:** H04, H14, H15, H36, H37, C02, and C03.
+
+#### Underlying issue
+
+Analyze already has useful per-artifact identity and dependency semantics, but
+successful results participate in session and invocation projections and are
+not expressed as the simple resumable queue operators need. The per-artifact
+transaction also mixes preparation, invocation, validation, installation, and
+evidence construction behind a broad parameter list.
+
+#### Required changes
+
+- Use only the session manifest's analyze artifact collection as durable
+  authority. Remove invocation projections and their validation, application,
+  and rollback machinery.
+- Execute the effective artifact set in deterministic dependency order.
+- After each artifact succeeds and its canonical output is validated and
+  materialized, update its current record, stale any transitive dependents whose
+  evidence is invalidated, atomically save the session manifest, and only then
+  continue.
+- If the checkpoint save fails, leave incidental output non-authoritative and
+  allow safe recomputation on a later run.
+- On artifact failure, persist its failed state, stale current transitive
+  dependents, mark the aggregate analyze stage failed, and stop before later
+  artifacts.
+- On an ordinary later run, retain valid successes, retry the first failed,
+  stale, or missing selected artifact, and continue after it succeeds.
+- Preserve established force and artifact-selection semantics: unqualified
+  force reruns the configured analyze set; selected force applies to the
+  selected artifacts and required dependency closure without rerunning
+  unrelated valid artifacts.
+- Retain fingerprints, output evidence, deterministic dependency ordering,
+  dependency-aware invalidation, and the rule that incidental files cannot
+  create current authority.
+- Give `analyzeStage.Run` a private session-only execution accumulator or
+  equivalent focused state. Split each artifact transaction into input/request
+  preparation, adapter invocation, validated materialization, and
+  record/evidence construction under one wrapper and narrow context.
+
+#### Target state
+
+Analyze is a deterministic resumable artifact queue backed by one manifest.
+Successful expensive work survives a later failure, invalid dependents cannot
+remain current, and ordinary resume continues at the first work that actually
+needs attention.
+
+### Workstream 7 — Configuration structure and semantic diagnostics
+
+**Status:** Pending
+
+**Audit sources:** H04, H14, H15, H17, H38, H39, H40, H41, H42, C01, and C06.
+
+#### Underlying issue
+
+Configuration behavior is appropriately strict, but unrelated validation
+domains share one large file, several functions combine distinct phases, legacy
+fields remain active, and tests treat exact diagnostic choreography as though
+it were a versioned semantic contract.
+
+#### Required changes
+
+- Complete the legacy configuration purge defined by the canonical-contract
+  workstream.
+- Split validation within package `config` into core pipeline/session,
+  external-adapter, publish/storage, and Scriptorium artifact/dependency
+  domains.
+- Separate Scriptorium artifact declaration and output checks,
+  dependency/input checks, and variable checks. Traverse declarations
+  deterministically where ordering affects reproducibility, without promising
+  one particular independent-error discovery order.
+- Separate Notarius reference-binding normalization from output-lane
+  normalization and collision checks; retain enablement and scalar checks in
+  the top-level validator.
+- Move the cohesive Audita validator with the other adapter validators without
+  fragmenting its straightforward guards.
+- Split composition-envelope shape decoding, cross-field semantic validation,
+  and removal of root-only metadata while retaining Narratio's product-specific
+  composition model.
+- Preserve strict YAML decoding, unknown-field rejection, additive imports with
+  no duplicate ownership, deterministic profile selection, meaningful overlay
+  behavior for explicit values and collections, centralized defaults, canonical
+  party and artifact-family resolution, and secret-free effective digests.
+- Preserve actionable field identification and enough source information to
+  correct composition conflicts. Stop preserving complete prose, punctuation,
+  formatting, validation call order, or unused per-node provenance.
+- Rewrite brittle tests to assert typed categories, field paths, structured
+  details, or the smallest stable semantic fragment. Retain exact strings only
+  for intentionally stable CLI or machine-consumed contracts.
+
+#### Target state
+
+Configuration remains strict, deterministic, and operator-friendly, but its
+implementation is organized by semantic owner and can improve diagnostics
+without treating private ordering or wording as compatibility behavior.
+
+### Workstream 8 — Remote publication, per-artifact locks, and publish decomposition
+
+**Status:** Pending
+
+**Audit sources:** H04, H15, H18, H44, and C05.
+
+#### Underlying issue
+
+Publish contains several cohesive concerns in one large file, while remote
+locks use a shared mutable YAML control plane with static precedence, merging,
+conditional replacement, and retry machinery. Omitting a locked artifact from
+a new immutable snapshot can also make the canonical artifact disappear rather
+than keep it frozen.
+
+#### Required changes
+
+- Split publish source discovery and opening, output-rule resolution, transfer
+  preparation and verification, immutable upload, and commit/pointer encoding
+  into focused files while retaining `publishStage.Run` as the transaction
+  owner.
+- Replace static pipeline locks and the central remote lock manifest with one
+  versioned JSON lock object per normalized stable source ID at the approved
+  hashed key. Validate that each payload source hashes to its object name.
+- Remove static/remote merging, precedence, configuration injection, shared
+  document replacement retries, and the retired lock schema and tests.
+- Extend the narrow object-store port and each implementation or fake with the
+  deletion operation needed to remove one lock.
+- Adapt lock list/add/remove behavior to independent objects: list validates the
+  prefix, add fails if present unless force replaces metadata, and remove treats
+  absence as idempotent success.
+- Check each relevant lock before spending work to publish its artifact and
+  recheck immediately before advancing the current pointer.
+- Build a new commit so an unlocked artifact uses its newly uploaded
+  declaration, while a locked artifact carries the exact prior immutable
+  declaration when one exists. A locked artifact never previously published may
+  remain absent without becoming a required-output error.
+- If the final recheck discovers a newly added lock, do not advance the pointer;
+  allow unselected immutable attempt objects to remain and rebuild on the next
+  run. A lock removed after the initial read may conservatively carry the prior
+  declaration once.
+- Allow carried declarations to reference an earlier immutable run prefix in
+  the same campaign and session, retaining source identity, originating run,
+  destination, checksum, size, and generation. Readers must use only objects
+  declared by the selected commit and must not scan mutable prefixes.
+- Distinguish newly published, carried-forward locked,
+  locked-but-never-published, optional missing, and unselected outcomes in
+  publish metadata and inspection.
+- Preserve the visible transaction boundary: immutable objects, immutable
+  commit, final lock revalidation, then one mutable current-pointer write.
+
+#### Target state
+
+Publish remains one coherent immutable-snapshot transaction with navigable
+internal files. Each lock is an independent remote object, changes to one lock
+cannot overwrite another, and later snapshots retain the exact prior canonical
+version of a locked artifact while unlocked artifacts advance.
+
+### Workstream 9 — Cleanup as the final pipeline stage
+
+**Status:** Pending
+
+**Audit sources:** H20, H21, C02, and C04.
+
+#### Underlying issue
+
+Local deletion currently occurs as a post-publish transaction with a special
+manifest obligation. A cleanup error can therefore make a successfully
+committed remote publication appear to have failed, and the extra state model
+duplicates ordinary stage retry behavior.
+
+#### Required changes
+
+- Add `cleanup` as the final canonical stage, after every stage that may need
+  run-local data or diagnostics.
+- Remove post-publish cleanup from the runner and delete the
+  `post_publish_cleanup` manifest model, persistence, compatibility code,
+  tests, and documentation.
+- Use existing cleanup configuration to select spool audio, run workspace data,
+  both, or neither.
+- Require evidence of a successful current remote publish before deleting data
+  whose retention depends on publication, and bind targets to that published
+  run rather than inferring arbitrary paths from current configuration.
+- Self-skip clearly when nothing is configured, treat already-absent eligible
+  targets as success, use the filesystem workstream's root-scoped safeguards,
+  and produce no canonical artifact output.
+- Record cleanup through ordinary session-manifest stage status. A cleanup
+  failure must leave publish success and remote current state unchanged; an
+  ordinary later run should retry cleanup while reusing valid upstream stages.
+- Include cleanup in the fixed invalidation relation so a changed or forced
+  prerequisite makes an earlier cleanup result stale where appropriate.
+- Ensure a selected range ending before cleanup performs no deletion, while a
+  default full run includes cleanup.
+
+#### Target state
+
+Remote publication and local deletion are distinct, understandable outcomes.
+Publish succeeds when its pointer becomes current; cleanup succeeds when its
+eligible local targets are absent. Cleanup failure is reported and retried as a
+cleanup-stage failure without republishing unchanged data.
+
+### Workstream 10 — Quality, testability, and documentation alignment
+
+**Status:** Pending (cross-cutting)
+
+**Audit sources:** H07, H08, H12, H13, H17, H18, H23, H25, C01, C02, C03,
+C04, C05, and C06.
+
+#### Underlying issue
+
+Most tests protect meaningful behavior, but some preserve mechanisms and
+contracts now approved for removal, some application tests rely on mutable
+global seams, and large structural failures can be difficult to diagnose.
+Canonical documentation also describes several filesystem mechanisms as
+durable policy.
+
+#### Required changes
+
+- Delete tests, fixtures, hooks, and helpers only when their underlying legacy
+  format, run-manifest model, cleanup obligation, central lock store, or FileOps
+  guarantee is removed.
+- Add or retain focused behavior coverage for atomic visibility, bounded limits
+  and adapter errors, group modes, validated downloads, Notarius rename,
+  destructive scope, restore symlink conflicts, secret redaction, session-only
+  progress, per-artifact checkpoints, planning/execution parity, lock
+  carry-forward, and idempotent cleanup.
+- Prefer real temporary filesystems, fake object stores, and controlled helper
+  processes at their current durable boundaries. Do not recreate removed
+  algorithms through private test hooks.
+- Replace affected mutable application globals with the invocation-scoped
+  dependency set defined by the runner workstream.
+- Use `cmp.Diff` for selected large composed configurations, analyze state,
+  restore plans, and publish mappings. Use direct or standard-library equality
+  for simple values.
+- Apply `slices` and `maps` only to shallow clones, membership checks, and
+  deterministic key handling in code already being touched, with nil, empty,
+  and ordering semantics verified.
+- Update architecture, operations, troubleshooting, integration, internal,
+  configuration, CLI, examples, and release documentation only where each is
+  the canonical owner of changed behavior. Remove retired claims rather than
+  maintaining compatibility narratives as current behavior.
+- Keep repository-wide tests and vet mandatory throughout. Apply focused race,
+  documentation/example, and cross-platform release-build checks in proportion
+  to each workstream, with the complete release-candidate validation at the
+  integrated boundary.
+
+#### Target state
+
+Tests protect retained outcomes and failure boundaries instead of obsolete
+mechanisms. Collaborators are injectable per invocation without a framework,
+large diffs are readable where useful, and canonical documentation accurately
+describes the simplified implemented contracts as each workstream lands.
+
+## Dependency Relationships
+
+These relationships constrain a future implementation plan without defining
+its numbered stages:
+
+- The Go 1.26 baseline is required before relying on the process-handle API;
+  the root-removal API is already present in the current Go 1.25 baseline.
+- The compatibility purge should remove legacy branches before maintainers
+  invest in decomposing the corresponding current-only previous-cache,
+  configuration, analyze, and publication paths.
+- Removing the run manifest establishes the session-only authority needed by
+  runner decomposition, per-artifact analyze checkpointing, and the ordinary
+  cleanup stage.
+- Runner lifecycle decomposition and its pre-execution decision model provide
+  the clean stage boundary used by analyze recovery and cleanup retry.
+- The per-artifact lock representation and carry-forward rules must be reflected
+  together in publish construction, commit validation, current-state reading,
+  operator commands, and tests.
+- Filesystem simplification and root-scoped destructive helpers must be
+  available to the cleanup stage, while restore correctness belongs in the same
+  filesystem change rather than the old facade.
+- Test and documentation alignment accompanies the workstream that changes the
+  behavior; it is not deferred to a final bulk cleanup.
+
+## Compatibility and Release Intent
+
+The compatibility purge intentionally rejects deprecated formats rather than
+silently migrating them. Migration guidance should direct users to regenerate,
+republish, or otherwise replace old state. No runtime compatibility layer should
+survive solely to make the internal refactor invisible.
+
+The approved audit decision is that this combined work remains eligible for a
+minor release because the removed compatibility surfaces were already
+deprecated and explicitly removable. Preparing or publishing that release is
+not part of this roadmap.
+
+## Consolidated Target State
+
+After this roadmap is implemented:
+
+- Narratio has one canonical configuration and state model with no deprecated
+  runtime fallback paths;
+- the session manifest alone owns durable local progress;
+- analyze preserves each valid successful artifact and resumes failed, stale,
+  or missing work in dependency order;
+- application orchestration has explicit private lifecycle phases and one
+  narrow planning/execution decision model;
+- ordinary filesystem work uses the standard library, while small focused
+  helpers retain atomic visibility, bounds, modes, destructive scope, restore
+  safety, and secret protection;
+- Notarius output is installed by one validated rename without a duplicate tree;
+- publish owns coherent remote publication, per-artifact locks preserve prior
+  canonical declarations, and cleanup is a separate final stage;
+- configuration remains strict and deterministic without freezing incidental
+  diagnostic choreography;
+- Windows subprocess ownership and transcription concurrency use the audited
+  Go 1.26 primitives, and subprocess mechanism is internally distinct from
+  Narratio policy; and
+- tests and documentation protect and describe the reduced observable contract.
+
+## Open Questions
+
+None. The audits and subsequent approved decisions resolve the product and
+architecture choices required to prepare a staged implementation plan.