Plan Go and process refactoring

docs/roadmap/library-audit.md

@@ -0,0 +1,395 @@
+# Reusable Library Extraction Audit
+
+## Status
+
+Complete. This audit reviews the Narratio code at `af5fbc1c4f8d` and records
+candidate-extraction decisions for future refactoring. It does not authorize an
+extraction, create a new module, or change application behavior.
+
+This is the third maintainability pass. It complements the two earlier passes:
+
+1. [`audit.md`](audit.md) asks whether implementation can be simplified while
+   preserving current behavior and contracts.
+2. [`contract-audit.md`](contract-audit.md) asks whether apparent contracts can
+   reasonably be changed to permit substantially simpler behavior.
+3. This document asks whether code that Narratio no longer needs should instead
+   be preserved as a generic, independently reusable library.
+
+The question in this pass is deliberately separate from whether Narratio should
+continue using an extracted library. Code may be valuable in a more demanding
+application while remaining the wrong dependency for Narratio.
+
+## Method and Decision Standard
+
+The review inventoried Narratio's internal packages, concentrated on mechanisms
+whose contracts can be stated without D&D or pipeline terminology, inspected
+their call sites and tests, and compared the most plausible candidates with the
+actual implementations in the neighboring Notarius and WeatherReporter
+repositories. It also checked the Go standard library and representative
+maintained or emerging third-party implementations before treating custom code
+as a library opportunity.
+
+A candidate is suitable for extraction only when all of the following are true:
+
+- its contract can be explained without Narratio types or product policy;
+- Narratio-specific configuration, paths, modes, diagnostics, and error prose
+  can remain in Narratio rather than leak into the generic API;
+- at least one credible independent consumer needs materially the same
+  guarantees;
+- extraction removes meaningful implementation and test ownership from
+  consumers instead of adding adapters around a small helper;
+- the generic API is narrow, stable, and independently testable on every
+  promised platform;
+- an existing maintained library does not already satisfy the need; and
+- the extracted module has an identified maintenance, compatibility, release,
+  and vulnerability-response owner.
+
+The verdicts used below are:
+
+- **Extract now:** a proven shared contract and consumer justify a separate
+  maintained module.
+- **Incubate:** the code contains a credible generic core, but extraction should
+  wait for a concrete trigger.
+- **Keep internal:** the implementation is coherent but belongs to Narratio's
+  domain or is too small to justify an external lifecycle.
+- **Simplify or delete:** independent reuse does not justify retaining the code
+  in Narratio; source history is enough to preserve ideas for later recovery.
+- **Prefer an established abstraction:** a future generic consumer should start
+  from an existing standard or library rather than Narratio's implementation.
+
+## Executive Summary
+
+No package should be extracted before the approved Narratio refactoring begins.
+The audit found one serious incubation candidate: the cross-platform subprocess
+supervision core. It implements a substantial and reasonably generic process-
+tree lifecycle that the Go standard library does not provide as one operation.
+It is nevertheless mixed with Narratio-owned environment, logging, redaction,
+filesystem, and diagnostic policy, and neither Notarius nor WeatherReporter is
+currently a matching second consumer. Extraction now would therefore invent a
+public API before its reuse requirements are known.
+
+The current FileOps subsystem may be useful source material for an application
+that genuinely needs hostile-race resistance, confined traversal, no-replace
+directory promotion, or power-loss durability. It is not a cohesive generic
+library as written, and its strongest guarantees are specifically the ones
+Narratio has decided to retire. The approved FileOps simplification should
+proceed. Git history is a sufficient preservation mechanism unless a real
+external consumer appears before deletion.
+
+Configuration composition, remote commit snapshots, artifact policy, session
+state, and pipeline recovery contain useful patterns, but their contracts are
+Narratio product contracts rather than generic library seams. Small path,
+bounded-I/O, locking, identifier, and atomic-write helpers do not carry enough
+independent value to justify separate versioning. Release automation is a
+possible shared tool or template, not a Go library, and current sibling
+repositories do not yet share one implementation that needs central ownership.
+
+| Candidate | Verdict | Refactoring consequence |
+| --- | --- | --- |
+| Cross-platform subprocess supervision | Incubate | Separate generic mechanism from Narratio policy locally; extract only after a second consumer and owner exist. |
+| Hardened FileOps subsystem | Simplify or delete | Follow `fileops.md`; preserve history, not a Narratio dependency on an overbuilt library. |
+| YAML composition and provenance | Keep internal | Retain as Narratio's configuration DSL; reconsider structural merge dependencies separately if they reduce code. |
+| Immutable remote commit and pointer protocol | Keep internal | Preserve as Narratio's publication contract; do not market it as a generic object-store transaction library. |
+| Path-safety and bounded-I/O helpers | Keep internal | Prefer small local helpers and newer standard-library primitives where semantics match. |
+| Atomic file replacement | Keep internal for Narratio | Use a small cross-platform helper or an established dependency after focused implementation research; do not extract a new module. |
+| Object-store interface and session locks | Keep internal | These are narrow ports and product coordination rules, not reusable implementations. |
+| Release scripts and checks | Keep repository-local | Share a template or standalone tool only if multiple repositories develop materially identical behavior and drift. |
+| Artifact, manifest, analyze, and pipeline policy | Keep internal | These are Narratio domain code even where individual algorithms are conventional. |
+
+## Candidate 1 — Cross-Platform Subprocess Supervision
+
+### Why it is a credible library seed
+
+`internal/adapters/subprocess` contains substantially more than a wrapper around
+`os/exec`. Its process owner:
+
+- starts each command in an owned Unix process group or Windows Job Object;
+- prevents the Windows child from running before job assignment;
+- terminates the owned tree on cancellation or timeout;
+- attempts graceful then forceful shutdown where the platform supports it;
+- reaps the leader and bounds waits when descendants retain stream handles;
+- distinguishes exit, timeout, cancellation, capture-limit, and cleanup
+  outcomes; and
+- has focused Unix and Windows tests for the lifecycle.
+
+That core contract is understandable without Narratio terminology and is large
+enough that a proven consumer could benefit from sharing it. The Go
+[`os/exec`](https://pkg.go.dev/os/exec) package supplies command execution,
+custom cancellation, and `WaitDelay`, but its default cancellation targets the
+leader process rather than establishing cross-platform process-tree ownership.
+
+### What is not generic
+
+The package currently also owns policy that must not cross an extraction
+boundary:
+
+- Narratio's inherited-environment allowlist and test-helper variables;
+- API-key environment names and extraction of their secret values;
+- fixed diagnostic size limits and application-specific owner wording;
+- log paths, group-owned file modes, and FileOps calls;
+- stderr-tail formatting and Python/file-descriptor diagnostic hints; and
+- atomic YAML and byte-writing helpers unrelated to process ownership.
+
+A reusable core should accept an explicit environment and ordinary `io.Writer`
+or capture interfaces. Narratio should remain responsible for choosing and
+redacting the environment, opening bounded diagnostic sinks, selecting paths
+and modes, and translating typed results into operator-facing errors.
+
+### Existing alternatives
+
+The standard library remains the foundation but does not provide the whole-tree
+contract. Third-party packages now exist in this space, including
+[`processkit-go`](https://github.com/ZelAnton/processkit-go), but the candidates
+found during this audit are comparatively new or offer a materially broader
+contract than Narratio needs. They should be reevaluated at extraction time;
+the existence and maturity of this category are changing quickly, and copying
+Narratio code into a new module is not automatically preferable.
+
+### Decision and extraction trigger
+
+**Verdict: Incubate; do not extract now.**
+
+During Narratio refactoring, make the internal mechanism/policy boundary clear
+and remove unrelated file-writing helpers from the package. Do not contort the
+internal API solely to resemble a hypothetical public module.
+
+Reopen extraction only when all of these triggers are present:
+
+1. a second production application needs whole-process-tree ownership on at
+   least two of Narratio's supported operating systems;
+2. its timeout, graceful-shutdown, cleanup, and result semantics substantially
+   overlap Narratio's rather than merely needing `exec.CommandContext`;
+3. maintained alternatives have been reevaluated and rejected for documented
+   technical or maintenance reasons;
+4. an independent module API can exclude all Narratio filesystem, environment,
+   redaction, and diagnostic policy; and
+5. a maintainer accepts cross-platform CI, semantic versioning, releases, and
+   security ownership for the module.
+
+If those conditions are met, extract the generic core from the simplified
+internal boundary and let Narratio consume it through a thin policy adapter.
+
+## Candidate 2 — Hardened FileOps
+
+### Why the code may be useful elsewhere
+
+The present `internal/fileops` implementation includes confined root operations,
+symlink-resistant traversal, identity rechecks, same-directory replacement,
+checksummed copy and download installation, directory tree validation, atomic
+no-replace promotion, mode enforcement, file and directory syncing, and
+platform-specific implementations. An application operating on an untrusted,
+concurrently modified tree could reasonably need some of those mechanisms.
+
+### Why it is not an extraction candidate as written
+
+The package is a bundle of distinct concerns rather than one small generic
+contract. It combines:
+
+- atomic single-file visibility and durability;
+- hostile-race-resistant traversal and copying;
+- directory transaction and no-replace semantics;
+- Narratio workspace ownership modes;
+- bounded application I/O; and
+- application-specific unsupported-platform and contextual errors.
+
+A credible external library would first need to choose which threat model and
+durability level it promises, define a much smaller API, specify filesystem and
+platform assumptions, and split policy such as modes and limits from mechanism.
+That is a new library design project, not a package move.
+
+There is also no matching current consumer nearby. Notarius implements its
+ordinary atomic byte write in a small local helper, and WeatherReporter likewise
+uses a focused same-directory temporary-file replacement helper. Neither needs
+Narratio's complete confined traversal and directory-promotion contract.
+
+Existing components cover portions of the space. Go provides
+[`os.Root`](https://pkg.go.dev/os#Root) for root-scoped operations; libraries
+such as [`renameio`](https://github.com/google/renameio) and
+[`moby/sys/atomicwriter`](https://github.com/moby/sys/tree/main/atomicwriter)
+address atomic file replacement with different platform, durability, and mode
+tradeoffs. No single dependency should be selected merely to preserve the
+current aggregate abstraction.
+
+### Decision
+
+**Verdict: Simplify or delete; do not extract before refactoring.**
+
+Implement the reduced contract in [`fileops.md`](fileops.md). Preserve the
+current implementation through version control and, if useful, a short design
+note identifying the historical commit. Do not make Narratio depend on an
+externalized version of guarantees that Narratio has explicitly rejected.
+
+If another application later demonstrates a need, design a new module from its
+requirements and selectively recover proven algorithms and tests from history.
+That work should not block or enlarge the Narratio refactor.
+
+## Candidate 3 — YAML Composition and Provenance
+
+`internal/config/composition.go` implements a presence-aware YAML tree with
+source identity, duplicate-key detection, additive disjoint-source composition,
+profile overlays, deterministic canonical rendering, semantic leaf projection,
+and source attribution. These mechanics look generic at first glance.
+
+The actual merge algebra is nevertheless Narratio's configuration language:
+root-only composition metadata, disjoint imports, named pipeline profiles,
+overlay ownership, atomic list treatment, null and kind-change policy, pipeline
+field provenance, effective digests, and strict decoding into `PipelineConfig`.
+The useful diagnostics also depend on those policy choices. Extracting only the
+tree mechanics would leave much of the code and all product decisions in
+Narratio, while extracting the whole package would publish Narratio's DSL as a
+supposedly generic API.
+
+Notarius does not implement the same import/profile language, and
+WeatherReporter's configuration merge is a short typed decode into its own
+configuration. They are not current consumers. Structural YAML libraries such
+as [`yaml/go-yaml-dom`](https://github.com/yaml/go-yaml-dom) now provide clone
+and configurable merge operations, but they do not by themselves establish
+Narratio's provenance and additive-ownership rules.
+
+**Verdict: Keep internal.** Keep product semantics explicit in Narratio. A
+future implementation pass may evaluate a maintained YAML DOM/merge dependency
+as a way to delete low-level node mechanics, but dependency adoption and public
+library extraction are different decisions. Do not create a generic module
+without a second consumer that independently chooses the same merge algebra.
+
+## Candidate 4 — Immutable Remote Commit Protocol
+
+The remote publication design uses immutable run-scoped objects, checksummed
+and generation-bound artifact descriptors, an immutable commit manifest, and a
+single mutable current pointer. The pattern is reusable in the abstract and
+provides coherent publication on an object store without pretending the store
+offers a multi-object transaction.
+
+Its concrete schema and validation are Narratio-specific: campaign and session
+identity, run IDs, session and run manifests, configured artifact sources,
+canonical S3 key layout, publication categories, previous-session resolution,
+and current-state reconstruction. The storage interface alone is not the
+protocol; extracting it would either export Narratio's schema or require a new
+generic content model and transaction API.
+
+Established content-addressed specifications already model similar generic
+ideas. The [OCI image specification](https://github.com/opencontainers/image-spec)
+defines descriptors, digests, manifests, indexes, and mutable names over
+content-addressed objects. Narratio should not migrate to OCI merely for reuse,
+but any future cross-application protocol should evaluate established formats
+before turning Narratio's private contract into a library.
+
+**Verdict: Keep internal.** Treat the implementation as a good domain protocol,
+not a generic object-store transaction package. Reconsider only if multiple
+applications need interoperable snapshots; at that point start with a shared
+protocol specification and compatibility requirements, not a code extraction.
+
+## Candidate 5 — Small Infrastructure Helpers
+
+### Path safety
+
+`internal/pathsafe` is small, pure, and widely used, but it encodes Narratio's
+slash-style relative destination rules and cross-platform object-key behavior.
+It is below the threshold where a separate module's dependency and release
+lifecycle would pay for itself. Newer Go path APIs such as
+[`filepath.IsLocal`](https://pkg.go.dev/path/filepath#IsLocal) should be used
+where their host-path semantics exactly match; object keys and canonical slash
+paths may still warrant a few local functions.
+
+**Verdict: Keep internal and small.**
+
+### Bounded I/O and streaming redaction
+
+Bounded reads, tail capture, and streaming secret redaction are useful generic
+techniques. Narratio's instances are tied to adapter-specific limits, owner
+names, failure wording, log files, and its API-key policy. Separating them into
+external modules would trade a few local helpers for additional APIs without a
+known consumer.
+
+**Verdict: Keep internal.** If subprocess incubation later proceeds, bounded
+capture may become part of that module only if consumers agree on typed limit
+semantics; secret selection and error prose remain application policy.
+
+### Atomic file replacement
+
+Same-directory temporary-file replacement is generic, but Narratio's reduced
+need is a small operation with explicit final mode and supported-platform
+behavior. Notarius and WeatherReporter each maintain a short local variant, but
+their differing modes, cancellation points, and parent-directory behavior show
+that superficial duplication is not yet one shared contract.
+
+**Verdict: Do not create a new library.** During FileOps implementation,
+reevaluate established atomic-writer packages against Narratio's exact Windows,
+mode, and atomic-visibility requirements. Otherwise retain a small local helper.
+
+### Object-store ports, file locks, IDs, and DAG helpers
+
+The object-store interfaces are deliberately narrow ports over provider SDKs;
+the implementations and error semantics remain application adapters. Session
+locks enforce Narratio's one-session execution rule. Run/session identifiers
+and artifact dependency traversal are bound to Narratio schemas and lifecycle.
+Extracting any of these would either expose domain types or create generic
+abstractions with less clarity than the current code.
+
+**Verdict: Keep internal.** Use established provider SDKs and standard-library
+algorithms directly rather than creating thin generic wrapper modules.
+
+## Candidate 6 — Release Automation
+
+Narratio's release scripts combine source-tree validation, asset-matrix builds,
+archive naming, checksum generation, Git state checks, tag creation, and push
+behavior. Those ideas are reusable, but the concrete release contract includes
+Narratio's binary name, supported target matrix, documentation, tag policy,
+asset layout, and responsibility boundary. Notarius currently has a narrower
+source-release check, while WeatherReporter documents its procedure without a
+matching shared script suite.
+
+**Verdict: Keep repository-local.** This is not a runtime Go library candidate.
+If two or more repositories later carry materially identical scripts and fixes
+begin to drift, extract a versioned standalone release tool or a maintained
+template with per-project declarative metadata. Until then, local scripts keep
+release changes atomic with each repository and avoid a bootstrap dependency in
+the release path.
+
+## Explicit Non-Candidates
+
+The following code is intentionally excluded from generic extraction despite
+containing reusable programming techniques:
+
+- artifact source classification and published-destination policy;
+- session and remote manifest schemas and validation;
+- stage ordering, selection, invalidation, resume, and force semantics;
+- analyze artifact checkpointing and dependency-aware recovery;
+- cleanup and remote lock policy;
+- Notarius, Seriatim, Scriptorium, Audita, WhisperX, and S3 adapters; and
+- configuration validation domains and party/artifact-family expansion.
+
+Their value comes from making Narratio's product rules explicit. Moving them to
+a generic module would hide ownership rather than create reuse. Conventional
+algorithms inside domain code are not by themselves library candidates.
+
+## Refactoring Guidance
+
+This audit does not add an extraction prerequisite to the approved refactoring.
+Use the following sequence:
+
+1. implement the FileOps and contract simplifications already recorded in the
+   roadmap documents;
+2. keep refactored domain ownership within Narratio unless a stage explicitly
+   calls for a package seam;
+3. while touching subprocess code, separate process ownership from Narratio's
+   diagnostics and filesystem policy, but retain an internal API;
+4. record deleted FileOps history rather than preserving dead compatibility or
+   an external dependency for hypothetical users; and
+5. revisit subprocess extraction only when the trigger conditions in this
+   audit are satisfied.
+
+An extraction, if later approved, should be planned and released as its own
+project. It should not be smuggled into a Narratio refactoring stage, because
+public API design, multi-platform CI, module ownership, migration, and release
+policy are independent deliverables.
+
+## Final State of This Audit
+
+The audit preserves one future option without delaying simplification:
+Narratio's subprocess supervision core is a credible library seed, while all
+other reviewed areas should remain internal, use established abstractions, or
+be simplified and deleted. No current code must be retained merely because it
+might someday be reusable. Version control preserves implementation history,
+and a demonstrated second consumer—not speculative generality—will trigger any
+future extraction.