narratio/docs/roadmap/library-audit.md

# Reusable Library Extraction Audit

## Status

Complete. This audit reviews the Narratio code at `af5fbc1c4f8d` and records
candidate-extraction decisions for future refactoring. It does not authorize an
extraction, create a new module, or change application behavior.

This is the third maintainability pass. It complements the two earlier passes:

1. [`audit.md`](audit.md) asks whether implementation can be simplified while
   preserving current behavior and contracts.
2. [`contract-audit.md`](contract-audit.md) asks whether apparent contracts can
   reasonably be changed to permit substantially simpler behavior.
3. This document asks whether code that Narratio no longer needs should instead
   be preserved as a generic, independently reusable library.

The question in this pass is deliberately separate from whether Narratio should
continue using an extracted library. Code may be valuable in a more demanding
application while remaining the wrong dependency for Narratio.

## Method and Decision Standard

The review inventoried Narratio's internal packages, concentrated on mechanisms
whose contracts can be stated without D&D or pipeline terminology, inspected
their call sites and tests, and compared the most plausible candidates with the
actual implementations in the neighboring Notarius and WeatherReporter
repositories. It also checked the Go standard library and representative
maintained or emerging third-party implementations before treating custom code
as a library opportunity.

A candidate is suitable for extraction only when all of the following are true:

- its contract can be explained without Narratio types or product policy;
- Narratio-specific configuration, paths, modes, diagnostics, and error prose
  can remain in Narratio rather than leak into the generic API;
- at least one credible independent consumer needs materially the same
  guarantees;
- extraction removes meaningful implementation and test ownership from
  consumers instead of adding adapters around a small helper;
- the generic API is narrow, stable, and independently testable on every
  promised platform;
- an existing maintained library does not already satisfy the need; and
- the extracted module has an identified maintenance, compatibility, release,
  and vulnerability-response owner.

The verdicts used below are:

- **Extract now:** a proven shared contract and consumer justify a separate
  maintained module.
- **Incubate:** the code contains a credible generic core, but extraction should
  wait for a concrete trigger.
- **Keep internal:** the implementation is coherent but belongs to Narratio's
  domain or is too small to justify an external lifecycle.
- **Simplify or delete:** independent reuse does not justify retaining the code
  in Narratio; source history is enough to preserve ideas for later recovery.
- **Prefer an established abstraction:** a future generic consumer should start
  from an existing standard or library rather than Narratio's implementation.

## Executive Summary

No package should be extracted before the approved Narratio refactoring begins.
The audit found one serious incubation candidate: the cross-platform subprocess
supervision core. It implements a substantial and reasonably generic process-
tree lifecycle that the Go standard library does not provide as one operation.
It is nevertheless mixed with Narratio-owned environment, logging, redaction,
filesystem, and diagnostic policy, and neither Notarius nor WeatherReporter is
currently a matching second consumer. Extraction now would therefore invent a
public API before its reuse requirements are known.

The current FileOps subsystem may be useful source material for an application
that genuinely needs hostile-race resistance, confined traversal, no-replace
directory promotion, or power-loss durability. It is not a cohesive generic
library as written, and its strongest guarantees are specifically the ones
Narratio has decided to retire. The approved FileOps simplification should
proceed. Git history is a sufficient preservation mechanism unless a real
external consumer appears before deletion.

Configuration composition, remote commit snapshots, artifact policy, session
state, and pipeline recovery contain useful patterns, but their contracts are
Narratio product contracts rather than generic library seams. Small path,
bounded-I/O, locking, identifier, and atomic-write helpers do not carry enough
independent value to justify separate versioning. Release automation is a
possible shared tool or template, not a Go library, and current sibling
repositories do not yet share one implementation that needs central ownership.

| Candidate | Verdict | Refactoring consequence |
| --- | --- | --- |
| Cross-platform subprocess supervision | Incubate | Separate generic mechanism from Narratio policy locally; extract only after a second consumer and owner exist. |
| Hardened FileOps subsystem | Simplify or delete | Follow `fileops.md`; preserve history, not a Narratio dependency on an overbuilt library. |
| YAML composition and provenance | Keep internal | Retain as Narratio's configuration DSL; reconsider structural merge dependencies separately if they reduce code. |
| Immutable remote commit and pointer protocol | Keep internal | Preserve as Narratio's publication contract; do not market it as a generic object-store transaction library. |
| Path-safety and bounded-I/O helpers | Keep internal | Prefer small local helpers and newer standard-library primitives where semantics match. |
| Atomic file replacement | Keep internal for Narratio | Use a small cross-platform helper or an established dependency after focused implementation research; do not extract a new module. |
| Object-store interface and session locks | Keep internal | These are narrow ports and product coordination rules, not reusable implementations. |
| Release scripts and checks | Keep repository-local | Share a template or standalone tool only if multiple repositories develop materially identical behavior and drift. |
| Artifact, manifest, analyze, and pipeline policy | Keep internal | These are Narratio domain code even where individual algorithms are conventional. |

## Candidate 1 — Cross-Platform Subprocess Supervision

### Why it is a credible library seed

`internal/adapters/subprocess` contains substantially more than a wrapper around
`os/exec`. Its process owner:

- starts each command in an owned Unix process group or Windows Job Object;
- prevents the Windows child from running before job assignment;
- terminates the owned tree on cancellation or timeout;
- attempts graceful then forceful shutdown where the platform supports it;
- reaps the leader and bounds waits when descendants retain stream handles;
- distinguishes exit, timeout, cancellation, capture-limit, and cleanup
  outcomes; and
- has focused Unix and Windows tests for the lifecycle.

That core contract is understandable without Narratio terminology and is large
enough that a proven consumer could benefit from sharing it. The Go
[`os/exec`](https://pkg.go.dev/os/exec) package supplies command execution,
custom cancellation, and `WaitDelay`, but its default cancellation targets the
leader process rather than establishing cross-platform process-tree ownership.

### What is not generic

The package currently also owns policy that must not cross an extraction
boundary:

- Narratio's inherited-environment allowlist and test-helper variables;
- API-key environment names and extraction of their secret values;
- fixed diagnostic size limits and application-specific owner wording;
- log paths, group-owned file modes, and FileOps calls;
- stderr-tail formatting and Python/file-descriptor diagnostic hints; and
- atomic YAML and byte-writing helpers unrelated to process ownership.

A reusable core should accept an explicit environment and ordinary `io.Writer`
or capture interfaces. Narratio should remain responsible for choosing and
redacting the environment, opening bounded diagnostic sinks, selecting paths
and modes, and translating typed results into operator-facing errors.

### Existing alternatives

The standard library remains the foundation but does not provide the whole-tree
contract. Third-party packages now exist in this space, including
[`processkit-go`](https://github.com/ZelAnton/processkit-go), but the candidates
found during this audit are comparatively new or offer a materially broader
contract than Narratio needs. They should be reevaluated at extraction time;
the existence and maturity of this category are changing quickly, and copying
Narratio code into a new module is not automatically preferable.

### Decision and extraction trigger

**Verdict: Incubate; do not extract now.**

During Narratio refactoring, make the internal mechanism/policy boundary clear
and remove unrelated file-writing helpers from the package. Do not contort the
internal API solely to resemble a hypothetical public module.

Reopen extraction only when all of these triggers are present:

1. a second production application needs whole-process-tree ownership on at
   least two of Narratio's supported operating systems;
2. its timeout, graceful-shutdown, cleanup, and result semantics substantially
   overlap Narratio's rather than merely needing `exec.CommandContext`;
3. maintained alternatives have been reevaluated and rejected for documented
   technical or maintenance reasons;
4. an independent module API can exclude all Narratio filesystem, environment,
   redaction, and diagnostic policy; and
5. a maintainer accepts cross-platform CI, semantic versioning, releases, and
   security ownership for the module.

If those conditions are met, extract the generic core from the simplified
internal boundary and let Narratio consume it through a thin policy adapter.

## Candidate 2 — Hardened FileOps

### Why the code may be useful elsewhere

The present `internal/fileops` implementation includes confined root operations,
symlink-resistant traversal, identity rechecks, same-directory replacement,
checksummed copy and download installation, directory tree validation, atomic
no-replace promotion, mode enforcement, file and directory syncing, and
platform-specific implementations. An application operating on an untrusted,
concurrently modified tree could reasonably need some of those mechanisms.

### Why it is not an extraction candidate as written

The package is a bundle of distinct concerns rather than one small generic
contract. It combines:

- atomic single-file visibility and durability;
- hostile-race-resistant traversal and copying;
- directory transaction and no-replace semantics;
- Narratio workspace ownership modes;
- bounded application I/O; and
- application-specific unsupported-platform and contextual errors.

A credible external library would first need to choose which threat model and
durability level it promises, define a much smaller API, specify filesystem and
platform assumptions, and split policy such as modes and limits from mechanism.
That is a new library design project, not a package move.

There is also no matching current consumer nearby. Notarius implements its
ordinary atomic byte write in a small local helper, and WeatherReporter likewise
uses a focused same-directory temporary-file replacement helper. Neither needs
Narratio's complete confined traversal and directory-promotion contract.

Existing components cover portions of the space. Go provides
[`os.Root`](https://pkg.go.dev/os#Root) for root-scoped operations; libraries
such as [`renameio`](https://github.com/google/renameio) and
[`moby/sys/atomicwriter`](https://github.com/moby/sys/tree/main/atomicwriter)
address atomic file replacement with different platform, durability, and mode
tradeoffs. No single dependency should be selected merely to preserve the
current aggregate abstraction.

### Decision

**Verdict: Simplify or delete; do not extract before refactoring.**

Implement the reduced contract in [`fileops.md`](fileops.md). Preserve the
current implementation through version control and, if useful, a short design
note identifying the historical commit. Do not make Narratio depend on an
externalized version of guarantees that Narratio has explicitly rejected.

If another application later demonstrates a need, design a new module from its
requirements and selectively recover proven algorithms and tests from history.
That work should not block or enlarge the Narratio refactor.

## Candidate 3 — YAML Composition and Provenance

`internal/config/composition.go` implements a presence-aware YAML tree with
source identity, duplicate-key detection, additive disjoint-source composition,
profile overlays, deterministic canonical rendering, semantic leaf projection,
and source attribution. These mechanics look generic at first glance.

The actual merge algebra is nevertheless Narratio's configuration language:
root-only composition metadata, disjoint imports, named pipeline profiles,
overlay ownership, atomic list treatment, null and kind-change policy, pipeline
field provenance, effective digests, and strict decoding into `PipelineConfig`.
The useful diagnostics also depend on those policy choices. Extracting only the
tree mechanics would leave much of the code and all product decisions in
Narratio, while extracting the whole package would publish Narratio's DSL as a
supposedly generic API.

Notarius does not implement the same import/profile language, and
WeatherReporter's configuration merge is a short typed decode into its own
configuration. They are not current consumers. Structural YAML libraries such
as [`yaml/go-yaml-dom`](https://github.com/yaml/go-yaml-dom) now provide clone
and configurable merge operations, but they do not by themselves establish
Narratio's provenance and additive-ownership rules.

**Verdict: Keep internal.** Keep product semantics explicit in Narratio. A
future implementation pass may evaluate a maintained YAML DOM/merge dependency
as a way to delete low-level node mechanics, but dependency adoption and public
library extraction are different decisions. Do not create a generic module
without a second consumer that independently chooses the same merge algebra.

## Candidate 4 — Immutable Remote Commit Protocol

The remote publication design uses immutable run-scoped objects, checksummed
and generation-bound artifact descriptors, an immutable commit manifest, and a
single mutable current pointer. The pattern is reusable in the abstract and
provides coherent publication on an object store without pretending the store
offers a multi-object transaction.

Its concrete schema and validation are Narratio-specific: campaign and session
identity, run IDs, session and run manifests, configured artifact sources,
canonical S3 key layout, publication categories, previous-session resolution,
and current-state reconstruction. The storage interface alone is not the
protocol; extracting it would either export Narratio's schema or require a new
generic content model and transaction API.

Established content-addressed specifications already model similar generic
ideas. The [OCI image specification](https://github.com/opencontainers/image-spec)
defines descriptors, digests, manifests, indexes, and mutable names over
content-addressed objects. Narratio should not migrate to OCI merely for reuse,
but any future cross-application protocol should evaluate established formats
before turning Narratio's private contract into a library.

**Verdict: Keep internal.** Treat the implementation as a good domain protocol,
not a generic object-store transaction package. Reconsider only if multiple
applications need interoperable snapshots; at that point start with a shared
protocol specification and compatibility requirements, not a code extraction.

## Candidate 5 — Small Infrastructure Helpers

### Path safety

`internal/pathsafe` is small, pure, and widely used, but it encodes Narratio's
slash-style relative destination rules and cross-platform object-key behavior.
It is below the threshold where a separate module's dependency and release
lifecycle would pay for itself. Newer Go path APIs such as
[`filepath.IsLocal`](https://pkg.go.dev/path/filepath#IsLocal) should be used
where their host-path semantics exactly match; object keys and canonical slash
paths may still warrant a few local functions.

**Verdict: Keep internal and small.**

### Bounded I/O and streaming redaction

Bounded reads, tail capture, and streaming secret redaction are useful generic
techniques. Narratio's instances are tied to adapter-specific limits, owner
names, failure wording, log files, and its API-key policy. Separating them into
external modules would trade a few local helpers for additional APIs without a
known consumer.

**Verdict: Keep internal.** If subprocess incubation later proceeds, bounded
capture may become part of that module only if consumers agree on typed limit
semantics; secret selection and error prose remain application policy.

### Atomic file replacement

Same-directory temporary-file replacement is generic, but Narratio's reduced
need is a small operation with explicit final mode and supported-platform
behavior. Notarius and WeatherReporter each maintain a short local variant, but
their differing modes, cancellation points, and parent-directory behavior show
that superficial duplication is not yet one shared contract.

**Verdict: Do not create a new library.** During FileOps implementation,
reevaluate established atomic-writer packages against Narratio's exact Windows,
mode, and atomic-visibility requirements. Otherwise retain a small local helper.

### Object-store ports, file locks, IDs, and DAG helpers

The object-store interfaces are deliberately narrow ports over provider SDKs;
the implementations and error semantics remain application adapters. Session
locks enforce Narratio's one-session execution rule. Run/session identifiers
and artifact dependency traversal are bound to Narratio schemas and lifecycle.
Extracting any of these would either expose domain types or create generic
abstractions with less clarity than the current code.

**Verdict: Keep internal.** Use established provider SDKs and standard-library
algorithms directly rather than creating thin generic wrapper modules.

## Candidate 6 — Release Automation

Narratio's release scripts combine source-tree validation, asset-matrix builds,
archive naming, checksum generation, Git state checks, tag creation, and push
behavior. Those ideas are reusable, but the concrete release contract includes
Narratio's binary name, supported target matrix, documentation, tag policy,
asset layout, and responsibility boundary. Notarius currently has a narrower
source-release check, while WeatherReporter documents its procedure without a
matching shared script suite.

**Verdict: Keep repository-local.** This is not a runtime Go library candidate.
If two or more repositories later carry materially identical scripts and fixes
begin to drift, extract a versioned standalone release tool or a maintained
template with per-project declarative metadata. Until then, local scripts keep
release changes atomic with each repository and avoid a bootstrap dependency in
the release path.

## Explicit Non-Candidates

The following code is intentionally excluded from generic extraction despite
containing reusable programming techniques:

- artifact source classification and published-destination policy;
- session and remote manifest schemas and validation;
- stage ordering, selection, invalidation, resume, and force semantics;
- analyze artifact checkpointing and dependency-aware recovery;
- cleanup and remote lock policy;
- Notarius, Seriatim, Scriptorium, Audita, WhisperX, and S3 adapters; and
- configuration validation domains and party/artifact-family expansion.

Their value comes from making Narratio's product rules explicit. Moving them to
a generic module would hide ownership rather than create reuse. Conventional
algorithms inside domain code are not by themselves library candidates.

## Refactoring Guidance

This audit does not add an extraction prerequisite to the approved refactoring.
Use the following sequence:

1. implement the FileOps and contract simplifications already recorded in the
   roadmap documents;
2. keep refactored domain ownership within Narratio unless a stage explicitly
   calls for a package seam;
3. while touching subprocess code, separate process ownership from Narratio's
   diagnostics and filesystem policy, but retain an internal API;
4. record deleted FileOps history rather than preserving dead compatibility or
   an external dependency for hypothetical users; and
5. revisit subprocess extraction only when the trigger conditions in this
   audit are satisfied.

An extraction, if later approved, should be planned and released as its own
project. It should not be smuggled into a Narratio refactoring stage, because
public API design, multi-platform CI, module ownership, migration, and release
policy are independent deliverables.

## Final State of This Audit

The audit preserves one future option without delaying simplification:
Narratio's subprocess supervision core is a credible library seed, while all
other reviewed areas should remain internal, use established abstractions, or
be simplified and deleted. No current code must be retained merely because it
might someday be reusable. Version control preserves implementation history,
and a demonstrated second consumer—not speculative generality—will trigger any
future extraction.