narratio/docs/roadmap/library-audit.md

Reusable Library Extraction Audit

Status

Complete. This audit reviews the Narratio code at af5fbc1c4f8d and records candidate-extraction decisions for future refactoring. It does not authorize an extraction, create a new module, or change application behavior.

This is the third maintainability pass. It complements the two earlier passes:

1. audit.md asks whether implementation can be simplified while preserving current behavior and contracts.
2. contract-audit.md asks whether apparent contracts can reasonably be changed to permit substantially simpler behavior.
3. This document asks whether code that Narratio no longer needs should instead be preserved as a generic, independently reusable library.

The question in this pass is deliberately separate from whether Narratio should continue using an extracted library. Code may be valuable in a more demanding application while remaining the wrong dependency for Narratio.

Method and Decision Standard

The review inventoried Narratio's internal packages, concentrated on mechanisms whose contracts can be stated without D&D or pipeline terminology, inspected their call sites and tests, and compared the most plausible candidates with the actual implementations in the neighboring Notarius and WeatherReporter repositories. It also checked the Go standard library and representative maintained or emerging third-party implementations before treating custom code as a library opportunity.

A candidate is suitable for extraction only when all of the following are true:

• its contract can be explained without Narratio types or product policy;
• Narratio-specific configuration, paths, modes, diagnostics, and error prose can remain in Narratio rather than leak into the generic API;
• at least one credible independent consumer needs materially the same guarantees;
• extraction removes meaningful implementation and test ownership from consumers instead of adding adapters around a small helper;
• the generic API is narrow, stable, and independently testable on every promised platform;
• an existing maintained library does not already satisfy the need; and
• the extracted module has an identified maintenance, compatibility, release, and vulnerability-response owner.

The verdicts used below are:

• Extract now: a proven shared contract and consumer justify a separate maintained module.
• Incubate: the code contains a credible generic core, but extraction should wait for a concrete trigger.
• Keep internal: the implementation is coherent but belongs to Narratio's domain or is too small to justify an external lifecycle.
• Simplify or delete: independent reuse does not justify retaining the code in Narratio; source history is enough to preserve ideas for later recovery.
• Prefer an established abstraction: a future generic consumer should start from an existing standard or library rather than Narratio's implementation.

Executive Summary

No package should be extracted before the approved Narratio refactoring begins. The audit found one serious incubation candidate: the cross-platform subprocess supervision core. It implements a substantial and reasonably generic process- tree lifecycle that the Go standard library does not provide as one operation. It is nevertheless mixed with Narratio-owned environment, logging, redaction, filesystem, and diagnostic policy, and neither Notarius nor WeatherReporter is currently a matching second consumer. Extraction now would therefore invent a public API before its reuse requirements are known.

The current FileOps subsystem may be useful source material for an application that genuinely needs hostile-race resistance, confined traversal, no-replace directory promotion, or power-loss durability. It is not a cohesive generic library as written, and its strongest guarantees are specifically the ones Narratio has decided to retire. The approved FileOps simplification should proceed. Git history is a sufficient preservation mechanism unless a real external consumer appears before deletion.

Configuration composition, remote commit snapshots, artifact policy, session state, and pipeline recovery contain useful patterns, but their contracts are Narratio product contracts rather than generic library seams. Small path, bounded-I/O, locking, identifier, and atomic-write helpers do not carry enough independent value to justify separate versioning. Release automation is a possible shared tool or template, not a Go library, and current sibling repositories do not yet share one implementation that needs central ownership.

Candidate	Verdict	Refactoring consequence
Cross-platform subprocess supervision	Incubate	Separate generic mechanism from Narratio policy locally; extract only after a second consumer and owner exist.
Hardened FileOps subsystem	Simplify or delete	Follow fileops.md; preserve history, not a Narratio dependency on an overbuilt library.
YAML composition and provenance	Keep internal	Retain as Narratio's configuration DSL; reconsider structural merge dependencies separately if they reduce code.
Immutable remote commit and pointer protocol	Keep internal	Preserve as Narratio's publication contract; do not market it as a generic object-store transaction library.
Path-safety and bounded-I/O helpers	Keep internal	Prefer small local helpers and newer standard-library primitives where semantics match.
Atomic file replacement	Keep internal for Narratio	Use a small cross-platform helper or an established dependency after focused implementation research; do not extract a new module.
Object-store interface and session locks	Keep internal	These are narrow ports and product coordination rules, not reusable implementations.
Release scripts and checks	Keep repository-local	Share a template or standalone tool only if multiple repositories develop materially identical behavior and drift.
Artifact, manifest, analyze, and pipeline policy	Keep internal	These are Narratio domain code even where individual algorithms are conventional.

Candidate 1 — Cross-Platform Subprocess Supervision

Why it is a credible library seed

internal/adapters/subprocess contains substantially more than a wrapper around os/exec. Its process owner:

• starts each command in an owned Unix process group or Windows Job Object;
• prevents the Windows child from running before job assignment;
• terminates the owned tree on cancellation or timeout;
• attempts graceful then forceful shutdown where the platform supports it;
• reaps the leader and bounds waits when descendants retain stream handles;
• distinguishes exit, timeout, cancellation, capture-limit, and cleanup outcomes; and
• has focused Unix and Windows tests for the lifecycle.

That core contract is understandable without Narratio terminology and is large enough that a proven consumer could benefit from sharing it. The Go os/exec package supplies command execution, custom cancellation, and WaitDelay, but its default cancellation targets the leader process rather than establishing cross-platform process-tree ownership.

What is not generic

The package currently also owns policy that must not cross an extraction boundary:

• Narratio's inherited-environment allowlist and test-helper variables;
• API-key environment names and extraction of their secret values;
• fixed diagnostic size limits and application-specific owner wording;
• log paths, group-owned file modes, and FileOps calls;
• stderr-tail formatting and Python/file-descriptor diagnostic hints; and
• atomic YAML and byte-writing helpers unrelated to process ownership.

A reusable core should accept an explicit environment and ordinary io.Writer or capture interfaces. Narratio should remain responsible for choosing and redacting the environment, opening bounded diagnostic sinks, selecting paths and modes, and translating typed results into operator-facing errors.

Existing alternatives

The standard library remains the foundation but does not provide the whole-tree contract. Third-party packages now exist in this space, including processkit-go, but the candidates found during this audit are comparatively new or offer a materially broader contract than Narratio needs. They should be reevaluated at extraction time; the existence and maturity of this category are changing quickly, and copying Narratio code into a new module is not automatically preferable.

Decision and extraction trigger

Verdict: Incubate; do not extract now.

During Narratio refactoring, make the internal mechanism/policy boundary clear and remove unrelated file-writing helpers from the package. Do not contort the internal API solely to resemble a hypothetical public module.

Reopen extraction only when all of these triggers are present:

1. a second production application needs whole-process-tree ownership on at least two of Narratio's supported operating systems;
2. its timeout, graceful-shutdown, cleanup, and result semantics substantially overlap Narratio's rather than merely needing exec.CommandContext;
3. maintained alternatives have been reevaluated and rejected for documented technical or maintenance reasons;
4. an independent module API can exclude all Narratio filesystem, environment, redaction, and diagnostic policy; and
5. a maintainer accepts cross-platform CI, semantic versioning, releases, and security ownership for the module.

If those conditions are met, extract the generic core from the simplified internal boundary and let Narratio consume it through a thin policy adapter.

Candidate 2 — Hardened FileOps

Why the code may be useful elsewhere

The present internal/fileops implementation includes confined root operations, symlink-resistant traversal, identity rechecks, same-directory replacement, checksummed copy and download installation, directory tree validation, atomic no-replace promotion, mode enforcement, file and directory syncing, and platform-specific implementations. An application operating on an untrusted, concurrently modified tree could reasonably need some of those mechanisms.

Why it is not an extraction candidate as written

The package is a bundle of distinct concerns rather than one small generic contract. It combines:

• atomic single-file visibility and durability;
• hostile-race-resistant traversal and copying;
• directory transaction and no-replace semantics;
• Narratio workspace ownership modes;
• bounded application I/O; and
• application-specific unsupported-platform and contextual errors.

A credible external library would first need to choose which threat model and durability level it promises, define a much smaller API, specify filesystem and platform assumptions, and split policy such as modes and limits from mechanism. That is a new library design project, not a package move.

There is also no matching current consumer nearby. Notarius implements its ordinary atomic byte write in a small local helper, and WeatherReporter likewise uses a focused same-directory temporary-file replacement helper. Neither needs Narratio's complete confined traversal and directory-promotion contract.

Existing components cover portions of the space. Go provides os.Root for root-scoped operations; libraries such as renameio and moby/sys/atomicwriter address atomic file replacement with different platform, durability, and mode tradeoffs. No single dependency should be selected merely to preserve the current aggregate abstraction.

Decision

Verdict: Simplify or delete; do not extract before refactoring.

Implement the reduced contract in fileops.md. Preserve the current implementation through version control and, if useful, a short design note identifying the historical commit. Do not make Narratio depend on an externalized version of guarantees that Narratio has explicitly rejected.

If another application later demonstrates a need, design a new module from its requirements and selectively recover proven algorithms and tests from history. That work should not block or enlarge the Narratio refactor.

Candidate 3 — YAML Composition and Provenance

internal/config/composition.go implements a presence-aware YAML tree with source identity, duplicate-key detection, additive disjoint-source composition, profile overlays, deterministic canonical rendering, semantic leaf projection, and source attribution. These mechanics look generic at first glance.

The actual merge algebra is nevertheless Narratio's configuration language: root-only composition metadata, disjoint imports, named pipeline profiles, overlay ownership, atomic list treatment, null and kind-change policy, pipeline field provenance, effective digests, and strict decoding into PipelineConfig. The useful diagnostics also depend on those policy choices. Extracting only the tree mechanics would leave much of the code and all product decisions in Narratio, while extracting the whole package would publish Narratio's DSL as a supposedly generic API.

Notarius does not implement the same import/profile language, and WeatherReporter's configuration merge is a short typed decode into its own configuration. They are not current consumers. Structural YAML libraries such as yaml/go-yaml-dom now provide clone and configurable merge operations, but they do not by themselves establish Narratio's provenance and additive-ownership rules.

Verdict: Keep internal. Keep product semantics explicit in Narratio. A future implementation pass may evaluate a maintained YAML DOM/merge dependency as a way to delete low-level node mechanics, but dependency adoption and public library extraction are different decisions. Do not create a generic module without a second consumer that independently chooses the same merge algebra.

Candidate 4 — Immutable Remote Commit Protocol

The remote publication design uses immutable run-scoped objects, checksummed and generation-bound artifact descriptors, an immutable commit manifest, and a single mutable current pointer. The pattern is reusable in the abstract and provides coherent publication on an object store without pretending the store offers a multi-object transaction.

Its concrete schema and validation are Narratio-specific: campaign and session identity, run IDs, session and run manifests, configured artifact sources, canonical S3 key layout, publication categories, previous-session resolution, and current-state reconstruction. The storage interface alone is not the protocol; extracting it would either export Narratio's schema or require a new generic content model and transaction API.

Established content-addressed specifications already model similar generic ideas. The OCI image specification defines descriptors, digests, manifests, indexes, and mutable names over content-addressed objects. Narratio should not migrate to OCI merely for reuse, but any future cross-application protocol should evaluate established formats before turning Narratio's private contract into a library.

Verdict: Keep internal. Treat the implementation as a good domain protocol, not a generic object-store transaction package. Reconsider only if multiple applications need interoperable snapshots; at that point start with a shared protocol specification and compatibility requirements, not a code extraction.

Candidate 5 — Small Infrastructure Helpers

Path safety

internal/pathsafe is small, pure, and widely used, but it encodes Narratio's slash-style relative destination rules and cross-platform object-key behavior. It is below the threshold where a separate module's dependency and release lifecycle would pay for itself. Newer Go path APIs such as filepath.IsLocal should be used where their host-path semantics exactly match; object keys and canonical slash paths may still warrant a few local functions.

Verdict: Keep internal and small.

Bounded I/O and streaming redaction

Bounded reads, tail capture, and streaming secret redaction are useful generic techniques. Narratio's instances are tied to adapter-specific limits, owner names, failure wording, log files, and its API-key policy. Separating them into external modules would trade a few local helpers for additional APIs without a known consumer.

Verdict: Keep internal. If subprocess incubation later proceeds, bounded capture may become part of that module only if consumers agree on typed limit semantics; secret selection and error prose remain application policy.

Atomic file replacement

Same-directory temporary-file replacement is generic, but Narratio's reduced need is a small operation with explicit final mode and supported-platform behavior. Notarius and WeatherReporter each maintain a short local variant, but their differing modes, cancellation points, and parent-directory behavior show that superficial duplication is not yet one shared contract.

Verdict: Do not create a new library. During FileOps implementation, reevaluate established atomic-writer packages against Narratio's exact Windows, mode, and atomic-visibility requirements. Otherwise retain a small local helper.

Object-store ports, file locks, IDs, and DAG helpers

The object-store interfaces are deliberately narrow ports over provider SDKs; the implementations and error semantics remain application adapters. Session locks enforce Narratio's one-session execution rule. Run/session identifiers and artifact dependency traversal are bound to Narratio schemas and lifecycle. Extracting any of these would either expose domain types or create generic abstractions with less clarity than the current code.

Verdict: Keep internal. Use established provider SDKs and standard-library algorithms directly rather than creating thin generic wrapper modules.

Candidate 6 — Release Automation

Narratio's release scripts combine source-tree validation, asset-matrix builds, archive naming, checksum generation, Git state checks, tag creation, and push behavior. Those ideas are reusable, but the concrete release contract includes Narratio's binary name, supported target matrix, documentation, tag policy, asset layout, and responsibility boundary. Notarius currently has a narrower source-release check, while WeatherReporter documents its procedure without a matching shared script suite.

Verdict: Keep repository-local. This is not a runtime Go library candidate. If two or more repositories later carry materially identical scripts and fixes begin to drift, extract a versioned standalone release tool or a maintained template with per-project declarative metadata. Until then, local scripts keep release changes atomic with each repository and avoid a bootstrap dependency in the release path.

Explicit Non-Candidates

The following code is intentionally excluded from generic extraction despite containing reusable programming techniques:

• artifact source classification and published-destination policy;
• session and remote manifest schemas and validation;
• stage ordering, selection, invalidation, resume, and force semantics;
• analyze artifact checkpointing and dependency-aware recovery;
• cleanup and remote lock policy;
• Notarius, Seriatim, Scriptorium, Audita, WhisperX, and S3 adapters; and
• configuration validation domains and party/artifact-family expansion.

Their value comes from making Narratio's product rules explicit. Moving them to a generic module would hide ownership rather than create reuse. Conventional algorithms inside domain code are not by themselves library candidates.

Refactoring Guidance

This audit does not add an extraction prerequisite to the approved refactoring. Use the following sequence:

1. implement the FileOps and contract simplifications already recorded in the roadmap documents;
2. keep refactored domain ownership within Narratio unless a stage explicitly calls for a package seam;
3. while touching subprocess code, separate process ownership from Narratio's diagnostics and filesystem policy, but retain an internal API;
4. record deleted FileOps history rather than preserving dead compatibility or an external dependency for hypothetical users; and
5. revisit subprocess extraction only when the trigger conditions in this audit are satisfied.

An extraction, if later approved, should be planned and released as its own project. It should not be smuggled into a Narratio refactoring stage, because public API design, multi-platform CI, module ownership, migration, and release policy are independent deliverables.

Final State of This Audit

The audit preserves one future option without delaying simplification: Narratio's subprocess supervision core is a credible library seed, while all other reviewed areas should remain internal, use established abstractions, or be simplified and deleted. No current code must be retained merely because it might someday be reusable. Version control preserves implementation history, and a demonstrated second consumer—not speculative generality—will trigger any future extraction.