narratio/docs/roadmap/configkit-adoption.md

ConfigKit Adoption and Platform Simplification

Status: Accepted; implementation pending

Purpose

Narratio currently owns a substantial YAML configuration subsystem for strict decoding, explicit imports, additive composition, profile overlays, presence tracking, provenance, and effective-value inspection. ConfigKit was extracted from this implementation and now provides those generic responsibilities as a maintained library.

Narratio 2.0 will adopt ConfigKit as its configuration foundation, remove the superseded generic infrastructure, and retain only Narratio-specific configuration policy and domain resolution. The same major release will narrow the supported platform set to the systems that Narratio is expected to run on in practice.

This roadmap defines the desired contracts and end state. The implementation plan owns sequencing and completion gates.

Goals

• Reduce Narratio-owned configuration code and tests by delegating generic YAML mechanics to ConfigKit.
• Establish one strict and explainable configuration lifecycle rather than retaining parallel or compatibility loaders.
• Preserve Narratio's application-specific defaults, validation, campaign/session resolution, party handling, artifact-family expansion, digests, and CLI policy.
• Make individual artifact definitions easy to organize, identify, inspect, and reuse as dependencies of later artifacts.
• Keep raw credentials out of YAML and adopt a predictable secrets-directory convention that works before configuration loading.
• Remove unsupported platform code and release obligations that provide no known user value.
• Deliver the resulting contract changes as Narratio v2.0.0 with a clear migration guide.

Platform Contract

Narratio's v2 platform matrix is:

Platform	Status	Required evidence and deliverables
Linux amd64	Primary, supported	Default tests, race checks, static analysis, build validation, and a release asset
Linux arm64	Primary, supported	Cross-build validation and a release asset; native validation is used when an appropriate runner is available
macOS arm64	Best effort	Cross-build validation and a release asset; native CI and platform-equivalent runtime guarantees are not required release gates
macOS amd64	Unsupported	No build gate, release asset, compatibility promise, or platform-specific maintenance
Windows	Unsupported	No build gate, release asset, compatibility promise, or platform-specific maintenance

Windows- and Intel-Mac-specific code, tests, CI entries, release packaging, and documentation are removed when they no longer serve a supported build. Portable code should remain portable where doing so is natural, but accidental compilation on an unsupported target does not create a support commitment.

Best-effort macOS arm64 support means Narratio intends to keep the target buildable and will consider reported defects when practical. Lack of a native runner or an unverified platform-specific behavior does not block a release. A target for which Narratio publishes an asset must still pass its configured cross-build gate.

ConfigKit Dependency and Ownership

Narratio will depend on the tagged ConfigKit module at an exact reviewed version, initially gitea.maximumdirect.net/eric/ConfigKit@v0.1.1. ConfigKit is pre-1.0 and may make compatibility changes in later minor versions, so dependency upgrades are intentional changes accompanied by contract review and focused Narratio integration tests; Narratio does not follow an unpinned branch or automatically accept a newer minor release.

ConfigKit owns:

• bounded file, byte, and reader input;
• the accepted YAML syntax and strict typed decoding;
• import declarations, traversal, containment, and resource limits;
• additive composition and conflict attribution;
• selected-profile overlay semantics;
• explicit input presence and source provenance;
• application-default and recorded-resolution coordination;
• structured configuration diagnostics;
• secret-reference parsing and resolution;
• secrets-directory loading and its safe report; and
• detached effective configuration inspection supplied by its public result.

Narratio owns:

• pipeline, campaign, session, party, and runtime domain types;
• configuration-file discovery and CLI selection;
• concrete defaults, normalization, and semantic validation;
• campaign/session identity and stable-input resolution;
• canonical party parsing requirements and authored-byte retention;
• character artifact-family expansion and generated ownership attribution;
• root-relative interpretation of Narratio-owned runtime paths;
• effective pipeline digests and semantic profile diffs;
• command output and exit behavior; and
• adapter and runtime initialization after configuration succeeds.

Narratio must use ConfigKit through its public package. It must not import ConfigKit's private YAML implementation or reproduce ConfigKit's parser, composition, profile, provenance, or secret-bootstrap mechanics locally.

Configuration Format

Narratio adopts ConfigKit's configuration conventions directly. The old Narratio composition: envelope, declared profile catalog, arbitrary overlay paths, and composition.default_profile are removed without a compatibility parser.

The root pipeline uses configkit.imports declarations. Import paths, optional imports, globs, recursion, ordering, composition conflicts, containment, and limits have the meanings defined by the pinned ConfigKit contract. Narratio does not add a second import syntax or narrower application-specific traversal implementation.

The unoverlaid root and its imports are the production configuration. A named profile is a ConfigKit overlay at profiles/<name>.yml; in particular, profiles/testing.yml contains inexpensive-model overrides and testing-only artifacts. Omitting --profile, or selecting ConfigKit's default profile, uses the production base. --profile testing applies the testing overlay. Narratio does not maintain a separate configured default-profile mechanism or validate unselected profile files.

ConfigKit's rules for explicit false, zero, empty, and null values replace Narratio's custom presence tracking. Application defaults and normalization use a ConfigKit definition and recorded resolution where the final value depends on whether input was present. NormalizeConfig.UnmarshalYAML and equivalent parser-coupled application hooks are removed.

Pipeline, campaign, session, and party inputs use the common ConfigKit loading boundary where its schema model can represent their types. Narratio may perform narrow pre- or post-load domain work, such as retaining authored party bytes, rejecting unresolved session-template placeholders, resolving stable inputs, and expanding artifact families. Such work must not recreate a second generic YAML loader.

Artifact Definition Ergonomics

The canonical v2 configuration layout defines each ordinary Scriptorium artifact in its own file under artifacts/. The root pipeline imports the directory through ConfigKit, for example:

configkit:
  imports:
    - glob: artifacts/*.yml

Each file contributes one ordinary entry at scriptorium.artifacts.<artifact_key>. The map key is the authoritative Narratio artifact key and therefore makes the artifact's internal identity explicit in the definition itself:

# artifacts/session_recap.yml
scriptorium:
  artifacts:
    session_recap:
      enabled: true
      prompt_id: dnd.session_recap
      output_path: artifacts/session_recap.md

The canonical reusable source ID is derived without another configured identifier: session_recap becomes narratio.artifact.session_recap. The recommended filename is <artifact_key>.yml, but the filename is organizational rather than an identity source. Renaming or moving a file without changing its map key does not change the artifact contract. ConfigKit additive composition rejects duplicate artifact keys.

The one-artifact-per-file layout is the documented happy path and is used by maintained examples. Narratio does not prohibit an inline artifact or another ConfigKit-compatible source organization merely because it differs from that layout. Artifact identity always comes from the effective map key, never implicitly from a filename or import position.

Artifact families use the corresponding artifact-families/*.yml convention, with one scriptorium.artifact_families.<family_key> entry per file. The family key is explicit. Concrete member keys and source IDs continue to use Narratio's existing deterministic family/member naming rule after canonical party resolution.

Narratio adds a read-only narratio config artifacts command using the ordinary pipeline, profile, and campaign selectors. It reports every effective ordinary artifact's key, canonical narratio.artifact.<key> source ID, defining source, enabled state, output path, and effective dependencies. Generated family members also report their family origin. Output ordering is deterministic by artifact key, and the command does not require a session or initialize runtime adapters.

A configured-artifact input creates an implicit dependency on the referenced artifact. For example:

inputs:
  recap:
    source: narratio.artifact.session_recap

The effective dependency graph is the union of explicit depends_on entries and dependencies inferred from narratio.artifact.<key> inputs. Inferred and explicit duplicates collapse to one dependency. depends_on remains available for an ordering requirement when the upstream output is not supplied as an input. Validation applies the same existence, self-reference, and cycle rules to both kinds of dependency, and diagnostics identify the declaration or input that introduced an invalid edge. Operators no longer need to repeat an artifact input in depends_on merely to make that input valid.

Effective Inspection, Digests, and Profile Diff

ConfigKit results are the authority for composed input presence, base/profile origins, diagnostics, and the initial typed effective configuration. Narratio's configuration commands adapt those results to existing application-facing command formats rather than exposing ConfigKit implementation types directly.

Party-dependent artifact families are expanded only after the selected campaign and canonical party are known. That derived state is intentionally outside ConfigKit's completed load snapshot. Narratio retains a narrow application-owned layer that:

• records generated family/member and publish-rule attribution;
• projects the final post-expansion pipeline for operator inspection;
• recomputes the effective pipeline digest after expansion; and
• ensures stages and adapters receive only the final concrete configuration.

Narratio continues to own semantic config diff behavior because ConfigKit does not provide configuration diffs or fingerprints. Each side of a profile comparison may be loaded independently through ConfigKit. The former guarantee that both sides come from one parsed source snapshot is removed; operators must not rely on a meaningful comparison while concurrently modifying its source files. Equality is still based on normalized effective values rather than YAML formatting or source layout.

Credential and Secrets Convention

Raw API keys and other credential values remain prohibited in pipeline, campaign, session, generated configuration, manifests, diagnostics, examples, and logs. Credential-bearing configuration fields use ConfigKit secret references: YAML contains an environment-variable name, while the resolved value remains available only through ConfigKit's trusted typed boundary. Fields should use credential names rather than legacy names ending in _env when the v2 schema is updated.

The pipeline.secrets.env_dir stanza is removed. Narratio selects a secrets directory before parsing the pipeline using this precedence:

1. an explicitly supplied --secrets-dir PATH; otherwise
2. the secrets subdirectory beside the selected root pipeline file.

The conventional directory is derived from the resolved root pipeline path, never from the current working directory, an imported file, or a selected profile. A relative explicit CLI path is resolved from the caller's current working directory, consistent with an ordinary command-line path argument.

If an explicitly selected directory is missing or invalid, the command fails with a clear bootstrap diagnostic. If the conventional directory is absent, Narratio performs no directory bootstrap and continues; pre-existing environment variables may satisfy all references. If the conventional directory exists but ConfigKit rejects or cannot read it, the command fails. Already-set environment variables, including empty values, retain ConfigKit's documented precedence.

Narratio explicitly calls configkit.LoadSecretsDir at most once for a command, before the first ConfigKit load that may resolve secret references and before any concurrent work or runtime client initialization. This application-level convention does not make ConfigKit's ordinary Load mutate process-global state. The bootstrap report is rendered or summarized through a safe Narratio command boundary without secret values.

Narratio adopts ConfigKit's secret-directory rules and limits without restoring its old permission-mode, selected-directory-symlink, invalid-entry, byte-limit, or newline-trimming behavior through compatibility wrappers. Configuration inspection may require the referenced environment variables or the selected secrets directory because an ordinary successful ConfigKit load resolves effective typed secret references. Inspection output may show non-secret reference names where appropriate, but must never expose resolved values.

Validation and Diagnostics

Narratio's semantic rules remain recognizable application code registered with the appropriate ConfigKit definition. Validation continues to reject invalid pipeline, campaign, session, party, adapter, publish, and artifact relationships before runtime work where practical. Validation order and exact prose are not compatibility contracts.

ConfigKit diagnostic codes, paths, and source locations should be preserved through the application error boundary where useful. Narratio adds command context and exit classification but does not parse ConfigKit error strings or expose raw parser errors. Secret values must remain absent from success and failure diagnostics.

Compatibility and Migration

This work intentionally changes supported platforms, pipeline composition syntax, profile layout and selection, secret configuration, YAML semantics, diagnostics, and some inspection prerequisites. It is released as Narratio v2.0.0.

There is no runtime compatibility layer for the old composition or pipeline.secrets syntax. Maintained examples are converted to the ConfigKit format, and the v2 release documentation provides a concise migration guide that covers:

• converting imports to configkit.imports declarations;
• folding the production overlay into the base configuration;
• placing testing overrides in profiles/testing.yml;
• moving ordinary artifacts and artifact families into the recommended individual definition files and using their map keys as canonical identities;
• removing redundant depends_on entries when a configured-artifact input now supplies the same dependency;
• moving secret files to the conventional sibling secrets/ directory or using --secrets-dir;
• renaming credential-reference fields where applicable;
• ConfigKit's stricter YAML and explicit-value semantics;
• the independent-load behavior of config diff; and
• the removal of Windows and macOS amd64 artifacts.

Obsolete application implementation, tests, documentation, examples, CI paths, and release machinery are removed rather than left deprecated inside the v2 codebase.

Required Changes

• Add and pin the ConfigKit module dependency and establish a small application integration boundary for definitions, result adaptation, and diagnostics.
• Replace Narratio's generic YAML parsing, composition, import, profile, presence, and provenance implementation with ConfigKit public APIs.
• Express existing defaults, presence-sensitive normalization, and semantic validation through ConfigKit definitions without moving Narratio domain policy into the library.
• Adapt pipeline/campaign/session/party loading and preserve the existing application resolution steps that occur after their independent typed loads.
• Rework effective configuration, sources, and diff commands around ConfigKit results plus Narratio's post-party generated-attribution layer.
• Add the maintained per-artifact and per-family file layouts, the read-only config artifacts catalog, and effective dependency inference from configured artifact inputs.
• Convert credential configuration to typed references, add the common --secrets-dir option, implement conventional-directory selection, and replace the custom API-key directory loader with configkit.LoadSecretsDir.
• Update adapters to consume resolved credentials or safe environment reference names from typed configuration without independently rediscovering secret values.
• Remove Windows and macOS amd64 build targets, release assets, CI entries, platform-specific implementation, and obsolete tests and documentation.
• Convert maintained examples and add v2 migration and release documentation.
• Retain focused Narratio tests for domain policy and assembled application behavior while deleting parser/composer/bootstrap mechanism tests now owned by ConfigKit. Tests remain deterministic, offline, credential-free, and aligned with the repository testing policy.
• Update canonical architecture, configuration, CLI, operations, development, internal, troubleshooting, and release documentation only as the corresponding behavior is implemented.

Target End State

Narratio v2.0.0 has one configuration infrastructure dependency and no duplicate YAML composition subsystem. Operators maintain a production base configuration, optional explicit imports, and fixed named overlays such as testing. ConfigKit provides strict parsing, composition, typed loading, provenance, diagnostics, and credential resolution; Narratio supplies domain meaning and its post-campaign artifact expansion.

Operators can add an artifact by creating one discoverable YAML file whose map key is its stable Narratio identity. config artifacts exposes that key, its canonical source ID, origin, output, and effective dependency set. A later artifact can reuse the output through narratio.artifact.<key> without repeating the same edge in depends_on, while explicit ordering-only dependencies remain supported.

Secrets normally live in a predictable secrets/ directory beside the selected pipeline, with --secrets-dir available for deployment-specific placement. Raw credential values never enter YAML or inspection output. The bootstrap is explicit in application startup and ordinary ConfigKit loading remains free of process-global mutation.

Release and CI maintenance covers Linux amd64 and arm64 as the primary platforms and macOS arm64 on a best-effort basis. Windows and macOS amd64 carry no remaining support or release burden. The repository contains no legacy configuration parser, profile compatibility layer, obsolete platform implementation, or redundant tests for behavior owned by ConfigKit.

Deferred Work

• A ConfigKit reference-only inspection or typed prepare/finalize lifecycle may be considered upstream if credential-free inspection becomes a demonstrated shared requirement. Narratio does not implement a private substitute in this migration.
• ConfigKit hot reload, remote providers, automatic configuration migration, and application command frameworks remain outside Narratio's scope.
• Native macOS arm64 CI may be added later when a reliable runner is available; it is not required for v2.0.0.