narratio/docs/roadmap/configkit-adoption.md

# ConfigKit Adoption and Platform Simplification

**Status:** Accepted; implementation pending

## Purpose

Narratio currently owns a substantial YAML configuration subsystem for strict
decoding, explicit imports, additive composition, profile overlays, presence
tracking, provenance, and effective-value inspection. ConfigKit was extracted
from this implementation and now provides those generic responsibilities as a
maintained library.

Narratio 2.0 will adopt ConfigKit as its configuration foundation, remove the
superseded generic infrastructure, and retain only Narratio-specific configuration
policy and domain resolution. The same major release will narrow the supported
platform set to the systems that Narratio is expected to run on in practice.

This roadmap defines the desired contracts and end state. The
[implementation plan](implementation.md) owns sequencing and completion gates.

## Goals

- Reduce Narratio-owned configuration code and tests by delegating generic YAML
  mechanics to ConfigKit.
- Establish one strict and explainable configuration lifecycle rather than
  retaining parallel or compatibility loaders.
- Preserve Narratio's application-specific defaults, validation, campaign/session
  resolution, party handling, artifact-family expansion, digests, and CLI policy.
- Make individual artifact definitions easy to organize, identify, inspect, and
  reuse as dependencies of later artifacts.
- Keep raw credentials out of YAML and adopt a predictable secrets-directory
  convention that works before configuration loading.
- Remove unsupported platform code and release obligations that provide no known
  user value.
- Deliver the resulting contract changes as Narratio v2.0.0 with a clear migration
  guide.

## Platform Contract

Narratio's v2 platform matrix is:

| Platform | Status | Required evidence and deliverables |
| --- | --- | --- |
| Linux amd64 | Primary, supported | Default tests, race checks, static analysis, build validation, and a release asset |
| Linux arm64 | Primary, supported | Cross-build validation and a release asset; native validation is used when an appropriate runner is available |
| macOS arm64 | Best effort | Cross-build validation and a release asset; native CI and platform-equivalent runtime guarantees are not required release gates |
| macOS amd64 | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance |
| Windows | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance |

Windows- and Intel-Mac-specific code, tests, CI entries, release packaging, and
documentation are removed when they no longer serve a supported build. Portable
code should remain portable where doing so is natural, but accidental compilation
on an unsupported target does not create a support commitment.

Best-effort macOS arm64 support means Narratio intends to keep the target buildable
and will consider reported defects when practical. Lack of a native runner or an
unverified platform-specific behavior does not block a release. A target for which
Narratio publishes an asset must still pass its configured cross-build gate.

## ConfigKit Dependency and Ownership

Narratio will depend on the tagged ConfigKit module at an exact reviewed version,
initially `gitea.maximumdirect.net/eric/ConfigKit@v0.1.1`. ConfigKit is pre-1.0 and
may make compatibility changes in later minor versions, so dependency upgrades are
intentional changes accompanied by contract review and focused Narratio integration
tests; Narratio does not follow an unpinned branch or automatically accept a newer
minor release.

ConfigKit owns:

- bounded file, byte, and reader input;
- the accepted YAML syntax and strict typed decoding;
- import declarations, traversal, containment, and resource limits;
- additive composition and conflict attribution;
- selected-profile overlay semantics;
- explicit input presence and source provenance;
- application-default and recorded-resolution coordination;
- structured configuration diagnostics;
- secret-reference parsing and resolution;
- secrets-directory loading and its safe report; and
- detached effective configuration inspection supplied by its public result.

Narratio owns:

- pipeline, campaign, session, party, and runtime domain types;
- configuration-file discovery and CLI selection;
- concrete defaults, normalization, and semantic validation;
- campaign/session identity and stable-input resolution;
- canonical party parsing requirements and authored-byte retention;
- character artifact-family expansion and generated ownership attribution;
- root-relative interpretation of Narratio-owned runtime paths;
- effective pipeline digests and semantic profile diffs;
- command output and exit behavior; and
- adapter and runtime initialization after configuration succeeds.

Narratio must use ConfigKit through its public package. It must not import
ConfigKit's private YAML implementation or reproduce ConfigKit's parser,
composition, profile, provenance, or secret-bootstrap mechanics locally.

## Configuration Format

Narratio adopts ConfigKit's configuration conventions directly. The old Narratio
`composition:` envelope, declared profile catalog, arbitrary overlay paths, and
`composition.default_profile` are removed without a compatibility parser.

The root pipeline uses `configkit.imports` declarations. Import paths, optional
imports, globs, recursion, ordering, composition conflicts, containment, and limits
have the meanings defined by the pinned ConfigKit contract. Narratio does not add a
second import syntax or narrower application-specific traversal implementation.

The unoverlaid root and its imports are the production configuration. A named
profile is a ConfigKit overlay at `profiles/<name>.yml`; in particular,
`profiles/testing.yml` contains inexpensive-model overrides and testing-only
artifacts. Omitting `--profile`, or selecting ConfigKit's `default` profile, uses the
production base. `--profile testing` applies the testing overlay. Narratio does not
maintain a separate configured default-profile mechanism or validate unselected
profile files.

ConfigKit's rules for explicit false, zero, empty, and null values replace
Narratio's custom presence tracking. Application defaults and normalization use a
ConfigKit definition and recorded resolution where the final value depends on
whether input was present. `NormalizeConfig.UnmarshalYAML` and equivalent
parser-coupled application hooks are removed.

Pipeline, campaign, session, and party inputs use the common ConfigKit loading
boundary where its schema model can represent their types. Narratio may perform
narrow pre- or post-load domain work, such as retaining authored party bytes,
rejecting unresolved session-template placeholders, resolving stable inputs, and
expanding artifact families. Such work must not recreate a second generic YAML
loader.

## Artifact Definition Ergonomics

The canonical v2 configuration layout defines each ordinary Scriptorium artifact
in its own file under `artifacts/`. The root pipeline imports the directory through
ConfigKit, for example:

```yaml
configkit:
  imports:
    - glob: artifacts/*.yml
```

Each file contributes one ordinary entry at
`scriptorium.artifacts.<artifact_key>`. The map key is the authoritative Narratio
artifact key and therefore makes the artifact's internal identity explicit in the
definition itself:

```yaml
# artifacts/session_recap.yml
scriptorium:
  artifacts:
    session_recap:
      enabled: true
      prompt_id: dnd.session_recap
      output_path: artifacts/session_recap.md
```

The canonical reusable source ID is derived without another configured identifier:
`session_recap` becomes `narratio.artifact.session_recap`. The recommended filename
is `<artifact_key>.yml`, but the filename is organizational rather than an identity
source. Renaming or moving a file without changing its map key does not change the
artifact contract. ConfigKit additive composition rejects duplicate artifact keys.

The one-artifact-per-file layout is the documented happy path and is used by
maintained examples. Narratio does not prohibit an inline artifact or another
ConfigKit-compatible source organization merely because it differs from that
layout. Artifact identity always comes from the effective map key, never implicitly
from a filename or import position.

Artifact families use the corresponding `artifact-families/*.yml` convention, with
one `scriptorium.artifact_families.<family_key>` entry per file. The family key is
explicit. Concrete member keys and source IDs continue to use Narratio's existing
deterministic family/member naming rule after canonical party resolution.

Narratio adds a read-only `narratio config artifacts` command using the ordinary
pipeline, profile, and campaign selectors. It reports every effective ordinary
artifact's key, canonical `narratio.artifact.<key>` source ID, defining source,
enabled state, output path, and effective dependencies. Generated family members
also report their family origin. Output ordering is deterministic by artifact key,
and the command does not require a session or initialize runtime adapters.

A configured-artifact input creates an implicit dependency on the referenced
artifact. For example:

```yaml
inputs:
  recap:
    source: narratio.artifact.session_recap
```

The effective dependency graph is the union of explicit `depends_on` entries and
dependencies inferred from `narratio.artifact.<key>` inputs. Inferred and explicit
duplicates collapse to one dependency. `depends_on` remains available for an
ordering requirement when the upstream output is not supplied as an input.
Validation applies the same existence, self-reference, and cycle rules to both
kinds of dependency, and diagnostics identify the declaration or input that
introduced an invalid edge. Operators no longer need to repeat an artifact input in
`depends_on` merely to make that input valid.

## Effective Inspection, Digests, and Profile Diff

ConfigKit results are the authority for composed input presence, base/profile
origins, diagnostics, and the initial typed effective configuration. Narratio's
configuration commands adapt those results to existing application-facing command
formats rather than exposing ConfigKit implementation types directly.

Party-dependent artifact families are expanded only after the selected campaign and
canonical party are known. That derived state is intentionally outside ConfigKit's
completed load snapshot. Narratio retains a narrow application-owned layer that:

- records generated family/member and publish-rule attribution;
- projects the final post-expansion pipeline for operator inspection;
- recomputes the effective pipeline digest after expansion; and
- ensures stages and adapters receive only the final concrete configuration.

Narratio continues to own semantic `config diff` behavior because ConfigKit does
not provide configuration diffs or fingerprints. Each side of a profile comparison
may be loaded independently through ConfigKit. The former guarantee that both sides
come from one parsed source snapshot is removed; operators must not rely on a
meaningful comparison while concurrently modifying its source files. Equality is
still based on normalized effective values rather than YAML formatting or source
layout.

## Credential and Secrets Convention

Raw API keys and other credential values remain prohibited in pipeline, campaign,
session, generated configuration, manifests, diagnostics, examples, and logs.
Credential-bearing configuration fields use ConfigKit secret references: YAML
contains an environment-variable name, while the resolved value remains available
only through ConfigKit's trusted typed boundary. Fields should use credential names
rather than legacy names ending in `_env` when the v2 schema is updated.

The `pipeline.secrets.env_dir` stanza is removed. Narratio selects a secrets
directory before parsing the pipeline using this precedence:

1. an explicitly supplied `--secrets-dir PATH`; otherwise
2. the `secrets` subdirectory beside the selected root pipeline file.

The conventional directory is derived from the resolved root pipeline path, never
from the current working directory, an imported file, or a selected profile. A
relative explicit CLI path is resolved from the caller's current working directory,
consistent with an ordinary command-line path argument.

If an explicitly selected directory is missing or invalid, the command fails with a
clear bootstrap diagnostic. If the conventional directory is absent, Narratio
performs no directory bootstrap and continues; pre-existing environment variables
may satisfy all references. If the conventional directory exists but ConfigKit
rejects or cannot read it, the command fails. Already-set environment variables,
including empty values, retain ConfigKit's documented precedence.

Narratio explicitly calls `configkit.LoadSecretsDir` at most once for a command,
before the first ConfigKit load that may resolve secret references and before any
concurrent work or runtime client initialization. This application-level convention
does not make ConfigKit's ordinary `Load` mutate process-global state. The bootstrap
report is rendered or summarized through a safe Narratio command boundary without
secret values.

Narratio adopts ConfigKit's secret-directory rules and limits without restoring its
old permission-mode, selected-directory-symlink, invalid-entry, byte-limit, or
newline-trimming behavior through compatibility wrappers. Configuration inspection
may require the referenced environment variables or the selected secrets directory
because an ordinary successful ConfigKit load resolves effective typed secret
references. Inspection output may show non-secret reference names where appropriate,
but must never expose resolved values.

## Validation and Diagnostics

Narratio's semantic rules remain recognizable application code registered with the
appropriate ConfigKit definition. Validation continues to reject invalid pipeline,
campaign, session, party, adapter, publish, and artifact relationships before
runtime work where practical. Validation order and exact prose are not compatibility
contracts.

ConfigKit diagnostic codes, paths, and source locations should be preserved through
the application error boundary where useful. Narratio adds command context and exit
classification but does not parse ConfigKit error strings or expose raw parser
errors. Secret values must remain absent from success and failure diagnostics.

## Compatibility and Migration

This work intentionally changes supported platforms, pipeline composition syntax,
profile layout and selection, secret configuration, YAML semantics, diagnostics,
and some inspection prerequisites. It is released as Narratio v2.0.0.

There is no runtime compatibility layer for the old `composition` or
`pipeline.secrets` syntax. Maintained examples are converted to the ConfigKit
format, and the v2 release documentation provides a concise migration guide that
covers:

- converting imports to `configkit.imports` declarations;
- folding the production overlay into the base configuration;
- placing testing overrides in `profiles/testing.yml`;
- moving ordinary artifacts and artifact families into the recommended individual
  definition files and using their map keys as canonical identities;
- removing redundant `depends_on` entries when a configured-artifact input now
  supplies the same dependency;
- moving secret files to the conventional sibling `secrets/` directory or using
  `--secrets-dir`;
- renaming credential-reference fields where applicable;
- ConfigKit's stricter YAML and explicit-value semantics;
- the independent-load behavior of `config diff`; and
- the removal of Windows and macOS amd64 artifacts.

Obsolete application implementation, tests, documentation, examples, CI paths, and
release machinery are removed rather than left deprecated inside the v2 codebase.

## Required Changes

- Add and pin the ConfigKit module dependency and establish a small application
  integration boundary for definitions, result adaptation, and diagnostics.
- Replace Narratio's generic YAML parsing, composition, import, profile, presence,
  and provenance implementation with ConfigKit public APIs.
- Express existing defaults, presence-sensitive normalization, and semantic
  validation through ConfigKit definitions without moving Narratio domain policy
  into the library.
- Adapt pipeline/campaign/session/party loading and preserve the existing
  application resolution steps that occur after their independent typed loads.
- Rework effective configuration, sources, and diff commands around ConfigKit
  results plus Narratio's post-party generated-attribution layer.
- Add the maintained per-artifact and per-family file layouts, the read-only
  `config artifacts` catalog, and effective dependency inference from configured
  artifact inputs.
- Convert credential configuration to typed references, add the common
  `--secrets-dir` option, implement conventional-directory selection, and replace
  the custom API-key directory loader with `configkit.LoadSecretsDir`.
- Update adapters to consume resolved credentials or safe environment reference
  names from typed configuration without independently rediscovering secret values.
- Remove Windows and macOS amd64 build targets, release assets, CI entries,
  platform-specific implementation, and obsolete tests and documentation.
- Convert maintained examples and add v2 migration and release documentation.
- Retain focused Narratio tests for domain policy and assembled application
  behavior while deleting parser/composer/bootstrap mechanism tests now owned by
  ConfigKit. Tests remain deterministic, offline, credential-free, and aligned with
  the repository testing policy.
- Update canonical architecture, configuration, CLI, operations, development,
  internal, troubleshooting, and release documentation only as the corresponding
  behavior is implemented.

## Target End State

Narratio v2.0.0 has one configuration infrastructure dependency and no duplicate
YAML composition subsystem. Operators maintain a production base configuration,
optional explicit imports, and fixed named overlays such as `testing`. ConfigKit
provides strict parsing, composition, typed loading, provenance, diagnostics, and
credential resolution; Narratio supplies domain meaning and its post-campaign
artifact expansion.

Operators can add an artifact by creating one discoverable YAML file whose map key
is its stable Narratio identity. `config artifacts` exposes that key, its canonical
source ID, origin, output, and effective dependency set. A later artifact can reuse
the output through `narratio.artifact.<key>` without repeating the same edge in
`depends_on`, while explicit ordering-only dependencies remain supported.

Secrets normally live in a predictable `secrets/` directory beside the selected
pipeline, with `--secrets-dir` available for deployment-specific placement. Raw
credential values never enter YAML or inspection output. The bootstrap is explicit
in application startup and ordinary ConfigKit loading remains free of process-global
mutation.

Release and CI maintenance covers Linux amd64 and arm64 as the primary platforms
and macOS arm64 on a best-effort basis. Windows and macOS amd64 carry no remaining
support or release burden. The repository contains no legacy configuration parser,
profile compatibility layer, obsolete platform implementation, or redundant tests
for behavior owned by ConfigKit.

## Deferred Work

- A ConfigKit reference-only inspection or typed prepare/finalize lifecycle may be
  considered upstream if credential-free inspection becomes a demonstrated shared
  requirement. Narratio does not implement a private substitute in this migration.
- ConfigKit hot reload, remote providers, automatic configuration migration, and
  application command frameworks remain outside Narratio's scope.
- Native macOS arm64 CI may be added later when a reliable runner is available; it
  is not required for v2.0.0.