From 1558c9ccaeb30aeaf17d71fc4b669a1341d5e018 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 7 Sep 2026 03:21:06 +0000 Subject: [PATCH] Plan ConfigKit adoption and platform simplification --- docs/roadmap/configkit-adoption.md | 375 ++++++++++++++++++++++ docs/roadmap/implementation.md | 480 +++++++++++++++++++++++++++++ 2 files changed, 855 insertions(+) create mode 100644 docs/roadmap/configkit-adoption.md create mode 100644 docs/roadmap/implementation.md diff --git a/docs/roadmap/configkit-adoption.md b/docs/roadmap/configkit-adoption.md new file mode 100644 index 0000000..ab3fb2d --- /dev/null +++ b/docs/roadmap/configkit-adoption.md @@ -0,0 +1,375 @@ +# ConfigKit Adoption and Platform Simplification + +**Status:** Accepted; implementation pending + +## Purpose + +Narratio currently owns a substantial YAML configuration subsystem for strict +decoding, explicit imports, additive composition, profile overlays, presence +tracking, provenance, and effective-value inspection. ConfigKit was extracted +from this implementation and now provides those generic responsibilities as a +maintained library. + +Narratio 2.0 will adopt ConfigKit as its configuration foundation, remove the +superseded generic infrastructure, and retain only Narratio-specific configuration +policy and domain resolution. The same major release will narrow the supported +platform set to the systems that Narratio is expected to run on in practice. + +This roadmap defines the desired contracts and end state. The +[implementation plan](implementation.md) owns sequencing and completion gates. + +## Goals + +- Reduce Narratio-owned configuration code and tests by delegating generic YAML + mechanics to ConfigKit. +- Establish one strict and explainable configuration lifecycle rather than + retaining parallel or compatibility loaders. +- Preserve Narratio's application-specific defaults, validation, campaign/session + resolution, party handling, artifact-family expansion, digests, and CLI policy. +- Make individual artifact definitions easy to organize, identify, inspect, and + reuse as dependencies of later artifacts. +- Keep raw credentials out of YAML and adopt a predictable secrets-directory + convention that works before configuration loading. +- Remove unsupported platform code and release obligations that provide no known + user value. +- Deliver the resulting contract changes as Narratio v2.0.0 with a clear migration + guide. + +## Platform Contract + +Narratio's v2 platform matrix is: + +| Platform | Status | Required evidence and deliverables | +| --- | --- | --- | +| Linux amd64 | Primary, supported | Default tests, race checks, static analysis, build validation, and a release asset | +| Linux arm64 | Primary, supported | Cross-build validation and a release asset; native validation is used when an appropriate runner is available | +| macOS arm64 | Best effort | Cross-build validation and a release asset; native CI and platform-equivalent runtime guarantees are not required release gates | +| macOS amd64 | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance | +| Windows | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance | + +Windows- and Intel-Mac-specific code, tests, CI entries, release packaging, and +documentation are removed when they no longer serve a supported build. Portable +code should remain portable where doing so is natural, but accidental compilation +on an unsupported target does not create a support commitment. + +Best-effort macOS arm64 support means Narratio intends to keep the target buildable +and will consider reported defects when practical. Lack of a native runner or an +unverified platform-specific behavior does not block a release. A target for which +Narratio publishes an asset must still pass its configured cross-build gate. + +## ConfigKit Dependency and Ownership + +Narratio will depend on the tagged ConfigKit module at an exact reviewed version, +initially `gitea.maximumdirect.net/eric/ConfigKit@v0.1.1`. ConfigKit is pre-1.0 and +may make compatibility changes in later minor versions, so dependency upgrades are +intentional changes accompanied by contract review and focused Narratio integration +tests; Narratio does not follow an unpinned branch or automatically accept a newer +minor release. + +ConfigKit owns: + +- bounded file, byte, and reader input; +- the accepted YAML syntax and strict typed decoding; +- import declarations, traversal, containment, and resource limits; +- additive composition and conflict attribution; +- selected-profile overlay semantics; +- explicit input presence and source provenance; +- application-default and recorded-resolution coordination; +- structured configuration diagnostics; +- secret-reference parsing and resolution; +- secrets-directory loading and its safe report; and +- detached effective configuration inspection supplied by its public result. + +Narratio owns: + +- pipeline, campaign, session, party, and runtime domain types; +- configuration-file discovery and CLI selection; +- concrete defaults, normalization, and semantic validation; +- campaign/session identity and stable-input resolution; +- canonical party parsing requirements and authored-byte retention; +- character artifact-family expansion and generated ownership attribution; +- root-relative interpretation of Narratio-owned runtime paths; +- effective pipeline digests and semantic profile diffs; +- command output and exit behavior; and +- adapter and runtime initialization after configuration succeeds. + +Narratio must use ConfigKit through its public package. It must not import +ConfigKit's private YAML implementation or reproduce ConfigKit's parser, +composition, profile, provenance, or secret-bootstrap mechanics locally. + +## Configuration Format + +Narratio adopts ConfigKit's configuration conventions directly. The old Narratio +`composition:` envelope, declared profile catalog, arbitrary overlay paths, and +`composition.default_profile` are removed without a compatibility parser. + +The root pipeline uses `configkit.imports` declarations. Import paths, optional +imports, globs, recursion, ordering, composition conflicts, containment, and limits +have the meanings defined by the pinned ConfigKit contract. Narratio does not add a +second import syntax or narrower application-specific traversal implementation. + +The unoverlaid root and its imports are the production configuration. A named +profile is a ConfigKit overlay at `profiles/.yml`; in particular, +`profiles/testing.yml` contains inexpensive-model overrides and testing-only +artifacts. Omitting `--profile`, or selecting ConfigKit's `default` profile, uses the +production base. `--profile testing` applies the testing overlay. Narratio does not +maintain a separate configured default-profile mechanism or validate unselected +profile files. + +ConfigKit's rules for explicit false, zero, empty, and null values replace +Narratio's custom presence tracking. Application defaults and normalization use a +ConfigKit definition and recorded resolution where the final value depends on +whether input was present. `NormalizeConfig.UnmarshalYAML` and equivalent +parser-coupled application hooks are removed. + +Pipeline, campaign, session, and party inputs use the common ConfigKit loading +boundary where its schema model can represent their types. Narratio may perform +narrow pre- or post-load domain work, such as retaining authored party bytes, +rejecting unresolved session-template placeholders, resolving stable inputs, and +expanding artifact families. Such work must not recreate a second generic YAML +loader. + +## Artifact Definition Ergonomics + +The canonical v2 configuration layout defines each ordinary Scriptorium artifact +in its own file under `artifacts/`. The root pipeline imports the directory through +ConfigKit, for example: + +```yaml +configkit: + imports: + - glob: artifacts/*.yml +``` + +Each file contributes one ordinary entry at +`scriptorium.artifacts.`. The map key is the authoritative Narratio +artifact key and therefore makes the artifact's internal identity explicit in the +definition itself: + +```yaml +# artifacts/session_recap.yml +scriptorium: + artifacts: + session_recap: + enabled: true + prompt_id: dnd.session_recap + output_path: artifacts/session_recap.md +``` + +The canonical reusable source ID is derived without another configured identifier: +`session_recap` becomes `narratio.artifact.session_recap`. The recommended filename +is `.yml`, but the filename is organizational rather than an identity +source. Renaming or moving a file without changing its map key does not change the +artifact contract. ConfigKit additive composition rejects duplicate artifact keys. + +The one-artifact-per-file layout is the documented happy path and is used by +maintained examples. Narratio does not prohibit an inline artifact or another +ConfigKit-compatible source organization merely because it differs from that +layout. Artifact identity always comes from the effective map key, never implicitly +from a filename or import position. + +Artifact families use the corresponding `artifact-families/*.yml` convention, with +one `scriptorium.artifact_families.` entry per file. The family key is +explicit. Concrete member keys and source IDs continue to use Narratio's existing +deterministic family/member naming rule after canonical party resolution. + +Narratio adds a read-only `narratio config artifacts` command using the ordinary +pipeline, profile, and campaign selectors. It reports every effective ordinary +artifact's key, canonical `narratio.artifact.` source ID, defining source, +enabled state, output path, and effective dependencies. Generated family members +also report their family origin. Output ordering is deterministic by artifact key, +and the command does not require a session or initialize runtime adapters. + +A configured-artifact input creates an implicit dependency on the referenced +artifact. For example: + +```yaml +inputs: + recap: + source: narratio.artifact.session_recap +``` + +The effective dependency graph is the union of explicit `depends_on` entries and +dependencies inferred from `narratio.artifact.` inputs. Inferred and explicit +duplicates collapse to one dependency. `depends_on` remains available for an +ordering requirement when the upstream output is not supplied as an input. +Validation applies the same existence, self-reference, and cycle rules to both +kinds of dependency, and diagnostics identify the declaration or input that +introduced an invalid edge. Operators no longer need to repeat an artifact input in +`depends_on` merely to make that input valid. + +## Effective Inspection, Digests, and Profile Diff + +ConfigKit results are the authority for composed input presence, base/profile +origins, diagnostics, and the initial typed effective configuration. Narratio's +configuration commands adapt those results to existing application-facing command +formats rather than exposing ConfigKit implementation types directly. + +Party-dependent artifact families are expanded only after the selected campaign and +canonical party are known. That derived state is intentionally outside ConfigKit's +completed load snapshot. Narratio retains a narrow application-owned layer that: + +- records generated family/member and publish-rule attribution; +- projects the final post-expansion pipeline for operator inspection; +- recomputes the effective pipeline digest after expansion; and +- ensures stages and adapters receive only the final concrete configuration. + +Narratio continues to own semantic `config diff` behavior because ConfigKit does +not provide configuration diffs or fingerprints. Each side of a profile comparison +may be loaded independently through ConfigKit. The former guarantee that both sides +come from one parsed source snapshot is removed; operators must not rely on a +meaningful comparison while concurrently modifying its source files. Equality is +still based on normalized effective values rather than YAML formatting or source +layout. + +## Credential and Secrets Convention + +Raw API keys and other credential values remain prohibited in pipeline, campaign, +session, generated configuration, manifests, diagnostics, examples, and logs. +Credential-bearing configuration fields use ConfigKit secret references: YAML +contains an environment-variable name, while the resolved value remains available +only through ConfigKit's trusted typed boundary. Fields should use credential names +rather than legacy names ending in `_env` when the v2 schema is updated. + +The `pipeline.secrets.env_dir` stanza is removed. Narratio selects a secrets +directory before parsing the pipeline using this precedence: + +1. an explicitly supplied `--secrets-dir PATH`; otherwise +2. the `secrets` subdirectory beside the selected root pipeline file. + +The conventional directory is derived from the resolved root pipeline path, never +from the current working directory, an imported file, or a selected profile. A +relative explicit CLI path is resolved from the caller's current working directory, +consistent with an ordinary command-line path argument. + +If an explicitly selected directory is missing or invalid, the command fails with a +clear bootstrap diagnostic. If the conventional directory is absent, Narratio +performs no directory bootstrap and continues; pre-existing environment variables +may satisfy all references. If the conventional directory exists but ConfigKit +rejects or cannot read it, the command fails. Already-set environment variables, +including empty values, retain ConfigKit's documented precedence. + +Narratio explicitly calls `configkit.LoadSecretsDir` at most once for a command, +before the first ConfigKit load that may resolve secret references and before any +concurrent work or runtime client initialization. This application-level convention +does not make ConfigKit's ordinary `Load` mutate process-global state. The bootstrap +report is rendered or summarized through a safe Narratio command boundary without +secret values. + +Narratio adopts ConfigKit's secret-directory rules and limits without restoring its +old permission-mode, selected-directory-symlink, invalid-entry, byte-limit, or +newline-trimming behavior through compatibility wrappers. Configuration inspection +may require the referenced environment variables or the selected secrets directory +because an ordinary successful ConfigKit load resolves effective typed secret +references. Inspection output may show non-secret reference names where appropriate, +but must never expose resolved values. + +## Validation and Diagnostics + +Narratio's semantic rules remain recognizable application code registered with the +appropriate ConfigKit definition. Validation continues to reject invalid pipeline, +campaign, session, party, adapter, publish, and artifact relationships before +runtime work where practical. Validation order and exact prose are not compatibility +contracts. + +ConfigKit diagnostic codes, paths, and source locations should be preserved through +the application error boundary where useful. Narratio adds command context and exit +classification but does not parse ConfigKit error strings or expose raw parser +errors. Secret values must remain absent from success and failure diagnostics. + +## Compatibility and Migration + +This work intentionally changes supported platforms, pipeline composition syntax, +profile layout and selection, secret configuration, YAML semantics, diagnostics, +and some inspection prerequisites. It is released as Narratio v2.0.0. + +There is no runtime compatibility layer for the old `composition` or +`pipeline.secrets` syntax. Maintained examples are converted to the ConfigKit +format, and the v2 release documentation provides a concise migration guide that +covers: + +- converting imports to `configkit.imports` declarations; +- folding the production overlay into the base configuration; +- placing testing overrides in `profiles/testing.yml`; +- moving ordinary artifacts and artifact families into the recommended individual + definition files and using their map keys as canonical identities; +- removing redundant `depends_on` entries when a configured-artifact input now + supplies the same dependency; +- moving secret files to the conventional sibling `secrets/` directory or using + `--secrets-dir`; +- renaming credential-reference fields where applicable; +- ConfigKit's stricter YAML and explicit-value semantics; +- the independent-load behavior of `config diff`; and +- the removal of Windows and macOS amd64 artifacts. + +Obsolete application implementation, tests, documentation, examples, CI paths, and +release machinery are removed rather than left deprecated inside the v2 codebase. + +## Required Changes + +- Add and pin the ConfigKit module dependency and establish a small application + integration boundary for definitions, result adaptation, and diagnostics. +- Replace Narratio's generic YAML parsing, composition, import, profile, presence, + and provenance implementation with ConfigKit public APIs. +- Express existing defaults, presence-sensitive normalization, and semantic + validation through ConfigKit definitions without moving Narratio domain policy + into the library. +- Adapt pipeline/campaign/session/party loading and preserve the existing + application resolution steps that occur after their independent typed loads. +- Rework effective configuration, sources, and diff commands around ConfigKit + results plus Narratio's post-party generated-attribution layer. +- Add the maintained per-artifact and per-family file layouts, the read-only + `config artifacts` catalog, and effective dependency inference from configured + artifact inputs. +- Convert credential configuration to typed references, add the common + `--secrets-dir` option, implement conventional-directory selection, and replace + the custom API-key directory loader with `configkit.LoadSecretsDir`. +- Update adapters to consume resolved credentials or safe environment reference + names from typed configuration without independently rediscovering secret values. +- Remove Windows and macOS amd64 build targets, release assets, CI entries, + platform-specific implementation, and obsolete tests and documentation. +- Convert maintained examples and add v2 migration and release documentation. +- Retain focused Narratio tests for domain policy and assembled application + behavior while deleting parser/composer/bootstrap mechanism tests now owned by + ConfigKit. Tests remain deterministic, offline, credential-free, and aligned with + the repository testing policy. +- Update canonical architecture, configuration, CLI, operations, development, + internal, troubleshooting, and release documentation only as the corresponding + behavior is implemented. + +## Target End State + +Narratio v2.0.0 has one configuration infrastructure dependency and no duplicate +YAML composition subsystem. Operators maintain a production base configuration, +optional explicit imports, and fixed named overlays such as `testing`. ConfigKit +provides strict parsing, composition, typed loading, provenance, diagnostics, and +credential resolution; Narratio supplies domain meaning and its post-campaign +artifact expansion. + +Operators can add an artifact by creating one discoverable YAML file whose map key +is its stable Narratio identity. `config artifacts` exposes that key, its canonical +source ID, origin, output, and effective dependency set. A later artifact can reuse +the output through `narratio.artifact.` without repeating the same edge in +`depends_on`, while explicit ordering-only dependencies remain supported. + +Secrets normally live in a predictable `secrets/` directory beside the selected +pipeline, with `--secrets-dir` available for deployment-specific placement. Raw +credential values never enter YAML or inspection output. The bootstrap is explicit +in application startup and ordinary ConfigKit loading remains free of process-global +mutation. + +Release and CI maintenance covers Linux amd64 and arm64 as the primary platforms +and macOS arm64 on a best-effort basis. Windows and macOS amd64 carry no remaining +support or release burden. The repository contains no legacy configuration parser, +profile compatibility layer, obsolete platform implementation, or redundant tests +for behavior owned by ConfigKit. + +## Deferred Work + +- A ConfigKit reference-only inspection or typed prepare/finalize lifecycle may be + considered upstream if credential-free inspection becomes a demonstrated shared + requirement. Narratio does not implement a private substitute in this migration. +- ConfigKit hot reload, remote providers, automatic configuration migration, and + application command frameworks remain outside Narratio's scope. +- Native macOS arm64 CI may be added later when a reliable runner is available; it + is not required for v2.0.0. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..efdcf72 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,480 @@ +# ConfigKit Adoption Implementation Plan + +Status: Pending + +Implement the [ConfigKit adoption roadmap](configkit-adoption.md) in the order +below. The target is Narratio v2.0.0: ConfigKit v0.1.1, the reduced platform +matrix, and the artifact-authoring workflow described there. Each stage is a +bounded implementation assignment for a gpt-5.6-terra coding agent. Mark a stage +complete only after its changes and acceptance checks succeed; record a short +completion and verification note beside its status. + +## Instructions for every stage + +- Read `docs/development.md` and all three policies under `docs/policy/`. + Read the relevant roadmap section and the stage-specific sources listed below. + Paths identify starting points, not an exhaustive edit allowlist; inspect callers + and tests before changing a package boundary. +- Use the tagged ConfigKit public API. The local `../ConfigKit` checkout is useful + reference material, but verify its version before relying on it. The module + cache for v0.1.1 is an alternative. Do not require a sibling checkout, a local + `replace`, a Go workspace, an upstream patch, or an unpinned revision to build. +- Keep Narratio domain policy in `internal/config`, startup and selectors in + `internal/app`, and credential delivery in the owning adapters. Do not build + a general configuration framework around ConfigKit. +- Keep each stage buildable and run focused behavioral tests for its changed + boundary. Temporary migration helpers may exist between stages only where + named here; they must not introduce runtime format auto-detection or fallback. +- Update each canonical current-behavior document when its behavior changes. + The roadmap remains the owner of unimplemented work. Preserve historical + release notes; their descriptions of old platforms and formats are history. +- Tests use real ConfigKit and temporary files, with fake external adapters or + storage where needed. Use synthetic credentials and isolate environment + mutation; such tests must not run in parallel. Retain domain and security + coverage, remove superseded mechanism tests, and avoid exact diagnostic prose + or private callback choreography assertions. +- Do not tag or publish v2.0.0 as part of this plan. Release preparation belongs + here; release authorization and tag publication are a later task. The existing + release completion boundary remains successful upstream tag push, with no CI + polling or waiting. + +## Decisions that apply across stages + +The roadmap is authoritative for product scope. These implementation choices +make that scope concrete: + +- Enable ConfigKit imports and profile selection for pipeline files only. + Campaign, session, and party are single-source definitions with imports and + profiles disabled. Preserve their domain schemas and source-relative inputs. + Omitted or empty profile and the exact name `default` select the base; other + names use ConfigKit's fixed `profiles/.yml` convention. +- Defaults apply to absent fields. Explicit empty/zero/false/null values follow + ConfigKit decoding and then domain validation; do not refill invalid explicit + values with legacy defaults. Optional null stage sections remain absent; + required sections fail domain validation rather than being resurrected. +- Keep ConfigKit metadata/result snapshots separate from runtime-only Narratio + state. ConfigKit discards ignored and unexported fields when detaching values; + attach root paths, retained party bytes, family catalogs, and digest metadata + after loading. No validator may mutate bound configuration. +- Use one Narratio-owned safe effective projection for the final expanded + pipeline. It contains configuration values and credential reference names, + never resolved credential bytes. It feeds show, semantic diff, digest, and + catalog data. ConfigKit metadata remains the authority for input provenance; + Narratio adds only domain-generated attribution. +- Rename `storage.s3.access_key_id_env` to `access_key_id`, + `storage.s3.secret_access_key_env` to `secret_access_key`, and + `audita.llm_api_key_env` to `llm_api_key`. Each is an optional `*configkit.Secret`. + An explicit reference always resolves during loading, even if the associated + stage will not execute. Require both S3 fields together or neither. With neither, + retain the AWS SDK default credential chain; do not synthesize mandatory secret + references that would break role-based authentication or local-only inspection. + With no Audita reference, preserve its existing downstream credential discovery. + Explicit references replace Narratio's old per-adapter environment lookup. +- Keep absent S3 credentials out of defaults; migration examples explicitly name + the existing AWS variables when static credentials are intended. A non-nil + missing/empty reference fails instead of silently falling back. This preserves + provider-managed credentials while making explicit configuration trustworthy. +- `config artifacts` uses the existing configuration command selectors and text + output convention. No new JSON mode, artifact creation wizard, inferred filename + identity, or profile catalog is added. +- Preserve the roadmap's unique base artifact definition rule explicitly: + ConfigKit rejects conflicting leaves but can compose disjoint children of the + same mapping. Use its public source metadata to reject an ordinary artifact key + declared by multiple base files, even if their fields are disjoint. Apply the + same rule to family keys. A selected profile may override or extend a definition. + This is a narrow Narratio declaration rule, not a different merge algorithm. +- Relax only the obsolete configuration/secrets filesystem restrictions identified + by the roadmap. Do not remove session restore, cleanup, publication, workspace + confinement, or ordinary shared-workspace mode guarantees as collateral work. + +## Stage 1 — Reduce the platform and release matrix + +Status: Pending + +Read: roadmap “Platform Contract”; `scripts/build-release-assets.sh`, +`scripts/check-release-candidate.sh`, `.woodpecker/`, `internal/releasecheck`, +`docs/release.md`, and platform-specific files in application, file operations, +and subprocess packages. + +1. Make Linux amd64, Linux arm64, and macOS arm64 the only maintained build and + release targets. Remove Windows suffix/packaging branches and macOS amd64 + targets from scripts, workflows, tests, and current documentation. +2. Delete Windows-only implementations and tests; simplify paired build-tag + files where the remaining implementations can be shared. Preserve portable + code when retaining it adds no special support machinery. Preserve functioning + Unix subprocess cancellation, locking, and file operations on both supported OSes. +3. Retain the local Linux verification gates and cross-build gates for all three + published assets. Do not introduce a native macOS runner requirement or claim + that cross-compilation verifies runtime behavior. +4. Update architecture/operations platform statements and contributor/release + guidance. Keep existing release protections and the no-CI-wait boundary intact. + +Acceptance: focused platform/subprocess/fileops/releasecheck tests pass; build all +three targets with `CGO_ENABLED=0`; verify the asset builder produces exactly the +three expected versioned binaries and the host binary reports the supplied version. +Existing release tests still prove guarded publication. No current support promise +or asset list retains Windows or macOS amd64. + +## Stage 2 — Separate configuration normalization from validation + +Status: Pending + +Read: `internal/config/load.go`, `defaults.go`, `normalize.go`, `validate*.go`, +`artifact_families.go`, and ConfigKit's “Recorded application resolution”, +“Semantic validation”, and “Defaults and typed decoding” contracts. + +1. Separate value-producing normalization from read-only validation, including + Notarius reference/output normalization currently performed by + `validateNotarius`. Audit all validators for map, slice, pointer, and scalar + mutations, including changes that would occur only on error paths. +2. Keep reusable domain normalization ordinary package code. Validation checks the + already normalized candidate and must work for both loaded and programmatically + constructed values without silently fixing them. +3. Inventory defaults in `load.go`: static defaults, per-map-entry/list-element + defaults, and inherited or presence-sensitive defaults. Retain named constants + in `defaults.go`. This inventory should guide code and concise comments, not + become a permanent parallel defaults document. +4. Keep the current loader calling normalization explicitly before validation + during this preparatory stage. The ConfigKit resolver replaces that loading + choreography in Stage 4. Preserve family expansion as an explicit domain step. + +Acceptance: focused configuration tests prove normalization is deterministic and +validation does not mutate accepted or rejected inputs. Cover representative +Notarius normalization/collision handling and existing artifact/publish rules. +Do not add a separate mutation test for every trivial validator. + +## Stage 3 — Replace composition-dependent effective projection + +Status: Pending + +Read: `internal/config/effective_pipeline.go`, `effective_digest.go`, +`pipeline_composition.go`, `internal/app/config_commands.go`, the config-diff +implementation, and `docs/internal/configuration.md`. + +1. Introduce one package-owned, schema-specific safe projection of the effective + pipeline. Use explicit typed projections with stable configuration field names; + preserve null versus empty containers and exact integers. Ordinary nested + configuration types may be reused where they contain no runtime state. + Credential fields must be projected explicitly as reference names when Stage 7 + changes their types. Do not introduce a generic reflection schema engine. +2. Render this projection as deterministic YAML using the existing YAML encoder; + encoding output is not a replacement YAML loader. Produce semantic records + directly from the safe projection using standard JSON encoding/decoding with + number preservation if a plain value tree is useful. Do not reparse YAML or + reconstruct `compositionNode` trees to compare or hash effective values. +3. Make show, diff values, and digest use that common projection. Mapping order is + immaterial; sequences remain atomic for diff; nil and empty remain distinct. + Exclude family declarations once expanded, all runtime metadata, profile names, + source locations, and raw credentials. Preserve logical Notarius path handling + so moving an equivalent bundle does not change its digest. +4. Version the changed digest encoding explicitly using the package's digest + version constant. Digest/profile metadata remains provenance, never a blanket + resume cache key. Keep stage/artifact semantic evidence behavior intact. +5. Decouple enumeration of effective field paths from the legacy composition + renderer. Input ownership can still come from the current loader until Stage 4. + +Acceptance: show/diff/digest tests cover equivalent monolithic/split values, +different map insertion order, null/empty distinction, family-expanded outputs, +and bundle relocation. A meaningful artifact change changes the digest; formatting +and profile naming alone do not. Existing semantic resume tests pass. No effective +projection consumer needs the custom composition parser/renderer. + +## Stage 4 — Cut pipeline loading over to ConfigKit + +Status: Pending + +Read: ConfigKit v0.1.1 `docs/config.md`, `docs/consumers/README.md`, and its +Narratio-shaped compiled example; Narratio `internal/config/load.go`, +`composition.go`, `pipeline_composition.go`, and profile/composition tests. + +1. Pin `gitea.maximumdirect.net/eric/ConfigKit v0.1.1` in `go.mod`/`go.sum`. + Implement a small pipeline definition using `Load`, `File`, enabled imports, + `Options.Profile`, defaults, recorded resolution, and read-only validation. +2. Migrate static defaults to fresh default values. Apply map-entry, list-element, + and inherited defaults with `Resolution.Default`/`IsExplicit`; use recorded + normalization for transformations from Stage 2. Do not call old post-decode + zero-filling defaults. Remove `NormalizeConfig.UnmarshalYAML` and its private + presence bit. Include publication list-entry defaults, optional section + allocation, and trim's Scriptorium requirement in this conversion. +3. Perform file-local semantic checks in the definition. Defer checks needing + canonical party expansion or campaign/session context to the existing explicit + resolution boundary; do not reject references to future concrete family members + merely because the campaign has not loaded yet. Run full checks after expansion. +4. Replace pipeline composition and profile loaders. `configkit.imports` is the + only directive; no `composition` or `_configkit` alias is accepted. Pass profile + selection directly under the agreed convention; unselected overlays are not read. + `LoadPipelineProfilePair` may remain a thin pair of independent loads, with no + shared parse cache or paired source-snapshot promise. +5. Adapt ConfigKit sources and metadata to Narratio's existing reporting records. + Translate paths structurally using public path APIs, not error-message parsing. + Preserve multiple contributing locations for composite/profile-owned mappings + and distinguish defaults/resolution from authored values. + Check artifact/family map-entry ownership for duplicate base declarations using + those locations; ignore synthetic defaults and allow the selected overlay. +6. Attach runtime metadata only after load succeeds. Preserve root-relative runtime + path interpretation and capture logical paths before making them absolute. + Retain family declarations and original ConfigKit metadata for later expansion. +7. Replace generic mechanism tests with a small integration set proving actual + Narratio wiring. Convert fixtures affected by this stage. Delete + `composition.go`, obsolete pipeline traversal/overlay/presence machinery, and + their redundant tests once the callers are migrated. + +Acceptance: a real root with imports and a testing overlay loads; duplicate claims +fail with useful source attribution, including two base files contributing disjoint +fields to one artifact or family key; base/default selection works; malformed +unselected profiles do not fail base loading. Omitted defaults and representative +explicit false/zero/empty/null inputs obey the new semantics. Invalid domain values +still fail. Commands use the new pipeline loader with no format fallback. Update +`docs/config.md` and configuration internals for the implemented boundary. + +## Stage 5 — Migrate campaign, session, and canonical party loading + +Status: Pending + +Read: `internal/config/load.go`, `party.go`, `party_resolution.go`, +`artifact_families.go`, session-template tests, `internal/app/config_loader.go`, +and ConfigKit source/schema/version contracts. + +1. Use separate ConfigKit definitions for campaign, session, and party. Replace + `decodeStrictYAML*` and the custom party YAML-node parser. Model the party input + as the existing string schema version and character-keyed typed map, with the + existing `alias` list, ordered class list, and optional positive class level. +2. Keep party domain checks: valid IDs, required display names and classes, + trimmed/control-free names, case-insensitive name/alias ambiguity, and duplicate + class checks. Validate the required version through `VersionRule`. Keep an + exact independent copy of authored bytes outside ConfigKit's schema object. +3. Preserve authored character order using successful metadata source coordinates + for the single party input when constructing `CanonicalParty.Characters`. + Preserve class/alias sequence order naturally. Family expansion and players + projection continue using their existing deterministic ID sorting. Do not keep + a second YAML parse just to recover mapping order. +4. Preserve the narrow session placeholder rejection before typed load, expected + session/previous-session identity checks, campaign selection, stable-input + precedence, and local versus remotely downloaded session provenance. + Retain the initially loaded pipeline/campaign during remote session acquisition. +5. Where bytes must be retained or inspected before `ConfigKit.Bytes`, bound the + initial read too, using the existing fileops bounded-read helper and a named + package constant of 8 MiB matching the pinned source limit. Otherwise prefer + `ConfigKit.File`. Do not read an unbounded file and rely on later rejection. +6. Complete post-party family and publish-rule expansion, then full cross-config + validation and digest refresh. Keep generated origins outside the library + snapshot. Remove unused decoder plumbing and obsolete party parsing tests. + +Acceptance: retained authored party bytes are identical; domain errors, aliases, +class order, family origins, and derived players are correct. Local and remote +session loads preserve identity, input path bases, and the original pipeline. +Unknown fields and disabled directives fail through ConfigKit. Focused config, +prepare/extract, and application loading tests pass. + +## Stage 6 — Move secrets bootstrap before all configuration loads + +Status: Pending + +Read: roadmap “Credential and Secrets Convention”; ConfigKit's bootstrap +contract; `internal/app/secrets_env*.go`, `config_loader.go`, `config_commands.go`, +`object_store.go`, command dependencies, and command argument owners. + +1. Add a shared `--secrets-dir` selector to every CLI path that loads a pipeline, + including configuration inspection/diff, execution aliases, planning, publish, + and session/storage operations. Route all through one command-owned bootstrap + context. Help/version and commands that do not load configuration do not bootstrap. +2. Discover the root pipeline first. An explicit relative directory is based on + caller CWD; otherwise choose `secrets/` beside the resolved root pipeline. + An explicitly empty flag is invalid. Missing conventional directories are + skipped; missing explicit directories and unreadable/invalid existing directories + fail. Use ConfigKit's selected-directory symlink behavior without custom rejection. +3. Call `LoadSecretsDir` at most once per command before any ConfigKit load or + adapter creation, including both loads of config diff and remote session lookup. + Use invocation state, not a package-global `sync.Once`. A new command gets its + own bootstrap; library config loaders themselves never mutate the environment. +4. Present safe bootstrap warnings/failures through stderr or the existing safe + diagnostic boundary, never into YAML show output. Preserve the partial report + on bootstrap failure. Do not add rollback or reinterpret ConfigKit file rules. +5. Remove late bootstrap calls in object-store and runner composition, the custom + directory reader and permission platform files, `pipeline.secrets`, its defaults + and validators, and their obsolete tests. Convert affected fixtures immediately. + +Acceptance: test absent conventional directory, explicit override from another +CWD, invalid explicit directory, already-set empty/nonempty environment values, +and bootstrap failure before adapter construction. A diff invocation and a remote +session acquisition each bootstrap once. Config inspection requires no session, +workspace writes, manifests, or runtime clients. Update CLI, config, operations, +architecture, and troubleshooting to the new bootstrap/permission contract. + +## Stage 7 — Resolve typed credentials and protect all output boundaries + +Status: Pending + +Read: ConfigKit “Typed secret references” and rendering contracts; +`internal/config/config.go`, `defaults.go`, `validate_adapters.go`, the safe +projection from Stage 3, `internal/adapters/storage/s3_backend.go`, +`internal/adapters/audita/subprocess.go`, and their composition callers. + +1. Implement the three renamed optional typed fields and S3 pair policy above. + Reject old `_env` schema keys; remove their default-name and lookup helpers. + Convert maintained fixtures with synthetic environment setup where references + are present. Do not add credentials to unrelated local pipeline fixtures. +2. Pass resolved S3 values to the SDK adapter's explicit static provider only when + the pair is configured. With neither, preserve its default provider chain. + Pass Audita the resolved value and safe reference name; deliver the value only + through the child environment's existing `AUDITA_LLM_API_KEY` override. + Do not re-read the environment for an explicitly resolved reference. +3. Adapt every serialization, digest, source/diff output, diagnostic, generated + invocation config, manifest, and provenance caller to safe reference names. + Never YAML-marshal ConfigKit's opaque secret directly or call `Value()` in an + inspection path. Secret changes alone must not alter effective digests or + invalidate artifact evidence; reference-name changes may change provenance. +4. Keep ConfigKit errors structured and safe. Domain validation messages must + avoid raw credential values; do not unwrap private parser errors or interpolate + whole trusted configuration structs into error messages. +5. Preserve downstream tool ownership for Notarius/Scriptorium credentials that + Narratio does not configure. Do not add a parallel credential schema for them. + +Acceptance: use distinctive synthetic secrets to prove no leakage in show, sources, +diff, catalog-ready projections, success/failure diagnostics, manifests, or generated +Audita configuration. Verify the child receives the credential and an environment +change after loading does not change explicit delivery. Test S3 paired references, +partial-pair rejection, missing effective references, and nil SDK fallback offline. +Update adapter and credential documentation and remove obsolete lookup tests. + +## Stage 8 — Infer and normalize artifact dependencies once + +Status: Pending + +Read: `internal/config/validate_scriptorium.go`, `artifact_families.go`, +`internal/artifactpolicy`, `internal/artifacts`, analyze planning/execution, and +their artifact dependency and semantic-evidence tests. + +1. At the final configuration resolution boundary, union explicit `depends_on` + with configured-artifact input references. Use the existing source-ID parser; + do not infer edges from filenames, arbitrary strings, or non-artifact inputs. +2. Infer after family member inputs become concrete `narratio.artifact.` + sources. Normalize, deduplicate, and sort the effective dependency list once. + Keep explicit ordering-only dependencies valid. Repeated finalization must not + accumulate edges or alter ordering. Store source-edge attribution separately + from the effective graph for diagnostics and generated provenance. +3. Apply existing existence, self-reference, enabled-dependency, output, and cycle + validation to the union. Remove the error requiring an input to be repeated in + `depends_on`. Retain diagnostics pointing to the explicit list entry or the + input that contributed an invalid edge, including family origin where relevant. +4. Ensure execution, targeted analysis, planning, prerequisite closure, reuse, + forced regeneration, and semantic evidence consume this same concrete graph. + No adapter or command may infer a second graph. Refresh final projection/digest + after inference; equivalent explicit and inferred declarations compare equally. + +Acceptance: cover input-only dependency ordering, duplicate explicit/inferred edges, +ordering-only edges, missing/self/cyclic references, and rewritten family inputs. +Use an assembled fake-artifact workflow to prove upstream output is supplied to a +dependent artifact and failure/resume retains successful predecessors. Existing +force/selection and invalidation tests remain valid. Update config and analyze +internals with the implemented inference rule. + +## Stage 9 — Add the effective artifact catalog command + +Status: Pending + +Read: `internal/app/config_commands.go`, config diff/source rendering, the safe +projection and family-origin catalogs, and `docs/cli.md`. + +1. Add `narratio config artifacts` to dispatch and help. Reuse ordinary pipeline, + profile, campaign, campaign-file, and secrets-directory parsing/resolution. + Resolve a campaign when families require it or when explicitly selected; a + pipeline with only ordinary artifacts can be inspected without a campaign. +2. Render one stable record per effective concrete artifact in key order with + key, `narratio.artifact.` source ID, defining source location(s), enabled + state, output path, and sorted effective dependencies. Family members additionally + identify family key, character ID, defining family source, and party source. +3. Derive identity from the map key. A profile may contribute to an existing + artifact, so report contributing definition locations instead of inventing + a single filename owner. Use a deterministic escaped tabular text rendering; + include a header and no records for an empty catalog. Propagate writer errors. +4. Use the same finalized state as show/sources/diff and runtime configuration. + No session download, adapter initialization, manifest access, or workspace + materialization is allowed. Secret bootstrap remains the agreed startup exception. + +Acceptance: CLI tests cover ordinary and generated artifacts, disabled entries, +profile-added/overridden artifacts, empty catalog, no-campaign ordinary inspection, +required-campaign failure for families, stable ordering, and no runtime side effects. +Check identity stays the same after moving an artifact file without changing its key. +Document the command in its canonical CLI owner and link to config layout examples. + +## Stage 10 — Finish the operator layout and v2 migration documentation + +Status: Pending + +Read: roadmap “Artifact Definition Ergonomics” and “Compatibility and Migration”; +`examples/README.md`, existing example bundles and their loader test, +`docs/config.md`, `docs/operations.md`, `docs/release.md`, and internal configuration +and artifact documentation. + +1. Convert maintained pipeline examples to a production base with explicit + `configkit.imports`, `artifacts/.yml`, `artifact-families/.yml`, and + `profiles/testing.yml`. Each recommended artifact/family file defines exactly + one explicit map entry. Fold old production overlays into the base. +2. Include an input-only dependency example and at least two character-family + examples sharing a campaign-owned party roster. Keep supported inline artifacts + legal; file layout is guidance, not a filename-based validation requirement. +3. Update the example validation harness to load root configurations and selected + testing profiles, not attempt to load each imported fragment as a full pipeline. + Use synthetic environment setup for explicit references. No example includes + real credentials or depends on local tools/cloud services to validate. +4. Consolidate canonical documentation updates from earlier stages. Remove current + references to old composition/profile catalogs, default-profile configuration, + secrets.env_dir, credential `_env` fields, credential-free inspection guarantees, + paired diff snapshots, and removed platform support. Explain explicit-value and + null changes without promising reloadable inspection output. +5. Add `docs/releases/v2.0.0.md` using the release-note structure, with a concrete + migration checklist linking to the canonical config/CLI docs and examples. + Include static S3-reference migration versus SDK credential discovery, new + inspection prerequisites, artifact input dependency inference, and platform + removals. Do not claim publication or edit older immutable release notes. + +Acceptance: `go test ./internal/doccheck` and +`go test ./internal/config -run '^TestExamplesLoadAndValidate$'` pass. Walk through +the new operator examples with config validate/show/sources/diff/artifacts offline. +Their identities, family members, origins, and testing overrides agree. + +## Stage 11 — Verify the complete migration and remove remaining scaffolding + +Status: Pending + +Read: the whole feature roadmap, completed stage notes, and the current test and +release procedures. Review the diff from the commit introducing this plan. + +1. Audit every roadmap requirement against current code and executable evidence. + Remove temporary migration adapters and any residual generic YAML parser, + composer, profile traversal, explicit-presence store, old secrets loader, + decoder hooks, or unused tests. `gopkg.in/yaml.v3` may remain for output encoding + and unrelated external formats; it must not supply a second configuration loader. +2. Confirm result-owned snapshots versus final expanded configuration are handled + consistently; every inspection/runtime path sees normalized inferred dependencies + and generated publish rules. Recheck secret safety and semantic resume boundaries. +3. Inspect current platform conditions and asset lists. Retain historical notes but + no active Windows/macOS amd64 obligation. Compare owned production/test code + before and after; report removals and any remaining complexity with reasons. + Do not chase a LOC quota by removing useful domain checks or regressions. +4. Run the repository gates with the maintained Go 1.26 toolchain: + + ```sh + go test ./... + go test -race ./... + go vet ./... + go build ./... + go test ./internal/doccheck + go test ./internal/config -run '^TestExamplesLoadAndValidate$' + ``` + +5. Run `scripts/check-release-candidate.sh v2.0.0` after the release note and all + candidate changes exist. This also verifies module hygiene and all three + release assets with `GOWORK=off`. Resolve failures; do not bypass checks or + publish a tag. Record native versus cross-build evidence accurately. + +Acceptance: all gates pass and every roadmap outcome has direct evidence. No +dependency on the sibling ConfigKit checkout remains. Record completion and any +best-effort platform limitations, then hand off for a separate release decision. + +## Open Questions + +None required before implementation. The agreed roadmap and the explicit choices +above resolve the migration decisions. Upstream credential-free inspection and +native macOS arm64 CI remain deferred, not prerequisites for this plan.