eric/narratio
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 27 Wiki Activity

Plan ConfigKit adoption and platform simplification

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-07 03:21:06 +00:00
parent b68435128b
commit 1558c9ccae
2 changed files with 855 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

375
docs/roadmap/configkit-adoption.md Normal file
View File

@@ -0,0 +1,375 @@
# ConfigKit Adoption and Platform Simplification
**Status:** Accepted; implementation pending
## Purpose
Narratio currently owns a substantial YAML configuration subsystem for strict
decoding, explicit imports, additive composition, profile overlays, presence
tracking, provenance, and effective-value inspection. ConfigKit was extracted
from this implementation and now provides those generic responsibilities as a
maintained library.
Narratio 2.0 will adopt ConfigKit as its configuration foundation, remove the
superseded generic infrastructure, and retain only Narratio-specific configuration
policy and domain resolution. The same major release will narrow the supported
platform set to the systems that Narratio is expected to run on in practice.
This roadmap defines the desired contracts and end state. The
[implementation plan](implementation.md) owns sequencing and completion gates.
## Goals
- Reduce Narratio-owned configuration code and tests by delegating generic YAML
mechanics to ConfigKit.
- Establish one strict and explainable configuration lifecycle rather than
retaining parallel or compatibility loaders.
- Preserve Narratio's application-specific defaults, validation, campaign/session
resolution, party handling, artifact-family expansion, digests, and CLI policy.
- Make individual artifact definitions easy to organize, identify, inspect, and
reuse as dependencies of later artifacts.
- Keep raw credentials out of YAML and adopt a predictable secrets-directory
convention that works before configuration loading.
- Remove unsupported platform code and release obligations that provide no known
user value.
- Deliver the resulting contract changes as Narratio v2.0.0 with a clear migration
guide.
## Platform Contract
Narratio's v2 platform matrix is:
| Platform | Status | Required evidence and deliverables |
| --- | --- | --- |
| Linux amd64 | Primary, supported | Default tests, race checks, static analysis, build validation, and a release asset |
| Linux arm64 | Primary, supported | Cross-build validation and a release asset; native validation is used when an appropriate runner is available |
| macOS arm64 | Best effort | Cross-build validation and a release asset; native CI and platform-equivalent runtime guarantees are not required release gates |
| macOS amd64 | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance |
| Windows | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance |
Windows- and Intel-Mac-specific code, tests, CI entries, release packaging, and
documentation are removed when they no longer serve a supported build. Portable
code should remain portable where doing so is natural, but accidental compilation
on an unsupported target does not create a support commitment.
Best-effort macOS arm64 support means Narratio intends to keep the target buildable
and will consider reported defects when practical. Lack of a native runner or an
unverified platform-specific behavior does not block a release. A target for which
Narratio publishes an asset must still pass its configured cross-build gate.
## ConfigKit Dependency and Ownership
Narratio will depend on the tagged ConfigKit module at an exact reviewed version,
initially `gitea.maximumdirect.net/eric/ConfigKit@v0.1.1`. ConfigKit is pre-1.0 and
may make compatibility changes in later minor versions, so dependency upgrades are
intentional changes accompanied by contract review and focused Narratio integration
tests; Narratio does not follow an unpinned branch or automatically accept a newer
minor release.
ConfigKit owns:
- bounded file, byte, and reader input;
- the accepted YAML syntax and strict typed decoding;
- import declarations, traversal, containment, and resource limits;
- additive composition and conflict attribution;
- selected-profile overlay semantics;
- explicit input presence and source provenance;
- application-default and recorded-resolution coordination;
- structured configuration diagnostics;
- secret-reference parsing and resolution;
- secrets-directory loading and its safe report; and
- detached effective configuration inspection supplied by its public result.
Narratio owns:
- pipeline, campaign, session, party, and runtime domain types;
- configuration-file discovery and CLI selection;
- concrete defaults, normalization, and semantic validation;
- campaign/session identity and stable-input resolution;
- canonical party parsing requirements and authored-byte retention;
- character artifact-family expansion and generated ownership attribution;
- root-relative interpretation of Narratio-owned runtime paths;
- effective pipeline digests and semantic profile diffs;
- command output and exit behavior; and
- adapter and runtime initialization after configuration succeeds.
Narratio must use ConfigKit through its public package. It must not import
ConfigKit's private YAML implementation or reproduce ConfigKit's parser,
composition, profile, provenance, or secret-bootstrap mechanics locally.
## Configuration Format
Narratio adopts ConfigKit's configuration conventions directly. The old Narratio
`composition:` envelope, declared profile catalog, arbitrary overlay paths, and
`composition.default_profile` are removed without a compatibility parser.
The root pipeline uses `configkit.imports` declarations. Import paths, optional
imports, globs, recursion, ordering, composition conflicts, containment, and limits
have the meanings defined by the pinned ConfigKit contract. Narratio does not add a
second import syntax or narrower application-specific traversal implementation.
The unoverlaid root and its imports are the production configuration. A named
profile is a ConfigKit overlay at `profiles/<name>.yml`; in particular,
`profiles/testing.yml` contains inexpensive-model overrides and testing-only
artifacts. Omitting `--profile`, or selecting ConfigKit's `default` profile, uses the
production base. `--profile testing` applies the testing overlay. Narratio does not
maintain a separate configured default-profile mechanism or validate unselected
profile files.
ConfigKit's rules for explicit false, zero, empty, and null values replace
Narratio's custom presence tracking. Application defaults and normalization use a
ConfigKit definition and recorded resolution where the final value depends on
whether input was present. `NormalizeConfig.UnmarshalYAML` and equivalent
parser-coupled application hooks are removed.
Pipeline, campaign, session, and party inputs use the common ConfigKit loading
boundary where its schema model can represent their types. Narratio may perform
narrow pre- or post-load domain work, such as retaining authored party bytes,
rejecting unresolved session-template placeholders, resolving stable inputs, and
expanding artifact families. Such work must not recreate a second generic YAML
loader.
## Artifact Definition Ergonomics
The canonical v2 configuration layout defines each ordinary Scriptorium artifact
in its own file under `artifacts/`. The root pipeline imports the directory through
ConfigKit, for example:
```yaml
configkit:
imports:
- glob: artifacts/*.yml
```
Each file contributes one ordinary entry at
`scriptorium.artifacts.<artifact_key>`. The map key is the authoritative Narratio
artifact key and therefore makes the artifact's internal identity explicit in the
definition itself:
```yaml
# artifacts/session_recap.yml
scriptorium:
artifacts:
session_recap:
enabled: true
prompt_id: dnd.session_recap
output_path: artifacts/session_recap.md
```
The canonical reusable source ID is derived without another configured identifier:
`session_recap` becomes `narratio.artifact.session_recap`. The recommended filename
is `<artifact_key>.yml`, but the filename is organizational rather than an identity
source. Renaming or moving a file without changing its map key does not change the
artifact contract. ConfigKit additive composition rejects duplicate artifact keys.
The one-artifact-per-file layout is the documented happy path and is used by
maintained examples. Narratio does not prohibit an inline artifact or another
ConfigKit-compatible source organization merely because it differs from that
layout. Artifact identity always comes from the effective map key, never implicitly
from a filename or import position.
Artifact families use the corresponding `artifact-families/*.yml` convention, with
one `scriptorium.artifact_families.<family_key>` entry per file. The family key is
explicit. Concrete member keys and source IDs continue to use Narratio's existing
deterministic family/member naming rule after canonical party resolution.
Narratio adds a read-only `narratio config artifacts` command using the ordinary
pipeline, profile, and campaign selectors. It reports every effective ordinary
artifact's key, canonical `narratio.artifact.<key>` source ID, defining source,
enabled state, output path, and effective dependencies. Generated family members
also report their family origin. Output ordering is deterministic by artifact key,
and the command does not require a session or initialize runtime adapters.
A configured-artifact input creates an implicit dependency on the referenced
artifact. For example:
```yaml
inputs:
recap:
source: narratio.artifact.session_recap
```
The effective dependency graph is the union of explicit `depends_on` entries and
dependencies inferred from `narratio.artifact.<key>` inputs. Inferred and explicit
duplicates collapse to one dependency. `depends_on` remains available for an
ordering requirement when the upstream output is not supplied as an input.
Validation applies the same existence, self-reference, and cycle rules to both
kinds of dependency, and diagnostics identify the declaration or input that
introduced an invalid edge. Operators no longer need to repeat an artifact input in
`depends_on` merely to make that input valid.
## Effective Inspection, Digests, and Profile Diff
ConfigKit results are the authority for composed input presence, base/profile
origins, diagnostics, and the initial typed effective configuration. Narratio's
configuration commands adapt those results to existing application-facing command
formats rather than exposing ConfigKit implementation types directly.
Party-dependent artifact families are expanded only after the selected campaign and
canonical party are known. That derived state is intentionally outside ConfigKit's
completed load snapshot. Narratio retains a narrow application-owned layer that:
- records generated family/member and publish-rule attribution;
- projects the final post-expansion pipeline for operator inspection;
- recomputes the effective pipeline digest after expansion; and
- ensures stages and adapters receive only the final concrete configuration.
Narratio continues to own semantic `config diff` behavior because ConfigKit does
not provide configuration diffs or fingerprints. Each side of a profile comparison
may be loaded independently through ConfigKit. The former guarantee that both sides
come from one parsed source snapshot is removed; operators must not rely on a
meaningful comparison while concurrently modifying its source files. Equality is
still based on normalized effective values rather than YAML formatting or source
layout.
## Credential and Secrets Convention
Raw API keys and other credential values remain prohibited in pipeline, campaign,
session, generated configuration, manifests, diagnostics, examples, and logs.
Credential-bearing configuration fields use ConfigKit secret references: YAML
contains an environment-variable name, while the resolved value remains available
only through ConfigKit's trusted typed boundary. Fields should use credential names
rather than legacy names ending in `_env` when the v2 schema is updated.
The `pipeline.secrets.env_dir` stanza is removed. Narratio selects a secrets
directory before parsing the pipeline using this precedence:
1. an explicitly supplied `--secrets-dir PATH`; otherwise
2. the `secrets` subdirectory beside the selected root pipeline file.
The conventional directory is derived from the resolved root pipeline path, never
from the current working directory, an imported file, or a selected profile. A
relative explicit CLI path is resolved from the caller's current working directory,
consistent with an ordinary command-line path argument.
If an explicitly selected directory is missing or invalid, the command fails with a
clear bootstrap diagnostic. If the conventional directory is absent, Narratio
performs no directory bootstrap and continues; pre-existing environment variables
may satisfy all references. If the conventional directory exists but ConfigKit
rejects or cannot read it, the command fails. Already-set environment variables,
including empty values, retain ConfigKit's documented precedence.
Narratio explicitly calls `configkit.LoadSecretsDir` at most once for a command,
before the first ConfigKit load that may resolve secret references and before any
concurrent work or runtime client initialization. This application-level convention
does not make ConfigKit's ordinary `Load` mutate process-global state. The bootstrap
report is rendered or summarized through a safe Narratio command boundary without
secret values.
Narratio adopts ConfigKit's secret-directory rules and limits without restoring its
old permission-mode, selected-directory-symlink, invalid-entry, byte-limit, or
newline-trimming behavior through compatibility wrappers. Configuration inspection
may require the referenced environment variables or the selected secrets directory
because an ordinary successful ConfigKit load resolves effective typed secret
references. Inspection output may show non-secret reference names where appropriate,
but must never expose resolved values.
## Validation and Diagnostics
Narratio's semantic rules remain recognizable application code registered with the
appropriate ConfigKit definition. Validation continues to reject invalid pipeline,
campaign, session, party, adapter, publish, and artifact relationships before
runtime work where practical. Validation order and exact prose are not compatibility
contracts.
ConfigKit diagnostic codes, paths, and source locations should be preserved through
the application error boundary where useful. Narratio adds command context and exit
classification but does not parse ConfigKit error strings or expose raw parser
errors. Secret values must remain absent from success and failure diagnostics.
## Compatibility and Migration
This work intentionally changes supported platforms, pipeline composition syntax,
profile layout and selection, secret configuration, YAML semantics, diagnostics,
and some inspection prerequisites. It is released as Narratio v2.0.0.
There is no runtime compatibility layer for the old `composition` or
`pipeline.secrets` syntax. Maintained examples are converted to the ConfigKit
format, and the v2 release documentation provides a concise migration guide that
covers:
- converting imports to `configkit.imports` declarations;
- folding the production overlay into the base configuration;
- placing testing overrides in `profiles/testing.yml`;
- moving ordinary artifacts and artifact families into the recommended individual
definition files and using their map keys as canonical identities;
- removing redundant `depends_on` entries when a configured-artifact input now
supplies the same dependency;
- moving secret files to the conventional sibling `secrets/` directory or using
`--secrets-dir`;
- renaming credential-reference fields where applicable;
- ConfigKit's stricter YAML and explicit-value semantics;
- the independent-load behavior of `config diff`; and
- the removal of Windows and macOS amd64 artifacts.
Obsolete application implementation, tests, documentation, examples, CI paths, and
release machinery are removed rather than left deprecated inside the v2 codebase.
## Required Changes
- Add and pin the ConfigKit module dependency and establish a small application
integration boundary for definitions, result adaptation, and diagnostics.
- Replace Narratio's generic YAML parsing, composition, import, profile, presence,
and provenance implementation with ConfigKit public APIs.
- Express existing defaults, presence-sensitive normalization, and semantic
validation through ConfigKit definitions without moving Narratio domain policy
into the library.
- Adapt pipeline/campaign/session/party loading and preserve the existing
application resolution steps that occur after their independent typed loads.
- Rework effective configuration, sources, and diff commands around ConfigKit
results plus Narratio's post-party generated-attribution layer.
- Add the maintained per-artifact and per-family file layouts, the read-only
`config artifacts` catalog, and effective dependency inference from configured
artifact inputs.
- Convert credential configuration to typed references, add the common
`--secrets-dir` option, implement conventional-directory selection, and replace
the custom API-key directory loader with `configkit.LoadSecretsDir`.
- Update adapters to consume resolved credentials or safe environment reference
names from typed configuration without independently rediscovering secret values.
- Remove Windows and macOS amd64 build targets, release assets, CI entries,
platform-specific implementation, and obsolete tests and documentation.
- Convert maintained examples and add v2 migration and release documentation.
- Retain focused Narratio tests for domain policy and assembled application
behavior while deleting parser/composer/bootstrap mechanism tests now owned by
ConfigKit. Tests remain deterministic, offline, credential-free, and aligned with
the repository testing policy.
- Update canonical architecture, configuration, CLI, operations, development,
internal, troubleshooting, and release documentation only as the corresponding
behavior is implemented.
## Target End State
Narratio v2.0.0 has one configuration infrastructure dependency and no duplicate
YAML composition subsystem. Operators maintain a production base configuration,
optional explicit imports, and fixed named overlays such as `testing`. ConfigKit
provides strict parsing, composition, typed loading, provenance, diagnostics, and
credential resolution; Narratio supplies domain meaning and its post-campaign
artifact expansion.
Operators can add an artifact by creating one discoverable YAML file whose map key
is its stable Narratio identity. `config artifacts` exposes that key, its canonical
source ID, origin, output, and effective dependency set. A later artifact can reuse
the output through `narratio.artifact.<key>` without repeating the same edge in
`depends_on`, while explicit ordering-only dependencies remain supported.
Secrets normally live in a predictable `secrets/` directory beside the selected
pipeline, with `--secrets-dir` available for deployment-specific placement. Raw
credential values never enter YAML or inspection output. The bootstrap is explicit
in application startup and ordinary ConfigKit loading remains free of process-global
mutation.
Release and CI maintenance covers Linux amd64 and arm64 as the primary platforms
and macOS arm64 on a best-effort basis. Windows and macOS amd64 carry no remaining
support or release burden. The repository contains no legacy configuration parser,
profile compatibility layer, obsolete platform implementation, or redundant tests
for behavior owned by ConfigKit.
## Deferred Work
- A ConfigKit reference-only inspection or typed prepare/finalize lifecycle may be
considered upstream if credential-free inspection becomes a demonstrated shared
requirement. Narratio does not implement a private substitute in this migration.
- ConfigKit hot reload, remote providers, automatic configuration migration, and
application command frameworks remain outside Narratio's scope.
- Native macOS arm64 CI may be added later when a reliable runner is available; it
is not required for v2.0.0.

480
docs/roadmap/implementation.md Normal file
View File

@@ -0,0 +1,480 @@
# ConfigKit Adoption Implementation Plan
Status: Pending
Implement the [ConfigKit adoption roadmap](configkit-adoption.md) in the order
below. The target is Narratio v2.0.0: ConfigKit v0.1.1, the reduced platform
matrix, and the artifact-authoring workflow described there. Each stage is a
bounded implementation assignment for a gpt-5.6-terra coding agent. Mark a stage
complete only after its changes and acceptance checks succeed; record a short
completion and verification note beside its status.
## Instructions for every stage
- Read `docs/development.md` and all three policies under `docs/policy/`.
Read the relevant roadmap section and the stage-specific sources listed below.
Paths identify starting points, not an exhaustive edit allowlist; inspect callers
and tests before changing a package boundary.
- Use the tagged ConfigKit public API. The local `../ConfigKit` checkout is useful
reference material, but verify its version before relying on it. The module
cache for v0.1.1 is an alternative. Do not require a sibling checkout, a local
`replace`, a Go workspace, an upstream patch, or an unpinned revision to build.
- Keep Narratio domain policy in `internal/config`, startup and selectors in
`internal/app`, and credential delivery in the owning adapters. Do not build
a general configuration framework around ConfigKit.
- Keep each stage buildable and run focused behavioral tests for its changed
boundary. Temporary migration helpers may exist between stages only where
named here; they must not introduce runtime format auto-detection or fallback.
- Update each canonical current-behavior document when its behavior changes.
The roadmap remains the owner of unimplemented work. Preserve historical
release notes; their descriptions of old platforms and formats are history.
- Tests use real ConfigKit and temporary files, with fake external adapters or
storage where needed. Use synthetic credentials and isolate environment
mutation; such tests must not run in parallel. Retain domain and security
coverage, remove superseded mechanism tests, and avoid exact diagnostic prose
or private callback choreography assertions.
- Do not tag or publish v2.0.0 as part of this plan. Release preparation belongs
here; release authorization and tag publication are a later task. The existing
release completion boundary remains successful upstream tag push, with no CI
polling or waiting.
## Decisions that apply across stages
The roadmap is authoritative for product scope. These implementation choices
make that scope concrete:
- Enable ConfigKit imports and profile selection for pipeline files only.
Campaign, session, and party are single-source definitions with imports and
profiles disabled. Preserve their domain schemas and source-relative inputs.
Omitted or empty profile and the exact name `default` select the base; other
names use ConfigKit's fixed `profiles/<name>.yml` convention.
- Defaults apply to absent fields. Explicit empty/zero/false/null values follow
ConfigKit decoding and then domain validation; do not refill invalid explicit
values with legacy defaults. Optional null stage sections remain absent;
required sections fail domain validation rather than being resurrected.
- Keep ConfigKit metadata/result snapshots separate from runtime-only Narratio
state. ConfigKit discards ignored and unexported fields when detaching values;
attach root paths, retained party bytes, family catalogs, and digest metadata
after loading. No validator may mutate bound configuration.
- Use one Narratio-owned safe effective projection for the final expanded
pipeline. It contains configuration values and credential reference names,
never resolved credential bytes. It feeds show, semantic diff, digest, and
catalog data. ConfigKit metadata remains the authority for input provenance;
Narratio adds only domain-generated attribution.
- Rename `storage.s3.access_key_id_env` to `access_key_id`,
`storage.s3.secret_access_key_env` to `secret_access_key`, and
`audita.llm_api_key_env` to `llm_api_key`. Each is an optional `*configkit.Secret`.
An explicit reference always resolves during loading, even if the associated
stage will not execute. Require both S3 fields together or neither. With neither,
retain the AWS SDK default credential chain; do not synthesize mandatory secret
references that would break role-based authentication or local-only inspection.
With no Audita reference, preserve its existing downstream credential discovery.
Explicit references replace Narratio's old per-adapter environment lookup.
- Keep absent S3 credentials out of defaults; migration examples explicitly name
the existing AWS variables when static credentials are intended. A non-nil
missing/empty reference fails instead of silently falling back. This preserves
provider-managed credentials while making explicit configuration trustworthy.
- `config artifacts` uses the existing configuration command selectors and text
output convention. No new JSON mode, artifact creation wizard, inferred filename
identity, or profile catalog is added.
- Preserve the roadmap's unique base artifact definition rule explicitly:
ConfigKit rejects conflicting leaves but can compose disjoint children of the
same mapping. Use its public source metadata to reject an ordinary artifact key
declared by multiple base files, even if their fields are disjoint. Apply the
same rule to family keys. A selected profile may override or extend a definition.
This is a narrow Narratio declaration rule, not a different merge algorithm.
- Relax only the obsolete configuration/secrets filesystem restrictions identified
by the roadmap. Do not remove session restore, cleanup, publication, workspace
confinement, or ordinary shared-workspace mode guarantees as collateral work.
## Stage 1 — Reduce the platform and release matrix
Status: Pending
Read: roadmap “Platform Contract”; `scripts/build-release-assets.sh`,
`scripts/check-release-candidate.sh`, `.woodpecker/`, `internal/releasecheck`,
`docs/release.md`, and platform-specific files in application, file operations,
and subprocess packages.
1. Make Linux amd64, Linux arm64, and macOS arm64 the only maintained build and
release targets. Remove Windows suffix/packaging branches and macOS amd64
targets from scripts, workflows, tests, and current documentation.
2. Delete Windows-only implementations and tests; simplify paired build-tag
files where the remaining implementations can be shared. Preserve portable
code when retaining it adds no special support machinery. Preserve functioning
Unix subprocess cancellation, locking, and file operations on both supported OSes.
3. Retain the local Linux verification gates and cross-build gates for all three
published assets. Do not introduce a native macOS runner requirement or claim
that cross-compilation verifies runtime behavior.
4. Update architecture/operations platform statements and contributor/release
guidance. Keep existing release protections and the no-CI-wait boundary intact.
Acceptance: focused platform/subprocess/fileops/releasecheck tests pass; build all
three targets with `CGO_ENABLED=0`; verify the asset builder produces exactly the
three expected versioned binaries and the host binary reports the supplied version.
Existing release tests still prove guarded publication. No current support promise
or asset list retains Windows or macOS amd64.
## Stage 2 — Separate configuration normalization from validation
Status: Pending
Read: `internal/config/load.go`, `defaults.go`, `normalize.go`, `validate*.go`,
`artifact_families.go`, and ConfigKit's “Recorded application resolution”,
“Semantic validation”, and “Defaults and typed decoding” contracts.
1. Separate value-producing normalization from read-only validation, including
Notarius reference/output normalization currently performed by
`validateNotarius`. Audit all validators for map, slice, pointer, and scalar
mutations, including changes that would occur only on error paths.
2. Keep reusable domain normalization ordinary package code. Validation checks the
already normalized candidate and must work for both loaded and programmatically
constructed values without silently fixing them.
3. Inventory defaults in `load.go`: static defaults, per-map-entry/list-element
defaults, and inherited or presence-sensitive defaults. Retain named constants
in `defaults.go`. This inventory should guide code and concise comments, not
become a permanent parallel defaults document.
4. Keep the current loader calling normalization explicitly before validation
during this preparatory stage. The ConfigKit resolver replaces that loading
choreography in Stage 4. Preserve family expansion as an explicit domain step.
Acceptance: focused configuration tests prove normalization is deterministic and
validation does not mutate accepted or rejected inputs. Cover representative
Notarius normalization/collision handling and existing artifact/publish rules.
Do not add a separate mutation test for every trivial validator.
## Stage 3 — Replace composition-dependent effective projection
Status: Pending
Read: `internal/config/effective_pipeline.go`, `effective_digest.go`,
`pipeline_composition.go`, `internal/app/config_commands.go`, the config-diff
implementation, and `docs/internal/configuration.md`.
1. Introduce one package-owned, schema-specific safe projection of the effective
pipeline. Use explicit typed projections with stable configuration field names;
preserve null versus empty containers and exact integers. Ordinary nested
configuration types may be reused where they contain no runtime state.
Credential fields must be projected explicitly as reference names when Stage 7
changes their types. Do not introduce a generic reflection schema engine.
2. Render this projection as deterministic YAML using the existing YAML encoder;
encoding output is not a replacement YAML loader. Produce semantic records
directly from the safe projection using standard JSON encoding/decoding with
number preservation if a plain value tree is useful. Do not reparse YAML or
reconstruct `compositionNode` trees to compare or hash effective values.
3. Make show, diff values, and digest use that common projection. Mapping order is
immaterial; sequences remain atomic for diff; nil and empty remain distinct.
Exclude family declarations once expanded, all runtime metadata, profile names,
source locations, and raw credentials. Preserve logical Notarius path handling
so moving an equivalent bundle does not change its digest.
4. Version the changed digest encoding explicitly using the package's digest
version constant. Digest/profile metadata remains provenance, never a blanket
resume cache key. Keep stage/artifact semantic evidence behavior intact.
5. Decouple enumeration of effective field paths from the legacy composition
renderer. Input ownership can still come from the current loader until Stage 4.
Acceptance: show/diff/digest tests cover equivalent monolithic/split values,
different map insertion order, null/empty distinction, family-expanded outputs,
and bundle relocation. A meaningful artifact change changes the digest; formatting
and profile naming alone do not. Existing semantic resume tests pass. No effective
projection consumer needs the custom composition parser/renderer.
## Stage 4 — Cut pipeline loading over to ConfigKit
Status: Pending
Read: ConfigKit v0.1.1 `docs/config.md`, `docs/consumers/README.md`, and its
Narratio-shaped compiled example; Narratio `internal/config/load.go`,
`composition.go`, `pipeline_composition.go`, and profile/composition tests.
1. Pin `gitea.maximumdirect.net/eric/ConfigKit v0.1.1` in `go.mod`/`go.sum`.
Implement a small pipeline definition using `Load`, `File`, enabled imports,
`Options.Profile`, defaults, recorded resolution, and read-only validation.
2. Migrate static defaults to fresh default values. Apply map-entry, list-element,
and inherited defaults with `Resolution.Default`/`IsExplicit`; use recorded
normalization for transformations from Stage 2. Do not call old post-decode
zero-filling defaults. Remove `NormalizeConfig.UnmarshalYAML` and its private
presence bit. Include publication list-entry defaults, optional section
allocation, and trim's Scriptorium requirement in this conversion.
3. Perform file-local semantic checks in the definition. Defer checks needing
canonical party expansion or campaign/session context to the existing explicit
resolution boundary; do not reject references to future concrete family members
merely because the campaign has not loaded yet. Run full checks after expansion.
4. Replace pipeline composition and profile loaders. `configkit.imports` is the
only directive; no `composition` or `_configkit` alias is accepted. Pass profile
selection directly under the agreed convention; unselected overlays are not read.
`LoadPipelineProfilePair` may remain a thin pair of independent loads, with no
shared parse cache or paired source-snapshot promise.
5. Adapt ConfigKit sources and metadata to Narratio's existing reporting records.
Translate paths structurally using public path APIs, not error-message parsing.
Preserve multiple contributing locations for composite/profile-owned mappings
and distinguish defaults/resolution from authored values.
Check artifact/family map-entry ownership for duplicate base declarations using
those locations; ignore synthetic defaults and allow the selected overlay.
6. Attach runtime metadata only after load succeeds. Preserve root-relative runtime
path interpretation and capture logical paths before making them absolute.
Retain family declarations and original ConfigKit metadata for later expansion.
7. Replace generic mechanism tests with a small integration set proving actual
Narratio wiring. Convert fixtures affected by this stage. Delete
`composition.go`, obsolete pipeline traversal/overlay/presence machinery, and
their redundant tests once the callers are migrated.
Acceptance: a real root with imports and a testing overlay loads; duplicate claims
fail with useful source attribution, including two base files contributing disjoint
fields to one artifact or family key; base/default selection works; malformed
unselected profiles do not fail base loading. Omitted defaults and representative
explicit false/zero/empty/null inputs obey the new semantics. Invalid domain values
still fail. Commands use the new pipeline loader with no format fallback. Update
`docs/config.md` and configuration internals for the implemented boundary.
## Stage 5 — Migrate campaign, session, and canonical party loading
Status: Pending
Read: `internal/config/load.go`, `party.go`, `party_resolution.go`,
`artifact_families.go`, session-template tests, `internal/app/config_loader.go`,
and ConfigKit source/schema/version contracts.
1. Use separate ConfigKit definitions for campaign, session, and party. Replace
`decodeStrictYAML*` and the custom party YAML-node parser. Model the party input
as the existing string schema version and character-keyed typed map, with the
existing `alias` list, ordered class list, and optional positive class level.
2. Keep party domain checks: valid IDs, required display names and classes,
trimmed/control-free names, case-insensitive name/alias ambiguity, and duplicate
class checks. Validate the required version through `VersionRule`. Keep an
exact independent copy of authored bytes outside ConfigKit's schema object.
3. Preserve authored character order using successful metadata source coordinates
for the single party input when constructing `CanonicalParty.Characters`.
Preserve class/alias sequence order naturally. Family expansion and players
projection continue using their existing deterministic ID sorting. Do not keep
a second YAML parse just to recover mapping order.
4. Preserve the narrow session placeholder rejection before typed load, expected
session/previous-session identity checks, campaign selection, stable-input
precedence, and local versus remotely downloaded session provenance.
Retain the initially loaded pipeline/campaign during remote session acquisition.
5. Where bytes must be retained or inspected before `ConfigKit.Bytes`, bound the
initial read too, using the existing fileops bounded-read helper and a named
package constant of 8 MiB matching the pinned source limit. Otherwise prefer
`ConfigKit.File`. Do not read an unbounded file and rely on later rejection.
6. Complete post-party family and publish-rule expansion, then full cross-config
validation and digest refresh. Keep generated origins outside the library
snapshot. Remove unused decoder plumbing and obsolete party parsing tests.
Acceptance: retained authored party bytes are identical; domain errors, aliases,
class order, family origins, and derived players are correct. Local and remote
session loads preserve identity, input path bases, and the original pipeline.
Unknown fields and disabled directives fail through ConfigKit. Focused config,
prepare/extract, and application loading tests pass.
## Stage 6 — Move secrets bootstrap before all configuration loads
Status: Pending
Read: roadmap “Credential and Secrets Convention”; ConfigKit's bootstrap
contract; `internal/app/secrets_env*.go`, `config_loader.go`, `config_commands.go`,
`object_store.go`, command dependencies, and command argument owners.
1. Add a shared `--secrets-dir` selector to every CLI path that loads a pipeline,
including configuration inspection/diff, execution aliases, planning, publish,
and session/storage operations. Route all through one command-owned bootstrap
context. Help/version and commands that do not load configuration do not bootstrap.
2. Discover the root pipeline first. An explicit relative directory is based on
caller CWD; otherwise choose `secrets/` beside the resolved root pipeline.
An explicitly empty flag is invalid. Missing conventional directories are
skipped; missing explicit directories and unreadable/invalid existing directories
fail. Use ConfigKit's selected-directory symlink behavior without custom rejection.
3. Call `LoadSecretsDir` at most once per command before any ConfigKit load or
adapter creation, including both loads of config diff and remote session lookup.
Use invocation state, not a package-global `sync.Once`. A new command gets its
own bootstrap; library config loaders themselves never mutate the environment.
4. Present safe bootstrap warnings/failures through stderr or the existing safe
diagnostic boundary, never into YAML show output. Preserve the partial report
on bootstrap failure. Do not add rollback or reinterpret ConfigKit file rules.
5. Remove late bootstrap calls in object-store and runner composition, the custom
directory reader and permission platform files, `pipeline.secrets`, its defaults
and validators, and their obsolete tests. Convert affected fixtures immediately.
Acceptance: test absent conventional directory, explicit override from another
CWD, invalid explicit directory, already-set empty/nonempty environment values,
and bootstrap failure before adapter construction. A diff invocation and a remote
session acquisition each bootstrap once. Config inspection requires no session,
workspace writes, manifests, or runtime clients. Update CLI, config, operations,
architecture, and troubleshooting to the new bootstrap/permission contract.
## Stage 7 — Resolve typed credentials and protect all output boundaries
Status: Pending
Read: ConfigKit “Typed secret references” and rendering contracts;
`internal/config/config.go`, `defaults.go`, `validate_adapters.go`, the safe
projection from Stage 3, `internal/adapters/storage/s3_backend.go`,
`internal/adapters/audita/subprocess.go`, and their composition callers.
1. Implement the three renamed optional typed fields and S3 pair policy above.
Reject old `_env` schema keys; remove their default-name and lookup helpers.
Convert maintained fixtures with synthetic environment setup where references
are present. Do not add credentials to unrelated local pipeline fixtures.
2. Pass resolved S3 values to the SDK adapter's explicit static provider only when
the pair is configured. With neither, preserve its default provider chain.
Pass Audita the resolved value and safe reference name; deliver the value only
through the child environment's existing `AUDITA_LLM_API_KEY` override.
Do not re-read the environment for an explicitly resolved reference.
3. Adapt every serialization, digest, source/diff output, diagnostic, generated
invocation config, manifest, and provenance caller to safe reference names.
Never YAML-marshal ConfigKit's opaque secret directly or call `Value()` in an
inspection path. Secret changes alone must not alter effective digests or
invalidate artifact evidence; reference-name changes may change provenance.
4. Keep ConfigKit errors structured and safe. Domain validation messages must
avoid raw credential values; do not unwrap private parser errors or interpolate
whole trusted configuration structs into error messages.
5. Preserve downstream tool ownership for Notarius/Scriptorium credentials that
Narratio does not configure. Do not add a parallel credential schema for them.
Acceptance: use distinctive synthetic secrets to prove no leakage in show, sources,
diff, catalog-ready projections, success/failure diagnostics, manifests, or generated
Audita configuration. Verify the child receives the credential and an environment
change after loading does not change explicit delivery. Test S3 paired references,
partial-pair rejection, missing effective references, and nil SDK fallback offline.
Update adapter and credential documentation and remove obsolete lookup tests.
## Stage 8 — Infer and normalize artifact dependencies once
Status: Pending
Read: `internal/config/validate_scriptorium.go`, `artifact_families.go`,
`internal/artifactpolicy`, `internal/artifacts`, analyze planning/execution, and
their artifact dependency and semantic-evidence tests.
1. At the final configuration resolution boundary, union explicit `depends_on`
with configured-artifact input references. Use the existing source-ID parser;
do not infer edges from filenames, arbitrary strings, or non-artifact inputs.
2. Infer after family member inputs become concrete `narratio.artifact.<key>`
sources. Normalize, deduplicate, and sort the effective dependency list once.
Keep explicit ordering-only dependencies valid. Repeated finalization must not
accumulate edges or alter ordering. Store source-edge attribution separately
from the effective graph for diagnostics and generated provenance.
3. Apply existing existence, self-reference, enabled-dependency, output, and cycle
validation to the union. Remove the error requiring an input to be repeated in
`depends_on`. Retain diagnostics pointing to the explicit list entry or the
input that contributed an invalid edge, including family origin where relevant.
4. Ensure execution, targeted analysis, planning, prerequisite closure, reuse,
forced regeneration, and semantic evidence consume this same concrete graph.
No adapter or command may infer a second graph. Refresh final projection/digest
after inference; equivalent explicit and inferred declarations compare equally.
Acceptance: cover input-only dependency ordering, duplicate explicit/inferred edges,
ordering-only edges, missing/self/cyclic references, and rewritten family inputs.
Use an assembled fake-artifact workflow to prove upstream output is supplied to a
dependent artifact and failure/resume retains successful predecessors. Existing
force/selection and invalidation tests remain valid. Update config and analyze
internals with the implemented inference rule.
## Stage 9 — Add the effective artifact catalog command
Status: Pending
Read: `internal/app/config_commands.go`, config diff/source rendering, the safe
projection and family-origin catalogs, and `docs/cli.md`.
1. Add `narratio config artifacts` to dispatch and help. Reuse ordinary pipeline,
profile, campaign, campaign-file, and secrets-directory parsing/resolution.
Resolve a campaign when families require it or when explicitly selected; a
pipeline with only ordinary artifacts can be inspected without a campaign.
2. Render one stable record per effective concrete artifact in key order with
key, `narratio.artifact.<key>` source ID, defining source location(s), enabled
state, output path, and sorted effective dependencies. Family members additionally
identify family key, character ID, defining family source, and party source.
3. Derive identity from the map key. A profile may contribute to an existing
artifact, so report contributing definition locations instead of inventing
a single filename owner. Use a deterministic escaped tabular text rendering;
include a header and no records for an empty catalog. Propagate writer errors.
4. Use the same finalized state as show/sources/diff and runtime configuration.
No session download, adapter initialization, manifest access, or workspace
materialization is allowed. Secret bootstrap remains the agreed startup exception.
Acceptance: CLI tests cover ordinary and generated artifacts, disabled entries,
profile-added/overridden artifacts, empty catalog, no-campaign ordinary inspection,
required-campaign failure for families, stable ordering, and no runtime side effects.
Check identity stays the same after moving an artifact file without changing its key.
Document the command in its canonical CLI owner and link to config layout examples.
## Stage 10 — Finish the operator layout and v2 migration documentation
Status: Pending
Read: roadmap “Artifact Definition Ergonomics” and “Compatibility and Migration”;
`examples/README.md`, existing example bundles and their loader test,
`docs/config.md`, `docs/operations.md`, `docs/release.md`, and internal configuration
and artifact documentation.
1. Convert maintained pipeline examples to a production base with explicit
`configkit.imports`, `artifacts/<key>.yml`, `artifact-families/<key>.yml`, and
`profiles/testing.yml`. Each recommended artifact/family file defines exactly
one explicit map entry. Fold old production overlays into the base.
2. Include an input-only dependency example and at least two character-family
examples sharing a campaign-owned party roster. Keep supported inline artifacts
legal; file layout is guidance, not a filename-based validation requirement.
3. Update the example validation harness to load root configurations and selected
testing profiles, not attempt to load each imported fragment as a full pipeline.
Use synthetic environment setup for explicit references. No example includes
real credentials or depends on local tools/cloud services to validate.
4. Consolidate canonical documentation updates from earlier stages. Remove current
references to old composition/profile catalogs, default-profile configuration,
secrets.env_dir, credential `_env` fields, credential-free inspection guarantees,
paired diff snapshots, and removed platform support. Explain explicit-value and
null changes without promising reloadable inspection output.
5. Add `docs/releases/v2.0.0.md` using the release-note structure, with a concrete
migration checklist linking to the canonical config/CLI docs and examples.
Include static S3-reference migration versus SDK credential discovery, new
inspection prerequisites, artifact input dependency inference, and platform
removals. Do not claim publication or edit older immutable release notes.
Acceptance: `go test ./internal/doccheck` and
`go test ./internal/config -run '^TestExamplesLoadAndValidate$'` pass. Walk through
the new operator examples with config validate/show/sources/diff/artifacts offline.
Their identities, family members, origins, and testing overrides agree.
## Stage 11 — Verify the complete migration and remove remaining scaffolding
Status: Pending
Read: the whole feature roadmap, completed stage notes, and the current test and
release procedures. Review the diff from the commit introducing this plan.
1. Audit every roadmap requirement against current code and executable evidence.
Remove temporary migration adapters and any residual generic YAML parser,
composer, profile traversal, explicit-presence store, old secrets loader,
decoder hooks, or unused tests. `gopkg.in/yaml.v3` may remain for output encoding
and unrelated external formats; it must not supply a second configuration loader.
2. Confirm result-owned snapshots versus final expanded configuration are handled
consistently; every inspection/runtime path sees normalized inferred dependencies
and generated publish rules. Recheck secret safety and semantic resume boundaries.
3. Inspect current platform conditions and asset lists. Retain historical notes but
no active Windows/macOS amd64 obligation. Compare owned production/test code
before and after; report removals and any remaining complexity with reasons.
Do not chase a LOC quota by removing useful domain checks or regressions.
4. Run the repository gates with the maintained Go 1.26 toolchain:
```sh
go test ./...
go test -race ./...
go vet ./...
go build ./...
go test ./internal/doccheck
go test ./internal/config -run '^TestExamplesLoadAndValidate$'
```
5. Run `scripts/check-release-candidate.sh v2.0.0` after the release note and all
candidate changes exist. This also verifies module hygiene and all three
release assets with `GOWORK=off`. Resolve failures; do not bypass checks or
publish a tag. Record native versus cross-build evidence accurately.
Acceptance: all gates pass and every roadmap outcome has direct evidence. No
dependency on the sibling ConfigKit checkout remains. Record completion and any
best-effort platform limitations, then hand off for a separate release decision.
## Open Questions
None required before implementation. The agreed roadmap and the explicit choices
above resolve the migration decisions. Upstream credential-free inspection and
native macOS arm64 CI remain deferred, not prerequisites for this plan.