narratio/docs/roadmap/implementation.md

ConfigKit Adoption Implementation Plan

Status: Pending

Implement the ConfigKit adoption roadmap in the order below. The target is Narratio v2.0.0: ConfigKit v0.1.1, the reduced platform matrix, and the artifact-authoring workflow described there. Each stage is a bounded implementation assignment for a gpt-5.6-terra coding agent. Mark a stage complete only after its changes and acceptance checks succeed; record a short completion and verification note beside its status.

Instructions for every stage

• Read docs/development.md and all three policies under docs/policy/. Read the relevant roadmap section and the stage-specific sources listed below. Paths identify starting points, not an exhaustive edit allowlist; inspect callers and tests before changing a package boundary.
• Use the tagged ConfigKit public API. The local ../ConfigKit checkout is useful reference material, but verify its version before relying on it. The module cache for v0.1.1 is an alternative. Do not require a sibling checkout, a local replace, a Go workspace, an upstream patch, or an unpinned revision to build.
• Keep Narratio domain policy in internal/config, startup and selectors in internal/app, and credential delivery in the owning adapters. Do not build a general configuration framework around ConfigKit.
• Keep each stage buildable and run focused behavioral tests for its changed boundary. Temporary migration helpers may exist between stages only where named here; they must not introduce runtime format auto-detection or fallback.
• Update each canonical current-behavior document when its behavior changes. The roadmap remains the owner of unimplemented work. Preserve historical release notes; their descriptions of old platforms and formats are history.
• Tests use real ConfigKit and temporary files, with fake external adapters or storage where needed. Use synthetic credentials and isolate environment mutation; such tests must not run in parallel. Retain domain and security coverage, remove superseded mechanism tests, and avoid exact diagnostic prose or private callback choreography assertions.
• Do not tag or publish v2.0.0 as part of this plan. Release preparation belongs here; release authorization and tag publication are a later task. The existing release completion boundary remains successful upstream tag push, with no CI polling or waiting.

Decisions that apply across stages

The roadmap is authoritative for product scope. These implementation choices make that scope concrete:

• Enable ConfigKit imports and profile selection for pipeline files only. Campaign, session, and party are single-source definitions with imports and profiles disabled. Preserve their domain schemas and source-relative inputs. Omitted or empty profile and the exact name default select the base; other names use ConfigKit's fixed profiles/<name>.yml convention.
• Defaults apply to absent fields. Explicit empty/zero/false/null values follow ConfigKit decoding and then domain validation; do not refill invalid explicit values with legacy defaults. Optional null stage sections remain absent; required sections fail domain validation rather than being resurrected.
• Keep ConfigKit metadata/result snapshots separate from runtime-only Narratio state. ConfigKit discards ignored and unexported fields when detaching values; attach root paths, retained party bytes, family catalogs, and digest metadata after loading. No validator may mutate bound configuration.
• Use one Narratio-owned safe effective projection for the final expanded pipeline. It contains configuration values and credential reference names, never resolved credential bytes. It feeds show, semantic diff, digest, and catalog data. ConfigKit metadata remains the authority for input provenance; Narratio adds only domain-generated attribution.
• Rename storage.s3.access_key_id_env to access_key_id, storage.s3.secret_access_key_env to secret_access_key, and audita.llm_api_key_env to llm_api_key. Each is an optional *configkit.Secret. An explicit reference always resolves during loading, even if the associated stage will not execute. Require both S3 fields together or neither. With neither, retain the AWS SDK default credential chain; do not synthesize mandatory secret references that would break role-based authentication or local-only inspection. With no Audita reference, preserve its existing downstream credential discovery. Explicit references replace Narratio's old per-adapter environment lookup.
• Keep absent S3 credentials out of defaults; migration examples explicitly name the existing AWS variables when static credentials are intended. A non-nil missing/empty reference fails instead of silently falling back. This preserves provider-managed credentials while making explicit configuration trustworthy.
• config artifacts uses the existing configuration command selectors and text output convention. No new JSON mode, artifact creation wizard, inferred filename identity, or profile catalog is added.
• Preserve the roadmap's unique base artifact definition rule explicitly: ConfigKit rejects conflicting leaves but can compose disjoint children of the same mapping. Use its public source metadata to reject an ordinary artifact key declared by multiple base files, even if their fields are disjoint. Apply the same rule to family keys. A selected profile may override or extend a definition. This is a narrow Narratio declaration rule, not a different merge algorithm.
• Relax only the obsolete configuration/secrets filesystem restrictions identified by the roadmap. Do not remove session restore, cleanup, publication, workspace confinement, or ordinary shared-workspace mode guarantees as collateral work.

Stage 1 — Reduce the platform and release matrix

Status: Pending

Read: roadmap “Platform Contract”; scripts/build-release-assets.sh, scripts/check-release-candidate.sh, .woodpecker/, internal/releasecheck, docs/release.md, and platform-specific files in application, file operations, and subprocess packages.

1. Make Linux amd64, Linux arm64, and macOS arm64 the only maintained build and release targets. Remove Windows suffix/packaging branches and macOS amd64 targets from scripts, workflows, tests, and current documentation.
2. Delete Windows-only implementations and tests; simplify paired build-tag files where the remaining implementations can be shared. Preserve portable code when retaining it adds no special support machinery. Preserve functioning Unix subprocess cancellation, locking, and file operations on both supported OSes.
3. Retain the local Linux verification gates and cross-build gates for all three published assets. Do not introduce a native macOS runner requirement or claim that cross-compilation verifies runtime behavior.
4. Update architecture/operations platform statements and contributor/release guidance. Keep existing release protections and the no-CI-wait boundary intact.

Acceptance: focused platform/subprocess/fileops/releasecheck tests pass; build all three targets with CGO_ENABLED=0; verify the asset builder produces exactly the three expected versioned binaries and the host binary reports the supplied version. Existing release tests still prove guarded publication. No current support promise or asset list retains Windows or macOS amd64.

Stage 2 — Separate configuration normalization from validation

Status: Pending

Read: internal/config/load.go, defaults.go, normalize.go, validate*.go, artifact_families.go, and ConfigKit's “Recorded application resolution”, “Semantic validation”, and “Defaults and typed decoding” contracts.

1. Separate value-producing normalization from read-only validation, including Notarius reference/output normalization currently performed by validateNotarius. Audit all validators for map, slice, pointer, and scalar mutations, including changes that would occur only on error paths.
2. Keep reusable domain normalization ordinary package code. Validation checks the already normalized candidate and must work for both loaded and programmatically constructed values without silently fixing them.
3. Inventory defaults in load.go: static defaults, per-map-entry/list-element defaults, and inherited or presence-sensitive defaults. Retain named constants in defaults.go. This inventory should guide code and concise comments, not become a permanent parallel defaults document.
4. Keep the current loader calling normalization explicitly before validation during this preparatory stage. The ConfigKit resolver replaces that loading choreography in Stage 4. Preserve family expansion as an explicit domain step.

Acceptance: focused configuration tests prove normalization is deterministic and validation does not mutate accepted or rejected inputs. Cover representative Notarius normalization/collision handling and existing artifact/publish rules. Do not add a separate mutation test for every trivial validator.

Stage 3 — Replace composition-dependent effective projection

Status: Pending

Read: internal/config/effective_pipeline.go, effective_digest.go, pipeline_composition.go, internal/app/config_commands.go, the config-diff implementation, and docs/internal/configuration.md.

1. Introduce one package-owned, schema-specific safe projection of the effective pipeline. Use explicit typed projections with stable configuration field names; preserve null versus empty containers and exact integers. Ordinary nested configuration types may be reused where they contain no runtime state. Credential fields must be projected explicitly as reference names when Stage 7 changes their types. Do not introduce a generic reflection schema engine.
2. Render this projection as deterministic YAML using the existing YAML encoder; encoding output is not a replacement YAML loader. Produce semantic records directly from the safe projection using standard JSON encoding/decoding with number preservation if a plain value tree is useful. Do not reparse YAML or reconstruct compositionNode trees to compare or hash effective values.
3. Make show, diff values, and digest use that common projection. Mapping order is immaterial; sequences remain atomic for diff; nil and empty remain distinct. Exclude family declarations once expanded, all runtime metadata, profile names, source locations, and raw credentials. Preserve logical Notarius path handling so moving an equivalent bundle does not change its digest.
4. Version the changed digest encoding explicitly using the package's digest version constant. Digest/profile metadata remains provenance, never a blanket resume cache key. Keep stage/artifact semantic evidence behavior intact.
5. Decouple enumeration of effective field paths from the legacy composition renderer. Input ownership can still come from the current loader until Stage 4.

Acceptance: show/diff/digest tests cover equivalent monolithic/split values, different map insertion order, null/empty distinction, family-expanded outputs, and bundle relocation. A meaningful artifact change changes the digest; formatting and profile naming alone do not. Existing semantic resume tests pass. No effective projection consumer needs the custom composition parser/renderer.

Stage 4 — Cut pipeline loading over to ConfigKit

Status: Pending

Read: ConfigKit v0.1.1 docs/config.md, docs/consumers/README.md, and its Narratio-shaped compiled example; Narratio internal/config/load.go, composition.go, pipeline_composition.go, and profile/composition tests.

1. Pin gitea.maximumdirect.net/eric/ConfigKit v0.1.1 in go.mod/go.sum. Implement a small pipeline definition using Load, File, enabled imports, Options.Profile, defaults, recorded resolution, and read-only validation.
2. Migrate static defaults to fresh default values. Apply map-entry, list-element, and inherited defaults with Resolution.Default/IsExplicit; use recorded normalization for transformations from Stage 2. Do not call old post-decode zero-filling defaults. Remove NormalizeConfig.UnmarshalYAML and its private presence bit. Include publication list-entry defaults, optional section allocation, and trim's Scriptorium requirement in this conversion.
3. Perform file-local semantic checks in the definition. Defer checks needing canonical party expansion or campaign/session context to the existing explicit resolution boundary; do not reject references to future concrete family members merely because the campaign has not loaded yet. Run full checks after expansion.
4. Replace pipeline composition and profile loaders. configkit.imports is the only directive; no composition or _configkit alias is accepted. Pass profile selection directly under the agreed convention; unselected overlays are not read. LoadPipelineProfilePair may remain a thin pair of independent loads, with no shared parse cache or paired source-snapshot promise.
5. Adapt ConfigKit sources and metadata to Narratio's existing reporting records. Translate paths structurally using public path APIs, not error-message parsing. Preserve multiple contributing locations for composite/profile-owned mappings and distinguish defaults/resolution from authored values. Check artifact/family map-entry ownership for duplicate base declarations using those locations; ignore synthetic defaults and allow the selected overlay.
6. Attach runtime metadata only after load succeeds. Preserve root-relative runtime path interpretation and capture logical paths before making them absolute. Retain family declarations and original ConfigKit metadata for later expansion.
7. Replace generic mechanism tests with a small integration set proving actual Narratio wiring. Convert fixtures affected by this stage. Delete composition.go, obsolete pipeline traversal/overlay/presence machinery, and their redundant tests once the callers are migrated.

Acceptance: a real root with imports and a testing overlay loads; duplicate claims fail with useful source attribution, including two base files contributing disjoint fields to one artifact or family key; base/default selection works; malformed unselected profiles do not fail base loading. Omitted defaults and representative explicit false/zero/empty/null inputs obey the new semantics. Invalid domain values still fail. Commands use the new pipeline loader with no format fallback. Update docs/config.md and configuration internals for the implemented boundary.

Stage 5 — Migrate campaign, session, and canonical party loading

Status: Pending

Read: internal/config/load.go, party.go, party_resolution.go, artifact_families.go, session-template tests, internal/app/config_loader.go, and ConfigKit source/schema/version contracts.

1. Use separate ConfigKit definitions for campaign, session, and party. Replace decodeStrictYAML* and the custom party YAML-node parser. Model the party input as the existing string schema version and character-keyed typed map, with the existing alias list, ordered class list, and optional positive class level.
2. Keep party domain checks: valid IDs, required display names and classes, trimmed/control-free names, case-insensitive name/alias ambiguity, and duplicate class checks. Validate the required version through VersionRule. Keep an exact independent copy of authored bytes outside ConfigKit's schema object.
3. Preserve authored character order using successful metadata source coordinates for the single party input when constructing CanonicalParty.Characters. Preserve class/alias sequence order naturally. Family expansion and players projection continue using their existing deterministic ID sorting. Do not keep a second YAML parse just to recover mapping order.
4. Preserve the narrow session placeholder rejection before typed load, expected session/previous-session identity checks, campaign selection, stable-input precedence, and local versus remotely downloaded session provenance. Retain the initially loaded pipeline/campaign during remote session acquisition.
5. Where bytes must be retained or inspected before ConfigKit.Bytes, bound the initial read too, using the existing fileops bounded-read helper and a named package constant of 8 MiB matching the pinned source limit. Otherwise prefer ConfigKit.File. Do not read an unbounded file and rely on later rejection.
6. Complete post-party family and publish-rule expansion, then full cross-config validation and digest refresh. Keep generated origins outside the library snapshot. Remove unused decoder plumbing and obsolete party parsing tests.

Acceptance: retained authored party bytes are identical; domain errors, aliases, class order, family origins, and derived players are correct. Local and remote session loads preserve identity, input path bases, and the original pipeline. Unknown fields and disabled directives fail through ConfigKit. Focused config, prepare/extract, and application loading tests pass.

Stage 6 — Move secrets bootstrap before all configuration loads

Status: Pending

Read: roadmap “Credential and Secrets Convention”; ConfigKit's bootstrap contract; internal/app/secrets_env*.go, config_loader.go, config_commands.go, object_store.go, command dependencies, and command argument owners.

1. Add a shared --secrets-dir selector to every CLI path that loads a pipeline, including configuration inspection/diff, execution aliases, planning, publish, and session/storage operations. Route all through one command-owned bootstrap context. Help/version and commands that do not load configuration do not bootstrap.
2. Discover the root pipeline first. An explicit relative directory is based on caller CWD; otherwise choose secrets/ beside the resolved root pipeline. An explicitly empty flag is invalid. Missing conventional directories are skipped; missing explicit directories and unreadable/invalid existing directories fail. Use ConfigKit's selected-directory symlink behavior without custom rejection.
3. Call LoadSecretsDir at most once per command before any ConfigKit load or adapter creation, including both loads of config diff and remote session lookup. Use invocation state, not a package-global sync.Once. A new command gets its own bootstrap; library config loaders themselves never mutate the environment.
4. Present safe bootstrap warnings/failures through stderr or the existing safe diagnostic boundary, never into YAML show output. Preserve the partial report on bootstrap failure. Do not add rollback or reinterpret ConfigKit file rules.
5. Remove late bootstrap calls in object-store and runner composition, the custom directory reader and permission platform files, pipeline.secrets, its defaults and validators, and their obsolete tests. Convert affected fixtures immediately.

Acceptance: test absent conventional directory, explicit override from another CWD, invalid explicit directory, already-set empty/nonempty environment values, and bootstrap failure before adapter construction. A diff invocation and a remote session acquisition each bootstrap once. Config inspection requires no session, workspace writes, manifests, or runtime clients. Update CLI, config, operations, architecture, and troubleshooting to the new bootstrap/permission contract.

Stage 7 — Resolve typed credentials and protect all output boundaries

Status: Pending

Read: ConfigKit “Typed secret references” and rendering contracts; internal/config/config.go, defaults.go, validate_adapters.go, the safe projection from Stage 3, internal/adapters/storage/s3_backend.go, internal/adapters/audita/subprocess.go, and their composition callers.

1. Implement the three renamed optional typed fields and S3 pair policy above. Reject old _env schema keys; remove their default-name and lookup helpers. Convert maintained fixtures with synthetic environment setup where references are present. Do not add credentials to unrelated local pipeline fixtures.
2. Pass resolved S3 values to the SDK adapter's explicit static provider only when the pair is configured. With neither, preserve its default provider chain. Pass Audita the resolved value and safe reference name; deliver the value only through the child environment's existing AUDITA_LLM_API_KEY override. Do not re-read the environment for an explicitly resolved reference.
3. Adapt every serialization, digest, source/diff output, diagnostic, generated invocation config, manifest, and provenance caller to safe reference names. Never YAML-marshal ConfigKit's opaque secret directly or call Value() in an inspection path. Secret changes alone must not alter effective digests or invalidate artifact evidence; reference-name changes may change provenance.
4. Keep ConfigKit errors structured and safe. Domain validation messages must avoid raw credential values; do not unwrap private parser errors or interpolate whole trusted configuration structs into error messages.
5. Preserve downstream tool ownership for Notarius/Scriptorium credentials that Narratio does not configure. Do not add a parallel credential schema for them.

Acceptance: use distinctive synthetic secrets to prove no leakage in show, sources, diff, catalog-ready projections, success/failure diagnostics, manifests, or generated Audita configuration. Verify the child receives the credential and an environment change after loading does not change explicit delivery. Test S3 paired references, partial-pair rejection, missing effective references, and nil SDK fallback offline. Update adapter and credential documentation and remove obsolete lookup tests.

Stage 8 — Infer and normalize artifact dependencies once

Status: Pending

Read: internal/config/validate_scriptorium.go, artifact_families.go, internal/artifactpolicy, internal/artifacts, analyze planning/execution, and their artifact dependency and semantic-evidence tests.

1. At the final configuration resolution boundary, union explicit depends_on with configured-artifact input references. Use the existing source-ID parser; do not infer edges from filenames, arbitrary strings, or non-artifact inputs.
2. Infer after family member inputs become concrete narratio.artifact.<key> sources. Normalize, deduplicate, and sort the effective dependency list once. Keep explicit ordering-only dependencies valid. Repeated finalization must not accumulate edges or alter ordering. Store source-edge attribution separately from the effective graph for diagnostics and generated provenance.
3. Apply existing existence, self-reference, enabled-dependency, output, and cycle validation to the union. Remove the error requiring an input to be repeated in depends_on. Retain diagnostics pointing to the explicit list entry or the input that contributed an invalid edge, including family origin where relevant.
4. Ensure execution, targeted analysis, planning, prerequisite closure, reuse, forced regeneration, and semantic evidence consume this same concrete graph. No adapter or command may infer a second graph. Refresh final projection/digest after inference; equivalent explicit and inferred declarations compare equally.

Acceptance: cover input-only dependency ordering, duplicate explicit/inferred edges, ordering-only edges, missing/self/cyclic references, and rewritten family inputs. Use an assembled fake-artifact workflow to prove upstream output is supplied to a dependent artifact and failure/resume retains successful predecessors. Existing force/selection and invalidation tests remain valid. Update config and analyze internals with the implemented inference rule.

Stage 9 — Add the effective artifact catalog command

Status: Pending

Read: internal/app/config_commands.go, config diff/source rendering, the safe projection and family-origin catalogs, and docs/cli.md.

1. Add narratio config artifacts to dispatch and help. Reuse ordinary pipeline, profile, campaign, campaign-file, and secrets-directory parsing/resolution. Resolve a campaign when families require it or when explicitly selected; a pipeline with only ordinary artifacts can be inspected without a campaign.
2. Render one stable record per effective concrete artifact in key order with key, narratio.artifact.<key> source ID, defining source location(s), enabled state, output path, and sorted effective dependencies. Family members additionally identify family key, character ID, defining family source, and party source.
3. Derive identity from the map key. A profile may contribute to an existing artifact, so report contributing definition locations instead of inventing a single filename owner. Use a deterministic escaped tabular text rendering; include a header and no records for an empty catalog. Propagate writer errors.
4. Use the same finalized state as show/sources/diff and runtime configuration. No session download, adapter initialization, manifest access, or workspace materialization is allowed. Secret bootstrap remains the agreed startup exception.

Acceptance: CLI tests cover ordinary and generated artifacts, disabled entries, profile-added/overridden artifacts, empty catalog, no-campaign ordinary inspection, required-campaign failure for families, stable ordering, and no runtime side effects. Check identity stays the same after moving an artifact file without changing its key. Document the command in its canonical CLI owner and link to config layout examples.

Stage 10 — Finish the operator layout and v2 migration documentation

Status: Pending

Read: roadmap “Artifact Definition Ergonomics” and “Compatibility and Migration”; examples/README.md, existing example bundles and their loader test, docs/config.md, docs/operations.md, docs/release.md, and internal configuration and artifact documentation.

1. Convert maintained pipeline examples to a production base with explicit configkit.imports, artifacts/<key>.yml, artifact-families/<key>.yml, and profiles/testing.yml. Each recommended artifact/family file defines exactly one explicit map entry. Fold old production overlays into the base.
2. Include an input-only dependency example and at least two character-family examples sharing a campaign-owned party roster. Keep supported inline artifacts legal; file layout is guidance, not a filename-based validation requirement.
3. Update the example validation harness to load root configurations and selected testing profiles, not attempt to load each imported fragment as a full pipeline. Use synthetic environment setup for explicit references. No example includes real credentials or depends on local tools/cloud services to validate.
4. Consolidate canonical documentation updates from earlier stages. Remove current references to old composition/profile catalogs, default-profile configuration, secrets.env_dir, credential _env fields, credential-free inspection guarantees, paired diff snapshots, and removed platform support. Explain explicit-value and null changes without promising reloadable inspection output.
5. Add docs/releases/v2.0.0.md using the release-note structure, with a concrete migration checklist linking to the canonical config/CLI docs and examples. Include static S3-reference migration versus SDK credential discovery, new inspection prerequisites, artifact input dependency inference, and platform removals. Do not claim publication or edit older immutable release notes.

Acceptance: go test ./internal/doccheck and go test ./internal/config -run '^TestExamplesLoadAndValidate$' pass. Walk through the new operator examples with config validate/show/sources/diff/artifacts offline. Their identities, family members, origins, and testing overrides agree.

Stage 11 — Verify the complete migration and remove remaining scaffolding

Status: Pending

Read: the whole feature roadmap, completed stage notes, and the current test and release procedures. Review the diff from the commit introducing this plan.

1. Audit every roadmap requirement against current code and executable evidence. Remove temporary migration adapters and any residual generic YAML parser, composer, profile traversal, explicit-presence store, old secrets loader, decoder hooks, or unused tests. gopkg.in/yaml.v3 may remain for output encoding and unrelated external formats; it must not supply a second configuration loader.

2. Confirm result-owned snapshots versus final expanded configuration are handled consistently; every inspection/runtime path sees normalized inferred dependencies and generated publish rules. Recheck secret safety and semantic resume boundaries.

3. Inspect current platform conditions and asset lists. Retain historical notes but no active Windows/macOS amd64 obligation. Compare owned production/test code before and after; report removals and any remaining complexity with reasons. Do not chase a LOC quota by removing useful domain checks or regressions.

4. Run the repository gates with the maintained Go 1.26 toolchain:

   go test ./...
   go test -race ./...
   go vet ./...
   go build ./...
   go test ./internal/doccheck
   go test ./internal/config -run '^TestExamplesLoadAndValidate$'
   

5. Run scripts/check-release-candidate.sh v2.0.0 after the release note and all candidate changes exist. This also verifies module hygiene and all three release assets with GOWORK=off. Resolve failures; do not bypass checks or publish a tag. Record native versus cross-build evidence accurately.

Acceptance: all gates pass and every roadmap outcome has direct evidence. No dependency on the sibling ConfigKit checkout remains. Record completion and any best-effort platform limitations, then hand off for a separate release decision.

Open Questions

None required before implementation. The agreed roadmap and the explicit choices above resolve the migration decisions. Upstream credential-free inspection and native macOS arm64 CI remain deferred, not prerequisites for this plan.