diff --git a/docs/roadmap/configkit-adoption.md b/docs/roadmap/configkit-adoption.md deleted file mode 100644 index ab3fb2d..0000000 --- a/docs/roadmap/configkit-adoption.md +++ /dev/null @@ -1,375 +0,0 @@ -# ConfigKit Adoption and Platform Simplification - -**Status:** Accepted; implementation pending - -## Purpose - -Narratio currently owns a substantial YAML configuration subsystem for strict -decoding, explicit imports, additive composition, profile overlays, presence -tracking, provenance, and effective-value inspection. ConfigKit was extracted -from this implementation and now provides those generic responsibilities as a -maintained library. - -Narratio 2.0 will adopt ConfigKit as its configuration foundation, remove the -superseded generic infrastructure, and retain only Narratio-specific configuration -policy and domain resolution. The same major release will narrow the supported -platform set to the systems that Narratio is expected to run on in practice. - -This roadmap defines the desired contracts and end state. The -[implementation plan](implementation.md) owns sequencing and completion gates. - -## Goals - -- Reduce Narratio-owned configuration code and tests by delegating generic YAML - mechanics to ConfigKit. -- Establish one strict and explainable configuration lifecycle rather than - retaining parallel or compatibility loaders. -- Preserve Narratio's application-specific defaults, validation, campaign/session - resolution, party handling, artifact-family expansion, digests, and CLI policy. -- Make individual artifact definitions easy to organize, identify, inspect, and - reuse as dependencies of later artifacts. -- Keep raw credentials out of YAML and adopt a predictable secrets-directory - convention that works before configuration loading. -- Remove unsupported platform code and release obligations that provide no known - user value. -- Deliver the resulting contract changes as Narratio v2.0.0 with a clear migration - guide. - -## Platform Contract - -Narratio's v2 platform matrix is: - -| Platform | Status | Required evidence and deliverables | -| --- | --- | --- | -| Linux amd64 | Primary, supported | Default tests, race checks, static analysis, build validation, and a release asset | -| Linux arm64 | Primary, supported | Cross-build validation and a release asset; native validation is used when an appropriate runner is available | -| macOS arm64 | Best effort | Cross-build validation and a release asset; native CI and platform-equivalent runtime guarantees are not required release gates | -| macOS amd64 | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance | -| Windows | Unsupported | No build gate, release asset, compatibility promise, or platform-specific maintenance | - -Windows- and Intel-Mac-specific code, tests, CI entries, release packaging, and -documentation are removed when they no longer serve a supported build. Portable -code should remain portable where doing so is natural, but accidental compilation -on an unsupported target does not create a support commitment. - -Best-effort macOS arm64 support means Narratio intends to keep the target buildable -and will consider reported defects when practical. Lack of a native runner or an -unverified platform-specific behavior does not block a release. A target for which -Narratio publishes an asset must still pass its configured cross-build gate. - -## ConfigKit Dependency and Ownership - -Narratio will depend on the tagged ConfigKit module at an exact reviewed version, -initially `gitea.maximumdirect.net/eric/ConfigKit@v0.1.1`. ConfigKit is pre-1.0 and -may make compatibility changes in later minor versions, so dependency upgrades are -intentional changes accompanied by contract review and focused Narratio integration -tests; Narratio does not follow an unpinned branch or automatically accept a newer -minor release. - -ConfigKit owns: - -- bounded file, byte, and reader input; -- the accepted YAML syntax and strict typed decoding; -- import declarations, traversal, containment, and resource limits; -- additive composition and conflict attribution; -- selected-profile overlay semantics; -- explicit input presence and source provenance; -- application-default and recorded-resolution coordination; -- structured configuration diagnostics; -- secret-reference parsing and resolution; -- secrets-directory loading and its safe report; and -- detached effective configuration inspection supplied by its public result. - -Narratio owns: - -- pipeline, campaign, session, party, and runtime domain types; -- configuration-file discovery and CLI selection; -- concrete defaults, normalization, and semantic validation; -- campaign/session identity and stable-input resolution; -- canonical party parsing requirements and authored-byte retention; -- character artifact-family expansion and generated ownership attribution; -- root-relative interpretation of Narratio-owned runtime paths; -- effective pipeline digests and semantic profile diffs; -- command output and exit behavior; and -- adapter and runtime initialization after configuration succeeds. - -Narratio must use ConfigKit through its public package. It must not import -ConfigKit's private YAML implementation or reproduce ConfigKit's parser, -composition, profile, provenance, or secret-bootstrap mechanics locally. - -## Configuration Format - -Narratio adopts ConfigKit's configuration conventions directly. The old Narratio -`composition:` envelope, declared profile catalog, arbitrary overlay paths, and -`composition.default_profile` are removed without a compatibility parser. - -The root pipeline uses `configkit.imports` declarations. Import paths, optional -imports, globs, recursion, ordering, composition conflicts, containment, and limits -have the meanings defined by the pinned ConfigKit contract. Narratio does not add a -second import syntax or narrower application-specific traversal implementation. - -The unoverlaid root and its imports are the production configuration. A named -profile is a ConfigKit overlay at `profiles/.yml`; in particular, -`profiles/testing.yml` contains inexpensive-model overrides and testing-only -artifacts. Omitting `--profile`, or selecting ConfigKit's `default` profile, uses the -production base. `--profile testing` applies the testing overlay. Narratio does not -maintain a separate configured default-profile mechanism or validate unselected -profile files. - -ConfigKit's rules for explicit false, zero, empty, and null values replace -Narratio's custom presence tracking. Application defaults and normalization use a -ConfigKit definition and recorded resolution where the final value depends on -whether input was present. `NormalizeConfig.UnmarshalYAML` and equivalent -parser-coupled application hooks are removed. - -Pipeline, campaign, session, and party inputs use the common ConfigKit loading -boundary where its schema model can represent their types. Narratio may perform -narrow pre- or post-load domain work, such as retaining authored party bytes, -rejecting unresolved session-template placeholders, resolving stable inputs, and -expanding artifact families. Such work must not recreate a second generic YAML -loader. - -## Artifact Definition Ergonomics - -The canonical v2 configuration layout defines each ordinary Scriptorium artifact -in its own file under `artifacts/`. The root pipeline imports the directory through -ConfigKit, for example: - -```yaml -configkit: - imports: - - glob: artifacts/*.yml -``` - -Each file contributes one ordinary entry at -`scriptorium.artifacts.`. The map key is the authoritative Narratio -artifact key and therefore makes the artifact's internal identity explicit in the -definition itself: - -```yaml -# artifacts/session_recap.yml -scriptorium: - artifacts: - session_recap: - enabled: true - prompt_id: dnd.session_recap - output_path: artifacts/session_recap.md -``` - -The canonical reusable source ID is derived without another configured identifier: -`session_recap` becomes `narratio.artifact.session_recap`. The recommended filename -is `.yml`, but the filename is organizational rather than an identity -source. Renaming or moving a file without changing its map key does not change the -artifact contract. ConfigKit additive composition rejects duplicate artifact keys. - -The one-artifact-per-file layout is the documented happy path and is used by -maintained examples. Narratio does not prohibit an inline artifact or another -ConfigKit-compatible source organization merely because it differs from that -layout. Artifact identity always comes from the effective map key, never implicitly -from a filename or import position. - -Artifact families use the corresponding `artifact-families/*.yml` convention, with -one `scriptorium.artifact_families.` entry per file. The family key is -explicit. Concrete member keys and source IDs continue to use Narratio's existing -deterministic family/member naming rule after canonical party resolution. - -Narratio adds a read-only `narratio config artifacts` command using the ordinary -pipeline, profile, and campaign selectors. It reports every effective ordinary -artifact's key, canonical `narratio.artifact.` source ID, defining source, -enabled state, output path, and effective dependencies. Generated family members -also report their family origin. Output ordering is deterministic by artifact key, -and the command does not require a session or initialize runtime adapters. - -A configured-artifact input creates an implicit dependency on the referenced -artifact. For example: - -```yaml -inputs: - recap: - source: narratio.artifact.session_recap -``` - -The effective dependency graph is the union of explicit `depends_on` entries and -dependencies inferred from `narratio.artifact.` inputs. Inferred and explicit -duplicates collapse to one dependency. `depends_on` remains available for an -ordering requirement when the upstream output is not supplied as an input. -Validation applies the same existence, self-reference, and cycle rules to both -kinds of dependency, and diagnostics identify the declaration or input that -introduced an invalid edge. Operators no longer need to repeat an artifact input in -`depends_on` merely to make that input valid. - -## Effective Inspection, Digests, and Profile Diff - -ConfigKit results are the authority for composed input presence, base/profile -origins, diagnostics, and the initial typed effective configuration. Narratio's -configuration commands adapt those results to existing application-facing command -formats rather than exposing ConfigKit implementation types directly. - -Party-dependent artifact families are expanded only after the selected campaign and -canonical party are known. That derived state is intentionally outside ConfigKit's -completed load snapshot. Narratio retains a narrow application-owned layer that: - -- records generated family/member and publish-rule attribution; -- projects the final post-expansion pipeline for operator inspection; -- recomputes the effective pipeline digest after expansion; and -- ensures stages and adapters receive only the final concrete configuration. - -Narratio continues to own semantic `config diff` behavior because ConfigKit does -not provide configuration diffs or fingerprints. Each side of a profile comparison -may be loaded independently through ConfigKit. The former guarantee that both sides -come from one parsed source snapshot is removed; operators must not rely on a -meaningful comparison while concurrently modifying its source files. Equality is -still based on normalized effective values rather than YAML formatting or source -layout. - -## Credential and Secrets Convention - -Raw API keys and other credential values remain prohibited in pipeline, campaign, -session, generated configuration, manifests, diagnostics, examples, and logs. -Credential-bearing configuration fields use ConfigKit secret references: YAML -contains an environment-variable name, while the resolved value remains available -only through ConfigKit's trusted typed boundary. Fields should use credential names -rather than legacy names ending in `_env` when the v2 schema is updated. - -The `pipeline.secrets.env_dir` stanza is removed. Narratio selects a secrets -directory before parsing the pipeline using this precedence: - -1. an explicitly supplied `--secrets-dir PATH`; otherwise -2. the `secrets` subdirectory beside the selected root pipeline file. - -The conventional directory is derived from the resolved root pipeline path, never -from the current working directory, an imported file, or a selected profile. A -relative explicit CLI path is resolved from the caller's current working directory, -consistent with an ordinary command-line path argument. - -If an explicitly selected directory is missing or invalid, the command fails with a -clear bootstrap diagnostic. If the conventional directory is absent, Narratio -performs no directory bootstrap and continues; pre-existing environment variables -may satisfy all references. If the conventional directory exists but ConfigKit -rejects or cannot read it, the command fails. Already-set environment variables, -including empty values, retain ConfigKit's documented precedence. - -Narratio explicitly calls `configkit.LoadSecretsDir` at most once for a command, -before the first ConfigKit load that may resolve secret references and before any -concurrent work or runtime client initialization. This application-level convention -does not make ConfigKit's ordinary `Load` mutate process-global state. The bootstrap -report is rendered or summarized through a safe Narratio command boundary without -secret values. - -Narratio adopts ConfigKit's secret-directory rules and limits without restoring its -old permission-mode, selected-directory-symlink, invalid-entry, byte-limit, or -newline-trimming behavior through compatibility wrappers. Configuration inspection -may require the referenced environment variables or the selected secrets directory -because an ordinary successful ConfigKit load resolves effective typed secret -references. Inspection output may show non-secret reference names where appropriate, -but must never expose resolved values. - -## Validation and Diagnostics - -Narratio's semantic rules remain recognizable application code registered with the -appropriate ConfigKit definition. Validation continues to reject invalid pipeline, -campaign, session, party, adapter, publish, and artifact relationships before -runtime work where practical. Validation order and exact prose are not compatibility -contracts. - -ConfigKit diagnostic codes, paths, and source locations should be preserved through -the application error boundary where useful. Narratio adds command context and exit -classification but does not parse ConfigKit error strings or expose raw parser -errors. Secret values must remain absent from success and failure diagnostics. - -## Compatibility and Migration - -This work intentionally changes supported platforms, pipeline composition syntax, -profile layout and selection, secret configuration, YAML semantics, diagnostics, -and some inspection prerequisites. It is released as Narratio v2.0.0. - -There is no runtime compatibility layer for the old `composition` or -`pipeline.secrets` syntax. Maintained examples are converted to the ConfigKit -format, and the v2 release documentation provides a concise migration guide that -covers: - -- converting imports to `configkit.imports` declarations; -- folding the production overlay into the base configuration; -- placing testing overrides in `profiles/testing.yml`; -- moving ordinary artifacts and artifact families into the recommended individual - definition files and using their map keys as canonical identities; -- removing redundant `depends_on` entries when a configured-artifact input now - supplies the same dependency; -- moving secret files to the conventional sibling `secrets/` directory or using - `--secrets-dir`; -- renaming credential-reference fields where applicable; -- ConfigKit's stricter YAML and explicit-value semantics; -- the independent-load behavior of `config diff`; and -- the removal of Windows and macOS amd64 artifacts. - -Obsolete application implementation, tests, documentation, examples, CI paths, and -release machinery are removed rather than left deprecated inside the v2 codebase. - -## Required Changes - -- Add and pin the ConfigKit module dependency and establish a small application - integration boundary for definitions, result adaptation, and diagnostics. -- Replace Narratio's generic YAML parsing, composition, import, profile, presence, - and provenance implementation with ConfigKit public APIs. -- Express existing defaults, presence-sensitive normalization, and semantic - validation through ConfigKit definitions without moving Narratio domain policy - into the library. -- Adapt pipeline/campaign/session/party loading and preserve the existing - application resolution steps that occur after their independent typed loads. -- Rework effective configuration, sources, and diff commands around ConfigKit - results plus Narratio's post-party generated-attribution layer. -- Add the maintained per-artifact and per-family file layouts, the read-only - `config artifacts` catalog, and effective dependency inference from configured - artifact inputs. -- Convert credential configuration to typed references, add the common - `--secrets-dir` option, implement conventional-directory selection, and replace - the custom API-key directory loader with `configkit.LoadSecretsDir`. -- Update adapters to consume resolved credentials or safe environment reference - names from typed configuration without independently rediscovering secret values. -- Remove Windows and macOS amd64 build targets, release assets, CI entries, - platform-specific implementation, and obsolete tests and documentation. -- Convert maintained examples and add v2 migration and release documentation. -- Retain focused Narratio tests for domain policy and assembled application - behavior while deleting parser/composer/bootstrap mechanism tests now owned by - ConfigKit. Tests remain deterministic, offline, credential-free, and aligned with - the repository testing policy. -- Update canonical architecture, configuration, CLI, operations, development, - internal, troubleshooting, and release documentation only as the corresponding - behavior is implemented. - -## Target End State - -Narratio v2.0.0 has one configuration infrastructure dependency and no duplicate -YAML composition subsystem. Operators maintain a production base configuration, -optional explicit imports, and fixed named overlays such as `testing`. ConfigKit -provides strict parsing, composition, typed loading, provenance, diagnostics, and -credential resolution; Narratio supplies domain meaning and its post-campaign -artifact expansion. - -Operators can add an artifact by creating one discoverable YAML file whose map key -is its stable Narratio identity. `config artifacts` exposes that key, its canonical -source ID, origin, output, and effective dependency set. A later artifact can reuse -the output through `narratio.artifact.` without repeating the same edge in -`depends_on`, while explicit ordering-only dependencies remain supported. - -Secrets normally live in a predictable `secrets/` directory beside the selected -pipeline, with `--secrets-dir` available for deployment-specific placement. Raw -credential values never enter YAML or inspection output. The bootstrap is explicit -in application startup and ordinary ConfigKit loading remains free of process-global -mutation. - -Release and CI maintenance covers Linux amd64 and arm64 as the primary platforms -and macOS arm64 on a best-effort basis. Windows and macOS amd64 carry no remaining -support or release burden. The repository contains no legacy configuration parser, -profile compatibility layer, obsolete platform implementation, or redundant tests -for behavior owned by ConfigKit. - -## Deferred Work - -- A ConfigKit reference-only inspection or typed prepare/finalize lifecycle may be - considered upstream if credential-free inspection becomes a demonstrated shared - requirement. Narratio does not implement a private substitute in this migration. -- ConfigKit hot reload, remote providers, automatic configuration migration, and - application command frameworks remain outside Narratio's scope. -- Native macOS arm64 CI may be added later when a reliable runner is available; it - is not required for v2.0.0. diff --git a/docs/roadmap/configkit-v0.3.0.md b/docs/roadmap/configkit-v0.3.0.md deleted file mode 100644 index 11797f1..0000000 --- a/docs/roadmap/configkit-v0.3.0.md +++ /dev/null @@ -1,149 +0,0 @@ -# Adopt ConfigKit v0.3.0 - -Status: Implemented — 2026-09-16 - -Execution order and stage acceptance checks belong to the -[implementation plan](implementation.md). This roadmap owns scope, policy, and -the target end state. - -## Goal - -Upgrade Narratio from ConfigKit v0.1.2 to the released v0.3.0 while preserving -Narratio's configuration contracts. Keep this a bounded dependency-maintenance -change, not a new configuration feature or a second configuration refactor. - -The upgrade also includes correcting the test-isolation problems discovered -during evaluation so that validation does not depend on whether the developer -has installed Narratio configuration on the host. - -## Background and Evaluation - -ConfigKit v0.3.0 adds callback-scoped `Resolution.InputInfo` and -`Resolution.Source` queries and separates its public API from its private engine. -The public imports, signatures, secret handling, and existing metadata APIs, -including `Metadata.HistoricalPaths`, remain supported. Its minimum Go version -remains 1.25; Narratio's maintained Go 1.26 baseline needs no change. The private -YAML dependency remains `go.yaml.in/yaml/v4 v4.0.0-rc.6`. - -The intervening v0.2.0 release allows decimal-integer schema-version input to -reach an application's version checker as text. It also rejects signed sequence -index tokens such as `+0` and `-0`; literal mapping keys are unaffected. -Narratio constructs sequence paths using ConfigKit's index API and does not need -a user-facing path migration. - -ConfigKit now treats macOS as best-effort. This does not require changing -Narratio's Linux amd64/arm64 support or its own best-effort macOS arm64 -cross-build gate. - -The 2026-09-16 evaluation used the released dependency through a temporary module -file, without changing Narratio's tracked files: - -- Build, vet, and Linux arm64/macOS arm64 cross-builds passed. -- The ordinary suite exposed one upgrade-related test expectation mismatch: - integer party schema versions still fail, but with an application-level - unsupported-version diagnostic instead of `version.type`. -- Configuration race tests passed with that expectation adjusted in a temporary - test overlay. -- Three CLI cases failed on both v0.1.2 and v0.3.0 because they assumed no default - pipeline configuration was installed. These are pre-existing isolation gaps, - not demonstrated ConfigKit regressions. Other packages passed the ordinary - suite; the overall suite was not green. - -Upstream references are the tagged `docs/releases/v0.2.0.md` and -`docs/releases/v0.3.0.md` in the ConfigKit repository. Narratio's current owners -remain the [configuration reference](../config.md), -[configuration internals](../internal/configuration.md), and -[architecture policy](../policy/architecture.md). - -## Required Changes - -### Released dependency - -Pin `gitea.maximumdirect.net/eric/ConfigKit v0.3.0` in `go.mod`, refresh -`go.sum`, and retain a tidy module graph. Do not introduce local replacements, -a sibling-checkout requirement, vendoring, or workspace-only dependency wiring. -Production code must use only ConfigKit's public root package. - -### Party-version regression coverage - -Update `TestParsePartyCanonicalRejectsInvalidDocuments` in -`internal/config/party_test.go` so that integer `schema_version: 1` is tested as -an unsupported version rather than a ConfigKit type error. Preserve the required -`narratio.party.v1` identifier: the upgrade must not make integer party versions -valid or introduce coercion for ordinary string fields. - -Keep assertions focused on rejection and the unsupported-version meaning, not -exact diagnostic prose. Retain meaningful coverage for malformed version input -and strict ordinary-field decoding without duplicating ConfigKit's parser suite. - -### Host-independent CLI tests - -Remove installed-configuration assumptions from these failing cases: - -- `TestExecuteAnalyzeMissingConfigUsesRunStageLoadingPath` and - `TestExecutePublishMissingConfigUsesRunStageLoadingPath` in - `internal/app/analyze_artifacts_commands_test.go`; -- the `run missing config uses defaults` case of - `TestExecuteMissingRequiredFlags` in `internal/app/commands_test.go`. - -Use the existing command dependency/discovery boundaries where possible, or a -narrow invocation-local seam if needed, to exercise missing-config behavior -against controlled temporary locations. Preserve coverage that these commands -route through ordinary configuration discovery. Inspect adjacent missing-config -tests for the same assumption and repair affected cases within this boundary. - -Tests must neither read operator credentials/configuration nor create, rename, -delete, or require absence of files in real system configuration locations. -Avoid process-global search-path mutation and do not change production search -order merely to satisfy tests. Follow the -[testing policy](../policy/testing.md) when selecting the narrowest durable seam. - -### Documentation - -Update current documentation only where adoption changes an actual claim. -Record the dependency upgrade and diagnostic adjustment in the applicable -unreleased release note; do not rewrite already-published historical notes. -No operator configuration migration is expected. If no unreleased note exists, -retain the summary in the implementation completion record until a separate -release task selects a version; published notes remain immutable. - -## Preserved Behavior and Non-Goals - -- Pipeline paths remain relative to the root pipeline file, not the importing - file that declares a value. Do not adopt the new resolver provenance APIs - merely because they are available. -- Imports, profile selection, explicit-null/default semantics, unique base - artifact ownership, party expansion, dependency inference, and safe - show/diff/source/catalog projections retain their existing behavior. -- Resolve only effective credential references. Profiles can replace or remove - unused base credentials, and hidden duplicate definitions remain rejected - through historical metadata without independently loading the base. -- Secret bootstrap, redaction, runtime adapter delivery, and resume evidence - remain unchanged. The separate - [Notarius credential-forwarding proposal](notarius-credential-forwarding.md) - is not implemented by this upgrade. -- No new schema versions, CLI flags, path policy, supported platforms, Go - baseline change, or upstream modifications are required. -- No performance improvement is claimed. The primary benefit is maintaining - compatibility with the current upstream library and its explicit public API. - -## Validation and Target End State - -The released dependency works without consumer production-code migration except -any narrowly justified testability wiring. Existing valid configurations keep -their meaning, and integer party versions remain invalid with a truthful -unsupported-version diagnostic. Missing-config tests pass regardless of host -installation state. - -Acceptance requires the full validation set in the -[development guide](../development.md): ordinary and race tests, vet, build, -documentation links, and maintained examples. Verify module tidiness and all -three Narratio cross-build targets. Recheck profile credential replacement/null -removal, hidden duplicate ownership, provenance, and credential-safe inspection -through existing regression coverage; add tests only for uncovered meaningful -risks. Validation must not rely on skipped failures or temporary test overlays. - -Before publication, run the normal -[release-candidate procedure](../release.md). Cross-compilation is not native -macOS runtime evidence. Tagging, pushing, and release publication are separate -actions, not part of implementing this roadmap. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index b1a5af5..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,317 +0,0 @@ -# Notarius Credential Forwarding Implementation Plan - -Status: Complete — 2026-09-16 - -Release-ready summary: Narratio now accepts an optional typed -`notarius.llm_api_key` environment reference, resolves it through the existing -command bootstrap, forwards only that selected credential to Notarius, and -redacts its value from captured subprocess streams. Existing configurations -require no migration, and credential-only changes do not invalidate completed -extraction work. Version selection and release publication remain separate. - -Implement the [feature roadmap](notarius-credential-forwarding.md) in the order -below. Each stage is a bounded assignment for a gpt-5.6-terra coding agent. -Record a short completion note and executed checks beside each stage's status; -do not mark it complete until its acceptance criteria pass. The completed -ConfigKit upgrade plan is preserved in Git history. - -## Instructions and fixed decisions - -- Read `docs/development.md`, all documents under `docs/policy/`, the feature - roadmap, and each stage's listed sources. Inspect callers and existing tests - before editing; file lists are starting points, not edit allowlists. -- Implement one optional `notarius.llm_api_key` field. It contains an environment - variable name, not a value. Omission and null mean no credential forwarding. - Every surviving non-null reference resolves during ordinary configuration - loading, even when extract is disabled, omitted from the range, or resumable. -- Reuse ConfigKit v0.3.0 and invocation-scoped secret bootstrap. Do not upgrade - dependencies, modify upstream repositories, add flags, infer provider settings, - or add a general environment map. Keep unrelated work out of scope. -- Keep the credential opaque across configuration and stage orchestration: - both `NotariusConfig.LLMAPIKey` and `notarius.RunRequest.LLMAPIKey` use - `*configkit.Secret`. This is the adapter's explicit typed-credential boundary - contract, using only ConfigKit's public root API. It avoids another credential - abstraction and avoids introducing raw-secret string fields in stage requests. - Only the Notarius adapter calls `Value()` to construct the child environment. -- A supplied adapter credential must already be resolved. Reject unresolved or - empty programmatic references before process launch with a value-free error. - Do not resolve again from the process environment or silently drop an invalid - credential. Nil remains supported by all existing callers and fakes. -- The adapter forwards `EnvName()` unchanged through `EnvOverrides` and - includes that name in `SensitiveEnvNames`. Do not widen the shared inherited - environment or hardcode a provider-specific destination variable. -- Neither credential name nor value participates in extract's semantic resume - fingerprint. Safe configuration projections include at most the name: - renaming can change the effective digest, value rotation cannot. Do not bump - the extract fingerprint version or add digest-wide invalidation. -- Never serialize or log a raw subprocess request/environment map. Tests use - disposable synthetic credentials and assertions that do not print values on - failure. Environment-mutating tests are nonparallel and restore state. -- Retain the roadmap's downstream responsibility boundary: redact captured - streams, but do not scan/rewrite arbitrary Notarius files or claim protection - against encoded secrets or a malicious executable. -- Each stage must compile and keep existing tests green. Stage 1's configuration - support is an intermediate state, not a releasable completed feature. -- No release, tag, push, upstream changes, or CI polling are part of this plan. - Preserve published release notes and unrelated roadmaps. - -## Stage 1 — Add typed configuration and safe projections - -Status: Complete — 2026-09-16 - -Completion: Added the optional typed Notarius credential reference and a safe -direct-YAML projection. Existing ConfigKit loading, effective projections, -schema-derived provenance, normalization copies, and effective digests carry -the field without exposing its resolved value. Coverage now protects omitted -and null values, custom and invalid references, disabled-section resolution, -profile replacement/removal, root/profile ownership, safe YAML/JSON output, and -reference-name versus value-rotation digest behavior. - -Checks: - -- `GOWORK=off go test ./internal/config` — pass. -- `GOWORK=off go test ./...` — pass; maintained examples load and validate. -- `git diff --check` — pass. - -Read: `internal/config/config.go`, `credential_projection.go`, -`effective_projection.go`, `effective_pipeline.go`, `effective_digest.go`, -`pipeline_configkit.go`, `pipeline_normalize.go`, `validate_notarius.go`, -`notarius_test.go`, `load_validate_test.go`, and `pipeline_configkit_test.go`. -Also read `docs/config.md` and `docs/internal/configuration.md`. - -1. Add `LLMAPIKey *configkit.Secret` to `NotariusConfig`, with - `yaml:"llm_api_key,omitempty" json:"llm_api_key,omitempty"`. Leave its default - nil. Do not add a new environment lookup, codec, or secret-validation framework; - ConfigKit already owns loaded reference syntax and resolution. -2. Add `NotariusConfig.MarshalYAML` in `credential_projection.go`, following - the existing Audita/S3 pattern: preserve all other Notarius fields and their - tags, and project the credential through `secretReferenceName`. - This protects direct YAML serialization as well as effective inspection. - JSON uses ConfigKit's safe secret representation. Preserve existing behavior - of nil stage sections, reference/output maps, and logical path normalization. -3. Verify that the existing effective projection, schema-derived metadata paths, - copy/normalization helpers, source reporting, and digest capture carry the new - field correctly. Make only narrowly necessary changes. Do not add another - projection or provenance store, and do not mutate or resolve the secret in - normalization. -4. Extend package-boundary tests for: - - omitted/null references; a valid custom reference; missing, empty, and - invalid reference names rejected without exposing values; - - selected profiles replacing or nulling an unavailable base reference, - while references in disabled Notarius configuration still resolve; - - safe effective YAML/JSON and direct pipeline YAML containing the reference - name, never its value; nil references retain the previous projection shape; - - correct root/profile provenance for the new field; - - unchanged safe digest after rotating a value under the same name, and - observable projection/diff identity changes after renaming the reference. - Reuse existing credential-projection and profile regression test owners. - Avoid asserting the entire configuration or complete diagnostic wording. - -Acceptance: - -- `GOWORK=off go test ./internal/config` -- `GOWORK=off go test ./...` -- Existing examples still load without adding new required credentials. -- No adapter delivery yet, but all serialization paths for the new field are safe. - -## Stage 2 — Deliver the selected credential and register redaction - -Status: Complete — 2026-09-16 - -Completion: Extended the typed Notarius request boundary with the resolved -credential reference, passed it from extraction, and taught the subprocess -adapter to reject invalid programmatic references before launch. The adapter -now forwards exactly the configured name and loaded value through the minimal -child environment and registers the custom name for streaming redaction. -Focused coverage verifies nil and invalid references, unchanged arguments, -load-time value capture, exact-name delivery, unrelated-parent exclusion, and -stdout/stderr redaction for successful receipts and process failures without -putting credentials in stage metadata. - -Checks: - -- `GOWORK=off go test ./internal/adapters/notarius ./internal/adapters/subprocess ./internal/stage` — pass. -- `GOWORK=off go test -race ./internal/adapters/notarius ./internal/stage` — pass. -- `GOWORK=off go test ./...` — pass. -- `git diff --check` — pass. - -Read: `internal/adapters/notarius/runner.go`, `subprocess.go`, `fake.go`, -`subprocess_test.go`; `internal/adapters/subprocess/run.go`, -`diagnostics.go`, `run_test.go`, `execution_test.go`; -`internal/stage/extract.go`, `extract_test.go`; and -`docs/internal/adapters.md`, `docs/internal/stage-extract.md`. - -1. Add the optional opaque field to `notarius.RunRequest` and document that it - is an already-resolved reference. Pass `notariusConfig.LLMAPIKey` when extract - constructs the request. Do not add credentials to stage metadata, receipts, - output definitions, manifests, reference arguments, or generated files. - Keep fakes compatible and avoid dumping credential-bearing requests in tests. -2. Validate any non-nil request credential before subprocess launch. Obtain the - name and value only at this adapter boundary; a non-nil unresolved reference - is an error, not an instruction to reread the environment. -3. In `SubprocessRunner.Run`, supply exactly the selected credential override - and its sensitive name to the shared subprocess request. Nil adds no override - or sensitive name. Preserve arguments, reference ordering, timeout, working - directory, log/receipt destinations, process lifecycle, and bundle validation. -4. Extend the extract request test to show the selected resolved reference - reaches the adapter without adding raw values to result metadata. Construct - resolved test credentials through the real loader, not reflection or private - ConfigKit state. -5. Extend adapter tests for nil, resolved custom name, and unresolved reference. - Assert custom override/name wiring and unchanged argv without printing the - environment map. A caller changing the parent environment after loading must - not change the resolved credential delivered in this request. -6. Extend the existing real subprocess test using a controlled executable: - - check that the selected value arrives under exactly the configured name, - and an unrelated synthetic parent credential is absent; - - report only success/failure of those checks, not an environment dump; - - emit the selected canary through stdout/stderr in success and nonzero-exit - cases, and assert its absence from persisted captured streams and returned - errors, with the existing redaction marker present; - - keep a valid successful bundle/receipt fixture; put any stdout canary in an - ignorable JSON field rather than corrupting the expected receipt protocol. - Use a custom name outside the built-in sensitive-name list. Verify actual - redaction on disk, not only the request's `SensitiveEnvNames`. -7. Reuse shared subprocess timeout/cancellation and streaming-boundary coverage. - Do not duplicate its whole test matrix or add a new redactor. If adapter - control flow bypasses that shared path, add a focused regression for it. - -Acceptance: - -- `GOWORK=off go test ./internal/adapters/notarius ./internal/adapters/subprocess ./internal/stage` -- `GOWORK=off go test -race ./internal/adapters/notarius ./internal/stage` -- Delivery is opt-in and uses the loaded value. Known and custom names receive - the same captured-stream protection, including subprocess failure diagnostics. - -## Stage 3 — Verify command lifecycle, inspection, and resume invariants - -Status: Complete — 2026-09-16 - -Completion: Extended command-boundary fixtures to cover Notarius credentials -from conventional and explicit secret directories, environment precedence, and -the existing-empty failure rule. Inspection coverage now proves validation, -safe show/sources/diff output, ownership, profile replacement and null removal, -and unavailable-reference rejection. Lifecycle tests verify manifests and the -prepared effective pipeline omit credential bytes, while adding, removing, -renaming, or rotating a valid loaded credential leaves extract's semantic -fingerprint, resume eligibility, and downstream success unchanged without a -second Notarius invocation. No fingerprint version, schema, or invalidation -code changed. - -Checks: - -- `GOWORK=off go test ./internal/app ./internal/config ./internal/stage` — pass. -- `GOWORK=off go test -race ./internal/app ./internal/config ./internal/stage` — pass. -- `GOWORK=off go test ./...` — pass. -- `git diff --check` — pass. - -Read: `internal/app/secrets_bootstrap.go`, `secrets_bootstrap_test.go`, -`config_commands_test.go`, `config_loader.go`, `extract_lifecycle_test.go`, -`semantic_resume_test.go`; `internal/stage/semantic_contracts_delivery.go`, -`semantic_contracts_delivery_test.go`, `prepare.go`, and their focused tests. - -1. Extend existing command credential fixtures with the Notarius reference. - Exercise real pipeline loading/bootstrap with a controlled pipeline path: - conventional and explicit secret directories, environment precedence, and - an existing empty environment variable remaining an error rather than being - replaced from a file. Reuse established fixture helpers and avoid a second - exhaustive bootstrap suite. -2. Prove public config validation accepts the resolved field and rejects an - unavailable one before any adapter execution. Exercise show, sources, and - diff with a custom reference name and a synthetic canary: reference names and - ownership are visible, raw values are absent, and profile null removal works. -3. Cover prepared/resolved pipeline diagnostics with the Notarius canary using - the actual persistence path. Assert that manifests and Narratio-generated - configuration contain no credential bytes. Do not assert sanitation of - arbitrary files authored directly by the downstream fixture. -4. Extend extract semantic fingerprint and resume tests to cover credential - addition/removal, reference-name changes, and value rotation after valid load. - The semantic fingerprint and resume eligibility must remain unchanged. - Verify a previously successful extraction is reused without invoking Notarius; - retain normal selected-reference validation at command loading. -5. Keep effective digest/provenance distinct from semantic resume identity. - Do not change the runner's invalidation relation, fingerprint version, or - schema to satisfy these tests. No blanket cache key may include credentials. - -Acceptance: - -- `GOWORK=off go test ./internal/app ./internal/config ./internal/stage` -- `GOWORK=off go test -race ./internal/app ./internal/config ./internal/stage` -- Tests do not depend on real system configuration, installed Notarius, or a - provider. No secret value is printed even if a non-disclosure assertion fails. -- Credential changes alone do not reexecute a completed extraction or stale - dependent artifacts. - -## Stage 4 — Document the operator contract and validate the whole feature - -Status: Complete — 2026-09-16 - -Completion: Updated the canonical configuration, CLI, Notarius integration, -internal configuration/adapter/extract, troubleshooting, and example guidance. -The maintained annotated pipeline demonstrates the optional reference with a -synthetic-value test fixture, and published release notes remain unchanged. -The full repository validation and all three maintained asset builds succeeded; -the macOS arm64 result is cross-compilation evidence only. - -Checks: - -- `GOWORK=off go mod tidy -diff` — pass with no changes. -- `GOWORK=off go mod verify` — pass. -- `GOWORK=off go test -count=1 ./...` — pass. -- `GOWORK=off go test -race -count=1 ./...` — pass. -- `GOWORK=off go vet ./...` — pass. -- `GOWORK=off go build ./...` — pass. -- `GOWORK=off go test ./internal/doccheck` — pass. -- `GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$'` — pass. -- `git diff --check` and `gofmt -l` on changed Go files — pass. -- `scripts/build-release-assets.sh v0.0.0-test ` — pass for - Linux amd64, Linux arm64, and macOS arm64 cross-compilation. - -Read: `docs/config.md`, `docs/cli.md`, `docs/integrations/notarius.md`, -`docs/internal/configuration.md`, `docs/internal/adapters.md`, -`docs/internal/stage-extract.md`, `docs/troubleshooting.md`, -`examples/README.md`, `docs/release.md`, and the feature roadmap. - -1. Update canonical documentation with the field's type, absence/null behavior, - environment-name matching with Notarius/PromptKit, existing bootstrap - precedence, and the load-time requirement even when extraction is reusable. - State the redaction/downstream responsibility boundary and operational-only - resume treatment in their appropriate owners; link rather than duplicate. - Do not advertise new CLI flags or automatic provider-name discovery. -2. Add a maintained, secret-free example of the optional reference in an - appropriate existing Notarius example. Keep ordinary examples runnable without - real keys: if a fixture actively declares the reference, its example test must - supply and restore a synthetic environment value. Never add real secret files. -3. Preserve immutable release notes, including v2.0.0. Update an existing - unreleased note if one exists; otherwise record a concise release-ready summary - here for the separately authorized release task. Do not choose/tag a new - version merely to complete this stage. -4. Run the repository checks below against normal module files. Inspect the diff - for unintended dependency, platform, CLI, schema, and fingerprint changes. - Mark the roadmap and all stages complete only after acceptance succeeds. - Retain other roadmap and audit files. - -Required validation: - -```sh -GOWORK=off go mod tidy -diff -GOWORK=off go mod verify -GOWORK=off go test -count=1 ./... -GOWORK=off go test -race -count=1 ./... -GOWORK=off go vet ./... -GOWORK=off go build ./... -GOWORK=off go test ./internal/doccheck -GOWORK=off go test ./internal/config -run '^TestExamplesLoadAndValidate$' -git diff --check -``` - -Check changed Go files with `gofmt -l`. Run -`scripts/build-release-assets.sh v0.0.0-test OUTPUT_DIR` with an empty absolute -temporary directory created by `mktemp -d`. All three maintained targets must -build; report macOS cross-compilation as such, not native runtime validation. -No live model calls, tag publication, or CI monitoring are required. - -## Open Questions - -None. The operator contract, optionality, delivery boundary, redaction scope, -and resume policy are settled. Release version selection is a separate task. diff --git a/docs/roadmap/notarius-credential-forwarding.md b/docs/roadmap/notarius-credential-forwarding.md deleted file mode 100644 index fa33310..0000000 --- a/docs/roadmap/notarius-credential-forwarding.md +++ /dev/null @@ -1,259 +0,0 @@ -# Typed Notarius Credential Forwarding - -Status: Complete — 2026-09-16 - -## Summary - -Allow an operator to configure one typed credential reference for the Notarius -stage. Narratio resolves that reference through its existing ConfigKit secrets -bootstrap and supplies the credential to the Notarius subprocess under the -referenced environment-variable name. - -The target is a bounded configuration-and-adapter feature that reuses ConfigKit -typed secrets and Narratio's existing subprocess environment and redaction -support. No upstream change or general environment-management framework is -required. This roadmap owns the desired end state and policy; the -[implementation plan](implementation.md) owns execution order and concrete -implementation steps. - -## Problem - -An enabled Notarius pipeline may select a PromptKit profile that requires an -API key, such as a profile whose `api_key_env` is `OPENROUTER_API_KEY`. -Narratio already supports environment and secrets-directory bootstrap for typed -credentials, but the Notarius configuration section has no credential field. - -Consequently, an operator cannot declare the Notarius credential dependency in -Narratio configuration. Supplying the variable to Narratio's environment or -conventional secrets directory is not sufficient under the current public -contract: the Notarius invocation receives Narratio's minimal child -environment, and no selected Notarius credential is forwarded. - -This leaves an otherwise valid configured extraction stage unable to -authenticate to a provider without an out-of-band invocation mechanism. - -## Required Operator-Facing API - -Add one optional typed secret reference at: - -```text -pipeline.notarius.llm_api_key -``` - -Illustrative addition to an otherwise valid pipeline: - -```yaml -notarius: - llm_api_key: OPENROUTER_API_KEY -``` - -`llm_api_key` contains an environment-variable name, never a raw credential. -Its value uses the same typed-secret and ConfigKit bootstrap semantics as other -Narratio credential references: - -- when the field is absent, Narratio forwards no Notarius credential and - preserves the current behavior; -- when the field is present, its value must be a valid typed secret reference - that resolves to a nonempty credential during configuration loading; -- an existing process environment value takes precedence over a value loaded - from the selected secrets directory; -- `--secrets-dir` and conventional sibling `secrets/` discovery retain their - existing meanings; and -- a selected profile may replace or explicitly remove the field under ordinary - ConfigKit overlay semantics. - -An explicit null has the same credential-delivery meaning as absence. Every -non-null reference surviving the selected profile must resolve, including during -inspection or when extract will not execute. Unselected or removed base -references are not independently resolved. Do not introduce stage-dependent -secret loading or a second environment lookup at execution time. - -For a configured reference such as `OPENROUTER_API_KEY`, the Notarius child -process receives the resolved credential under exactly that environment name. -This lets the operator coordinate the Narratio field with the `api_key_env` -selected by the Notarius PromptKit profile without introducing -provider-specific configuration in Narratio. - -No new CLI flag is required. The existing Notarius argument vector, pipeline -configuration path, working directory, reference bindings, receipt, and output -contracts remain unchanged. - -## Required Architecture and Changes - -### Configuration and inspection - -Add an optional `*configkit.Secret` to `NotariusConfig`, following the existing -typed credential pattern. ConfigKit owns reference syntax, effective-profile -resolution, and access to the resolved value; the application continues to own -the one-per-invocation secrets-directory bootstrap. The field defaults to absent. - -Extend the relevant safe configuration projections, serialization, provenance, -and tests so this field is visible by its public reference name. Review both -effective inspection and generated/persisted configuration paths; protecting -`config show` alone is insufficient. Do not serialize an adapter invocation -request or a raw credential-bearing structure for diagnostics. - -### Extract and adapter delivery - -Carry the optional credential through extract's existing Notarius invocation -boundary. Keep transport-specific environment construction in the Notarius -adapter and expose the resolved value only at the trusted delivery boundary. -Reuse the subprocess runner's `EnvOverrides` for the selected name/value pair -and `SensitiveEnvNames` to register that same name for redaction. Register custom -names as well as familiar provider names; the built-in sensitive-name list is -not sufficient for this feature. - -The child retains Narratio's existing minimal inherited environment plus the -explicit credential override. There is no broad process-environment inheritance, -provider-name remapping, or forwarding of unrelated bootstrapped secrets. An -absent reference adds neither an override nor a credential requirement. - -### Ownership - -Keep configuration semantics in `internal/config`, bootstrap in `internal/app`, -invocation orchestration in `internal/stage`, and environment/redaction delivery -in `internal/adapters/notarius` and the shared subprocess boundary. Reuse the -existing helpers rather than introducing another secret resolver or redactor. -The [architecture policy](../policy/architecture.md), -[configuration internals](../internal/configuration.md), and -[adapter internals](../internal/adapters.md) remain the canonical owners of the -implemented system's boundaries. - -## Inspection and Diagnostic Contract - -- `narratio config validate` resolves the configured secret reference and - fails before stage execution when it is missing or empty. -- `narratio config show` may expose the non-secret reference name, such as - `OPENROUTER_API_KEY`, but never the resolved credential value. -- `narratio config sources` and `narratio config diff` retain their existing - secret-safe behavior. -- Configuration digests, manifests, generated configuration, command - arguments, persisted diagnostics, receipts, and other operator-visible - output must not contain the resolved credential value. -- Errors may identify the field or unresolved environment-variable name but - must not include credential bytes. - -The selected credential must receive the existing streaming redaction protection -on captured stdout and stderr, including logs, captured receipt output, and -stderr-derived errors on nonzero exit, timeout, or cancellation. Test the delivery -of the dynamically selected sensitive name; do not create a provider-specific -redaction exception or a second log-sanitization mechanism. - -### Downstream responsibility boundary - -Narratio must not place credential values in its authored configuration, -arguments, manifests, or metadata, and must redact the selected value from -subprocess streams it captures. Notarius remains responsible for keeping -credentials out of files it writes directly, including bundle artifacts and -diagnostics. Captured-stream redaction is not a filesystem sandbox and does not -guarantee sanitization of arbitrary downstream files or transformed/encoded -secret values. This feature does not add recursive scanning or rewriting of -Notarius output bundles. Such rewriting could corrupt valid artifacts and would -not establish a general non-disclosure guarantee. - -## Resume and Configuration Identity - -Authentication is operational configuration, not extraction content. Neither -the credential value nor its reference name becomes part of extract's semantic -resume fingerprint. Credential rotation, renaming, addition, or removal alone -must not invalidate a successfully recorded extraction or its dependents. - -The effective configuration projection may include the public reference name, -so changing that name can appear in `config diff` and change the effective -configuration digest. Rotating only the resolved value must not change that -digest. Preserve the existing distinction between configuration provenance and -stage-owned resume evidence; do not add a blanket digest-based invalidation. -Normal configuration loading still requires any selected reference to resolve -even when the prior extraction can be reused. - -## Compatibility - -The field is optional and defaults to absent. Existing configurations retain -their current behavior without migration. - -The credential name belongs to deployment configuration rather than to a -specific provider. Narratio does not infer a name from a Notarius pipeline, -profile ID, backend, model, or configuration file. Operators must select a name -that agrees with the Notarius PromptKit profile they deploy. - -## Acceptance Criteria - -- A pipeline with Notarius enabled and no `notarius.llm_api_key` continues to - load and invoke Notarius with no forwarded provider credential. -- A pipeline may set `notarius.llm_api_key` to a valid environment-variable - reference. -- The reference resolves from an existing process environment value or from - Narratio's existing explicit or conventional secrets-directory bootstrap. -- A missing or empty configured credential fails configuration loading before - Notarius execution. -- A successful Notarius invocation receives the resolved value under exactly - the configured reference name. -- Unrelated credentials present in Narratio's environment or secrets directory - are not made available to Notarius merely because this field is configured. -- Base and named-profile configuration follow ordinary ConfigKit replacement, - explicit-null, source-reporting, diff, and secret-resolution semantics. -- Effective inspection exposes at most the safe reference name and never the - resolved value. -- No raw credential appears in Narratio-authored command arguments, generated - configuration, manifests, metadata, or test output. Captured receipt/log streams - and errors redact the selected value within the responsibility boundary above. -- A custom environment-variable name not present in the built-in sensitive-name - list receives the same redaction protection as a known provider name, on both - successful and failing subprocess execution. -- Credential rotation or reference-name changes do not alter extract's semantic - fingerprint or invalidate a reusable extraction. Inspection reflects name - changes without revealing values; value-only rotation leaves the safe digest - unchanged. -- The existing Notarius CLI arguments and result-validation contract remain - unchanged. -- Focused tests use synthetic credentials and a controlled Notarius substitute; - they do not contact a model provider or require a real API key. -- Configuration, CLI, integration, example, troubleshooting, and release - documentation are updated only where their existing canonical ownership - requires it when the feature is implemented. - -Use focused behavioral tests at the configuration, extract/adapter delivery, -and captured-output boundaries. Retain shared subprocess tests for its generic -streaming guarantees rather than duplicating their full matrix. Verify the -assembled invocation with a controlled executable and synthetic credentials, -including proof that an unrelated credential is not forwarded. Follow the -[testing policy](../policy/testing.md) and the normal -[development validation](../development.md#validation); no live-provider -authentication is an acceptance gate. - -## Target End State - -An operator supplies a named API-key file through the existing secrets directory -or sets that environment variable, declares `notarius.llm_api_key`, and configures -the matching `api_key_env` in Notarius's selected PromptKit profile. Narratio -validates the reference, delivers only the selected credential to Notarius, and -keeps its own configuration and diagnostics credential-safe. Existing pipelines -need no migration, and changing authentication does not rerun completed analysis -work. No new CLI, dependency upgrade, or upstream feature is required. - -## Non-goals - -- Adding an arbitrary environment-variable map to the Notarius stage. -- Forwarding every variable loaded from the secrets directory. -- Inferring provider, backend, profile, model, or credential names from - Notarius configuration. -- Managing secret files, credential values, rotation, encryption, or an - external secret store. -- Changing Notarius configuration syntax, PromptKit profile syntax, CLI flags, - receipt schemas, pipeline definitions, or output bundles. -- Supporting more than one Notarius provider credential in this feature. -- Validating a credential with its provider during configuration loading. -- Adding credentials to command-line arguments, ordinary YAML values, or - generated invocation files. -- Scanning or sanitizing arbitrary files created directly by downstream tools. -- Changing resume semantics, pipeline path bases, artifact schemas, or the - existing minimal inherited subprocess environment. - -## Consumer Dependency - -The D&D Ansible pipeline requires this operator-facing contract before it can -provide complete suite-managed credential delivery. Its intended deployment -places files named for configured secret references in Narratio's conventional -`secrets/` directory. That deployment can select and provision the value only -after Narratio provides a typed field that forwards the selected Notarius -credential under the same reference name. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index e7c0129..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,331 +0,0 @@ -# ConfigKit Migration Review — Required Remediations - -Reviewed 2026-09-07 at `f7a40af`, against the changes after `1558c9c`, the -[feature roadmap](configkit-adoption.md), and the completed -[implementation plan](implementation.md). - -The main migration is implemented. The following findings are reproducible gaps -in the intended behavior, not proposals to restore retired compatibility or -filesystem restrictions. P2 denotes a correctness issue to fix before release; -P3 denotes an operator-facing completeness issue also required by the accepted -plan. No application code was changed during this review. - -## 1. Derive explicit publish destinations after family expansion - -Status: Resolved — P2 - -Resolution: Missing destinations are now derived through one shared domain helper -both during ConfigKit resolution and again after concrete family expansion. Explicit -destinations remain unchanged, while unknown sources still fail read-only publish -validation. Regression coverage verifies resolved runtime and effective inspection -output for generated members, explicit overrides, and unknown sources. - -**Evidence:** `internal/config/pipeline_configkit.go:208` derives missing publish -destinations during the ConfigKit resolver. A concrete family member does not -exist yet, so the resolver ignores the failed derivation. After family expansion, -`internal/config/artifact_families.go:141` calls the now-read-only publish -validator, which rejects the still-empty destination. The previous implementation -could derive it during that later validation. - -**Reproduction:** In an otherwise valid pipeline, define a family `meta` over -`party.characters`, with output pattern `artifacts/{character_id}.md`. Select a -valid campaign whose party contains `alice`, and add: - -```yaml -publish: - outputs: - - source: narratio.artifact.meta_alice -``` - -`config validate --config --campaign-file ` fails with -`pipeline.publish.outputs[0].dest is required when destination cannot be derived -from "narratio.artifact.meta_alice"`. Adding `dest: artifacts/alice.md` makes the -same configuration pass. The destination is therefore available and derivable; -the lifecycle omits the required final normalization step. - -**Required change:** Complete publish destination resolution after concrete -artifacts exist, before publish validation and final projection/digest generation. -Share the domain derivation logic and keep validation read-only. Preserve -explicit destinations and generated family publish-rule collision checks. Do not -require redundant `dest` fields or revive mutation inside validators. - -**Acceptance:** Test an explicit publish source naming a generated member with -omitted destination, an explicit destination override, and a genuinely unknown -source. Validate both effective inspection and resolved runtime configuration. - -## 2. Stop re-defaulting an explicitly empty Notarius working directory - -Status: Resolved — P2 - -Resolution: Removed both legacy blank-value fallbacks after ConfigKit resolution. -An omitted working directory is still derived by the presence-aware resolver, an -explicit relative directory is still rooted at the pipeline, and explicit empty or -whitespace-only values remain empty and fail enabled-Notarius validation. Existing -relocation-invariant digest coverage remains green. - -**Evidence:** ConfigKit's resolver correctly uses presence-aware defaulting at -`internal/config/pipeline_configkit.go:180`. However, -`internal/config/load.go:454` subsequently replaces any blank working directory -with the config file's directory, irrespective of input presence. -`internal/config/effective_digest.go` also applies a blank-value fallback while -capturing logical paths. These retained legacy fallbacks bypass the new rule that -defaults apply only to absent fields. - -**Reproduction:** Set `notarius.enabled: true`, a valid config path, pipeline ID -and output map, and `notarius.working_directory: ""`. With -`config_path: notarius/config.yml`, `config show` succeeds and reports -`working_directory: notarius`; validation never sees the explicit invalid blank. - -**Required change:** Let the ConfigKit resolver own the absent-field default. -The subsequent runtime and logical-path conversions must preserve explicit -emptiness so enabled-Notarius validation can reject it. Remove duplicate -value-based defaulting rather than adding another private presence mechanism. - -**Acceptance:** Omission still derives the expected directory; explicit empty -and whitespace-only values fail when Notarius is enabled; a configured relative -directory resolves from the pipeline root. Preserve digest relocation invariance. - -## 3. Enforce unique base definitions even when a profile removes them - -Status: Resolved — P2 - -Resolution: Loading now discovers base declarations through ConfigKit's public -historical-path enumeration and metadata history from the selected result, -without separately resolving the base (see finding 7). Duplicate ordinary or -family definitions therefore fail even when an overlay nulls their container or -all of Scriptorium. A single base owner -with a legitimate profile override remains supported. - -**Evidence:** `rejectDuplicateBaseDefinitions` at -`internal/config/pipeline_configkit.go:354` enumerates only artifact/family keys -remaining in the final typed configuration. It inspects history for those keys -but never examines base declarations removed by an overlay. - -**Reproduction:** Import `first.yml` defining -`scriptorium.artifacts.recap.prompt_id` and `second.yml` defining -`scriptorium.artifacts.recap.output_path`. Base validation correctly rejects the -two-file declaration. Add `profiles/clear.yml` containing: - -```yaml -scriptorium: - artifacts: null -``` - -The same root with `--profile clear` passes. A selected profile can thus hide the -base ownership conflict, contrary to the plan's unique base-definition rule. -This concerns disjoint contributions to one artifact, not ConfigKit's generic -conflicting-leaf checks, which still work. - -**Required change:** Discover the complete base declaration set through public -ConfigKit metadata/history, including declarations displaced by an overlay, and -validate ownership independently of final effective-key survival. Keep legitimate -profile overrides and profile-only definitions supported. Do not reparse source -YAML or add a second composer. - -**Acceptance:** Cover duplicate ordinary and family definitions hidden by a null -artifact/family container or null Scriptorium section. Base and selected-profile -loads must reject the same base conflict, while a single base declaration with a -valid overlay remains accepted. - -## 4. Preserve null versus empty collections in effective YAML - -Status: Resolved — P2 - -Resolution: Effective YAML is now emitted from the credential-safe JSON value -projection through an output-only YAML-node conversion. Explicit nulls, empty maps, -empty slices, booleans, strings, and exact JSON numbers retain their semantic types. -Public show/diff tests cover null and empty maps and slices, while digest, -credential-redaction, deterministic output, and exact-integer coverage remain intact. - -**Evidence:** `internal/config/effective_pipeline.go:154` marshals the typed -projection directly with `yaml.Marshal`. The encoder emits nil slices as `[]` -and nil maps as `{}`, while the JSON projection used by diff and digest preserves -them as `null`. The shared input type does not guarantee shared output semantics. - -**Reproduction:** Set `audita.modules: null` in a valid base and -`audita.modules: []` in `profiles/empty.yml`. Both `config show` invocations emit -`modules: []`, but `config diff default empty` reports: - -```text -changed audita.modules null [] -``` - -The commands disagree about the effective value. The same renderer also supplies -the resolved pipeline diagnostic written by prepare. - -**Required change:** Render YAML from a credential-safe projection that explicitly -preserves nulls and empty collections, consistently with the semantic projection. -Keep exact integers and safe credential reference names. An output-only conversion -is appropriate; do not restore YAML decoding or a configuration parser. This -requires truthful inspection, not a guarantee that inspection output is reloadable. - -**Acceptance:** Verify null versus empty maps and slices through the public show -boundary, and agreement with diff/digest distinctions. Retain credential-leakage -and exact-integer regression coverage. - -## 5. Complete provenance for generated artifacts and inferred dependencies - -Status: Resolved — P3 - -Resolution: Generated catalog records now collect the complete base/profile family -definition set from ConfigKit history. Effective source ownership for materialized -dependency lists is rebuilt narrowly from retained explicit-list and input-edge -origins, including mixed and generated-family dependencies. CLI tests verify a -profiled family catalog agrees with generated source attribution and that inferred -ordinary edges name their input contributor instead of the central default. - -Two new reporting paths lose available declaration information: - -- **Profile contributions to family members:** - `internal/config/artifact_catalog.go:52` fills a generated member's definitions - from the single `member.Source`. That source is chosen by the first matching - non-default family field in `artifactFamilySource`, not the full set of effective - contributors. Define family `meta` in the base and override its `prompt_id` in - `profiles/testing.yml`. `config artifacts --profile testing` reports only the - base file for `meta_alice`, even though `config sources` correctly attributes - that member's prompt to the testing overlay. -- **Inferred ordinary dependencies:** With `second.inputs.prior.source` set to - `narratio.artifact.first` and no explicit dependency list, `config sources` - reports `scriptorium.artifacts.second.depends_on` as `default / default`. - `finalizeArtifactDependencies` stores the input-edge origin, but the effective - ownership projection retains the pre-inference default attribution from - `internal/config/effective_digest.go:55` and does not consult that origin. - -**Impact:** Operators inspecting a testing profile or following a dependency can -be directed to the wrong file or told a user-derived value is a central default. -Runtime ordering and generated artifact values were correct in these reproductions. - -**Required change:** Use ConfigKit's contributing locations for family definition -sets, as for ordinary artifacts, and add narrow generated attribution for inferred -dependency lists using their retained edge origins. Preserve family/character/party -identity and deterministic ordering. Avoid introducing a second general provenance -store or choosing a file based on the first alphabetically visited field. - -**Acceptance:** A profiled family catalog exposes both relevant files; an inferred -edge points back to its input declaration; mixed explicit/inferred dependencies -retain their relevant contributors. Verify catalog and sources agree after expansion. - -## 6. Surface successful secrets bootstrap reports, including skipped empty variables - -Status: Resolved — P3 - -Resolution: Successful bootstrap reports now flow once through the command -diagnostics channel, while direct internal command calls default that channel to -discard. ConfigKit's value-free report makes skipped existing empty variables -explicit and remains available before later typed-load failures; structured output -stays isolated on stdout. Command-level coverage verifies successful reporting, -skipped-empty reporting, value redaction, clean config-show output, and one report -for a two-profile diff. - -**Evidence:** `internal/app/secrets_bootstrap.go:36` discards the report whenever -`LoadSecretsDir` returns no error. Reports are only rendered when directory -bootstrap itself fails. This does not implement the roadmap's safe bootstrap -reporting boundary or Stage 6's warning presentation requirement. - -**Reproduction:** Place a synthetic key in `secrets/NARRATIO_REVIEW_KEY`, reference -it from `audita.llm_api_key`, and invoke config validation with -`NARRATIO_REVIEW_KEY=` in the environment. ConfigKit correctly skips the file -because the variable already exists, then typed loading fails with `secret.empty`. -Narratio shows only that load failure, omitting the bootstrap report's -`skipped_existing` and `ExistingEmpty` information. A successful load likewise -discards all bootstrap results. - -**Required change:** Retain and render or summarize the safe report once per -invocation through a diagnostic output channel, including when a later typed load -fails. Make skipped-existing-empty outcomes explicit. Preserve process environment -precedence and the existing partial report on failure; keep values out of all -diagnostics and keep structured/YAML stdout uncontaminated. Do not add bootstrap -rollback, retries, or repeated directory loading. - -**Acceptance:** Command-level tests demonstrate safe success/report output and a -useful skipped-empty diagnostic, without leaking a synthetic secret or polluting -show output. A two-profile diff still bootstraps and reports only once. - -## 7. Do not resolve unused base credentials when loading a profile - -Status: Resolved — P2 - -Resolution: Upgraded to released ConfigKit v0.1.2 and replaced the extra base load -with `Metadata.HistoricalPaths()` enumeration of declaration paths in the selected -result. Ownership checks still inspect base-layer history for both ordinary -artifacts and families, even when overlays remove them. Regression tests verify -that a profile can replace an unset production credential with a resolvable testing -reference or remove it with null. Existing hidden-duplicate and legitimate-override -tests remain intact. No local dependency replacement or second composer is needed. - -Follow-up review: 2026-09-07 at `3ce1981`. The original six reproductions are -addressed, but the fix for finding 3 introduces this regression. - -**Evidence:** `internal/config/pipeline_configkit.go:49-57` loads the base again -without a profile to check duplicate base definitions. This calls the complete -typed ConfigKit loader, including secret resolution, domain resolution, and -validation—not just declaration ownership inspection. Consequently, loading a -valid testing profile also requires credentials from the unselected base. - -**Reproduction:** In an otherwise valid base, set `audita.llm_api_key` to -`NARRATIO_REVIEW_PROD_KEY`. In `profiles/testing.yml`, replace it with -`NARRATIO_REVIEW_TEST_KEY`. Leave the production variable unset and supply only -the testing variable with a synthetic value. `config validate --config ---profile testing` fails with `secret.unset` at the base credential declaration. -Supplying both variables makes it pass. A profile that instead sets -`audita.llm_api_key: null` also fails when the unused production variable is -unset. Both failures were reproduced through the current CLI. - -**Impact:** Operators cannot use an independent testing credential environment -or remove a production credential requirement through a profile. Commands fail -despite having all credentials required by their effective configuration. The -extra full load also unnecessarily repeats configuration processing. - -**Required change:** Enforce unique base ownership without requiring the -unselected base to succeed as a separately resolved runtime configuration. -Inspect complete declaration history through ConfigKit's public metadata, -including declarations displaced by overlays. If additional upstream metadata -support is needed, address that explicitly rather than requiring unused secrets, -suppressing arbitrary load failures, reparsing YAML, or restoring a second -composer. Preserve the duplicate-definition protection added for finding 3. - -**Acceptance:** Add public-boundary regression tests for a profile replacing an -unset production credential with an available testing credential, and for a -profile removing that credential with null. Both must load successfully without -the production variable. Retain failures for duplicate base ordinary/family -definitions hidden by null overlays, along with legitimate profile override and -profile-only definition coverage. Resolve secrets only for the selected effective -configuration. - -## Follow-up verification - -After adopting ConfigKit v0.1.2 and completing finding 7, the full -`scripts/check-release-candidate.sh v2.0.0` check passed: module hygiene, -formatting/whitespace, uncached tests and race tests, vet, build, documentation, -maintained examples, and Linux amd64/arm64 plus macOS arm64 release cross-builds. -`go mod verify` also passed. The new credential regressions failed before the -ownership-check fix and passed afterward. No local module replacements, release -tags, or pushes were used. - -At `3ce1981`, `go test ./...`, `go test -race ./...`, `go vet ./...`, and -`go build ./...` passed. The new finding was reproduced with a freshly built CLI, -temporary configuration files, and synthetic credentials; no external adapters -or cloud services were invoked. That review pass changed only this review record. - -## Original verification and scope - -The following checks passed at the reviewed commit: - -- `go test ./...` -- `go test -race ./...` -- `go vet ./...` -- `go build ./...` -- The release asset builder for `v2.0.0`, producing exactly Linux amd64, Linux - arm64, and macOS arm64 assets; the Linux amd64 binary reported the expected version. - -The existing full suites include documentation and maintained-example checks. -The findings above were additionally reproduced through a freshly built CLI with -isolated temporary YAML fixtures and synthetic credentials. No live downstream -tools or cloud services were invoked. Linux arm64 and macOS arm64 were cross-built, -not tested natively. No tag was created or pushed. - -The migration removes a net 1,235 production Go lines and 843 test Go lines versus -`1558c9c` (physical diff lines, including comments and whitespace). ConfigKit owns -generic loading/composition; Narratio retains domain resolution, orchestration, -safe inspection, and adapter delivery. These findings require completing that -design, not reversing the simplification.