notarius/docs/roadmap/implementation.md

# ConfigKit Inheritance Implementation Plan

## Purpose And Scope

Implement [ConfigKit 0.5 And Configuration Inheritance](configkit.md). That
feature roadmap owns the intended behavior and policy choices. This document
owns the ordered implementation work. Each numbered stage is one prompt for a
Luna coding agent; complete its acceptance criteria before starting the next.

The starting baseline pins ConfigKit v0.4.2 and PromptKit v0.11.2, uses ordinary
`configkit.Load`, and accepts Notarius configuration version 5. Upgrade only
ConfigKit to v0.5.0, source commit
`dc09959cca4e105f49ec316c6d4ba585bd2340bd`. The release tag is annotated: use
`git rev-parse 'v0.5.0^{commit}'` when verifying the source commit in a checkout.

The implementation adds supported native configuration inheritance and a
maintained example. It does not require changing pipeline execution, artifact
schemas, or release tooling. Release preparation and publication remain a
separate task; this feature belongs in a future minor release.

## Working Instructions For Every Stage

- Read `docs/development.md`, `docs/policy/agents.md`, and the feature roadmap.
  Use the development guide to select additional applicable policies, including
  architecture, testing, documentation, and OSS for the changes being made.
- Inspect the named code and tests before editing. Prefer the repository's
  code knowledge graph for code discovery. Inspect the pinned upstream release
  and public API when an integration detail needs confirmation; do not assume
  a sibling checkout's current branch equals the released version.
- Keep ordinary `configkit.Load`, typed defaults, and the existing private
  configuration boundary. ConfigKit handles `extends` before decoding; add no
  `Extends` field, second merge pass, application template registry, or opt-in
  flag. Accept valid native inheritance locations under existing domain rules.
- Make production changes only where a concrete integration failure requires
  them. A stage may consist of focused tests and documentation when the
  existing adapter already behaves correctly. Fix a regression rather than
  weakening an established expectation to accommodate the upgrade.
- Test Notarius's boundary with the real ConfigKit dependency and small,
  deterministic fixtures. Reuse existing helpers and coverage; do not replicate
  ConfigKit's complete test matrix, add diagnostic snapshots, or assert
  incidental output formatting. Keep the default suite offline and independent
  of credentials and model calls.
- Use `GOWORK=off` for Go commands so a sibling checkout cannot substitute for
  the pinned modules. Update canonical documentation as each behavior lands.
  Review the diff, run the stage's checks, and commit the coherent result under
  the agent policy. Record any unresolved failure in the handoff rather than
  marking that stage complete. Do not push or publish a release as part of this
  plan.

## Stage 1 — Upgrade The Dependency And Its Notices

> Completion: complete.

Goal: establish the released dependency baseline without redesigning the
configuration adapter.

Inspect:

- `go.mod`, `go.sum`, and `internal/core/config/load.go`.
- `docs/integrations/pkg-configkit.md` and the ConfigKit entry in
  `docs/oss/attribution.md`.
- ConfigKit's pinned `docs/releases/v0.5.0.md`, `go.mod`, `COPYRIGHT`, and
  `LICENSE`; consult the sibling checkout with `git show v0.5.0:<path>`.

Work:

1. Run `GOWORK=off go get gitea.maximumdirect.net/eric/ConfigKit@v0.5.0`, then
   `GOWORK=off go mod tidy`. Review the selected dependency graph. Retain
   PromptKit v0.11.2 and unrelated pins unless module resolution requires a
   change; explain any such change in the handoff.
2. Search maintained configuration, fixtures, YAML tags, and documentation for
   ordinary uses of the key or entry name `extends`. For example, use
   `rg -n '\bextends\b' examples internal docs assets`. Classify any matches:
   new inheritance instructions are valid; ordinary field/key uses collide
   with the new reserved directive. Rename actual maintained collisions and
   their consumers together, without an alias or bypass. Do not rename the
   unrelated PromptKit `base_profile` setting.
3. Run focused existing tests. Address compile/API compatibility failures with
   the smallest necessary adapter changes. Native inheritance needs no
   registration or load option.
4. Update the integration document's pin and brief description to v0.5.0;
   replace the v0.4.2 patch-only explanation with the new composition boundary.
   Link to pinned upstream release/configuration documentation.
5. Update only ConfigKit's attribution version and exact source/notice links
   to the target commit, preserving the verified copyright and
   `LGPL-3.0-or-later` expression. Reuse the existing LGPL/GPL texts when
   unchanged. Do not refresh unrelated component attribution or policies.

Acceptance criteria:

- The declared and selected ConfigKit module is v0.5.0, without a workspace or
  local `replace` dependency.
- Existing configuration and CLI tests pass; maintained ordinary `extends`
  collisions, if any, have been resolved explicitly.
- Integration and attribution documents agree with the actual dependency and
  imported source commit. The adapter still uses ordinary `Load`.

Validation:

```sh
GOWORK=off go list -m gitea.maximumdirect.net/eric/ConfigKit
GOWORK=off go test ./internal/core/config ./internal/cli
git diff --check
```

Prompt size: one dependency upgrade, its notices, and demonstrated compatibility
adjustments. Leave new inheritance coverage to the following stages.

## Stage 2 — Prove Pipeline Composition And Runtime Conversion

Goal: establish whole-pipeline reuse through the real configuration boundary.

Inspect `load.go`, `document.go`, `loaded_config.go`, `pipeline_document.go`,
`document_test.go`, `pipeline_document_test.go`, and
`effective_config_contract_test.go` under `internal/core/config`. Reuse the
existing test catalog for runtime resolution.

Work:

1. Add focused loading coverage in `internal/core/config/inheritance_test.go`.
   Define a valid parent pipeline and a child with `extends: parent` plus a
   pipeline-level override. Assert the inherited input, bindings, and ordered
   steps survive conversion, while the child retains its own pipeline ID and
   the parent keeps its own settings.
   Keep the fixture small rather than copying the complete D&D example.
2. Resolve the resulting child with the existing catalog. Assert its selected
   ID and overridden setting reach the resolved pipeline. Use the catalog's
   registered module keys and avoid generated-reference requirements in this
   small fixture; the full generated-reference handoff is covered in Stage 7.
   Reuse existing resolution tests for unrelated defaults and digests.
3. Add one replacement case: the child supplies a complete different `steps`
   list. Assert replacement and ordering, with no parent steps appended or
   merged by ID. Both resulting pipelines must satisfy ordinary validation.
4. Add one native nested-mapping case using sibling artifact lanes in an
   `artifacts` mapping: a child lane inherits a valid parent lane and overrides
   a binding value. This proves that Notarius does not limit inheritance to
   pipeline mappings. Other upstream-supported mapping shapes need no custom
   permission mechanism.
5. In `docs/config.md`, add the canonical inheritance explanation: exact
   immediate-sibling lookup, one parent, source composition before expansion,
   recursive mapping combination, sequence/scalar/null replacement, normal
   validation of parent and child, and all effective relationships being
   resolved. State that step-sequence entries cannot extend each other.
   Include a small illustrative pipeline snippet and the reserved `extends`
   migration note. Distinguish configuration overlays and PromptKit profiles.

Acceptance criteria:

- Loading and resolving a child yields an ordinary independently identified
  pipeline with the expected inherited composition and explicit override.
- Replacement lists retain only the child list, in its configured order.
- Valid nested artifact-lane inheritance works through native ConfigKit
  behavior without extra runtime types or application merging.
- Configuration version remains 5 and the user reference describes the
  implemented syntax and reserved-key compatibility impact.

Validation:

```sh
GOWORK=off go test ./internal/core/config
git diff --check
```

Prompt size: small composition fixtures, runtime handoff, and one canonical
syntax section. Presence-sensitive cases belong to Stage 3.

## Stage 3 — Preserve Presence And Existing Domain Validation

Goal: ensure inherited input remains explicit through Notarius normalization,
defaulting, and validator-chain conversion.

Inspect `operational_document.go`, `pipeline_document.go`,
`document_normalizer.go`, `loaded_config.go`, and their existing presence tests.
Extend `inheritance_test.go` rather than adding a parallel fixture framework.

Work:

1. Cover inherited `structured_output_repair_attempts: 0`: it remains a
   populated zero override after loading and resolution, rather than reverting
   to a pipeline or prompt default. Also cover a child explicitly overriding a
   nonzero parent budget with zero.
2. Cover an inherited supported `validators: []` on a chunk or extraction
   binding. Its runtime override retains `Set=true` and an empty chain. Cover
   a child replacing a nonempty parent chain with `[]`; do not concatenate it
   or reinstall default validators because the child list is empty.
3. Cover an inherited false module-option value, such as
   `output.options.evidence_context.enabled: false`, remaining false in runtime
   configuration. Inspect the decoded document's input metadata to establish
   explicit presence where needed, rather than relying on truthiness.
4. Cover inherited or child-supplied null for a field already rejected when
   explicit, such as `structured_output_repair_attempts`. Require the existing
   contextual load failure. Inheritance's support for null does not make that
   field nullable. Reuse existing direct-input null coverage.
5. Use a small overlay fixture to demonstrate the sequencing: an overlay that
   changes the parent before expansion changes the child's inherited value;
   a child's own explicit replacement wins over the parent's value. Verify
   `extends: null` cancellation using a child with its own valid input binding,
   and verify omission of the directive in an overlay preserves the base
   relationship. An explicit override record for the parent value must be
   reflected in an inheriting child as well.
6. If any integration defect appears, correct Notarius's existing presence or
   conversion handling, using ConfigKit's metadata. Add no application
   inheritance evaluator. Refine `docs/config.md` to explain explicit
   zero/false/empty/null and cancellation without promising universal validity.

Acceptance criteria:

- Inherited and direct replacements retain their existing explicit-value
  semantics, including zero repair budgets and empty validator overrides.
- Existing null restrictions are neither relaxed nor expanded.
- Overlay/explicit-override composition precedes inheritance, and cancellation
  behaves as described by the pinned upstream contract.

Validation:

```sh
GOWORK=off go test ./internal/core/config
git diff --check
```

Prompt size: presence and precedence at one package boundary. This stage does
not change defaults or add CLI flags for arbitrary configuration overrides.

## Stage 4 — Verify Declaration-Relative Paths Through Inheritance

Goal: keep inherited resource paths anchored to the original declaration.

Inspect `document_paths.go`, `document_paths_test.go`, the original-path map in
`pipeline_document.go`, and sequence mutation recording in
`document_normalizer.go`. Consult ConfigKit's public `InputInfo`, `Origin`, and
`Source` contracts at v0.5.0.

Work:

1. Extend real-file tests using `t.TempDir()`: import parent and child pipeline
   definitions from different subdirectories beneath one configuration root.
   Give the parent pipeline a relative external reference and one reference
   inside an inherited ordered step. Assert both inherited paths resolve from
   the parent's file; a direct child reference replacement resolves from the
   child's file. Reference files may be minimal public fixture data.
2. Add an overlay replacement case where the parent reference is declared by
   the overlay and subsequently inherited. Its base must be the overlay's
   directory. A nested reference-mapping inheritance case must preserve its
   original contributor when copied into a derived pipeline.
3. Combine one inherited path case with whitespace-normalized pipeline or
   reference keys. The existing `originalPath` mechanism must still find the
   input's origin. ConfigKit exact sibling matching happens before Notarius
   trims names; do not pre-normalize the inheritance lookup.
4. Include a legitimate `../` reference, retaining the current host-path
   behavior. Reuse existing coverage for built-in destination bases, explicit
   CLI destination bases, absolute paths, and empty cache roots unless the
   upgrade reveals a distinct regression.
5. Fix `documentPathBase` or related origin handling only if these tests expose
   a failure. Query the surviving value's contributor, not the `extends`
   control location or an arbitrary sibling. Keep the existing host-path
   resolution model; do not replace it with resource opening through
   `ResourceContext.FS`, which could introduce containment restrictions.
6. Clarify inherited declaring-file behavior in `docs/config.md`. Update the
   loading/path explanation in `docs/internal/configuration.md` to reflect
   expansion before decoding and source-aware recorded normalization.

Acceptance criteria:

- Imported, nested, and overlaid inherited paths use the file supplying the
  surviving value; direct child replacements use their own declaring file.
- Normalized identifiers and inherited ordered-step references retain correct
  bases without unrecorded sequence mutation failures.
- Existing path access and destination policies are preserved. No new path
  containment, existence, or resource-format restriction is introduced.

Validation:

```sh
GOWORK=off go test ./internal/core/config
git diff --check
```

Prompt size: one path-resolution seam and focused real-file regressions.
Inspection rendering is checked separately in Stage 5.

## Stage 5 — Preserve Inspection, Derivation, And Ownership

Goal: prove inherited configuration remains correctly inspectable after
Notarius's recorded normalization and runtime conversion.

Inspect `loaded_config.go`, `loaded_config_test.go`, `document_normalizer.go`,
`redaction_test.go`, and the new inherited path fixtures. Review upstream
metadata/history APIs rather than inventing a new provenance format.

Work:

1. Through `LoadedConfig.WriteEffective`, check a derived pipeline's expanded
   values and resolved reference paths agree with its runtime configuration.
   Verify consumed `extends` directives are absent from the effective typed
   configuration. Assert selected values/paths, not the full YAML document.
2. Through `WriteExplanation`, inspect one inherited scalar and one inherited
   path that Notarius normalizes. Assert useful parent/source and inheritance
   evidence remains available alongside the final normalization result.
   Use structured upstream metadata within the configuration package for
   precise provenance checks, and minimal stable fragments for rendered text.
   Do not require a new Notarius renderer or exact diagnostic wording.
3. Add a synthetic credential-shaped option to a parent and inherit it into a
   child. Verify effective and explanation views redact its value, including
   inherited/displaced history where present. Use only fake secret strings and
   the existing sensitivity predicate; add no new credential rules.
4. Extend the existing ownership test: mutating a child's runtime nested map
   must not mutate the parent's runtime map or the frozen inspection snapshot.
   Retain the `RuntimeConfig()` detached-copy behavior without a new cloning
   mechanism unless the test demonstrates a conversion defect.
5. Update `docs/internal/configuration.md` to describe inherited explicit input,
   retained derivation, normalization, and the detached runtime boundary. Repair
   any affected stale loading terminology, such as calling a configuration
   overlay a profile. Link to `docs/config.md` for syntax and field rules.

Acceptance criteria:

- Effective paths agree with runtime paths; explanations retain useful source
  and transfer evidence after normalization.
- Existing redaction applies to inherited values and history.
- Parent, child, returned copies, and inspection snapshots have the established
  ownership behavior. Upstream inspection types stay private to configuration.

Validation:

```sh
GOWORK=off go test ./internal/core/config
git diff --check
```

Prompt size: configuration inspection and ownership, with focused tests and its
internal documentation.

## Stage 6 — Exercise CLI Selection And Failure Reporting

Goal: make derived pipelines first-class through the existing commands and
preserve the normal load-error path for invalid inheritance.

Inspect `internal/cli/configkit_commands_test.go`, `command_contract_test.go`,
and the configuration command/load helpers. Use the existing injected catalog
and fake collaborators; do not construct providers or make LLM calls.

Work:

1. Add a small valid parent/child fixture. Exercise `pipelines list` and
   `config validate --pipeline child`; assert the child can be selected by its
   own ID and listing remains sorted. Reuse existing command rendering tests
   rather than duplicating every output mode.
2. Exercise `config effective` and `config explain` for that child to prove CLI
   wiring reaches the safe upstream inspection views. Assert successful status
   and relevant inherited evidence, not a whole-output snapshot.
3. Add representative invalid load cases for a missing parent, a cycle, and an
   invalid directive type. Keep one invalid relationship in an unselected
   pipeline to prove selecting a different valid pipeline cannot hide it.
   Assert runtime-error status, no success output, and meaningful safe
   configuration diagnostics. Reuse package-level diagnostics for more detailed
   path checks; ConfigKit owns the full malformed-directive matrix.
4. Check exact-key behavior with a whitespace-bearing parent name: the matching
   declaration works and existing identifier normalization follows expansion.
   A mismatch should produce the upstream missing-parent failure. No automatic
   trimming, aliases, or application-specific inheritance restrictions are
   added.
5. Change production CLI code only if these tests expose a wiring defect.
   `docs/cli.md` may gain a short link to configuration inheritance where useful,
   but commands and flags remain the same.

Acceptance criteria:

- Derived pipelines use the ordinary selection, listing, validation, and
  inspection commands.
- Invalid relationships reach the existing load-error reporting path, even
  when their entries are not selected for execution.
- Read-only command tests require neither credentials nor provider work.

Validation:

```sh
GOWORK=off go test ./internal/cli ./internal/core/config
git diff --check
```

Prompt size: command-boundary behavior and representative error propagation.

## Stage 7 — Publish And Test The Development Pipeline Example

Goal: give operators one complete, runnable demonstration without duplicating
the full pipeline.

Inspect `examples/dnd-complete.config.yml`,
`examples/pipelines/dnd-session.yml`, `examples/overlays/development.yml`,
`internal/cli/example_contract_test.go`, and `maintainedExampleFiles` in
`internal/cli/production_contract_test.go`.

Work:

1. Add this sibling to `examples/pipelines/dnd-session.yml`:

   ```yaml
   dnd-session-development:
     extends: dnd-session
     structured_output_repair_attempts: 0
   ```

   Keep it beneath `pipelines`, beside the complete parent. Explain with a short
   comment that this development variant disables PromptKit structural repairs
   while inheriting the steps and configured producer-stage retry budgets.
   It uses the parent's existing `dnd-extraction` profile.
2. Update the complete example's expected pipeline IDs to include the child in
   sorted order. Keep the minimal example and existing operational overlay.
   Do not add a third standalone configuration or duplicate the parent's steps.
3. Extend maintained-example coverage to load, resolve, and materialize external
   references for both complete-example pipelines. Compare the child to the
   parent for step/lane composition and generated-reference bindings, and assert
   the selected zero structural-repair setting reaches eligible resolved
   bindings. Do not assert equal pipeline digests: their IDs/settings differ.
4. Update `TestMaintainedExamplesValidateEffectiveProfilesOffline` to validate
   every expected pipeline in each maintained example, including the new child,
   with credentials absent. Retain the existing two-root-example inventory.
5. Update `docs/config.md`'s maintained-example guidance to show selection of the
   variant and explain that operators may instead override `llm_profile` when
   supplying another profile. Link to the inherited fragment. Add only a short
   configuration link in `docs/consumers/dnd-pipeline.md` if useful; keep its
   complete parent workflow and artifact instructions intact.

Acceptance criteria:

- The existing complete root imports both valid pipelines, and the derived
  pipeline contains only the relationship and its intentional override.
- Both variants resolve and validate offline using the maintained profile,
  external references, generated bindings, and full module catalog.
- Maintained examples and navigation accurately explain the variant without
  confusing it with the operational overlay or PromptKit profile inheritance.

Validation:

```sh
GOWORK=off go test ./internal/cli -run 'TestMaintained'
GOWORK=off go test ./internal/core/config ./internal/cli
git diff --check
```

Prompt size: one example fragment, its existing test inventory, and focused
operator guidance. No full live extraction run is needed.

## Stage 8 — Consolidate Documentation And Verify The Complete Upgrade

Goal: finish with an internally consistent, validated upgrade ready for review.

Inspect all changed files, `docs/config.md`,
`docs/internal/configuration.md`, `docs/integrations/pkg-configkit.md`,
`docs/oss/attribution.md`, and affected CLI/consumer navigation. Compare against
the feature roadmap's acceptance criteria.

Work:

1. Confirm canonical ownership: configuration defines syntax/precedence/path
   rules; internals explain the adapter; integration documentation identifies
   the supported upstream contract; attribution records the imported version
   and notices. Remove duplicate definitions introduced during earlier stages.
2. Check the documentation covers the reserved key in every input mapping,
   exact sibling naming, normal validation of all definitions, replacement of
   step/validator sequences, source-relative inherited paths, explicit values,
   and cancellation. Do not describe deferred `LoadLayered`, filesystem source,
   mapping source, or underlay workflows as available Notarius features.
3. Search for stale ConfigKit v0.4.2/current-pin claims in active documentation
   and dependency files, ordinary reserved-key collisions, and confusing profile
   terminology. Leave historical releases and archived roadmaps intact. Verify
   relative document links locally and pinned external links where accessible;
   no new documentation-checking tool or recurring network gate is required.
4. Run the full offline default suite, vet, and native build. Build to a
   `mktemp -d` destination so no binary is left in the repository. Run
   `GOWORK=off go mod tidy -diff`; an unexpected diff must be resolved and
   reviewed rather than accepted silently.
5. Review the final implementation for unintended restrictions or guarantees:
   no new schema version, inheritance whitelist, abstract-template exemption,
   custom list merge, host-path restriction, configuration source tier,
   additional CLI knobs, or upstream types leaking into runtime contracts.
6. Report the implemented behavior, successful checks, and any concrete gaps.
   Keep roadmap retirement and release publication for a later request.

Acceptance criteria:

- Every required outcome in `configkit.md` is implemented and covered at the
  appropriate boundary, with no unexplained scope expansion.
- Maintained examples and canonical documents agree with the selected module
  and observed loading behavior. Dependency metadata is tidy.
- Full offline tests, vet, build, and relevant diff/link checks pass. Any
  unavailable optional external-link verification is reported accurately.

Validation:

```sh
GOWORK=off go test ./...
GOWORK=off go vet ./...
build_dir=$(mktemp -d)
GOWORK=off go build -o "$build_dir/notarius" ./cmd/notarius
GOWORK=off go mod tidy -diff
git diff --check
git status --short
```

These are development checks for this upgrade, not additional permanent release
gates. Do not repeat successful checks for unchanged content solely to satisfy
another handoff step; rerun the affected checks after any correction.

Prompt size: final review, small documentation consolidation, and aggregate
verification. A newly discovered substantial defect must be reported with its
evidence and required follow-up rather than folded into an unrelated refactor.

## Open Questions

None at this baseline. The dependency target, native inheritance semantics,
path policy, example override, documentation owners, and deferred features are
specified. Whether an adapter needs a small correction is determined by the
integration tests above, rather than a new product-policy decision.