notarius/docs/roadmap/implementation.md

ConfigKit Inheritance Implementation Plan

Purpose And Scope

Implement ConfigKit 0.5 And Configuration Inheritance. That feature roadmap owns the intended behavior and policy choices. This document owns the ordered implementation work. Each numbered stage is one prompt for a Luna coding agent; complete its acceptance criteria before starting the next.

The starting baseline pins ConfigKit v0.4.2 and PromptKit v0.11.2, uses ordinary configkit.Load, and accepts Notarius configuration version 5. Upgrade only ConfigKit to v0.5.0, source commit dc09959cca4e105f49ec316c6d4ba585bd2340bd. The release tag is annotated: use git rev-parse 'v0.5.0^{commit}' when verifying the source commit in a checkout.

The implementation adds supported native configuration inheritance and a maintained example. It does not require changing pipeline execution, artifact schemas, or release tooling. Release preparation and publication remain a separate task; this feature belongs in a future minor release.

Working Instructions For Every Stage

• Read docs/development.md, docs/policy/agents.md, and the feature roadmap. Use the development guide to select additional applicable policies, including architecture, testing, documentation, and OSS for the changes being made.
• Inspect the named code and tests before editing. Prefer the repository's code knowledge graph for code discovery. Inspect the pinned upstream release and public API when an integration detail needs confirmation; do not assume a sibling checkout's current branch equals the released version.
• Keep ordinary configkit.Load, typed defaults, and the existing private configuration boundary. ConfigKit handles extends before decoding; add no Extends field, second merge pass, application template registry, or opt-in flag. Accept valid native inheritance locations under existing domain rules.
• Make production changes only where a concrete integration failure requires them. A stage may consist of focused tests and documentation when the existing adapter already behaves correctly. Fix a regression rather than weakening an established expectation to accommodate the upgrade.
• Test Notarius's boundary with the real ConfigKit dependency and small, deterministic fixtures. Reuse existing helpers and coverage; do not replicate ConfigKit's complete test matrix, add diagnostic snapshots, or assert incidental output formatting. Keep the default suite offline and independent of credentials and model calls.
• Use GOWORK=off for Go commands so a sibling checkout cannot substitute for the pinned modules. Update canonical documentation as each behavior lands. Review the diff, run the stage's checks, and commit the coherent result under the agent policy. Record any unresolved failure in the handoff rather than marking that stage complete. Do not push or publish a release as part of this plan.

Stage 1 — Upgrade The Dependency And Its Notices

Completion: complete.

Goal: establish the released dependency baseline without redesigning the configuration adapter.

Inspect:

• go.mod, go.sum, and internal/core/config/load.go.
• docs/integrations/pkg-configkit.md and the ConfigKit entry in docs/oss/attribution.md.
• ConfigKit's pinned docs/releases/v0.5.0.md, go.mod, COPYRIGHT, and LICENSE; consult the sibling checkout with git show v0.5.0:<path>.

Work:

1. Run GOWORK=off go get gitea.maximumdirect.net/eric/ConfigKit@v0.5.0, then GOWORK=off go mod tidy. Review the selected dependency graph. Retain PromptKit v0.11.2 and unrelated pins unless module resolution requires a change; explain any such change in the handoff.
2. Search maintained configuration, fixtures, YAML tags, and documentation for ordinary uses of the key or entry name extends. For example, use rg -n '\bextends\b' examples internal docs assets. Classify any matches: new inheritance instructions are valid; ordinary field/key uses collide with the new reserved directive. Rename actual maintained collisions and their consumers together, without an alias or bypass. Do not rename the unrelated PromptKit base_profile setting.
3. Run focused existing tests. Address compile/API compatibility failures with the smallest necessary adapter changes. Native inheritance needs no registration or load option.
4. Update the integration document's pin and brief description to v0.5.0; replace the v0.4.2 patch-only explanation with the new composition boundary. Link to pinned upstream release/configuration documentation.
5. Update only ConfigKit's attribution version and exact source/notice links to the target commit, preserving the verified copyright and LGPL-3.0-or-later expression. Reuse the existing LGPL/GPL texts when unchanged. Do not refresh unrelated component attribution or policies.

Acceptance criteria:

• The declared and selected ConfigKit module is v0.5.0, without a workspace or local replace dependency.
• Existing configuration and CLI tests pass; maintained ordinary extends collisions, if any, have been resolved explicitly.
• Integration and attribution documents agree with the actual dependency and imported source commit. The adapter still uses ordinary Load.

Validation:

GOWORK=off go list -m gitea.maximumdirect.net/eric/ConfigKit
GOWORK=off go test ./internal/core/config ./internal/cli
git diff --check

Prompt size: one dependency upgrade, its notices, and demonstrated compatibility adjustments. Leave new inheritance coverage to the following stages.

Stage 2 — Prove Pipeline Composition And Runtime Conversion

Completion: complete.

Goal: establish whole-pipeline reuse through the real configuration boundary.

Inspect load.go, document.go, loaded_config.go, pipeline_document.go, document_test.go, pipeline_document_test.go, and effective_config_contract_test.go under internal/core/config. Reuse the existing test catalog for runtime resolution.

Work:

1. Add focused loading coverage in internal/core/config/inheritance_test.go. Define a valid parent pipeline and a child with extends: parent plus a pipeline-level override. Assert the inherited input, bindings, and ordered steps survive conversion, while the child retains its own pipeline ID and the parent keeps its own settings. Keep the fixture small rather than copying the complete D&D example.
2. Resolve the resulting child with the existing catalog. Assert its selected ID and overridden setting reach the resolved pipeline. Use the catalog's registered module keys and avoid generated-reference requirements in this small fixture; the full generated-reference handoff is covered in Stage 7. Reuse existing resolution tests for unrelated defaults and digests.
3. Add one replacement case: the child supplies a complete different steps list. Assert replacement and ordering, with no parent steps appended or merged by ID. Both resulting pipelines must satisfy ordinary validation.
4. Add one native nested-mapping case using sibling artifact lanes in an artifacts mapping: a child lane inherits a valid parent lane and overrides a binding value. This proves that Notarius does not limit inheritance to pipeline mappings. Other upstream-supported mapping shapes need no custom permission mechanism.
5. In docs/config.md, add the canonical inheritance explanation: exact immediate-sibling lookup, one parent, source composition before expansion, recursive mapping combination, sequence/scalar/null replacement, normal validation of parent and child, and all effective relationships being resolved. State that step-sequence entries cannot extend each other. Include a small illustrative pipeline snippet and the reserved extends migration note. Distinguish configuration overlays and PromptKit profiles.

Acceptance criteria:

• Loading and resolving a child yields an ordinary independently identified pipeline with the expected inherited composition and explicit override.
• Replacement lists retain only the child list, in its configured order.
• Valid nested artifact-lane inheritance works through native ConfigKit behavior without extra runtime types or application merging.
• Configuration version remains 5 and the user reference describes the implemented syntax and reserved-key compatibility impact.

Validation:

GOWORK=off go test ./internal/core/config
git diff --check

Prompt size: small composition fixtures, runtime handoff, and one canonical syntax section. Presence-sensitive cases belong to Stage 3.

Stage 3 — Preserve Presence And Existing Domain Validation

Completion: complete.

Goal: ensure inherited input remains explicit through Notarius normalization, defaulting, and validator-chain conversion.

Inspect operational_document.go, pipeline_document.go, document_normalizer.go, loaded_config.go, and their existing presence tests. Extend inheritance_test.go rather than adding a parallel fixture framework.

Work:

1. Cover inherited structured_output_repair_attempts: 0: it remains a populated zero override after loading and resolution, rather than reverting to a pipeline or prompt default. Also cover a child explicitly overriding a nonzero parent budget with zero.
2. Cover an inherited supported validators: [] on a chunk or extraction binding. Its runtime override retains Set=true and an empty chain. Cover a child replacing a nonempty parent chain with []; do not concatenate it or reinstall default validators because the child list is empty.
3. Cover an inherited false module-option value, such as output.options.evidence_context.enabled: false, remaining false in runtime configuration. Inspect the decoded document's input metadata to establish explicit presence where needed, rather than relying on truthiness.
4. Cover inherited or child-supplied null for a field already rejected when explicit, such as structured_output_repair_attempts. Require the existing contextual load failure. Inheritance's support for null does not make that field nullable. Reuse existing direct-input null coverage.
5. Use a small overlay fixture to demonstrate the sequencing: an overlay that changes the parent before expansion changes the child's inherited value; a child's own explicit replacement wins over the parent's value. Verify extends: null cancellation using a child with its own valid input binding, and verify omission of the directive in an overlay preserves the base relationship. An explicit override record for the parent value must be reflected in an inheriting child as well.
6. If any integration defect appears, correct Notarius's existing presence or conversion handling, using ConfigKit's metadata. Add no application inheritance evaluator. Refine docs/config.md to explain explicit zero/false/empty/null and cancellation without promising universal validity.

Acceptance criteria:

• Inherited and direct replacements retain their existing explicit-value semantics, including zero repair budgets and empty validator overrides.
• Existing null restrictions are neither relaxed nor expanded.
• Overlay/explicit-override composition precedes inheritance, and cancellation behaves as described by the pinned upstream contract.

Validation:

GOWORK=off go test ./internal/core/config
git diff --check

Prompt size: presence and precedence at one package boundary. This stage does not change defaults or add CLI flags for arbitrary configuration overrides.

Stage 4 — Verify Declaration-Relative Paths Through Inheritance

Completion: complete.

Goal: keep inherited resource paths anchored to the original declaration.

Inspect document_paths.go, document_paths_test.go, the original-path map in pipeline_document.go, and sequence mutation recording in document_normalizer.go. Consult ConfigKit's public InputInfo, Origin, and Source contracts at v0.5.0.

Work:

1. Extend real-file tests using t.TempDir(): import parent and child pipeline definitions from different subdirectories beneath one configuration root. Give the parent pipeline a relative external reference and one reference inside an inherited ordered step. Assert both inherited paths resolve from the parent's file; a direct child reference replacement resolves from the child's file. Reference files may be minimal public fixture data.
2. Add an overlay replacement case where the parent reference is declared by the overlay and subsequently inherited. Its base must be the overlay's directory. A nested reference-mapping inheritance case must preserve its original contributor when copied into a derived pipeline.
3. Combine one inherited path case with whitespace-normalized pipeline or reference keys. The existing originalPath mechanism must still find the input's origin. ConfigKit exact sibling matching happens before Notarius trims names; do not pre-normalize the inheritance lookup.
4. Include a legitimate ../ reference, retaining the current host-path behavior. Reuse existing coverage for built-in destination bases, explicit CLI destination bases, absolute paths, and empty cache roots unless the upgrade reveals a distinct regression.
5. Fix documentPathBase or related origin handling only if these tests expose a failure. Query the surviving value's contributor, not the extends control location or an arbitrary sibling. Keep the existing host-path resolution model; do not replace it with resource opening through ResourceContext.FS, which could introduce containment restrictions.
6. Clarify inherited declaring-file behavior in docs/config.md. Update the loading/path explanation in docs/internal/configuration.md to reflect expansion before decoding and source-aware recorded normalization.

Acceptance criteria:

• Imported, nested, and overlaid inherited paths use the file supplying the surviving value; direct child replacements use their own declaring file.
• Normalized identifiers and inherited ordered-step references retain correct bases without unrecorded sequence mutation failures.
• Existing path access and destination policies are preserved. No new path containment, existence, or resource-format restriction is introduced.

Validation:

GOWORK=off go test ./internal/core/config
git diff --check

Prompt size: one path-resolution seam and focused real-file regressions. Inspection rendering is checked separately in Stage 5.

Stage 5 — Preserve Inspection, Derivation, And Ownership

Completion: complete.

Goal: prove inherited configuration remains correctly inspectable after Notarius's recorded normalization and runtime conversion.

Inspect loaded_config.go, loaded_config_test.go, document_normalizer.go, redaction_test.go, and the new inherited path fixtures. Review upstream metadata/history APIs rather than inventing a new provenance format.

Work:

1. Through LoadedConfig.WriteEffective, check a derived pipeline's expanded values and resolved reference paths agree with its runtime configuration. Verify consumed extends directives are absent from the effective typed configuration. Assert selected values/paths, not the full YAML document.
2. Through WriteExplanation, inspect one inherited scalar and one inherited path that Notarius normalizes. Assert useful parent/source and inheritance evidence remains available alongside the final normalization result. Use structured upstream metadata within the configuration package for precise provenance checks, and minimal stable fragments for rendered text. Do not require a new Notarius renderer or exact diagnostic wording.
3. Add a synthetic credential-shaped option to a parent and inherit it into a child. Verify effective and explanation views redact its value, including inherited/displaced history where present. Use only fake secret strings and the existing sensitivity predicate; add no new credential rules.
4. Extend the existing ownership test: mutating a child's runtime nested map must not mutate the parent's runtime map or the frozen inspection snapshot. Retain the RuntimeConfig() detached-copy behavior without a new cloning mechanism unless the test demonstrates a conversion defect.
5. Update docs/internal/configuration.md to describe inherited explicit input, retained derivation, normalization, and the detached runtime boundary. Repair any affected stale loading terminology, such as calling a configuration overlay a profile. Link to docs/config.md for syntax and field rules.

Acceptance criteria:

• Effective paths agree with runtime paths; explanations retain useful source and transfer evidence after normalization.
• Existing redaction applies to inherited values and history.
• Parent, child, returned copies, and inspection snapshots have the established ownership behavior. Upstream inspection types stay private to configuration.

Validation:

GOWORK=off go test ./internal/core/config
git diff --check

Prompt size: configuration inspection and ownership, with focused tests and its internal documentation.

Stage 6 — Exercise CLI Selection And Failure Reporting

Completion: complete.

Goal: make derived pipelines first-class through the existing commands and preserve the normal load-error path for invalid inheritance.

Inspect internal/cli/configkit_commands_test.go, command_contract_test.go, and the configuration command/load helpers. Use the existing injected catalog and fake collaborators; do not construct providers or make LLM calls.

Work:

1. Add a small valid parent/child fixture. Exercise pipelines list and config validate --pipeline child; assert the child can be selected by its own ID and listing remains sorted. Reuse existing command rendering tests rather than duplicating every output mode.
2. Exercise config effective and config explain for that child to prove CLI wiring reaches the safe upstream inspection views. Assert successful status and relevant inherited evidence, not a whole-output snapshot.
3. Add representative invalid load cases for a missing parent, a cycle, and an invalid directive type. Keep one invalid relationship in an unselected pipeline to prove selecting a different valid pipeline cannot hide it. Assert runtime-error status, no success output, and meaningful safe configuration diagnostics. Reuse package-level diagnostics for more detailed path checks; ConfigKit owns the full malformed-directive matrix.
4. Check exact-key behavior with a whitespace-bearing parent name: the matching declaration works and existing identifier normalization follows expansion. A mismatch should produce the upstream missing-parent failure. No automatic trimming, aliases, or application-specific inheritance restrictions are added.
5. Change production CLI code only if these tests expose a wiring defect. docs/cli.md may gain a short link to configuration inheritance where useful, but commands and flags remain the same.

Acceptance criteria:

• Derived pipelines use the ordinary selection, listing, validation, and inspection commands.
• Invalid relationships reach the existing load-error reporting path, even when their entries are not selected for execution.
• Read-only command tests require neither credentials nor provider work.

Validation:

GOWORK=off go test ./internal/cli ./internal/core/config
git diff --check

Prompt size: command-boundary behavior and representative error propagation.

Stage 7 — Publish And Test The Development Pipeline Example

Completion: complete.

Goal: give operators one complete, runnable demonstration without duplicating the full pipeline.

Inspect examples/dnd-complete.config.yml, examples/pipelines/dnd-session.yml, examples/overlays/development.yml, internal/cli/example_contract_test.go, and maintainedExampleFiles in internal/cli/production_contract_test.go.

Work:

1. Add this sibling to examples/pipelines/dnd-session.yml:

   dnd-session-development:
     extends: dnd-session
     structured_output_repair_attempts: 0
   

   Keep it beneath pipelines, beside the complete parent. Explain with a short comment that this development variant disables PromptKit structural repairs while inheriting the steps and configured producer-stage retry budgets. It uses the parent's existing dnd-extraction profile.

2. Update the complete example's expected pipeline IDs to include the child in sorted order. Keep the minimal example and existing operational overlay. Do not add a third standalone configuration or duplicate the parent's steps.

3. Extend maintained-example coverage to load, resolve, and materialize external references for both complete-example pipelines. Compare the child to the parent for step/lane composition and generated-reference bindings, and assert the selected zero structural-repair setting reaches eligible resolved bindings. Do not assert equal pipeline digests: their IDs/settings differ.

4. Update TestMaintainedExamplesValidateEffectiveProfilesOffline to validate every expected pipeline in each maintained example, including the new child, with credentials absent. Retain the existing two-root-example inventory.

5. Update docs/config.md's maintained-example guidance to show selection of the variant and explain that operators may instead override llm_profile when supplying another profile. Link to the inherited fragment. Add only a short configuration link in docs/consumers/dnd-pipeline.md if useful; keep its complete parent workflow and artifact instructions intact.

Acceptance criteria:

• The existing complete root imports both valid pipelines, and the derived pipeline contains only the relationship and its intentional override.
• Both variants resolve and validate offline using the maintained profile, external references, generated bindings, and full module catalog.
• Maintained examples and navigation accurately explain the variant without confusing it with the operational overlay or PromptKit profile inheritance.

Validation:

GOWORK=off go test ./internal/cli -run 'TestMaintained'
GOWORK=off go test ./internal/core/config ./internal/cli
git diff --check

Prompt size: one example fragment, its existing test inventory, and focused operator guidance. No full live extraction run is needed.

Stage 8 — Consolidate Documentation And Verify The Complete Upgrade

Completion: complete.

Goal: finish with an internally consistent, validated upgrade ready for review.

Inspect all changed files, docs/config.md, docs/internal/configuration.md, docs/integrations/pkg-configkit.md, docs/oss/attribution.md, and affected CLI/consumer navigation. Compare against the feature roadmap's acceptance criteria.

Work:

1. Confirm canonical ownership: configuration defines syntax/precedence/path rules; internals explain the adapter; integration documentation identifies the supported upstream contract; attribution records the imported version and notices. Remove duplicate definitions introduced during earlier stages.
2. Check the documentation covers the reserved key in every input mapping, exact sibling naming, normal validation of all definitions, replacement of step/validator sequences, source-relative inherited paths, explicit values, and cancellation. Do not describe deferred LoadLayered, filesystem source, mapping source, or underlay workflows as available Notarius features.
3. Search for stale ConfigKit v0.4.2/current-pin claims in active documentation and dependency files, ordinary reserved-key collisions, and confusing profile terminology. Leave historical releases and archived roadmaps intact. Verify relative document links locally and pinned external links where accessible; no new documentation-checking tool or recurring network gate is required.
4. Run the full offline default suite, vet, and native build. Build to a mktemp -d destination so no binary is left in the repository. Run GOWORK=off go mod tidy -diff; an unexpected diff must be resolved and reviewed rather than accepted silently.
5. Review the final implementation for unintended restrictions or guarantees: no new schema version, inheritance whitelist, abstract-template exemption, custom list merge, host-path restriction, configuration source tier, additional CLI knobs, or upstream types leaking into runtime contracts.
6. Report the implemented behavior, successful checks, and any concrete gaps. Keep roadmap retirement and release publication for a later request.

Acceptance criteria:

• Every required outcome in configkit.md is implemented and covered at the appropriate boundary, with no unexplained scope expansion.
• Maintained examples and canonical documents agree with the selected module and observed loading behavior. Dependency metadata is tidy.
• Full offline tests, vet, build, and relevant diff/link checks pass. Any unavailable optional external-link verification is reported accurately.

Validation:

GOWORK=off go test ./...
GOWORK=off go vet ./...
build_dir=$(mktemp -d)
GOWORK=off go build -o "$build_dir/notarius" ./cmd/notarius
GOWORK=off go mod tidy -diff
git diff --check
git status --short

These are development checks for this upgrade, not additional permanent release gates. Do not repeat successful checks for unchanged content solely to satisfy another handoff step; rerun the affected checks after any correction.

Prompt size: final review, small documentation consolidation, and aggregate verification. A newly discovered substantial defect must be reported with its evidence and required follow-up rather than folded into an unrelated refactor.

Open Questions

None at this baseline. The dependency target, native inheritance semantics, path policy, example override, documentation owners, and deferred features are specified. Whether an adapter needs a small correction is determined by the integration tests above, rather than a new product-policy decision.