ConfigKit/docs/roadmap/implementation.md

Sibling Inheritance Implementation Plan

Scope and working rules

Implement the agreed named-inheritance roadmap, starting from 01e38b5. The roadmap owns feature policy; this document owns execution and the concrete implementation choices below. Follow stages in order. Each stage is a focused task for a gpt-5.6-terra coding agent and should end with relevant passing checks and a plain-English commit. Do not publish or tag a release during this sprint.

Read docs/development.md, docs/policy/agents.md, and task-relevant policies. Preserve unrelated work. The root owns public types; the private engine owns processing. Reuse the parser-independent document representation and existing helpers. Do not add dependencies, configurable keywords, registration APIs, merge strategies, background work, or consumer-specific semantics.

Stages 1–6 develop private machinery without activating inheritance in public loading. Stage 7 connects the complete pipeline. Intermediate commits must not publish resolved values with incomplete provenance or redaction. Private names and file splits below are guidance, not new public contracts.

Decisions carried into implementation

• extends is reserved input syntax. A string names an exact sibling mapping; null cancels a relationship; omission retains lower contributions. Other types, missing/non-mapping parents, self-reference, and cycles are errors. Empty strings are exact names. Root and direct sequence-element mappings cannot use string references; null needs no lookup.
• Strict composition still rejects overlapping non-mapping contributions, including duplicate extends definitions. Complete all source-tier and CLI precedence before expanding the effective input. Overwritten earlier references are historical evidence, not active relationships to validate.
• Expand enclosing parent chains before resolving nested relationships in each destination. Reusable parent templates retain unresolved nested instructions. Sort mapping keys lexicographically, follow parent dependencies as needed, and visit sequences by index. Sorting never determines value precedence.
• Parent-to-child combination uses recursive overlay, including explicit zero/false/empty/null replacement and whole-sequence replacement. Consume the current mapping's own relationship during enclosing expansion; nested relationships travel with nested payload until their scope is resolved.
• Inherited values count as explicit input for presence/defaulting queries. Provenance distinguishes direct input from inherited input.
• Version checking stays at its existing main-input point. Expansion follows source assembly and precedes defaults/decoding, recorded resolution, secret lookup, validation, and publication. Invalid effective relationships fail even in entries the application does not select for execution.
• Preserve source records and passive resource contexts. Inheritance is neither a new acquired source nor another precedence layer.

Concrete metadata representation

Extend engine and root Origin with Inheritance []InheritanceStep. Define root-owned and engine counterpart records with From Path, To Path, Reference Path, and Locations []Location. A step describes a payload-path transfer. Reference is the destination's controlling extends path; Locations identifies that directive's contributing source coordinates. Paths are concrete semantic paths, not names requiring another lookup. Records contain no values.

Keep Origin.Layer and Origin.Locations for actual value contributors. Order steps by derivation; composite origins retain contributing branches rather than inventing a single chain. Project concrete descendant paths when queried. Keep original contributor paths privately alongside locations, so synthetic sources do not depend on file coordinates to identify original semantic paths. Direct replacement keeps its own effective origin; displaced inheritance stays in history.

Add inspection-only ChangeInheritance = "inheritance", like ChangeDecoding; do not accept it through Resolution.Record. Inheritance events identify parent/control dependencies through existing Change, ValueChange, and Explanation surfaces. No separate public query family is needed. Effective inherited origins retain source layers and transfer evidence; expansion history events use the new operation kind.

Budgets and diagnostics

Keep acquired-input accounting unchanged. Use defaultProjectionLimits() for expanded tree depth/nodes, counted by logical materialized nodes rather than unique pointers; report CodeSchemaProjectionLimit. Check during materialization before uncontrolled allocation and typed decoding. Do not charge copies as acquired input or add a chain-length cap.

Extend historyNodesWithin to retained inheritance templates and snapshots, counting shared immutable document nodes once under the existing historyNodeLimit; report CodeExplainLimitExceeded. Compact relationship records refer to shared evidence instead of copying a full chain/tree per field. Do not redefine node limits as byte limits or introduce a new public limit. Check retained growth as it is accumulated and release temporary templates. Use private small test budgets.

Add mirrored engine/root codes: CodeInheritanceInvalid (inheritance.invalid) for invalid type/position or non-mapping parent; CodeInheritanceMissing (inheritance.missing); CodeInheritanceCycle (inheritance.cycle); and CodeInheritanceVersion (inheritance.version). Use directive paths/locations as primary evidence and related definition locations for dependencies/cycles. Messages must not interpolate raw reference strings or payload. Existing syntax, composition, decode, and budget codes retain their meanings.

Stage 1 — Prepare shared recursive combination

Completed 2026-09-29. Recursive mapping overlay now uses a private value-combination walk with path-aware add, combine, and replacement callbacks; the existing overlay adapter continues to own source origins and history, while strict composition remains separate. A focused table covers recursive and atomic replacements, empty mappings, event paths, and detached inputs. Focused composition/overlay/override/underlay tests and ./scripts/check.sh passed.

Inspect internal/engine/overlay.go, compose.go, mapping_index.go, history.go, and internal/document/document.go. applyOverlay already shares mergeOverlayMappings across underlays, main-over-underlay assembly, and selected overlays. Strict composition uses mergeDisjointMappings after claim validation; history replay has a separate value-only traversal.

Extract or adapt the smallest private recursive-overlay primitive needed for arbitrary mapping roots, preserving ordered entries and indexed lookup. Keep source-tier bookkeeping in its adapter. Provide path/replacement information for inheritance's later adapter; do not model inheritance as a fake layer. Reuse document.Clone and mappingIndex. Keep strict composition distinct. Consolidate value-only replay only if doing so preserves its behavior without extra machinery; do not build a public or general-purpose merge framework.

Validation: existing compose/overlay/override and underlay tests, plus a compact table covering recursion, atomic sequence/scalar/null replacement, empty mappings, and detached ownership. Existing behavioral expectations should not change. Run ./scripts/check.sh before committing.

Stage 2 — Expand one sibling scope

Completed 2026-09-29. A private, non-activated resolver indexes sibling definitions, follows parent dependencies with an explicit stack, memoizes unresolved enclosing templates, consumes each mapping's own relationship, and retains transfer/control evidence. Safe diagnostics cover missing or non-mapping parents, invalid reference types, and cycles. Focused tests include forward and backward references, shared chains, exact unusual names, null cancellation, independent results, and a deep cycle. ./scripts/check.sh passed.

Add a private inheritance resolver, callable only from engine tests initially. Index effective sibling definitions. Resolve single-parent dependencies with visiting/resolved states and memoized enclosing-chain templates. Use an explicit work stack so graph chain length is not tied to the Go call stack. Overlay parent payload with child payload using Stage 1. Consume the current mapping's own extends, leaving nested instructions intact. Handle null before lookup.

Implement internal diagnostic codes and deterministic traversal. Preserve locations and collect transfer/control evidence for Stage 4. Keep original input unchanged and templates private; a child must not mutate its parent. Failed expansion returns diagnostics and no usable expanded result.

Validation: forward/backward references, chains, shared parents, child replacement, null cancellation, exact unusual/empty names, missing/non-mapping parents, invalid types, self/long cycles, and input independence. Assert results and diagnostic fields, not stack layout or exact prose. Run focused tests and the everyday checker.

Stage 3 — Resolve nested destination scopes and bound expansion

Completed 2026-09-29. Private expansion now walks from enclosing mappings into destination-local nested scopes, including mappings within sequences. Root and direct sequence-element string references fail; null cancellation is consumed. Logical materialization uses an early non-directive lower bound and final node/depth checks under projection limits, with no global cache. Focused tests cover destination-specific nested values, three-generation chains, relationship replacement/cancellation, a destination-only cycle, sequence scopes, invalid positions, and budget boundaries. ./scripts/check.sh passed.

Extend to the complete effective tree. Distinguish enclosing-chain templates from fully resolved destination subtrees. After materializing an enclosing mapping, resolve its nested sibling scopes independently. Visit sequence-contained mappings; reject string references on root/direct sequence elements without ancestor lookup.

Scope memoization to each effective destination namespace. Copying a template must not copy its already-resolved nested children. Preserve inherited nested control evidence and support cancellation through a child's nested null directive. Whole- container replacement removes those active relationships at that destination; the parent's own surviving relationships still require validation.

Apply the selected expanded-tree budgets, counting logical output from shared templates and releasing transient state. No global caches or unrelated-scope lookup.

Validation: the roadmap's nested primary/secondary example; three-generation enclosing chains; nested reference replacement/cancellation; destination-only cycles; scopes inside lists; root/list-element errors; and parent names sorting before or after children. Exercise small node/depth budgets at and beyond their boundaries. Run the everyday checker.

Stage 4 — Preserve derivation, presence, and resource origins

Completed 2026-09-29. Private expansion now returns the effective tree, per-path origins, shared linked transfer traces, control records, and displaced replacement evidence. Contributor records retain original semantic paths and source locations through chains, nested destination scopes, composite mappings, and sequence descendants; direct child replacements have independent effective origins. Clones detach mutable slices and paths, and origin containment checks include derivation. Focused source/context, provenance, replacement, and detachment tests and ./scripts/check.sh passed. Public loading remains unchanged.

Implement the selected metadata representation privately in the engine. Carry original semantic contributor paths, locations/layers, transfer edges, and control evidence through copies and recursive combination. Preserve composite origins and sequence-descendant origins, including atomic sequence contributions. Do not infer ownership from the last matching layer: multiple sources can share a layer.

Return an expanded tree, origins, and inheritance evidence for the decoded metadata pipeline. Presence comes from expanded input. Direct replacement removes effective inheritance at replaced paths but retains displaced evidence. Extend cloning, comparison/deduplication, and private carriers; share immutable internal evidence while detaching query outputs.

Validation: inherited zero/false/null presence, chains and nested transfers, composites, same-layer multiple sources, and replacement evidence. Demonstrate that an inherited relative resource points to its original source context rather than the child's directory. Inheritance does not open referenced resources. Run the everyday checker.

Stage 5 — Integrate historical inspection

Extend history.go, load_metadata.go, explain.go, and explanation rendering. Retain original source applications; represent structural expansion separately, between source replay and canonical decoding, followed by recorded application resolution. Do not attribute inherited values or consumed directives to decoding.

Destination history must show parent contribution and child replacement when both exist, even if expansion leaves the effective child value unchanged. Include derivation dependencies, preserve source history at original paths, and make chains navigable through concrete paths. Do not fabricate repeated chronological source loading at every child. Explain consumed directives as historical control input absent from effective metadata/configuration. Preserve cancellation and superseded directive evidence.

Extend historical path enumeration, ancestor replacement, origin comparison, and retained-node accounting. Keep typed reconciliation separate. Share scope evidence and project relevant changes instead of storing a full snapshot per leaf.

Validation: inherited values, equal child replacements, displaced inherited subtrees, parent source changes, consumed/canceled directives, destination-specific nested results, and subsequent application-derived replacement. Check agreement among final values, History, Explain, and historical paths. Exercise small history budgets. Run the everyday checker.

Stage 6 — Preserve sensitivity through inheritance

Connect transfer dependencies to existing secret classification and historical redaction. Classify both source and destination paths under schema and Definition.Secret rules; propagate sensitivity through chains/nested copies. Include displaced evidence and control history. Preserve existing conservative historical redaction without declaring unrelated siblings secret.

Copy unresolved secret references only; lookup remains at the existing lifecycle point. Credentials never enter expansion trees/history. Independent replacements retain their own effective provenance while previous secrets remain redacted. Queries and renderers must agree.

Validation: source-only and destination-only classification, shared parents, transitive inheritance, replacement/removal, and lookup after expansion. Use disposable canaries across affected inspection/failure surfaces. Avoid real credentials and parallel environment mutation. Run focused secret/history tests and the everyday checker.

Stage 7 — Activate the common lifecycle and facade metadata

Insert expansion in executeLoadRequest after ordinary applyInputLayers or layered assembleLayeredInput, before decodeLoadInput invokes defaults/decoding. Include plain single-source loads, creating the minimal composition carrier needed to preserve original history. Update load_result.go and inspection carriers with Stages 4–6 evidence. Preserve no-input behavior and use a no-directive fast path where practical. Do not repeat acquisition, version callbacks, or user resolution.

Keep the version path main-owned. Detect inherited contributions, replacement, or directive removal affecting it, including protected ancestor replacement; report CodeInheritanceVersion. Do not reject an entire enclosing mapping merely because it contains an unchanged version field. A version rule targeting an extends field cannot silently lose its checked declaration when the directive is consumed. Keep the checker at its original point and retain accepted version representations.

Mirror records/codes in root records.go, conversions, clones, and API comments. Ensure root Origin.Inheritance and InheritanceStep expose only root types and detached slices/paths. Keep ChangeInheritance inspection-only. Update root/engine IsExplicit comments, including resolver-entry queries. Preserve SourceContext.

Do not treat extends as the acquisition-time configkit directive. Imports, selected overlays, Mapping input, and CLI overrides carry it through combination until expansion. Keep existing configkit validation and CLI path restrictions. Typed defaults and values added by application resolution do not initiate another structural pass; inheritance processes assembled source input only.

Validation: public ordinary/layered loads; imports and strict conflicts; multiple underlays; selected overlay and CLI changes to parent/child/reference; resolver defaulting using inherited presence; version ownership and callback timing; typed unknown-field behavior after consumption; no later hooks on expansion failure; partial bases with normal validation; and no-directive regressions. Include a filesystem resource-origin example and mapping/bytes equivalence case, not a full duplication of every mechanism test. Run ./scripts/check.sh and go test -race ./... for ownership changes.

Stage 8 — Validate scaling and complete documentation

Add focused benchmarks beside assembly/underlay benchmarks: long chains, wide shared-parent scopes, nested destination expansion, and replacements. Evaluate allocations/work against input, materialized output, and retained history, including sorting costs. Fix repeated ancestor work or whole-tree copying without changing semantics. No unconditional O(n) promise or timing release gate.

Add a maintained consumer example showing one-load inheritance, explicit false, nested destination behavior, and source/derivation inspection. Update docs/config.md as behavior owner, root comments as API owners, README/consumer guidance as summaries, and internal overview/development test map as navigation. Update architecture/deferred-scope descriptions for implemented inheritance. Explain reserved-key migration, null cancellation, no registration, sibling lookup, source-first ordering, presence versus provenance, and unchanged version/lifecycle policy. No release-policy or platform-gate changes are part of this feature.

Write the next available ADR (currently 0009) recording ConfigKit's chosen architecture and rationale. Keep downstream-specific discussion out; rejected choices belong only in Alternatives Considered. Do not edit accepted ADR decision bodies or silently accept earlier drafts. Mark the feature roadmap's implementation status accurately and link the implemented contract. Release notes/tagging remain separate release-preparation work.

Run the everyday checker, reuse race evidence if ownership code is unchanged, and run benchmarks for evidence without pass/fail thresholds. Review the final diff against the roadmap, source equivalence, confidentiality, and public boundary. Confirm no independent merge rule, registration API, extra release gate, or background/resource lifecycle was introduced. Commit and report validation and any remaining limitations.