ConfigKit/docs/roadmap/implementation.md

# Sibling Inheritance Implementation Plan

## Scope and working rules

Implement the agreed [named-inheritance roadmap](named-inheritance.md), starting
from `01e38b5`. The roadmap owns feature policy; this document owns execution and
the concrete implementation choices below. Follow stages in order. Each stage is
a focused task for a gpt-5.6-terra coding agent and should end with relevant
passing checks and a plain-English commit. Do not publish or tag a release during
this sprint.

Read `docs/development.md`, `docs/policy/agents.md`, and task-relevant policies.
Preserve unrelated work. The root owns public types; the private engine owns
processing. Reuse the parser-independent document representation and existing
helpers. Do not add dependencies, configurable keywords, registration APIs,
merge strategies, background work, or consumer-specific semantics.

Stages 1–6 develop private machinery without activating inheritance in public
loading. Stage 7 connects the complete pipeline. Intermediate commits must not
publish resolved values with incomplete provenance or redaction. Private names
and file splits below are guidance, not new public contracts.

## Decisions carried into implementation

- `extends` is reserved input syntax. A string names an exact sibling mapping;
  null cancels a relationship; omission retains lower contributions. Other types,
  missing/non-mapping parents, self-reference, and cycles are errors. Empty strings
  are exact names. Root and direct sequence-element mappings cannot use string
  references; null needs no lookup.
- Strict composition still rejects overlapping non-mapping contributions,
  including duplicate `extends` definitions. Complete all source-tier and CLI
  precedence before expanding the effective input. Overwritten earlier references
  are historical evidence, not active relationships to validate.
- Expand enclosing parent chains before resolving nested relationships in each
  destination. Reusable parent templates retain unresolved nested instructions.
  Sort mapping keys lexicographically, follow parent dependencies as needed, and
  visit sequences by index. Sorting never determines value precedence.
- Parent-to-child combination uses recursive overlay, including explicit
  zero/false/empty/null replacement and whole-sequence replacement. Consume the
  current mapping's own relationship during enclosing expansion; nested
  relationships travel with nested payload until their scope is resolved.
- Inherited values count as explicit input for presence/defaulting queries.
  Provenance distinguishes direct input from inherited input.
- Version checking stays at its existing main-input point. Expansion follows
  source assembly and precedes defaults/decoding, recorded resolution, secret
  lookup, validation, and publication. Invalid effective relationships fail even
  in entries the application does not select for execution.
- Preserve source records and passive resource contexts. Inheritance is neither
  a new acquired source nor another precedence layer.

### Concrete metadata representation

Extend engine and root `Origin` with `Inheritance []InheritanceStep`. Define
root-owned and engine counterpart records with `From Path`, `To Path`,
`Reference Path`, and `Locations []Location`. A step describes a payload-path
transfer. `Reference` is the destination's controlling `extends` path; `Locations`
identifies that directive's contributing source coordinates. Paths are concrete
semantic paths, not names requiring another lookup. Records contain no values.

Keep `Origin.Layer` and `Origin.Locations` for actual value contributors. Order
steps by derivation; composite origins retain contributing branches rather than
inventing a single chain. Project concrete descendant paths when queried. Keep
original contributor paths privately alongside locations, so synthetic sources
do not depend on file coordinates to identify original semantic paths. Direct
replacement keeps its own effective origin; displaced inheritance stays in history.

Add inspection-only `ChangeInheritance = "inheritance"`, like `ChangeDecoding`;
do not accept it through `Resolution.Record`. Inheritance events identify
parent/control dependencies through existing `Change`, `ValueChange`, and
`Explanation` surfaces. No separate public query family is needed. Effective
inherited origins retain source layers and transfer evidence; expansion history
events use the new operation kind.

### Budgets and diagnostics

Keep acquired-input accounting unchanged. Use `defaultProjectionLimits()` for
expanded tree depth/nodes, counted by logical materialized nodes rather than
unique pointers; report `CodeSchemaProjectionLimit`. Check during materialization
before uncontrolled allocation and typed decoding. Do not charge copies as
acquired input or add a chain-length cap.

Extend `historyNodesWithin` to retained inheritance templates and snapshots,
counting shared immutable document nodes once under the existing `historyNodeLimit`;
report `CodeExplainLimitExceeded`. Compact relationship records refer to shared
evidence instead of copying a full chain/tree per field. Do not redefine node
limits as byte limits or introduce a new public limit. Check retained growth as
it is accumulated and release temporary templates. Use private small test budgets.

Add mirrored engine/root codes: `CodeInheritanceInvalid` (`inheritance.invalid`)
for invalid type/position or non-mapping parent; `CodeInheritanceMissing`
(`inheritance.missing`); `CodeInheritanceCycle` (`inheritance.cycle`); and
`CodeInheritanceVersion` (`inheritance.version`). Use directive paths/locations
as primary evidence and related definition locations for dependencies/cycles.
Messages must not interpolate raw reference strings or payload. Existing syntax,
composition, decode, and budget codes retain their meanings.

## Stage 1 — Prepare shared recursive combination

**Completed 2026-09-29.** Recursive mapping overlay now uses a private
value-combination walk with path-aware add, combine, and replacement callbacks;
the existing overlay adapter continues to own source origins and history, while
strict composition remains separate. A focused table covers recursive and
atomic replacements, empty mappings, event paths, and detached inputs. Focused
composition/overlay/override/underlay tests and `./scripts/check.sh` passed.

Inspect `internal/engine/overlay.go`, `compose.go`, `mapping_index.go`,
`history.go`, and `internal/document/document.go`. `applyOverlay` already shares
`mergeOverlayMappings` across underlays, main-over-underlay assembly, and selected
overlays. Strict composition uses `mergeDisjointMappings` after claim validation;
history replay has a separate value-only traversal.

Extract or adapt the smallest private recursive-overlay primitive needed for
arbitrary mapping roots, preserving ordered entries and indexed lookup. Keep
source-tier bookkeeping in its adapter. Provide path/replacement information for
inheritance's later adapter; do not model inheritance as a fake layer. Reuse
`document.Clone` and `mappingIndex`. Keep strict composition distinct. Consolidate
value-only replay only if doing so preserves its behavior without extra machinery;
do not build a public or general-purpose merge framework.

Validation: existing compose/overlay/override and underlay tests, plus a compact
table covering recursion, atomic sequence/scalar/null replacement, empty mappings,
and detached ownership. Existing behavioral expectations should not change.
Run `./scripts/check.sh` before committing.

## Stage 2 — Expand one sibling scope

**Completed 2026-09-29.** A private, non-activated resolver indexes sibling
definitions, follows parent dependencies with an explicit stack, memoizes
unresolved enclosing templates, consumes each mapping's own relationship, and
retains transfer/control evidence. Safe diagnostics cover missing or non-mapping
parents, invalid reference types, and cycles. Focused tests include forward and
backward references, shared chains, exact unusual names, null cancellation,
independent results, and a deep cycle. `./scripts/check.sh` passed.

Add a private inheritance resolver, callable only from engine tests initially.
Index effective sibling definitions. Resolve single-parent dependencies with
visiting/resolved states and memoized enclosing-chain templates. Use an explicit
work stack so graph chain length is not tied to the Go call stack. Overlay parent
payload with child payload using Stage 1. Consume the current mapping's own
`extends`, leaving nested instructions intact. Handle null before lookup.

Implement internal diagnostic codes and deterministic traversal. Preserve
locations and collect transfer/control evidence for Stage 4. Keep original input
unchanged and templates private; a child must not mutate its parent. Failed
expansion returns diagnostics and no usable expanded result.

Validation: forward/backward references, chains, shared parents, child replacement,
null cancellation, exact unusual/empty names, missing/non-mapping parents, invalid
types, self/long cycles, and input independence. Assert results and diagnostic
fields, not stack layout or exact prose. Run focused tests and the everyday checker.

## Stage 3 — Resolve nested destination scopes and bound expansion

**Completed 2026-09-29.** Private expansion now walks from enclosing mappings
into destination-local nested scopes, including mappings within sequences.
Root and direct sequence-element string references fail; null cancellation is
consumed. Logical materialization uses an early non-directive lower bound and
final node/depth checks under projection limits, with no global cache. Focused
tests cover destination-specific nested values, three-generation chains,
relationship replacement/cancellation, a destination-only cycle, sequence
scopes, invalid positions, and budget boundaries. `./scripts/check.sh` passed.

Extend to the complete effective tree. Distinguish enclosing-chain templates from
fully resolved destination subtrees. After materializing an enclosing mapping,
resolve its nested sibling scopes independently. Visit sequence-contained mappings;
reject string references on root/direct sequence elements without ancestor lookup.

Scope memoization to each effective destination namespace. Copying a template must
not copy its already-resolved nested children. Preserve inherited nested control
evidence and support cancellation through a child's nested null directive. Whole-
container replacement removes those active relationships at that destination;
the parent's own surviving relationships still require validation.

Apply the selected expanded-tree budgets, counting logical output from shared
templates and releasing transient state. No global caches or unrelated-scope lookup.

Validation: the roadmap's nested `primary`/`secondary` example; three-generation
enclosing chains; nested reference replacement/cancellation; destination-only
cycles; scopes inside lists; root/list-element errors; and parent names sorting
before or after children. Exercise small node/depth budgets at and beyond their
boundaries. Run the everyday checker.

## Stage 4 — Preserve derivation, presence, and resource origins

**Completed 2026-09-29.** Private expansion now returns the effective tree,
per-path origins, shared linked transfer traces, control records, and displaced
replacement evidence. Contributor records retain original semantic paths and
source locations through chains, nested destination scopes, composite mappings,
and sequence descendants; direct child replacements have independent effective
origins. Clones detach mutable slices and paths, and origin containment checks
include derivation. Focused source/context, provenance, replacement, and
detachment tests and `./scripts/check.sh` passed. Public loading remains unchanged.

Implement the selected metadata representation privately in the engine. Carry
original semantic contributor paths, locations/layers, transfer edges, and control
evidence through copies and recursive combination. Preserve composite origins and
sequence-descendant origins, including atomic sequence contributions. Do not infer
ownership from the last matching layer: multiple sources can share a layer.

Return an expanded tree, origins, and inheritance evidence for the decoded metadata
pipeline. Presence comes from expanded input. Direct replacement removes effective
inheritance at replaced paths but retains displaced evidence. Extend cloning,
comparison/deduplication, and private carriers; share immutable internal evidence
while detaching query outputs.

Validation: inherited zero/false/null presence, chains and nested transfers,
composites, same-layer multiple sources, and replacement evidence. Demonstrate
that an inherited relative resource points to its original source context rather
than the child's directory. Inheritance does not open referenced resources.
Run the everyday checker.

## Stage 5 — Integrate historical inspection

**Completed 2026-09-29.** Source replay stays at original paths, followed by
projected structural control/transfer events before typed reconciliation and
application resolution. History and explanations preserve consumed/cancelled
directives, equal-valued child replacements, displaced descendants, nested
destination results, and parent source updates. Shared retained evidence is
counted with a bounded incremental check; focused tests and `./scripts/check.sh`
passed. Public loading remains unchanged.

Extend `history.go`, `load_metadata.go`, `explain.go`, and explanation rendering.
Retain original source applications; represent structural expansion separately,
between source replay and canonical decoding, followed by recorded application
resolution. Do not attribute inherited values or consumed directives to decoding.

Destination history must show parent contribution and child replacement when both
exist, even if expansion leaves the effective child value unchanged. Include
derivation dependencies, preserve source history at original paths, and make
chains navigable through concrete paths. Do not fabricate repeated chronological
source loading at every child. Explain consumed directives as historical control
input absent from effective metadata/configuration. Preserve cancellation and
superseded directive evidence.

Extend historical path enumeration, ancestor replacement, origin comparison, and
retained-node accounting. Keep typed reconciliation separate. Share scope evidence
and project relevant changes instead of storing a full snapshot per leaf.

Validation: inherited values, equal child replacements, displaced inherited
subtrees, parent source changes, consumed/canceled directives, destination-specific
nested results, and subsequent application-derived replacement. Check agreement
among final values, `History`, `Explain`, and historical paths. Exercise small
history budgets. Run the everyday checker.

## Stage 6 — Preserve sensitivity through inheritance

**Completed 2026-09-29.** Sensitivity classification now follows corresponding
source/destination paths through structural transfers and controls to a fixed
point, while leaving neighboring fields unclassified. Historical replacement
evidence remains redacted, and inherited secret references remain unresolved
until the existing typed lookup point. Focused secret/history tests and
`./scripts/check.sh` passed. Public loading remains unchanged.

Connect transfer dependencies to existing secret classification and historical
redaction. Classify both source and destination paths under schema and
`Definition.Secret` rules; propagate sensitivity through chains/nested copies.
Include displaced evidence and control history. Preserve existing conservative
historical redaction without declaring unrelated siblings secret.

Copy unresolved secret references only; lookup remains at the existing lifecycle
point. Credentials never enter expansion trees/history. Independent replacements
retain their own effective provenance while previous secrets remain redacted.
Queries and renderers must agree.

Validation: source-only and destination-only classification, shared parents,
transitive inheritance, replacement/removal, and lookup after expansion. Use
disposable canaries across affected inspection/failure surfaces. Avoid real
credentials and parallel environment mutation. Run focused secret/history tests
and the everyday checker.

## Stage 7 — Activate the common lifecycle and facade metadata

**Completed 2026-09-29.** Ordinary and layered loads now expand assembled
input before decoding while retaining original source applications, provenance,
history, sensitivity, and resource contexts. Main-owned version paths reject
structural changes without rejecting unrelated inheritance. Root records and
conversions expose detached transfer evidence; public tests cover precedence,
presence, resource context, version timing, cancellation, imports, and secrets.
`./scripts/check.sh` and `go test -race ./...` passed.

Insert expansion in `executeLoadRequest` after ordinary `applyInputLayers` or
layered `assembleLayeredInput`, before `decodeLoadInput` invokes defaults/decoding.
Include plain single-source loads, creating the minimal composition carrier needed
to preserve original history. Update `load_result.go` and inspection carriers with
Stages 4–6 evidence. Preserve no-input behavior and use a no-directive fast path
where practical. Do not repeat acquisition, version callbacks, or user resolution.

Keep the version path main-owned. Detect inherited contributions, replacement, or
directive removal affecting it, including protected ancestor replacement; report
`CodeInheritanceVersion`. Do not reject an entire enclosing mapping merely because
it contains an unchanged version field. A version rule targeting an `extends`
field cannot silently lose its checked declaration when the directive is consumed.
Keep the checker at its original point and retain accepted version representations.

Mirror records/codes in root `records.go`, conversions, clones, and API comments.
Ensure root `Origin.Inheritance` and `InheritanceStep` expose only root types and
detached slices/paths. Keep `ChangeInheritance` inspection-only. Update root/engine
`IsExplicit` comments, including resolver-entry queries. Preserve `SourceContext`.

Do not treat `extends` as the acquisition-time `configkit` directive. Imports,
selected overlays, Mapping input, and CLI overrides carry it through combination
until expansion. Keep existing `configkit` validation and CLI path restrictions.
Typed defaults and values added by application resolution do not initiate another
structural pass; inheritance processes assembled source input only.

Validation: public ordinary/layered loads; imports and strict conflicts; multiple
underlays; selected overlay and CLI changes to parent/child/reference; resolver
defaulting using inherited presence; version ownership and callback timing;
typed unknown-field behavior after consumption; no later hooks on expansion
failure; partial bases with normal validation; and no-directive regressions.
Include a filesystem resource-origin example and mapping/bytes equivalence case,
not a full duplication of every mechanism test. Run `./scripts/check.sh` and
`go test -race ./...` for ownership changes.

## Stage 8 — Validate scaling and complete documentation

**Completed 2026-09-29.** Benchmarks cover chains, shared-parent width, nested
destinations, replacements, and full inspected loads. Skipping repeated scope
assembly when no immediate relationship exists reduced measured work while a
single boundary clone preserves caller-input ownership. The maintained preset
example, configuration contract, consumer guidance, internal navigation,
architecture/deferred scope, and ADR 0009 now describe the implemented feature.
`./scripts/check.sh`, `go test -race ./...`, and the focused benchmarks passed.
No release was tagged or published.

Add focused benchmarks beside assembly/underlay benchmarks: long chains, wide
shared-parent scopes, nested destination expansion, and replacements. Evaluate
allocations/work against input, materialized output, and retained history,
including sorting costs. Fix repeated ancestor work or whole-tree copying without
changing semantics. No unconditional O(n) promise or timing release gate.

Add a maintained consumer example showing one-load inheritance, explicit false,
nested destination behavior, and source/derivation inspection. Update
`docs/config.md` as behavior owner, root comments as API owners, README/consumer
guidance as summaries, and internal overview/development test map as navigation.
Update architecture/deferred-scope descriptions for implemented inheritance.
Explain reserved-key migration, null cancellation, no registration, sibling lookup,
source-first ordering, presence versus provenance, and unchanged version/lifecycle
policy. No release-policy or platform-gate changes are part of this feature.

Write the next available ADR (currently 0009) recording ConfigKit's chosen
architecture and rationale. Keep downstream-specific discussion out; rejected
choices belong only in Alternatives Considered. Do not edit accepted ADR decision
bodies or silently accept earlier drafts. Mark the feature roadmap's implementation
status accurately and link the implemented contract. Release notes/tagging remain
separate release-preparation work.

Run the everyday checker, reuse race evidence if ownership code is unchanged, and
run benchmarks for evidence without pass/fail thresholds. Review the final diff
against the roadmap, source equivalence, confidentiality, and public boundary.
Confirm no independent merge rule, registration API, extra release gate, or
background/resource lifecycle was introduced. Commit and report validation and
any remaining limitations.