diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..a054911 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,288 @@ +# Sibling Inheritance Implementation Plan + +## Scope and working rules + +Implement the agreed [named-inheritance roadmap](named-inheritance.md), starting +from `01e38b5`. The roadmap owns feature policy; this document owns execution and +the concrete implementation choices below. Follow stages in order. Each stage is +a focused task for a gpt-5.6-terra coding agent and should end with relevant +passing checks and a plain-English commit. Do not publish or tag a release during +this sprint. + +Read `docs/development.md`, `docs/policy/agents.md`, and task-relevant policies. +Preserve unrelated work. The root owns public types; the private engine owns +processing. Reuse the parser-independent document representation and existing +helpers. Do not add dependencies, configurable keywords, registration APIs, +merge strategies, background work, or consumer-specific semantics. + +Stages 1–6 develop private machinery without activating inheritance in public +loading. Stage 7 connects the complete pipeline. Intermediate commits must not +publish resolved values with incomplete provenance or redaction. Private names +and file splits below are guidance, not new public contracts. + +## Decisions carried into implementation + +- `extends` is reserved input syntax. A string names an exact sibling mapping; + null cancels a relationship; omission retains lower contributions. Other types, + missing/non-mapping parents, self-reference, and cycles are errors. Empty strings + are exact names. Root and direct sequence-element mappings cannot use string + references; null needs no lookup. +- Strict composition still rejects overlapping non-mapping contributions, + including duplicate `extends` definitions. Complete all source-tier and CLI + precedence before expanding the effective input. Overwritten earlier references + are historical evidence, not active relationships to validate. +- Expand enclosing parent chains before resolving nested relationships in each + destination. Reusable parent templates retain unresolved nested instructions. + Sort mapping keys lexicographically, follow parent dependencies as needed, and + visit sequences by index. Sorting never determines value precedence. +- Parent-to-child combination uses recursive overlay, including explicit + zero/false/empty/null replacement and whole-sequence replacement. Consume the + current mapping's own relationship during enclosing expansion; nested + relationships travel with nested payload until their scope is resolved. +- Inherited values count as explicit input for presence/defaulting queries. + Provenance distinguishes direct input from inherited input. +- Version checking stays at its existing main-input point. Expansion follows + source assembly and precedes defaults/decoding, recorded resolution, secret + lookup, validation, and publication. Invalid effective relationships fail even + in entries the application does not select for execution. +- Preserve source records and passive resource contexts. Inheritance is neither + a new acquired source nor another precedence layer. + +### Concrete metadata representation + +Extend engine and root `Origin` with `Inheritance []InheritanceStep`. Define +root-owned and engine counterpart records with `From Path`, `To Path`, +`Reference Path`, and `Locations []Location`. A step describes a payload-path +transfer. `Reference` is the destination's controlling `extends` path; `Locations` +identifies that directive's contributing source coordinates. Paths are concrete +semantic paths, not names requiring another lookup. Records contain no values. + +Keep `Origin.Layer` and `Origin.Locations` for actual value contributors. Order +steps by derivation; composite origins retain contributing branches rather than +inventing a single chain. Project concrete descendant paths when queried. Keep +original contributor paths privately alongside locations, so synthetic sources +do not depend on file coordinates to identify original semantic paths. Direct +replacement keeps its own effective origin; displaced inheritance stays in history. + +Add inspection-only `ChangeInheritance = "inheritance"`, like `ChangeDecoding`; +do not accept it through `Resolution.Record`. Inheritance events identify +parent/control dependencies through existing `Change`, `ValueChange`, and +`Explanation` surfaces. No separate public query family is needed. Effective +inherited origins retain source layers and transfer evidence; expansion history +events use the new operation kind. + +### Budgets and diagnostics + +Keep acquired-input accounting unchanged. Use `defaultProjectionLimits()` for +expanded tree depth/nodes, counted by logical materialized nodes rather than +unique pointers; report `CodeSchemaProjectionLimit`. Check during materialization +before uncontrolled allocation and typed decoding. Do not charge copies as +acquired input or add a chain-length cap. + +Extend `historyNodesWithin` to retained inheritance templates and snapshots, +counting shared immutable document nodes once under the existing `historyNodeLimit`; +report `CodeExplainLimitExceeded`. Compact relationship records refer to shared +evidence instead of copying a full chain/tree per field. Do not redefine node +limits as byte limits or introduce a new public limit. Check retained growth as +it is accumulated and release temporary templates. Use private small test budgets. + +Add mirrored engine/root codes: `CodeInheritanceInvalid` (`inheritance.invalid`) +for invalid type/position or non-mapping parent; `CodeInheritanceMissing` +(`inheritance.missing`); `CodeInheritanceCycle` (`inheritance.cycle`); and +`CodeInheritanceVersion` (`inheritance.version`). Use directive paths/locations +as primary evidence and related definition locations for dependencies/cycles. +Messages must not interpolate raw reference strings or payload. Existing syntax, +composition, decode, and budget codes retain their meanings. + +## Stage 1 — Prepare shared recursive combination + +Inspect `internal/engine/overlay.go`, `compose.go`, `mapping_index.go`, +`history.go`, and `internal/document/document.go`. `applyOverlay` already shares +`mergeOverlayMappings` across underlays, main-over-underlay assembly, and selected +overlays. Strict composition uses `mergeDisjointMappings` after claim validation; +history replay has a separate value-only traversal. + +Extract or adapt the smallest private recursive-overlay primitive needed for +arbitrary mapping roots, preserving ordered entries and indexed lookup. Keep +source-tier bookkeeping in its adapter. Provide path/replacement information for +inheritance's later adapter; do not model inheritance as a fake layer. Reuse +`document.Clone` and `mappingIndex`. Keep strict composition distinct. Consolidate +value-only replay only if doing so preserves its behavior without extra machinery; +do not build a public or general-purpose merge framework. + +Validation: existing compose/overlay/override and underlay tests, plus a compact +table covering recursion, atomic sequence/scalar/null replacement, empty mappings, +and detached ownership. Existing behavioral expectations should not change. +Run `./scripts/check.sh` before committing. + +## Stage 2 — Expand one sibling scope + +Add a private inheritance resolver, callable only from engine tests initially. +Index effective sibling definitions. Resolve single-parent dependencies with +visiting/resolved states and memoized enclosing-chain templates. Use an explicit +work stack so graph chain length is not tied to the Go call stack. Overlay parent +payload with child payload using Stage 1. Consume the current mapping's own +`extends`, leaving nested instructions intact. Handle null before lookup. + +Implement internal diagnostic codes and deterministic traversal. Preserve +locations and collect transfer/control evidence for Stage 4. Keep original input +unchanged and templates private; a child must not mutate its parent. Failed +expansion returns diagnostics and no usable expanded result. + +Validation: forward/backward references, chains, shared parents, child replacement, +null cancellation, exact unusual/empty names, missing/non-mapping parents, invalid +types, self/long cycles, and input independence. Assert results and diagnostic +fields, not stack layout or exact prose. Run focused tests and the everyday checker. + +## Stage 3 — Resolve nested destination scopes and bound expansion + +Extend to the complete effective tree. Distinguish enclosing-chain templates from +fully resolved destination subtrees. After materializing an enclosing mapping, +resolve its nested sibling scopes independently. Visit sequence-contained mappings; +reject string references on root/direct sequence elements without ancestor lookup. + +Scope memoization to each effective destination namespace. Copying a template must +not copy its already-resolved nested children. Preserve inherited nested control +evidence and support cancellation through a child's nested null directive. Whole- +container replacement removes those active relationships at that destination; +the parent's own surviving relationships still require validation. + +Apply the selected expanded-tree budgets, counting logical output from shared +templates and releasing transient state. No global caches or unrelated-scope lookup. + +Validation: the roadmap's nested `primary`/`secondary` example; three-generation +enclosing chains; nested reference replacement/cancellation; destination-only +cycles; scopes inside lists; root/list-element errors; and parent names sorting +before or after children. Exercise small node/depth budgets at and beyond their +boundaries. Run the everyday checker. + +## Stage 4 — Preserve derivation, presence, and resource origins + +Implement the selected metadata representation privately in the engine. Carry +original semantic contributor paths, locations/layers, transfer edges, and control +evidence through copies and recursive combination. Preserve composite origins and +sequence-descendant origins, including atomic sequence contributions. Do not infer +ownership from the last matching layer: multiple sources can share a layer. + +Return an expanded tree, origins, and inheritance evidence for the decoded metadata +pipeline. Presence comes from expanded input. Direct replacement removes effective +inheritance at replaced paths but retains displaced evidence. Extend cloning, +comparison/deduplication, and private carriers; share immutable internal evidence +while detaching query outputs. + +Validation: inherited zero/false/null presence, chains and nested transfers, +composites, same-layer multiple sources, and replacement evidence. Demonstrate +that an inherited relative resource points to its original source context rather +than the child's directory. Inheritance does not open referenced resources. +Run the everyday checker. + +## Stage 5 — Integrate historical inspection + +Extend `history.go`, `load_metadata.go`, `explain.go`, and explanation rendering. +Retain original source applications; represent structural expansion separately, +between source replay and canonical decoding, followed by recorded application +resolution. Do not attribute inherited values or consumed directives to decoding. + +Destination history must show parent contribution and child replacement when both +exist, even if expansion leaves the effective child value unchanged. Include +derivation dependencies, preserve source history at original paths, and make +chains navigable through concrete paths. Do not fabricate repeated chronological +source loading at every child. Explain consumed directives as historical control +input absent from effective metadata/configuration. Preserve cancellation and +superseded directive evidence. + +Extend historical path enumeration, ancestor replacement, origin comparison, and +retained-node accounting. Keep typed reconciliation separate. Share scope evidence +and project relevant changes instead of storing a full snapshot per leaf. + +Validation: inherited values, equal child replacements, displaced inherited +subtrees, parent source changes, consumed/canceled directives, destination-specific +nested results, and subsequent application-derived replacement. Check agreement +among final values, `History`, `Explain`, and historical paths. Exercise small +history budgets. Run the everyday checker. + +## Stage 6 — Preserve sensitivity through inheritance + +Connect transfer dependencies to existing secret classification and historical +redaction. Classify both source and destination paths under schema and +`Definition.Secret` rules; propagate sensitivity through chains/nested copies. +Include displaced evidence and control history. Preserve existing conservative +historical redaction without declaring unrelated siblings secret. + +Copy unresolved secret references only; lookup remains at the existing lifecycle +point. Credentials never enter expansion trees/history. Independent replacements +retain their own effective provenance while previous secrets remain redacted. +Queries and renderers must agree. + +Validation: source-only and destination-only classification, shared parents, +transitive inheritance, replacement/removal, and lookup after expansion. Use +disposable canaries across affected inspection/failure surfaces. Avoid real +credentials and parallel environment mutation. Run focused secret/history tests +and the everyday checker. + +## Stage 7 — Activate the common lifecycle and facade metadata + +Insert expansion in `executeLoadRequest` after ordinary `applyInputLayers` or +layered `assembleLayeredInput`, before `decodeLoadInput` invokes defaults/decoding. +Include plain single-source loads, creating the minimal composition carrier needed +to preserve original history. Update `load_result.go` and inspection carriers with +Stages 4–6 evidence. Preserve no-input behavior and use a no-directive fast path +where practical. Do not repeat acquisition, version callbacks, or user resolution. + +Keep the version path main-owned. Detect inherited contributions, replacement, or +directive removal affecting it, including protected ancestor replacement; report +`CodeInheritanceVersion`. Do not reject an entire enclosing mapping merely because +it contains an unchanged version field. A version rule targeting an `extends` +field cannot silently lose its checked declaration when the directive is consumed. +Keep the checker at its original point and retain accepted version representations. + +Mirror records/codes in root `records.go`, conversions, clones, and API comments. +Ensure root `Origin.Inheritance` and `InheritanceStep` expose only root types and +detached slices/paths. Keep `ChangeInheritance` inspection-only. Update root/engine +`IsExplicit` comments, including resolver-entry queries. Preserve `SourceContext`. + +Do not treat `extends` as the acquisition-time `configkit` directive. Imports, +selected overlays, Mapping input, and CLI overrides carry it through combination +until expansion. Keep existing `configkit` validation and CLI path restrictions. +Typed defaults and values added by application resolution do not initiate another +structural pass; inheritance processes assembled source input only. + +Validation: public ordinary/layered loads; imports and strict conflicts; multiple +underlays; selected overlay and CLI changes to parent/child/reference; resolver +defaulting using inherited presence; version ownership and callback timing; +typed unknown-field behavior after consumption; no later hooks on expansion +failure; partial bases with normal validation; and no-directive regressions. +Include a filesystem resource-origin example and mapping/bytes equivalence case, +not a full duplication of every mechanism test. Run `./scripts/check.sh` and +`go test -race ./...` for ownership changes. + +## Stage 8 — Validate scaling and complete documentation + +Add focused benchmarks beside assembly/underlay benchmarks: long chains, wide +shared-parent scopes, nested destination expansion, and replacements. Evaluate +allocations/work against input, materialized output, and retained history, +including sorting costs. Fix repeated ancestor work or whole-tree copying without +changing semantics. No unconditional O(n) promise or timing release gate. + +Add a maintained consumer example showing one-load inheritance, explicit false, +nested destination behavior, and source/derivation inspection. Update +`docs/config.md` as behavior owner, root comments as API owners, README/consumer +guidance as summaries, and internal overview/development test map as navigation. +Update architecture/deferred-scope descriptions for implemented inheritance. +Explain reserved-key migration, null cancellation, no registration, sibling lookup, +source-first ordering, presence versus provenance, and unchanged version/lifecycle +policy. No release-policy or platform-gate changes are part of this feature. + +Write the next available ADR (currently 0009) recording ConfigKit's chosen +architecture and rationale. Keep downstream-specific discussion out; rejected +choices belong only in Alternatives Considered. Do not edit accepted ADR decision +bodies or silently accept earlier drafts. Mark the feature roadmap's implementation +status accurately and link the implemented contract. Release notes/tagging remain +separate release-preparation work. + +Run the everyday checker, reuse race evidence if ownership code is unchanged, and +run benchmarks for evidence without pass/fail thresholds. Review the final diff +against the roadmap, source equivalence, confidentiality, and public boundary. +Confirm no independent merge rule, registration API, extra release gate, or +background/resource lifecycle was introduced. Commit and report validation and +any remaining limitations. diff --git a/docs/roadmap/named-inheritance.md b/docs/roadmap/named-inheritance.md index 941b3c3..74d4422 100644 --- a/docs/roadmap/named-inheritance.md +++ b/docs/roadmap/named-inheritance.md @@ -1,7 +1,7 @@ # Built-In Sibling Configuration Inheritance Status: agreed feature design; not implemented. This document defines the intended -end state. A separate implementation plan will define execution steps. +end state. The [implementation plan](implementation.md) defines execution steps. ## Purpose and priorities @@ -275,9 +275,9 @@ output than the source declarations. Use representative chains, shared-base tree wide collections, and replacement cases to evaluate growth. Performance evidence is not a new release timing gate. -Existing acquired-input accounting remains in force. The eventual implementation -plan must define how expanded output and retained derivation fit existing decoded -tree/history limits, including where expansion is checked. Avoid silently charging +Existing acquired-input accounting remains in force. Expanded output and retained +derivation respect existing decoded tree/history limits; the implementation plan +defines accounting and check placement. Avoid silently charging one inherited value as a newly acquired source or inventing arbitrary entry/chain caps. Any necessary new public limit requires an explicit design decision. @@ -302,7 +302,7 @@ A new ADR should record the selected inheritance architecture. Architecture/deferred-scope documentation must recognize the new structural responsibility; public API, configuration, consumer, and internal documentation should describe implemented behavior when it lands. -Implementation sequencing belongs in a separate future plan. Reserving `extends` +Implementation sequencing belongs in the implementation plan. Reserving `extends` changes behavior for applications that currently use it as ordinary data. Document that compatibility change and the required migration rather than describing this as a purely additive opt-in feature. Configurations without `extends` retain their