ConfigKit/docs/roadmap/implementation.md

Sequence Identity And Combination Implementation Plan

Status: implementation in progress; Stages 1–10 complete.

Audience: a Luna coding agent implementing one numbered stage per prompt, in order. This plan starts at commit 4f941f8ba4c9a9ed0afa87b07fe6ba29f131d1c6. It implements the settled sequence roadmap, ADR 0017, and ADR 0018.

Working Rules

• Read development guidance and the applicable policies, especially agents, architecture, testing, and documentation. Prefer the repository knowledge graph for code discovery.
• The roadmap owns the selected feature semantics. This plan supplies execution order and implementation choices, not additional product requirements. Preserve unchanged source, parser, version, schema, ownership, and inspection contracts.
• Complete each stage's implementation and behavioral tests together. Record its completion and validation here, then commit the nonempty unit with a short, plain-English message. Preserve unrelated changes. Do not publish intermediate stages or a release, update consumers' dependencies, or wait for CI.
• Run focused tests while iterating and ./scripts/check.sh before handing off each code stage. Use the installed compatible Go toolchain with dependencies already acquired, as the development guide describes. Add race coverage at the ownership milestones below. These are implementation checks, not new release gates. Reuse successful checks for unchanged content rather than repeating them at a documentation-only milestone. Do not require human platform validation.
• Update affected declaration comments and canonical current-behavior prose when behavior lands. Keep incomplete target behavior in this roadmap/plan until it works. The final documentation stage reconciles the cross-feature narrative; it is not permission to leave misleading current contracts in earlier stages.
• Transitional internal adapters are acceptable where specified. Remove them by completion. Do not add public compatibility aliases, permanent feature flags, alternate semantics, new dependencies, new resource caps, or an extension/plugin framework for this work.
• Preparation stages keep existing public behavior working and test new machinery privately. Activation stages integrate values, policy, metadata, and relevant failure handling together; do not defer correct inspection to a later stage.

Implementation Decisions

Public Registration Surface

Replace SequenceComposition and Definition.SequenceCompositions with:

type SequenceIdentity struct { /* private, root-owned registration state */ }
type SequenceSelector []SequenceSelectorSegment
type SequenceSelectorSegment struct { /* private literal-or-any-mapping-key state */ }

func SequenceLiteralKey(key string) SequenceSelectorSegment
func SequenceAnyMappingKey() SequenceSelectorSegment
func SequenceIdentityAt(path Path, fields ...string) SequenceIdentity
func SequenceIdentityMatching(selector SequenceSelector, fields ...string) SequenceIdentity

// Field on Definition[T]:
// SequenceIdentities []SequenceIdentity

For example, the reusable definition can contain both registrations:

SequenceIdentities: []configkit.SequenceIdentity{
    configkit.SequenceIdentityAt(configkit.Keys("prompts"), "id"),
    configkit.SequenceIdentityMatching(configkit.SequenceSelector{
        configkit.SequenceLiteralKey("collections"),
        configkit.SequenceAnyMappingKey(),
        configkit.SequenceLiteralKey("prompts"),
    }, "id"),
},

The exact constructor converts mapping-only Path segments into literal selector segments. Preserve invalid index segments long enough to report an ordinary definition error; do not panic, silently drop them, or change Path semantics. The zero selector is the exact root selector, harmless under the existing mapping-root schema. A zero selector segment denotes the literal empty key, not a wildcard. Empty field names are valid literal field names; an empty field set is invalid. Constructors snapshot slices; load-entry conversion also takes an owned snapshot. Definition validation, not construction, reports errors. Root types and functions must not expose internal types or bridge methods.

Identity Representation And Matching

• Normalize identity field sets once: deduplicate and sort exact field names. Use the same normalizer for YAML and application registration.
• Encode a record's identity as a hashable, type-tagged, length-delimited tuple in normalized field order. Mapping lookup indexes avoid scanning record fields repeatedly. Distinguish structural-payload lookup from final typed-payload and parent-reference lookup: only fields acting as directives are excluded, not every occurrence of the same literal spelling. Never include tuple encodings in diagnostics.
• Normalize finite JSON numbers to sign, significant decimal digits, and a signed base-10 exponent. Strip insignificant zeroes and give all zero spellings one representation. Use math/big.Int for exponent arithmetic, not a power-of-ten expansion or a float conversion. Thus large exponent spellings do not allocate storage proportional to their mathematical magnitude. This comparison-only representation does not rewrite document nodes or typed projection.
• Compile selectors into a mapping-segment trie with literal and wildcard edges. Carry applicable trie states through traversal; stop at sequence-index edges. Index concrete matches within one assembly/inheritance scope where useful, but do not save an assembly-only destination list for final validation.
• Check registration overlap pairwise once during definition validation using equal length and compatible corresponding segments. This straightforward O(R² × D) check, for R registrations and maximum selector depth D, is proportionate for registration definitions. Equivalent constraints coalesce; conflicting overlaps fail even with no matching input. Do not impose a registration-count limit to simplify this operation.

Policy And Evidence Storage

Keep application constraints separate from YAML policy. Add private engine collection-policy records containing normalized identity, explicit mode, and declaration locations. Track whether identity is pending or belongs to a retained collection. Retain declaration presence separately from inferred defaults.

Use the existing acquisition/item scopes as ownership anchors, with an opaque per-load owner token when a retained record needs one. Store item-local policies relative to their owning record, and project concrete paths when that record is placed. Do not key ownership solely by numeric index, identity tuple, or source filename: all three can coincide for a replacement that must lose old policies. Inheritance clones the relevant scoped state; it does not create registrations.

Resolve policies through shared engine helpers used by strict composition, precedence, and inheritance. Do not attach engine policy to public values or add policy fields to the parser-independent document.Node. The sidecar can be extended with concrete-path indexes; it need not become a general event store.

Sequence combination returns the resulting list plus explicit input-to-output item projections and retained/replaced contribution information. Use that same result for values, policy transfer, origins, source destinations, sensitivity, and historical replay. Extend the existing sequence-history descriptors for noncontiguous upsert positions rather than inferring them from equal values. Sharing immutable internal evidence is acceptable under existing ownership rules; independently returned mutable records must remain detached.

Diagnostics

Use existing options.invalid for invalid application definitions and existing compose.invalid / compose.conflict for malformed or duplicate strict-peer contributions. Add root-owned and engine diagnostic constants CodeSequenceInvalid (sequence.invalid) and CodeSequenceConflict (sequence.conflict) for malformed collection declarations/operations or final identities, and conflicting policy / precedence duplicates respectively. Final duplicate identities use sequence.conflict. Keep existing inheritance invalid/missing/cycle codes for record relationships. Use existing override and schema/decode codes at the CLI boundary; do not invent a new error taxonomy there.

Report collection/item paths and available locations, not identity values, normalized tuple strings, raw parent references, or raw hook errors. Declaration conflicts include both sites. Definition overlap errors identify registration positions without fabricating a concrete matched configuration path. Exact English diagnostic wording is not a new guarantee.

Stage 1 — Implement Shared Scalar Identity Primitives

Scope: new private identity helpers and their focused tests; no load-path change.

1. Add internal/engine/sequence_identity.go. Implement field-set normalization, scalar tuple construction, and exact decimal normalization as chosen above. Accept string, number, boolean, and explicit null nodes; distinguish categories and absence. Exclude fields acting as structural directives from payload lookup.
2. Validate mapping records, complete direct-field sets, and uniqueness using one tuple-to-first-record index. Return paths and source locations, not values. Support caller-selected diagnostic families for strict and other boundaries.
3. Cover equivalent field sets; empty strings and null; missing/non-scalar fields; literal punctuation; compound tuple boundaries; 1/1.0/1e0; signed zero; close unequal decimals; and large exponents without expansion. Verify that node text remains unchanged. Include extends: null as a structural field that cannot supply source identity.

Validation: go test ./internal/engine -run 'SequenceIdentity|SequenceNumber' and the everyday check. Retain the old registration path until its adapters are replaced; do not generalize unrelated YAML scalar decoding.

Completion: Implemented and verified with the focused identity tests and ./scripts/check.sh.

Stage 2 — Compile Exact And Dynamic Application Constraints

Scope: private registration model, selector matching, overlap validation.

1. Extend sequence_registration.go with the common registration representation and compiled selector trie. Represent literal and arbitrary-mapping-key segments structurally, not with a magic string. Snapshot registration inputs.
2. Validate field sets and mapping-only selectors before source acquisition. Preserve existing destination validation without adding restrictions on identity field spelling. Root and unused selectors remain harmless.
3. Implement full-destination matching, equivalent overlap coalescing, and incompatible-overlap errors. Exact rules have no priority over dynamic rules. Keep a temporary conversion from the current single-field registration so existing callers still use their current behavior.
4. Test exact and wildcard matches, multiple wildcard levels, empty/punctuation keys, sequence-index nonmatches, disjoint lengths/literals, equivalent sets, incompatible overlaps with absent input, and independent uniqueness scopes.

Validation: focused registration/selector tests and the everyday check. This stage does not export the replacement API or add final-result validation yet.

Completion: Implemented and verified with focused registration/selector tests and ./scripts/check.sh.

Stage 3 — Parse Collection Declarations Without Filesystem Coupling

Scope: private parser and directive extraction helpers, initially unit-tested without activating collection combination in public loads. Keep source-form activation out of this stage; ordinary sources and selected overlays are wired in Stages 13 and 14 respectively.

1. Add sequence_declaration.go for source-root configkit.sequences. Each entry has a required existing literal destination form and optional key and mode. key is a string or nonempty string list; mode is exactly replace, append, or upsert. Use shared identity normalization. A destination-only entry can be a harmless no-op; do not add an unrequested ban on empty declarations.
2. Split common root-directive extraction from import-specific parsing in import_spec.go / parse.go. Extract imports and sequences together, retaining source-local locations and explicit presence. Reject unrecognized directive fields and wrong supplied field types using existing strict parsing style.
3. Keep declaration destinations relative to source placement, with item-local coordinates retained until their scope is assembled. Do not project them prematurely to a guessed numeric index.
4. Give extraction an explicit overlay context: sequences are allowed, imports are not. Import enablement remains separately enforced; do not turn on filesystem access because a source declares sequences. Keep enabled source-version checks in raw source coordinates before consuming controls; extend existing actual header/control-overlap handling to sequence directives without treating an absent header beneath a control as an actual overlap.
5. Test key-only, mode-only, combined, repeated-field-set equivalence, no-op, malformed types/modes/destinations, mixed imports/sequences, and overlay rules. Do not add import-level key/mode shorthand or sequence wrappers.

Validation: focused declaration/parser tests, existing import-spec tests, and the everyday check. Existing import-only behavior remains intact during preparation.

Completion: Implemented and verified with focused declaration/parser and import-spec tests plus ./scripts/check.sh.

Stage 4 — Reconcile Declarations And Select Collection Operations

Scope: pure agreement and operation-selection helpers, independent of loading and of the ownership transformations introduced next.

1. Add collection_policy.go. Gather the current scope's declarations before combining peers. Reconcile equivalent keys and agreeing explicit modes; conflicting peers fail. Omitted fields have no preference. Apply authoritative application constraints even for key-only declarations without a value.
2. Implement shared operation selection: current explicit mode, retained explicit mode, inferred upsert with effective identity, otherwise replace. Persist only explicit modes. Policy-only declarations must not replay earlier combinations.
3. Accept eligible lower policy as an input; do not decide record survival or ancestor removal inside declaration reconciliation. Retain declaration locations/presence separately from inferred mode and application constraints.
4. Test equivalent/different field sets, agreeing/conflicting peer modes, omission, explicit default modes, and operation inference. Include key-only declarations agreeing or conflicting with application constraints before a sequence exists.

Validation: focused policy-agreement/selection tests and the everyday check. Do not activate collection combination or implement the ownership layer here.

Completion: Implemented and verified with focused collection-policy tests and ./scripts/check.sh.

Stage 5 — Track Policy Lifetime And Record Ownership

Scope: private scoped-state transformations built on Stage 4's resolution helpers.

1. Implement key agreement across retained values, replacement reset, pending key declarations, and immediate validation when a key-only declaration identifies retained records. Changing mode to replace without a replacement value does not reset identity.
2. Implement scope/record ownership transfer and retirement hooks. Actual ancestor replacement removes descendant collection identity and removed-record policy; independently owned modes and application constraints survive as specified in the roadmap. Do not repeatedly reapply old declarations and resurrect reset identity. Pending declarations retain their owning-scope rules.
3. Use item/acquisition scope anchors and record-owned relative policy as selected above. Provide transformations for retention, whole-record replacement, whole-sequence replacement, and inheritance copies without wiring them into public loading yet. Application registrations remain destination constraints.
4. Use table-driven tests derived from the roadmap's lifetime, persistence, and replacement tables, including absent/non-sequence destinations, equal outer identities with distinct record owners, and independent root versus item policy.

Validation: focused collection-policy tests and the everyday check. This stage must not add path-existence, shape, nullability, or global YAML final-check rules.

Completion: Implemented and verified with focused collection-policy tests, go test -race ./internal/engine, and ./scripts/check.sh.

Stage 6 — Implement The Shared Sequence Operations

Scope: a pure document-level combination helper, not separate implementations for overlays and inheritance.

1. Add sequence_combine.go, accepting lower/higher sequence nodes, resolved policy, and combination context. Reuse Stage 1 identity extraction and indexes.
2. Implement replace, duplicate-rejecting append, and whole-record upsert. Upsert retains matched lower positions and appends new higher identities in order. Return item projections and replacement/retention evidence with the new value.
3. Keep peer validation distinct: duplicate identities within any contribution or among peers fail under every mode. Do not merge record payloads or sort by ID.
4. Cover absent versus null/non-sequence lower values; higher omission, [], null, and non-sequence values; missing identity under append/upsert; empty inputs; stable ordering; equivalent/different keys; and explicit replacement identity reset. An empty input does not bypass kind checks or identity requirements.
5. Verify clones do not alias mutable input nodes or policy slices. Build indexes once per operation rather than scanning the accumulated list for every record.

Validation: focused combination tests and the everyday check. Do not change the ordinary recursive mapping rule or add unkeyed append.

Completion: Implemented and verified with focused sequence-combination tests and ./scripts/check.sh.

Stage 7 — Extend Sequence History Representation And Replay

Scope: represent and replay sequence operations without yet changing all metadata projection consumers or public precedence.

1. Extend sequenceHistoryContribution, sequence replay in history.go, and its contribution indexes to consume Stage 6's explicit item mappings. Existing append ranges may remain a compact case; upsert cannot be represented by one offset.
2. Preserve authored raw contribution snapshots and displaced values. A replaced record's omitted fields must appear as removed, not as retained lower payload. Preserve equal-valued replacement events. Keep existing index-based query and atomic sequence-diff APIs; add no public identity-selector path syntax.
3. Exercise synthetic append/upsert/replace histories, noncontiguous matches, nested paths, and discarded descendants. Compare replayed final values with the actual combination result, including first/last and unchanged neighboring records. Reuse existing escaped-path and detached-query tests rather than duplicating them.

Validation: sequence/history/replay tests and the everyday check. Keep current append/replace callers working through the extended descriptors.

Completion: Implemented and verified with focused sequence-history tests and ./scripts/check.sh.

Stage 8 — Project Sequence Origins And Inspection Evidence

Scope: use the prepared item mappings consistently in metadata consumers.

1. Project composition origins, nested source destinations, historical paths, and sensitivity through Stage 6's item mapping and Stage 7's history descriptors. Physical source IDs/locations and resource contexts continue to identify the declaring source. Reuse one projection mechanism rather than searching again by identity or equal payload.
2. Test synthetic noncontiguous upserts, replaced descendants, nested source destinations, unchanged neighbors, and secret-marked replaced records. Verify projected queries agree with replayed and actual final values, retaining displaced evidence and detached results.
3. Keep the new precedence operations unactivated until their value, policy, and projection machinery are integrated in the later load-path stages.

Validation: metadata/history/sequence/provenance/sensitivity tests, go test -race ./internal/engine, and the everyday check.

Completion: Implemented and verified with focused projection, history, sequence, provenance, and sensitivity tests, go test -race ./internal/engine, and ./scripts/check.sh.

Stage 9 — Separate Per-Input Acquisition From Composition

Scope: a behavior-preserving acquired-input handoff; grouped underlay coordination is the next stage.

1. In load_input.go, imports.go, and related graph helpers, retain acquisition scopes, raw contributions, inclusion edges, source records, contexts, and extracted declarations without irrevocably composing them. Reuse existing deferAssembly and acquired-scope mechanisms instead of adding a second loader. Keep already acquired document handles closed as before.
2. Separate acquisition from the per-input composition entry. Retain pending item/helper scope boundaries, namespace identities, inclusion/repetition checks, source records, accounting, and scheduler inputs. Do not reacquire sources to compose them later or change grouped coordination yet.
3. Preserve existing version checks for this refactor. Retain a small adapter invoking current composition semantics at the existing lifecycle point. Preserve source-version callbacks at acquisition, including header consumption.
4. Run existing acquisition, dependency scheduling, item placement, source-version, ordinary/layered equivalence, resource-lifetime, and source-context tests. Add only targeted regressions for any newly split handoff not already covered.

Validation: the focused suites above and the everyday check. This refactor must not change source IDs or turn acquisition order into precedence order.

Completion: Implemented and verified with focused acquisition, scheduling, placement, source-version, layered-equivalence, resource-lifetime, and context tests plus ./scripts/check.sh.

Stage 10 — Separate Underlay-Group Acquisition From Composition

Scope: behavior-preserving group coordination in load_request.go, using the per-input handoff already prepared in Stage 9.

1. Separate acquiring an underlay group from scheduling/composing its acquired scopes. Preserve namespace reanchoring, shared inclusion/repetition state, item-local helper scopes, source/context aggregation, and group accounting.
2. Keep authored acquisition order: main and depth-first imports, each underlay and input, overlay, then synthetic CLI records. Preserve underlay indexes and current version-check timing in this refactor; policy-dependent layered composition is activated only in Stage 15.
3. Expose the common scope-composition handoff with inputs for accumulated lower policy and compiled application constraints. Initially adapt it to the current semantics. Do not add another loader or repeat acquisition/version callbacks.
4. Test the split with existing grouped/dependency/version/destination coverage, including empty inputs/groups and repeated sources within versus across tiers. Confirm unchanged values, source ordering, contexts, and failure evidence.

Validation: grouped/layered, import scheduling, version, destination, and resource-context tests plus the everyday check.

Completion: Implemented and verified with focused grouped/layered, import-scheduling, version, destination, and resource-context tests plus ./scripts/check.sh.

Stage 11 — Compose Strict Collections With Resolved Identity

Scope: the policy-aware strict composition core. Nested pending-item scope integration is deferred to Stage 12, not absorbed into this prompt.

1. Extend composeDocumentsWithPolicy and validateRegisteredSequences to use normalized field sets and scalar tuples. Resolve current declarations, applicable application constraints, and eligible lower identity before peer composition, using the Stage 4 agreement and Stage 5 lifetime helpers.
2. Use the existing dependency-ready order for contributions and authored list order. Validate every identified sequence, including singleton contributions; strictly append distinct peers and reject duplicates under all modes.
3. For explicit replacement, do not use obsolete lower YAML identity to authorize otherwise unkeyed peer composition. Still apply independently registered application constraints and genuinely applicable declarations.
4. Attach Stages 7–8 evidence and Stage 5 scoped policy to composed results. Test completed-document contributions with lower-established identity, declaration conflicts, compound/scalar IDs, singleton validation, and replacement resets.

Validation: composition and sequence identity tests plus the everyday check. Keep unregistered non-mapping conflicts unchanged. Public load activation remains in the later integration stages.

Stage 12 — Integrate Policy-Aware Item And Helper Scopes

Scope: adapt pending-item/helper composition to Stage 11's core using the acquired scope boundaries from Stages 9–10.

1. Gather declarations before composing each relevant scope, complete item-local helper imports before item placement, and consume applicable non-main version headers before checking record payload identity. Keep relative policy with its owner until concrete placement is known.
2. Apply eligible lower identity at each scope, but prevent a replacement outer record from borrowing nested identity owned by the discarded lower record. Preserve independently applicable policy and application constraints.
3. Retained structural directives cannot serve as identity fields. A helper may supply payload identity components. Do not require a helper-built compound parent reference to be complete at this point; validate the completed surviving reference during inheritance.
4. Project item evidence and owned policies through Stages 7–8 machinery. Test nested items, helper-supplied identity, header consumption, declaration scope, discarded owner identity, and exact diagnostic paths with no value disclosure.

Validation: item/destination/import/composition/source-version tests and the everyday check. Keep current public behavior until load-path activation.

Stage 13 — Activate Ordinary Source Collection Declarations

Scope: ordinary Load base-source assembly, excluding activation of the new selected-overlay policy surface until Stage 14.

1. Wire common directive extraction into every ordinary source form: file, native filesystem, bytes, reader, mapping, imported mapping, and sequence item. Declarations work without Imports; disabled imports retain their existing rejection semantics. NoSource remains absence rather than an empty document.
2. Use Stages 11–12 composition and Stage 5 scoped policy. Strip collection controls from application decoding, but retain their locations/evidence for their structural lifetime and diagnostics.
3. Integrate policy/projections together with the values. Cover inline/imported equivalence, placement-relative declarations, singleton and peer validation, and declarations without accompanying values. Maintain ordinary loading's existing non-collection overlay behavior while preparing its policy handoff.
4. Update the implemented declaration/composition contract sections. Collection precedence, inheritance, and later CLI behavior remain incomplete until their integration stages; do not claim them implemented here.

Validation: ordinary-load, import, composition, and metadata tests plus the everyday check. Include one public ordinary-load declaration example test.

Stage 14 — Activate Selected Overlay Collection Policies

Scope: ordinary loading's selected overlay and shared precedence integration, using the source/declaration handoff already active in Stage 13.

1. Permit configkit.sequences in selected overlays while continuing to reject overlay imports. Preserve source-version header/control handling and ordinary overlay namespace selection. Consume controls before application decoding.
2. Use Stage 6's sequence operations and Stages 4–5 policy helpers in the assembly accumulator. Mappings still recurse and unrelated sequences still replace. Wire value, policy retirement/transfer, and Stages 7–8 inspection projections together, not as a later best-effort repair pass.
3. Cover append/upsert/replace, empty/null/kind cases, key-only/mode-only changes, retained explicit modes, overlay identity agreement, and independently owned policies. Check values, history, origins, and discarded descendants together.
4. Update the selected-overlay and collection-operation contract sections. Make the shared precedence entry reusable by layered loading without adding another operation implementation or turning overlay imports on.

Validation: ordinary-load, overlay, composition, policy, and metadata tests plus the everyday check.

Stage 15 — Integrate Lower Policy Across Layered Assembly

Scope: source-backed underlay coordination and the layered selected-overlay path, using the prepared acquisition, ownership, strict-scope, and precedence machinery. Do not absorb unfinished work from Stages 5–14 into this stage.

1. Acquire main first, but assemble underlays from lowest to highest, strictly composing each group with applicable accumulated lower policy before applying its one precedence operation. Then compose/apply main and the selected overlay. Do not sequentially upsert peer sources as if they were different tiers.
2. Pass lower identity into higher strict scopes before peer composition. Cover an otherwise empty underlay establishing identity, inherited explicit mode, and replacement that deliberately discards the old YAML key. Invalid lower tiers still fail even if a higher replacement would hide their values.
3. Keep main-only version ownership and every enabled per-source version rule. In default source-version mode, run the main checker once on the composed main tier before overlay/inheritance/defaults, never on underlay-augmented values. Its former main-before-underlay composition timing need not persist: main composition now depends on lower policy. Enabled-mode source checks keep acquisition order and exact source-local header handling. No extra callbacks or second acquisition pass are permitted.
4. Keep histories in precedence order, sources in acquisition order, underlay indexes stable across empty groups, and shared budgets intact. Update the layered contract's acquisition-versus-composition description accordingly. Remove the acquisition/composition adapters superseded by this integration; the old public-registration adapter remains until Stage 22.
5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main, explicit empty groups, layered overlays, version protection and diagnostics, and equivalence with ordinary loading where the inputs express the same model.

Validation: layered/grouped, source-version, dependency scheduling, destination, and metadata tests plus the everyday check.

Stage 16 — Verify Replacement And Nested Policy Ownership End To End

Scope: the consequential lifetime cases across the now-active source assembly.

1. Exercise an item-local nested collection through retaining the outer record, upserting the same outer ID, and replacing the entire outer sequence. Retained items keep their local key/mode; replacements do not inherit discarded policy merely because identity, source label, or final index is equal.
2. Exercise ancestor replacement and later reintroduction. Removed collection identity does not reappear; independently declared append mode can survive and require renewed identity. Cover renewed YAML key, application constraint, explicit replace, and the intended failure with no available identity.
3. Cover key-only validation of retained records, pending declarations, a no-value replace declaration that cannot authorize key changes, and independent policy whose owning scope survives a record replacement.
4. Correct any integration defects in the Stage 5 ownership transformations; consolidate tests with their mechanism owners. Do not introduce special-case ownership by numeric path or a historical identity cache to make cases pass.
5. Verify frozen metadata, origin/resource context, sensitivity of discarded records, and mutable independence across repeated loads for these scenarios.

Validation: focused policy/sequence/layered/ownership tests, go test -race ./internal/engine, and the everyday check.

Stage 17 — Prepare Inheritance Policy And Template Transport

Scope: behavior-preserving inheritance state transport, before activating collection-aware inheritance operations in Stage 18.

1. Extend inheritance templates/expansion sidecars to carry collection identity and record-local policy alongside nodes and origins. Keep existing scheduler keys, cancellation, version protection, missing-parent/cycle diagnostics, and destination-local evaluation of nested templates.
2. Provide owned-state copy/transfer helpers from Stage 5 and path projections from Stage 8. Preserve original nested instructions for each destination's later evaluation rather than retaining only a parent's expanded output. Keep application registration matching separate from copied YAML policy.
3. Test template reuse at multiple destinations, independent scoped state, and retained instruction locations/resource context. Preserve current inheritance values/history while these new helpers remain preparatory; do not activate a value change without its corresponding policy/evidence handling.

Validation: existing inheritance, ownership, history, and resource-context tests, focused transport tests, and the everyday check.

Stage 18 — Apply Collection Policy During Mapping-Sibling Inheritance

Scope: activate shared collection operations using the prepared template/state handoff; existing mapping-sibling relationship selection remains unchanged.

1. Use the shared sequence combiner at the inheriting destination: parent lower, child higher, destination explicit mode or identity-based default. Copy YAML identity where applicable, but do not automatically copy a parent collection's explicit mode to a different destination. Match application constraints at the destination, including newly created matching paths.
2. Transfer record-owned state only with surviving records using Stage 17's helpers. Integrate inheritance evidence with Stage 8's projections in the same activation, preserving origins, historical sensitivity, resource context, cancellation, and existing expansion/history accounting.
3. Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key with inferred upsert, destination explicit mode, incompatible keys, application constraints, cancellation, and policy at an inherited new namespace.
4. Update current mapping-sibling inheritance documentation for collection operations. Identified-record parent lookup remains incomplete until Stage 21.

Validation: inheritance, history, provenance, sensitivity, and resource-context tests plus the everyday check. No identified-record parent lookup is active yet.

Stage 19 — Build Identified-Record Inheritance Graphs

Scope: relationship extraction/validation and shared scheduling for sequence scopes; test these helpers before activating payload expansion.

1. Extend sibling-scope discovery to identified sequences. Build the tuple-to-item lookup once using effective identity. Every record supplies its own complete identity before inheritance; neither parent lookup nor typed defaults supplies a missing child field.
2. Parse a surviving record's extends as null cancellation or a mapping with exactly the normalized identity fields and supported scalar components. Mapping-sibling references keep their existing string form. Do not accept a scalar shorthand, multiple parents, positional or cross-collection lookup.
3. Parent-reference mappings have already used ordinary strict mapping composition: disjoint helper fields can complete one reference, while equal overlapping fields conflict. Validate completeness/extra fields only now on the surviving reference. Discarded references remain historical evidence.
4. Build one-parent dependency edges and reuse scheduleDependencies with stable semantic-pointer keys. Missing parents, self-reference, and longer cycles use existing inheritance diagnostics without identity values. Keep authored output positions independent from graph schedule.
5. Cover compound and null references, numeric equality, forward edges, chains, unrelated equal identities in different lists, malformed refs, missing parents, cycles, helper-built references, and overwritten/cancelled relationships.

Validation: focused inheritance graph/reference tests and the everyday check.

Stage 20 — Prepare Reusable Inheritance Scope Execution

Scope: remove mapping-container assumptions from the private template executor without activating identified-record inheritance in public loads.

1. Refactor template execution to separate relationship scheduling/payload expansion from reading and rebuilding its scope container. Preserve current mapping-sibling output, locations, cancellation, evidence, and budgets.
2. Provide private mapping- and identified-sequence-scope adapters using Stage 19's relationship graphs. The sequence adapter preserves output positions and exposes child identity fields to the payload-expansion step. Keep this private; do not add a public generic inheritance or merge callback API.
3. Reuse Stage 17's template/policy transport and Stage 18's collection-aware payload operations. Existing origin/history/retained-node handling must remain available to both adapters rather than duplicating an entire executor.
4. Run existing mapping-inheritance regressions and privately exercise a small sequence graph through the adapter. Check output order, independent templates, and operation/evidence paths before public activation in Stage 21.

Validation: inheritance execution, scheduling, history, and budget tests plus the everyday check. This refactor does not introduce new parent-reference forms or change public loading semantics yet.

Stage 21 — Resolve Record Inheritance With Ownership And Evidence

Scope: integrate Stage 19's graphs and Stage 20's executor into the structural expansion phase, with values and inspection working together.

1. Expand parent payload into the child while preserving the child's own identity fields and their origin/presence. Exclude parent identity fields from payload inheritance. Recursively combine mappings and use destination collection policy for nested sequences; retain whole-record upsert at the earlier boundary.
2. Consume the child's directive before decoding. Keep parents as ordinary records subject to the same schema/semantic validation; create no abstract templates or acquired source records.
3. Resolve only records surviving tier assembly. Cover parent upsert/removal, replacement of a child without extends, self-reference not exposing a lower predecessor, and nested destination-local relationships. Enclosing scopes finish before nested scopes; assembled list order stays unchanged.
4. Carry policy, origin/history, sensitivity, resource context, and existing expansion/retained-history accounting through copies. Do not widen budgets or require a new inspection-path API.
5. Add public-load regressions for complete inheritance chains and nested collections, including secret canaries on success and failure. Update the implemented inheritance contract and consumer guidance for this scope.

Validation: all inheritance and cross-feature inspection tests, go test -race ./internal/engine, and the everyday check.

Stage 22 — Expose The Root-Owned Registration API

Scope: the settled facade surface and migration of in-repository callers.

1. Implement the exact public names/signatures above in a focused root file, and add Definition.SequenceIdentities. Convert root records/selectors explicitly into the engine model. Snapshot fields, selectors, and paths on construction and loading, preserving immutable definition reuse.
2. Remove SequenceComposition, Definition.SequenceCompositions, and the private old-shape adapters. Migrate all maintained code/tests/examples; do not edit historical release notes solely to erase the old name or add deprecated aliases.
3. Cover mixed exact/dynamic constructors, compound scalar fields, literal * versus wildcard segments, root/unused selectors, invalid index paths and empty field sets, overlap failure before source access, and independent loads.
4. Prove dynamic constraints apply before strict composition and at inherited destinations, without consumer namespace enumeration. Keep final-result enforcement described as pending until Stage 27 actually adds it.
5. Verify the public boundary and declaration documentation. Do not expose engine enums, policy stores, parser nodes, or internal methods.

Validation: root sequence/facade, registration, public/dependency-boundary, and consumer example tests plus the everyday check.

Stage 23 — Separate Reusable CLI Override Preparation

Scope: mechanical preparation refactor and a privately tested post-structural validation entry, without changing current CLI processing order.

1. Refactor override preparation so labels, paths, snapshots, type support, disjointness, version protection, and existing budgets remain reusable outside the source assembly accumulator. Allocate synthetic sources once in the same order; do not reopen or reinterpret CLI values as YAML.
2. Prepare post-structural validation that rejects extends as well as configkit in supplied paths and nested values, using their established structural roles. Test this entry privately; retain the existing entry's behavior until the lifecycle switches in Stage 26. Do not reject previously valid early-CLI instructions before that switch or add unrelated restrictions.
3. Test detached prepared batches, synthetic source ordering, and the new directive validation. Reuse existing label/path/cycle/type-support/budget tests rather than adding another inventory of the same preparation cases.

Validation: override preparation/source tests and the everyday check. Keep ordinary/layered behavior unchanged; typed assignment is the next stage.

Stage 24 — Implement Schema-Aware Post-Decode CLI Assignment

Scope: a private typed assignment helper using Stage 23's prepared overrides, without activating the later lifecycle order yet.

1. Add applyDecodedOverrides, using existing schema/decode mechanisms to bind only supplied override subtrees against a detached copy of the decoded state. Mappings recursively overlay, sequences replace, and explicit zero/empty/null retain normal schema rules. Current typed values supply retained children, not new defaults for replaced list items. Do not call Definition.Defaults again or re-decode the whole projected configuration as newly authored input.
2. Return the updated typed value, canonical post-CLI projection, presence/origin changes, and existing raw CLI history contributions. Preserve a pre-CLI typed projection for decoding reconciliation/history. No resolver runs on a failed override; no partially applied result is returned.
3. Test unknown/type-invalid fields, mapping retention, sequence replacement, nested maps/lists, named/text/duration/secret schema fields, explicit null/false/ zero, discarded children, structural controls, and mutable ownership.

Validation: override, decode, schema, projection, and ownership tests plus the everyday check. Only the helper is added here; the shared lifecycle does not switch until its inspection/context support is prepared in Stage 25.

Stage 25 — Prepare Post-Decode CLI Inspection And Effective Context

Scope: prepare history, resolution-input, and secret-lookup context handling; retain the existing public CLI lifecycle until Stage 26.

1. Extend existing replay/reconciliation to represent the target order: source tiers, inheritance, typed decoding/defaults, CLI, application resolution. Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and post-CLI projection distinct where needed. Use the existing history machinery, not a second replay framework, and retain legacy input compatibility until the lifecycle switch. Do not mislabel CLI changes as defaults or inheritance.
2. Prepare resolution-entry metadata from Stage 24's result so explicit presence, origins, source records, and resource contexts describe the overridden state.
3. Prepare the effective reference/location handoff for secret lookup and final validation using the post-override/post-resolution values, not stale structural input. Preserve existing no-credential-byte and ownership behavior.
4. Test the target pipeline with privately assembled snapshots: equal-value CLI replacement, replaced descendants, decoding reconciliation before CLI, later recorded resolver transformations, explicit-presence defaults, and an overridden secret reference. Verify current public-load regressions still pass unchanged.

Validation: history/metadata/resolution/secret tests and the everyday check. Public activation must not require a later stage to repair these mechanisms.

Stage 26 — Activate CLI After Decoding In The Shared Lifecycle

Scope: activate ADR 0018 through both loading paths using Stages 23–25's complete assignment, inspection, and effective-value handoffs.

1. Remove CLI assignment from ordinary/layered structural assembly. Invoke Stage 24 after inheritance and successful typed-default creation/decoding, immediately before application resolution. Activate Stage 23's post-structural validation. Keep the single shared lifecycle, not separate CLI implementations.
2. Wire Stage 25's history and context handoffs in the same change. Resolution receives explicit-presence/origin metadata and effective CLI values; Resolution.Default preserves them while recorded transformations can change them normally. Secret lookup uses effective references and locations. Preserve exactly-once default-factory behavior and synthetic source ordering.
3. Cover NoSource, ordinary/layered equivalence, CLI clearing/replacing an identified list despite append/upsert mode, derived values seeing CLI input, equal-value provenance, and resolver changes after CLI. Add explicit regressions that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a relationship. Update obsolete early-CLI tests deliberately.
4. Update the lifecycle, override, history, and secret-reference contracts and affected examples to describe the now-implemented order.
5. Remove the superseded early-CLI application paths, old preparation entry, and temporary replay compatibility paths. Keep their still-relevant behavioral tests under the new lifecycle rather than postponing code cleanup to the documentation stage.

Validation: override/lifecycle/resolution/inheritance/history/secret tests, go test -race ./..., and the everyday check.

Stage 27 — Enforce Final Application Identity Constraints

Scope: the final-result guarantee, not another combination or inheritance phase.

1. Add shared final identity validation after effective secret lookup and before application semantic validation/publication in the common lifecycle. Use the canonical effective typed projection, which retains secret references rather than credential bytes. Preserve the existing validated publication snapshot and detachment behavior.
2. Traverse the actual final tree with the compiled selector trie, validating each matching sequence independently with Stage 1 primitives. This must discover paths introduced by typed defaults, CLI, and callbacks, not only assembly paths. Support root/namespaced rules and unrelated unselected sequences.
3. Validate actual sequences only. Missing paths, null, and other shapes remain the schema/application validator's responsibility. Identity extraction here examines effective typed payload, not whether a field was authored. Do not recheck YAML-only policies or every intermediate callback state.
4. Cover duplicate/missing/non-scalar identities produced at each late phase, valid null/scalar identities, numeric equality after typed conversion, dynamic namespaces created by each late phase, harmless unmatched rules, concrete-path diagnostics with available origins, and secret-value non-disclosure.
5. Prove both Load and LoadLayered use this guarantee and return no usable result on failure. Update registration comments and canonical contract with this final boundary, excluding later caller mutations from the guarantee.

Validation: final identity, selector, facade, defaults/resolution/override, and secret tests plus the everyday check.

Stage 28 — Exercise Representative Consumer Workflows

Scope: a small number of meaningful integrations, not a Cartesian test matrix.

1. Extend examples/consumers with an executable source-backed catalog: underlay records, main fragments, selected overlay append/upsert/replace, and explicit record inheritance. Demonstrate complete replacement of a matched record, stable order, and final inspection without application merge code.
2. Add a namespaced typed-map example using one wildcard registration plus an exact root registration, with independent uniqueness. Show YAML-declared identity without registration separately so the final guarantee is not implied.
3. Include one integrated CLI/resolution case and one representative provenance/ resource-context/secret-history case where those interactions add protection beyond mechanism tests. Keep application domain validation recognizable.
4. Audit coverage against the roadmap's work inventory, especially helper-built parent references, version-header consumption, pending declarations, lower identity before higher peers, replacement ownership, and dynamic destinations introduced through inheritance and late typed changes. Add only missing, consequential regressions at their narrowest stable owners.
5. Consolidate obsolete/brittle tests. Do not restore old replacement semantics merely to keep fixtures unchanged, and do not preserve duplicates of the same semantic table across every layer.

Validation: go test ./examples/consumers, focused root integration tests, go test -race ./..., and the everyday check.

Stage 29 — Reconcile Documentation

Scope: finish the current-contract narrative. Code cleanup belongs with the integrations that supersede each adapter, not in this documentation prompt.

1. Reconcile docs/config.md for declarations, identity equality, selectors, operations/persistence, ownership, inheritance, final validation, and CLI order. Keep its operation and presence tables authoritative; Go comments own exported symbols and consumer guidance links rather than copying full contracts.
2. Update docs/consumers/README.md, relevant root examples, README.md, docs/internal/overview.md, docs/internal/imports.md, and docs/development.md where the implemented structure or reading/test map has changed. Remove stale claims about string-only registration, universal sequence replacement, early CLI, and overlay prohibition of all directives.
3. Remove implemented portions from the deferred identified-record discussion in future.md, retaining genuinely deferred recursive body merging and other still-relevant ideas. Preserve accepted ADR decision content; update only status or implementation references where appropriate. Architecture already adopts the decisions: remove pending-implementation wording when true, without adding new policy. Leave release version selection/publication to the release task.
4. Search maintained prose/examples for obsolete registration names and lifecycle claims; retain intentional historical references. Follow canonical ownership rather than copying the complete contract into each guide.

Validation: local documentation-link checks and whitespace review; run relevant example tests if executable examples changed. Reuse Stage 28's successful code/race validation when its covered content is unchanged.

Stage 30 — Audit Scaling And Verify The Completed Target

Scope: a bounded implementation review and final validation, not another broad refactor or expansion of the feature/performance scope.

1. Inspect the completed code for duplicate combination/identity algorithms, per-record scans of whole collections, repeated declaration gathering, numeric magnitude expansion, stale ownership/path caches, and unnecessary copies. Prefer shared indexed mechanisms without pooling/global caches. Reuse existing composition/underlay/inheritance benchmarks and add a focused mixed-upsert or wildcard-collection benchmark only where existing fixtures cannot exercise the new scaling risk. Record commands and qualitative findings here; timing or allocation thresholds and a quiet-host benchmark campaign are not gates.
2. Confirm transitional adapters/dead early-CLI paths have been removed. Search current code and maintained examples for SequenceCompositions and SequenceComposition; only intentional historical references may remain. Confirm no new dependencies, public internal-type exposure, background work, release machinery, or additional limits were introduced. Review the full diff against the roadmap, not merely passing tests. Record bounded review findings here; significant corrective work requires its own focused unit rather than being hidden inside this stage. Do not declare completion while a target behavior remains incorrect or missing.

Validation: ./scripts/check.sh, go test -race ./..., and the development guide's local documentation/public-boundary checks, reusing successful checks from Stages 28–29 if their covered content is unchanged. Record completed scope, validation, benchmark commands/qualitative findings, and any real remaining issues accurately. No new timing thresholds or release gates are introduced.

Acceptance: all roadmap operations and lifecycle boundaries work through ordinary and layered loads; values and inspection agree; application constraints hold on successful final results; deferred behavior and YAML-only guarantees have not expanded. All thirty stages are complete and their nonempty units committed.

Open Questions

None. Feature choices are settled in the roadmap and accepted ADRs; this plan selects the remaining API, representation, validation, and execution decisions.