ConfigKit/docs/roadmap/implementation.md

# Sequence Identity And Combination Implementation Plan

Status: implementation in progress; Stages 1–11 complete.

Audience: a Luna coding agent implementing one numbered stage per prompt, in
order. This plan starts at commit
`4f941f8ba4c9a9ed0afa87b07fe6ba29f131d1c6`. It implements the settled
[sequence roadmap](sequences.md), [ADR 0017](../adr/0017-collection-sequence-policy.md),
and [ADR 0018](../adr/0018-cli-after-structural-resolution.md).

## Working Rules

- Read [development guidance](../development.md) and the applicable policies,
  especially [agents](../policy/agents.md), [architecture](../policy/architecture.md),
  [testing](../policy/testing.md), and [documentation](../policy/documentation.md).
  Prefer the repository knowledge graph for code discovery.
- The roadmap owns the selected feature semantics. This plan supplies execution
  order and implementation choices, not additional product requirements. Preserve
  unchanged source, parser, version, schema, ownership, and inspection contracts.
- Complete each stage's implementation and behavioral tests together. Record its
  completion and validation here, then commit the nonempty unit with a short,
  plain-English message. Preserve unrelated changes. Do not publish intermediate
  stages or a release, update consumers' dependencies, or wait for CI.
- Run focused tests while iterating and `./scripts/check.sh` before handing off
  each code stage. Use the installed compatible Go toolchain with dependencies
  already acquired, as the development guide describes. Add race coverage at the
  ownership milestones below. These are implementation checks, not new release
  gates. Reuse successful checks for unchanged content rather than repeating them
  at a documentation-only milestone. Do not require human platform validation.
- Update affected declaration comments and canonical current-behavior prose when
  behavior lands. Keep incomplete target behavior in this roadmap/plan until it
  works. The final documentation stage reconciles the cross-feature narrative;
  it is not permission to leave misleading current contracts in earlier stages.
- Transitional internal adapters are acceptable where specified. Remove them by
  completion. Do not add public compatibility aliases, permanent feature flags,
  alternate semantics, new dependencies, new resource caps, or an extension/plugin
  framework for this work.
- Preparation stages keep existing public behavior working and test new machinery
  privately. Activation stages integrate values, policy, metadata, and relevant
  failure handling together; do not defer correct inspection to a later stage.

## Implementation Decisions

### Public Registration Surface

Replace `SequenceComposition` and `Definition.SequenceCompositions` with:

```go
type SequenceIdentity struct { /* private, root-owned registration state */ }
type SequenceSelector []SequenceSelectorSegment
type SequenceSelectorSegment struct { /* private literal-or-any-mapping-key state */ }

func SequenceLiteralKey(key string) SequenceSelectorSegment
func SequenceAnyMappingKey() SequenceSelectorSegment
func SequenceIdentityAt(path Path, fields ...string) SequenceIdentity
func SequenceIdentityMatching(selector SequenceSelector, fields ...string) SequenceIdentity

// Field on Definition[T]:
// SequenceIdentities []SequenceIdentity
```

For example, the reusable definition can contain both registrations:

```go
SequenceIdentities: []configkit.SequenceIdentity{
    configkit.SequenceIdentityAt(configkit.Keys("prompts"), "id"),
    configkit.SequenceIdentityMatching(configkit.SequenceSelector{
        configkit.SequenceLiteralKey("collections"),
        configkit.SequenceAnyMappingKey(),
        configkit.SequenceLiteralKey("prompts"),
    }, "id"),
},
```

The exact constructor converts mapping-only `Path` segments into literal selector
segments. Preserve invalid index segments long enough to report an ordinary
definition error; do not panic, silently drop them, or change `Path` semantics.
The zero selector is the exact root selector, harmless under the existing
mapping-root schema. A zero selector segment denotes the literal empty key,
not a wildcard. Empty field names are valid literal field names; an empty field
set is invalid. Constructors snapshot slices; load-entry conversion also takes
an owned snapshot. Definition validation, not construction, reports errors.
Root types and functions must not expose internal types or bridge methods.

### Identity Representation And Matching

- Normalize identity field sets once: deduplicate and sort exact field names.
  Use the same normalizer for YAML and application registration.
- Encode a record's identity as a hashable, type-tagged, length-delimited tuple in
  normalized field order. Mapping lookup indexes avoid scanning record fields
  repeatedly. Distinguish structural-payload lookup from final typed-payload and
  parent-reference lookup: only fields acting as directives are excluded, not
  every occurrence of the same literal spelling. Never include tuple encodings
  in diagnostics.
- Normalize finite JSON numbers to sign, significant decimal digits, and a signed
  base-10 exponent. Strip insignificant zeroes and give all zero spellings one
  representation. Use `math/big.Int` for exponent arithmetic, not a power-of-ten
  expansion or a float conversion. Thus large exponent spellings do not allocate
  storage proportional to their mathematical magnitude. This comparison-only
  representation does not rewrite document nodes or typed projection.
- Compile selectors into a mapping-segment trie with literal and wildcard edges.
  Carry applicable trie states through traversal; stop at sequence-index edges.
  Index concrete matches within one assembly/inheritance scope where useful, but
  do not save an assembly-only destination list for final validation.
- Check registration overlap pairwise once during definition validation using
  equal length and compatible corresponding segments. This straightforward
  `O(R² × D)` check, for R registrations and maximum selector depth D, is
  proportionate for registration definitions. Equivalent constraints coalesce;
  conflicting overlaps fail even with no matching input.
  Do not impose a registration-count limit to simplify this operation.

### Policy And Evidence Storage

Keep application constraints separate from YAML policy. Add private engine
collection-policy records containing normalized identity, explicit mode, and
declaration locations. Track whether identity is pending or belongs to a retained
collection. Retain declaration presence separately from inferred defaults.

Use the existing acquisition/item scopes as ownership anchors, with an opaque
per-load owner token when a retained record needs one. Store item-local policies
relative to their owning record, and project concrete paths when that record is
placed. Do not key ownership solely by numeric index, identity tuple, or source
filename: all three can coincide for a replacement that must lose old policies.
Inheritance clones the relevant scoped state; it does not create registrations.

Resolve policies through shared engine helpers used by strict composition,
precedence, and inheritance. Do not attach engine policy to public values or add
policy fields to the parser-independent `document.Node`. The sidecar can be
extended with concrete-path indexes; it need not become a general event store.

Sequence combination returns the resulting list plus explicit input-to-output
item projections and retained/replaced contribution information. Use that same
result for values, policy transfer, origins, source destinations, sensitivity,
and historical replay. Extend the existing sequence-history descriptors for
noncontiguous upsert positions rather than inferring them from equal values.
Sharing immutable internal evidence is acceptable under existing ownership rules;
independently returned mutable records must remain detached.

### Diagnostics

Use existing `options.invalid` for invalid application definitions and existing
`compose.invalid` / `compose.conflict` for malformed or duplicate strict-peer
contributions. Add root-owned and engine diagnostic constants
`CodeSequenceInvalid` (`sequence.invalid`) and `CodeSequenceConflict`
(`sequence.conflict`) for malformed collection declarations/operations or final
identities, and conflicting policy / precedence duplicates respectively.
Final duplicate identities use `sequence.conflict`. Keep existing inheritance
invalid/missing/cycle codes for record relationships. Use existing override and
schema/decode codes at the CLI boundary; do not invent a new error taxonomy there.

Report collection/item paths and available locations, not identity values,
normalized tuple strings, raw parent references, or raw hook errors. Declaration
conflicts include both sites. Definition overlap errors identify registration
positions without fabricating a concrete matched configuration path. Exact
English diagnostic wording is not a new guarantee.

## Stage 1 — Implement Shared Scalar Identity Primitives

Scope: new private identity helpers and their focused tests; no load-path change.

1. Add `internal/engine/sequence_identity.go`. Implement field-set normalization,
   scalar tuple construction, and exact decimal normalization as chosen above.
   Accept string, number, boolean, and explicit null nodes; distinguish categories
   and absence. Exclude fields acting as structural directives from payload lookup.
2. Validate mapping records, complete direct-field sets, and uniqueness using one
   tuple-to-first-record index. Return paths and source locations, not values.
   Support caller-selected diagnostic families for strict and other boundaries.
3. Cover equivalent field sets; empty strings and null; missing/non-scalar fields;
   literal punctuation; compound tuple boundaries; `1`/`1.0`/`1e0`; signed zero;
   close unequal decimals; and large exponents without expansion. Verify that
   node text remains unchanged. Include `extends: null` as a structural field
   that cannot supply source identity.

Validation: `go test ./internal/engine -run 'SequenceIdentity|SequenceNumber'`
and the everyday check. Retain the old registration path until its adapters are
replaced; do not generalize unrelated YAML scalar decoding.

**Completion:** Implemented and verified with the focused identity tests and
`./scripts/check.sh`.

## Stage 2 — Compile Exact And Dynamic Application Constraints

Scope: private registration model, selector matching, overlap validation.

1. Extend `sequence_registration.go` with the common registration representation
   and compiled selector trie. Represent literal and arbitrary-mapping-key segments
   structurally, not with a magic string. Snapshot registration inputs.
2. Validate field sets and mapping-only selectors before source acquisition.
   Preserve existing destination validation without adding restrictions on identity
   field spelling. Root and unused selectors remain harmless.
3. Implement full-destination matching, equivalent overlap coalescing, and
   incompatible-overlap errors. Exact rules have no priority over dynamic rules.
   Keep a temporary conversion from the current single-field registration so
   existing callers still use their current behavior.
4. Test exact and wildcard matches, multiple wildcard levels, empty/punctuation
   keys, sequence-index nonmatches, disjoint lengths/literals, equivalent sets,
   incompatible overlaps with absent input, and independent uniqueness scopes.

Validation: focused registration/selector tests and the everyday check. This
stage does not export the replacement API or add final-result validation yet.

**Completion:** Implemented and verified with focused registration/selector
tests and `./scripts/check.sh`.

## Stage 3 — Parse Collection Declarations Without Filesystem Coupling

Scope: private parser and directive extraction helpers, initially unit-tested
without activating collection combination in public loads. Keep source-form
activation out of this stage; ordinary sources and selected overlays are wired
in Stages 13 and 14 respectively.

1. Add `sequence_declaration.go` for source-root `configkit.sequences`. Each entry
   has a required existing literal `destination` form and optional `key` and
   `mode`. `key` is a string or nonempty string list; `mode` is exactly replace,
   append, or upsert. Use shared identity normalization. A destination-only entry
   can be a harmless no-op; do not add an unrequested ban on empty declarations.
2. Split common root-directive extraction from import-specific parsing in
   `import_spec.go` / `parse.go`. Extract imports and sequences together, retaining
   source-local locations and explicit presence. Reject unrecognized directive
   fields and wrong supplied field types using existing strict parsing style.
3. Keep declaration destinations relative to source placement, with item-local
   coordinates retained until their scope is assembled. Do not project them
   prematurely to a guessed numeric index.
4. Give extraction an explicit overlay context: sequences are allowed, imports
   are not. Import enablement remains separately enforced; do not turn on
   filesystem access because a source declares sequences. Keep enabled
   source-version checks in raw source coordinates before consuming controls;
   extend existing actual header/control-overlap handling to sequence directives
   without treating an absent header beneath a control as an actual overlap.
5. Test key-only, mode-only, combined, repeated-field-set equivalence, no-op,
   malformed types/modes/destinations, mixed imports/sequences, and overlay rules.
   Do not add import-level key/mode shorthand or sequence wrappers.

Validation: focused declaration/parser tests, existing import-spec tests, and the
everyday check. Existing import-only behavior remains intact during preparation.

**Completion:** Implemented and verified with focused declaration/parser and
import-spec tests plus `./scripts/check.sh`.

## Stage 4 — Reconcile Declarations And Select Collection Operations

Scope: pure agreement and operation-selection helpers, independent of loading
and of the ownership transformations introduced next.

1. Add `collection_policy.go`. Gather the current scope's declarations before
   combining peers. Reconcile equivalent keys and agreeing explicit modes;
   conflicting peers fail. Omitted fields have no preference. Apply authoritative
   application constraints even for key-only declarations without a value.
2. Implement shared operation selection: current explicit mode, retained explicit
   mode, inferred upsert with effective identity, otherwise replace. Persist only
   explicit modes. Policy-only declarations must not replay earlier combinations.
3. Accept eligible lower policy as an input; do not decide record survival or
   ancestor removal inside declaration reconciliation. Retain declaration
   locations/presence separately from inferred mode and application constraints.
4. Test equivalent/different field sets, agreeing/conflicting peer modes,
   omission, explicit default modes, and operation inference. Include key-only
   declarations agreeing or conflicting with application constraints before a
   sequence exists.

Validation: focused policy-agreement/selection tests and the everyday check.
Do not activate collection combination or implement the ownership layer here.

**Completion:** Implemented and verified with focused collection-policy tests and
`./scripts/check.sh`.

## Stage 5 — Track Policy Lifetime And Record Ownership

Scope: private scoped-state transformations built on Stage 4's resolution helpers.

1. Implement key agreement across retained values, replacement reset, pending
   key declarations, and immediate validation when a key-only declaration
   identifies retained records.
   Changing mode to replace without a replacement value does not reset identity.
2. Implement scope/record ownership transfer and retirement hooks. Actual ancestor
   replacement removes descendant collection identity and removed-record policy;
   independently owned modes and application constraints survive as specified in
   the roadmap. Do not repeatedly reapply old declarations and resurrect reset
   identity. Pending declarations retain their owning-scope rules.
3. Use item/acquisition scope anchors and record-owned relative policy as selected
   above. Provide transformations for retention, whole-record replacement,
   whole-sequence replacement, and inheritance copies without wiring them into
   public loading yet. Application registrations remain destination constraints.
4. Use table-driven tests derived from the roadmap's lifetime, persistence, and
   replacement tables, including absent/non-sequence destinations, equal outer
   identities with distinct record owners, and independent root versus item policy.

Validation: focused collection-policy tests and the everyday check. This stage
must not add path-existence, shape, nullability, or global YAML final-check rules.

**Completion:** Implemented and verified with focused collection-policy tests,
`go test -race ./internal/engine`, and `./scripts/check.sh`.

## Stage 6 — Implement The Shared Sequence Operations

Scope: a pure document-level combination helper, not separate implementations
for overlays and inheritance.

1. Add `sequence_combine.go`, accepting lower/higher sequence nodes, resolved
   policy, and combination context. Reuse Stage 1 identity extraction and indexes.
2. Implement replace, duplicate-rejecting append, and whole-record upsert. Upsert
   retains matched lower positions and appends new higher identities in order.
   Return item projections and replacement/retention evidence with the new value.
3. Keep peer validation distinct: duplicate identities within any contribution or
   among peers fail under every mode. Do not merge record payloads or sort by ID.
4. Cover absent versus null/non-sequence lower values; higher omission, `[]`, null,
   and non-sequence values; missing identity under append/upsert; empty inputs;
   stable ordering; equivalent/different keys; and explicit replacement identity
   reset. An empty input does not bypass kind checks or identity requirements.
5. Verify clones do not alias mutable input nodes or policy slices. Build indexes
   once per operation rather than scanning the accumulated list for every record.

Validation: focused combination tests and the everyday check. Do not change the
ordinary recursive mapping rule or add unkeyed append.

**Completion:** Implemented and verified with focused sequence-combination tests
and `./scripts/check.sh`.

## Stage 7 — Extend Sequence History Representation And Replay

Scope: represent and replay sequence operations without yet changing all metadata
projection consumers or public precedence.

1. Extend `sequenceHistoryContribution`, sequence replay in `history.go`, and
   its contribution indexes to consume Stage 6's explicit item mappings. Existing
   append ranges may remain a compact case; upsert cannot be represented by one
   offset.
2. Preserve authored raw contribution snapshots and displaced values. A replaced
   record's omitted fields must appear as removed, not as retained lower payload.
   Preserve equal-valued replacement events. Keep existing index-based query and
   atomic sequence-diff APIs; add no public identity-selector path syntax.
3. Exercise synthetic append/upsert/replace histories, noncontiguous matches,
   nested paths, and discarded descendants. Compare replayed final values with
   the actual combination result, including first/last and unchanged neighboring
   records. Reuse existing escaped-path and detached-query tests rather than
   duplicating them.

Validation: sequence/history/replay tests and the everyday check. Keep current
append/replace callers working through the extended descriptors.

**Completion:** Implemented and verified with focused sequence-history tests and
`./scripts/check.sh`.

## Stage 8 — Project Sequence Origins And Inspection Evidence

Scope: use the prepared item mappings consistently in metadata consumers.

1. Project composition origins, nested source destinations, historical paths, and
   sensitivity through Stage 6's item mapping and Stage 7's history descriptors.
   Physical source IDs/locations and resource contexts continue to identify the
   declaring source. Reuse one projection mechanism rather than searching again
   by identity or equal payload.
2. Test synthetic noncontiguous upserts, replaced descendants, nested source
   destinations, unchanged neighbors, and secret-marked replaced records. Verify
   projected queries agree with replayed and actual final values, retaining
   displaced evidence and detached results.
3. Keep the new precedence operations unactivated until their value, policy, and
   projection machinery are integrated in the later load-path stages.

Validation: metadata/history/sequence/provenance/sensitivity tests,
`go test -race ./internal/engine`, and the everyday check.

**Completion:** Implemented and verified with focused projection, history,
sequence, provenance, and sensitivity tests, `go test -race ./internal/engine`,
and `./scripts/check.sh`.

## Stage 9 — Separate Per-Input Acquisition From Composition

Scope: a behavior-preserving acquired-input handoff; grouped underlay coordination
is the next stage.

1. In `load_input.go`, `imports.go`, and related graph helpers, retain acquisition
   scopes, raw contributions, inclusion edges, source records,
   contexts, and extracted declarations without irrevocably composing them.
   Reuse existing `deferAssembly` and acquired-scope mechanisms instead of adding
   a second loader. Keep already acquired document handles closed as before.
2. Separate acquisition from the per-input composition entry. Retain pending
   item/helper scope boundaries, namespace identities, inclusion/repetition checks,
   source records, accounting, and scheduler inputs. Do not reacquire sources to
   compose them later or change grouped coordination yet.
3. Preserve existing version checks for this refactor. Retain a small adapter
   invoking current composition semantics at the existing lifecycle point.
   Preserve source-version callbacks at acquisition, including header consumption.
4. Run existing acquisition, dependency scheduling, item placement, source-version,
   ordinary/layered equivalence, resource-lifetime, and source-context tests. Add
   only targeted regressions for any newly split handoff not already covered.

Validation: the focused suites above and the everyday check. This refactor must
not change source IDs or turn acquisition order into precedence order.

**Completion:** Implemented and verified with focused acquisition, scheduling,
placement, source-version, layered-equivalence, resource-lifetime, and context
tests plus `./scripts/check.sh`.

## Stage 10 — Separate Underlay-Group Acquisition From Composition

Scope: behavior-preserving group coordination in `load_request.go`, using the
per-input handoff already prepared in Stage 9.

1. Separate acquiring an underlay group from scheduling/composing its acquired
   scopes. Preserve namespace reanchoring, shared inclusion/repetition state,
   item-local helper scopes, source/context aggregation, and group accounting.
2. Keep authored acquisition order: main and depth-first imports, each underlay
   and input, overlay, then synthetic CLI records. Preserve underlay indexes and
   current version-check timing in this refactor; policy-dependent layered
   composition is activated only in Stage 15.
3. Expose the common scope-composition handoff with inputs for accumulated lower
   policy and compiled application constraints. Initially adapt it to the current
   semantics. Do not add another loader or repeat acquisition/version callbacks.
4. Test the split with existing grouped/dependency/version/destination coverage,
   including empty inputs/groups and repeated sources within versus across tiers.
   Confirm unchanged values, source ordering, contexts, and failure evidence.

Validation: grouped/layered, import scheduling, version, destination, and
resource-context tests plus the everyday check.

**Completion:** Implemented and verified with focused grouped/layered,
import-scheduling, version, destination, and resource-context tests plus
`./scripts/check.sh`.

## Stage 11 — Compose Strict Collections With Resolved Identity

Scope: the policy-aware strict composition core. Nested pending-item scope
integration is deferred to Stage 12, not absorbed into this prompt.

1. Extend `composeDocumentsWithPolicy` and `validateRegisteredSequences` to use
   normalized field sets and scalar tuples. Resolve current declarations,
   applicable application constraints, and eligible lower identity before peer
   composition, using the Stage 4 agreement and Stage 5 lifetime helpers.
2. Use the existing dependency-ready order for contributions and authored list
   order. Validate every identified sequence, including singleton contributions;
   strictly append distinct peers and reject duplicates under all modes.
3. For explicit replacement, do not use obsolete lower YAML identity to authorize
   otherwise unkeyed peer composition. Still apply independently registered
   application constraints and genuinely applicable declarations.
4. Attach Stages 7–8 evidence and Stage 5 scoped policy to composed results. Test
   completed-document contributions with lower-established identity, declaration
   conflicts, compound/scalar IDs, singleton validation, and replacement resets.

Validation: composition and sequence identity tests plus the everyday check.
Keep unregistered non-mapping conflicts unchanged. Public load activation remains
in the later integration stages.

**Completion:** Implemented and verified with focused composition, collection
policy, and sequence identity tests plus `./scripts/check.sh`.

## Stage 12 — Integrate Policy-Aware Item And Helper Scopes

Scope: adapt pending-item/helper composition to Stage 11's core using the acquired
scope boundaries from Stages 9–10.

1. Gather declarations before composing each relevant scope, complete item-local
   helper imports before item placement, and consume applicable non-main version
   headers before checking record payload identity. Keep relative policy with its
   owner until concrete placement is known.
2. Apply eligible lower identity at each scope, but prevent a replacement outer
   record from borrowing nested identity owned by the discarded lower record.
   Preserve independently applicable policy and application constraints.
3. Retained structural directives cannot serve as identity fields. A helper may
   supply payload identity components. Do not require a helper-built compound
   parent reference to be complete at this point; validate the completed surviving
   reference during inheritance.
4. Project item evidence and owned policies through Stages 7–8 machinery. Test
   nested items, helper-supplied identity, header consumption, declaration scope,
   discarded owner identity, and exact diagnostic paths with no value disclosure.

Validation: item/destination/import/composition/source-version tests and the
everyday check. Keep current public behavior until load-path activation.

## Stage 13 — Activate Ordinary Source Collection Declarations

Scope: ordinary `Load` base-source assembly, excluding activation of the new
selected-overlay policy surface until Stage 14.

1. Wire common directive extraction into every ordinary source form: file,
   native filesystem, bytes, reader, mapping, imported mapping, and sequence item.
   Declarations work without `Imports`; disabled imports retain their existing
   rejection semantics. `NoSource` remains absence rather than an empty document.
2. Use Stages 11–12 composition and Stage 5 scoped policy. Strip collection
   controls from application decoding, but retain their locations/evidence for
   their structural lifetime and diagnostics.
3. Integrate policy/projections together with the values. Cover inline/imported
   equivalence, placement-relative declarations, singleton and peer validation,
   and declarations without accompanying values. Maintain ordinary loading's
   existing non-collection overlay behavior while preparing its policy handoff.
4. Update the implemented declaration/composition contract sections. Collection
   precedence, inheritance, and later CLI behavior remain incomplete until their
   integration stages; do not claim them implemented here.

Validation: ordinary-load, import, composition, and metadata tests plus
the everyday check. Include one public ordinary-load declaration example test.

## Stage 14 — Activate Selected Overlay Collection Policies

Scope: ordinary loading's selected overlay and shared precedence integration,
using the source/declaration handoff already active in Stage 13.

1. Permit `configkit.sequences` in selected overlays while continuing to reject
   overlay imports. Preserve source-version header/control handling and ordinary
   overlay namespace selection. Consume controls before application decoding.
2. Use Stage 6's sequence operations and Stages 4–5 policy helpers in the assembly
   accumulator. Mappings still recurse and unrelated sequences still replace.
   Wire value, policy retirement/transfer, and Stages 7–8 inspection projections
   together, not as a later best-effort repair pass.
3. Cover append/upsert/replace, empty/null/kind cases, key-only/mode-only changes,
   retained explicit modes, overlay identity agreement, and independently owned
   policies. Check values, history, origins, and discarded descendants together.
4. Update the selected-overlay and collection-operation contract sections. Make
   the shared precedence entry reusable by layered loading without adding another
   operation implementation or turning overlay imports on.

Validation: ordinary-load, overlay, composition, policy, and metadata tests plus
the everyday check.

## Stage 15 — Integrate Lower Policy Across Layered Assembly

Scope: source-backed underlay coordination and the layered selected-overlay path,
using the prepared acquisition, ownership, strict-scope, and precedence machinery.
Do not absorb unfinished work from Stages 5–14 into this stage.

1. Acquire main first, but assemble underlays from lowest to highest, strictly
   composing each group with applicable accumulated lower policy before applying
   its one precedence operation. Then compose/apply main and the selected overlay.
   Do not sequentially upsert peer sources as if they were different tiers.
2. Pass lower identity into higher strict scopes before peer composition. Cover an
   otherwise empty underlay establishing identity, inherited explicit mode, and
   replacement that deliberately discards the old YAML key. Invalid lower tiers
   still fail even if a higher replacement would hide their values.
3. Keep main-only version ownership and every enabled per-source version rule.
   In default source-version mode, run the main checker once on the composed
   main tier before overlay/inheritance/defaults, never on underlay-augmented
   values. Its former main-before-underlay composition timing need not persist:
   main composition now depends on lower policy. Enabled-mode source checks keep
   acquisition order and exact source-local header handling. No extra callbacks
   or second acquisition pass are permitted.
4. Keep histories in precedence order, sources in acquisition order, underlay
   indexes stable across empty groups, and shared budgets intact. Update the
   layered contract's acquisition-versus-composition description accordingly.
   Remove the acquisition/composition adapters superseded by this integration;
   the old public-registration adapter remains until Stage 22.
5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main,
   explicit empty groups, layered overlays, version protection and diagnostics,
   and equivalence with ordinary loading where the inputs express the same model.

Validation: layered/grouped, source-version, dependency scheduling, destination,
and metadata tests plus the everyday check.

## Stage 16 — Verify Replacement And Nested Policy Ownership End To End

Scope: the consequential lifetime cases across the now-active source assembly.

1. Exercise an item-local nested collection through retaining the outer record,
   upserting the same outer ID, and replacing the entire outer sequence. Retained
   items keep their local key/mode; replacements do not inherit discarded policy
   merely because identity, source label, or final index is equal.
2. Exercise ancestor replacement and later reintroduction. Removed collection
   identity does not reappear; independently declared append mode can survive
   and require renewed identity. Cover renewed YAML key, application constraint,
   explicit replace, and the intended failure with no available identity.
3. Cover key-only validation of retained records, pending declarations, a no-value
   replace declaration that cannot authorize key changes, and independent policy
   whose owning scope survives a record replacement.
4. Correct any integration defects in the Stage 5 ownership transformations;
   consolidate tests with their mechanism owners. Do not introduce special-case
   ownership by numeric path or a historical identity cache to make cases pass.
5. Verify frozen metadata, origin/resource context, sensitivity of discarded
   records, and mutable independence across repeated loads for these scenarios.

Validation: focused policy/sequence/layered/ownership tests,
`go test -race ./internal/engine`, and the everyday check.

## Stage 17 — Prepare Inheritance Policy And Template Transport

Scope: behavior-preserving inheritance state transport, before activating
collection-aware inheritance operations in Stage 18.

1. Extend inheritance templates/expansion sidecars to carry collection identity
   and record-local policy alongside nodes and origins. Keep existing scheduler
   keys, cancellation, version protection, missing-parent/cycle diagnostics, and
   destination-local evaluation of nested templates.
2. Provide owned-state copy/transfer helpers from Stage 5 and path projections from
   Stage 8. Preserve original nested instructions for each destination's later
   evaluation rather than retaining only a parent's expanded output. Keep
   application registration matching separate from copied YAML policy.
3. Test template reuse at multiple destinations, independent scoped state, and
   retained instruction locations/resource context. Preserve current inheritance
   values/history while these new helpers remain preparatory; do not activate a
   value change without its corresponding policy/evidence handling.

Validation: existing inheritance, ownership, history, and resource-context tests,
focused transport tests, and the everyday check.

## Stage 18 — Apply Collection Policy During Mapping-Sibling Inheritance

Scope: activate shared collection operations using the prepared template/state
handoff; existing mapping-sibling relationship selection remains unchanged.

1. Use the shared sequence combiner at the inheriting destination: parent lower,
   child higher, destination explicit mode or identity-based default. Copy YAML
   identity where applicable, but do not automatically copy a parent collection's
   explicit mode to a different destination. Match application constraints at the
   destination, including newly created matching paths.
2. Transfer record-owned state only with surviving records using Stage 17's
   helpers. Integrate inheritance evidence with Stage 8's projections in the same
   activation, preserving origins, historical sensitivity, resource context,
   cancellation, and existing expansion/history accounting.
3. Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key
   with inferred upsert, destination explicit mode, incompatible keys, application
   constraints, cancellation, and policy at an inherited new namespace.
4. Update current mapping-sibling inheritance documentation for collection
   operations. Identified-record parent lookup remains incomplete until Stage 21.

Validation: inheritance, history, provenance, sensitivity, and resource-context
tests plus the everyday check. No identified-record parent lookup is active yet.

## Stage 19 — Build Identified-Record Inheritance Graphs

Scope: relationship extraction/validation and shared scheduling for sequence
scopes; test these helpers before activating payload expansion.

1. Extend sibling-scope discovery to identified sequences. Build the tuple-to-item
   lookup once using effective identity. Every record supplies its own complete
   identity before inheritance; neither parent lookup nor typed defaults supplies
   a missing child field.
2. Parse a surviving record's `extends` as null cancellation or a mapping with
   exactly the normalized identity fields and supported scalar components.
   Mapping-sibling references keep their existing string form. Do not accept a
   scalar shorthand, multiple parents, positional or cross-collection lookup.
3. Parent-reference mappings have already used ordinary strict mapping
   composition: disjoint helper fields can complete one reference, while equal
   overlapping fields conflict. Validate completeness/extra fields only now on
   the surviving reference. Discarded references remain historical evidence.
4. Build one-parent dependency edges and reuse `scheduleDependencies` with stable
   semantic-pointer keys. Missing parents, self-reference, and longer cycles
   use existing inheritance diagnostics without identity values. Keep authored
   output positions independent from graph schedule.
5. Cover compound and null references, numeric equality, forward edges, chains,
   unrelated equal identities in different lists, malformed refs, missing parents,
   cycles, helper-built references, and overwritten/cancelled relationships.

Validation: focused inheritance graph/reference tests and the everyday check.

## Stage 20 — Prepare Reusable Inheritance Scope Execution

Scope: remove mapping-container assumptions from the private template executor
without activating identified-record inheritance in public loads.

1. Refactor template execution to separate relationship scheduling/payload
   expansion from reading and rebuilding its scope container. Preserve current
   mapping-sibling output, locations, cancellation, evidence, and budgets.
2. Provide private mapping- and identified-sequence-scope adapters using Stage 19's
   relationship graphs. The sequence adapter preserves output positions and
   exposes child identity fields to the payload-expansion step. Keep this private;
   do not add a public generic inheritance or merge callback API.
3. Reuse Stage 17's template/policy transport and Stage 18's collection-aware
   payload operations. Existing origin/history/retained-node handling must remain
   available to both adapters rather than duplicating an entire executor.
4. Run existing mapping-inheritance regressions and privately exercise a small
   sequence graph through the adapter. Check output order, independent templates,
   and operation/evidence paths before public activation in Stage 21.

Validation: inheritance execution, scheduling, history, and budget tests plus
the everyday check. This refactor does not introduce new parent-reference forms
or change public loading semantics yet.

## Stage 21 — Resolve Record Inheritance With Ownership And Evidence

Scope: integrate Stage 19's graphs and Stage 20's executor into the structural
expansion phase, with values and inspection working together.

1. Expand parent payload into the child while preserving the child's own identity
   fields and their origin/presence. Exclude parent identity fields from payload
   inheritance. Recursively combine mappings and use destination collection policy
   for nested sequences; retain whole-record upsert at the earlier boundary.
2. Consume the child's directive before decoding. Keep parents as ordinary records
   subject to the same schema/semantic validation; create no abstract templates or
   acquired source records.
3. Resolve only records surviving tier assembly. Cover parent upsert/removal,
   replacement of a child without `extends`, self-reference not exposing a lower
   predecessor, and nested destination-local relationships. Enclosing scopes
   finish before nested scopes; assembled list order stays unchanged.
4. Carry policy, origin/history, sensitivity, resource context, and existing
   expansion/retained-history accounting through copies. Do not widen budgets or
   require a new inspection-path API.
5. Add public-load regressions for complete inheritance chains and nested
   collections, including secret canaries on success and failure. Update the
   implemented inheritance contract and consumer guidance for this scope.

Validation: all inheritance and cross-feature inspection tests,
`go test -race ./internal/engine`, and the everyday check.

## Stage 22 — Expose The Root-Owned Registration API

Scope: the settled facade surface and migration of in-repository callers.

1. Implement the exact public names/signatures above in a focused root file, and
   add `Definition.SequenceIdentities`. Convert root records/selectors explicitly
   into the engine model. Snapshot fields, selectors, and paths on construction
   and loading, preserving immutable definition reuse.
2. Remove `SequenceComposition`, `Definition.SequenceCompositions`, and the private
   old-shape adapters. Migrate all maintained code/tests/examples; do not edit
   historical release notes solely to erase the old name or add deprecated aliases.
3. Cover mixed exact/dynamic constructors, compound scalar fields, literal `*`
   versus wildcard segments, root/unused selectors, invalid index paths and empty
   field sets, overlap failure before source access, and independent loads.
4. Prove dynamic constraints apply before strict composition and at inherited
   destinations, without consumer namespace enumeration. Keep final-result
   enforcement described as pending until Stage 27 actually adds it.
5. Verify the public boundary and declaration documentation. Do not expose engine
   enums, policy stores, parser nodes, or internal methods.

Validation: root sequence/facade, registration, public/dependency-boundary, and
consumer example tests plus the everyday check.

## Stage 23 — Separate Reusable CLI Override Preparation

Scope: mechanical preparation refactor and a privately tested post-structural
validation entry, without changing current CLI processing order.

1. Refactor override preparation so labels, paths, snapshots, type support,
   disjointness, version protection, and existing budgets remain reusable outside
   the source assembly accumulator. Allocate synthetic sources once in the same
   order; do not reopen or reinterpret CLI values as YAML.
2. Prepare post-structural validation that rejects `extends` as well as
   `configkit` in supplied paths and nested values, using their established
   structural roles. Test this entry privately; retain the existing entry's
   behavior until the lifecycle switches in Stage 26. Do not reject previously
   valid early-CLI instructions before that switch or add unrelated restrictions.
3. Test detached prepared batches, synthetic source ordering, and the new
   directive validation. Reuse existing label/path/cycle/type-support/budget tests
   rather than adding another inventory of the same preparation cases.

Validation: override preparation/source tests and the everyday check. Keep
ordinary/layered behavior unchanged; typed assignment is the next stage.

## Stage 24 — Implement Schema-Aware Post-Decode CLI Assignment

Scope: a private typed assignment helper using Stage 23's prepared overrides,
without activating the later lifecycle order yet.

1. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind
   only supplied override subtrees against a detached copy of the decoded state.
   Mappings recursively overlay, sequences replace, and explicit zero/empty/null
   retain normal schema rules. Current typed values supply retained children,
   not new defaults for replaced list items. Do not call `Definition.Defaults`
   again or re-decode the whole projected configuration as newly authored input.
2. Return the updated typed value, canonical post-CLI projection, presence/origin
   changes, and existing raw CLI history contributions. Preserve a pre-CLI typed
   projection for decoding reconciliation/history. No resolver runs on a failed
   override; no partially applied result is returned.
3. Test unknown/type-invalid fields, mapping retention, sequence replacement,
   nested maps/lists, named/text/duration/secret schema fields, explicit null/false/
   zero, discarded children, structural controls, and mutable ownership.

Validation: override, decode, schema, projection, and ownership tests plus the
everyday check. Only the helper is added here; the shared lifecycle does not
switch until its inspection/context support is prepared in Stage 25.

## Stage 25 — Prepare Post-Decode CLI Inspection And Effective Context

Scope: prepare history, resolution-input, and secret-lookup context handling;
retain the existing public CLI lifecycle until Stage 26.

1. Extend existing replay/reconciliation to represent the target order:
   source tiers, inheritance, typed decoding/defaults, CLI, application resolution.
   Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and
   post-CLI projection distinct where needed. Use the existing history machinery,
   not a second replay framework, and retain legacy input compatibility until the
   lifecycle switch. Do not mislabel CLI changes as defaults or inheritance.
2. Prepare resolution-entry metadata from Stage 24's result so explicit presence,
   origins, source records, and resource contexts describe the overridden state.
3. Prepare the effective reference/location handoff for secret lookup and final
   validation using the post-override/post-resolution values, not stale structural
   input. Preserve existing no-credential-byte and ownership behavior.
4. Test the target pipeline with privately assembled snapshots: equal-value CLI
   replacement, replaced descendants, decoding reconciliation before CLI, later
   recorded resolver transformations, explicit-presence defaults, and an overridden
   secret reference. Verify current public-load regressions still pass unchanged.

Validation: history/metadata/resolution/secret tests and the everyday check.
Public activation must not require a later stage to repair these mechanisms.

## Stage 26 — Activate CLI After Decoding In The Shared Lifecycle

Scope: activate ADR 0018 through both loading paths using Stages 23–25's complete
assignment, inspection, and effective-value handoffs.

1. Remove CLI assignment from ordinary/layered structural assembly. Invoke
   Stage 24 after inheritance and successful typed-default creation/decoding,
   immediately before application resolution. Activate Stage 23's post-structural
   validation. Keep the single shared lifecycle, not separate CLI implementations.
2. Wire Stage 25's history and context handoffs in the same change. Resolution
   receives explicit-presence/origin metadata and effective CLI values;
   `Resolution.Default` preserves them while recorded transformations can change
   them normally. Secret lookup uses effective references and locations. Preserve
   exactly-once default-factory behavior and synthetic source ordering.
3. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an
   identified list despite append/upsert mode, derived values seeing CLI input,
   equal-value provenance, and resolver changes after CLI. Add explicit regressions
   that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a
   relationship. Update obsolete early-CLI tests deliberately.
4. Update the lifecycle, override, history, and secret-reference contracts and
   affected examples to describe the now-implemented order.
5. Remove the superseded early-CLI application paths, old preparation entry, and
   temporary replay compatibility paths. Keep their still-relevant behavioral
   tests under the new lifecycle rather than postponing code cleanup to the
   documentation stage.

Validation: override/lifecycle/resolution/inheritance/history/secret tests,
`go test -race ./...`, and the everyday check.

## Stage 27 — Enforce Final Application Identity Constraints

Scope: the final-result guarantee, not another combination or inheritance phase.

1. Add shared final identity validation after effective secret lookup and before
   application semantic validation/publication in the common lifecycle. Use the
   canonical effective typed projection, which retains secret references rather
   than credential bytes. Preserve the existing validated publication snapshot
   and detachment behavior.
2. Traverse the actual final tree with the compiled selector trie, validating each
   matching sequence independently with Stage 1 primitives. This must discover
   paths introduced by typed defaults, CLI, and callbacks, not only assembly paths.
   Support root/namespaced rules and unrelated unselected sequences.
3. Validate actual sequences only. Missing paths, null, and other shapes remain
   the schema/application validator's responsibility. Identity extraction here
   examines effective typed payload, not whether a field was authored. Do not
   recheck YAML-only policies or every intermediate callback state.
4. Cover duplicate/missing/non-scalar identities produced at each late phase,
   valid null/scalar identities, numeric equality after typed conversion, dynamic
   namespaces created by each late phase, harmless unmatched rules, concrete-path
   diagnostics with available origins, and secret-value non-disclosure.
5. Prove both `Load` and `LoadLayered` use this guarantee and return no usable
   result on failure. Update registration comments and canonical contract with
   this final boundary, excluding later caller mutations from the guarantee.

Validation: final identity, selector, facade, defaults/resolution/override, and
secret tests plus the everyday check.

## Stage 28 — Exercise Representative Consumer Workflows

Scope: a small number of meaningful integrations, not a Cartesian test matrix.

1. Extend `examples/consumers` with an executable source-backed catalog: underlay
   records, main fragments, selected overlay append/upsert/replace, and explicit
   record inheritance. Demonstrate complete replacement of a matched record,
   stable order, and final inspection without application merge code.
2. Add a namespaced typed-map example using one wildcard registration plus an
   exact root registration, with independent uniqueness. Show YAML-declared
   identity without registration separately so the final guarantee is not implied.
3. Include one integrated CLI/resolution case and one representative provenance/
   resource-context/secret-history case where those interactions add protection
   beyond mechanism tests. Keep application domain validation recognizable.
4. Audit coverage against the roadmap's work inventory, especially helper-built
   parent references, version-header consumption, pending declarations, lower
   identity before higher peers, replacement ownership, and dynamic destinations
   introduced through inheritance and late typed changes. Add only missing,
   consequential regressions at their narrowest stable owners.
5. Consolidate obsolete/brittle tests. Do not restore old replacement semantics
   merely to keep fixtures unchanged, and do not preserve duplicates of the same
   semantic table across every layer.

Validation: `go test ./examples/consumers`, focused root integration tests,
`go test -race ./...`, and the everyday check.

## Stage 29 — Reconcile Documentation

Scope: finish the current-contract narrative. Code cleanup belongs with the
integrations that supersede each adapter, not in this documentation prompt.

1. Reconcile `docs/config.md` for declarations, identity equality, selectors,
   operations/persistence, ownership, inheritance, final validation, and CLI order.
   Keep its operation and presence tables authoritative; Go comments own exported
   symbols and consumer guidance links rather than copying full contracts.
2. Update `docs/consumers/README.md`, relevant root examples, `README.md`,
   `docs/internal/overview.md`, `docs/internal/imports.md`, and
   `docs/development.md` where the implemented structure or reading/test map has
   changed. Remove stale claims about string-only registration, universal sequence
   replacement, early CLI, and overlay prohibition of all directives.
3. Remove implemented portions from the deferred identified-record discussion in
   `future.md`, retaining genuinely deferred recursive body merging and other
   still-relevant ideas. Preserve accepted ADR decision content; update only status
   or implementation references where appropriate. Architecture already adopts
   the decisions: remove pending-implementation wording when true, without adding
   new policy. Leave release version selection/publication to the release task.
4. Search maintained prose/examples for obsolete registration names and lifecycle
   claims; retain intentional historical references. Follow canonical ownership
   rather than copying the complete contract into each guide.

Validation: local documentation-link checks and whitespace review; run relevant
example tests if executable examples changed. Reuse Stage 28's successful
code/race validation when its covered content is unchanged.

## Stage 30 — Audit Scaling And Verify The Completed Target

Scope: a bounded implementation review and final validation, not another broad
refactor or expansion of the feature/performance scope.

1. Inspect the completed code for duplicate combination/identity algorithms,
   per-record scans of whole collections, repeated declaration gathering, numeric
   magnitude expansion, stale ownership/path caches, and unnecessary copies.
   Prefer shared indexed mechanisms without pooling/global caches. Reuse existing
   composition/underlay/inheritance benchmarks and add a focused mixed-upsert or
   wildcard-collection benchmark only where existing fixtures cannot exercise the
   new scaling risk. Record commands and qualitative findings here; timing or
   allocation thresholds and a quiet-host benchmark campaign are not gates.
2. Confirm transitional adapters/dead early-CLI paths have been removed. Search
   current code and maintained examples for `SequenceCompositions` and
   `SequenceComposition`; only intentional historical references may remain.
   Confirm no new dependencies, public internal-type exposure, background work,
   release machinery, or additional limits were introduced. Review the full diff
   against the roadmap, not merely passing tests. Record bounded review findings
   here; significant corrective
   work requires its own focused unit rather than being hidden inside this stage.
   Do not declare completion while a target behavior remains incorrect or missing.

Validation: `./scripts/check.sh`, `go test -race ./...`, and the development
guide's local documentation/public-boundary checks, reusing successful checks
from Stages 28–29 if their covered content is unchanged. Record completed scope,
validation, benchmark commands/qualitative findings, and any real remaining
issues accurately. No new timing thresholds or release gates are introduced.

Acceptance: all roadmap operations and lifecycle boundaries work through ordinary
and layered loads; values and inspection agree; application constraints hold on
successful final results; deferred behavior and YAML-only guarantees have not
expanded. All thirty stages are complete and their nonempty units committed.

## Open Questions

None. Feature choices are settled in the roadmap and accepted ADRs; this plan
selects the remaining API, representation, validation, and execution decisions.