diff --git a/docs/development.md b/docs/development.md index bb74b59..3a52a65 100644 --- a/docs/development.md +++ b/docs/development.md @@ -167,6 +167,21 @@ The subsequent import-integration audit and completed remediation record are preserved at `6903c9e:docs/roadmap/remediation.md`; retrieve them with `git show 6903c9e:docs/roadmap/remediation.md`. +The completed sequence-identity/collection-policy roadmap and implementation +plan are preserved in Git history at `d0a605a`: + +```sh +git show d0a605a:docs/roadmap/sequences.md +git show d0a605a:docs/roadmap/implementation.md +``` + +Current behavior belongs to the configuration contract's +[strict composition](config.md#strict-document-composition), +[sibling inheritance](config.md#sibling-inheritance), and +[CLI overrides](config.md#explicit-cli-overrides) sections. +The full [remediation record](roadmap/remediation.md) remains available, including +the open algorithm-audit findings. + Potential enhancements are tracked in [future features](roadmap/future.md). Install the Go version required by `go.mod` or a compatible newer toolchain. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index b91ce9a..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,1073 +0,0 @@ -# Sequence Identity And Combination Implementation Plan - -Status: all thirty stages complete. - -Audience: a Luna coding agent implementing one numbered stage per prompt, in -order. This plan starts at commit -`4f941f8ba4c9a9ed0afa87b07fe6ba29f131d1c6`. It implements the settled -[sequence roadmap](sequences.md), [ADR 0017](../adr/0017-collection-sequence-policy.md), -and [ADR 0018](../adr/0018-cli-after-structural-resolution.md). - -## Working Rules - -- Read [development guidance](../development.md) and the applicable policies, - especially [agents](../policy/agents.md), [architecture](../policy/architecture.md), - [testing](../policy/testing.md), and [documentation](../policy/documentation.md). - Prefer the repository knowledge graph for code discovery. -- The roadmap owns the selected feature semantics. This plan supplies execution - order and implementation choices, not additional product requirements. Preserve - unchanged source, parser, version, schema, ownership, and inspection contracts. -- Complete each stage's implementation and behavioral tests together. Record its - completion and validation here, then commit the nonempty unit with a short, - plain-English message. Preserve unrelated changes. Do not publish intermediate - stages or a release, update consumers' dependencies, or wait for CI. -- Run focused tests while iterating and `./scripts/check.sh` before handing off - each code stage. Use the installed compatible Go toolchain with dependencies - already acquired, as the development guide describes. Add race coverage at the - ownership milestones below. These are implementation checks, not new release - gates. Reuse successful checks for unchanged content rather than repeating them - at a documentation-only milestone. Do not require human platform validation. -- Update affected declaration comments and canonical current-behavior prose when - behavior lands. Keep incomplete target behavior in this roadmap/plan until it - works. The final documentation stage reconciles the cross-feature narrative; - it is not permission to leave misleading current contracts in earlier stages. -- Transitional internal adapters are acceptable where specified. Remove them by - completion. Do not add public compatibility aliases, permanent feature flags, - alternate semantics, new dependencies, new resource caps, or an extension/plugin - framework for this work. -- Preparation stages keep existing public behavior working and test new machinery - privately. Activation stages integrate values, policy, metadata, and relevant - failure handling together; do not defer correct inspection to a later stage. - -## Implementation Decisions - -### Public Registration Surface - -Replace `SequenceComposition` and `Definition.SequenceCompositions` with: - -```go -type SequenceIdentity struct { /* private, root-owned registration state */ } -type SequenceSelector []SequenceSelectorSegment -type SequenceSelectorSegment struct { /* private literal-or-any-mapping-key state */ } - -func SequenceLiteralKey(key string) SequenceSelectorSegment -func SequenceAnyMappingKey() SequenceSelectorSegment -func SequenceIdentityAt(path Path, fields ...string) SequenceIdentity -func SequenceIdentityMatching(selector SequenceSelector, fields ...string) SequenceIdentity - -// Field on Definition[T]: -// SequenceIdentities []SequenceIdentity -``` - -For example, the reusable definition can contain both registrations: - -```go -SequenceIdentities: []configkit.SequenceIdentity{ - configkit.SequenceIdentityAt(configkit.Keys("prompts"), "id"), - configkit.SequenceIdentityMatching(configkit.SequenceSelector{ - configkit.SequenceLiteralKey("collections"), - configkit.SequenceAnyMappingKey(), - configkit.SequenceLiteralKey("prompts"), - }, "id"), -}, -``` - -The exact constructor converts mapping-only `Path` segments into literal selector -segments. Preserve invalid index segments long enough to report an ordinary -definition error; do not panic, silently drop them, or change `Path` semantics. -The zero selector is the exact root selector, harmless under the existing -mapping-root schema. A zero selector segment denotes the literal empty key, -not a wildcard. Empty field names are valid literal field names; an empty field -set is invalid. Constructors snapshot slices; load-entry conversion also takes -an owned snapshot. Definition validation, not construction, reports errors. -Root types and functions must not expose internal types or bridge methods. - -### Identity Representation And Matching - -- Normalize identity field sets once: deduplicate and sort exact field names. - Use the same normalizer for YAML and application registration. -- Encode a record's identity as a hashable, type-tagged, length-delimited tuple in - normalized field order. Mapping lookup indexes avoid scanning record fields - repeatedly. Distinguish structural-payload lookup from final typed-payload and - parent-reference lookup: only fields acting as directives are excluded, not - every occurrence of the same literal spelling. Never include tuple encodings - in diagnostics. -- Normalize finite JSON numbers to sign, significant decimal digits, and a signed - base-10 exponent. Strip insignificant zeroes and give all zero spellings one - representation. Use `math/big.Int` for exponent arithmetic, not a power-of-ten - expansion or a float conversion. Thus large exponent spellings do not allocate - storage proportional to their mathematical magnitude. This comparison-only - representation does not rewrite document nodes or typed projection. -- Compile selectors into a mapping-segment trie with literal and wildcard edges. - Carry applicable trie states through traversal; stop at sequence-index edges. - Index concrete matches within one assembly/inheritance scope where useful, but - do not save an assembly-only destination list for final validation. -- Check registration overlap pairwise once during definition validation using - equal length and compatible corresponding segments. This straightforward - `O(R² × D)` check, for R registrations and maximum selector depth D, is - proportionate for registration definitions. Equivalent constraints coalesce; - conflicting overlaps fail even with no matching input. - Do not impose a registration-count limit to simplify this operation. - -### Policy And Evidence Storage - -Keep application constraints separate from YAML policy. Add private engine -collection-policy records containing normalized identity, explicit mode, and -declaration locations. Track whether identity is pending or belongs to a retained -collection. Retain declaration presence separately from inferred defaults. - -Use the existing acquisition/item scopes as ownership anchors, with an opaque -per-load owner token when a retained record needs one. Store item-local policies -relative to their owning record, and project concrete paths when that record is -placed. Do not key ownership solely by numeric index, identity tuple, or source -filename: all three can coincide for a replacement that must lose old policies. -Inheritance clones the relevant scoped state; it does not create registrations. - -Resolve policies through shared engine helpers used by strict composition, -precedence, and inheritance. Do not attach engine policy to public values or add -policy fields to the parser-independent `document.Node`. The sidecar can be -extended with concrete-path indexes; it need not become a general event store. - -Sequence combination returns the resulting list plus explicit input-to-output -item projections and retained/replaced contribution information. Use that same -result for values, policy transfer, origins, source destinations, sensitivity, -and historical replay. Extend the existing sequence-history descriptors for -noncontiguous upsert positions rather than inferring them from equal values. -Sharing immutable internal evidence is acceptable under existing ownership rules; -independently returned mutable records must remain detached. - -### Diagnostics - -Use existing `options.invalid` for invalid application definitions and existing -`compose.invalid` / `compose.conflict` for malformed or duplicate strict-peer -contributions. Add root-owned and engine diagnostic constants -`CodeSequenceInvalid` (`sequence.invalid`) and `CodeSequenceConflict` -(`sequence.conflict`) for malformed collection declarations/operations or final -identities, and conflicting policy / precedence duplicates respectively. -Final duplicate identities use `sequence.conflict`. Keep existing inheritance -invalid/missing/cycle codes for record relationships. Use existing override and -schema/decode codes at the CLI boundary; do not invent a new error taxonomy there. - -Report collection/item paths and available locations, not identity values, -normalized tuple strings, raw parent references, or raw hook errors. Declaration -conflicts include both sites. Definition overlap errors identify registration -positions without fabricating a concrete matched configuration path. Exact -English diagnostic wording is not a new guarantee. - -## Stage 1 — Implement Shared Scalar Identity Primitives - -Scope: new private identity helpers and their focused tests; no load-path change. - -1. Add `internal/engine/sequence_identity.go`. Implement field-set normalization, - scalar tuple construction, and exact decimal normalization as chosen above. - Accept string, number, boolean, and explicit null nodes; distinguish categories - and absence. Exclude fields acting as structural directives from payload lookup. -2. Validate mapping records, complete direct-field sets, and uniqueness using one - tuple-to-first-record index. Return paths and source locations, not values. - Support caller-selected diagnostic families for strict and other boundaries. -3. Cover equivalent field sets; empty strings and null; missing/non-scalar fields; - literal punctuation; compound tuple boundaries; `1`/`1.0`/`1e0`; signed zero; - close unequal decimals; and large exponents without expansion. Verify that - node text remains unchanged. Include `extends: null` as a structural field - that cannot supply source identity. - -Validation: `go test ./internal/engine -run 'SequenceIdentity|SequenceNumber'` -and the everyday check. Retain the old registration path until its adapters are -replaced; do not generalize unrelated YAML scalar decoding. - -**Completion:** Implemented and verified with the focused identity tests and -`./scripts/check.sh`. - -## Stage 2 — Compile Exact And Dynamic Application Constraints - -Scope: private registration model, selector matching, overlap validation. - -1. Extend `sequence_registration.go` with the common registration representation - and compiled selector trie. Represent literal and arbitrary-mapping-key segments - structurally, not with a magic string. Snapshot registration inputs. -2. Validate field sets and mapping-only selectors before source acquisition. - Preserve existing destination validation without adding restrictions on identity - field spelling. Root and unused selectors remain harmless. -3. Implement full-destination matching, equivalent overlap coalescing, and - incompatible-overlap errors. Exact rules have no priority over dynamic rules. - Keep a temporary conversion from the current single-field registration so - existing callers still use their current behavior. -4. Test exact and wildcard matches, multiple wildcard levels, empty/punctuation - keys, sequence-index nonmatches, disjoint lengths/literals, equivalent sets, - incompatible overlaps with absent input, and independent uniqueness scopes. - -Validation: focused registration/selector tests and the everyday check. This -stage does not export the replacement API or add final-result validation yet. - -**Completion:** Implemented and verified with focused registration/selector -tests and `./scripts/check.sh`. - -## Stage 3 — Parse Collection Declarations Without Filesystem Coupling - -Scope: private parser and directive extraction helpers, initially unit-tested -without activating collection combination in public loads. Keep source-form -activation out of this stage; ordinary sources and selected overlays are wired -in Stages 13 and 14 respectively. - -1. Add `sequence_declaration.go` for source-root `configkit.sequences`. Each entry - has a required existing literal `destination` form and optional `key` and - `mode`. `key` is a string or nonempty string list; `mode` is exactly replace, - append, or upsert. Use shared identity normalization. A destination-only entry - can be a harmless no-op; do not add an unrequested ban on empty declarations. -2. Split common root-directive extraction from import-specific parsing in - `import_spec.go` / `parse.go`. Extract imports and sequences together, retaining - source-local locations and explicit presence. Reject unrecognized directive - fields and wrong supplied field types using existing strict parsing style. -3. Keep declaration destinations relative to source placement, with item-local - coordinates retained until their scope is assembled. Do not project them - prematurely to a guessed numeric index. -4. Give extraction an explicit overlay context: sequences are allowed, imports - are not. Import enablement remains separately enforced; do not turn on - filesystem access because a source declares sequences. Keep enabled - source-version checks in raw source coordinates before consuming controls; - extend existing actual header/control-overlap handling to sequence directives - without treating an absent header beneath a control as an actual overlap. -5. Test key-only, mode-only, combined, repeated-field-set equivalence, no-op, - malformed types/modes/destinations, mixed imports/sequences, and overlay rules. - Do not add import-level key/mode shorthand or sequence wrappers. - -Validation: focused declaration/parser tests, existing import-spec tests, and the -everyday check. Existing import-only behavior remains intact during preparation. - -**Completion:** Implemented and verified with focused declaration/parser and -import-spec tests plus `./scripts/check.sh`. - -## Stage 4 — Reconcile Declarations And Select Collection Operations - -Scope: pure agreement and operation-selection helpers, independent of loading -and of the ownership transformations introduced next. - -1. Add `collection_policy.go`. Gather the current scope's declarations before - combining peers. Reconcile equivalent keys and agreeing explicit modes; - conflicting peers fail. Omitted fields have no preference. Apply authoritative - application constraints even for key-only declarations without a value. -2. Implement shared operation selection: current explicit mode, retained explicit - mode, inferred upsert with effective identity, otherwise replace. Persist only - explicit modes. Policy-only declarations must not replay earlier combinations. -3. Accept eligible lower policy as an input; do not decide record survival or - ancestor removal inside declaration reconciliation. Retain declaration - locations/presence separately from inferred mode and application constraints. -4. Test equivalent/different field sets, agreeing/conflicting peer modes, - omission, explicit default modes, and operation inference. Include key-only - declarations agreeing or conflicting with application constraints before a - sequence exists. - -Validation: focused policy-agreement/selection tests and the everyday check. -Do not activate collection combination or implement the ownership layer here. - -**Completion:** Implemented and verified with focused collection-policy tests and -`./scripts/check.sh`. - -## Stage 5 — Track Policy Lifetime And Record Ownership - -Scope: private scoped-state transformations built on Stage 4's resolution helpers. - -1. Implement key agreement across retained values, replacement reset, pending - key declarations, and immediate validation when a key-only declaration - identifies retained records. - Changing mode to replace without a replacement value does not reset identity. -2. Implement scope/record ownership transfer and retirement hooks. Actual ancestor - replacement removes descendant collection identity and removed-record policy; - independently owned modes and application constraints survive as specified in - the roadmap. Do not repeatedly reapply old declarations and resurrect reset - identity. Pending declarations retain their owning-scope rules. -3. Use item/acquisition scope anchors and record-owned relative policy as selected - above. Provide transformations for retention, whole-record replacement, - whole-sequence replacement, and inheritance copies without wiring them into - public loading yet. Application registrations remain destination constraints. -4. Use table-driven tests derived from the roadmap's lifetime, persistence, and - replacement tables, including absent/non-sequence destinations, equal outer - identities with distinct record owners, and independent root versus item policy. - -Validation: focused collection-policy tests and the everyday check. This stage -must not add path-existence, shape, nullability, or global YAML final-check rules. - -**Completion:** Implemented and verified with focused collection-policy tests, -`go test -race ./internal/engine`, and `./scripts/check.sh`. - -## Stage 6 — Implement The Shared Sequence Operations - -Scope: a pure document-level combination helper, not separate implementations -for overlays and inheritance. - -1. Add `sequence_combine.go`, accepting lower/higher sequence nodes, resolved - policy, and combination context. Reuse Stage 1 identity extraction and indexes. -2. Implement replace, duplicate-rejecting append, and whole-record upsert. Upsert - retains matched lower positions and appends new higher identities in order. - Return item projections and replacement/retention evidence with the new value. -3. Keep peer validation distinct: duplicate identities within any contribution or - among peers fail under every mode. Do not merge record payloads or sort by ID. -4. Cover absent versus null/non-sequence lower values; higher omission, `[]`, null, - and non-sequence values; missing identity under append/upsert; empty inputs; - stable ordering; equivalent/different keys; and explicit replacement identity - reset. An empty input does not bypass kind checks or identity requirements. -5. Verify clones do not alias mutable input nodes or policy slices. Build indexes - once per operation rather than scanning the accumulated list for every record. - -Validation: focused combination tests and the everyday check. Do not change the -ordinary recursive mapping rule or add unkeyed append. - -**Completion:** Implemented and verified with focused sequence-combination tests -and `./scripts/check.sh`. - -## Stage 7 — Extend Sequence History Representation And Replay - -Scope: represent and replay sequence operations without yet changing all metadata -projection consumers or public precedence. - -1. Extend `sequenceHistoryContribution`, sequence replay in `history.go`, and - its contribution indexes to consume Stage 6's explicit item mappings. Existing - append ranges may remain a compact case; upsert cannot be represented by one - offset. -2. Preserve authored raw contribution snapshots and displaced values. A replaced - record's omitted fields must appear as removed, not as retained lower payload. - Preserve equal-valued replacement events. Keep existing index-based query and - atomic sequence-diff APIs; add no public identity-selector path syntax. -3. Exercise synthetic append/upsert/replace histories, noncontiguous matches, - nested paths, and discarded descendants. Compare replayed final values with - the actual combination result, including first/last and unchanged neighboring - records. Reuse existing escaped-path and detached-query tests rather than - duplicating them. - -Validation: sequence/history/replay tests and the everyday check. Keep current -append/replace callers working through the extended descriptors. - -**Completion:** Implemented and verified with focused sequence-history tests and -`./scripts/check.sh`. - -## Stage 8 — Project Sequence Origins And Inspection Evidence - -Scope: use the prepared item mappings consistently in metadata consumers. - -1. Project composition origins, nested source destinations, historical paths, and - sensitivity through Stage 6's item mapping and Stage 7's history descriptors. - Physical source IDs/locations and resource contexts continue to identify the - declaring source. Reuse one projection mechanism rather than searching again - by identity or equal payload. -2. Test synthetic noncontiguous upserts, replaced descendants, nested source - destinations, unchanged neighbors, and secret-marked replaced records. Verify - projected queries agree with replayed and actual final values, retaining - displaced evidence and detached results. -3. Keep the new precedence operations unactivated until their value, policy, and - projection machinery are integrated in the later load-path stages. - -Validation: metadata/history/sequence/provenance/sensitivity tests, -`go test -race ./internal/engine`, and the everyday check. - -**Completion:** Implemented and verified with focused projection, history, -sequence, provenance, and sensitivity tests, `go test -race ./internal/engine`, -and `./scripts/check.sh`. - -## Stage 9 — Separate Per-Input Acquisition From Composition - -Scope: a behavior-preserving acquired-input handoff; grouped underlay coordination -is the next stage. - -1. In `load_input.go`, `imports.go`, and related graph helpers, retain acquisition - scopes, raw contributions, inclusion edges, source records, - contexts, and extracted declarations without irrevocably composing them. - Reuse existing `deferAssembly` and acquired-scope mechanisms instead of adding - a second loader. Keep already acquired document handles closed as before. -2. Separate acquisition from the per-input composition entry. Retain pending - item/helper scope boundaries, namespace identities, inclusion/repetition checks, - source records, accounting, and scheduler inputs. Do not reacquire sources to - compose them later or change grouped coordination yet. -3. Preserve existing version checks for this refactor. Retain a small adapter - invoking current composition semantics at the existing lifecycle point. - Preserve source-version callbacks at acquisition, including header consumption. -4. Run existing acquisition, dependency scheduling, item placement, source-version, - ordinary/layered equivalence, resource-lifetime, and source-context tests. Add - only targeted regressions for any newly split handoff not already covered. - -Validation: the focused suites above and the everyday check. This refactor must -not change source IDs or turn acquisition order into precedence order. - -**Completion:** Implemented and verified with focused acquisition, scheduling, -placement, source-version, layered-equivalence, resource-lifetime, and context -tests plus `./scripts/check.sh`. - -## Stage 10 — Separate Underlay-Group Acquisition From Composition - -Scope: behavior-preserving group coordination in `load_request.go`, using the -per-input handoff already prepared in Stage 9. - -1. Separate acquiring an underlay group from scheduling/composing its acquired - scopes. Preserve namespace reanchoring, shared inclusion/repetition state, - item-local helper scopes, source/context aggregation, and group accounting. -2. Keep authored acquisition order: main and depth-first imports, each underlay - and input, overlay, then synthetic CLI records. Preserve underlay indexes and - current version-check timing in this refactor; policy-dependent layered - composition is activated only in Stage 15. -3. Expose the common scope-composition handoff with inputs for accumulated lower - policy and compiled application constraints. Initially adapt it to the current - semantics. Do not add another loader or repeat acquisition/version callbacks. -4. Test the split with existing grouped/dependency/version/destination coverage, - including empty inputs/groups and repeated sources within versus across tiers. - Confirm unchanged values, source ordering, contexts, and failure evidence. - -Validation: grouped/layered, import scheduling, version, destination, and -resource-context tests plus the everyday check. - -**Completion:** Implemented and verified with focused grouped/layered, -import-scheduling, version, destination, and resource-context tests plus -`./scripts/check.sh`. - -## Stage 11 — Compose Strict Collections With Resolved Identity - -Scope: the policy-aware strict composition core. Nested pending-item scope -integration is deferred to Stage 12, not absorbed into this prompt. - -1. Extend `composeDocumentsWithPolicy` and `validateRegisteredSequences` to use - normalized field sets and scalar tuples. Resolve current declarations, - applicable application constraints, and eligible lower identity before peer - composition, using the Stage 4 agreement and Stage 5 lifetime helpers. -2. Use the existing dependency-ready order for contributions and authored list - order. Validate every identified sequence, including singleton contributions; - strictly append distinct peers and reject duplicates under all modes. -3. For explicit replacement, do not use obsolete lower YAML identity to authorize - otherwise unkeyed peer composition. Still apply independently registered - application constraints and genuinely applicable declarations. -4. Attach Stages 7–8 evidence and Stage 5 scoped policy to composed results. Test - completed-document contributions with lower-established identity, declaration - conflicts, compound/scalar IDs, singleton validation, and replacement resets. - -Validation: composition and sequence identity tests plus the everyday check. -Keep unregistered non-mapping conflicts unchanged. Public load activation remains -in the later integration stages. - -**Completion:** Implemented and verified with focused composition, collection -policy, and sequence identity tests plus `./scripts/check.sh`. - -## Stage 12 — Integrate Policy-Aware Item And Helper Scopes - -Scope: adapt pending-item/helper composition to Stage 11's core using the acquired -scope boundaries from Stages 9–10. - -1. Gather declarations before composing each relevant scope, complete item-local - helper imports before item placement, and consume applicable non-main version - headers before checking record payload identity. Keep relative policy with its - owner until concrete placement is known. -2. Apply eligible lower identity at each scope, but prevent a replacement outer - record from borrowing nested identity owned by the discarded lower record. - Preserve independently applicable policy and application constraints. -3. Retained structural directives cannot serve as identity fields. A helper may - supply payload identity components. Do not require a helper-built compound - parent reference to be complete at this point; validate the completed surviving - reference during inheritance. -4. Project item evidence and owned policies through Stages 7–8 machinery. Test - nested items, helper-supplied identity, header consumption, declaration scope, - discarded owner identity, and exact diagnostic paths with no value disclosure. - -Validation: item/destination/import/composition/source-version tests and the -everyday check. Keep current public behavior until load-path activation. - -**Completion:** Implemented and verified with focused item-scope, destination, -import, composition, collection-policy, and source-version tests plus -`./scripts/check.sh`. - -## Stage 13 — Activate Ordinary Source Collection Declarations - -Scope: ordinary `Load` base-source assembly, excluding activation of the new -selected-overlay policy surface until Stage 14. - -1. Wire common directive extraction into every ordinary source form: file, - native filesystem, bytes, reader, mapping, imported mapping, and sequence item. - Declarations work without `Imports`; disabled imports retain their existing - rejection semantics. `NoSource` remains absence rather than an empty document. -2. Use Stages 11–12 composition and Stage 5 scoped policy. Strip collection - controls from application decoding, but retain their locations/evidence for - their structural lifetime and diagnostics. -3. Integrate policy/projections together with the values. Cover inline/imported - equivalence, placement-relative declarations, singleton and peer validation, - and declarations without accompanying values. Maintain ordinary loading's - existing non-collection overlay behavior while preparing its policy handoff. -4. Update the implemented declaration/composition contract sections. Collection - precedence, inheritance, and later CLI behavior remain incomplete until their - integration stages; do not claim them implemented here. - -Validation: ordinary-load, import, composition, and metadata tests plus -the everyday check. Include one public ordinary-load declaration example test. - -**Completion:** Implemented and verified with ordinary-load, import, composition, -metadata, sequence-item, and source-version tests, the public ordinary-load -example, and `./scripts/check.sh`. - -## Stage 14 — Activate Selected Overlay Collection Policies - -Scope: ordinary loading's selected overlay and shared precedence integration, -using the source/declaration handoff already active in Stage 13. - -1. Permit `configkit.sequences` in selected overlays while continuing to reject - overlay imports. Preserve source-version header/control handling and ordinary - overlay namespace selection. Consume controls before application decoding. -2. Use Stage 6's sequence operations and Stages 4–5 policy helpers in the assembly - accumulator. Mappings still recurse and unrelated sequences still replace. - Wire value, policy retirement/transfer, and Stages 7–8 inspection projections - together, not as a later best-effort repair pass. -3. Cover append/upsert/replace, empty/null/kind cases, key-only/mode-only changes, - retained explicit modes, overlay identity agreement, and independently owned - policies. Check values, history, origins, and discarded descendants together. -4. Update the selected-overlay and collection-operation contract sections. Make - the shared precedence entry reusable by layered loading without adding another - operation implementation or turning overlay imports on. - -Validation: ordinary-load, overlay, composition, policy, and metadata tests plus -the everyday check. - -**Completion:** Implemented and verified with ordinary overlay-operation, -sequence-policy, source-version, and metadata tests. Validation commands: -`go test ./internal/engine -count=1` and `./scripts/check.sh`. - -## Stage 15 — Integrate Lower Policy Across Layered Assembly - -Scope: source-backed underlay coordination and the layered selected-overlay path, -using the prepared acquisition, ownership, strict-scope, and precedence machinery. -Do not absorb unfinished work from Stages 5–14 into this stage. - -1. Acquire main first, but assemble underlays from lowest to highest, strictly - composing each group with applicable accumulated lower policy before applying - its one precedence operation. Then compose/apply main and the selected overlay. - Do not sequentially upsert peer sources as if they were different tiers. -2. Pass lower identity into higher strict scopes before peer composition. Cover an - otherwise empty underlay establishing identity, inherited explicit mode, and - replacement that deliberately discards the old YAML key. Invalid lower tiers - still fail even if a higher replacement would hide their values. -3. Keep main-only version ownership and every enabled per-source version rule. - In default source-version mode, run the main checker once on the composed - main tier before overlay/inheritance/defaults, never on underlay-augmented - values. Its former main-before-underlay composition timing need not persist: - main composition now depends on lower policy. Enabled-mode source checks keep - acquisition order and exact source-local header handling. No extra callbacks - or second acquisition pass are permitted. -4. Keep histories in precedence order, sources in acquisition order, underlay - indexes stable across empty groups, and shared budgets intact. Update the - layered contract's acquisition-versus-composition description accordingly. - Remove the acquisition/composition adapters superseded by this integration; - the old public-registration adapter remains until Stage 22. -5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main, - explicit empty groups, layered overlays, version protection and diagnostics, - and equivalence with ordinary loading where the inputs express the same model. - -Validation: layered/grouped, source-version, dependency scheduling, destination, -and metadata tests plus the everyday check. - -**Completion:** Implemented lower collection-policy propagation across underlay -groups, main composition, and layered overlays. Verified policy-only lower -tiers, inherited identity and explicit mode, explicit replacement, source -version behavior, sequence history, and consumer examples. Validation: -`go test ./internal/engine -count=1` and `./scripts/check.sh`. - -## Stage 16 — Verify Replacement And Nested Policy Ownership End To End - -Scope: the consequential lifetime cases across the now-active source assembly. - -1. Exercise an item-local nested collection through retaining the outer record, - upserting the same outer ID, and replacing the entire outer sequence. Retained - items keep their local key/mode; replacements do not inherit discarded policy - merely because identity, source label, or final index is equal. -2. Exercise ancestor replacement and later reintroduction. Removed collection - identity does not reappear; independently declared append mode can survive - and require renewed identity. Cover renewed YAML key, application constraint, - explicit replace, and the intended failure with no available identity. -3. Cover key-only validation of retained records, pending declarations, a no-value - replace declaration that cannot authorize key changes, and independent policy - whose owning scope survives a record replacement. -4. Correct any integration defects in the Stage 5 ownership transformations; - consolidate tests with their mechanism owners. Do not introduce special-case - ownership by numeric path or a historical identity cache to make cases pass. -5. Verify frozen metadata, origin/resource context, sensitivity of discarded - records, and mutable independence across repeated loads for these scenarios. - -Validation: focused policy/sequence/layered/ownership tests, -`go test -race ./internal/engine`, and the everyday check. - -**Completion:** Updated layered precedence to transfer incoming item-owned -collection policies through sequence identity projections. Verified retained, -upserted, and replaced records, ancestor retirement and reintroduction, renewed -identity, and append-mode ownership. Validation: `go test -race ./internal/engine` -and `./scripts/check.sh`. - -## Stage 17 — Prepare Inheritance Policy And Template Transport - -Scope: behavior-preserving inheritance state transport, before activating -collection-aware inheritance operations in Stage 18. - -1. Extend inheritance templates/expansion sidecars to carry collection identity - and record-local policy alongside nodes and origins. Keep existing scheduler - keys, cancellation, version protection, missing-parent/cycle diagnostics, and - destination-local evaluation of nested templates. -2. Provide owned-state copy/transfer helpers from Stage 5 and path projections from - Stage 8. Preserve original nested instructions for each destination's later - evaluation rather than retaining only a parent's expanded output. Keep - application registration matching separate from copied YAML policy. -3. Test template reuse at multiple destinations, independent scoped state, and - retained instruction locations/resource context. Preserve current inheritance - values/history while these new helpers remain preparatory; do not activate a - value change without its corresponding policy/evidence handling. - -Validation: existing inheritance, ownership, history, and resource-context tests, -focused transport tests, and the everyday check. - -**Completion:** Implemented detached collection-policy transport through -inheritance templates and expansion operations while preserving current values -and nested instructions. Verified focused inheritance/history/resource tests, -`go test -race ./internal/engine`, and `./scripts/check.sh`. - -## Stage 18 — Apply Collection Policy During Mapping-Sibling Inheritance - -Scope: activate shared collection operations using the prepared template/state -handoff; existing mapping-sibling relationship selection remains unchanged. - -1. Use the shared sequence combiner at the inheriting destination: parent lower, - child higher, destination explicit mode or identity-based default. Copy YAML - identity where applicable, but do not automatically copy a parent collection's - explicit mode to a different destination. Match application constraints at the - destination, including newly created matching paths. -2. Transfer record-owned state only with surviving records using Stage 17's - helpers. Integrate inheritance evidence with Stage 8's projections in the same - activation, preserving origins, historical sensitivity, resource context, - cancellation, and existing expansion/history accounting. -3. Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key - with inferred upsert, destination explicit mode, incompatible keys, application - constraints, cancellation, and policy at an inherited new namespace. -4. Update current mapping-sibling inheritance documentation for collection - operations. Identified-record parent lookup remains incomplete until Stage 21. - -Validation: inheritance, history, provenance, sensitivity, and resource-context -tests plus the everyday check. No identified-record parent lookup is active yet. - -**Completion:** Activated collection-aware sequence operations at mapping-sibling -inheritance destinations, including identity transfer, destination-local modes, -record-owned policy projection, and application registrations. Verified focused -inheritance and sequence tests, `go test -race ./internal/engine`, and -`./scripts/check.sh`. - -## Stage 19 — Build Identified-Record Inheritance Graphs - -Scope: relationship extraction/validation and shared scheduling for sequence -scopes; test these helpers before activating payload expansion. - -1. Extend sibling-scope discovery to identified sequences. Build the tuple-to-item - lookup once using effective identity. Every record supplies its own complete - identity before inheritance; neither parent lookup nor typed defaults supplies - a missing child field. -2. Parse a surviving record's `extends` as null cancellation or a mapping with - exactly the normalized identity fields and supported scalar components. - Mapping-sibling references keep their existing string form. Do not accept a - scalar shorthand, multiple parents, positional or cross-collection lookup. -3. Parent-reference mappings have already used ordinary strict mapping - composition: disjoint helper fields can complete one reference, while equal - overlapping fields conflict. Validate completeness/extra fields only now on - the surviving reference. Discarded references remain historical evidence. -4. Build one-parent dependency edges and reuse `scheduleDependencies` with stable - semantic-pointer keys. Missing parents, self-reference, and longer cycles - use existing inheritance diagnostics without identity values. Keep authored - output positions independent from graph schedule. -5. Cover compound and null references, numeric equality, forward edges, chains, - unrelated equal identities in different lists, malformed refs, missing parents, - cycles, helper-built references, and overwritten/cancelled relationships. - -Validation: focused inheritance graph/reference tests and the everyday check. - -**Completion:** Added private identified-sequence graph validation and stable -dependency scheduling for complete scalar identity references, without changing -the active mapping-sibling expansion path. Verified focused graph/reference tests -and `./scripts/check.sh`. - -## Stage 20 — Prepare Reusable Inheritance Scope Execution - -Scope: remove mapping-container assumptions from the private template executor -without activating identified-record inheritance in public loads. - -1. Refactor template execution to separate relationship scheduling/payload - expansion from reading and rebuilding its scope container. Preserve current - mapping-sibling output, locations, cancellation, evidence, and budgets. -2. Provide private mapping- and identified-sequence-scope adapters using Stage 19's - relationship graphs. The sequence adapter preserves output positions and - exposes child identity fields to the payload-expansion step. Keep this private; - do not add a public generic inheritance or merge callback API. -3. Reuse Stage 17's template/policy transport and Stage 18's collection-aware - payload operations. Existing origin/history/retained-node handling must remain - available to both adapters rather than duplicating an entire executor. -4. Run existing mapping-inheritance regressions and privately exercise a small - sequence graph through the adapter. Check output order, independent templates, - and operation/evidence paths before public activation in Stage 21. - -Validation: inheritance execution, scheduling, history, and budget tests plus -the everyday check. This refactor does not introduce new parent-reference forms -or change public loading semantics yet. - -**Completion:** Implemented reusable mapping and identified-sequence scope -execution with focused inheritance/scheduling/history/budget tests, -`go test -race ./internal/engine`, and `./scripts/check.sh`. - -## Stage 21 — Resolve Record Inheritance With Ownership And Evidence - -Scope: integrate Stage 19's graphs and Stage 20's executor into the structural -expansion phase, with values and inspection working together. - -1. Expand parent payload into the child while preserving the child's own identity - fields and their origin/presence. Exclude parent identity fields from payload - inheritance. Recursively combine mappings and use destination collection policy - for nested sequences; retain whole-record upsert at the earlier boundary. -2. Consume the child's directive before decoding. Keep parents as ordinary records - subject to the same schema/semantic validation; create no abstract templates or - acquired source records. -3. Resolve only records surviving tier assembly. Cover parent upsert/removal, - replacement of a child without `extends`, self-reference not exposing a lower - predecessor, and nested destination-local relationships. Enclosing scopes - finish before nested scopes; assembled list order stays unchanged. -4. Carry policy, origin/history, sensitivity, resource context, and existing - expansion/retained-history accounting through copies. Do not widen budgets or - require a new inspection-path API. -5. Add public-load regressions for complete inheritance chains and nested - collections, including secret canaries on success and failure. Update the - implemented inheritance contract and consumer guidance for this scope. - -Validation: all inheritance and cross-feature inspection tests, -`go test -race ./internal/engine`, and the everyday check. - -**Completion:** Activated identified-record inheritance after tier assembly, -including nested collection relationships, ownership and inspection evidence. -Verified inheritance/inspection regressions, `go test -race ./internal/engine`, -and `./scripts/check.sh`. - -## Stage 22 — Expose The Root-Owned Registration API - -Scope: the settled facade surface and migration of in-repository callers. - -1. Implement the exact public names/signatures above in a focused root file, and - add `Definition.SequenceIdentities`. Convert root records/selectors explicitly - into the engine model. Snapshot fields, selectors, and paths on construction - and loading, preserving immutable definition reuse. -2. Remove `SequenceComposition`, `Definition.SequenceCompositions`, and the private - old-shape adapters. Migrate all maintained code/tests/examples; do not edit - historical release notes solely to erase the old name or add deprecated aliases. -3. Cover mixed exact/dynamic constructors, compound scalar fields, literal `*` - versus wildcard segments, root/unused selectors, invalid index paths and empty - field sets, overlap failure before source access, and independent loads. -4. Prove dynamic constraints apply before strict composition and at inherited - destinations, without consumer namespace enumeration. Keep final-result - enforcement described as pending until Stage 27 actually adds it. -5. Verify the public boundary and declaration documentation. Do not expose engine - enums, policy stores, parser nodes, or internal methods. - -Validation: root sequence/facade, registration, public/dependency-boundary, and -consumer example tests plus the everyday check. - -**Completion:** Exposed the root-owned exact and dynamic sequence identity -registration API, migrated maintained callers, and removed the old single-field -registration adapters. Verified facade, registration, inheritance, -public/dependency-boundary, and consumer example tests plus `./scripts/check.sh`. - -## Stage 23 — Separate Reusable CLI Override Preparation - -Scope: mechanical preparation refactor and a privately tested post-structural -validation entry, without changing current CLI processing order. - -1. Refactor override preparation so labels, paths, snapshots, type support, - disjointness, version protection, and existing budgets remain reusable outside - the source assembly accumulator. Allocate synthetic sources once in the same - order; do not reopen or reinterpret CLI values as YAML. -2. Prepare post-structural validation that rejects `extends` as well as - `configkit` in supplied paths and nested values, using their established - structural roles. Test this entry privately; retain the existing entry's - behavior until the lifecycle switches in Stage 26. Do not reject previously - valid early-CLI instructions before that switch or add unrelated restrictions. -3. Test detached prepared batches, synthetic source ordering, and the new - directive validation. Reuse existing label/path/cycle/type-support/budget tests - rather than adding another inventory of the same preparation cases. - -Validation: override preparation/source tests and the everyday check. Keep -ordinary/layered behavior unchanged; typed assignment is the next stage. - -**Completion:** Extracted detached override preparation batches with stable -synthetic source ordering and added a privately exercised post-structural -directive check for `extends`. Existing early override behavior remains active. -Verified focused override tests and `./scripts/check.sh`. - -## Stage 24 — Implement Schema-Aware Post-Decode CLI Assignment - -Scope: a private typed assignment helper using Stage 23's prepared overrides, -without activating the later lifecycle order yet. - -1. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind - only supplied override subtrees against a detached copy of the decoded state. - Mappings recursively overlay, sequences replace, and explicit zero/empty/null - retain normal schema rules. Current typed values supply retained children, - not new defaults for replaced list items. Do not call `Definition.Defaults` - again or re-decode the whole projected configuration as newly authored input. -2. Return the updated typed value, canonical post-CLI projection, presence/origin - changes, and existing raw CLI history contributions. Preserve a pre-CLI typed - projection for decoding reconciliation/history. No resolver runs on a failed - override; no partially applied result is returned. -3. Test unknown/type-invalid fields, mapping retention, sequence replacement, - nested maps/lists, named/text/duration/secret schema fields, explicit null/false/ - zero, discarded children, structural controls, and mutable ownership. - -Validation: override, decode, schema, projection, and ownership tests plus the -everyday check. Only the helper is added here; the shared lifecycle does not -switch until its inspection/context support is prepared in Stage 25. - -**Completion:** Added private schema-aware post-decode CLI assignment with -detached typed results, canonical projection, updated presence/origins, and -preserved pre-CLI/raw override snapshots. Verified focused override, decode, -schema, projection, and ownership tests plus `./scripts/check.sh`. - -## Stage 25 — Prepare Post-Decode CLI Inspection And Effective Context - -Scope: prepare history, resolution-input, and secret-lookup context handling; -retain the existing public CLI lifecycle until Stage 26. - -1. Extend existing replay/reconciliation to represent the target order: - source tiers, inheritance, typed decoding/defaults, CLI, application resolution. - Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and - post-CLI projection distinct where needed. Use the existing history machinery, - not a second replay framework, and retain legacy input compatibility until the - lifecycle switch. Do not mislabel CLI changes as defaults or inheritance. -2. Prepare resolution-entry metadata from Stage 24's result so explicit presence, - origins, source records, and resource contexts describe the overridden state. -3. Prepare the effective reference/location handoff for secret lookup and final - validation using the post-override/post-resolution values, not stale structural - input. Preserve existing no-credential-byte and ownership behavior. -4. Test the target pipeline with privately assembled snapshots: equal-value CLI - replacement, replaced descendants, decoding reconciliation before CLI, later - recorded resolver transformations, explicit-presence defaults, and an overridden - secret reference. Verify current public-load regressions still pass unchanged. - -Validation: history/metadata/resolution/secret tests and the everyday check. -Public activation must not require a later stage to repair these mechanisms. - -**Completion:** Prepared detached post-decode inspection, resolution metadata, -and effective secret-reference/location handoffs using existing history -machinery. Verified the private target pipeline and unchanged public-load -regressions with focused history/metadata/resolution/secret tests and -`./scripts/check.sh`. - -## Stage 26 — Activate CLI After Decoding In The Shared Lifecycle - -Scope: activate ADR 0018 through both loading paths using Stages 23–25's complete -assignment, inspection, and effective-value handoffs. - -1. Remove CLI assignment from ordinary/layered structural assembly. Invoke - Stage 24 after inheritance and successful typed-default creation/decoding, - immediately before application resolution. Activate Stage 23's post-structural - validation. Keep the single shared lifecycle, not separate CLI implementations. -2. Wire Stage 25's history and context handoffs in the same change. Resolution - receives explicit-presence/origin metadata and effective CLI values; - `Resolution.Default` preserves them while recorded transformations can change - them normally. Secret lookup uses effective references and locations. Preserve - exactly-once default-factory behavior and synthetic source ordering. -3. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an - identified list despite append/upsert mode, derived values seeing CLI input, - equal-value provenance, and resolver changes after CLI. Add explicit regressions - that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a - relationship. Update obsolete early-CLI tests deliberately. -4. Update the lifecycle, override, history, and secret-reference contracts and - affected examples to describe the now-implemented order. -5. Remove the superseded early-CLI application paths, old preparation entry, and - temporary replay compatibility paths. Keep their still-relevant behavioral - tests under the new lifecycle rather than postponing code cleanup to the - documentation stage. - -Validation: override/lifecycle/resolution/inheritance/history/secret tests, -`go test -race ./...`, and the everyday check. - -**Completion:** Activated typed CLI assignment after structural expansion and -decoding in the shared load lifecycle, including resolution, history, and -effective secret-reference context. Removed early application and replay fallback -paths, updated contracts and examples, and revised obsolete relationship-change -expectations. Verified focused lifecycle and secret tests, `go test -race ./...`, -and `./scripts/check.sh`. - -## Stage 27 — Enforce Final Application Identity Constraints - -Scope: the final-result guarantee, not another combination or inheritance phase. - -1. Add shared final identity validation after effective secret lookup and before - application semantic validation/publication in the common lifecycle. Use the - canonical effective typed projection, which retains secret references rather - than credential bytes. Preserve the existing validated publication snapshot - and detachment behavior. -2. Traverse the actual final tree with the compiled selector trie, validating each - matching sequence independently with Stage 1 primitives. This must discover - paths introduced by typed defaults, CLI, and callbacks, not only assembly paths. - Support root/namespaced rules and unrelated unselected sequences. -3. Validate actual sequences only. Missing paths, null, and other shapes remain - the schema/application validator's responsibility. Identity extraction here - examines effective typed payload, not whether a field was authored. Do not - recheck YAML-only policies or every intermediate callback state. -4. Cover duplicate/missing/non-scalar identities produced at each late phase, - valid null/scalar identities, numeric equality after typed conversion, dynamic - namespaces created by each late phase, harmless unmatched rules, concrete-path - diagnostics with available origins, and secret-value non-disclosure. -5. Prove both `Load` and `LoadLayered` use this guarantee and return no usable - result on failure. Update registration comments and canonical contract with - this final boundary, excluding later caller mutations from the guarantee. - -Validation: final identity, selector, facade, defaults/resolution/override, and -secret tests plus the everyday check. - -**Completion:** Added shared final identity validation to the ordinary and layered -load lifecycle after effective secret lookup and before application validation. -It traverses the final typed projection with compiled selectors and reports -concrete paths and available locations without exposing identity or credential -values. Added facade and lifecycle coverage for late defaults, CLI, resolution, -dynamic namespaces, typed numeric equality, ignored unmatched shapes, and nil -results on failure. Updated registration comments and the configuration contract. -Verified focused facade and engine tests plus `./scripts/check.sh`. - -## Stage 28 — Exercise Representative Consumer Workflows - -Scope: a small number of meaningful integrations, not a Cartesian test matrix. - -1. Extend `examples/consumers` with an executable source-backed catalog: underlay - records, main fragments, selected overlay append/upsert/replace, and explicit - record inheritance. Demonstrate complete replacement of a matched record, - stable order, and final inspection without application merge code. -2. Add a namespaced typed-map example using one wildcard registration plus an - exact root registration, with independent uniqueness. Show YAML-declared - identity without registration separately so the final guarantee is not implied. -3. Include one integrated CLI/resolution case and one representative provenance/ - resource-context/secret-history case where those interactions add protection - beyond mechanism tests. Keep application domain validation recognizable. -4. Audit coverage against the roadmap's work inventory, especially helper-built - parent references, version-header consumption, pending declarations, lower - identity before higher peers, replacement ownership, and dynamic destinations - introduced through inheritance and late typed changes. Add only missing, - consequential regressions at their narrowest stable owners. -5. Consolidate obsolete/brittle tests. Do not restore old replacement semantics - merely to keep fixtures unchanged, and do not preserve duplicates of the same - semantic table across every layer. - -Validation: `go test ./examples/consumers`, focused root integration tests, -`go test -race ./...`, and the everyday check. - -**Completion:** Added an executable source-backed catalog that combines an -underlay identity declaration, imported main fragments, identified record -inheritance, all three selected overlay operations, typed CLI assignment, and -recorded application resolution. Its checks demonstrate stable operation order, -complete matched-record replacement, final metadata and history inspection, -inherited resource lookup through source context, and secret-safe explanation. -Added a typed-map example with exact and wildcard registrations, independent -uniqueness across concrete collections, and a YAML-only identity example that -shows the absence of final application enforcement without registration. Audited -the existing helper-composed parent reference, source-version control consumption, -pending declarations, lower-tier identity before peer composition, and policy -replacement ownership coverage; added a consumer regression for a dynamic -collection destination introduced by inheritance. Verified -`go test ./examples/consumers -count=1`, -`go test . -run 'Facade.*(Inheritance|Sequence|Override)' -count=1`, -`go test -race ./...`, and `./scripts/check.sh`. - -## Stage 29 — Reconcile Documentation - -Scope: finish the current-contract narrative. Code cleanup belongs with the -integrations that supersede each adapter, not in this documentation prompt. - -1. Reconcile `docs/config.md` for declarations, identity equality, selectors, - operations/persistence, ownership, inheritance, final validation, and CLI order. - Keep its operation and presence tables authoritative; Go comments own exported - symbols and consumer guidance links rather than copying full contracts. -2. Update `docs/consumers/README.md`, relevant root examples, `README.md`, - `docs/internal/overview.md`, `docs/internal/imports.md`, and - `docs/development.md` where the implemented structure or reading/test map has - changed. Remove stale claims about string-only registration, universal sequence - replacement, early CLI, and overlay prohibition of all directives. -3. Remove implemented portions from the deferred identified-record discussion in - `future.md`, retaining genuinely deferred recursive body merging and other - still-relevant ideas. Preserve accepted ADR decision content; update only status - or implementation references where appropriate. Architecture already adopts - the decisions: remove pending-implementation wording when true, without adding - new policy. Leave release version selection/publication to the release task. -4. Search maintained prose/examples for obsolete registration names and lifecycle - claims; retain intentional historical references. Follow canonical ownership - rather than copying the complete contract into each guide. - -Validation: local documentation-link checks and whitespace review; run relevant -example tests if executable examples changed. Reuse Stage 28's successful -code/race validation when its covered content is unchanged. - -**Completion:** Reconciled the configuration contract for identity equality, -selectors, declarations, operation selection and presence, ownership, -inheritance, final registered-identity validation, and post-decode CLI ordering. -Updated consumer, root, internal, development, and architecture guidance; removed -implemented behavior from the deferred feature list while retaining recursive -body merging as future work. Updated ADR implementation references and preserved -intentional historical references. Verified maintained prose/example searches, -`go test . -run '^TestLocalDocumentationLinks$' -count=1`, and `git diff --check`. - -## Stage 30 — Audit Scaling And Verify The Completed Target - -Scope: a bounded implementation review and final validation, not another broad -refactor or expansion of the feature/performance scope. - -1. Inspect the completed code for duplicate combination/identity algorithms, - per-record scans of whole collections, repeated declaration gathering, numeric - magnitude expansion, stale ownership/path caches, and unnecessary copies. - Prefer shared indexed mechanisms without pooling/global caches. Reuse existing - composition/underlay/inheritance benchmarks and add a focused mixed-upsert or - wildcard-collection benchmark only where existing fixtures cannot exercise the - new scaling risk. Record commands and qualitative findings here; timing or - allocation thresholds and a quiet-host benchmark campaign are not gates. -2. Confirm transitional adapters/dead early-CLI paths have been removed. Search - current code and maintained examples for `SequenceCompositions` and - `SequenceComposition`; only intentional historical references may remain. - Confirm no new dependencies, public internal-type exposure, background work, - release machinery, or additional limits were introduced. Review the full diff - against the roadmap, not merely passing tests. Record bounded review findings - here; significant corrective - work requires its own focused unit rather than being hidden inside this stage. - Do not declare completion while a target behavior remains incorrect or missing. - -Validation: `./scripts/check.sh`, `go test -race ./...`, and the development -guide's local documentation/public-boundary checks, reusing successful checks -from Stages 28–29 if their covered content is unchanged. Record completed scope, -validation, benchmark commands/qualitative findings, and any real remaining -issues accurately. No new timing thresholds or release gates are introduced. - -**Completion:** Audited the shared indexed sequence-combination and identity -validation paths, declaration gathering, numeric normalization, ownership and -selector-path handling, adapters, dependencies, boundaries, and limits. Removed -per-declaration temporary-slice allocation while retaining detached declaration -ownership. Added a mixed-size identified-sequence upsert benchmark because the -existing assembly benchmarks did not exercise this path. Its allocations grew -roughly in proportion to record count from 128 to 2,048 records; elapsed times -were similarly proportional in this short run and are descriptive only. - -Benchmark command: - -```sh -go test ./internal/engine -run '^$' -bench '^(BenchmarkComposeWideMappings|BenchmarkUnderlay(BroadCatalog|ManySmallTiers)|BenchmarkOverlay(RecursiveMappings|ManyApplications)|BenchmarkInheritance(Expansion|IndependentScopes)|BenchmarkSequenceMixedUpsert)$' -benchmem -benchtime=100ms -``` - -The benchmark reported mixed-upsert results of 278,009 B and 2,337 allocations -per operation at 128 records; 1,122,177 B and 9,253 allocations at 512; and -4,521,839 B and 36,925 allocations at 2,048. No thresholds are implied. The -code review found shared combination and identity algorithms, indexed matching -and mapping access, declaration grouping by destination, coefficient/exponent -numeric normalization without magnitude expansion, operation-local ownership -and selector traversal, no remaining early-CLI adapter, and no newly introduced -dependencies, public internal types, background work, release machinery, or -sequence-specific resource limits. Obsolete registration names remain only in -historical or roadmap text. - -Verified with `./scripts/check.sh`, `go test -race ./...`, -`go test . -run 'TestLocalDocumentationLinks|TestDependencyImportBoundary|TestPublicFacadeBoundary' -count=1`, -`git diff --check`, and the benchmark command above. No remaining implementation -issues were found. - -Acceptance: all roadmap operations and lifecycle boundaries work through ordinary -and layered loads; values and inspection agree; application constraints hold on -successful final results; deferred behavior and YAML-only guarantees have not -expanded. All thirty stages are complete and their nonempty units committed. - -## Open Questions - -None. Feature choices are settled in the roadmap and accepted ADRs; this plan -selects the remaining API, representation, validation, and execution decisions. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md index 3203ba1..bbadd76 100644 --- a/docs/roadmap/remediation.md +++ b/docs/roadmap/remediation.md @@ -6,8 +6,9 @@ gate. Reviewed commit: `0be7cdb00eff576fda6b30c0600b2134d4e2e086`. Implementation range: `bc6b977..0be7cdb`. -Authority: the [sequence roadmap](sequences.md), -[implementation plan](implementation.md), +Authority: the sequence roadmap and implementation plan preserved at +`d0a605a:docs/roadmap/sequences.md` and +`d0a605a:docs/roadmap/implementation.md`, [ADR 0017](../adr/0017-collection-sequence-policy.md), and [ADR 0018](../adr/0018-cli-after-structural-resolution.md). diff --git a/docs/roadmap/sequences.md b/docs/roadmap/sequences.md deleted file mode 100644 index 93f7135..0000000 --- a/docs/roadmap/sequences.md +++ /dev/null @@ -1,998 +0,0 @@ -# Sequence Identity And Combination Roadmap - -Status: settled feature roadmap; implementation is specified in the -[staged implementation plan](implementation.md). This document defines the -targeted end state, not an implemented contract. The implementation plan owns -execution order, exact Go names and signatures, bookkeeping, and validation. - -## Purpose And Scope - -Allow configuration authors to assemble collections from independently authored -records, declare identity in configuration, and progressively extend a baseline -collection through higher layers. Authors can replace the entire collection, -append distinct records, or replace complete records by identity while retaining -unmatched records. Sibling inheritance uses the same sequence operations and -identity-based defaults. Within an identified sequence, a record can explicitly -inherit settings from another record in that sequence while keeping its own -identity. - -The motivating use case is a prompt catalog whose source-backed baseline supplies -records and whose application or installation layers add or supersede records. -Applications can optionally impose identity constraints that configuration must -respect and that hold in the final loaded configuration. These constraints can -cover exact destinations or collections beneath dynamically named mapping keys, -without requiring the application to discover those names before loading. - -The feature covers ordinary and layered loading, imported items, inline -sequences, selected overlays, and sibling inheritance. Typed defaults, CLI -assignments, and application resolution retain their own combination semantics; -CLI application moves later in the loading lifecycle. - -Automatic recursive merging of matching records remains deferred -(see [future work](future.md#1-recursive-body-merging-for-identified-records)); -explicit inheritance between distinct records is included. The initial -declaration surface is `configkit.sequences`; `key` and `mode` shorthand on -imports is deferred. Application lists remain -ordinary YAML sequences without control wrappers. Unkeyed sequence concatenation -is outside this feature. - -## Current Baseline And Required Changes - -The [import contract](../config.md#explicit-imports) owns existing selectors, -destinations, placement, and ordering. `placement: sequence-item` already places -each selected source mapping as one item at its effective destination. Item-local -helper imports can contribute fields before placement. - -The [composition contract](../config.md#strict-document-composition), -[layered-loading contract](../config.md#layered-loading), and -[inheritance contract](../config.md#sibling-inheritance) describe the implemented -baseline: - -| Area | Current behavior | Target behavior | -| --- | --- | --- | -| Identity declaration | `Definition.SequenceCompositions` registers one literal string field at each registered mapping path. | YAML collection declarations and optional authoritative application registrations support field sets and scalar identity components. | -| Application registration destinations | Registrations select exact literal mapping paths. | Exact and structural selectors can impose identity beneath arbitrary mapping keys, independently at each matched destination. | -| Strict composition | Registered sequences append distinct string identities and reject duplicates; repeated unregistered sequences conflict. | Collections with effective identity append distinct records and reject duplicates. Applicable lower-tier identity is available before composing higher-tier peers. Unidentified sequences retain strict-conflict behavior. | -| Source precedence and selected overlays | A present higher sequence replaces the lower sequence wholesale. | Collection policy selects `replace`, `append`, or `upsert`; overlays can declare policy for inline sequences. | -| Sibling inheritance | An inheriting mapping recursively overlays its resolved parent; supplied sequences replace wholesale. | Mappings retain recursive combination, while sequences use destination policy and the same three operations. | -| Sequence-member inheritance | A sequence element has no sibling scope for its own inheritance relationship. | An identified sequence provides a sibling scope: a record can inherit from one record in the same sequence, selected by complete identity. | -| CLI overrides | Apply before inheritance and typed defaults/decoding. | Apply after inheritance and typed defaults/decoding, before application resolution. | -| Identity enforcement | Registration governs strict sequence composition. | Structural identity checks cover identified combinations; application registrations additionally constrain final effective sequences. | - -Source-backed underlays participate in source assembly. `Definition.Defaults` -provides typed defaults later; extending source combination does not turn typed -default lists into source underlays. - -## Collection Declarations - -Use a source-root `configkit.sequences` block containing collection declarations. -Each entry identifies a `destination` and may declare `key`, `mode`, or both. -Identity and mode can be supplied independently. This form serves imported items, -inline records, ordinary mapping imports, and inline sequences in selected -overlays. - -```yaml -configkit: - sequences: - - destination: [prompts] - key: [namespace, id] - mode: append - imports: - - dir: prompts - pattern: '*.yaml' - destination: [prompts] - placement: sequence-item -``` - -Each selected file contributes a record such as: - -```yaml -namespace: support -id: welcome -text: Welcome -``` - -Here append retains a lower-layer collection and adds distinct higher-layer -records. Identity and combination controls belong to the collection declaration, -so imported and inline contributions share one policy surface. - -### Destinations And Scope - -`destination` uses the existing sequence of literal mapping keys. Resolve it -relative to the declaring source's placement: a fragment mounted under `catalog` -can declare `[prompts]` locally. A sequence-item source can declare a nested -collection within its own item without referring to an eventual numeric index. -Preserve source and item-local scope boundaries. - -A collection's scope is its resolved destination within its composition scope. -Resolve relative destinations before reconciling declarations. Different imported -items retain separate scopes for their nested collections. A declaration governs -all contributions to its collection in the tier, including imported and inline -records; it does not govern only records in the declaring file. - -Gather declarations before composing the relevant scope so their effect does not -depend on which contribution is assembled first. Retain applicable controls -through source precedence and inheritance, then keep them out of application -decoding. - -Sequence declarations are independent of filesystem import enablement. Selected -overlays may contain `configkit.sequences`, extending the current blanket -prohibition on overlay `configkit` declarations. This does not enable imports -in overlays. - -### Overlay Authoring - -For an existing sequence `[foo, bar]`, an overlay can replace `foo` in full: - -```yaml -configkit: - sequences: - - destination: [prompts] - key: id - mode: upsert -prompts: - - id: foo - text: Revised definition -``` - -The result is `[foo*, bar]`. The explicit upsert mode can be omitted when no -different explicit mode applies from a lower tier: effective identity then -selects upsert. To add a distinct record instead: - -```yaml -configkit: - sequences: - - destination: [prompts] - key: id - mode: append -prompts: - - id: baz - text: Another prompt -``` - -The result is `[foo, bar, baz]`. To replace the whole sequence: - -```yaml -configkit: - sequences: - - destination: [prompts] - mode: replace -prompts: - - id: bar - text: Bar definition - - id: baz - text: Baz definition -``` - -The result is `[bar, baz]`. Explicitly selecting the default operation is valid. -With no applicable explicit mode, omitting this declaration selects replace for -an unidentified collection and upsert for an identified one. A retained explicit -mode continues to govern. - -## Identity Definitions And Constraints - -### Field Sets - -`key` accepts a string or a nonempty list of strings naming direct, literal -fields in each record. A name such as `metadata.id` denotes that literal field, -not a nested lookup. Treat field names as a set: remove repeated names and -ignore authored order when comparing definitions. - -Thus `id`, `[id]`, and `[id, id]` are equivalent; `[namespace, id]`, -`[id, namespace]`, and `[namespace, id, id]` are also equivalent. An empty list -is invalid because it supplies no identity fields. These rules apply to both -configuration declarations and application registrations. - -Use a consistent field order internally to construct identity tuples. Preserve -component types and tuple boundaries so unstructured concatenation cannot cause -otherwise distinct identities to collide. - -### Scalar Components And Equality - -Each identified record must be a mapping with every designated identity field -present. Components may be strings, numbers, booleans, or explicit null. Missing -fields and mapping or sequence components are invalid. Present null differs -from absence; two single-field null identities are equal. Compound identities -are equal when all corresponding components are equal. - -Strings compare exactly, including empty strings, without trimming, case folding, -or normalization. Scalar categories remain distinct: `"1"`, `1`, `true`, and -null are different identities. - -Numbers compare by mathematical value across integer, fractional, and exponent -spellings: `1`, `1.0`, and `1e0` are equal, as are `0` and `-0`. Use exact numeric -normalization for identity comparison without rounding through a floating-point -conversion. This does not rewrite authored values or change existing typed -decoding and projection semantics. The parser's rejection of non-finite numeric -forms remains unchanged. - -### Application-Owned Constraints - -Retain an optional application interface for registering collections and imposing -their identity definitions. Configuration may define identity where no application -constraint exists. When a declaration omits `key`, an applicable registration -supplies identity. An explicit equivalent definition is accepted, and a different -definition fails under every mode, including replace. - -For example, if the application requires `id`, `key: [id, name]` is invalid: -that compound identity would allow repeated `id` values with different names. -A registration is an authoritative constraint, not an omission-filling default. -Copying a collection through inheritance does not create another application -registration; registrations constrain their registered destinations. - -Identity constraints validate actual sequences. They do not independently require -a path to exist, be non-null, or have sequence shape. The application schema and -validation own those requirements. Final-result enforcement for application -registrations is defined under [Validation And Diagnostics](#validation-and-diagnostics). - -### Application Registration API - -Use one public sequence-identity registration type, collected in one field on -`Definition`. Each registration contains an explicit destination selector and -its identity fields. Exact and dynamic registrations share matching, overlap -validation, identity comparison, and lifecycle behavior. - -Provide two constructors returning that same registration type: - -- An exact-path convenience constructor accepting a literal `Path` and identity - field names. It converts the path into a selector containing literal segments. -- An explicit-selector constructor accepting a structural selector and identity - field names, for destinations that include arbitrary mapping keys. - -Both constructors accept identity fields as variadic strings, so one-field and -compound identities use the same construction form. Apply the settled field-set -rules, including deduplication. Invalid registrations, including an empty identity -field set, fail during application-definition validation. - -Existing `Path` values retain their literal meaning throughout ConfigKit. The -exact-path constructor supplies convenience at the registration boundary; dynamic -matching uses the explicit selector type. Both forms can appear together in the -same definition field. - -Replace the current registration shape with this model. Preserving the existing -`SequenceComposition` fields and `Definition.SequenceCompositions` spelling is not -a requirement, and no compatibility facade is needed solely to retain that syntax. -Choose names centered on sequence identity, reflecting its role in composition, -inheritance, and final validation. Exact Go type, field, and constructor names -and signatures belong in the implementation plan. - -### Dynamic Application Registration Destinations - -Allow applications to register identity for collections whose containing mapping -keys become known only during loading. For example: - -```yaml -collections: - weatherreporter: - prompts: - - id: foo.bar - text: Report the weather. - homewatch: - prompts: - - id: foo.bar - text: Report the household status. -``` - -One application registration can require `id` for every namespace's `prompts` -list. The equal IDs above are valid because each list has its own uniqueness -scope. An optional root `prompts` list can have a separate exact registration. - -Use an explicit application-side structural selector composed of literal mapping -keys and an arbitrary-mapping-key segment. Conceptually, the namespace rule is: - -```text -Literal("collections") -> AnyMappingKey -> Literal("prompts") -``` - -This notation explains the selector; it is not a Go signature or YAML syntax. -Each segment consumes exactly one mapping level. Match the complete destination, -not an arbitrary prefix or suffix. Literal keys remain exact, including empty -strings and keys containing punctuation such as `*`, `/`, or `.`. An arbitrary-key -segment is distinct from any literal spelling and does not match a sequence -index. Existing `Path` values retain their literal meaning. - -This capability belongs to the reusable application definition and applies to -ordinary and layered loads. YAML collection destinations and import destinations -retain their existing literal syntax. Recursive search, regular expressions, -filesystem glob rules, sequence-index selectors, and custom matching callbacks -are outside this addition. - -At each concrete matched destination, apply the same authoritative identity -constraint as an exact registration. Use the settled field-set and scalar-equality -rules. Do not pool identities across destinations or require records to repeat -the containing mapping key as an identity component. Applications retain ownership -of namespace interpretation, qualified runtime IDs, and domain validation. - -Matching supplies effective identity before relevant strict composition and -precedence operations. Without an applicable explicit mode, this identity selects -upsert under the existing default rule. It also supplies identity for inheritance -within the matched sequence. YAML declarations must agree with it under every -mode, including replace. These are ordinary application constraints with a broader -destination selector, not a separate final-validation facility. - -### Registration Overlap - -Equivalent application constraints coalesce at any shared destination. Equivalence -uses the identity field-set rules: for example, `id` and `[id, id]` agree. -Incompatible constraints fail; exact registrations do not override broader -selectors, and registration order grants no precedence. - -Detect incompatible overlapping registrations while validating the application -definition, before source acquisition. For these fixed-depth mapping selectors, -overlap exists when the selectors have the same number of segments and every -corresponding pair can match a common key. Unequal literal keys at any position -make the selectors disjoint; an arbitrary-key segment can overlap a literal or -another arbitrary-key segment. Compare segment structure rather than rendered -path strings. No configuration pre-read is needed. - -Thus an `id` rule for all namespace prompt lists conflicts with a `[id, name]` -rule for one named namespace, even if that namespace is absent from a particular -load. This is an invalid application definition. A valid selector matching no -collections is harmless: registration does not require a path to exist, create -collections, or impose shape and nullability requirements. - -### Matching Across The Loading Lifecycle - -Match application selectors against effective configuration destinations after -accounting for input placement. Physical filenames and source-local paths do not -determine applicability. Make constraints available before composing contributions -at those destinations, including namespaces first introduced by imports, mounted -inputs, or higher tiers. Apply matching constraints to key-only YAML declarations -as well, even before their sequence exists. Consumers need neither a preliminary -load nor source inspection to enumerate namespaces. - -During inheritance, apply registrations at the inheriting destination, including -newly created matching destinations. Copying a collection does not transfer its -application registrations to a different path. The existing YAML-policy ownership -and transfer rules remain separate from application selector matching. - -During final validation, find matching actual sequences in the final effective -configuration. This includes namespaces introduced or replaced by typed defaults, -CLI overrides, and application resolution. Reusing only destinations found during -source assembly would miss these values. Use the same selector meaning and -identity validator in both phases; later assignments do not trigger composition -or inheritance again. YAML-only identities retain their existing enforcement -boundary. - -## Strict Composition And Precedence Operations - -### Compose Peers Before Applying Precedence - -Resolve applicable identity and mode policy, strictly compose the tier's peer -contributions, then apply one selected operation per collection against the -accumulated lower value. The tier operation does not sequentially replace peer -source contributions. Selected overlay contributions use the same combination -semantics. - -Within a strict tier, equivalent identity definitions coalesce and different -field sets fail. Agreeing explicit modes coalesce; conflicting explicit modes -fail. Omission expresses no preference. A root declaration or a later declaration -has no precedence over conflicting peers. Explicitly selecting a default mode -is still a real instruction and can conflict with another explicit mode. - -Identified peer sequences append distinct records. Duplicate identities among -peers or within a contributed list fail under all three modes, including replace -and upsert. Equal or disjoint record payloads do not excuse a duplicate identity. -Unidentified sequences retain their current strict-composition conflict behavior. - -For example, higher-tier fragments `[C]` and `[D]` with an agreed identity assemble -as `[C, D]`. Against lower `[A, B]`, append or upsert produces `[A, B, C, D]`, -while replace produces `[C, D]`. Peers `[B]` and `[B*]` conflict even under -upsert because neither has higher precedence. - -### Available Operations - -Use one spelling for each mode: `replace`, `append`, and `upsert`. - -| Mode | Behavior at source-precedence and inheritance boundaries | -| --- | --- | -| `replace` | Replace the complete lower sequence with the assembled higher sequence. | -| `append` | Retain lower records and append higher records; duplicate identities fail. | -| `upsert` | Replace matching records in full, retain unmatched lower records, and append new identities. | - -Append and upsert require effective identity. ConfigKit does not infer identity -from an `id` field or record position. Whole-sequence replacement does not require -identity agreement with lower YAML policy, but application constraints still apply. - -Letters denote record identities; `B*` has the same identity as `B` with a different -payload: - -| Lower | Higher | Mode | Result | -| --- | --- | --- | --- | -| `[A, B]` | `[C]` | `append` | `[A, B, C]` | -| `[A, B]` | `[C]` | `replace` | `[C]` | -| `[A, B]` | `[B*]` | `append` | Duplicate-identity error | -| `[A, B]` | `[B*]` | `replace` | `[B*]` | -| `[A, B]` | `[B*]` | `upsert` | `[A, B*]` | -| `[A, B]` | `[B*, C]` | `upsert` | `[A, B*, C]` | - -Upsert replaces the whole matched record. If lower `B` has `id`, `text`, and -`temperature`, while higher `B` has only `id` and `text`, the result at this -boundary has only `id` and `text`. Subsequent defaulting or other lifecycle -operations retain their own semantics. Mappings continue to combine recursively; -matching sequence records do not recursively merge fields. - -### Mode Selection And Persistence - -Select the operation in this order: - -1. Reconcile explicit modes within the current tier; conflicting peers fail. -2. Use the current tier's explicit mode if supplied. -3. Otherwise retain an applicable explicit mode from a lower tier. -4. With no explicit mode, infer upsert when effective identity is available. -5. Otherwise infer replace. - -Effective identity can come from an application registration, current declarations, -or applicable lower policy, subject to the identity agreement and lifetime rules -below. Determine the inferred mode from available identity before applying the -operation. A plain higher-tier sequence can therefore use upsert without repeating -a key declaration. - -An explicit mode persists at its applicable destination until a higher-priority -explicit declaration changes it. New sequence values alone do not reset it, and -an unrelated overlay leaves it intact. A mode-only declaration can change policy -without supplying records; it neither clears the collection nor retroactively -recombines earlier tiers. Later source contributions and inheritance use the -resulting policy. - -| Lower explicit mode | Higher declaration | Effective mode | -| --- | --- | --- | -| `append` | Mode omitted | `append` | -| `append` | `upsert` | `upsert` | -| `upsert` | `replace` | `replace` | -| None | Mode omitted, effective identity available | Inferred `upsert` | -| None | Mode omitted, no effective identity | Inferred `replace` | - -Only explicit modes persist. Inferred defaults are recomputed from effective -identity when needed. Explicit replace continues to apply in subsequent tiers -until another mode is declared; introducing identity alone does not override it. -Mode persistence does not preserve YAML identity discarded by replacement, nor -policies owned by discarded records. Source precedence and inheritance use the -same destination policy, without separate per-phase settings. CLI assignments -remain outside mode selection. - -### Presence, Empty Sequences, Null, And Kind Mismatches - -At a collection's precedence boundary: - -| Higher contribution | `replace` | `append` / `upsert` | -| --- | --- | --- | -| Absent | Leave the lower value unchanged. | Leave the lower value unchanged. | -| Explicit `[]` | Clear the sequence. | Contribute no records. | -| Explicit null or another non-sequence value | Ordinary replacement, subject to application schema. | Error: the operation requires a sequence. | - -A declaration alone does not clear a collection. An optional import selecting -nothing contributes nothing and does not erase existing records. An absent lower -collection can be initialized by an incoming sequence under any mode, subject to -identity requirements. Explicit lower null or another non-sequence value is not -absence: append and upsert fail against it, while replacement remains available. -An empty incoming sequence bypasses neither kind checks nor identity requirements. - -With effective identity and no explicit mode, `prompts: []` uses upsert and does -not clear the collection. Clearing requires replace, selected in the current tier -or retained from lower policy. Replacing an identified collection with null also -requires replace and a schema permitting null. Collection null is distinct from -a present null identity component inside a record. - -### Ordering - -Retain existing dependency-ready source scheduling within each tier and authored -item order within lists. Gathering declarations does not change assembly order; -identity values do not sort records. - -- Append keeps lower records in order, followed by higher records in assembled order. -- Replace keeps only higher records in their assembled order. -- Upsert keeps a matched record's lower position and appends new identities in - higher-tier order. - -For example, lower `[A, B, C]` and higher `[C*, A*, D]` produce `[A*, B, C*, D]` -under upsert. Whole-sequence replacement gives the higher tier control of the -complete sequence order. - -## Policy Agreement, Lifetime, And Ownership - -### Identity Agreement And Replacement - -Append and upsert use one agreed identity for all participating records: - -| Lower identity | Higher declaration | Result when combining records | -| --- | --- | --- | -| `id` | Omitted | Use `id`. | -| `id` | `id`, `[id]`, or `[id, id]` | Accept the equivalent definition. | -| `id` | `[namespace, id]` | Error: different definitions. | -| None | `id` | Establish `id` and validate all participating records. | - -Introducing identity for an unkeyed lower sequence requires checking both lower -and incoming records, including uniqueness within each contribution. With no -identity from the application or either participating collection, append and -upsert fail. - -YAML identity follows the collection that survives an operation: - -| Operation | Effective YAML identity | -| --- | --- | -| Append or upsert | Retain or establish the agreed identity. | -| Whole-sequence replacement | Discard lower YAML identity and use the replacing tier's identity declaration, if any. | -| Replacement by null or another non-sequence value | Discard lower YAML identity and retain any identity explicitly declared by the replacing tier. | - -Application constraints remain authoritative under every operation. Without an -application constraint, replacement can introduce a different identity, such as -changing from `id` to `name`. Replacing a sequence without a new key declaration -or application registration produces an unkeyed sequence. No identity-removal -directive is needed. - -The reset occurs only when the collection value is actually replaced. Declaring -replace without supplying a replacement does not authorize changing the identity -of retained records. Explicit mode lifetime is separate from YAML identity -lifetime. - -### Ancestor Replacement - -Replacing an enclosing mapping can remove a collection without replacing that -collection directly. Apply the same identity and ownership rules to this removal: -discard the removed collection's lower YAML identity, and discard policies owned -by any removed records. Reintroducing the same path later does not restore those -discarded policies. - -Independently declared explicit modes retain their existing persistence rules. -Removing a descendant collection does not remove a mode whose declaring scope -survives. Application registrations remain applicable at matching destinations, -including destinations reintroduced later. Identity newly declared by the -replacing tier follows the existing declaration and ownership rules. - -For example, a root-level declaration establishes `key: id` and `mode: append` -for `[catalog, prompts]`. A higher tier replaces `catalog` with null, removing -the prompts collection and its YAML identity. The independently declared append -mode survives. If a later tier recreates `catalog.prompts`, append requires -identity supplied by a new declaration or an application registration. Without -that identity, loading fails; the later tier can instead explicitly choose -replace. The replacement of `catalog` itself follows ordinary ancestor-value -precedence and does not attempt to append null to the removed prompts list. - -### Declarations Without Sequence Values - -A key declaration without accompanying records applies immediately to an existing -retained sequence: check agreement and validate its records, without recombining -earlier tiers. For example, a higher tier that only declares -`destination: [prompts], key: id` identifies and validates the retained lower -`prompts` sequence. - -If no collection exists, retain the declaration in its owning scope for later -contributions. An otherwise empty underlay can establish identity for main-tier -fragments. An explicit identity declaration can also remain pending at a -non-sequence destination; it does not independently impose a shape requirement. -The presence and kind rules still determine whether an attempted operation fails. -Item-local declarations retain their ownership boundaries. - -### Lower-Tier Identity During Strict Composition - -Resolve applicable identity before composing a higher tier's peers, using -application registrations, available lower-tier policy, and declarations gathered -within the higher tier's scope. Authors need not repeat a baseline key solely -because the higher tier has multiple fragments. - -For example, an underlay establishes `key: id` for `prompts`, and two main-tier -fragments contribute `[foo]` and `[bar]` without a key declaration. They strictly -compose as `[foo, bar]` before the main tier combines with the accumulated lower -collection. If both contribute `foo`, composition fails; upsert does not give -precedence to peers. - -Lower YAML identity remains subject to agreement, replacement, and ownership. -A replacing collection must not be constrained by obsolete lower identity just -because policy resolution precedes composition. Likewise, a replacement record -cannot borrow nested identity owned by the discarded lower record, even when -outer identity or numeric position matches. Application constraints still apply. -Higher-tier declarations do not retroactively repair invalid lower-tier composition. - -### Nested Record Ownership - -Attach item-local policies to their owning contributions so they follow surviving -records independently of numeric indexes. Whole-record upsert discards the old -record's local policies along with its fields. Whole-sequence replacement discards -policies owned by removed records. A replacement record supplies its own local -declarations; equal outer identity or final index does not transfer discarded -policies. - -For example, a source imported as one record into `prompts` can declare: - -```yaml -configkit: - sequences: - - destination: [steps] - key: id - mode: append -id: foo -steps: - - id: prepare -``` - -The declaration governs `steps` inside that `foo` record. Retaining `foo` retains -its local policy, including the explicit mode. Upserting a new `foo` discards the -old `steps` and its declaration even though the outer record keeps its position. - -Policies independently applicable at a destination are not discarded merely -because an individual record is replaced. Ownership determines which declarations -are removed. Application registrations remain constraints at their registered -destinations. Inheritance copies transfer identity and choose operations as -specified next. - -## Sibling Inheritance - -Complete source-tier assembly before resolving inheritance. Retain existing -mapping-sibling scopes and add identified-sequence scopes as described below. -Both use the existing dependency-ready algorithm. At each inheritance combination, -the resolved parent's collection is the lower value and the inheriting mapping's -local collection is the higher value. Mappings combine recursively; upsert within -an identified sequence remains whole-record replacement. Explicit record -inheritance is a separate construction step after those replacements. - -Use the inheriting destination collection's policy: an applicable explicit mode, -including one retained through source tiers, wins. Otherwise infer upsert with -effective identity and replace without it. There is no separate default-append -rule for inheritance. The same destination policy governs source-precedence and -inheritance combinations. - -YAML identity follows a collection copied through inheritance, subject to agreement -and replacement rules. A source collection's mode does not automatically become an -instruction at a different inheriting destination. For example, source `steps` -has `key: id` and `mode: append`; an inheriting mapping supplies local `steps` -without an applicable explicit mode. The copied identity selects upsert there. -An explicit destination append instead selects append. A matching destination key -agrees; a different key conflicts when combining records, while explicit replace -can establish new YAML identity subject to application constraints. - -For example: - -```yaml -configkit: - sequences: - - destination: [parent, items] - key: id -child: - items: - - id: foo - value: A - - id: bar - value: B -parent: - extends: child - items: - - id: baz - value: C -``` - -Despite the names, `parent` inherits from `child`. Identity at `[parent, items]` -selects upsert, producing `[foo, bar, baz]`. If the local sequence instead supplies -a revised `foo`, the result is `[foo*, bar]`. Append rejects that repeated identity; -replace retains only the local sequence. - -The same ordering, presence, and kind rules apply at inheritance boundaries. -An omitted local sequence retains the inherited sequence. Explicit `[]` clears it -under replace and contributes no records under append or upsert. Retain controls -until inheritance uses them, then exclude them from application decoding. - -### Inheritance Between Identified Sequence Records - -An identified sequence supplies a sibling scope for its records. Identity can -come from an application registration or an applicable YAML declaration. A record -may inherit from one other record in that same sequence using an `extends` -mapping that selects the parent's complete identity: - -```yaml -configkit: - sequences: - - destination: [prompts] - key: id -prompts: - - id: base - model: standard - temperature: 0.5 - - id: concise - extends: {id: base} - temperature: 0.2 -``` - -The final `concise` record is: - -```yaml -id: concise -model: standard -temperature: 0.2 -``` - -The `base` record remains in the collection. Parents are ordinary records subject -to the same decoding and validation as other records; there is no abstract -template category. - -Use mapping-form references for both single and compound keys. With -`key: [namespace, id]`, a reference can be -`extends: {namespace: shared, id: base}`. Match components using the collection's -field-set and scalar-equality rules, including mathematical numeric equality. -`extends: {id: null}` selects a null-valued identity; `extends: null` cancels a -relationship. - -Parent-reference mappings use ordinary strict mapping composition. Item-local -helper imports may supply disjoint fields of a compound reference; overlapping -fields conflict, even when equal. For example, a helper's -`extends: {namespace: shared}` and the importing record's `extends: {id: base}` -compose into `extends: {namespace: shared, id: base}`. This assembles one record; -it does not permit strict peers to contribute duplicate record identities. - -Before parent lookup, require the surviving effective reference mapping to contain -exactly the collection's identity fields, with supported scalar values. Missing -or extra fields fail. Validate the completed reference during inheritance, rather -than requiring each helper contribution to be independently complete. Superseded -references remain historical evidence; earlier strict-composition conflicts -remain errors. - -Every record must supply its complete own identity before inheritance. Preserve -the child's identity fields and inherit only the parent's nonidentity payload; -inheritance cannot supply a missing child identity. Child payload fields take -precedence, mappings combine recursively, and nested sequences use the inheriting -destination's collection policy. Consume the record's directive before decoding. - -Support one parent within the same identified sequence, including forward -references and chains. Missing parents and cycles, including self-reference, -fail the load. Cross-collection references, position-based references, multiple -parents, and inheritance between unidentified sequence members are outside this -scope. Existing string-form references between mapping siblings retain their -meaning; they do not become shorthand for sequence identity references. - -Resolve relationships against records surviving source-tier assembly and selected -overlays. A higher layer can replace a parent before inheritance runs; removing -that parent makes a surviving reference fail. Whole-record upsert discards the -old record's directive along with its payload. Omitting `extends` in the -replacement record does not retain the displaced record's relationship. An -explicit reference to the record's own identity is a cycle, not a way to access -its lower-tier predecessor. - -Reuse Kahn's algorithm with the existing stable canonical semantic-pointer key -for dependency-ready ordering. Build parent lookup from the sequence's effective -identity, and preserve the assembled list order independently of resolution -order. Enclosing relationships resolve before nested relationships. Parent -templates retain nested instructions for evaluation at each inheriting -destination, following the existing destination-local inheritance rule. Copied -collection identity, destination operation selection, and record-owned policy -transfer follow the rules above. - -Record inheritance runs in the existing structural phase, before typed defaults, -decoding, CLI overrides, and application resolution. Those later stages do not -trigger another inheritance pass. Preserve inherited presence, origin and -history evidence, sensitivity, declaring resource contexts, and existing -expansion budgets just as for mapping-sibling inheritance. Identity lookup must -not expose identity values in diagnostics. Existing index-based inspection paths -remain sufficient; no new identity-selector path API is needed. - -## Loading Lifecycle - -Use the following order: - -1. Assemble source tiers and selected overlays, resolving policy before each - tier's strict composition and applying collection operations at precedence - boundaries. -2. Resolve sibling inheritance in mappings and identified sequences, then apply - typed defaults and decode the assembled configuration. -3. Apply explicit CLI overrides to that configuration and check supplied values - against the application schema. -4. Run application resolution against the overridden configuration. -5. Resolve effective secret references, validate the final configuration including - application-registered identities, and publish it. - -### Typed Defaults And Application Resolution - -Source-backed underlays, including application-supplied mapping sources, -participate in sequence combination. Typed `Definition.Defaults` remains the -initial decoded state: supplied sequences replace default sequences. Source -append does not append to a typed default list. - -Application resolution retains assignment and recorded-transformation semantics. -ConfigKit does not automatically combine a callback's assigned sequence with its -previous value. Application-registered identities receive final validation -independently of how defaults, CLI overrides, or callbacks produce their values. - -### CLI Precedence - -CLI overrides are the highest-precedence input to application resolution. -Supplied sequences replace whole sequences, including sequences inside supplied -mappings. Mappings retain recursive override behavior; empty sequences and -permitted nulls retain explicit replacement meaning. Configuration append or -upsert does not reinterpret CLI assignments. Inheritance and defaulting do not -run again afterward: overriding assembled `[A, B]` with `[C]` presents `[C]` to -application resolution without inheritance reintroducing prior records. - -Application resolution sees CLI values and their explicit-presence and source -metadata. `Resolution.Default` preserves explicit CLI values; normalization and -recorded application transformations may still change them. For example, a derived -URL can use a CLI hostname, while an explicit CLI URL can be preserved by an -absence-based default. Retain recording and provenance requirements for those -transformations. - -Reject structural directives such as `extends` and `configkit` in CLI override -paths or supplied values. CLI input supplies application values after structural -processing; it cannot supply or cancel inheritance directives, change completed -relationships, or prevent earlier inheritance failures. - -Typed decoding also precedes CLI application, so a CLI replacement cannot repair -an earlier decoding failure. A source string in an integer field fails before a -valid CLI integer can apply. This changes current override-before-decoding -behavior. Semantic validation remains after overrides and application resolution. - -Secret lookup uses effective references after overrides and resolution. Neither -stage bypasses final application identity constraints or semantic validation. - -## Validation And Diagnostics - -### Structural Identity Checks - -Allow item-local helper imports to complete and consume applicable non-main -source-version headers before checking payload identity for collection assembly. -A helper can supply an identity component; a consumed header cannot supply one -after removal. - -Structural controls are likewise unavailable as source identity components. -Check identity against application payload fields, excluding fields acting as -structural directives even when the engine retains them for later processing. -For example, `key: extends` with an item containing `extends: null` does not -supply a null identity: that field cancels inheritance and is consumed. Report -the ordinary missing-identity-component error at the collection/item path using -available origins, without exposing values. Do not defer this failure until -directive consumption or final application validation. - -This rule follows a field's structural role; it does not introduce a broader -restriction on unrelated literal field names. Final application identity checks -continue to examine the effective typed payload under their existing rules. - -Validate each identified contribution before combination and check uniqueness in -the resulting identified collection, including inheritance operations. Upsert -matches across precedence are replacements; duplicates within a contribution -remain errors. Later inheritance, typed defaults, or callbacks cannot repair an -identity missing at an earlier assembly boundary. - -YAML-only identities are enforced during structural composition, precedence -assembly, and identified inheritance operations. They have no final-result -guarantee after typed defaulting, CLI overrides, or application resolution. Those -stages can produce duplicate identities or otherwise violate a YAML-only identity. -Applications needing a final guarantee register the constraint through the -application-owned interface. - -Adding record inheritance does not expand this guarantee: a YAML declaration -can enable parent lookup without imposing final validation on values later -produced by defaults, CLI overrides, or application resolution. - -### Final Application Constraints - -On successful loading, final effective collections covered by application -registrations satisfy their declared identity constraints. Retain assembly-time -checks and validate final effective sequences before publication, after defaults, -inheritance, overrides, application resolution, and secret lookup. - -This covers both exact and dynamic application registrations. Evaluate dynamic -selectors against the final effective configuration, with independent validation -for each concrete matched sequence as described under -[Matching Across The Loading Lifecycle](#matching-across-the-loading-lifecycle). - -ConfigKit performs these identity checks; consumers register the constraint -without needing to repeat duplicate detection in `Definition.Validate`. -Application-owned semantic validation continues to cover domain-specific rules. - -This guarantee covers actual sequences, not path existence or allowed shape. -It does not require every intermediate callback state to satisfy the constraint -and does not cover caller mutations after loading returns. Final checks examine -effective typed values; assembly checks validate authored contributions. A final -scalar alone does not prove that a source explicitly supplied that field. - -Use the existing canonical final-value representation and shared publication -path as the foundation. The guarantee requires neither a new mutation-history -system nor exact causal reconstruction for records supplied by defaults or -callbacks. - -### Diagnostics And Inspection - -Identity diagnostics identify collection and item paths using available origin -information without exposing identity values. Conflicting declarations identify -the collection destination and declaration locations. - -For dynamic registrations, use concrete matched collection and item paths for -value diagnostics, with available origins and existing sensitivity rules. -Definition-time overlap errors identify the conflicting registrations; they do -not require or invent a concrete destination from a loaded document. - -Correctness includes final values, projected item paths, source destinations, -provenance, history, sensitivity, and passive resource contexts. Appending can -change final indexes; replacing changes which contributions survive. Preserve -the existing loading and inspection contracts through both operations, including -policy ownership and inheritance transfer. - -## Work Needed From The Baseline - -The baseline needs the following connected capabilities to reach the target -state. Execution order and test sequencing belong to the -[implementation plan](implementation.md): - -- **Declarations and public API:** extend application registration to the settled - identity model with one public registration type and definition field, an - exact-path convenience constructor, and an explicit-selector constructor. - Preserve literal `Path` semantics. Parse and consume `configkit.sequences` across - supported source forms and selected overlays; preserve placement-relative and - item-local scopes. Keep sequence declarations independent of filesystem import - enablement. -- **Identity and policy representation:** implement field-set comparison, typed - scalar identity tuples with exact numeric equality, declaration agreement, - pending declarations, explicit-mode persistence, replacement lifetime, and - record-owned policy transfer. Choose bookkeeping proportional to these semantics. - Handle descendant collection removal through ancestor replacement, preserving - independently declared modes and application constraints while discarding - removed collection identity and record-owned policies. -- **Registration matching:** support literal and arbitrary-mapping-key segments, - detect incompatible selector overlap during definition validation, and resolve - applicable constraints at concrete placed destinations. Reuse the identity - rules and validator for exact and dynamic registrations. Keep matching separate - from source discovery and from YAML declaration ownership. -- **Assembly handoffs:** make accumulated lower policy available before higher-tier - strict composition. Separate source acquisition from policy-aware composition - or otherwise revise assembly sequencing; current main-before-underlay composition - is not a design constraint. Preserve distinctions among acquisition order, - dependency-ready scheduling within tiers, and precedence order. Account for - source records, callbacks, main-version checks, and failure diagnostics. Retain - version ownership and validation semantics; higher replacement cannot hide - lower-tier structural failures. -- **Combination and inheritance:** implement the three operations with strict peer - handling, whole-record replacement, presence rules, stable ordering, and the - destination policy rules for inheritance. Extend sibling-scope discovery and - parent lookup to identified sequences, validate mapping-form references, and - preserve child identities while copying parent payloads. Reuse the shared - dependency-ready scheduler and destination-local nested resolution. Preserve - provenance, history, sensitivity, paths, and resource contexts as records survive, - move, or disappear. -- **Lifecycle and validation:** move CLI application after inheritance and typed - defaults/decoding, preserve explicit-presence and source metadata for application - resolution, reject CLI structural directives, and add final application identity - validation through the shared publication path. Keep YAML-only enforcement - within structural operations. Match dynamic registrations during assembly and - inheritance, and against the final effective configuration rather than a saved - list of source-backed destinations. - Exclude structural controls from source identity checks even while retaining - those controls for later inheritance processing. -- **Documentation and verification:** exercise ordinary and layered loading, - imported and inline collections, item-local scopes, version-header consumption, - selected overlays, inheritance, typed defaults, CLI, callbacks, and inspection. - Cover single and compound parent references, scalar equality, forward references - and chains, missing parents and cycles, parent replacement/removal, child identity - preservation, nested destination policy, cancellation, and stable output order. - Verify helper-composed references, overlapping reference fields, missing or extra - fields, and replacement of references together with their containing records. - Verify independent namespace uniqueness, root and namespaced registrations, - literal punctuation keys, equivalent and incompatible overlaps, and conflicts - detected even without a matching input destination. Cover namespaces introduced - by imports, placement, higher tiers, inheritance, defaults, CLI, and callbacks; - preserve ordinary strict-peer and upsert behavior. Check YAML disagreement under - replace, harmless absent or empty collections, schema-owned shape requirements, - unrelated sequences remaining unselected, and concrete-path diagnostics. - Verify both constructors feed the same registration model, including mixed - exact and dynamic registrations, compound identities, and definition-time - rejection of empty identity field sets. - Cover ancestor replacement followed by collection reintroduction, including a - surviving append mode with and without renewed identity, explicit replacement, - and application-registered identity. Verify that `extends: null` cannot supply - an identity component and that structural-control exclusion uses ordinary - value-free missing-identity diagnostics. - Update the public contract and consumer examples when behavior lands. - -The collection-policy and CLI-ordering decisions are accepted in -[ADR 0017](../adr/0017-collection-sequence-policy.md) for collection policy and -[ADR 0018](../adr/0018-cli-after-structural-resolution.md) for CLI ordering. -Their scoped supersession is recorded in the affected earlier ADRs, and the -[architecture policy](../policy/architecture.md) reflects the accepted decisions. -These govern the target architecture; the configuration contract continues to -describe implemented behavior until the feature lands. This roadmap does not -select migration machinery. - -ADR 0017 now includes inheritance within identified sequences; -[ADR 0009](../adr/0009-sibling-inheritance.md) records that scope extension in its -status while preserving its decision content. The architecture policy is aligned. -Dynamic selectors extend the destinations covered by application registration; -they retain the accepted application authority and final-validation boundary. - -No open feature decisions remain. The implementation plan translates this end -state into reviewable stages without adding guarantees, restrictions, or -combination semantics.