# Sequence Identity And Combination Roadmap Status: settled feature roadmap, ready for a staged implementation plan. This is the targeted end state, not an implemented contract. The subsequent plan will specify execution order, the exact Go API, internal bookkeeping, and validation stages. ## Purpose And Scope Allow configuration authors to assemble collections from independently authored records, declare identity in configuration, and progressively extend a baseline collection through higher layers. Authors can replace the entire collection, append distinct records, or replace complete records by identity while retaining unmatched records. Sibling inheritance uses the same sequence operations and identity-based defaults. The motivating use case is a prompt catalog whose source-backed baseline supplies records and whose application or installation layers add or supersede records. Applications can optionally impose identity constraints that configuration must respect and that hold in the final loaded configuration. The feature covers ordinary and layered loading, imported items, inline sequences, selected overlays, and sibling inheritance. Typed defaults, CLI assignments, and application resolution retain their own combination semantics; CLI application moves later in the loading lifecycle. This scope is narrower than the deferred [identified-record sequences proposal](future.md#1-identified-record-sequences). Recursive merging of matching records and record-specific inheritance remain separate candidates. The initial declaration surface is `configkit.sequences`; `key` and `mode` shorthand on imports is deferred. Application lists remain ordinary YAML sequences without control wrappers. Unkeyed sequence concatenation is outside this feature. ## Current Baseline And Required Changes The [import contract](../config.md#explicit-imports) owns existing selectors, destinations, placement, and ordering. `placement: sequence-item` already places each selected source mapping as one item at its effective destination. Item-local helper imports can contribute fields before placement. The [composition contract](../config.md#strict-document-composition), [layered-loading contract](../config.md#layered-loading), and [inheritance contract](../config.md#sibling-inheritance) describe the implemented baseline: | Area | Current behavior | Target behavior | | --- | --- | --- | | Identity declaration | `Definition.SequenceCompositions` registers one literal string field at each registered mapping path. | YAML collection declarations and optional authoritative application registrations support field sets and scalar identity components. | | Strict composition | Registered sequences append distinct string identities and reject duplicates; repeated unregistered sequences conflict. | Collections with effective identity append distinct records and reject duplicates. Applicable lower-tier identity is available before composing higher-tier peers. Unidentified sequences retain strict-conflict behavior. | | Source precedence and selected overlays | A present higher sequence replaces the lower sequence wholesale. | Collection policy selects `replace`, `append`, or `upsert`; overlays can declare policy for inline sequences. | | Sibling inheritance | An inheriting mapping recursively overlays its resolved parent; supplied sequences replace wholesale. | Mappings retain recursive combination, while sequences use destination policy and the same three operations. | | CLI overrides | Apply before inheritance and typed defaults/decoding. | Apply after inheritance and typed defaults/decoding, before application resolution. | | Identity enforcement | Registration governs strict sequence composition. | Structural identity checks cover identified combinations; application registrations additionally constrain final effective sequences. | Source-backed underlays participate in source assembly. `Definition.Defaults` provides typed defaults later; extending source combination does not turn typed default lists into source underlays. ## Collection Declarations Use a source-root `configkit.sequences` block containing collection declarations. Each entry identifies a `destination` and may declare `key`, `mode`, or both. Identity and mode can be supplied independently. This form serves imported items, inline records, ordinary mapping imports, and inline sequences in selected overlays. ```yaml configkit: sequences: - destination: [prompts] key: [namespace, id] mode: append imports: - dir: prompts pattern: '*.yaml' destination: [prompts] placement: sequence-item ``` Each selected file contributes a record such as: ```yaml namespace: support id: welcome text: Welcome ``` Here append retains a lower-layer collection and adds distinct higher-layer records. Identity and combination controls belong to the collection declaration, so imported and inline contributions share one policy surface. ### Destinations And Scope `destination` uses the existing sequence of literal mapping keys. Resolve it relative to the declaring source's placement: a fragment mounted under `catalog` can declare `[prompts]` locally. A sequence-item source can declare a nested collection within its own item without referring to an eventual numeric index. Preserve source and item-local scope boundaries. A collection's scope is its resolved destination within its composition scope. Resolve relative destinations before reconciling declarations. Different imported items retain separate scopes for their nested collections. A declaration governs all contributions to its collection in the tier, including imported and inline records; it does not govern only records in the declaring file. Gather declarations before composing the relevant scope so their effect does not depend on which contribution is assembled first. Retain applicable controls through source precedence and inheritance, then keep them out of application decoding. Sequence declarations are independent of filesystem import enablement. Selected overlays may contain `configkit.sequences`, extending the current blanket prohibition on overlay `configkit` declarations. This does not enable imports in overlays. ### Overlay Authoring For an existing sequence `[foo, bar]`, an overlay can replace `foo` in full: ```yaml configkit: sequences: - destination: [prompts] key: id mode: upsert prompts: - id: foo text: Revised definition ``` The result is `[foo*, bar]`. The explicit upsert mode can be omitted when no different explicit mode applies from a lower tier: effective identity then selects upsert. To add a distinct record instead: ```yaml configkit: sequences: - destination: [prompts] key: id mode: append prompts: - id: baz text: Another prompt ``` The result is `[foo, bar, baz]`. To replace the whole sequence: ```yaml configkit: sequences: - destination: [prompts] mode: replace prompts: - id: bar text: Bar definition - id: baz text: Baz definition ``` The result is `[bar, baz]`. Explicitly selecting the default operation is valid. With no applicable explicit mode, omitting this declaration selects replace for an unidentified collection and upsert for an identified one. A retained explicit mode continues to govern. ## Identity Definitions And Constraints ### Field Sets `key` accepts a string or a nonempty list of strings naming direct, literal fields in each record. A name such as `metadata.id` denotes that literal field, not a nested lookup. Treat field names as a set: remove repeated names and ignore authored order when comparing definitions. Thus `id`, `[id]`, and `[id, id]` are equivalent; `[namespace, id]`, `[id, namespace]`, and `[namespace, id, id]` are also equivalent. An empty list is invalid because it supplies no identity fields. These rules apply to both configuration declarations and application registrations. Use a consistent field order internally to construct identity tuples. Preserve component types and tuple boundaries so unstructured concatenation cannot cause otherwise distinct identities to collide. ### Scalar Components And Equality Each identified record must be a mapping with every designated identity field present. Components may be strings, numbers, booleans, or explicit null. Missing fields and mapping or sequence components are invalid. Present null differs from absence; two single-field null identities are equal. Compound identities are equal when all corresponding components are equal. Strings compare exactly, including empty strings, without trimming, case folding, or normalization. Scalar categories remain distinct: `"1"`, `1`, `true`, and null are different identities. Numbers compare by mathematical value across integer, fractional, and exponent spellings: `1`, `1.0`, and `1e0` are equal, as are `0` and `-0`. Use exact numeric normalization for identity comparison without rounding through a floating-point conversion. This does not rewrite authored values or change existing typed decoding and projection semantics. The parser's rejection of non-finite numeric forms remains unchanged. ### Application-Owned Constraints Retain an optional application interface for registering collections and imposing their identity definitions. Configuration may define identity where no application constraint exists. When a declaration omits `key`, an applicable registration supplies identity. An explicit equivalent definition is accepted, and a different definition fails under every mode, including replace. For example, if the application requires `id`, `key: [id, name]` is invalid: that compound identity would allow repeated `id` values with different names. A registration is an authoritative constraint, not an omission-filling default. Copying a collection through inheritance does not create another application registration; registrations constrain their registered destinations. Identity constraints validate actual sequences. They do not independently require a path to exist, be non-null, or have sequence shape. The application schema and validation own those requirements. Final-result enforcement for application registrations is defined under [Validation And Diagnostics](#validation-and-diagnostics). ## Strict Composition And Precedence Operations ### Compose Peers Before Applying Precedence Resolve applicable identity and mode policy, strictly compose the tier's peer contributions, then apply one selected operation per collection against the accumulated lower value. The tier operation does not sequentially replace peer source contributions. Selected overlay contributions use the same combination semantics. Within a strict tier, equivalent identity definitions coalesce and different field sets fail. Agreeing explicit modes coalesce; conflicting explicit modes fail. Omission expresses no preference. A root declaration or a later declaration has no precedence over conflicting peers. Explicitly selecting a default mode is still a real instruction and can conflict with another explicit mode. Identified peer sequences append distinct records. Duplicate identities among peers or within a contributed list fail under all three modes, including replace and upsert. Equal or disjoint record payloads do not excuse a duplicate identity. Unidentified sequences retain their current strict-composition conflict behavior. For example, higher-tier fragments `[C]` and `[D]` with an agreed identity assemble as `[C, D]`. Against lower `[A, B]`, append or upsert produces `[A, B, C, D]`, while replace produces `[C, D]`. Peers `[B]` and `[B*]` conflict even under upsert because neither has higher precedence. ### Available Operations Use one spelling for each mode: `replace`, `append`, and `upsert`. | Mode | Behavior at source-precedence and inheritance boundaries | | --- | --- | | `replace` | Replace the complete lower sequence with the assembled higher sequence. | | `append` | Retain lower records and append higher records; duplicate identities fail. | | `upsert` | Replace matching records in full, retain unmatched lower records, and append new identities. | Append and upsert require effective identity. ConfigKit does not infer identity from an `id` field or record position. Whole-sequence replacement does not require identity agreement with lower YAML policy, but application constraints still apply. Letters denote record identities; `B*` has the same identity as `B` with a different payload: | Lower | Higher | Mode | Result | | --- | --- | --- | --- | | `[A, B]` | `[C]` | `append` | `[A, B, C]` | | `[A, B]` | `[C]` | `replace` | `[C]` | | `[A, B]` | `[B*]` | `append` | Duplicate-identity error | | `[A, B]` | `[B*]` | `replace` | `[B*]` | | `[A, B]` | `[B*]` | `upsert` | `[A, B*]` | | `[A, B]` | `[B*, C]` | `upsert` | `[A, B*, C]` | Upsert replaces the whole matched record. If lower `B` has `id`, `text`, and `temperature`, while higher `B` has only `id` and `text`, the result at this boundary has only `id` and `text`. Subsequent defaulting or other lifecycle operations retain their own semantics. Mappings continue to combine recursively; matching sequence records do not recursively merge fields. ### Mode Selection And Persistence Select the operation in this order: 1. Reconcile explicit modes within the current tier; conflicting peers fail. 2. Use the current tier's explicit mode if supplied. 3. Otherwise retain an applicable explicit mode from a lower tier. 4. With no explicit mode, infer upsert when effective identity is available. 5. Otherwise infer replace. Effective identity can come from an application registration, current declarations, or applicable lower policy, subject to the identity agreement and lifetime rules below. Determine the inferred mode from available identity before applying the operation. A plain higher-tier sequence can therefore use upsert without repeating a key declaration. An explicit mode persists at its applicable destination until a higher-priority explicit declaration changes it. New sequence values alone do not reset it, and an unrelated overlay leaves it intact. A mode-only declaration can change policy without supplying records; it neither clears the collection nor retroactively recombines earlier tiers. Later source contributions and inheritance use the resulting policy. | Lower explicit mode | Higher declaration | Effective mode | | --- | --- | --- | | `append` | Mode omitted | `append` | | `append` | `upsert` | `upsert` | | `upsert` | `replace` | `replace` | | None | Mode omitted, effective identity available | Inferred `upsert` | | None | Mode omitted, no effective identity | Inferred `replace` | Only explicit modes persist. Inferred defaults are recomputed from effective identity when needed. Explicit replace continues to apply in subsequent tiers until another mode is declared; introducing identity alone does not override it. Mode persistence does not preserve YAML identity discarded by replacement, nor policies owned by discarded records. Source precedence and inheritance use the same destination policy, without separate per-phase settings. CLI assignments remain outside mode selection. ### Presence, Empty Sequences, Null, And Kind Mismatches At a collection's precedence boundary: | Higher contribution | `replace` | `append` / `upsert` | | --- | --- | --- | | Absent | Leave the lower value unchanged. | Leave the lower value unchanged. | | Explicit `[]` | Clear the sequence. | Contribute no records. | | Explicit null or another non-sequence value | Ordinary replacement, subject to application schema. | Error: the operation requires a sequence. | A declaration alone does not clear a collection. An optional import selecting nothing contributes nothing and does not erase existing records. An absent lower collection can be initialized by an incoming sequence under any mode, subject to identity requirements. Explicit lower null or another non-sequence value is not absence: append and upsert fail against it, while replacement remains available. An empty incoming sequence bypasses neither kind checks nor identity requirements. With effective identity and no explicit mode, `prompts: []` uses upsert and does not clear the collection. Clearing requires replace, selected in the current tier or retained from lower policy. Replacing an identified collection with null also requires replace and a schema permitting null. Collection null is distinct from a present null identity component inside a record. ### Ordering Retain existing dependency-ready source scheduling within each tier and authored item order within lists. Gathering declarations does not change assembly order; identity values do not sort records. - Append keeps lower records in order, followed by higher records in assembled order. - Replace keeps only higher records in their assembled order. - Upsert keeps a matched record's lower position and appends new identities in higher-tier order. For example, lower `[A, B, C]` and higher `[C*, A*, D]` produce `[A*, B, C*, D]` under upsert. Whole-sequence replacement gives the higher tier control of the complete sequence order. ## Policy Agreement, Lifetime, And Ownership ### Identity Agreement And Replacement Append and upsert use one agreed identity for all participating records: | Lower identity | Higher declaration | Result when combining records | | --- | --- | --- | | `id` | Omitted | Use `id`. | | `id` | `id`, `[id]`, or `[id, id]` | Accept the equivalent definition. | | `id` | `[namespace, id]` | Error: different definitions. | | None | `id` | Establish `id` and validate all participating records. | Introducing identity for an unkeyed lower sequence requires checking both lower and incoming records, including uniqueness within each contribution. With no identity from the application or either participating collection, append and upsert fail. YAML identity follows the collection that survives an operation: | Operation | Effective YAML identity | | --- | --- | | Append or upsert | Retain or establish the agreed identity. | | Whole-sequence replacement | Discard lower YAML identity and use the replacing tier's identity declaration, if any. | | Replacement by null or another non-sequence value | Discard lower YAML identity and retain any identity explicitly declared by the replacing tier. | Application constraints remain authoritative under every operation. Without an application constraint, replacement can introduce a different identity, such as changing from `id` to `name`. Replacing a sequence without a new key declaration or application registration produces an unkeyed sequence. No identity-removal directive is needed. The reset occurs only when the collection value is actually replaced. Declaring replace without supplying a replacement does not authorize changing the identity of retained records. Explicit mode lifetime is separate from YAML identity lifetime. ### Declarations Without Sequence Values A key declaration without accompanying records applies immediately to an existing retained sequence: check agreement and validate its records, without recombining earlier tiers. For example, a higher tier that only declares `destination: [prompts], key: id` identifies and validates the retained lower `prompts` sequence. If no collection exists, retain the declaration in its owning scope for later contributions. An otherwise empty underlay can establish identity for main-tier fragments. An explicit identity declaration can also remain pending at a non-sequence destination; it does not independently impose a shape requirement. The presence and kind rules still determine whether an attempted operation fails. Item-local declarations retain their ownership boundaries. ### Lower-Tier Identity During Strict Composition Resolve applicable identity before composing a higher tier's peers, using application registrations, available lower-tier policy, and declarations gathered within the higher tier's scope. Authors need not repeat a baseline key solely because the higher tier has multiple fragments. For example, an underlay establishes `key: id` for `prompts`, and two main-tier fragments contribute `[foo]` and `[bar]` without a key declaration. They strictly compose as `[foo, bar]` before the main tier combines with the accumulated lower collection. If both contribute `foo`, composition fails; upsert does not give precedence to peers. Lower YAML identity remains subject to agreement, replacement, and ownership. A replacing collection must not be constrained by obsolete lower identity just because policy resolution precedes composition. Likewise, a replacement record cannot borrow nested identity owned by the discarded lower record, even when outer identity or numeric position matches. Application constraints still apply. Higher-tier declarations do not retroactively repair invalid lower-tier composition. ### Nested Record Ownership Attach item-local policies to their owning contributions so they follow surviving records independently of numeric indexes. Whole-record upsert discards the old record's local policies along with its fields. Whole-sequence replacement discards policies owned by removed records. A replacement record supplies its own local declarations; equal outer identity or final index does not transfer discarded policies. For example, a source imported as one record into `prompts` can declare: ```yaml configkit: sequences: - destination: [steps] key: id mode: append id: foo steps: - id: prepare ``` The declaration governs `steps` inside that `foo` record. Retaining `foo` retains its local policy, including the explicit mode. Upserting a new `foo` discards the old `steps` and its declaration even though the outer record keeps its position. Policies independently applicable at a destination are not discarded merely because an individual record is replaced. Ownership determines which declarations are removed. Application registrations remain constraints at their registered destinations. Inheritance copies transfer identity and choose operations as specified next. ## Sibling Inheritance Complete source-tier assembly before resolving inheritance. Retain the existing dependency-ready algorithm and sibling scope rules. At each inheritance combination, the resolved parent's collection is the lower value and the inheriting mapping's local collection is the higher value. Mappings combine recursively; identified sequence records remain whole-record replacements. Use the inheriting destination collection's policy: an applicable explicit mode, including one retained through source tiers, wins. Otherwise infer upsert with effective identity and replace without it. There is no separate default-append rule for inheritance. The same destination policy governs source-precedence and inheritance combinations. YAML identity follows a collection copied through inheritance, subject to agreement and replacement rules. A source collection's mode does not automatically become an instruction at a different inheriting destination. For example, source `steps` has `key: id` and `mode: append`; an inheriting mapping supplies local `steps` without an applicable explicit mode. The copied identity selects upsert there. An explicit destination append instead selects append. A matching destination key agrees; a different key conflicts when combining records, while explicit replace can establish new YAML identity subject to application constraints. For example: ```yaml configkit: sequences: - destination: [parent, items] key: id child: items: - id: foo value: A - id: bar value: B parent: extends: child items: - id: baz value: C ``` Despite the names, `parent` inherits from `child`. Identity at `[parent, items]` selects upsert, producing `[foo, bar, baz]`. If the local sequence instead supplies a revised `foo`, the result is `[foo*, bar]`. Append rejects that repeated identity; replace retains only the local sequence. The same ordering, presence, and kind rules apply at inheritance boundaries. An omitted local sequence retains the inherited sequence. Explicit `[]` clears it under replace and contributes no records under append or upsert. Retain controls until inheritance uses them, then exclude them from application decoding. ## Loading Lifecycle Use the following order: 1. Assemble source tiers and selected overlays, resolving policy before each tier's strict composition and applying collection operations at precedence boundaries. 2. Resolve sibling inheritance, then apply typed defaults and decode the assembled configuration. 3. Apply explicit CLI overrides to that configuration and check supplied values against the application schema. 4. Run application resolution against the overridden configuration. 5. Resolve effective secret references, validate the final configuration including application-registered identities, and publish it. ### Typed Defaults And Application Resolution Source-backed underlays, including application-supplied mapping sources, participate in sequence combination. Typed `Definition.Defaults` remains the initial decoded state: supplied sequences replace default sequences. Source append does not append to a typed default list. Application resolution retains assignment and recorded-transformation semantics. ConfigKit does not automatically combine a callback's assigned sequence with its previous value. Application-registered identities receive final validation independently of how defaults, CLI overrides, or callbacks produce their values. ### CLI Precedence CLI overrides are the highest-precedence input to application resolution. Supplied sequences replace whole sequences, including sequences inside supplied mappings. Mappings retain recursive override behavior; empty sequences and permitted nulls retain explicit replacement meaning. Configuration append or upsert does not reinterpret CLI assignments. Inheritance and defaulting do not run again afterward: overriding assembled `[A, B]` with `[C]` presents `[C]` to application resolution without inheritance reintroducing prior records. Application resolution sees CLI values and their explicit-presence and source metadata. `Resolution.Default` preserves explicit CLI values; normalization and recorded application transformations may still change them. For example, a derived URL can use a CLI hostname, while an explicit CLI URL can be preserved by an absence-based default. Retain recording and provenance requirements for those transformations. Reject structural directives such as `extends` and `configkit` in CLI override paths or supplied values. CLI input supplies application values after structural processing; it cannot supply or cancel inheritance directives, change completed relationships, or prevent earlier inheritance failures. Typed decoding also precedes CLI application, so a CLI replacement cannot repair an earlier decoding failure. A source string in an integer field fails before a valid CLI integer can apply. This changes current override-before-decoding behavior. Semantic validation remains after overrides and application resolution. Secret lookup uses effective references after overrides and resolution. Neither stage bypasses final application identity constraints or semantic validation. ## Validation And Diagnostics ### Structural Identity Checks Allow item-local helper imports to complete and consume applicable non-main source-version headers before checking payload identity for collection assembly. A helper can supply an identity component; a consumed header cannot supply one after removal. Validate each identified contribution before combination and check uniqueness in the resulting identified collection, including inheritance operations. Upsert matches across precedence are replacements; duplicates within a contribution remain errors. Later inheritance, typed defaults, or callbacks cannot repair an identity missing at an earlier assembly boundary. YAML-only identities are enforced during structural composition, precedence assembly, and identified inheritance operations. They have no final-result guarantee after typed defaulting, CLI overrides, or application resolution. Those stages can produce duplicate identities or otherwise violate a YAML-only identity. Applications needing a final guarantee register the constraint through the application-owned interface. ### Final Application Constraints On successful loading, final effective collections covered by application registrations satisfy their declared identity constraints. Retain assembly-time checks and validate final effective sequences before publication, after defaults, inheritance, overrides, application resolution, and secret lookup. This guarantee covers actual sequences, not path existence or allowed shape. It does not require every intermediate callback state to satisfy the constraint and does not cover caller mutations after loading returns. Final checks examine effective typed values; assembly checks validate authored contributions. A final scalar alone does not prove that a source explicitly supplied that field. Use the existing canonical final-value representation and shared publication path as the foundation. The guarantee requires neither a new mutation-history system nor exact causal reconstruction for records supplied by defaults or callbacks. ### Diagnostics And Inspection Identity diagnostics identify collection and item paths using available origin information without exposing identity values. Conflicting declarations identify the collection destination and declaration locations. Correctness includes final values, projected item paths, source destinations, provenance, history, sensitivity, and passive resource contexts. Appending can change final indexes; replacing changes which contributions survive. Preserve the existing loading and inspection contracts through both operations, including policy ownership and inheritance transfer. ## Work Needed From The Baseline The implementation plan should cover these connected changes, with execution order and test sequencing defined there: - **Declarations and public API:** extend application registration to the settled identity model; parse and consume `configkit.sequences` across supported source forms and selected overlays; preserve placement-relative and item-local scopes. Keep sequence declarations independent of filesystem import enablement. - **Identity and policy representation:** implement field-set comparison, typed scalar identity tuples with exact numeric equality, declaration agreement, pending declarations, explicit-mode persistence, replacement lifetime, and record-owned policy transfer. Choose bookkeeping proportional to these semantics. - **Assembly handoffs:** make accumulated lower policy available before higher-tier strict composition. Separate source acquisition from policy-aware composition or otherwise revise assembly sequencing; current main-before-underlay composition is not a design constraint. Preserve distinctions among acquisition order, dependency-ready scheduling within tiers, and precedence order. Account for source records, callbacks, main-version checks, and failure diagnostics. Retain version ownership and validation semantics; higher replacement cannot hide lower-tier structural failures. - **Combination and inheritance:** implement the three operations with strict peer handling, whole-record replacement, presence rules, stable ordering, and the destination policy rules for inheritance. Preserve provenance, history, sensitivity, paths, and resource contexts as records survive, move, or disappear. - **Lifecycle and validation:** move CLI application after inheritance and typed defaults/decoding, preserve explicit-presence and source metadata for application resolution, reject CLI structural directives, and add final application identity validation through the shared publication path. Keep YAML-only enforcement within structural operations. - **Documentation and verification:** exercise ordinary and layered loading, imported and inline collections, item-local scopes, version-header consumption, selected overlays, inheritance, typed defaults, CLI, callbacks, and inspection. Update the public contract and consumer examples when behavior lands. The durable architectural changes are accepted in [ADR 0017](../adr/0017-collection-sequence-policy.md) for collection policy and [ADR 0018](../adr/0018-cli-after-structural-resolution.md) for CLI ordering. Their scoped supersession is recorded in the affected earlier ADRs, and the [architecture policy](../policy/architecture.md) reflects the accepted decisions. These govern the target architecture; the configuration contract continues to describe implemented behavior until the feature lands. This roadmap does not select migration machinery. No open feature decisions remain. The implementation plan should translate this end state into reviewable stages without adding new guarantees, restrictions, or combination semantics.