ConfigKit/docs/roadmap/sequences.md

Sequence Identity And Cross-Tier Combination

Status: draft proposal for discussion. This document captures the requested capabilities, settled design choices, and initial recommendations. Choices under Open Decisions remain unsettled. It is not an implementation plan or an implemented contract. The declaration structure is settled under OD-2; mode names and default selection are settled under OD-5.

Motivation

Allow configuration authors to assemble collections from independently authored records, declare their identity in configuration, and progressively extend a baseline collection through higher layers. Authors should also be able to replace the accumulated collection or replace individual records by identity.

The motivating example is a prompt collection whose defaults supply some records and whose application or installation layers add or supersede records. Across tiers, authors need three operations: whole-sequence replacement, duplicate-rejecting append, and whole-record replacement by identity. The latter retains unmatched lower-tier records and adds higher-tier records with new identities. Matching records are replaced in full.

Current Baseline

The import contract owns current selector, destination, placement, and ordering behavior. An import with placement: sequence-item places each selected source mapping as one item at its effective destination. Item-local helper imports can contribute fields before placement.

The application currently registers sequence identity through Definition.SequenceCompositions, using one literal string field at each registered mapping path. Within a strict tier, distinct identities append and duplicate identities fail. Repeated unregistered sequences conflict. Across precedence tiers, a higher sequence replaces the lower sequence wholesale. The composition contract owns these rules; layered loading owns tier precedence.

Source-backed underlays and Definition.Defaults have different lifecycle roles. Underlays participate in source assembly; typed defaults are applied later. Extending one does not automatically imply extending the other.

Requested Capabilities

The proposed configkit.sequences block introduces a key declaration that accepts either one field or a list of fields for a compound identity. Fields are treated as a set; identity components may be strings, numbers, booleans, or explicitly supplied null values. Applications retain an optional interface for registering a collection and imposing its identity definition. Configuration declarations must conform to an application-owned constraint when one exists. The exact Go interface remains to be designed.

configkit:
  sequences:
    - destination: [prompts]
      key: [namespace, id]
      mode: append
  imports:
    - dir: prompts
      pattern: '*.yaml'
      destination: [prompts]
      placement: sequence-item

Each selected file would contribute a record such as:

namespace: support
id: welcome
text: Welcome

Here mode: append illustrates retaining a lower-layer collection and adding distinct records from the higher layer. The mode field expresses the operation; OD-5 defines its names and effective-identity default. Identity and combination controls belong to the collection declaration; import shorthand is deferred.

Letters below denote record identities; B* is a higher-tier record with the same identity as B and a different payload. The outcomes and ordering shown are settled below.

Lower-tier sequence	Higher-tier sequence	Operation	Proposed result
[A, B]	[C]	append	[A, B, C]
[A, B]	[C]	replace	[C]
[A, B]	[B*]	append	Duplicate-identity error
[A, B]	[B*]	replace	[B*]
[A, B]	[B*]	upsert	[A, B*]
[A, B]	[B*, C]	upsert	[A, B*, C]

Whole-sequence replacement discards the lower sequence. A repeated identity in the retained combined sequence is an error under the proposed append semantics, including when the records have identical or disjoint payloads.

Settled Design Choices

Collection Declarations

OD-2: settled.

Use a root configkit.sequences block containing a sequence of collection declarations. Each entry identifies a destination and may declare key, mode, or both. Application sequences remain ordinary YAML data. This declaration form serves imported items, inline records, and sequences supplied by ordinary mapping imports, including inline sequences in selected overlays.

destination uses the existing sequence of literal mapping keys. Resolve it relative to the declaring source's placement: a fragment mounted under catalog can declare [prompts] locally. A sequence-item source can declare a nested collection within its own item without referring to an eventual numeric index. Preserve those source and item-local scope boundaries.

Identity and mode can be supplied independently. Omitted identity can use an application registration; an explicit matching declaration is accepted, and a conflicting declaration is invalid. Whole-sequence replacement does not itself require identity matching, but application-owned constraints still apply. Identity-policy lifetime across tiers is settled in OD-3 below.

Mode describes the current tier's combination instruction. A lower tier's append does not silently select append for a later overlay. Gather collection declarations before composing their relevant scope so their effect does not depend on which contribution is assembled first. Identity agreement is settled in OD-3; operation agreement and omission among peers are settled in OD-4.

Permit sequence declarations in selected overlays and make their use independent of filesystem import enablement. This deliberately changes the current blanket prohibition on overlay configkit declarations. Enabling imports in overlays is not part of this decision. Consume sequence controls before application decoding, as with other structural directives.

For example, an overlay can replace a complete record by identity:

configkit:
  sequences:
    - destination: [prompts]
      key: id
      mode: upsert

prompts:
  - id: foo
    text: Revised definition

The explicit mode: upsert line can be omitted here because key: id supplies an effective identity. An overlay can instead select duplicate-rejecting append:

configkit:
  sequences:
    - destination: [prompts]
      key: id
      mode: append

prompts:
  - id: baz
    text: Another prompt

Or it can explicitly select whole-sequence replacement:

configkit:
  sequences:
    - destination: [prompts]
      mode: replace

prompts:
  - id: bar
    text: Bar definition
  - id: baz
    text: Baz definition

Explicitly selecting the default operation is valid. Removing the last example's configkit block produces the same replacement behavior only when no effective identity is available. For an identified collection, explicit mode: replace overrides the inferred upsert; it is not redundant. OD-5 defines the selection rule for replace, append, and upsert.

Start with this common collection declaration and defer key and mode shorthand on imports. A single declaration form covers inline overlays and mixed inline/imported collections without duplicate authoring surfaces. Local wrappers around sequence values would require distinguishing application data from control mappings; the existing reserved namespace keeps that distinction clear.

Identity Definitions And Lifetime

OD-3: settled.

Field Sets

key accepts a string or a nonempty list of strings naming direct, literal fields in each record. key: id and key: [id] are equivalent. A name such as metadata.id denotes that literal field, not a nested lookup.

Treat the selected fields as a set: remove repeated names and disregard authored field order when comparing definitions. For example, [namespace, id], [id, namespace], and [namespace, id, id] define the same identity. An empty list is invalid because it supplies no identity fields. These equivalence rules also apply when comparing configuration declarations with application registrations.

Use a consistent field order internally to construct compound identity tuples. Preserve component types and tuple boundaries; concatenating unstructured values must not cause otherwise distinct identities to collide.

Scalar Components And Equality

Each identified record must be a mapping, and every designated identity field must be present. Components may be strings, numbers, booleans, or explicit null. Missing fields and mapping or sequence components are invalid. A present null is a valid component and is distinct from an absent field. Two single-field null identities are equal; in a compound identity, all components must compare equal.

Strings compare exactly, with no trimming, case folding, or normalization. Empty strings remain valid. Scalar categories remain distinct: the string "1", the number 1, the boolean true, and null are different identities.

Numbers compare by mathematical value across integer, fractional, and exponent spellings: 1, 1.0, and 1e0 are equal, as are 0 and -0. Use exact numeric normalization for identity comparison without rounding through a floating-point conversion. This comparison does not rewrite authored values or alter existing typed decoding and projection semantics. The existing parser's rejection of non-finite numeric forms remains unchanged.

Agreement And Introduction

Within a strict tier, identity declarations for the same collection must agree. Equivalent field sets coalesce; different sets produce an error. Across tiers, append and replacement by identity use one agreed definition for the records being combined:

Lower collection identity	Higher declaration	Result when combining records
id	Omitted	Use id
id	id, [id], or [id, id]	Accept the equivalent definition
id	[namespace, id]	Error: different identity definitions
None	id	Establish id and validate all participating records under it

An application registration supplies identity when configuration omits it and remains authoritative over explicit declarations. If neither the application nor either participating collection supplies an identity, append and replacement by identity fail; ConfigKit does not infer an id convention. Introducing identity for a previously unkeyed lower sequence requires checking both lower and incoming records, including uniqueness within each contribution.

Replacement And Policy Lifetime

YAML-owned identity follows the collection that survives an operation:

Operation	Effective YAML-owned identity
Append	Retain or establish the agreed identity
Replacement by identity	Retain or establish the agreed identity
Whole-sequence replacement	Use the replacing tier's declaration and discard the lower collection's YAML-only identity

Without an application constraint, whole-sequence replacement may introduce a different identity, such as replacing an id-keyed collection with a name-keyed collection. A whole-sequence replacement with no new identity declaration and no application registration produces an unkeyed sequence. No separate identity removal directive is needed.

Application registrations continue to constrain the replacement collection. If the application requires id, declaring name is invalid under every mode. The policy reset above applies when a sequence is actually replaced; effects of absent contributions, null collections, and kind mismatches remain under OD-6. These structural rules do not settle final-result enforcement for YAML-only identity declarations, which remains the OD-1 follow-up.

Combination Scope, Declarations, And Ordering

OD-4: settled.

One Operation Per Collection Per Tier

Strictly compose a tier's contributions first, then apply one selected operation per collection against the accumulated lower sequence. A collection declaration governs all contributions to that collection in its tier, including imported and inline records; it does not govern only records in the declaring file.

Collection scope is the resolved destination within its composition scope. Resolve relative destinations first and preserve separate item-local scopes for collections inside different imported items. Apply the same combination semantics to selected overlay contributions.

For example, with an agreed identity, higher-tier fragments contributing [C] and [D] assemble [C, D]. Against lower [A, B], append or replacement by identity produces [A, B, C, D]; whole-sequence replacement produces [C, D]. The tier operation does not sequentially replace peer source contributions.

Operation Agreement And Omission

Coalesce agreeing explicit modes for the same collection within a tier and reject conflicting explicit modes. Omission expresses no preference; apply the default only after reconciling declarations.

Modes declared within one tier	Effective operation
append and append	append
append and omitted	append
replace and omitted	replace
append and replace	Error
All omitted	upsert with effective identity; otherwise replace

Explicitly selecting the default is a real instruction and can conflict with another explicit selection. Different modes in different tiers are valid; a lower tier's mode does not carry forward. Neither a root declaration nor a later declaration wins over conflicting peers. Diagnostics identify the collection destination and conflicting declaration locations. OD-5 defines mode names and the default-selection rule.

Strict Duplicate Handling

Duplicate identities among peer contributions or within one contributed list remain strict-composition errors under all three modes. For example, peer fragments contributing [B] and [B*] conflict even under replacement by identity: neither has higher precedence. Lower [B] and higher [B*] can instead produce [B*] at a precedence boundary. Whole-sequence replacement does not permit duplicate identities within an identified higher-tier sequence.

Ordering

Retain existing dependency-ready source scheduling within each tier and authored item order within each list. Gathering collection declarations does not change assembly order, and identity values do not sort records.

Across tiers:

• Append retains lower records in order, followed by higher records in assembled order.
• Whole-sequence replacement retains only the higher sequence, in its assembled order.
• Replacement by identity replaces matched records in their existing lower positions and appends new identities in higher-tier order.

For example, lower [A, B, C] and higher [C*, A*, D] produce [A*, B, C*, D] under replacement by identity. Updating a record's contents preserves its position; whole-sequence replacement gives the higher tier control of the complete sequence order.

OD-5 settles default selection and operation naming. Absent contributions, empty sequences, null, and kind mismatches remain under OD-6.

Mode Names And Effective-Identity Defaults

OD-5: settled.

Use the mode names replace, append, and upsert, with one spelling per operation:

Mode	Behavior between tiers
replace	Replace the complete lower sequence with the assembled higher sequence
append	Retain lower records and append higher records; duplicate identities fail
upsert	Replace matching records in full, retain unmatched lower records, and append new identities

upsert preserves the existing positions of matching lower records and adds new identities in higher-tier order, as settled in OD-4. It does not recursively merge fields within matching records.

Select the operation at each collection's precedence boundary in this order:

1. Reconcile explicit modes within the current tier under OD-4; conflicting explicit modes fail.
2. If an explicit mode is supplied, use it.
3. Otherwise, select upsert when an effective identity is available.
4. Otherwise, select replace.

Effective identity may come from an application registration, a current-tier declaration, or the lower collection, subject to OD-3's agreement rules. Selection depends on identity availability, not on whether the current tier explicitly repeats a key declaration. A plain overlay sequence can therefore default to upsert when identity was established elsewhere. A lower tier's mode does not carry forward.

Determine the inferred mode using the available identity before applying the operation. Explicit replace still follows OD-3: discard the lower collection's YAML-only identity with the lower sequence, use the replacing tier's identity declaration if supplied, and preserve application-owned constraints. Selecting append or upsert explicitly still requires an effective identity.

This default governs combination between tiers. Within a tier, identified sequences append distinct records and reject duplicate identities regardless of mode. Unidentified sequences retain their existing strict-composition conflict behavior; this decision does not introduce unkeyed concatenation.

Whole-Record Replacement By Identity

The upsert operation replaces each matching lower record with the complete higher record, retains unmatched lower records, and adds new identities. For example, given these lower and higher records:

# Lower record
id: B
text: Original
temperature: 0.5

# Higher record
id: B
text: Revised

The resulting record at this precedence boundary contains id and text. The lower record's temperature is discarded. Subsequent defaulting or other lifecycle operations retain their separately defined semantics. Recursive field-by-field overlay of matching records is outside this selected operation.

Selected Overlay Control

Selected overlay authors must be able to choose any of the three operations for a sequence supplied by the overlay, including records authored inline. The operation applies against the existing effective sequence at that path. For an existing sequence [foo, bar]:

Overlay contribution	Selected operation	Result
[foo*]	Replacement by identity	[foo*, bar]
[baz]	Append	[foo, bar, baz]
[bar, baz]	Whole-sequence replacement	[bar, baz]

Here foo* has the same identity as foo and supplies its complete replacement record. Append retains its duplicate-identity error semantics. The retained records, replacement behavior, and ordering shown are settled, including the OD-4 rule that matched records retain their lower positions.

The settled OD-2 collection declaration supplies the overlay authoring surface. OD-5 supplies operation names and effective-identity default selection. The current selected-overlay contract remains the baseline until the new controls are implemented.

Optional Application-Owned Identity Constraints

Retain an interface through which applications may register a collection and impose its unique-key definition. If the application requires key: id, a configuration declaration of key: [id, name] is invalid: the compound key would permit repeated id values with different names. The application identity constraint applies whichever cross-tier operation configuration selects.

This is an authoritative constraint, distinct from an application default that only fills an omission. Configuration can define identity for collections without an application-owned constraint. Omitted configuration identity uses a registered definition when present, and an explicit matching declaration is accepted, as settled in OD-2. The final-result guarantee for application-owned constraints is settled below; YAML-only enforcement remains an open follow-up to OD-1.

Final-Result Guarantee For Application Registrations

OD-1: settled for application-owned constraints.

On successful loading, the final effective collections covered by application registrations satisfy their declared identity constraints.

Retain assembly-time identity checks and validate the final effective collections before publication. This covers the loaded state after defaults, inheritance, overrides, and application resolution have taken effect. Those operations retain their separately defined combination semantics. The guarantee does not require every intermediate callback state to satisfy the constraint and does not extend to caller mutations after loading returns.

Keep the promise limited to the declared identity constraints. It does not by itself require registered paths to exist or settle null and non-sequence handling; those collection boundaries remain under OD-6. Final validation examines effective typed values, while assembly-time checks retain their role in validating authored contributions. A final scalar value alone does not prove that the field was explicitly supplied in a source.

The existing canonical final-value representation and shared publication path provide the implementation foundation. Diagnostics should identify the affected collection and item paths, using available origin information without exposing identity values. The guarantee requires neither a new mutation-history system nor exact causal reconstruction for records supplied by defaults or callbacks. Final enforcement for identities declared only in YAML is not settled here.

Architectural Direction

The settled collection declaration gives imported and inline records at the same destination a shared identity policy. Declarations are gathered before composing the relevant scope.

OD-3 settles identity agreement and lifetime; OD-4 settles tier-wide operation selection, strict peer composition, and ordering at precedence boundaries.

This requires collection policy to survive import expansion through precedence assembly. Correctness includes final values, projected item paths, source destinations, provenance, history, sensitivity, and passive resource contexts. Appending changes final indexes; replacing changes which contributions survive. The implementation must account for both through the existing loading lifecycle.

Scope And Decision Ownership

This is narrower than the deferred identified-record sequences proposal, which also considers recursive merging of matching records and record-specific inheritance. Those capabilities remain separate candidates. Compound keys, duplicate-rejecting append, and whole-record replacement by identity do not require adopting them.

The proposal would extend identity declaration ownership and revise the cross-tier replacement decision in ADR 0014. If adopted, record the architectural change in a superseding decision and reconcile affected architecture policy before implementation. The current ADR and contracts remain authoritative until that decision is made. This draft neither changes them nor selects migration machinery.

A staged implementation plan should follow resolution of the questions below. It should cover ordinary and layered loading, item-local scopes, source-version processing, identity validation, precedence, and inspection together. Public API design and test sequencing belong to that later plan.

Open Decisions

OD-1 Follow-Up. Final Validation For YAML-Only Identity Declarations

Application-registered collections have the settled final-result guarantee above. Should a configuration-only identity declaration carry the same guarantee, or govern structural assembly alone?

Discussion basis: Declaration scope and identity-policy lifetime are settled in OD-2 and OD-3. Revisit final enforcement using the effective identity of each retained collection. The application-owned guarantee remains independent of this choice.

Alternatives: Applying the same final check to YAML-declared identities would give both declaration forms consistent enforcement, but requires carrying the effective policy through later lifecycle stages, including nested records. Limiting YAML-only identity to assembly keeps policy lifetime narrower, but later operations could produce a final collection that no longer satisfies that identity declaration.

OD-6. What Do Absence, Empty Sequences, And Null Mean?

Can a declaration selecting nothing affect a lower collection? How do explicit empty sequences, null, and non-sequence values interact with all three operations?

Initial recommendation: An optional import selecting nothing contributes nothing and does not clear lower values. An explicitly supplied empty sequence with whole-sequence replacement can clear a collection; append or replacement by identity with an empty sequence contributes no records. Preserve the distinction between absence and supplied values. Decide null and kind-mismatch handling explicitly rather than silently treating them as empty sequences.

Alternative: Treat an explicitly declared replacement operation as a reset even when its import selects nothing. This provides a compact clearing mechanism, but makes an optional missing file or empty search destructive. Keeping clearing as an explicit value avoids that coupling.

OD-7. Which Lifecycle Stages Participate?

Do the new cross-tier operations extend only source-backed tiers, or also Definition.Defaults, inheritance, CLI overrides, and application-created records? At which boundaries must identity be validated, including helper-supplied fields and consumed source version headers?

Initial recommendation: Start with source-backed strict composition and precedence assembly. Validate records at the relevant collection assembly boundaries and uniqueness across each combined result. Preserve existing typed-default and application-resolution semantics unless their inclusion is explicitly chosen. Selected-overlay combination is included in the settled scope; its declaration mechanism is settled in OD-2. Account for item-local helpers and source-version consumption before deciding identity is available, consistent with the current lifecycle.

The settled OD-1 guarantee covers final validation of application constraints. Validating the final collection does not itself change how defaults, inheritance, CLI overrides, or callbacks combine values.

Alternative: Extend the combination operations through defaulting, inheritance, overrides, and application resolution. This would support progressively extending collections across more of the lifecycle, but requires additional combination, callback, and inspection rules. Supporting typed default lists is a separate scope choice from extending source-backed baseline collections; the application-owned final-result guarantee remains independent of this combination-scope decision.