ConfigKit/docs/roadmap/sequence-composition.md

Identity-Based Sequence Composition

Status: agreed feature direction; not implemented. Resolve the open decisions below before implementation, then execute the plan in order.

Goal and current baseline

Allow consumers to opt selected sequences into strict composition by one explicit string identity field. Independent fragments can contribute distinct records to the same list; repeated identities fail composition. Keep records as sequences for typed decoding, inspection, and effective YAML.

Current strict composition recursively combines mappings and rejects every other overlap. Current precedence assembly and selected overlays replace sequences wholesale. There is no sequence-composition registration in Definition today.

This is a narrower feature than the identified record sequences candidate. That candidate's composite identities, recursive merging of matching records, identity-based overlays, and record inheritance are not part of this plan.

Agreed behavior

Registration and identities

• Consumers explicitly designate each sequence path and one direct identity field, such as prompts and id. Unregistered sequences retain current rules.
• Records in a participating sequence must be mappings containing the designated field as a string. Missing, null, and non-string identities are invalid; identities are not coerced from numbers or supplied by defaults or resolution.
• Compare strings exactly, without trimming, case folding, or normalization. Whether the empty string is accepted is an open decision below.
• Retain identity fields as ordinary record content in the resulting sequence. They are not consumed as metadata or converted into public mapping keys.
• This is a source-composition rule, not a guarantee of identity uniqueness after application defaults and callbacks. Those retain their existing behavior; applications can enforce final domain constraints through validation.

Strict composition

For a registered prompts sequence with identity field id, two separate source fragments could contribute:

prompts:
  - id: prompt_one
    text: "foo"

prompts:
  - id: prompt_two
    text: "bar"

The composed value is:

prompts:
  - id: prompt_one
    text: "foo"
  - id: prompt_two
    text: "bar"

• Append distinct records in contribution order, preserving each source list's order. Use existing deterministic input/import traversal within a tier; identity values do not sort the result.
• Reject repeated identities both across contributions and within a single contributing sequence, including a sequence that has no peer contribution. Equal bodies and disjoint bodies both conflict. Never recursively combine records with the same identity.
• An empty sequence contributes no records. Composing empty sequences succeeds; an explicitly contributed empty sequence remains present, not absent.
• Apply these rules to main inputs and their imports, and to inputs and imports composing each underlay group. Honor input and relative-import destinations. Source acquisition kind must not change composition behavior.
• Invalid contributions and duplicate identities in a tier cannot be repaired by a higher-precedence replacement. Report a normal load error and return no usable partial result.

The analogy with keyed mappings is limited: distinct keys combine, but mappings at a shared key currently recurse. A shared record identity always conflicts under this feature.

Precedence and other operations

Sequence replacement remains the rule between underlay tiers, between underlays and the main tier, and for the selected overlay. An upper sequence replaces the complete lower sequence, even at a registered path; [] clears it. Identities may recur across those tiers without a cross-tier duplicate error. Null and cross-kind precedence replacements retain current semantics.

Whole-sequence CLI overrides, defaults, application resolution, and built-in sibling inheritance retain their existing semantics. This feature adds no record inheritance, identity selector syntax, record update/delete/rename operators, or identity-based behavior to effective-configuration diffs. Existing atomic sequence diffs continue to compare final frozen values.

Inspection and diagnostics

Appending records changes their effective indexes. Preserve the distinction between an original source position and its position in the assembled sequence. Effective presence, origins, resource contexts, history, and explanations must refer to the correct record and its original physical source locations.

Composition history must describe the accumulated sequence correctly, rather than replay each fragment as a whole-sequence replacement. Later actual replacements must still retain displaced history. Equal numeric indexes in different source lists must not make unrelated records share provenance.

Duplicate diagnostics must identify the conflicting source locations in deterministic order. Avoid including identity values in diagnostic text where sensitivity is not yet known; identify the sequence, identity field, and source locations instead. Apply existing sensitivity and disclosure rules to successful and failed inspection, including records whose effective indexes changed. Retain detached public records and input/result ownership guarantees.

Implementation plan

1. Settle the remaining contract choices

• Resolve the Open Decisions section and update this document with the choices.
• Select the public registration names, path interpretation, and diagnostic codes. Duplicate claims should use the existing composition-conflict category where appropriate; malformed registration and malformed records need distinct, actionable diagnostics.
• Record the significant composition extension in an ADR during implementation, under the architecture policy. No ADR is created by this roadmap-writing pass.

Completion: the public behavior and validation boundary are unambiguous, with no accidental commitment to the broader identified-records proposal.

2. Add registration and pass it through the loading paths

• Add root-owned public registration types and fields using the selected API. Convert them explicitly into private engine state; preserve the root facade boundary and reusable definition ownership.
• Validate registrations before source processing and pass the same policy through ordinary and layered loading, import assembly, and mounted inputs.
• Compile path lookup once per load if useful. No registrations must preserve current behavior without introducing a new global cache or merge framework.

Completion: public callers can select sequence paths consistently across load forms, and invalid options fail through the normal diagnostics boundary.

3. Extend strict composition and its retained evidence together

The current composeDocuments implementation in internal/engine/compose.go collects claims by path, rejects non-mapping overlaps, merges disjoint mappings, and retains origins and history applications. Changing only the final tree merge would leave these other representations wrong.

• Extend claim validation and tree assembly for registered sequences. Validate single contributions as well as overlaps. Track identities per sequence per strict tier, and append cloned records in stable order. Keep duplicate lookup proportional to record count rather than repeatedly scanning prior records.
• Keep original source locations while assigning effective indexes. Carry the necessary placement information into claims, origins, resource contexts, and history; do not expose a new public identity-addressing model.
• Update history/explanation replay to represent sequence accumulation. Ensure snapshots, historical paths, sensitivity classification, and subsequent whole-sequence replacement agree with the assembled tree.
• Integrate with ordinary input assembly and grouped assembly in internal/engine/load_input.go and internal/engine/load_request.go. Keep tier boundaries explicit so uniqueness is never enforced across precedence layers accidentally.
• Preserve parser/source limits and retained-node accounting. Keep source-header processing and structural-directive handling in their existing lifecycle positions; do not restore consumed headers to supply record identities.

Completion: final values and their inspection evidence agree across composition and subsequent replacement. No input mutation, source-context mix-up, or sensitive-value disclosure is introduced.

4. Verify behavior at its owning boundaries

• Composition tests: distinct and repeated identities, duplicates in one list, malformed records and identity types, empty strings according to the chosen decision, empty lists, ordering, missing paths, and null/cross-kind cases.
• Loading tests: imports, underlay groups, mounted and relative destinations, equivalent source forms, and conflicts hidden by later layers. Demonstrate that separate tiers may reuse identities and still replace whole lists.
• Metadata and inspection tests: records appended from the same source index acquire distinct final indexes while keeping correct locations, history, resource contexts, and sensitivity. Cover later replacement and historical redaction, including a failure before final classification.
• Public integration tests: registration conversion and reuse, normal load-error behavior, unchanged unregistered sequences, selected-overlay and CLI replacement, defaults/resolution behavior, and atomic effective diffs.
• Add a small maintained consumer example showing two fragments composing and a selected overlay replacing the result. Include only enough cases at each layer to catch distinct failure modes.

Run focused tests during development, then the everyday validation prescribed by the development guide. Run ownership or concurrency checks where required by the change. No independent performance benchmark campaign is part of this plan; keep measurement separate from the busy-host environment discussed in the performance audit.

Completion: new behavior is covered without weakening the existing atomic replacement and default-behavior regressions.

5. Publish the implemented contract in repository documentation

• Document the public API beside its declarations. Update the configuration contract's composition, inspection/history, and precedence sections, and the consumer guide and maintained example together when implementation lands.
• Update the architecture policy and component overview only where the implemented boundary changes. Keep the broader candidate in future.md clearly distinct from this delivered subset.
• Retire this completed roadmap through the usual documentation lifecycle. Release preparation and publication remain separate work.

Completion: current documentation describes the delivered feature, and no future proposal is presented as implemented behavior.

Open Decisions

1. Public registration ownership

Recommendation: place a collection of path/identity-field registrations on Definition, shared by ordinary and layered loading. This describes how the application's configuration is composed and avoids diverging registrations between load forms. Choose final type and field names before coding.

Alternative: place registrations on load options. That allows one definition to be composed differently per invocation, but duplicates configuration across ordinary/layered options and makes composition semantics less visibly part of the application's reusable definition.

2. Path matching, overlapping registrations, and unused rules

Recommendation: interpret paths in effective configuration coordinates after input/import placement, using literal mapping-key segments initially. Reject duplicate registrations for the same path; allow a registration to match no source value. These rules fit mounted collections without requiring discovery of which optional sources will be present. Restricting paths to mapping segments would be an explicit initial scope choice, not an existing library constraint.

Alternative: support exact sequence-index segments too, including registered sequences nested inside appended records. This is more expressive, but requires choosing whether a rule matches source indexes or composed indexes, and how matching behaves as earlier lists contribute records. Settle that interpretation and overlapping registration behavior before supporting these paths. Wildcard or identity-selector matching is not needed for the agreed use case.

3. Empty identity strings

Recommendation: accept the empty string as an exact identity; two empty identities conflict. Presence and string type are sufficient for the composition algorithm, and applications can require nonempty domain identifiers themselves.

Alternative: require nonempty identities. This catches likely authoring mistakes earlier, but adds a domain restriction beyond uniqueness and string type. Whitespace trimming or normalization would still remain out of scope.

4. Null and non-sequence values at registered paths

Recommendation: treat registration as selecting how actual sequences compose, not as a general schema assertion. Preserve existing rules for absent, null, scalar, and mapping values: non-sequence overlaps retain ordinary strict composition behavior, and final decoding/validation checks application types. Two sequence values use the new rule; sequence/null and sequence/mapping overlaps remain conflicts. No later layer can hide an actual strict-composition conflict.

Alternative: require every present non-null source value at the path to be a sequence, even when a later tier would replace it. This catches misplaced configuration earlier but makes registration an additional source-shape rule. If selected, define null handling explicitly rather than treating it as empty.

5. Validation of replacement-only source sequences

Recommendation: require well-formed, internally unique identities for actual source sequences at registered paths, including selected-overlay sequences. This makes source authoring consistent while preserving whole-sequence replacement and permitting identities to recur across tiers. Do not extend the rule to synthetic CLI replacements, defaults, or callback-generated values; final domain validation remains application-owned.

Alternative: validate only source sequences participating in a strict tier, leaving selected-overlay sequences subject solely to existing decoding and application validation. This keeps the check strictly tied to the composition operation, but allows the same list to succeed as an overlay and fail as a base contribution. Choose this boundary explicitly; replacement semantics alone do not settle whether an overlay's own duplicate identities are invalid.