ConfigKit/docs/roadmap/sequence-composition.md

Sequence-Item Placement and Identity-Based Composition

Status: agreed feature direction; not implemented. This roadmap defines the intended scope and remaining design choices. A separate staged implementation plan will be prepared after the roadmap is finalized.

Goal and current baseline

Let consumers assemble a list of records from independently authored mapping files. Placement makes a source mapping one sequence item; identity-based strict composition combines distinct items and rejects repeated identities. These are complementary operations, with explicit consumer opt-in.

Current input destinations place source mappings beneath mapping-key paths. Imports support immediate-parent-relative destinations. Current strict composition combines mappings recursively and rejects other overlaps, while precedence assembly and selected overlays replace sequences wholesale. Neither sequence-item placement nor sequence-composition registration exists today.

This roadmap covers a narrower feature than the identified record sequences candidate. Composite identities, recursive merging of matching records, identity-based overlays, and record inheritance are not requested here. That scope statement does not establish permanent architectural prohibitions.

Targeted behavior

PromptKit use case

Each file defines one prompt directly.

prompt_one.yml:

id: prompt.one
text: "foo"

prompt_two.yml:

id: prompt.two
text: "bar"

The consumer selects item placement at prompts for both sources and registers id as that sequence's identity field. Within one strict composition tier, the result is:

prompts:
  - id: prompt.one
    text: "foo"
  - id: prompt.two
    text: "bar"

The files need neither a filename-derived wrapper key nor their own list wrapper. Existing sources that already contain prompts: [...] can contribute to the same registered sequence under the same composition rule.

Explicit sequence-item placement

• The destination identifies where content belongs; an explicit placement mode selects ordinary mapping placement or placement as one sequence item. Preserve existing behavior when the new mode is not selected. Do not infer the mode from document shape, filename, or the presence of an id field.
• Support item placement for application-supplied inputs and YAML path/glob imports. Each matched file of an item-placement glob contributes one item, using existing match and traversal ordering. Physical import-path resolution remains relative to the declaring file.
• Wrap the source's application mapping as one item at the destination sequence; the new operation is not flattening a source sequence or splitting a mapping into several items. Existing supported source forms should receive equivalent placement behavior; this does not expand accepted source-root shapes.
• Item placement supplies structure, not identity values. Keep authored id and other fields in the item. Source directives and version headers retain their own processing rules; their interactions with the new placement context must be resolved as described under Open Decisions.
• Preserve existing absence distinctions: NoSource() and an unmatched optional import contribute nothing. An actual empty mapping contributes an empty item, which fails the agreed identity requirement when that item participates in a registered composition sequence.

Nested imports, placement without a composition registration, and the public representation of item placement remain decisions rather than implicit rules.

Identity-based strict composition

• Consumers designate sequence paths and one direct string identity field for each. Unregistered sequences retain their current composition behavior.
• Each participating record is a mapping with the designated string field. Missing, null, and non-string identities are invalid. Compare exact strings without trimming, case folding, normalization, or numeric coercion. Empty string handling remains open.
• Append distinct records in contribution order, preserving each source list's order. Use existing deterministic traversal within a strict tier rather than sorting by identity.
• Repeated identities conflict across contributions and within one contributing list, including a list with no peer contribution. Equal bodies and disjoint bodies both conflict; matching records do not recursively merge.
• An empty sequence contributes no records. Two empty sequences compose, and an explicitly contributed empty sequence remains present.
• Apply composition to main inputs and their imports and within each underlay group, including placed item contributions and already-authored sequences. Identity uniqueness is scoped to the sequence in that strict tier.
• A composition error cannot be hidden by a higher-precedence replacement. Use normal load errors and the existing failed-load result contract.

This differs from mapping composition at a shared key: mapping values can recurse, whereas a repeated record identity conflicts. It does not establish final uniqueness after defaults or application resolution. Identity values are not supplied by those later phases; final domain validation remains application-owned.

Precedence and other operations

Between underlay tiers, between underlays and the main tier, and in selected overlays, sequences continue to replace lower sequences wholesale. [] clears the lower list. The same identity can occur in different tiers without a cross-tier duplicate error. Item placement does not turn precedence assembly into append semantics.

Whole-sequence CLI overrides, defaults, application resolution, and existing sibling inheritance retain their behavior. This feature does not make records inherit from one another by identity. Effective-configuration diffs continue to compare final sequences atomically. Whether to validate identities in a replacement-only overlay source is an open decision, separate from replacement.

Existing inspection and ownership obligations

A field at source path text can become prompts[1].text; a record originally at source index zero can acquire a different effective index after composition. Keep original physical locations and resource contexts associated with the right record while reflecting effective placement in presence and origin queries.

History and explanations must represent accumulation within a tier and actual replacement between tiers. Replaying each contribution as a replacement would lose earlier records. Apply existing historical sensitivity and disclosure rules when indexes change or sequences are replaced, and preserve detached public records and existing input/result ownership. These are applications of current contracts, not a new promise of persistent identity-based paths across loads.

Work needed

The staged plan will sequence these areas after the decisions below are settled:

• Public API and import syntax: introduce the selected composition registration and placement controls at the existing root-owned boundaries; convert them into private engine state. Define invalid-option handling and retain current defaults for callers that do not opt in.
• Acquisition and placement: support item contributions in ordinary and layered loading, input/import destinations, and glob expansion. Implement the selected nested-import and source-version interpretation without changing acquisition identity or physical resource resolution.
• Composition and retained evidence: extend claim validation, tree assembly, origins, and history together. The current composer in internal/engine/compose.go collects claims by path and stores history applications as well as the final tree. Updating only the tree merge is insufficient. Keep tier boundaries explicit and preserve the existing limits and accounting through the new wrapper and accumulation operations.
• Inspection integration: carry source-to-effective placement through history, explanations, sensitivity, resource contexts, and public source records. Use the smallest internal representation that preserves those existing contracts; this roadmap does not prescribe a new indexing or caching subsystem.
• Behavioral verification: cover direct and glob-imported item sources, already-authored lists, mixed contributions, duplicate/malformed identities, empty and absent contributions, ordering, mounted paths, and nested imports. Verify whole-list replacement, errors hidden by later tiers, version and structural processing, remapped indexes, resource contexts, historical redaction, and detached records. Protect current unregistered behavior, defaults/resolution, and atomic diffs with proportionate regression coverage.
• Documentation and adoption: provide a maintained PromptKit-shaped example and update public declaration comments, configuration/inspection contracts, consumer guidance, and relevant internal documentation when behavior lands. Record the significant design decision under the architecture policy during implementation; no ADR is created by this roadmap update. Keep the broader candidate in future.md distinct from the implemented subset.

Completion means the PromptKit example works through public inputs and imports, composition and replacement behave as agreed, and configuration values agree with their inspection evidence. Validation follows the development guide. This documentation pass neither implements the feature nor runs performance benchmarks.

Open Decisions

Recommendations below are proposals, not accepted restrictions. Resolve them before preparing the staged implementation plan. Exact public names and error code spellings can be selected with the API design; implementation details that do not affect observable behavior need not become additional contracts.

1. Public controls and ownership

Recommendation: put sequence path/identity-field registrations on Definition and an explicit placement mode beside input destination controls, with equivalent YAML import syntax. Expose the corresponding choice for ordinary loading too. This separates reusable composition policy from placement of particular sources and keeps default placement unchanged.

Alternative: put composition registrations on load options, allowing different composition rules for each invocation of one definition. That offers flexibility but duplicates policy across ordinary/layered load forms. A dedicated item-input constructor is also viable instead of a mode field, but needs equivalent import syntax and a clear way to combine destination and source options.

2. Destination and registration paths

Recommendation: use literal mapping-key paths to identify destination sequences and match composition registrations after placement. This fits the existing destination vocabulary and the requested prompts use case. Preserve existing destination validation; do not silently add new reserved names. Keep empty/root item destinations invalid if they would replace the currently required mapping document root with a sequence. These are proposed scope choices, not already-approved new validation rules.

Allow unused registrations for optional configuration. Coalesce identical registrations and reject conflicting identity rules for the same path. For ancestor/descendant registrations, apply a rule only where its literal path actually reaches a sequence; it does not implicitly traverse every record.

Alternative: support explicit sequence-index paths, including sequences inside appended records, or sequence-valued configuration roots. These are expressive extensions, but require decisions about source versus effective indexes and root decoding. Nested-item destinations arising through imports also need the context rule in decision 4; they must not accidentally introduce wildcard path matching. Rejecting even identical duplicate registrations is another viable option, but catches redundant setup at the cost of an additional restriction.

3. Item placement without identity registration

Recommendation: allow item placement independently. It contributes a singleton sequence; without a registration, two present sequences still conflict during strict composition, while precedence still replaces them. This keeps placement orthogonal and allows a consumer to place a single item without inventing an identity rule.

Alternative: require every item destination to have a composition registration. This catches a likely setup omission early, but unnecessarily couples placement to identity validation for consumers that only need one item. Implicitly appending unregistered sequences would change the agreed opt-in composition boundary.

4. Imports declared inside an item-placed source

Recommendation: compose ordinary imports into that source's item mapping; interpret their relative destinations inside the item. An explicit item-placement import then contributes an item to a sequence within that context. A collection manifest can import multiple peer prompt files as items at prompts. Do not implicitly inherit item mode and turn every helper import into another prompt. This permits helper fragments without requiring each helper to declare an ID.

This recommendation introduces an item-local composition context. Specify how omitted and empty destinations resolve, how nested contexts are represented, and how registrations address any nested destination sequences consistently with decision 2 before accepting it.

Alternative: keep inherited destinations anchored at the enclosing collection and require an explicit mode on each imported contribution. That can simplify peer-item expansion, but ordinary helper mappings would collide with the collection sequence rather than extend the declaring record. Inheriting item mode as well would make peer imports concise, but needs an explicit way to request helpers within one record. Do not reject nested imports merely to avoid settling their interpretation.

5. Identity-validation boundary and structural processing

Recommendation: validate each contributed record after its source directives and applicable version-header processing, and after any item-local mapping composition chosen in decision 4, but before collection composition and later precedence replacement. Ordinary helper imports can therefore supply fields of one record; each completed record still needs its identity. Defaults and application callbacks do not supply missing identities at this stage.

Keep existing extends semantics: an item itself has no mapping-sibling namespace for record inheritance, while mappings inside an item may have one. Consumed structural fields or headers do not become application identity values. Determine whether a proposed identity-field selection overlaps such controls through the existing processing rules rather than inventing a blanket field-name blacklist.

Alternative: require the directly authored item file to contain its own ID before processing helpers. This simplifies early duplicate detection but adds a restriction not needed for the requested assembled-record behavior. Validating only the final configuration is not sufficient to catch the agreed strict-tier conflicts that later replacements would hide.

6. Empty identities and literal field names

Recommendation: accept empty string identity values, with repeated empty values conflicting. Interpret the designated direct field name literally, including unusual or empty YAML mapping keys where existing processing permits them. Presence and string type suffice for comparison; applications can impose stronger naming rules through their own validation.

Alternative: require nonempty identity values or nonempty field names. This can catch mistakes earlier but imposes additional domain conventions. Exact comparison without trimming or normalization remains the agreed behavior.

7. Non-sequence values at registered paths

Recommendation: let registration select how actual sequences compose, without asserting that every source must contain a sequence at that path. Missing values and non-sequence values retain existing composition and decoding behavior. Sequence/null and sequence/mapping overlaps still conflict; item placement contributes an actual sequence and follows those same rules.

Alternative: reject every present non-null non-sequence source value at a registered path, even if a higher tier would replace it. This catches misplaced configuration earlier but makes registration an extra source-shape validator. If selected, define null behavior explicitly rather than equating null with an empty sequence.

8. Validation of replacement-only sequences

Recommendation: apply identity validation where source sequences participate in strict composition, including singleton contributions, and preserve existing validation for replacement-only selected overlays and synthetic CLI overrides. Defaults and callbacks retain the agreed behavior. This keeps the new check tied to its purpose and avoids broadening source validation merely for uniformity. This revises the previous draft's recommendation to validate overlay identities.

Alternative: also require replacement-only source sequences to have valid, unique identities. Authoring rules would be more uniform, but this would reject some overlay configurations that do not need identity matching to replace a list. It needs explicit acceptance; whole-sequence replacement does not imply it.

9. Source-version checks for item-placed main inputs

Recommendation: keep enabled source-header checks source-local before item placement and preserve accepted header consumption/retention rules. Define the main header's effective path, protected-path behavior, and reported evidence using its actual item placement. Check ordinary and layered loading and all source-version modes, including default-mode effective-path interpretation. A sequence identity rule must not silently become a version policy.

Alternative: initially support item placement only for non-main contributions, leaving the main document in its existing mapping placement model. This covers a collection manifest importing prompt items but adds a source-role restriction beyond the requested general input capability. Prefer resolving the main-path semantics over imposing that restriction for implementation convenience.

10. Public placement evidence and diagnostics

Recommendation: retain SourceInfo.Destination as the effective container destination and add the minimum placement information needed to identify item mode. Use existing effective-path metadata for final indexes; a source need not have one permanent item index after replacement. Diagnostics should identify conflicting locations and paths without exposing identity values before sensitivity is known. Reuse existing error categories where they accurately describe the failure.

Alternative: report a resolved item path directly in source records, with explicit semantics for replaced items, helper sources, and different historical indexes. This is convenient for consumers but introduces a broader inspection contract. A permanent identity-addressed source path is not needed for this use case. Decide the public evidence shape without promising stable indexes across loads or changes in contribution order.