ConfigKit/docs/roadmap/sequence-composition.md

# Sequence-Item Placement and Identity-Based Composition

Status: agreed feature direction; not implemented. This roadmap defines the
intended scope and remaining design choices. A separate staged implementation
plan will be prepared after the roadmap is finalized.

## Goal and current baseline

Let consumers assemble a list of records from independently authored mapping
files. Placement makes a source mapping one sequence item; identity-based strict
composition combines distinct items and rejects repeated identities. These are
complementary operations, with explicit consumer opt-in.

Current [input destinations](../config.md#input-destinations) place source
mappings beneath mapping-key paths. Imports support immediate-parent-relative
destinations. Current [strict composition](../config.md#strict-document-composition)
combines mappings recursively and rejects other overlaps, while
[precedence assembly](../config.md#layered-loading) and
[selected overlays](../config.md#selected-overlays) replace sequences wholesale.
Neither sequence-item placement nor sequence-composition registration exists today.

This roadmap covers a narrower feature than the
[identified record sequences candidate](future.md#1-identified-record-sequences).
Composite identities, recursive merging of matching records, identity-based
overlays, and record inheritance are not requested here. That scope statement
does not establish permanent architectural prohibitions.

## Targeted behavior

### PromptKit use case

Each file defines one prompt directly.

`prompt_one.yml`:

```yaml
id: prompt.one
text: "foo"
```

`prompt_two.yml`:

```yaml
id: prompt.two
text: "bar"
```

The consumer selects item placement at `prompts` for both sources and registers
`id` as that sequence's identity field. Within one strict composition tier, the
result is:

```yaml
prompts:
  - id: prompt.one
    text: "foo"
  - id: prompt.two
    text: "bar"
```

The files need neither a filename-derived wrapper key nor their own list wrapper.
Existing sources that already contain `prompts: [...]` can contribute to the same
registered sequence under the same composition rule.

### Explicit sequence-item placement

- The destination identifies where content belongs; an explicit placement mode
  selects ordinary mapping placement or placement as one sequence item. Preserve
  existing behavior when the new mode is not selected. Do not infer the mode
  from document shape, filename, or the presence of an `id` field.
- Support item placement for application-supplied inputs and YAML path/glob
  imports. Each matched file of an item-placement glob contributes one item,
  using existing match and traversal ordering. Physical import-path resolution
  remains relative to the declaring file.
- Wrap the source's application mapping as one item at the destination sequence;
  the new operation is not flattening a source sequence or splitting a mapping
  into several items. Existing supported source forms should receive equivalent
  placement behavior; this does not expand accepted source-root shapes.
- Item placement supplies structure, not identity values. Keep authored `id` and
  other fields in the item. Source directives and version headers retain their
  own processing rules; their interactions with the new placement context must
  be resolved as described under Open Decisions.
- Preserve existing absence distinctions: `NoSource()` and an unmatched optional
  import contribute nothing. An actual empty mapping contributes an empty item,
  which fails the agreed identity requirement when that item participates in a
  registered composition sequence.

Nested imports, placement without a composition registration, and the public
representation of item placement remain decisions rather than implicit rules.

### Identity-based strict composition

- Consumers designate sequence paths and one direct string identity field for
  each. Unregistered sequences retain their current composition behavior.
- Each participating record is a mapping with the designated string field.
  Missing, null, and non-string identities are invalid. Compare exact strings
  without trimming, case folding, normalization, or numeric coercion. Empty
  string handling remains open.
- Append distinct records in contribution order, preserving each source list's
  order. Use existing deterministic traversal within a strict tier rather than
  sorting by identity.
- Repeated identities conflict across contributions and within one contributing
  list, including a list with no peer contribution. Equal bodies and disjoint
  bodies both conflict; matching records do not recursively merge.
- An empty sequence contributes no records. Two empty sequences compose, and an
  explicitly contributed empty sequence remains present.
- Apply composition to main inputs and their imports and within each underlay
  group, including placed item contributions and already-authored sequences.
  Identity uniqueness is scoped to the sequence in that strict tier.
- A composition error cannot be hidden by a higher-precedence replacement. Use
  normal load errors and the existing failed-load result contract.

This differs from mapping composition at a shared key: mapping values can recurse,
whereas a repeated record identity conflicts. It does not establish final
uniqueness after defaults or application resolution. Identity values are not
supplied by those later phases; final domain validation remains application-owned.

### Precedence and other operations

Between underlay tiers, between underlays and the main tier, and in selected
overlays, sequences continue to replace lower sequences wholesale. `[]` clears
the lower list. The same identity can occur in different tiers without a
cross-tier duplicate error. Item placement does not turn precedence assembly
into append semantics.

Whole-sequence CLI overrides, defaults, application resolution, and existing
sibling inheritance retain their behavior. This feature does not make records
inherit from one another by identity. Effective-configuration diffs continue to
compare final sequences atomically. Whether to validate identities in a
replacement-only overlay source is an open decision, separate from replacement.

### Existing inspection and ownership obligations

A field at source path `text` can become `prompts[1].text`; a record originally
at source index zero can acquire a different effective index after composition.
Keep original physical locations and resource contexts associated with the right
record while reflecting effective placement in presence and origin queries.

History and explanations must represent accumulation within a tier and actual
replacement between tiers. Replaying each contribution as a replacement would
lose earlier records. Apply existing historical sensitivity and disclosure rules
when indexes change or sequences are replaced, and preserve detached public
records and existing input/result ownership. These are applications of current
contracts, not a new promise of persistent identity-based paths across loads.

## Work needed

The staged plan will sequence these areas after the decisions below are settled:

- **Public API and import syntax:** introduce the selected composition
  registration and placement controls at the existing root-owned boundaries;
  convert them into private engine state. Define invalid-option handling and
  retain current defaults for callers that do not opt in.
- **Acquisition and placement:** support item contributions in ordinary and
  layered loading, input/import destinations, and glob expansion. Implement the
  selected nested-import and source-version interpretation without changing
  acquisition identity or physical resource resolution.
- **Composition and retained evidence:** extend claim validation, tree assembly,
  origins, and history together. The current composer in
  `internal/engine/compose.go` collects claims by path and stores history
  applications as well as the final tree. Updating only the tree merge is
  insufficient. Keep tier boundaries explicit and preserve the existing limits
  and accounting through the new wrapper and accumulation operations.
- **Inspection integration:** carry source-to-effective placement through history,
  explanations, sensitivity, resource contexts, and public source records. Use
  the smallest internal representation that preserves those existing contracts;
  this roadmap does not prescribe a new indexing or caching subsystem.
- **Behavioral verification:** cover direct and glob-imported item sources,
  already-authored lists, mixed contributions, duplicate/malformed identities,
  empty and absent contributions, ordering, mounted paths, and nested imports.
  Verify whole-list replacement, errors hidden by later tiers, version and
  structural processing, remapped indexes, resource contexts, historical
  redaction, and detached records. Protect current unregistered behavior,
  defaults/resolution, and atomic diffs with proportionate regression coverage.
- **Documentation and adoption:** provide a maintained PromptKit-shaped example
  and update public declaration comments, configuration/inspection contracts,
  consumer guidance, and relevant internal documentation when behavior lands.
  Record the significant design decision under the architecture policy during
  implementation; no ADR is created by this roadmap update. Keep the broader
  candidate in `future.md` distinct from the implemented subset.

Completion means the PromptKit example works through public inputs and imports,
composition and replacement behave as agreed, and configuration values agree
with their inspection evidence. Validation follows the
[development guide](../development.md#baseline-validation). This documentation
pass neither implements the feature nor runs performance benchmarks.

## Open Decisions

Recommendations below are proposals, not accepted restrictions. Resolve them
before preparing the staged implementation plan. Exact public names and error
code spellings can be selected with the API design; implementation details that
do not affect observable behavior need not become additional contracts.

### 1. Public controls and ownership

**Recommendation:** put sequence path/identity-field registrations on `Definition`
and an explicit placement mode beside input destination controls, with equivalent
YAML import syntax. Expose the corresponding choice for ordinary loading too.
This separates reusable composition policy from placement of particular sources
and keeps default placement unchanged.

**Alternative:** put composition registrations on load options, allowing different
composition rules for each invocation of one definition. That offers flexibility
but duplicates policy across ordinary/layered load forms. A dedicated item-input
constructor is also viable instead of a mode field, but needs equivalent import
syntax and a clear way to combine destination and source options.

### 2. Destination and registration paths

**Recommendation:** use literal mapping-key paths to identify destination
sequences and match composition registrations after placement. This fits the
existing destination vocabulary and the requested `prompts` use case. Preserve
existing destination validation; do not silently add new reserved names. Keep
empty/root item destinations invalid if they would replace the currently required
mapping document root with a sequence. These are proposed scope choices, not
already-approved new validation rules.

Allow unused registrations for optional configuration. Coalesce identical
registrations and reject conflicting identity rules for the same path. For
ancestor/descendant registrations, apply a rule only where its literal path
actually reaches a sequence; it does not implicitly traverse every record.

**Alternative:** support explicit sequence-index paths, including sequences inside
appended records, or sequence-valued configuration roots. These are expressive
extensions, but require decisions about source versus effective indexes and
root decoding. Nested-item destinations arising through imports also need the
context rule in decision 4; they must not accidentally introduce wildcard path
matching. Rejecting even identical duplicate registrations is another viable
option, but catches redundant setup at the cost of an additional restriction.

### 3. Item placement without identity registration

**Recommendation:** allow item placement independently. It contributes a singleton
sequence; without a registration, two present sequences still conflict during
strict composition, while precedence still replaces them. This keeps placement
orthogonal and allows a consumer to place a single item without inventing an
identity rule.

**Alternative:** require every item destination to have a composition registration.
This catches a likely setup omission early, but unnecessarily couples placement
to identity validation for consumers that only need one item. Implicitly appending
unregistered sequences would change the agreed opt-in composition boundary.

### 4. Imports declared inside an item-placed source

**Recommendation:** compose ordinary imports into that source's item mapping;
interpret their relative destinations inside the item. An explicit item-placement
import then contributes an item to a sequence within that context. A collection
manifest can import multiple peer prompt files as items at `prompts`. Do not
implicitly inherit item mode and turn every helper import into another prompt.
This permits helper fragments without requiring each helper to declare an ID.

This recommendation introduces an item-local composition context. Specify how
omitted and empty destinations resolve, how nested contexts are represented,
and how registrations address any nested destination sequences consistently
with decision 2 before accepting it.

**Alternative:** keep inherited destinations anchored at the enclosing collection
and require an explicit mode on each imported contribution. That can simplify
peer-item expansion, but ordinary helper mappings would collide with the
collection sequence rather than extend the declaring record. Inheriting item
mode as well would make peer imports concise, but needs an explicit way to
request helpers within one record. Do not reject nested imports merely to avoid
settling their interpretation.

### 5. Identity-validation boundary and structural processing

**Recommendation:** validate each contributed record after its source directives
and applicable version-header processing, and after any item-local mapping
composition chosen in decision 4, but before collection composition and later
precedence replacement. Ordinary helper imports can therefore supply fields of
one record; each completed record still needs its identity. Defaults and
application callbacks do not supply missing identities at this stage.

Keep existing `extends` semantics: an item itself has no mapping-sibling namespace
for record inheritance, while mappings inside an item may have one. Consumed
structural fields or headers do not become application identity values. Determine
whether a proposed identity-field selection overlaps such controls through the
existing processing rules rather than inventing a blanket field-name blacklist.

**Alternative:** require the directly authored item file to contain its own ID
before processing helpers. This simplifies early duplicate detection but adds a
restriction not needed for the requested assembled-record behavior. Validating
only the final configuration is not sufficient to catch the agreed strict-tier
conflicts that later replacements would hide.

### 6. Empty identities and literal field names

**Recommendation:** accept empty string identity values, with repeated empty
values conflicting. Interpret the designated direct field name literally,
including unusual or empty YAML mapping keys where existing processing permits
them. Presence and string type suffice for comparison; applications can impose
stronger naming rules through their own validation.

**Alternative:** require nonempty identity values or nonempty field names. This
can catch mistakes earlier but imposes additional domain conventions. Exact
comparison without trimming or normalization remains the agreed behavior.

### 7. Non-sequence values at registered paths

**Recommendation:** let registration select how actual sequences compose, without
asserting that every source must contain a sequence at that path. Missing values
and non-sequence values retain existing composition and decoding behavior.
Sequence/null and sequence/mapping overlaps still conflict; item placement
contributes an actual sequence and follows those same rules.

**Alternative:** reject every present non-null non-sequence source value at a
registered path, even if a higher tier would replace it. This catches misplaced
configuration earlier but makes registration an extra source-shape validator.
If selected, define null behavior explicitly rather than equating null with an
empty sequence.

### 8. Validation of replacement-only sequences

**Recommendation:** apply identity validation where source sequences participate
in strict composition, including singleton contributions, and preserve existing
validation for replacement-only selected overlays and synthetic CLI overrides.
Defaults and callbacks retain the agreed behavior. This keeps the new check tied
to its purpose and avoids broadening source validation merely for uniformity.
This revises the previous draft's recommendation to validate overlay identities.

**Alternative:** also require replacement-only source sequences to have valid,
unique identities. Authoring rules would be more uniform, but this would reject
some overlay configurations that do not need identity matching to replace a
list. It needs explicit acceptance; whole-sequence replacement does not imply it.

### 9. Source-version checks for item-placed main inputs

**Recommendation:** keep enabled source-header checks source-local before item
placement and preserve accepted header consumption/retention rules. Define the
main header's effective path, protected-path behavior, and reported evidence
using its actual item placement. Check ordinary and layered loading and all
source-version modes, including default-mode effective-path interpretation.
A sequence identity rule must not silently become a version policy.

**Alternative:** initially support item placement only for non-main contributions,
leaving the main document in its existing mapping placement model. This covers
a collection manifest importing prompt items but adds a source-role restriction
beyond the requested general input capability. Prefer resolving the main-path
semantics over imposing that restriction for implementation convenience.

### 10. Public placement evidence and diagnostics

**Recommendation:** retain `SourceInfo.Destination` as the effective
container destination and add the minimum placement information needed to
identify item mode. Use existing effective-path metadata for final indexes;
a source need not have one permanent item index after replacement. Diagnostics
should identify conflicting locations and paths without exposing identity
values before sensitivity is known. Reuse existing error categories where they
accurately describe the failure.

**Alternative:** report a resolved item path directly in source records, with
explicit semantics for replaced items, helper sources, and different historical
indexes. This is convenient for consumers but introduces a broader inspection
contract. A permanent identity-addressed source path is not needed for this use
case. Decide the public evidence shape without promising stable indexes across
loads or changes in contribution order.