Plan identity-based sequence composition

docs/roadmap/future.md

@@ -15,6 +15,11 @@ PromptKit repository at
 PromptKit selected mounted mappings instead; this feature is not a dependency
 of that design or a request to restore multiple installed prompt versions.
 
+A narrower [identity-based sequence composition roadmap](sequence-composition.md)
+now covers single-field identities, duplicate rejection during strict composition,
+and whole-sequence replacement between precedence layers. The additional
+capabilities below remain deferred and are not requirements of that roadmap.
+
 ### Motivation and proposed behavior
 
 Allow applications to declare that selected sequences contain records identified

docs/roadmap/sequence-composition.md

@@ -0,0 +1,288 @@
+# Identity-Based Sequence Composition
+
+Status: agreed feature direction; not implemented. Resolve the open decisions
+below before implementation, then execute the plan in order.
+
+## Goal and current baseline
+
+Allow consumers to opt selected sequences into strict composition by one explicit
+string identity field. Independent fragments can contribute distinct records to
+the same list; repeated identities fail composition. Keep records as sequences
+for typed decoding, inspection, and effective YAML.
+
+Current [strict composition](../config.md#strict-document-composition) recursively
+combines mappings and rejects every other overlap. Current
+[precedence assembly](../config.md#layered-loading) and
+[selected overlays](../config.md#selected-overlays) replace sequences wholesale.
+There is no sequence-composition registration in `Definition` today.
+
+This is a narrower feature than the
+[identified record sequences candidate](future.md#1-identified-record-sequences).
+That candidate's composite identities, recursive merging of matching records,
+identity-based overlays, and record inheritance are not part of this plan.
+
+## Agreed behavior
+
+### Registration and identities
+
+- Consumers explicitly designate each sequence path and one direct identity
+  field, such as `prompts` and `id`. Unregistered sequences retain current rules.
+- Records in a participating sequence must be mappings containing the designated
+  field as a string. Missing, null, and non-string identities are invalid;
+  identities are not coerced from numbers or supplied by defaults or resolution.
+- Compare strings exactly, without trimming, case folding, or normalization.
+  Whether the empty string is accepted is an open decision below.
+- Retain identity fields as ordinary record content in the resulting sequence.
+  They are not consumed as metadata or converted into public mapping keys.
+- This is a source-composition rule, not a guarantee of identity uniqueness after
+  application defaults and callbacks. Those retain their existing behavior;
+  applications can enforce final domain constraints through validation.
+
+### Strict composition
+
+For a registered `prompts` sequence with identity field `id`, two separate
+source fragments could contribute:
+
+```yaml
+prompts:
+  - id: prompt_one
+    text: "foo"
+```
+
+```yaml
+prompts:
+  - id: prompt_two
+    text: "bar"
+```
+
+The composed value is:
+
+```yaml
+prompts:
+  - id: prompt_one
+    text: "foo"
+  - id: prompt_two
+    text: "bar"
+```
+
+- Append distinct records in contribution order, preserving each source list's
+  order. Use existing deterministic input/import traversal within a tier;
+  identity values do not sort the result.
+- Reject repeated identities both across contributions and within a single
+  contributing sequence, including a sequence that has no peer contribution.
+  Equal bodies and disjoint bodies both conflict. Never recursively combine
+  records with the same identity.
+- An empty sequence contributes no records. Composing empty sequences succeeds;
+  an explicitly contributed empty sequence remains present, not absent.
+- Apply these rules to main inputs and their imports, and to inputs and imports
+  composing each underlay group. Honor input and relative-import destinations.
+  Source acquisition kind must not change composition behavior.
+- Invalid contributions and duplicate identities in a tier cannot be repaired
+  by a higher-precedence replacement. Report a normal load error and return no
+  usable partial result.
+
+The analogy with keyed mappings is limited: distinct keys combine, but mappings
+at a shared key currently recurse. A shared record identity always conflicts
+under this feature.
+
+### Precedence and other operations
+
+Sequence replacement remains the rule between underlay tiers, between underlays
+and the main tier, and for the selected overlay. An upper sequence replaces the
+complete lower sequence, even at a registered path; `[]` clears it. Identities
+may recur across those tiers without a cross-tier duplicate error. Null and
+cross-kind precedence replacements retain current semantics.
+
+Whole-sequence CLI overrides, defaults, application resolution, and built-in
+sibling inheritance retain their existing semantics. This feature adds no
+record inheritance, identity selector syntax, record update/delete/rename
+operators, or identity-based behavior to effective-configuration diffs. Existing
+atomic sequence diffs continue to compare final frozen values.
+
+### Inspection and diagnostics
+
+Appending records changes their effective indexes. Preserve the distinction
+between an original source position and its position in the assembled sequence.
+Effective presence, origins, resource contexts, history, and explanations must
+refer to the correct record and its original physical source locations.
+
+Composition history must describe the accumulated sequence correctly, rather
+than replay each fragment as a whole-sequence replacement. Later actual
+replacements must still retain displaced history. Equal numeric indexes in
+different source lists must not make unrelated records share provenance.
+
+Duplicate diagnostics must identify the conflicting source locations in
+deterministic order. Avoid including identity values in diagnostic text where
+sensitivity is not yet known; identify the sequence, identity field, and source
+locations instead. Apply existing sensitivity and disclosure rules to successful
+and failed inspection, including records whose effective indexes changed.
+Retain detached public records and input/result ownership guarantees.
+
+## Implementation plan
+
+### 1. Settle the remaining contract choices
+
+- Resolve the Open Decisions section and update this document with the choices.
+- Select the public registration names, path interpretation, and diagnostic codes.
+  Duplicate claims should use the existing composition-conflict category where
+  appropriate; malformed registration and malformed records need distinct,
+  actionable diagnostics.
+- Record the significant composition extension in an ADR during implementation,
+  under the architecture policy. No ADR is created by this roadmap-writing pass.
+
+Completion: the public behavior and validation boundary are unambiguous, with
+no accidental commitment to the broader identified-records proposal.
+
+### 2. Add registration and pass it through the loading paths
+
+- Add root-owned public registration types and fields using the selected API.
+  Convert them explicitly into private engine state; preserve the root facade
+  boundary and reusable definition ownership.
+- Validate registrations before source processing and pass the same policy
+  through ordinary and layered loading, import assembly, and mounted inputs.
+- Compile path lookup once per load if useful. No registrations must preserve
+  current behavior without introducing a new global cache or merge framework.
+
+Completion: public callers can select sequence paths consistently across load
+forms, and invalid options fail through the normal diagnostics boundary.
+
+### 3. Extend strict composition and its retained evidence together
+
+The current `composeDocuments` implementation in
+`internal/engine/compose.go` collects claims by path, rejects non-mapping
+overlaps, merges disjoint mappings, and retains origins and history applications.
+Changing only the final tree merge would leave these other representations wrong.
+
+- Extend claim validation and tree assembly for registered sequences. Validate
+  single contributions as well as overlaps. Track identities per sequence per
+  strict tier, and append cloned records in stable order. Keep duplicate lookup
+  proportional to record count rather than repeatedly scanning prior records.
+- Keep original source locations while assigning effective indexes. Carry the
+  necessary placement information into claims, origins, resource contexts, and
+  history; do not expose a new public identity-addressing model.
+- Update history/explanation replay to represent sequence accumulation. Ensure
+  snapshots, historical paths, sensitivity classification, and subsequent
+  whole-sequence replacement agree with the assembled tree.
+- Integrate with ordinary input assembly and grouped assembly in
+  `internal/engine/load_input.go` and `internal/engine/load_request.go`.
+  Keep tier boundaries explicit so uniqueness is never enforced across
+  precedence layers accidentally.
+- Preserve parser/source limits and retained-node accounting. Keep source-header
+  processing and structural-directive handling in their existing lifecycle
+  positions; do not restore consumed headers to supply record identities.
+
+Completion: final values and their inspection evidence agree across composition
+and subsequent replacement. No input mutation, source-context mix-up, or
+sensitive-value disclosure is introduced.
+
+### 4. Verify behavior at its owning boundaries
+
+- Composition tests: distinct and repeated identities, duplicates in one list,
+  malformed records and identity types, empty strings according to the chosen
+  decision, empty lists, ordering, missing paths, and null/cross-kind cases.
+- Loading tests: imports, underlay groups, mounted and relative destinations,
+  equivalent source forms, and conflicts hidden by later layers. Demonstrate
+  that separate tiers may reuse identities and still replace whole lists.
+- Metadata and inspection tests: records appended from the same source index
+  acquire distinct final indexes while keeping correct locations, history,
+  resource contexts, and sensitivity. Cover later replacement and historical
+  redaction, including a failure before final classification.
+- Public integration tests: registration conversion and reuse, normal load-error
+  behavior, unchanged unregistered sequences, selected-overlay and CLI
+  replacement, defaults/resolution behavior, and atomic effective diffs.
+- Add a small maintained consumer example showing two fragments composing and
+  a selected overlay replacing the result. Include only enough cases at each
+  layer to catch distinct failure modes.
+
+Run focused tests during development, then the everyday validation prescribed
+by the [development guide](../development.md#baseline-validation). Run ownership
+or concurrency checks where required by the change. No independent performance
+benchmark campaign is part of this plan; keep measurement separate from the
+busy-host environment discussed in the performance audit.
+
+Completion: new behavior is covered without weakening the existing atomic
+replacement and default-behavior regressions.
+
+### 5. Publish the implemented contract in repository documentation
+
+- Document the public API beside its declarations. Update the configuration
+  contract's composition, inspection/history, and precedence sections, and the
+  consumer guide and maintained example together when implementation lands.
+- Update the architecture policy and component overview only where the
+  implemented boundary changes. Keep the broader candidate in `future.md`
+  clearly distinct from this delivered subset.
+- Retire this completed roadmap through the usual documentation lifecycle.
+  Release preparation and publication remain separate work.
+
+Completion: current documentation describes the delivered feature, and no
+future proposal is presented as implemented behavior.
+
+## Open Decisions
+
+### 1. Public registration ownership
+
+**Recommendation:** place a collection of path/identity-field registrations on
+`Definition`, shared by ordinary and layered loading. This describes how the
+application's configuration is composed and avoids diverging registrations
+between load forms. Choose final type and field names before coding.
+
+**Alternative:** place registrations on load options. That allows one definition
+to be composed differently per invocation, but duplicates configuration across
+ordinary/layered options and makes composition semantics less visibly part of
+the application's reusable definition.
+
+### 2. Path matching, overlapping registrations, and unused rules
+
+**Recommendation:** interpret paths in effective configuration coordinates after
+input/import placement, using literal mapping-key segments initially. Reject
+duplicate registrations for the same path; allow a registration to match no
+source value. These rules fit mounted collections without requiring discovery
+of which optional sources will be present. Restricting paths to mapping segments
+would be an explicit initial scope choice, not an existing library constraint.
+
+**Alternative:** support exact sequence-index segments too, including registered
+sequences nested inside appended records. This is more expressive, but requires
+choosing whether a rule matches source indexes or composed indexes, and how
+matching behaves as earlier lists contribute records. Settle that interpretation
+and overlapping registration behavior before supporting these paths. Wildcard
+or identity-selector matching is not needed for the agreed use case.
+
+### 3. Empty identity strings
+
+**Recommendation:** accept the empty string as an exact identity; two empty
+identities conflict. Presence and string type are sufficient for the composition
+algorithm, and applications can require nonempty domain identifiers themselves.
+
+**Alternative:** require nonempty identities. This catches likely authoring
+mistakes earlier, but adds a domain restriction beyond uniqueness and string
+type. Whitespace trimming or normalization would still remain out of scope.
+
+### 4. Null and non-sequence values at registered paths
+
+**Recommendation:** treat registration as selecting how actual sequences
+compose, not as a general schema assertion. Preserve existing rules for absent,
+null, scalar, and mapping values: non-sequence overlaps retain ordinary strict
+composition behavior, and final decoding/validation checks application types.
+Two sequence values use the new rule; sequence/null and sequence/mapping overlaps
+remain conflicts. No later layer can hide an actual strict-composition conflict.
+
+**Alternative:** require every present non-null source value at the path to be a
+sequence, even when a later tier would replace it. This catches misplaced
+configuration earlier but makes registration an additional source-shape rule.
+If selected, define null handling explicitly rather than treating it as empty.
+
+### 5. Validation of replacement-only source sequences
+
+**Recommendation:** require well-formed, internally unique identities for actual
+source sequences at registered paths, including selected-overlay sequences.
+This makes source authoring consistent while preserving whole-sequence
+replacement and permitting identities to recur across tiers. Do not extend the
+rule to synthetic CLI replacements, defaults, or callback-generated values;
+final domain validation remains application-owned.
+
+**Alternative:** validate only source sequences participating in a strict tier,
+leaving selected-overlay sequences subject solely to existing decoding and
+application validation. This keeps the check strictly tied to the composition
+operation, but allows the same list to succeed as an overlay and fail as a base
+contribution. Choose this boundary explicitly; replacement semantics alone do
+not settle whether an overlay's own duplicate identities are invalid.