Settle sequence placement and composition roadmap decisions

docs/roadmap/sequence-composition.md

@@ -1,8 +1,8 @@
 # Sequence-Item Placement and Identity-Based Composition
 
-Status: agreed feature direction; not implemented. This roadmap defines the
-intended scope and remaining design choices. A separate staged implementation
-plan will be prepared after the roadmap is finalized.
+Status: design decisions settled; not implemented. This roadmap records the
+agreed scope and behavior for final review. A separate staged implementation
+plan will be prepared after that review.
 
 ## Goal and current baseline
 
@@ -77,15 +77,17 @@ registered sequence under the same composition rule.
   placement behavior; this does not expand accepted source-root shapes.
 - Item placement supplies structure, not identity values. Keep authored `id` and
   other fields in the item. Source directives and version headers retain their
-  own processing rules; their interactions with the new placement context must
-  be resolved as described under Open Decisions.
+  own processing rules, with the item-placement interpretation recorded under
+  Settled Decisions.
 - Preserve existing absence distinctions: `NoSource()` and an unmatched optional
   import contribute nothing. An actual empty mapping contributes an empty item,
   which fails the agreed identity requirement when that item participates in a
   registered composition sequence.
 
-Nested imports, placement without a composition registration, and the public
-representation of item placement remain decisions rather than implicit rules.
+Item placement is independent of composition registration. Ordinary imports
+inside an item compose into its mapping; public source evidence reports the
+placement mode and destination container. The settled decisions below define
+these behaviors in detail.
 
 ### Identity-based strict composition
 
@@ -94,7 +96,7 @@ representation of item placement remain decisions rather than implicit rules.
 - Each participating record is a mapping with the designated string field.
   Missing, null, and non-string identities are invalid. Compare exact strings
   without trimming, case folding, normalization, or numeric coercion. Empty
-  string handling remains open.
+  strings are valid identities; repeated empty identities conflict.
 - Append distinct records in contribution order, preserving each source list's
   order. Use existing deterministic traversal within a strict tier rather than
   sorting by identity.
@@ -125,8 +127,9 @@ into append semantics.
 Whole-sequence CLI overrides, defaults, application resolution, and existing
 sibling inheritance retain their behavior. This feature does not make records
 inherit from one another by identity. Effective-configuration diffs continue to
-compare final sequences atomically. Whether to validate identities in a
-replacement-only overlay source is an open decision, separate from replacement.
+compare final sequences atomically. Replacement-only selected overlays and
+synthetic CLI overrides do not acquire identity validation from composition
+registration; existing decoding and application validation still apply.
 
 ### Existing inspection and ownership obligations
 
@@ -144,7 +147,7 @@ contracts, not a new promise of persistent identity-based paths across loads.
 
 ## Work needed
 
-The staged plan will sequence these areas after the decisions below are settled:
+After final roadmap review, the staged plan will sequence these areas:
 
 - **Public API and import syntax:** introduce the selected composition
   registration and placement controls at the existing root-owned boundaries;
@@ -184,174 +187,133 @@ with their inspection evidence. Validation follows the
 [development guide](../development.md#baseline-validation). This documentation
 pass neither implements the feature nor runs performance benchmarks.
 
-## Open Decisions
+## Settled Decisions
 
-Recommendations below are proposals, not accepted restrictions. Resolve them
-before preparing the staged implementation plan. Exact public names and error
-code spellings can be selected with the API design; implementation details that
-do not affect observable behavior need not become additional contracts.
+The following choices are accepted for this feature. No substantive design
+choices remain open in this roadmap. Exact public names, YAML option names,
+and error-code spellings will be selected during API design in the staged plan.
+Internal representations remain implementation choices, subject to the existing
+contracts and the behavior recorded here.
 
 ### 1. Public controls and ownership
 
-**Recommendation:** put sequence path/identity-field registrations on `Definition`
-and an explicit placement mode beside input destination controls, with equivalent
-YAML import syntax. Expose the corresponding choice for ordinary loading too.
-This separates reusable composition policy from placement of particular sources
-and keeps default placement unchanged.
-
-**Alternative:** put composition registrations on load options, allowing different
-composition rules for each invocation of one definition. That offers flexibility
-but duplicates policy across ordinary/layered load forms. A dedicated item-input
-constructor is also viable instead of a mode field, but needs equivalent import
-syntax and a clear way to combine destination and source options.
+Put sequence path/identity-field registrations on `Definition`. Put an explicit
+placement mode beside input destination controls, with equivalent YAML import
+syntax and support in ordinary loading. This separates reusable composition
+policy from placement of individual sources. Default placement remains unchanged.
 
 ### 2. Destination and registration paths
 
-**Recommendation:** use literal mapping-key paths to identify destination
-sequences and match composition registrations after placement. This fits the
-existing destination vocabulary and the requested `prompts` use case. Preserve
-existing destination validation; do not silently add new reserved names. Keep
-empty/root item destinations invalid if they would replace the currently required
-mapping document root with a sequence. These are proposed scope choices, not
-already-approved new validation rules.
+Use literal mapping-key paths to identify destination sequences and match
+registrations at their effective location after placement. For example, a source
+mounted beneath `application` uses a registration at `application.prompts`.
+Preserve existing destination validation without adding new reserved names.
 
-Allow unused registrations for optional configuration. Coalesce identical
-registrations and reject conflicting identity rules for the same path. For
-ancestor/descendant registrations, apply a rule only where its literal path
-actually reaches a sequence; it does not implicitly traverse every record.
+Permit unused registrations, coalesce identical registrations, and reject
+conflicting identity fields for the same path. Ancestor/descendant registrations
+apply only where their literal paths reach sequences; they do not implicitly
+traverse every record.
 
-**Alternative:** support explicit sequence-index paths, including sequences inside
-appended records, or sequence-valued configuration roots. These are expressive
-extensions, but require decisions about source versus effective indexes and
-root decoding. Nested-item destinations arising through imports also need the
-context rule in decision 4; they must not accidentally introduce wildcard path
-matching. Rejecting even identical duplicate registrations is another viable
-option, but catches redundant setup at the cost of an additional restriction.
+Sequence-index selectors and wildcard matching are outside this initial feature.
+Item placement identifies a sequence beneath the configuration root; a root
+sequence would expand the existing mapping-root model and is not included.
+These are the selected feature boundaries, not permanent architectural exclusions.
+Nested placement follows decision 4 without expanding registration addressing.
 
 ### 3. Item placement without identity registration
 
-**Recommendation:** allow item placement independently. It contributes a singleton
-sequence; without a registration, two present sequences still conflict during
-strict composition, while precedence still replaces them. This keeps placement
-orthogonal and allows a consumer to place a single item without inventing an
-identity rule.
-
-**Alternative:** require every item destination to have a composition registration.
-This catches a likely setup omission early, but unnecessarily couples placement
-to identity validation for consumers that only need one item. Implicitly appending
-unregistered sequences would change the agreed opt-in composition boundary.
+Allow item placement independently. It contributes a singleton sequence without
+requiring an identity registration. Existing rules apply at unregistered paths:
+two present sequences conflict in one strict tier, and an upper-tier sequence
+replaces a lower one. Do not infer append semantics merely from item placement.
 
 ### 4. Imports declared inside an item-placed source
 
-**Recommendation:** compose ordinary imports into that source's item mapping;
-interpret their relative destinations inside the item. An explicit item-placement
-import then contributes an item to a sequence within that context. A collection
-manifest can import multiple peer prompt files as items at `prompts`. Do not
-implicitly inherit item mode and turn every helper import into another prompt.
-This permits helper fragments without requiring each helper to declare an ID.
+Compose ordinary imports into the declaring item's mapping. For an import
+declared at the item root, an omitted destination or `destination: []` means that
+item context; `[metadata]` places the helper inside the item's `metadata`.
+Further relative destinations continue from the immediately importing document's
+placement within that context. Physical import paths still resolve from the
+declaring file.
 
-This recommendation introduces an item-local composition context. Specify how
-omitted and empty destinations resolve, how nested contexts are represented,
-and how registrations address any nested destination sequences consistently
-with decision 2 before accepting it.
+Item mode is explicit on each import, not inherited automatically. A collection
+manifest explicitly imports peer prompt files as items at `prompts`. A helper
+import supplies fields of the declaring item and need not declare its own ID.
 
-**Alternative:** keep inherited destinations anchored at the enclosing collection
-and require an explicit mode on each imported contribution. That can simplify
-peer-item expansion, but ordinary helper mappings would collide with the
-collection sequence rather than extend the declaring record. Inheriting item
-mode as well would make peer imports concise, but needs an explicit way to
-request helpers within one record. Do not reject nested imports merely to avoid
-settling their interpretation.
+An explicitly item-placed import inside a prompt can contribute to a nested
+sequence. Registrations do not reach through outer sequence indexes in this
+initial feature, so those nested sequences retain atomic composition. Multiple
+contributions to such a nested sequence therefore follow ordinary strict
+conflict rules. This permits nested placement without introducing index selectors
+or wildcard matching.
 
 ### 5. Identity-validation boundary and structural processing
 
-**Recommendation:** validate each contributed record after its source directives
-and applicable version-header processing, and after any item-local mapping
-composition chosen in decision 4, but before collection composition and later
-precedence replacement. Ordinary helper imports can therefore supply fields of
-one record; each completed record still needs its identity. Defaults and
-application callbacks do not supply missing identities at this stage.
+Validate each completed record after applicable source-header processing and
+ordinary helper-import composition, before combining records in the collection
+and before later precedence replacement. A helper may supply the identity field;
+it need not be authored in the item file itself. Duplicate fields within the
+item still follow ordinary strict mapping composition. Defaults and application
+resolution cannot supply missing identities because they run later.
 
-Keep existing `extends` semantics: an item itself has no mapping-sibling namespace
-for record inheritance, while mappings inside an item may have one. Consumed
-structural fields or headers do not become application identity values. Determine
-whether a proposed identity-field selection overlaps such controls through the
-existing processing rules rather than inventing a blanket field-name blacklist.
-
-**Alternative:** require the directly authored item file to contain its own ID
-before processing helpers. This simplifies early duplicate detection but adds a
-restriction not needed for the requested assembled-record behavior. Validating
-only the final configuration is not sufficient to catch the agreed strict-tier
-conflicts that later replacements would hide.
+Preserve existing structural semantics. An item has no mapping-sibling namespace
+for record inheritance, while mappings inside it can use existing inheritance.
+Consumed structural fields and headers do not become application identity
+values. Existing processing rules govern overlap with those controls; no new
+identity-field-name blacklist is introduced.
 
 ### 6. Empty identities and literal field names
 
-**Recommendation:** accept empty string identity values, with repeated empty
-values conflicting. Interpret the designated direct field name literally,
-including unusual or empty YAML mapping keys where existing processing permits
-them. Presence and string type suffice for comparison; applications can impose
-stronger naming rules through their own validation.
-
-**Alternative:** require nonempty identity values or nonempty field names. This
-can catch mistakes earlier but imposes additional domain conventions. Exact
-comparison without trimming or normalization remains the agreed behavior.
+Accept empty string identities; repeated empty identities conflict. Interpret
+the designated direct field name literally, including unusual or empty YAML
+mapping keys where existing processing permits them. Presence and string type
+suffice for identity comparison. Nonempty-ID requirements, naming conventions,
+and whitespace restrictions remain application-owned validation choices.
 
 ### 7. Non-sequence values at registered paths
 
-**Recommendation:** let registration select how actual sequences compose, without
-asserting that every source must contain a sequence at that path. Missing values
-and non-sequence values retain existing composition and decoding behavior.
-Sequence/null and sequence/mapping overlaps still conflict; item placement
-contributes an actual sequence and follows those same rules.
-
-**Alternative:** reject every present non-null non-sequence source value at a
-registered path, even if a higher tier would replace it. This catches misplaced
-configuration earlier but makes registration an extra source-shape validator.
-If selected, define null behavior explicitly rather than equating null with an
-empty sequence.
+Registration selects how actual sequences compose; it is not an additional
+schema declaration. Missing and non-sequence values retain existing composition
+and decoding behavior. Sequence/null and sequence/mapping overlaps still
+conflict in strict composition. Item placement contributes an actual sequence
+and follows these rules. Registration does not independently reject every
+non-sequence source value that a later tier might replace.
 
 ### 8. Validation of replacement-only sequences
 
-**Recommendation:** apply identity validation where source sequences participate
-in strict composition, including singleton contributions, and preserve existing
-validation for replacement-only selected overlays and synthetic CLI overrides.
-Defaults and callbacks retain the agreed behavior. This keeps the new check tied
-to its purpose and avoids broadening source validation merely for uniformity.
-This revises the previous draft's recommendation to validate overlay identities.
+Apply identity validation to registered sequences participating in strict
+composition, including singleton contributions. Selected-overlay sequences and
+synthetic CLI replacements retain their existing validation. Defaults and
+application callbacks also retain their behavior.
 
-**Alternative:** also require replacement-only source sequences to have valid,
-unique identities. Authoring rules would be more uniform, but this would reject
-some overlay configurations that do not need identity matching to replace a
-list. It needs explicit acceptance; whole-sequence replacement does not imply it.
+An overlay can therefore replace a list with duplicate IDs unless the consuming
+application's validator rejects them. The identity rule governs composition,
+not final configuration validity.
 
 ### 9. Source-version checks for item-placed main inputs
 
-**Recommendation:** keep enabled source-header checks source-local before item
-placement and preserve accepted header consumption/retention rules. Define the
-main header's effective path, protected-path behavior, and reported evidence
-using its actual item placement. Check ordinary and layered loading and all
-source-version modes, including default-mode effective-path interpretation.
-A sequence identity rule must not silently become a version policy.
+Support item placement for main inputs as well as imported and underlay sources.
+Enabled source-version modes inspect declarations in source-local coordinates
+before placement. Default mode retains its existing effective-path interpretation.
+Accepted non-main headers are consumed; the main header remains application
+content, following existing rules.
 
-**Alternative:** initially support item placement only for non-main contributions,
-leaving the main document in its existing mapping placement model. This covers
-a collection manifest importing prompt items but adds a source-role restriction
-beyond the requested general input capability. Prefer resolving the main-path
-semantics over imposing that restriction for implementation convenience.
+The retained main header's effective location includes its actual item index.
+Existing ownership protection follows that location and must not accidentally
+protect another record because its source index differs from its effective
+index. Preserve source-local version evidence and diagnostics where the existing
+contract uses them. This feature introduces neither a new version policy nor
+an exception to existing version-path protections.
 
 ### 10. Public placement evidence and diagnostics
 
-**Recommendation:** retain `SourceInfo.Destination` as the effective
-container destination and add the minimum placement information needed to
-identify item mode. Use existing effective-path metadata for final indexes;
-a source need not have one permanent item index after replacement. Diagnostics
-should identify conflicting locations and paths without exposing identity
-values before sensitivity is known. Reuse existing error categories where they
-accurately describe the failure.
+Keep `SourceInfo.Destination` describing the effective destination container and
+expose the placement mode. Use existing effective-path metadata to describe
+resulting item paths, retaining physical source locations and resource contexts.
+Do not promise one permanent item index per source: replacement can remove its
+contribution, and contribution order can differ between loads.
 
-**Alternative:** report a resolved item path directly in source records, with
-explicit semantics for replaced items, helper sources, and different historical
-indexes. This is convenient for consumers but introduces a broader inspection
-contract. A permanent identity-addressed source path is not needed for this use
-case. Decide the public evidence shape without promising stable indexes across
-loads or changes in contribution order.
+Duplicate diagnostics identify conflicting locations and the sequence path
+without exposing potentially sensitive identity values. Reuse existing diagnostic
+categories where appropriate. Resolved historical item paths and persistent
+identity-addressed paths in source records are not part of this feature.