Settle sequence placement and composition roadmap decisions

docs/roadmap/sequence-composition.md

@@ -1,8 +1,8 @@
 # Sequence-Item Placement and Identity-Based Composition
 
-Status: agreed feature direction; not implemented. This roadmap defines the
+Status: design decisions settled; not implemented. This roadmap records the
-intended scope and remaining design choices. A separate staged implementation
+agreed scope and behavior for final review. A separate staged implementation
-plan will be prepared after the roadmap is finalized.
+plan will be prepared after that review.
 
 ## Goal and current baseline
 
@@ -77,15 +77,17 @@ registered sequence under the same composition rule.
   placement behavior; this does not expand accepted source-root shapes.
 - Item placement supplies structure, not identity values. Keep authored `id` and
   other fields in the item. Source directives and version headers retain their
-  own processing rules; their interactions with the new placement context must
+  own processing rules, with the item-placement interpretation recorded under
-  be resolved as described under Open Decisions.
+  Settled Decisions.
 - Preserve existing absence distinctions: `NoSource()` and an unmatched optional
   import contribute nothing. An actual empty mapping contributes an empty item,
   which fails the agreed identity requirement when that item participates in a
   registered composition sequence.
 
-Nested imports, placement without a composition registration, and the public
+Item placement is independent of composition registration. Ordinary imports
-representation of item placement remain decisions rather than implicit rules.
+inside an item compose into its mapping; public source evidence reports the
+placement mode and destination container. The settled decisions below define
+these behaviors in detail.
 
 ### Identity-based strict composition
 
@@ -94,7 +96,7 @@ representation of item placement remain decisions rather than implicit rules.
 - Each participating record is a mapping with the designated string field.
   Missing, null, and non-string identities are invalid. Compare exact strings
   without trimming, case folding, normalization, or numeric coercion. Empty
-  string handling remains open.
+  strings are valid identities; repeated empty identities conflict.
 - Append distinct records in contribution order, preserving each source list's
   order. Use existing deterministic traversal within a strict tier rather than
   sorting by identity.
@@ -125,8 +127,9 @@ into append semantics.
 Whole-sequence CLI overrides, defaults, application resolution, and existing
 sibling inheritance retain their behavior. This feature does not make records
 inherit from one another by identity. Effective-configuration diffs continue to
-compare final sequences atomically. Whether to validate identities in a
+compare final sequences atomically. Replacement-only selected overlays and
-replacement-only overlay source is an open decision, separate from replacement.
+synthetic CLI overrides do not acquire identity validation from composition
+registration; existing decoding and application validation still apply.
 
 ### Existing inspection and ownership obligations
 
@@ -144,7 +147,7 @@ contracts, not a new promise of persistent identity-based paths across loads.
 
 ## Work needed
 
-The staged plan will sequence these areas after the decisions below are settled:
+After final roadmap review, the staged plan will sequence these areas:
 
 - **Public API and import syntax:** introduce the selected composition
   registration and placement controls at the existing root-owned boundaries;
@@ -184,174 +187,133 @@ with their inspection evidence. Validation follows the
 [development guide](../development.md#baseline-validation). This documentation
 pass neither implements the feature nor runs performance benchmarks.
 
-## Open Decisions
+## Settled Decisions
 
-Recommendations below are proposals, not accepted restrictions. Resolve them
+The following choices are accepted for this feature. No substantive design
-before preparing the staged implementation plan. Exact public names and error
+choices remain open in this roadmap. Exact public names, YAML option names,
-code spellings can be selected with the API design; implementation details that
+and error-code spellings will be selected during API design in the staged plan.
-do not affect observable behavior need not become additional contracts.
+Internal representations remain implementation choices, subject to the existing
+contracts and the behavior recorded here.
 
 ### 1. Public controls and ownership
 
-**Recommendation:** put sequence path/identity-field registrations on `Definition`
+Put sequence path/identity-field registrations on `Definition`. Put an explicit
-and an explicit placement mode beside input destination controls, with equivalent
+placement mode beside input destination controls, with equivalent YAML import
-YAML import syntax. Expose the corresponding choice for ordinary loading too.
+syntax and support in ordinary loading. This separates reusable composition
-This separates reusable composition policy from placement of particular sources
+policy from placement of individual sources. Default placement remains unchanged.
-and keeps default placement unchanged.
-
-**Alternative:** put composition registrations on load options, allowing different
-composition rules for each invocation of one definition. That offers flexibility
-but duplicates policy across ordinary/layered load forms. A dedicated item-input
-constructor is also viable instead of a mode field, but needs equivalent import
-syntax and a clear way to combine destination and source options.
 
 ### 2. Destination and registration paths
 
-**Recommendation:** use literal mapping-key paths to identify destination
+Use literal mapping-key paths to identify destination sequences and match
-sequences and match composition registrations after placement. This fits the
+registrations at their effective location after placement. For example, a source
-existing destination vocabulary and the requested `prompts` use case. Preserve
+mounted beneath `application` uses a registration at `application.prompts`.
-existing destination validation; do not silently add new reserved names. Keep
+Preserve existing destination validation without adding new reserved names.
-empty/root item destinations invalid if they would replace the currently required
-mapping document root with a sequence. These are proposed scope choices, not
-already-approved new validation rules.
 
-Allow unused registrations for optional configuration. Coalesce identical
+Permit unused registrations, coalesce identical registrations, and reject
-registrations and reject conflicting identity rules for the same path. For
+conflicting identity fields for the same path. Ancestor/descendant registrations
-ancestor/descendant registrations, apply a rule only where its literal path
+apply only where their literal paths reach sequences; they do not implicitly
-actually reaches a sequence; it does not implicitly traverse every record.
+traverse every record.
 
-**Alternative:** support explicit sequence-index paths, including sequences inside
+Sequence-index selectors and wildcard matching are outside this initial feature.
-appended records, or sequence-valued configuration roots. These are expressive
+Item placement identifies a sequence beneath the configuration root; a root
-extensions, but require decisions about source versus effective indexes and
+sequence would expand the existing mapping-root model and is not included.
-root decoding. Nested-item destinations arising through imports also need the
+These are the selected feature boundaries, not permanent architectural exclusions.
-context rule in decision 4; they must not accidentally introduce wildcard path
+Nested placement follows decision 4 without expanding registration addressing.
-matching. Rejecting even identical duplicate registrations is another viable
-option, but catches redundant setup at the cost of an additional restriction.
 
 ### 3. Item placement without identity registration
 
-**Recommendation:** allow item placement independently. It contributes a singleton
+Allow item placement independently. It contributes a singleton sequence without
-sequence; without a registration, two present sequences still conflict during
+requiring an identity registration. Existing rules apply at unregistered paths:
-strict composition, while precedence still replaces them. This keeps placement
+two present sequences conflict in one strict tier, and an upper-tier sequence
-orthogonal and allows a consumer to place a single item without inventing an
+replaces a lower one. Do not infer append semantics merely from item placement.
-identity rule.
-
-**Alternative:** require every item destination to have a composition registration.
-This catches a likely setup omission early, but unnecessarily couples placement
-to identity validation for consumers that only need one item. Implicitly appending
-unregistered sequences would change the agreed opt-in composition boundary.
 
 ### 4. Imports declared inside an item-placed source
 
-**Recommendation:** compose ordinary imports into that source's item mapping;
+Compose ordinary imports into the declaring item's mapping. For an import
-interpret their relative destinations inside the item. An explicit item-placement
+declared at the item root, an omitted destination or `destination: []` means that
-import then contributes an item to a sequence within that context. A collection
+item context; `[metadata]` places the helper inside the item's `metadata`.
-manifest can import multiple peer prompt files as items at `prompts`. Do not
+Further relative destinations continue from the immediately importing document's
-implicitly inherit item mode and turn every helper import into another prompt.
+placement within that context. Physical import paths still resolve from the
-This permits helper fragments without requiring each helper to declare an ID.
+declaring file.
 
-This recommendation introduces an item-local composition context. Specify how
+Item mode is explicit on each import, not inherited automatically. A collection
-omitted and empty destinations resolve, how nested contexts are represented,
+manifest explicitly imports peer prompt files as items at `prompts`. A helper
-and how registrations address any nested destination sequences consistently
+import supplies fields of the declaring item and need not declare its own ID.
-with decision 2 before accepting it.
 
-**Alternative:** keep inherited destinations anchored at the enclosing collection
+An explicitly item-placed import inside a prompt can contribute to a nested
-and require an explicit mode on each imported contribution. That can simplify
+sequence. Registrations do not reach through outer sequence indexes in this
-peer-item expansion, but ordinary helper mappings would collide with the
+initial feature, so those nested sequences retain atomic composition. Multiple
-collection sequence rather than extend the declaring record. Inheriting item
+contributions to such a nested sequence therefore follow ordinary strict
-mode as well would make peer imports concise, but needs an explicit way to
+conflict rules. This permits nested placement without introducing index selectors
-request helpers within one record. Do not reject nested imports merely to avoid
+or wildcard matching.
-settling their interpretation.
 
 ### 5. Identity-validation boundary and structural processing
 
-**Recommendation:** validate each contributed record after its source directives
+Validate each completed record after applicable source-header processing and
-and applicable version-header processing, and after any item-local mapping
+ordinary helper-import composition, before combining records in the collection
-composition chosen in decision 4, but before collection composition and later
+and before later precedence replacement. A helper may supply the identity field;
-precedence replacement. Ordinary helper imports can therefore supply fields of
+it need not be authored in the item file itself. Duplicate fields within the
-one record; each completed record still needs its identity. Defaults and
+item still follow ordinary strict mapping composition. Defaults and application
-application callbacks do not supply missing identities at this stage.
+resolution cannot supply missing identities because they run later.
 
-Keep existing `extends` semantics: an item itself has no mapping-sibling namespace
+Preserve existing structural semantics. An item has no mapping-sibling namespace
-for record inheritance, while mappings inside an item may have one. Consumed
+for record inheritance, while mappings inside it can use existing inheritance.
-structural fields or headers do not become application identity values. Determine
+Consumed structural fields and headers do not become application identity
-whether a proposed identity-field selection overlaps such controls through the
+values. Existing processing rules govern overlap with those controls; no new
-existing processing rules rather than inventing a blanket field-name blacklist.
+identity-field-name blacklist is introduced.
-
-**Alternative:** require the directly authored item file to contain its own ID
-before processing helpers. This simplifies early duplicate detection but adds a
-restriction not needed for the requested assembled-record behavior. Validating
-only the final configuration is not sufficient to catch the agreed strict-tier
-conflicts that later replacements would hide.
 
 ### 6. Empty identities and literal field names
 
-**Recommendation:** accept empty string identity values, with repeated empty
+Accept empty string identities; repeated empty identities conflict. Interpret
-values conflicting. Interpret the designated direct field name literally,
+the designated direct field name literally, including unusual or empty YAML
-including unusual or empty YAML mapping keys where existing processing permits
+mapping keys where existing processing permits them. Presence and string type
-them. Presence and string type suffice for comparison; applications can impose
+suffice for identity comparison. Nonempty-ID requirements, naming conventions,
-stronger naming rules through their own validation.
+and whitespace restrictions remain application-owned validation choices.
-
-**Alternative:** require nonempty identity values or nonempty field names. This
-can catch mistakes earlier but imposes additional domain conventions. Exact
-comparison without trimming or normalization remains the agreed behavior.
 
 ### 7. Non-sequence values at registered paths
 
-**Recommendation:** let registration select how actual sequences compose, without
+Registration selects how actual sequences compose; it is not an additional
-asserting that every source must contain a sequence at that path. Missing values
+schema declaration. Missing and non-sequence values retain existing composition
-and non-sequence values retain existing composition and decoding behavior.
+and decoding behavior. Sequence/null and sequence/mapping overlaps still
-Sequence/null and sequence/mapping overlaps still conflict; item placement
+conflict in strict composition. Item placement contributes an actual sequence
-contributes an actual sequence and follows those same rules.
+and follows these rules. Registration does not independently reject every
-
+non-sequence source value that a later tier might replace.
-**Alternative:** reject every present non-null non-sequence source value at a
-registered path, even if a higher tier would replace it. This catches misplaced
-configuration earlier but makes registration an extra source-shape validator.
-If selected, define null behavior explicitly rather than equating null with an
-empty sequence.
 
 ### 8. Validation of replacement-only sequences
 
-**Recommendation:** apply identity validation where source sequences participate
+Apply identity validation to registered sequences participating in strict
-in strict composition, including singleton contributions, and preserve existing
+composition, including singleton contributions. Selected-overlay sequences and
-validation for replacement-only selected overlays and synthetic CLI overrides.
+synthetic CLI replacements retain their existing validation. Defaults and
-Defaults and callbacks retain the agreed behavior. This keeps the new check tied
+application callbacks also retain their behavior.
-to its purpose and avoids broadening source validation merely for uniformity.
-This revises the previous draft's recommendation to validate overlay identities.
 
-**Alternative:** also require replacement-only source sequences to have valid,
+An overlay can therefore replace a list with duplicate IDs unless the consuming
-unique identities. Authoring rules would be more uniform, but this would reject
+application's validator rejects them. The identity rule governs composition,
-some overlay configurations that do not need identity matching to replace a
+not final configuration validity.
-list. It needs explicit acceptance; whole-sequence replacement does not imply it.
 
 ### 9. Source-version checks for item-placed main inputs
 
-**Recommendation:** keep enabled source-header checks source-local before item
+Support item placement for main inputs as well as imported and underlay sources.
-placement and preserve accepted header consumption/retention rules. Define the
+Enabled source-version modes inspect declarations in source-local coordinates
-main header's effective path, protected-path behavior, and reported evidence
+before placement. Default mode retains its existing effective-path interpretation.
-using its actual item placement. Check ordinary and layered loading and all
+Accepted non-main headers are consumed; the main header remains application
-source-version modes, including default-mode effective-path interpretation.
+content, following existing rules.
-A sequence identity rule must not silently become a version policy.
 
-**Alternative:** initially support item placement only for non-main contributions,
+The retained main header's effective location includes its actual item index.
-leaving the main document in its existing mapping placement model. This covers
+Existing ownership protection follows that location and must not accidentally
-a collection manifest importing prompt items but adds a source-role restriction
+protect another record because its source index differs from its effective
-beyond the requested general input capability. Prefer resolving the main-path
+index. Preserve source-local version evidence and diagnostics where the existing
-semantics over imposing that restriction for implementation convenience.
+contract uses them. This feature introduces neither a new version policy nor
+an exception to existing version-path protections.
 
 ### 10. Public placement evidence and diagnostics
 
-**Recommendation:** retain `SourceInfo.Destination` as the effective
+Keep `SourceInfo.Destination` describing the effective destination container and
-container destination and add the minimum placement information needed to
+expose the placement mode. Use existing effective-path metadata to describe
-identify item mode. Use existing effective-path metadata for final indexes;
+resulting item paths, retaining physical source locations and resource contexts.
-a source need not have one permanent item index after replacement. Diagnostics
+Do not promise one permanent item index per source: replacement can remove its
-should identify conflicting locations and paths without exposing identity
+contribution, and contribution order can differ between loads.
-values before sensitivity is known. Reuse existing error categories where they
-accurately describe the failure.
 
-**Alternative:** report a resolved item path directly in source records, with
+Duplicate diagnostics identify conflicting locations and the sequence path
-explicit semantics for replaced items, helper sources, and different historical
+without exposing potentially sensitive identity values. Reuse existing diagnostic
-indexes. This is convenient for consumers but introduces a broader inspection
+categories where appropriate. Resolved historical item paths and persistent
-contract. A permanent identity-addressed source path is not needed for this use
+identity-addressed paths in source records are not part of this feature.
-case. Decide the public evidence shape without promising stable indexes across
-loads or changes in contribution order.