Settle collection declaration syntax in the sequence roadmap

docs/roadmap/sequences.md

@@ -3,8 +3,9 @@
 Status: draft proposal for discussion. This document captures the requested
 capabilities, settled design choices, and initial recommendations. Choices under
 Open Decisions remain unsettled.
-It is not an implementation plan or an implemented contract. All new syntax
+It is not an implementation plan or an implemented contract. The declaration
-shown below is illustrative.
+structure is settled under OD-2; operation names in examples remain illustrative
+pending OD-5.
 
 ## Motivation
 
@@ -42,22 +43,24 @@ later. Extending one does not automatically imply extending the other.
 
 ## Requested Capabilities
 
-The proposed import syntax introduces an identity declaration, tentatively
+The proposed `configkit.sequences` block introduces a `key` declaration that
-named `key`, that accepts either one field or a list of fields for a compound
+accepts either one field or a list of fields for a compound identity.
-identity. Applications retain an optional interface for registering a collection
+Applications retain an optional interface for registering a collection
 and imposing its identity definition. Configuration declarations must conform
-to an application-owned constraint when one exists. The exact Go interface and
+to an application-owned constraint when one exists. The exact Go interface
-configuration declaration site remain to be designed.
+remains to be designed.
 
 ```yaml
 configkit:
+  sequences:
+    - destination: [prompts]
+      key: [namespace, id]
+      mode: append
   imports:
     - dir: prompts
       pattern: '*.yaml'
       destination: [prompts]
       placement: sequence-item
-      key: [namespace, id]
-      mode: append
 ```
 
 Each selected file would contribute a record such as:
@@ -69,12 +72,11 @@ text: Welcome
 ```
 
 Here `mode: append` illustrates retaining a lower-layer collection and adding
-distinct records from the higher layer. A named operation is the current
+distinct records from the higher layer. The `mode` field expresses the operation;
-recommendation for expressing all three choices; field spelling, operation
+operation names and default behavior remain under OD-5. Identity and combination
-names, and default behavior remain open. The earlier `append` or inverse
+controls belong to the collection declaration; import shorthand is deferred.
-`replace` boolean proposals would need additional syntax for the third operation.
 
-The intended results can be expressed without settling the syntax. Letters
+The intended results can be expressed without settling operation names. Letters
 below denote record identities; `B*` is a higher-tier record with the same
 identity as `B` and a different payload. The outcomes are settled; the ordering
 shown is the recommendation discussed below.
@@ -94,6 +96,95 @@ including when the records have identical or disjoint payloads.
 
 ## Settled Design Choices
 
+### Collection Declarations
+
+**OD-2: settled.**
+
+Use a root `configkit.sequences` block containing a sequence of collection
+declarations. Each entry identifies a `destination` and may declare `key`,
+`mode`, or both. Application sequences remain ordinary YAML data. This declaration
+form serves imported items, inline records, and sequences supplied by ordinary
+mapping imports, including inline sequences in selected overlays.
+
+`destination` uses the existing sequence of literal mapping keys. Resolve it
+relative to the declaring source's placement: a fragment mounted under
+`catalog` can declare `[prompts]` locally. A sequence-item source can declare a
+nested collection within its own item without referring to an eventual numeric
+index. Preserve those source and item-local scope boundaries.
+
+Identity and mode can be supplied independently. Omitted identity can use an
+application registration; an explicit matching declaration is accepted, and a
+conflicting declaration is invalid. Whole-sequence replacement does not itself
+require identity matching, but application-owned constraints still apply.
+Identity-policy lifetime across tiers remains under OD-3.
+
+Mode describes the current tier's combination instruction. A lower tier's
+`append` does not silently select append for a later overlay. Gather collection
+declarations before composing their relevant scope so their effect does not
+depend on which contribution is assembled first. Agreement and omission among
+peer declarations remain under OD-4.
+
+Permit sequence declarations in selected overlays and make their use independent
+of filesystem import enablement. This deliberately changes the current blanket
+prohibition on overlay `configkit` declarations. Enabling imports in overlays
+is not part of this decision. Consume sequence controls before application
+decoding, as with other structural directives.
+
+For example, an overlay can replace a complete record by identity:
+
+```yaml
+configkit:
+  sequences:
+    - destination: [prompts]
+      key: id
+      mode: replace-by-key
+
+prompts:
+  - id: foo
+    text: Revised definition
+```
+
+An overlay can instead add a distinct record:
+
+```yaml
+configkit:
+  sequences:
+    - destination: [prompts]
+      key: id
+      mode: append
+
+prompts:
+  - id: baz
+    text: Another prompt
+```
+
+Or it can explicitly select whole-sequence replacement:
+
+```yaml
+configkit:
+  sequences:
+    - destination: [prompts]
+      mode: replace
+
+prompts:
+  - id: bar
+    text: Bar definition
+  - id: baz
+    text: Baz definition
+```
+
+Explicitly selecting the default operation is valid. Under the recommended
+whole-sequence replacement default, removing the last example's `configkit`
+block produces the same result. An author may keep it to make intent explicit.
+The names `replace`, `append`, and `replace-by-key` illustrate the three
+operations; exact names and the default remain under OD-5.
+
+Start with this common collection declaration and defer `key` and `mode`
+shorthand on imports. A single declaration form covers inline overlays and mixed
+inline/imported collections without duplicate authoring surfaces. Local wrappers
+around sequence values would require distinguishing application data from
+control mappings; the existing reserved namespace keeps that distinction clear.
+
 ### Whole-Record Replacement By Identity
 
 The third cross-tier operation replaces each matching lower record with the
@@ -136,12 +227,10 @@ record. Append retains its duplicate-identity error semantics. The retained
 records and replacement behavior are settled; ordering shown follows the
 recommendation under OD-4.
 
-An import-only declaration mechanism cannot cover inline overlay sequences.
+The settled OD-2 collection declaration supplies the overlay authoring surface.
-OD-2 must provide an overlay-authorable declaration site, and OD-5 must select
+OD-5 retains operation naming and default selection. The current
-operation names and omission behavior. The current
+[selected-overlay contract](../config.md#selected-overlays) remains the baseline
-[selected-overlay contract](../config.md#selected-overlays) prohibits `configkit`
+until the new controls are implemented.
-in overlays. Any new control syntax must explicitly account for that boundary;
-supporting sequence controls does not itself require enabling overlay imports.
 
 ### Optional Application-Owned Identity Constraints
 
@@ -153,9 +242,9 @@ constraint applies whichever cross-tier operation configuration selects.
 
 This is an authoritative constraint, distinct from an application default that
 only fills an omission. Configuration can define identity for collections
-without an application-owned constraint. The recommendation is for omitted
+without an application-owned constraint. Omitted configuration identity uses a
-configuration identity to inherit a registered definition and for an explicit
+registered definition when present, and an explicit matching declaration is
-matching declaration to be accepted. The final-result guarantee for
+accepted, as settled in OD-2. The final-result guarantee for
 application-owned constraints is settled below; YAML-only enforcement remains
 an open follow-up to OD-1.
 
@@ -189,18 +278,16 @@ Final enforcement for identities declared only in YAML is not settled here.
 
 ## Initial Architectural Direction
 
-Treat identity as a property of the destination collection, even if an import
+The settled collection declaration gives imported and inline records at the same
-introduces its declaration. That property needs to cover both imported items
+destination a shared identity policy. Declarations are gathered before composing
-and records authored directly at the same destination. Discovering collection
+the relevant scope.
-declarations before composing the relevant scope would make their application
-independent of which contribution happens to be assembled first.
 
 The initial recommendation is to compose each tier's contributions strictly,
 then apply one collection operation from the higher tier against the lower
-sequence. An individual import would therefore express policy for its tier's
+sequence. A collection declaration would therefore govern its tier's contribution
-contribution at that destination. It would not sequentially erase contributions
+at that destination, without sequentially erasing peer contributions. Multiple
-from peer imports. Multiple agreeing declarations could coalesce; conflicting
+agreeing declarations could coalesce; conflicting explicit identity or operation
-explicit identity or operation declarations would produce errors.
+declarations would produce errors.
 
 For append, the ordering recommendation is to retain lower-tier item order and
 then the higher tier's assembled item order. For replacement by identity, retain
@@ -246,9 +333,9 @@ Application-registered collections have the settled final-result guarantee
 above. Should a configuration-only identity declaration carry the same
 guarantee, or govern structural assembly alone?
 
-**Initial recommendation:** Settle this after choosing declaration scope in
+**Initial recommendation:** With declaration scope settled in OD-2, resolve
-OD-2 and identity-policy lifetime across replacement in OD-3. Final enforcement
+identity-policy lifetime across replacement in OD-3 before this choice. Final
-needs an unambiguous effective policy for each retained collection. The
+enforcement needs an unambiguous effective policy for each retained collection. The
 application-owned guarantee can proceed independently of this choice.
 
 **Alternatives:** Applying the same final check to YAML-declared identities
@@ -257,30 +344,6 @@ which policy survives replacement or follows nested records. Limiting YAML-only
 identity to assembly keeps policy lifetime narrower, but later operations could
 produce a final collection that no longer satisfies that identity declaration.
 
-### OD-2. Where Does Configuration Declare Collection Policy?
-
-Where should configuration declare collection identity and combination choices
-for imported items, inline sequences, and sequences supplied by ordinary mapping
-imports? Selected overlays must support author selection of all three operations
-for inline sequences. How does a declaration address a collection inside an
-imported item?
-
-**Initial recommendation:** Define one destination-scoped collection policy
-model first. Prefer a configuration-level collection declaration usable in base
-sources and selected overlays, with import annotations considered as shorthand.
-This gives inline and imported records a common policy surface. Collection
-identity must apply consistently to all participating records at that
-destination. Resolve relative destinations and item-local scopes using their
-existing ownership boundaries. Define the sequence controls accepted in overlays
-explicitly, including their relationship to the current `configkit` prohibition.
-
-**Alternative:** Put controls beside each sequence through a local wrapper or
-directive form. This keeps the operation close to the affected value and supports
-inline overlay records, but requires distinguishing control syntax from
-application data and defining how imported contributions share that policy.
-A configuration-level declaration adds path references and configuration
-structure; import shorthand also needs a clear relationship to that declaration.
-
 ### OD-3. What Constitutes Identity, And When Can It Change?
 
 Settle compound-key field selection, value types, missing/null components,
@@ -304,7 +367,7 @@ establish a new definition gives independently authored layers more flexibility.
 
 ### OD-4. What Does A Combination Choice Control?
 
-What happens when peer imports or collection declarations at the same destination
+What happens when peer collection declarations at the same destination
 request different operations, or when some contributions omit the operation?
 Settle ordering for replacement by identity, including where replaced records
 and new identities appear. Apply the same operation semantics to selected
@@ -314,18 +377,17 @@ overlay contributions.
 each tier, then apply its agreed operation at the precedence boundary. Reject
 conflicting explicit choices within a tier. This preserves peer composition and
 prevents declaration or scheduling order from granting replacement authority.
-Define how omitted choices inherit the effective collection operation once its
+Define how omitted choices use the effective operation within the current tier;
-declaration site is selected. Duplicate identities among peers would remain
+OD-2 settles that a lower tier's mode is not inherited by a later tier.
-strict-composition errors under all three cross-tier operations. For replacement
+Duplicate identities among peers would remain strict-composition errors under
+all three cross-tier operations. For replacement
 by identity, keep matched records in their lower-tier positions and append new
 identities in higher-tier order.
 
-**Alternative:** Treat each import as a sequential collection operation.
+**Alternative:** Apply collection declarations as sequential operations on their
-This permits deliberate resets among imports, but a replacement can discard peer
+associated source contributions. This permits deliberate resets within a tier,
-contributions and would require an explicitly selected operation-order model.
+but a replacement can discard peer contributions and would require an explicitly
-Another alternative is to allow the choice only on a collection declaration,
+selected operation-order model.
-which reduces conflicts between import annotations but changes the proposed
-authoring surface.
 
 For ordering, moving replaced records to their higher-tier contribution positions
 is another viable choice. It gives the higher layer more influence over ordering,
@@ -333,22 +395,23 @@ but changing a record's payload could also move it relative to retained records.
 
 ### OD-5. What Is The Default Operation And Its Syntax?
 
-Choose syntax for all three operations and decide what omission means.
+Choose names for the three `mode` values and decide the default operation.
+OD-2 settles the declaration structure and accepts explicit selection of the
+default. Omission among peer declarations remains under OD-4.
 
 **Initial recommendation:** Keep whole-sequence replacement as the cross-tier
 default and make append and replacement by identity explicit. Lists can represent
 complete ordered instructions as well as extensible catalogs; an explicit append
 communicates the intent to retain
-lower-layer records. Prefer one named operation with three values to interacting
+lower-layer records. Use names that distinguish whole-sequence from whole-record
-booleans; choose names that distinguish whole-sequence from whole-record
+replacement; the examples use `replace`, `append`, and `replace-by-key`.
-replacement.
 
 **Alternative:** Default identified collections to append or replacement by
 identity and require explicit whole-sequence replacement. These defaults make
 progressive collections concise, but retain lower-layer records unless an author
-deliberately replaces the collection. Extending the boolean proposal with a
+deliberately replaces the collection. For naming, `upsert` is a concise alternative
-separate duplicate-handling option is possible, but introduces interacting
+to `replace-by-key`, but requires explaining its whole-record replacement behavior
-settings whose combinations need explanation and validation.
+to authors unfamiliar with that term.
 
 ### OD-6. What Do Absence, Empty Sequences, And Null Mean?
 
@@ -381,7 +444,7 @@ precedence assembly. Validate records at the relevant collection assembly
 boundaries and uniqueness across each combined result. Preserve existing
 typed-default and application-resolution semantics unless their inclusion is
 explicitly chosen. Selected-overlay combination is included in the settled scope;
-its declaration mechanism remains under OD-2. Account for item-local helpers and
+its declaration mechanism is settled in OD-2. Account for item-local helpers and
 source-version consumption before deciding identity is available, consistent
 with the current lifecycle.