Record whole-record sequence replacement and application identity constraints

docs/roadmap/sequences.md

@@ -1,7 +1,8 @@
-# Configuration-Declared Sequence Identity And Layered Append
+# Sequence Identity And Cross-Tier Combination
 
 Status: draft proposal for discussion. This document captures the requested
-capabilities and initial recommendations; the open decisions are not settled.
+capabilities, settled design choices, and initial recommendations. Choices under
+Open Decisions remain unsettled.
 It is not an implementation plan or an implemented contract. All new syntax
 shown below is illustrative.
 
@@ -10,14 +11,14 @@ shown below is illustrative.
 Allow configuration authors to assemble collections from independently authored
 records, declare their identity in configuration, and progressively extend a
 baseline collection through higher layers. Authors should also be able to
-replace the accumulated collection explicitly.
+replace the accumulated collection or replace individual records by identity.
 
 The motivating example is a prompt collection whose defaults supply some
-records and whose application or installation layers add others. A repeated
+records and whose application or installation layers add or supersede records.
-identity should produce an error when records are combined. This proposal
+Across tiers, authors need three operations: whole-sequence replacement,
-concerns adding distinct records and replacing whole collections; matching
+duplicate-rejecting append, and whole-record replacement by identity. The latter
-records would not acquire field-by-field update semantics merely by adding
+retains unmatched lower-tier records and adds higher-tier records with new
-this feature.
+identities. Matching records are replaced in full.
 
 ## Current Baseline
 
@@ -43,9 +44,10 @@ later. Extending one does not automatically imply extending the other.
 
 The proposed import syntax introduces an identity declaration, tentatively
 named `key`, that accepts either one field or a list of fields for a compound
-identity. The original proposal is to move the application's sequence
+identity. Applications retain an optional interface for registering a collection
-registration into configuration. Whether to remove, retain, or adapt the Go
+and imposing its identity definition. Configuration declarations must conform
-registration remains open.
+to an application-owned constraint when one exists. The exact Go interface and
+configuration declaration site remain to be designed.
 
 ```yaml
 configkit:
@@ -55,7 +57,7 @@ configkit:
       destination: [prompts]
       placement: sequence-item
       key: [namespace, id]
-      append: true
+      mode: append
 ```
 
 Each selected file would contribute a record such as:
@@ -66,26 +68,71 @@ id: welcome
 text: Welcome
 ```
 
-Here `append: true` illustrates retaining a lower-layer collection and adding
+Here `mode: append` illustrates retaining a lower-layer collection and adding
-distinct records from the higher layer. The user also proposed `append: false`
+distinct records from the higher layer. A named operation is the current
-or an inverse `replace: true` choice for complete replacement. An explicit
+recommendation for expressing all three choices; field spelling, operation
-operation such as `mode: append` or `mode: replace` is another candidate;
+names, and default behavior remain open. The earlier `append` or inverse
-field spelling and default behavior remain open.
+`replace` boolean proposals would need additional syntax for the third operation.
 
 The intended results can be expressed without settling the syntax. Letters
-below denote record identities:
+below denote record identities; `B*` is a higher-tier record with the same
+identity as `B` and a different payload. The outcomes are settled; the ordering
+shown is the recommendation discussed below.
 
 | Lower-tier sequence | Higher-tier sequence | Operation | Proposed result |
 | --- | --- | --- | --- |
 | `[A, B]` | `[C]` | Append | `[A, B, C]` |
-| `[A, B]` | `[C]` | Replace | `[C]` |
+| `[A, B]` | `[C]` | Whole-sequence replacement | `[C]` |
-| `[A, B]` | `[B]` | Append | Duplicate-identity error |
+| `[A, B]` | `[B*]` | Append | Duplicate-identity error |
-| `[A, B]` | `[B]` | Replace | `[B]` |
+| `[A, B]` | `[B*]` | Whole-sequence replacement | `[B*]` |
+| `[A, B]` | `[B*]` | Replacement by identity | `[A, B*]` |
+| `[A, B]` | `[B*, C]` | Replacement by identity | `[A, B*, C]` |
 
-Replacement discards the lower sequence as a whole. A repeated identity in the
+Whole-sequence replacement discards the lower sequence. A repeated identity in the
 retained combined sequence is an error under the proposed append semantics,
 including when the records have identical or disjoint payloads.
 
+## Settled Design Choices
+
+### Whole-Record Replacement By Identity
+
+The third cross-tier operation replaces each matching lower record with the
+complete higher record, retains unmatched lower records, and adds new identities.
+For example, given these lower and higher records:
+
+```yaml
+# Lower record
+id: B
+text: Original
+temperature: 0.5
+```
+
+```yaml
+# Higher record
+id: B
+text: Revised
+```
+
+The resulting record at this precedence boundary contains `id` and `text`.
+The lower record's `temperature` is discarded. Subsequent defaulting or other
+lifecycle operations retain their separately defined semantics. Recursive
+field-by-field overlay of matching records is outside this selected operation.
+
+### Optional Application-Owned Identity Constraints
+
+Retain an interface through which applications may register a collection and
+impose its unique-key definition. If the application requires `key: id`, a
+configuration declaration of `key: [id, name]` is invalid: the compound key
+would permit repeated `id` values with different names. The application identity
+constraint applies whichever cross-tier operation configuration selects.
+
+This is an authoritative constraint, distinct from an application default that
+only fills an omission. Configuration can define identity for collections
+without an application-owned constraint. The recommendation is for omitted
+configuration identity to inherit a registered definition and for an explicit
+matching declaration to be accepted. Enforcement timing, including whether the
+constraint guarantees uniqueness in the final published result, remains OD-1.
+
 ## Initial Architectural Direction
 
 Treat identity as a property of the destination collection, even if an import
@@ -101,11 +148,13 @@ contribution at that destination. It would not sequentially erase contributions
 from peer imports. Multiple agreeing declarations could coalesce; conflicting
 explicit identity or operation declarations would produce errors.
 
-For append, retain lower-tier item order and then the higher tier's assembled
+For append, the ordering recommendation is to retain lower-tier item order and
-item order. Existing dependency-ready scheduling would continue to determine
+then the higher tier's assembled item order. For replacement by identity, retain
-assembly order within a tier. Replacement would retain only the higher tier's
+the position of each replaced record and append new identities in higher-tier
-assembled sequence. These are recommendations awaiting agreement, not changes
+order. Existing dependency-ready scheduling would continue to determine assembly
-to the current ordering contract.
+order within a tier. Whole-sequence replacement would retain only the higher
+tier's assembled sequence. Ordering and the treatment of peer declarations remain
+recommendations awaiting agreement, not changes to the current ordering contract.
 
 This requires collection policy to survive import expansion through precedence
 assembly. Correctness includes final values, projected item paths, source
@@ -118,11 +167,12 @@ The implementation must account for both through the existing loading lifecycle.
 This is narrower than the deferred
 [identified-record sequences proposal](future.md#1-identified-record-sequences),
 which also considers recursive merging of matching records and record-specific
-inheritance. Those capabilities remain separate candidates. Compound keys and
+inheritance. Those capabilities remain separate candidates. Compound keys,
-duplicate-rejecting append do not require adopting them.
+duplicate-rejecting append, and whole-record replacement by identity do not
+require adopting them.
 
-The proposal would revise the application-owned registration and cross-tier
+The proposal would extend identity declaration ownership and revise the
-replacement decisions in
+cross-tier replacement decision in
 [ADR 0014](../adr/0014-sequence-placement-and-composition.md). If adopted, record
 the architectural change in a superseding decision and reconcile affected
 architecture policy before implementation. The current ADR and contracts remain
@@ -136,23 +186,28 @@ design and test sequencing belong to that later plan.
 
 ## Open Decisions
 
-### OD-1. Who Owns Identity Policy?
+### OD-1. When Is An Application Identity Constraint Enforced?
 
-Should configuration replace `Definition.SequenceCompositions`, supplement it,
+Application authority over a registered collection's identity definition is
-or merely provide defaults for an application-owned policy?
+settled above. Does that registration govern assembly alone, or also guarantee
+valid, unique identities in the final published collection? The existing
+registration checks strict-composition contributions; later defaults,
+inheritance, overrides, and application callbacks have separate lifecycle roles.
 
-**Initial recommendation:** Let configuration own identity for collections
+**Initial recommendation:** Enforce an application-owned identity constraint
-assembled through the new mechanism, with applications retaining responsibility
+during participating collection assembly and validate it again on the final
-for their domain requirements. Resolve the treatment of existing Go registration
+effective collection before publication. This makes the application's constraint
-before removing it: applications may depend on a fixed identity definition even
+meaningful even when a later operation creates or changes records. It does not
-when configuration authors control source layout. Avoid two independent policy
+require applying append or replacement-by-identity semantics during those later
-systems whose precedence depends on acquisition order.
+operations, or validating every intermediate callback state. Whether YAML-only
+identity declarations also imply final validation remains to be decided.
 
-**Alternative:** Keep application registration authoritative and allow YAML to
+**Alternative:** Limit registration to structural assembly and leave final
-declare or confirm compatible policies. This preserves schema-owned identity
+uniqueness checks to application validation. This preserves the current phase
-constraints and supports programmatic use, but adds a reconciliation rule and
+boundary and reduces built-in validation work, but the registration alone would
-retains some application setup. YAML-only ownership is simpler for independently
+not establish a final-result invariant. Its public contract would need to state
-authored collections but moves enforcement of fixed domain identities elsewhere.
+that limitation clearly. Decide enforcement separately from the combination
+scope in OD-7.
 
 ### OD-2. Where Does Configuration Declare Collection Policy?
 
@@ -179,66 +234,80 @@ relationship to that declaration.
 
 Settle compound-key field selection, value types, missing/null components,
 empty and repeated field lists, empty identity strings, and agreement between
-declarations. Decide whether replacement can introduce a different identity
+declarations. Decide whether whole-sequence replacement can introduce a different identity
-definition after discarding the lower collection.
+definition after discarding the lower collection when the application has imposed
+no identity constraint. An application-owned constraint remains authoritative.
 
 **Initial recommendation:** Interpret `key: id` as `[id]` and a compound key as
 an ordered tuple of direct, literal string fields. Retain exact string comparison
 and require every component to be present. Preserve tuple boundaries rather than
-concatenating values. Identity declarations participating in an append must
+concatenating values. Identity declarations participating in append or replacement
-agree; equal definitions within a scope coalesce and conflicting definitions
+by identity must agree; equal definitions within a scope coalesce and conflicting
-fail. Detailed list validation and identity changes at replacement boundaries
+definitions fail. Detailed list validation and identity changes at whole-sequence
-remain to be settled.
+replacement boundaries remain to be settled.
 
 **Alternative:** Permit additional scalar types or nested field paths. These
 support more schemas but add type-equality and addressing rules. A load-wide fixed
 identity definition simplifies agreement across tiers; allowing a replacement to
 establish a new definition gives independently authored layers more flexibility.
 
-### OD-4. What Does An Import's Append Or Replace Choice Control?
+### OD-4. What Does An Import's Combination Choice Control?
 
 What happens when peer imports into the same destination request different
-operations, or when some contributions omit the operation?
+operations, or when some contributions omit the operation? Settle ordering for
+replacement by identity, including where replaced records and new identities
+appear.
 
 **Initial recommendation:** Assemble one strict sequence per destination in
 each tier, then apply its agreed operation at the precedence boundary. Reject
 conflicting explicit choices within a tier. This preserves peer composition and
 prevents declaration or scheduling order from granting replacement authority.
 Define how omitted choices inherit the effective collection operation once its
-declaration site is selected.
+declaration site is selected. Duplicate identities among peers would remain
+strict-composition errors under all three cross-tier operations. For replacement
+by identity, keep matched records in their lower-tier positions and append new
+identities in higher-tier order.
 
-**Alternative:** Treat each import as a sequential append-or-replace operation.
+**Alternative:** Treat each import as a sequential collection operation.
 This permits deliberate resets among imports, but a replacement can discard peer
 contributions and would require an explicitly selected operation-order model.
 Another alternative is to allow the choice only on a collection declaration,
 which reduces conflicts between import annotations but changes the proposed
 authoring surface.
 
+For ordering, moving replaced records to their higher-tier contribution positions
+is another viable choice. It gives the higher layer more influence over ordering,
+but changing a record's payload could also move it relative to retained records.
+
 ### OD-5. What Is The Default Operation And Its Syntax?
 
-Choose between `append`, an inverse `replace` option, or a named operation, and
+Choose syntax for all three operations and decide what omission means.
-decide what omission means.
 
-**Initial recommendation:** Keep replacement as the cross-tier default and
+**Initial recommendation:** Keep whole-sequence replacement as the cross-tier
-make append explicit. Lists can represent complete ordered instructions as well
+default and make append and replacement by identity explicit. Lists can represent
-as extensible catalogs; an explicit append communicates the intent to retain
+complete ordered instructions as well as extensible catalogs; an explicit append
-lower-layer records. Prefer a positive named operation if both modes need equal
+communicates the intent to retain
-prominence, while considering the simplicity of `append: true`.
+lower-layer records. Prefer one named operation with three values to interacting
+booleans; choose names that distinguish whole-sequence from whole-record
+replacement.
 
-**Alternative:** Default identified collections to append and require explicit
+**Alternative:** Default identified collections to append or replacement by
-replacement. This makes progressive collections concise, but retains lower-layer
+identity and require explicit whole-sequence replacement. These defaults make
-records unless an author deliberately clears them. An inverse `replace` boolean
+progressive collections concise, but retain lower-layer records unless an author
-expresses that model compactly; a named operation makes both choices explicit.
+deliberately replaces the collection. Extending the boolean proposal with a
+separate duplicate-handling option is possible, but introduces interacting
+settings whose combinations need explanation and validation.
 
 ### OD-6. What Do Absence, Empty Sequences, And Null Mean?
 
 Can a declaration selecting nothing affect a lower collection? How do explicit
-empty sequences, null, and non-sequence values interact with append or replace?
+empty sequences, null, and non-sequence values interact with all three operations?
 
 **Initial recommendation:** An optional import selecting nothing contributes
 nothing and does not clear lower values. An explicitly supplied empty sequence
-with replacement can clear a collection; append of an empty sequence adds no
+with whole-sequence replacement can clear a collection; append or replacement
-records. Preserve the distinction between absence and supplied values. Decide
+by identity with an empty sequence contributes no records. Preserve the
+distinction between absence and supplied values. Decide
 null and kind-mismatch handling explicitly rather than silently treating them
 as empty sequences.
 
@@ -249,22 +318,28 @@ as an explicit value avoids that coupling.
 
 ### OD-7. Which Lifecycle Stages Participate?
 
-Does append extend only source-backed tiers, or also `Definition.Defaults`,
+Do the new cross-tier operations extend only source-backed tiers, or also
-inheritance, CLI overrides, and application-created records? At which boundaries
+`Definition.Defaults`, inheritance, CLI overrides, and application-created records?
+At which boundaries
 must identity be validated, including helper-supplied fields and consumed source
 version headers?
 
 **Initial recommendation:** Start with source-backed strict composition and
 precedence assembly. Validate records at the relevant collection assembly
-boundaries and uniqueness across each retained append result. Preserve existing
+boundaries and uniqueness across each combined result. Preserve existing
 typed-default and application-resolution semantics unless their inclusion is
 explicitly chosen. Decide selected-overlay coverage with OD-2; it is part of
 source precedence but currently has no import declaration site. Account for
 item-local helpers and source-version consumption before deciding identity is
 available, consistent with the current lifecycle.
 
-**Alternative:** Apply collection policy throughout defaulting, inheritance,
+OD-1 separately addresses final validation of application constraints. Validating
-overrides, and application resolution, including final identity checks. This
+the final collection does not itself change how defaults, inheritance, CLI
-would provide a broader final-collection guarantee, but requires additional
+overrides, or callbacks combine values.
-combination, callback, and inspection rules. Supporting typed default lists is
+
-a separate scope choice from extending source-backed baseline collections.
+**Alternative:** Extend the combination operations through defaulting,
+inheritance, overrides, and application resolution. This would support
+progressively extending collections across more of the lifecycle, but requires
+additional combination, callback, and inspection rules. Supporting typed default
+lists is a separate scope choice from extending source-backed baseline
+collections; final identity enforcement remains the separate OD-1 decision.