Draft sequence identity and layered append proposal

docs/roadmap/sequences.md

@@ -0,0 +1,270 @@
+# Configuration-Declared Sequence Identity And Layered Append
+
+Status: draft proposal for discussion. This document captures the requested
+capabilities and initial recommendations; the open decisions are not settled.
+It is not an implementation plan or an implemented contract. All new syntax
+shown below is illustrative.
+
+## Motivation
+
+Allow configuration authors to assemble collections from independently authored
+records, declare their identity in configuration, and progressively extend a
+baseline collection through higher layers. Authors should also be able to
+replace the accumulated collection explicitly.
+
+The motivating example is a prompt collection whose defaults supply some
+records and whose application or installation layers add others. A repeated
+identity should produce an error when records are combined. This proposal
+concerns adding distinct records and replacing whole collections; matching
+records would not acquire field-by-field update semantics merely by adding
+this feature.
+
+## Current Baseline
+
+The [import contract](../config.md#explicit-imports) owns current selector,
+destination, placement, and ordering behavior. An import with
+`placement: sequence-item` places each selected source mapping as one item at
+its effective destination. Item-local helper imports can contribute fields
+before placement.
+
+The application currently registers sequence identity through
+`Definition.SequenceCompositions`, using one literal string field at each
+registered mapping path. Within a strict tier, distinct identities append and
+duplicate identities fail. Repeated unregistered sequences conflict. Across
+precedence tiers, a higher sequence replaces the lower sequence wholesale.
+The [composition contract](../config.md#strict-document-composition) owns these
+rules; [layered loading](../config.md#layered-loading) owns tier precedence.
+
+Source-backed underlays and `Definition.Defaults` have different lifecycle
+roles. Underlays participate in source assembly; typed defaults are applied
+later. Extending one does not automatically imply extending the other.
+
+## Requested Capabilities
+
+The proposed import syntax introduces an identity declaration, tentatively
+named `key`, that accepts either one field or a list of fields for a compound
+identity. The original proposal is to move the application's sequence
+registration into configuration. Whether to remove, retain, or adapt the Go
+registration remains open.
+
+```yaml
+configkit:
+  imports:
+    - dir: prompts
+      pattern: '*.yaml'
+      destination: [prompts]
+      placement: sequence-item
+      key: [namespace, id]
+      append: true
+```
+
+Each selected file would contribute a record such as:
+
+```yaml
+namespace: support
+id: welcome
+text: Welcome
+```
+
+Here `append: true` illustrates retaining a lower-layer collection and adding
+distinct records from the higher layer. The user also proposed `append: false`
+or an inverse `replace: true` choice for complete replacement. An explicit
+operation such as `mode: append` or `mode: replace` is another candidate;
+field spelling and default behavior remain open.
+
+The intended results can be expressed without settling the syntax. Letters
+below denote record identities:
+
+| Lower-tier sequence | Higher-tier sequence | Operation | Proposed result |
+| --- | --- | --- | --- |
+| `[A, B]` | `[C]` | Append | `[A, B, C]` |
+| `[A, B]` | `[C]` | Replace | `[C]` |
+| `[A, B]` | `[B]` | Append | Duplicate-identity error |
+| `[A, B]` | `[B]` | Replace | `[B]` |
+
+Replacement discards the lower sequence as a whole. A repeated identity in the
+retained combined sequence is an error under the proposed append semantics,
+including when the records have identical or disjoint payloads.
+
+## Initial Architectural Direction
+
+Treat identity as a property of the destination collection, even if an import
+introduces its declaration. That property needs to cover both imported items
+and records authored directly at the same destination. Discovering collection
+declarations before composing the relevant scope would make their application
+independent of which contribution happens to be assembled first.
+
+The initial recommendation is to compose each tier's contributions strictly,
+then apply one collection operation from the higher tier against the lower
+sequence. An individual import would therefore express policy for its tier's
+contribution at that destination. It would not sequentially erase contributions
+from peer imports. Multiple agreeing declarations could coalesce; conflicting
+explicit identity or operation declarations would produce errors.
+
+For append, retain lower-tier item order and then the higher tier's assembled
+item order. Existing dependency-ready scheduling would continue to determine
+assembly order within a tier. Replacement would retain only the higher tier's
+assembled sequence. These are recommendations awaiting agreement, not changes
+to the current ordering contract.
+
+This requires collection policy to survive import expansion through precedence
+assembly. Correctness includes final values, projected item paths, source
+destinations, provenance, history, sensitivity, and passive resource contexts.
+Appending changes final indexes; replacing changes which contributions survive.
+The implementation must account for both through the existing loading lifecycle.
+
+## Scope And Decision Ownership
+
+This is narrower than the deferred
+[identified-record sequences proposal](future.md#1-identified-record-sequences),
+which also considers recursive merging of matching records and record-specific
+inheritance. Those capabilities remain separate candidates. Compound keys and
+duplicate-rejecting append do not require adopting them.
+
+The proposal would revise the application-owned registration and cross-tier
+replacement decisions in
+[ADR 0014](../adr/0014-sequence-placement-and-composition.md). If adopted, record
+the architectural change in a superseding decision and reconcile affected
+architecture policy before implementation. The current ADR and contracts remain
+authoritative until that decision is made. This draft neither changes them nor
+selects migration machinery.
+
+A staged implementation plan should follow resolution of the questions below.
+It should cover ordinary and layered loading, item-local scopes, source-version
+processing, identity validation, precedence, and inspection together. Public API
+design and test sequencing belong to that later plan.
+
+## Open Decisions
+
+### OD-1. Who Owns Identity Policy?
+
+Should configuration replace `Definition.SequenceCompositions`, supplement it,
+or merely provide defaults for an application-owned policy?
+
+**Initial recommendation:** Let configuration own identity for collections
+assembled through the new mechanism, with applications retaining responsibility
+for their domain requirements. Resolve the treatment of existing Go registration
+before removing it: applications may depend on a fixed identity definition even
+when configuration authors control source layout. Avoid two independent policy
+systems whose precedence depends on acquisition order.
+
+**Alternative:** Keep application registration authoritative and allow YAML to
+declare or confirm compatible policies. This preserves schema-owned identity
+constraints and supports programmatic use, but adds a reconciliation rule and
+retains some application setup. YAML-only ownership is simpler for independently
+authored collections but moves enforcement of fixed domain identities elsewhere.
+
+### OD-2. Where Does Configuration Declare Collection Policy?
+
+Is `key` on a `sequence-item` import sufficient, or should configuration have a
+collection declaration that also covers inline sequences, ordinary mapping
+imports containing sequences, and selected overlays? How does such a declaration
+address a collection inside an imported item?
+
+**Initial recommendation:** Define one destination-scoped collection policy
+model first. Evaluate a configuration-level declaration alongside import syntax
+against these workflows before selecting the surface. Collection identity must
+apply consistently to all participating records at that destination, including
+inline records. Resolve relative destinations and item-local scopes using their
+existing ownership boundaries.
+
+**Alternative:** Limit this feature to policies declared on item imports. This
+keeps the initial syntax small and directly serves imported prompt collections,
+but leaves inline-only collections and overlay-authored append without a natural
+declaration site. A separate collection declaration covers those cases at the
+cost of more configuration structure; an import shorthand would require a clear
+relationship to that declaration.
+
+### OD-3. What Constitutes Identity, And When Can It Change?
+
+Settle compound-key field selection, value types, missing/null components,
+empty and repeated field lists, empty identity strings, and agreement between
+declarations. Decide whether replacement can introduce a different identity
+definition after discarding the lower collection.
+
+**Initial recommendation:** Interpret `key: id` as `[id]` and a compound key as
+an ordered tuple of direct, literal string fields. Retain exact string comparison
+and require every component to be present. Preserve tuple boundaries rather than
+concatenating values. Identity declarations participating in an append must
+agree; equal definitions within a scope coalesce and conflicting definitions
+fail. Detailed list validation and identity changes at replacement boundaries
+remain to be settled.
+
+**Alternative:** Permit additional scalar types or nested field paths. These
+support more schemas but add type-equality and addressing rules. A load-wide fixed
+identity definition simplifies agreement across tiers; allowing a replacement to
+establish a new definition gives independently authored layers more flexibility.
+
+### OD-4. What Does An Import's Append Or Replace Choice Control?
+
+What happens when peer imports into the same destination request different
+operations, or when some contributions omit the operation?
+
+**Initial recommendation:** Assemble one strict sequence per destination in
+each tier, then apply its agreed operation at the precedence boundary. Reject
+conflicting explicit choices within a tier. This preserves peer composition and
+prevents declaration or scheduling order from granting replacement authority.
+Define how omitted choices inherit the effective collection operation once its
+declaration site is selected.
+
+**Alternative:** Treat each import as a sequential append-or-replace operation.
+This permits deliberate resets among imports, but a replacement can discard peer
+contributions and would require an explicitly selected operation-order model.
+Another alternative is to allow the choice only on a collection declaration,
+which reduces conflicts between import annotations but changes the proposed
+authoring surface.
+
+### OD-5. What Is The Default Operation And Its Syntax?
+
+Choose between `append`, an inverse `replace` option, or a named operation, and
+decide what omission means.
+
+**Initial recommendation:** Keep replacement as the cross-tier default and
+make append explicit. Lists can represent complete ordered instructions as well
+as extensible catalogs; an explicit append communicates the intent to retain
+lower-layer records. Prefer a positive named operation if both modes need equal
+prominence, while considering the simplicity of `append: true`.
+
+**Alternative:** Default identified collections to append and require explicit
+replacement. This makes progressive collections concise, but retains lower-layer
+records unless an author deliberately clears them. An inverse `replace` boolean
+expresses that model compactly; a named operation makes both choices explicit.
+
+### OD-6. What Do Absence, Empty Sequences, And Null Mean?
+
+Can a declaration selecting nothing affect a lower collection? How do explicit
+empty sequences, null, and non-sequence values interact with append or replace?
+
+**Initial recommendation:** An optional import selecting nothing contributes
+nothing and does not clear lower values. An explicitly supplied empty sequence
+with replacement can clear a collection; append of an empty sequence adds no
+records. Preserve the distinction between absence and supplied values. Decide
+null and kind-mismatch handling explicitly rather than silently treating them
+as empty sequences.
+
+**Alternative:** Treat an explicitly declared replacement operation as a reset
+even when its import selects nothing. This provides a compact clearing mechanism,
+but makes an optional missing file or empty search destructive. Keeping clearing
+as an explicit value avoids that coupling.
+
+### OD-7. Which Lifecycle Stages Participate?
+
+Does append extend only source-backed tiers, or also `Definition.Defaults`,
+inheritance, CLI overrides, and application-created records? At which boundaries
+must identity be validated, including helper-supplied fields and consumed source
+version headers?
+
+**Initial recommendation:** Start with source-backed strict composition and
+precedence assembly. Validate records at the relevant collection assembly
+boundaries and uniqueness across each retained append result. Preserve existing
+typed-default and application-resolution semantics unless their inclusion is
+explicitly chosen. Decide selected-overlay coverage with OD-2; it is part of
+source precedence but currently has no import declaration site. Account for
+item-local helpers and source-version consumption before deciding identity is
+available, consistent with the current lifecycle.
+
+**Alternative:** Apply collection policy throughout defaulting, inheritance,
+overrides, and application resolution, including final identity checks. This
+would provide a broader final-collection guarantee, but requires additional
+combination, callback, and inspection rules. Supporting typed default lists is
+a separate scope choice from extending source-backed baseline collections.