From b58cd75c9855d5a7d5816888f4e78197a11ac8b0 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 20 Oct 2026 13:48:54 +0000 Subject: [PATCH] Draft sequence identity and layered append proposal --- docs/roadmap/sequences.md | 270 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 270 insertions(+) create mode 100644 docs/roadmap/sequences.md diff --git a/docs/roadmap/sequences.md b/docs/roadmap/sequences.md new file mode 100644 index 0000000..104366b --- /dev/null +++ b/docs/roadmap/sequences.md @@ -0,0 +1,270 @@ +# Configuration-Declared Sequence Identity And Layered Append + +Status: draft proposal for discussion. This document captures the requested +capabilities and initial recommendations; the open decisions are not settled. +It is not an implementation plan or an implemented contract. All new syntax +shown below is illustrative. + +## Motivation + +Allow configuration authors to assemble collections from independently authored +records, declare their identity in configuration, and progressively extend a +baseline collection through higher layers. Authors should also be able to +replace the accumulated collection explicitly. + +The motivating example is a prompt collection whose defaults supply some +records and whose application or installation layers add others. A repeated +identity should produce an error when records are combined. This proposal +concerns adding distinct records and replacing whole collections; matching +records would not acquire field-by-field update semantics merely by adding +this feature. + +## Current Baseline + +The [import contract](../config.md#explicit-imports) owns current selector, +destination, placement, and ordering behavior. An import with +`placement: sequence-item` places each selected source mapping as one item at +its effective destination. Item-local helper imports can contribute fields +before placement. + +The application currently registers sequence identity through +`Definition.SequenceCompositions`, using one literal string field at each +registered mapping path. Within a strict tier, distinct identities append and +duplicate identities fail. Repeated unregistered sequences conflict. Across +precedence tiers, a higher sequence replaces the lower sequence wholesale. +The [composition contract](../config.md#strict-document-composition) owns these +rules; [layered loading](../config.md#layered-loading) owns tier precedence. + +Source-backed underlays and `Definition.Defaults` have different lifecycle +roles. Underlays participate in source assembly; typed defaults are applied +later. Extending one does not automatically imply extending the other. + +## Requested Capabilities + +The proposed import syntax introduces an identity declaration, tentatively +named `key`, that accepts either one field or a list of fields for a compound +identity. The original proposal is to move the application's sequence +registration into configuration. Whether to remove, retain, or adapt the Go +registration remains open. + +```yaml +configkit: + imports: + - dir: prompts + pattern: '*.yaml' + destination: [prompts] + placement: sequence-item + key: [namespace, id] + append: true +``` + +Each selected file would contribute a record such as: + +```yaml +namespace: support +id: welcome +text: Welcome +``` + +Here `append: true` illustrates retaining a lower-layer collection and adding +distinct records from the higher layer. The user also proposed `append: false` +or an inverse `replace: true` choice for complete replacement. An explicit +operation such as `mode: append` or `mode: replace` is another candidate; +field spelling and default behavior remain open. + +The intended results can be expressed without settling the syntax. Letters +below denote record identities: + +| Lower-tier sequence | Higher-tier sequence | Operation | Proposed result | +| --- | --- | --- | --- | +| `[A, B]` | `[C]` | Append | `[A, B, C]` | +| `[A, B]` | `[C]` | Replace | `[C]` | +| `[A, B]` | `[B]` | Append | Duplicate-identity error | +| `[A, B]` | `[B]` | Replace | `[B]` | + +Replacement discards the lower sequence as a whole. A repeated identity in the +retained combined sequence is an error under the proposed append semantics, +including when the records have identical or disjoint payloads. + +## Initial Architectural Direction + +Treat identity as a property of the destination collection, even if an import +introduces its declaration. That property needs to cover both imported items +and records authored directly at the same destination. Discovering collection +declarations before composing the relevant scope would make their application +independent of which contribution happens to be assembled first. + +The initial recommendation is to compose each tier's contributions strictly, +then apply one collection operation from the higher tier against the lower +sequence. An individual import would therefore express policy for its tier's +contribution at that destination. It would not sequentially erase contributions +from peer imports. Multiple agreeing declarations could coalesce; conflicting +explicit identity or operation declarations would produce errors. + +For append, retain lower-tier item order and then the higher tier's assembled +item order. Existing dependency-ready scheduling would continue to determine +assembly order within a tier. Replacement would retain only the higher tier's +assembled sequence. These are recommendations awaiting agreement, not changes +to the current ordering contract. + +This requires collection policy to survive import expansion through precedence +assembly. Correctness includes final values, projected item paths, source +destinations, provenance, history, sensitivity, and passive resource contexts. +Appending changes final indexes; replacing changes which contributions survive. +The implementation must account for both through the existing loading lifecycle. + +## Scope And Decision Ownership + +This is narrower than the deferred +[identified-record sequences proposal](future.md#1-identified-record-sequences), +which also considers recursive merging of matching records and record-specific +inheritance. Those capabilities remain separate candidates. Compound keys and +duplicate-rejecting append do not require adopting them. + +The proposal would revise the application-owned registration and cross-tier +replacement decisions in +[ADR 0014](../adr/0014-sequence-placement-and-composition.md). If adopted, record +the architectural change in a superseding decision and reconcile affected +architecture policy before implementation. The current ADR and contracts remain +authoritative until that decision is made. This draft neither changes them nor +selects migration machinery. + +A staged implementation plan should follow resolution of the questions below. +It should cover ordinary and layered loading, item-local scopes, source-version +processing, identity validation, precedence, and inspection together. Public API +design and test sequencing belong to that later plan. + +## Open Decisions + +### OD-1. Who Owns Identity Policy? + +Should configuration replace `Definition.SequenceCompositions`, supplement it, +or merely provide defaults for an application-owned policy? + +**Initial recommendation:** Let configuration own identity for collections +assembled through the new mechanism, with applications retaining responsibility +for their domain requirements. Resolve the treatment of existing Go registration +before removing it: applications may depend on a fixed identity definition even +when configuration authors control source layout. Avoid two independent policy +systems whose precedence depends on acquisition order. + +**Alternative:** Keep application registration authoritative and allow YAML to +declare or confirm compatible policies. This preserves schema-owned identity +constraints and supports programmatic use, but adds a reconciliation rule and +retains some application setup. YAML-only ownership is simpler for independently +authored collections but moves enforcement of fixed domain identities elsewhere. + +### OD-2. Where Does Configuration Declare Collection Policy? + +Is `key` on a `sequence-item` import sufficient, or should configuration have a +collection declaration that also covers inline sequences, ordinary mapping +imports containing sequences, and selected overlays? How does such a declaration +address a collection inside an imported item? + +**Initial recommendation:** Define one destination-scoped collection policy +model first. Evaluate a configuration-level declaration alongside import syntax +against these workflows before selecting the surface. Collection identity must +apply consistently to all participating records at that destination, including +inline records. Resolve relative destinations and item-local scopes using their +existing ownership boundaries. + +**Alternative:** Limit this feature to policies declared on item imports. This +keeps the initial syntax small and directly serves imported prompt collections, +but leaves inline-only collections and overlay-authored append without a natural +declaration site. A separate collection declaration covers those cases at the +cost of more configuration structure; an import shorthand would require a clear +relationship to that declaration. + +### OD-3. What Constitutes Identity, And When Can It Change? + +Settle compound-key field selection, value types, missing/null components, +empty and repeated field lists, empty identity strings, and agreement between +declarations. Decide whether replacement can introduce a different identity +definition after discarding the lower collection. + +**Initial recommendation:** Interpret `key: id` as `[id]` and a compound key as +an ordered tuple of direct, literal string fields. Retain exact string comparison +and require every component to be present. Preserve tuple boundaries rather than +concatenating values. Identity declarations participating in an append must +agree; equal definitions within a scope coalesce and conflicting definitions +fail. Detailed list validation and identity changes at replacement boundaries +remain to be settled. + +**Alternative:** Permit additional scalar types or nested field paths. These +support more schemas but add type-equality and addressing rules. A load-wide fixed +identity definition simplifies agreement across tiers; allowing a replacement to +establish a new definition gives independently authored layers more flexibility. + +### OD-4. What Does An Import's Append Or Replace Choice Control? + +What happens when peer imports into the same destination request different +operations, or when some contributions omit the operation? + +**Initial recommendation:** Assemble one strict sequence per destination in +each tier, then apply its agreed operation at the precedence boundary. Reject +conflicting explicit choices within a tier. This preserves peer composition and +prevents declaration or scheduling order from granting replacement authority. +Define how omitted choices inherit the effective collection operation once its +declaration site is selected. + +**Alternative:** Treat each import as a sequential append-or-replace operation. +This permits deliberate resets among imports, but a replacement can discard peer +contributions and would require an explicitly selected operation-order model. +Another alternative is to allow the choice only on a collection declaration, +which reduces conflicts between import annotations but changes the proposed +authoring surface. + +### OD-5. What Is The Default Operation And Its Syntax? + +Choose between `append`, an inverse `replace` option, or a named operation, and +decide what omission means. + +**Initial recommendation:** Keep replacement as the cross-tier default and +make append explicit. Lists can represent complete ordered instructions as well +as extensible catalogs; an explicit append communicates the intent to retain +lower-layer records. Prefer a positive named operation if both modes need equal +prominence, while considering the simplicity of `append: true`. + +**Alternative:** Default identified collections to append and require explicit +replacement. This makes progressive collections concise, but retains lower-layer +records unless an author deliberately clears them. An inverse `replace` boolean +expresses that model compactly; a named operation makes both choices explicit. + +### OD-6. What Do Absence, Empty Sequences, And Null Mean? + +Can a declaration selecting nothing affect a lower collection? How do explicit +empty sequences, null, and non-sequence values interact with append or replace? + +**Initial recommendation:** An optional import selecting nothing contributes +nothing and does not clear lower values. An explicitly supplied empty sequence +with replacement can clear a collection; append of an empty sequence adds no +records. Preserve the distinction between absence and supplied values. Decide +null and kind-mismatch handling explicitly rather than silently treating them +as empty sequences. + +**Alternative:** Treat an explicitly declared replacement operation as a reset +even when its import selects nothing. This provides a compact clearing mechanism, +but makes an optional missing file or empty search destructive. Keeping clearing +as an explicit value avoids that coupling. + +### OD-7. Which Lifecycle Stages Participate? + +Does append extend only source-backed tiers, or also `Definition.Defaults`, +inheritance, CLI overrides, and application-created records? At which boundaries +must identity be validated, including helper-supplied fields and consumed source +version headers? + +**Initial recommendation:** Start with source-backed strict composition and +precedence assembly. Validate records at the relevant collection assembly +boundaries and uniqueness across each retained append result. Preserve existing +typed-default and application-resolution semantics unless their inclusion is +explicitly chosen. Decide selected-overlay coverage with OD-2; it is part of +source precedence but currently has no import declaration site. Account for +item-local helpers and source-version consumption before deciding identity is +available, consistent with the current lifecycle. + +**Alternative:** Apply collection policy throughout defaulting, inheritance, +overrides, and application resolution, including final identity checks. This +would provide a broader final-collection guarantee, but requires additional +combination, callback, and inspection rules. Supporting typed default lists is +a separate scope choice from extending source-backed baseline collections.