diff --git a/docs/roadmap/sequences.md b/docs/roadmap/sequences.md index 104366b..2022202 100644 --- a/docs/roadmap/sequences.md +++ b/docs/roadmap/sequences.md @@ -1,7 +1,8 @@ -# Configuration-Declared Sequence Identity And Layered Append +# Sequence Identity And Cross-Tier Combination Status: draft proposal for discussion. This document captures the requested -capabilities and initial recommendations; the open decisions are not settled. +capabilities, settled design choices, and initial recommendations. Choices under +Open Decisions remain unsettled. It is not an implementation plan or an implemented contract. All new syntax shown below is illustrative. @@ -10,14 +11,14 @@ shown below is illustrative. Allow configuration authors to assemble collections from independently authored records, declare their identity in configuration, and progressively extend a baseline collection through higher layers. Authors should also be able to -replace the accumulated collection explicitly. +replace the accumulated collection or replace individual records by identity. The motivating example is a prompt collection whose defaults supply some -records and whose application or installation layers add others. A repeated -identity should produce an error when records are combined. This proposal -concerns adding distinct records and replacing whole collections; matching -records would not acquire field-by-field update semantics merely by adding -this feature. +records and whose application or installation layers add or supersede records. +Across tiers, authors need three operations: whole-sequence replacement, +duplicate-rejecting append, and whole-record replacement by identity. The latter +retains unmatched lower-tier records and adds higher-tier records with new +identities. Matching records are replaced in full. ## Current Baseline @@ -43,9 +44,10 @@ later. Extending one does not automatically imply extending the other. The proposed import syntax introduces an identity declaration, tentatively named `key`, that accepts either one field or a list of fields for a compound -identity. The original proposal is to move the application's sequence -registration into configuration. Whether to remove, retain, or adapt the Go -registration remains open. +identity. Applications retain an optional interface for registering a collection +and imposing its identity definition. Configuration declarations must conform +to an application-owned constraint when one exists. The exact Go interface and +configuration declaration site remain to be designed. ```yaml configkit: @@ -55,7 +57,7 @@ configkit: destination: [prompts] placement: sequence-item key: [namespace, id] - append: true + mode: append ``` Each selected file would contribute a record such as: @@ -66,26 +68,71 @@ id: welcome text: Welcome ``` -Here `append: true` illustrates retaining a lower-layer collection and adding -distinct records from the higher layer. The user also proposed `append: false` -or an inverse `replace: true` choice for complete replacement. An explicit -operation such as `mode: append` or `mode: replace` is another candidate; -field spelling and default behavior remain open. +Here `mode: append` illustrates retaining a lower-layer collection and adding +distinct records from the higher layer. A named operation is the current +recommendation for expressing all three choices; field spelling, operation +names, and default behavior remain open. The earlier `append` or inverse +`replace` boolean proposals would need additional syntax for the third operation. The intended results can be expressed without settling the syntax. Letters -below denote record identities: +below denote record identities; `B*` is a higher-tier record with the same +identity as `B` and a different payload. The outcomes are settled; the ordering +shown is the recommendation discussed below. | Lower-tier sequence | Higher-tier sequence | Operation | Proposed result | | --- | --- | --- | --- | | `[A, B]` | `[C]` | Append | `[A, B, C]` | -| `[A, B]` | `[C]` | Replace | `[C]` | -| `[A, B]` | `[B]` | Append | Duplicate-identity error | -| `[A, B]` | `[B]` | Replace | `[B]` | +| `[A, B]` | `[C]` | Whole-sequence replacement | `[C]` | +| `[A, B]` | `[B*]` | Append | Duplicate-identity error | +| `[A, B]` | `[B*]` | Whole-sequence replacement | `[B*]` | +| `[A, B]` | `[B*]` | Replacement by identity | `[A, B*]` | +| `[A, B]` | `[B*, C]` | Replacement by identity | `[A, B*, C]` | -Replacement discards the lower sequence as a whole. A repeated identity in the +Whole-sequence replacement discards the lower sequence. A repeated identity in the retained combined sequence is an error under the proposed append semantics, including when the records have identical or disjoint payloads. +## Settled Design Choices + +### Whole-Record Replacement By Identity + +The third cross-tier operation replaces each matching lower record with the +complete higher record, retains unmatched lower records, and adds new identities. +For example, given these lower and higher records: + +```yaml +# Lower record +id: B +text: Original +temperature: 0.5 +``` + +```yaml +# Higher record +id: B +text: Revised +``` + +The resulting record at this precedence boundary contains `id` and `text`. +The lower record's `temperature` is discarded. Subsequent defaulting or other +lifecycle operations retain their separately defined semantics. Recursive +field-by-field overlay of matching records is outside this selected operation. + +### Optional Application-Owned Identity Constraints + +Retain an interface through which applications may register a collection and +impose its unique-key definition. If the application requires `key: id`, a +configuration declaration of `key: [id, name]` is invalid: the compound key +would permit repeated `id` values with different names. The application identity +constraint applies whichever cross-tier operation configuration selects. + +This is an authoritative constraint, distinct from an application default that +only fills an omission. Configuration can define identity for collections +without an application-owned constraint. The recommendation is for omitted +configuration identity to inherit a registered definition and for an explicit +matching declaration to be accepted. Enforcement timing, including whether the +constraint guarantees uniqueness in the final published result, remains OD-1. + ## Initial Architectural Direction Treat identity as a property of the destination collection, even if an import @@ -101,11 +148,13 @@ contribution at that destination. It would not sequentially erase contributions from peer imports. Multiple agreeing declarations could coalesce; conflicting explicit identity or operation declarations would produce errors. -For append, retain lower-tier item order and then the higher tier's assembled -item order. Existing dependency-ready scheduling would continue to determine -assembly order within a tier. Replacement would retain only the higher tier's -assembled sequence. These are recommendations awaiting agreement, not changes -to the current ordering contract. +For append, the ordering recommendation is to retain lower-tier item order and +then the higher tier's assembled item order. For replacement by identity, retain +the position of each replaced record and append new identities in higher-tier +order. Existing dependency-ready scheduling would continue to determine assembly +order within a tier. Whole-sequence replacement would retain only the higher +tier's assembled sequence. Ordering and the treatment of peer declarations remain +recommendations awaiting agreement, not changes to the current ordering contract. This requires collection policy to survive import expansion through precedence assembly. Correctness includes final values, projected item paths, source @@ -118,11 +167,12 @@ The implementation must account for both through the existing loading lifecycle. This is narrower than the deferred [identified-record sequences proposal](future.md#1-identified-record-sequences), which also considers recursive merging of matching records and record-specific -inheritance. Those capabilities remain separate candidates. Compound keys and -duplicate-rejecting append do not require adopting them. +inheritance. Those capabilities remain separate candidates. Compound keys, +duplicate-rejecting append, and whole-record replacement by identity do not +require adopting them. -The proposal would revise the application-owned registration and cross-tier -replacement decisions in +The proposal would extend identity declaration ownership and revise the +cross-tier replacement decision in [ADR 0014](../adr/0014-sequence-placement-and-composition.md). If adopted, record the architectural change in a superseding decision and reconcile affected architecture policy before implementation. The current ADR and contracts remain @@ -136,23 +186,28 @@ design and test sequencing belong to that later plan. ## Open Decisions -### OD-1. Who Owns Identity Policy? +### OD-1. When Is An Application Identity Constraint Enforced? -Should configuration replace `Definition.SequenceCompositions`, supplement it, -or merely provide defaults for an application-owned policy? +Application authority over a registered collection's identity definition is +settled above. Does that registration govern assembly alone, or also guarantee +valid, unique identities in the final published collection? The existing +registration checks strict-composition contributions; later defaults, +inheritance, overrides, and application callbacks have separate lifecycle roles. -**Initial recommendation:** Let configuration own identity for collections -assembled through the new mechanism, with applications retaining responsibility -for their domain requirements. Resolve the treatment of existing Go registration -before removing it: applications may depend on a fixed identity definition even -when configuration authors control source layout. Avoid two independent policy -systems whose precedence depends on acquisition order. +**Initial recommendation:** Enforce an application-owned identity constraint +during participating collection assembly and validate it again on the final +effective collection before publication. This makes the application's constraint +meaningful even when a later operation creates or changes records. It does not +require applying append or replacement-by-identity semantics during those later +operations, or validating every intermediate callback state. Whether YAML-only +identity declarations also imply final validation remains to be decided. -**Alternative:** Keep application registration authoritative and allow YAML to -declare or confirm compatible policies. This preserves schema-owned identity -constraints and supports programmatic use, but adds a reconciliation rule and -retains some application setup. YAML-only ownership is simpler for independently -authored collections but moves enforcement of fixed domain identities elsewhere. +**Alternative:** Limit registration to structural assembly and leave final +uniqueness checks to application validation. This preserves the current phase +boundary and reduces built-in validation work, but the registration alone would +not establish a final-result invariant. Its public contract would need to state +that limitation clearly. Decide enforcement separately from the combination +scope in OD-7. ### OD-2. Where Does Configuration Declare Collection Policy? @@ -179,66 +234,80 @@ relationship to that declaration. Settle compound-key field selection, value types, missing/null components, empty and repeated field lists, empty identity strings, and agreement between -declarations. Decide whether replacement can introduce a different identity -definition after discarding the lower collection. +declarations. Decide whether whole-sequence replacement can introduce a different identity +definition after discarding the lower collection when the application has imposed +no identity constraint. An application-owned constraint remains authoritative. **Initial recommendation:** Interpret `key: id` as `[id]` and a compound key as an ordered tuple of direct, literal string fields. Retain exact string comparison and require every component to be present. Preserve tuple boundaries rather than -concatenating values. Identity declarations participating in an append must -agree; equal definitions within a scope coalesce and conflicting definitions -fail. Detailed list validation and identity changes at replacement boundaries -remain to be settled. +concatenating values. Identity declarations participating in append or replacement +by identity must agree; equal definitions within a scope coalesce and conflicting +definitions fail. Detailed list validation and identity changes at whole-sequence +replacement boundaries remain to be settled. **Alternative:** Permit additional scalar types or nested field paths. These support more schemas but add type-equality and addressing rules. A load-wide fixed identity definition simplifies agreement across tiers; allowing a replacement to establish a new definition gives independently authored layers more flexibility. -### OD-4. What Does An Import's Append Or Replace Choice Control? +### OD-4. What Does An Import's Combination Choice Control? What happens when peer imports into the same destination request different -operations, or when some contributions omit the operation? +operations, or when some contributions omit the operation? Settle ordering for +replacement by identity, including where replaced records and new identities +appear. **Initial recommendation:** Assemble one strict sequence per destination in each tier, then apply its agreed operation at the precedence boundary. Reject conflicting explicit choices within a tier. This preserves peer composition and prevents declaration or scheduling order from granting replacement authority. Define how omitted choices inherit the effective collection operation once its -declaration site is selected. +declaration site is selected. Duplicate identities among peers would remain +strict-composition errors under all three cross-tier operations. For replacement +by identity, keep matched records in their lower-tier positions and append new +identities in higher-tier order. -**Alternative:** Treat each import as a sequential append-or-replace operation. +**Alternative:** Treat each import as a sequential collection operation. This permits deliberate resets among imports, but a replacement can discard peer contributions and would require an explicitly selected operation-order model. Another alternative is to allow the choice only on a collection declaration, which reduces conflicts between import annotations but changes the proposed authoring surface. +For ordering, moving replaced records to their higher-tier contribution positions +is another viable choice. It gives the higher layer more influence over ordering, +but changing a record's payload could also move it relative to retained records. + ### OD-5. What Is The Default Operation And Its Syntax? -Choose between `append`, an inverse `replace` option, or a named operation, and -decide what omission means. +Choose syntax for all three operations and decide what omission means. -**Initial recommendation:** Keep replacement as the cross-tier default and -make append explicit. Lists can represent complete ordered instructions as well -as extensible catalogs; an explicit append communicates the intent to retain -lower-layer records. Prefer a positive named operation if both modes need equal -prominence, while considering the simplicity of `append: true`. +**Initial recommendation:** Keep whole-sequence replacement as the cross-tier +default and make append and replacement by identity explicit. Lists can represent +complete ordered instructions as well as extensible catalogs; an explicit append +communicates the intent to retain +lower-layer records. Prefer one named operation with three values to interacting +booleans; choose names that distinguish whole-sequence from whole-record +replacement. -**Alternative:** Default identified collections to append and require explicit -replacement. This makes progressive collections concise, but retains lower-layer -records unless an author deliberately clears them. An inverse `replace` boolean -expresses that model compactly; a named operation makes both choices explicit. +**Alternative:** Default identified collections to append or replacement by +identity and require explicit whole-sequence replacement. These defaults make +progressive collections concise, but retain lower-layer records unless an author +deliberately replaces the collection. Extending the boolean proposal with a +separate duplicate-handling option is possible, but introduces interacting +settings whose combinations need explanation and validation. ### OD-6. What Do Absence, Empty Sequences, And Null Mean? Can a declaration selecting nothing affect a lower collection? How do explicit -empty sequences, null, and non-sequence values interact with append or replace? +empty sequences, null, and non-sequence values interact with all three operations? **Initial recommendation:** An optional import selecting nothing contributes nothing and does not clear lower values. An explicitly supplied empty sequence -with replacement can clear a collection; append of an empty sequence adds no -records. Preserve the distinction between absence and supplied values. Decide +with whole-sequence replacement can clear a collection; append or replacement +by identity with an empty sequence contributes no records. Preserve the +distinction between absence and supplied values. Decide null and kind-mismatch handling explicitly rather than silently treating them as empty sequences. @@ -249,22 +318,28 @@ as an explicit value avoids that coupling. ### OD-7. Which Lifecycle Stages Participate? -Does append extend only source-backed tiers, or also `Definition.Defaults`, -inheritance, CLI overrides, and application-created records? At which boundaries +Do the new cross-tier operations extend only source-backed tiers, or also +`Definition.Defaults`, inheritance, CLI overrides, and application-created records? +At which boundaries must identity be validated, including helper-supplied fields and consumed source version headers? **Initial recommendation:** Start with source-backed strict composition and precedence assembly. Validate records at the relevant collection assembly -boundaries and uniqueness across each retained append result. Preserve existing +boundaries and uniqueness across each combined result. Preserve existing typed-default and application-resolution semantics unless their inclusion is explicitly chosen. Decide selected-overlay coverage with OD-2; it is part of source precedence but currently has no import declaration site. Account for item-local helpers and source-version consumption before deciding identity is available, consistent with the current lifecycle. -**Alternative:** Apply collection policy throughout defaulting, inheritance, -overrides, and application resolution, including final identity checks. This -would provide a broader final-collection guarantee, but requires additional -combination, callback, and inspection rules. Supporting typed default lists is -a separate scope choice from extending source-backed baseline collections. +OD-1 separately addresses final validation of application constraints. Validating +the final collection does not itself change how defaults, inheritance, CLI +overrides, or callbacks combine values. + +**Alternative:** Extend the combination operations through defaulting, +inheritance, overrides, and application resolution. This would support +progressively extending collections across more of the lifecycle, but requires +additional combination, callback, and inspection rules. Supporting typed default +lists is a separate scope choice from extending source-backed baseline +collections; final identity enforcement remains the separate OD-1 decision.