diff --git a/docs/roadmap/roadmap.md b/docs/roadmap/roadmap.md new file mode 100644 index 0000000..ecc06f0 --- /dev/null +++ b/docs/roadmap/roadmap.md @@ -0,0 +1,669 @@ +# Sequence Identity And Combination Roadmap + +Status: settled feature roadmap, ready for a staged implementation plan. This +is the targeted end state, not an implemented contract. The subsequent plan will +specify execution order, the exact Go API, internal bookkeeping, and validation +stages. + +## Purpose And Scope + +Allow configuration authors to assemble collections from independently authored +records, declare identity in configuration, and progressively extend a baseline +collection through higher layers. Authors can replace the entire collection, +append distinct records, or replace complete records by identity while retaining +unmatched records. Sibling inheritance uses the same sequence operations and +identity-based defaults. + +The motivating use case is a prompt catalog whose source-backed baseline supplies +records and whose application or installation layers add or supersede records. +Applications can optionally impose identity constraints that configuration must +respect and that hold in the final loaded configuration. + +The feature covers ordinary and layered loading, imported items, inline +sequences, selected overlays, and sibling inheritance. Typed defaults, CLI +assignments, and application resolution retain their own combination semantics; +CLI application moves later in the loading lifecycle. + +This scope is narrower than the deferred +[identified-record sequences proposal](future.md#1-identified-record-sequences). +Recursive merging of matching records and record-specific inheritance remain +separate candidates. The initial declaration surface is `configkit.sequences`; +`key` and `mode` shorthand on imports is deferred. Application lists remain +ordinary YAML sequences without control wrappers. Unkeyed sequence concatenation +is outside this feature. + +## Current Baseline And Required Changes + +The [import contract](../config.md#explicit-imports) owns existing selectors, +destinations, placement, and ordering. `placement: sequence-item` already places +each selected source mapping as one item at its effective destination. Item-local +helper imports can contribute fields before placement. + +The [composition contract](../config.md#strict-document-composition), +[layered-loading contract](../config.md#layered-loading), and +[inheritance contract](../config.md#sibling-inheritance) describe the implemented +baseline: + +| Area | Current behavior | Target behavior | +| --- | --- | --- | +| Identity declaration | `Definition.SequenceCompositions` registers one literal string field at each registered mapping path. | YAML collection declarations and optional authoritative application registrations support field sets and scalar identity components. | +| Strict composition | Registered sequences append distinct string identities and reject duplicates; repeated unregistered sequences conflict. | Collections with effective identity append distinct records and reject duplicates. Applicable lower-tier identity is available before composing higher-tier peers. Unidentified sequences retain strict-conflict behavior. | +| Source precedence and selected overlays | A present higher sequence replaces the lower sequence wholesale. | Collection policy selects `replace`, `append`, or `upsert`; overlays can declare policy for inline sequences. | +| Sibling inheritance | An inheriting mapping recursively overlays its resolved parent; supplied sequences replace wholesale. | Mappings retain recursive combination, while sequences use destination policy and the same three operations. | +| CLI overrides | Apply before inheritance and typed defaults/decoding. | Apply after inheritance and typed defaults/decoding, before application resolution. | +| Identity enforcement | Registration governs strict sequence composition. | Structural identity checks cover identified combinations; application registrations additionally constrain final effective sequences. | + +Source-backed underlays participate in source assembly. `Definition.Defaults` +provides typed defaults later; extending source combination does not turn typed +default lists into source underlays. + +## Collection Declarations + +Use a source-root `configkit.sequences` block containing collection declarations. +Each entry identifies a `destination` and may declare `key`, `mode`, or both. +Identity and mode can be supplied independently. This form serves imported items, +inline records, ordinary mapping imports, and inline sequences in selected +overlays. + +```yaml +configkit: + sequences: + - destination: [prompts] + key: [namespace, id] + mode: append + imports: + - dir: prompts + pattern: '*.yaml' + destination: [prompts] + placement: sequence-item +``` + +Each selected file contributes a record such as: + +```yaml +namespace: support +id: welcome +text: Welcome +``` + +Here append retains a lower-layer collection and adds distinct higher-layer +records. Identity and combination controls belong to the collection declaration, +so imported and inline contributions share one policy surface. + +### Destinations And Scope + +`destination` uses the existing sequence of literal mapping keys. Resolve it +relative to the declaring source's placement: a fragment mounted under `catalog` +can declare `[prompts]` locally. A sequence-item source can declare a nested +collection within its own item without referring to an eventual numeric index. +Preserve source and item-local scope boundaries. + +A collection's scope is its resolved destination within its composition scope. +Resolve relative destinations before reconciling declarations. Different imported +items retain separate scopes for their nested collections. A declaration governs +all contributions to its collection in the tier, including imported and inline +records; it does not govern only records in the declaring file. + +Gather declarations before composing the relevant scope so their effect does not +depend on which contribution is assembled first. Retain applicable controls +through source precedence and inheritance, then keep them out of application +decoding. + +Sequence declarations are independent of filesystem import enablement. Selected +overlays may contain `configkit.sequences`, extending the current blanket +prohibition on overlay `configkit` declarations. This does not enable imports +in overlays. + +### Overlay Authoring + +For an existing sequence `[foo, bar]`, an overlay can replace `foo` in full: + +```yaml +configkit: + sequences: + - destination: [prompts] + key: id + mode: upsert +prompts: + - id: foo + text: Revised definition +``` + +The result is `[foo*, bar]`. The explicit upsert mode can be omitted when no +different explicit mode applies from a lower tier: effective identity then +selects upsert. To add a distinct record instead: + +```yaml +configkit: + sequences: + - destination: [prompts] + key: id + mode: append +prompts: + - id: baz + text: Another prompt +``` + +The result is `[foo, bar, baz]`. To replace the whole sequence: + +```yaml +configkit: + sequences: + - destination: [prompts] + mode: replace +prompts: + - id: bar + text: Bar definition + - id: baz + text: Baz definition +``` + +The result is `[bar, baz]`. Explicitly selecting the default operation is valid. +With no applicable explicit mode, omitting this declaration selects replace for +an unidentified collection and upsert for an identified one. A retained explicit +mode continues to govern. + +## Identity Definitions And Constraints + +### Field Sets + +`key` accepts a string or a nonempty list of strings naming direct, literal +fields in each record. A name such as `metadata.id` denotes that literal field, +not a nested lookup. Treat field names as a set: remove repeated names and +ignore authored order when comparing definitions. + +Thus `id`, `[id]`, and `[id, id]` are equivalent; `[namespace, id]`, +`[id, namespace]`, and `[namespace, id, id]` are also equivalent. An empty list +is invalid because it supplies no identity fields. These rules apply to both +configuration declarations and application registrations. + +Use a consistent field order internally to construct identity tuples. Preserve +component types and tuple boundaries so unstructured concatenation cannot cause +otherwise distinct identities to collide. + +### Scalar Components And Equality + +Each identified record must be a mapping with every designated identity field +present. Components may be strings, numbers, booleans, or explicit null. Missing +fields and mapping or sequence components are invalid. Present null differs +from absence; two single-field null identities are equal. Compound identities +are equal when all corresponding components are equal. + +Strings compare exactly, including empty strings, without trimming, case folding, +or normalization. Scalar categories remain distinct: `"1"`, `1`, `true`, and +null are different identities. + +Numbers compare by mathematical value across integer, fractional, and exponent +spellings: `1`, `1.0`, and `1e0` are equal, as are `0` and `-0`. Use exact numeric +normalization for identity comparison without rounding through a floating-point +conversion. This does not rewrite authored values or change existing typed +decoding and projection semantics. The parser's rejection of non-finite numeric +forms remains unchanged. + +### Application-Owned Constraints + +Retain an optional application interface for registering collections and imposing +their identity definitions. Configuration may define identity where no application +constraint exists. When a declaration omits `key`, an applicable registration +supplies identity. An explicit equivalent definition is accepted, and a different +definition fails under every mode, including replace. + +For example, if the application requires `id`, `key: [id, name]` is invalid: +that compound identity would allow repeated `id` values with different names. +A registration is an authoritative constraint, not an omission-filling default. +Copying a collection through inheritance does not create another application +registration; registrations constrain their registered destinations. + +Identity constraints validate actual sequences. They do not independently require +a path to exist, be non-null, or have sequence shape. The application schema and +validation own those requirements. Final-result enforcement for application +registrations is defined under [Validation And Diagnostics](#validation-and-diagnostics). + +## Strict Composition And Precedence Operations + +### Compose Peers Before Applying Precedence + +Resolve applicable identity and mode policy, strictly compose the tier's peer +contributions, then apply one selected operation per collection against the +accumulated lower value. The tier operation does not sequentially replace peer +source contributions. Selected overlay contributions use the same combination +semantics. + +Within a strict tier, equivalent identity definitions coalesce and different +field sets fail. Agreeing explicit modes coalesce; conflicting explicit modes +fail. Omission expresses no preference. A root declaration or a later declaration +has no precedence over conflicting peers. Explicitly selecting a default mode +is still a real instruction and can conflict with another explicit mode. + +Identified peer sequences append distinct records. Duplicate identities among +peers or within a contributed list fail under all three modes, including replace +and upsert. Equal or disjoint record payloads do not excuse a duplicate identity. +Unidentified sequences retain their current strict-composition conflict behavior. + +For example, higher-tier fragments `[C]` and `[D]` with an agreed identity assemble +as `[C, D]`. Against lower `[A, B]`, append or upsert produces `[A, B, C, D]`, +while replace produces `[C, D]`. Peers `[B]` and `[B*]` conflict even under +upsert because neither has higher precedence. + +### Available Operations + +Use one spelling for each mode: `replace`, `append`, and `upsert`. + +| Mode | Behavior at source-precedence and inheritance boundaries | +| --- | --- | +| `replace` | Replace the complete lower sequence with the assembled higher sequence. | +| `append` | Retain lower records and append higher records; duplicate identities fail. | +| `upsert` | Replace matching records in full, retain unmatched lower records, and append new identities. | + +Append and upsert require effective identity. ConfigKit does not infer identity +from an `id` field or record position. Whole-sequence replacement does not require +identity agreement with lower YAML policy, but application constraints still apply. + +Letters denote record identities; `B*` has the same identity as `B` with a different +payload: + +| Lower | Higher | Mode | Result | +| --- | --- | --- | --- | +| `[A, B]` | `[C]` | `append` | `[A, B, C]` | +| `[A, B]` | `[C]` | `replace` | `[C]` | +| `[A, B]` | `[B*]` | `append` | Duplicate-identity error | +| `[A, B]` | `[B*]` | `replace` | `[B*]` | +| `[A, B]` | `[B*]` | `upsert` | `[A, B*]` | +| `[A, B]` | `[B*, C]` | `upsert` | `[A, B*, C]` | + +Upsert replaces the whole matched record. If lower `B` has `id`, `text`, and +`temperature`, while higher `B` has only `id` and `text`, the result at this +boundary has only `id` and `text`. Subsequent defaulting or other lifecycle +operations retain their own semantics. Mappings continue to combine recursively; +matching sequence records do not recursively merge fields. + +### Mode Selection And Persistence + +Select the operation in this order: + +1. Reconcile explicit modes within the current tier; conflicting peers fail. +2. Use the current tier's explicit mode if supplied. +3. Otherwise retain an applicable explicit mode from a lower tier. +4. With no explicit mode, infer upsert when effective identity is available. +5. Otherwise infer replace. + +Effective identity can come from an application registration, current declarations, +or applicable lower policy, subject to the identity agreement and lifetime rules +below. Determine the inferred mode from available identity before applying the +operation. A plain higher-tier sequence can therefore use upsert without repeating +a key declaration. + +An explicit mode persists at its applicable destination until a higher-priority +explicit declaration changes it. New sequence values alone do not reset it, and +an unrelated overlay leaves it intact. A mode-only declaration can change policy +without supplying records; it neither clears the collection nor retroactively +recombines earlier tiers. Later source contributions and inheritance use the +resulting policy. + +| Lower explicit mode | Higher declaration | Effective mode | +| --- | --- | --- | +| `append` | Mode omitted | `append` | +| `append` | `upsert` | `upsert` | +| `upsert` | `replace` | `replace` | +| None | Mode omitted, effective identity available | Inferred `upsert` | +| None | Mode omitted, no effective identity | Inferred `replace` | + +Only explicit modes persist. Inferred defaults are recomputed from effective +identity when needed. Explicit replace continues to apply in subsequent tiers +until another mode is declared; introducing identity alone does not override it. +Mode persistence does not preserve YAML identity discarded by replacement, nor +policies owned by discarded records. Source precedence and inheritance use the +same destination policy, without separate per-phase settings. CLI assignments +remain outside mode selection. + +### Presence, Empty Sequences, Null, And Kind Mismatches + +At a collection's precedence boundary: + +| Higher contribution | `replace` | `append` / `upsert` | +| --- | --- | --- | +| Absent | Leave the lower value unchanged. | Leave the lower value unchanged. | +| Explicit `[]` | Clear the sequence. | Contribute no records. | +| Explicit null or another non-sequence value | Ordinary replacement, subject to application schema. | Error: the operation requires a sequence. | + +A declaration alone does not clear a collection. An optional import selecting +nothing contributes nothing and does not erase existing records. An absent lower +collection can be initialized by an incoming sequence under any mode, subject to +identity requirements. Explicit lower null or another non-sequence value is not +absence: append and upsert fail against it, while replacement remains available. +An empty incoming sequence bypasses neither kind checks nor identity requirements. + +With effective identity and no explicit mode, `prompts: []` uses upsert and does +not clear the collection. Clearing requires replace, selected in the current tier +or retained from lower policy. Replacing an identified collection with null also +requires replace and a schema permitting null. Collection null is distinct from +a present null identity component inside a record. + +### Ordering + +Retain existing dependency-ready source scheduling within each tier and authored +item order within lists. Gathering declarations does not change assembly order; +identity values do not sort records. + +- Append keeps lower records in order, followed by higher records in assembled order. +- Replace keeps only higher records in their assembled order. +- Upsert keeps a matched record's lower position and appends new identities in + higher-tier order. + +For example, lower `[A, B, C]` and higher `[C*, A*, D]` produce `[A*, B, C*, D]` +under upsert. Whole-sequence replacement gives the higher tier control of the +complete sequence order. + +## Policy Agreement, Lifetime, And Ownership + +### Identity Agreement And Replacement + +Append and upsert use one agreed identity for all participating records: + +| Lower identity | Higher declaration | Result when combining records | +| --- | --- | --- | +| `id` | Omitted | Use `id`. | +| `id` | `id`, `[id]`, or `[id, id]` | Accept the equivalent definition. | +| `id` | `[namespace, id]` | Error: different definitions. | +| None | `id` | Establish `id` and validate all participating records. | + +Introducing identity for an unkeyed lower sequence requires checking both lower +and incoming records, including uniqueness within each contribution. With no +identity from the application or either participating collection, append and +upsert fail. + +YAML identity follows the collection that survives an operation: + +| Operation | Effective YAML identity | +| --- | --- | +| Append or upsert | Retain or establish the agreed identity. | +| Whole-sequence replacement | Discard lower YAML identity and use the replacing tier's identity declaration, if any. | +| Replacement by null or another non-sequence value | Discard lower YAML identity and retain any identity explicitly declared by the replacing tier. | + +Application constraints remain authoritative under every operation. Without an +application constraint, replacement can introduce a different identity, such as +changing from `id` to `name`. Replacing a sequence without a new key declaration +or application registration produces an unkeyed sequence. No identity-removal +directive is needed. + +The reset occurs only when the collection value is actually replaced. Declaring +replace without supplying a replacement does not authorize changing the identity +of retained records. Explicit mode lifetime is separate from YAML identity +lifetime. + +### Declarations Without Sequence Values + +A key declaration without accompanying records applies immediately to an existing +retained sequence: check agreement and validate its records, without recombining +earlier tiers. For example, a higher tier that only declares +`destination: [prompts], key: id` identifies and validates the retained lower +`prompts` sequence. + +If no collection exists, retain the declaration in its owning scope for later +contributions. An otherwise empty underlay can establish identity for main-tier +fragments. An explicit identity declaration can also remain pending at a +non-sequence destination; it does not independently impose a shape requirement. +The presence and kind rules still determine whether an attempted operation fails. +Item-local declarations retain their ownership boundaries. + +### Lower-Tier Identity During Strict Composition + +Resolve applicable identity before composing a higher tier's peers, using +application registrations, available lower-tier policy, and declarations gathered +within the higher tier's scope. Authors need not repeat a baseline key solely +because the higher tier has multiple fragments. + +For example, an underlay establishes `key: id` for `prompts`, and two main-tier +fragments contribute `[foo]` and `[bar]` without a key declaration. They strictly +compose as `[foo, bar]` before the main tier combines with the accumulated lower +collection. If both contribute `foo`, composition fails; upsert does not give +precedence to peers. + +Lower YAML identity remains subject to agreement, replacement, and ownership. +A replacing collection must not be constrained by obsolete lower identity just +because policy resolution precedes composition. Likewise, a replacement record +cannot borrow nested identity owned by the discarded lower record, even when +outer identity or numeric position matches. Application constraints still apply. +Higher-tier declarations do not retroactively repair invalid lower-tier composition. + +### Nested Record Ownership + +Attach item-local policies to their owning contributions so they follow surviving +records independently of numeric indexes. Whole-record upsert discards the old +record's local policies along with its fields. Whole-sequence replacement discards +policies owned by removed records. A replacement record supplies its own local +declarations; equal outer identity or final index does not transfer discarded +policies. + +For example, a source imported as one record into `prompts` can declare: + +```yaml +configkit: + sequences: + - destination: [steps] + key: id + mode: append +id: foo +steps: + - id: prepare +``` + +The declaration governs `steps` inside that `foo` record. Retaining `foo` retains +its local policy, including the explicit mode. Upserting a new `foo` discards the +old `steps` and its declaration even though the outer record keeps its position. + +Policies independently applicable at a destination are not discarded merely +because an individual record is replaced. Ownership determines which declarations +are removed. Application registrations remain constraints at their registered +destinations. Inheritance copies transfer identity and choose operations as +specified next. + +## Sibling Inheritance + +Complete source-tier assembly before resolving inheritance. Retain the existing +dependency-ready algorithm and sibling scope rules. At each inheritance combination, +the resolved parent's collection is the lower value and the inheriting mapping's +local collection is the higher value. Mappings combine recursively; identified +sequence records remain whole-record replacements. + +Use the inheriting destination collection's policy: an applicable explicit mode, +including one retained through source tiers, wins. Otherwise infer upsert with +effective identity and replace without it. There is no separate default-append +rule for inheritance. The same destination policy governs source-precedence and +inheritance combinations. + +YAML identity follows a collection copied through inheritance, subject to agreement +and replacement rules. A source collection's mode does not automatically become an +instruction at a different inheriting destination. For example, source `steps` +has `key: id` and `mode: append`; an inheriting mapping supplies local `steps` +without an applicable explicit mode. The copied identity selects upsert there. +An explicit destination append instead selects append. A matching destination key +agrees; a different key conflicts when combining records, while explicit replace +can establish new YAML identity subject to application constraints. + +For example: + +```yaml +configkit: + sequences: + - destination: [parent, items] + key: id +child: + items: + - id: foo + value: A + - id: bar + value: B +parent: + extends: child + items: + - id: baz + value: C +``` + +Despite the names, `parent` inherits from `child`. Identity at `[parent, items]` +selects upsert, producing `[foo, bar, baz]`. If the local sequence instead supplies +a revised `foo`, the result is `[foo*, bar]`. Append rejects that repeated identity; +replace retains only the local sequence. + +The same ordering, presence, and kind rules apply at inheritance boundaries. +An omitted local sequence retains the inherited sequence. Explicit `[]` clears it +under replace and contributes no records under append or upsert. Retain controls +until inheritance uses them, then exclude them from application decoding. + +## Loading Lifecycle + +Use the following order: + +1. Assemble source tiers and selected overlays, resolving policy before each + tier's strict composition and applying collection operations at precedence + boundaries. +2. Resolve sibling inheritance, then apply typed defaults and decode the assembled + configuration. +3. Apply explicit CLI overrides to that configuration and check supplied values + against the application schema. +4. Run application resolution against the overridden configuration. +5. Resolve effective secret references, validate the final configuration including + application-registered identities, and publish it. + +### Typed Defaults And Application Resolution + +Source-backed underlays, including application-supplied mapping sources, +participate in sequence combination. Typed `Definition.Defaults` remains the +initial decoded state: supplied sequences replace default sequences. Source +append does not append to a typed default list. + +Application resolution retains assignment and recorded-transformation semantics. +ConfigKit does not automatically combine a callback's assigned sequence with its +previous value. Application-registered identities receive final validation +independently of how defaults, CLI overrides, or callbacks produce their values. + +### CLI Precedence + +CLI overrides are the highest-precedence input to application resolution. +Supplied sequences replace whole sequences, including sequences inside supplied +mappings. Mappings retain recursive override behavior; empty sequences and +permitted nulls retain explicit replacement meaning. Configuration append or +upsert does not reinterpret CLI assignments. Inheritance and defaulting do not +run again afterward: overriding assembled `[A, B]` with `[C]` presents `[C]` to +application resolution without inheritance reintroducing prior records. + +Application resolution sees CLI values and their explicit-presence and source +metadata. `Resolution.Default` preserves explicit CLI values; normalization and +recorded application transformations may still change them. For example, a derived +URL can use a CLI hostname, while an explicit CLI URL can be preserved by an +absence-based default. Retain recording and provenance requirements for those +transformations. + +Reject structural directives such as `extends` and `configkit` in CLI override +paths or supplied values. CLI input supplies application values after structural +processing; it cannot supply or cancel inheritance directives, change completed +relationships, or prevent earlier inheritance failures. + +Typed decoding also precedes CLI application, so a CLI replacement cannot repair +an earlier decoding failure. A source string in an integer field fails before a +valid CLI integer can apply. This changes current override-before-decoding +behavior. Semantic validation remains after overrides and application resolution. + +Secret lookup uses effective references after overrides and resolution. Neither +stage bypasses final application identity constraints or semantic validation. + +## Validation And Diagnostics + +### Structural Identity Checks + +Allow item-local helper imports to complete and consume applicable non-main +source-version headers before checking payload identity for collection assembly. +A helper can supply an identity component; a consumed header cannot supply one +after removal. + +Validate each identified contribution before combination and check uniqueness in +the resulting identified collection, including inheritance operations. Upsert +matches across precedence are replacements; duplicates within a contribution +remain errors. Later inheritance, typed defaults, or callbacks cannot repair an +identity missing at an earlier assembly boundary. + +YAML-only identities are enforced during structural composition, precedence +assembly, and identified inheritance operations. They have no final-result +guarantee after typed defaulting, CLI overrides, or application resolution. Those +stages can produce duplicate identities or otherwise violate a YAML-only identity. +Applications needing a final guarantee register the constraint through the +application-owned interface. + +### Final Application Constraints + +On successful loading, final effective collections covered by application +registrations satisfy their declared identity constraints. Retain assembly-time +checks and validate final effective sequences before publication, after defaults, +inheritance, overrides, application resolution, and secret lookup. + +This guarantee covers actual sequences, not path existence or allowed shape. +It does not require every intermediate callback state to satisfy the constraint +and does not cover caller mutations after loading returns. Final checks examine +effective typed values; assembly checks validate authored contributions. A final +scalar alone does not prove that a source explicitly supplied that field. + +Use the existing canonical final-value representation and shared publication +path as the foundation. The guarantee requires neither a new mutation-history +system nor exact causal reconstruction for records supplied by defaults or +callbacks. + +### Diagnostics And Inspection + +Identity diagnostics identify collection and item paths using available origin +information without exposing identity values. Conflicting declarations identify +the collection destination and declaration locations. + +Correctness includes final values, projected item paths, source destinations, +provenance, history, sensitivity, and passive resource contexts. Appending can +change final indexes; replacing changes which contributions survive. Preserve +the existing loading and inspection contracts through both operations, including +policy ownership and inheritance transfer. + +## Work Needed From The Baseline + +The implementation plan should cover these connected changes, with execution +order and test sequencing defined there: + +- **Declarations and public API:** extend application registration to the settled + identity model; parse and consume `configkit.sequences` across supported source + forms and selected overlays; preserve placement-relative and item-local scopes. + Keep sequence declarations independent of filesystem import enablement. +- **Identity and policy representation:** implement field-set comparison, typed + scalar identity tuples with exact numeric equality, declaration agreement, + pending declarations, explicit-mode persistence, replacement lifetime, and + record-owned policy transfer. Choose bookkeeping proportional to these semantics. +- **Assembly handoffs:** make accumulated lower policy available before higher-tier + strict composition. Separate source acquisition from policy-aware composition + or otherwise revise assembly sequencing; current main-before-underlay composition + is not a design constraint. Preserve distinctions among acquisition order, + dependency-ready scheduling within tiers, and precedence order. Account for + source records, callbacks, main-version checks, and failure diagnostics. Retain + version ownership and validation semantics; higher replacement cannot hide + lower-tier structural failures. +- **Combination and inheritance:** implement the three operations with strict peer + handling, whole-record replacement, presence rules, stable ordering, and the + destination policy rules for inheritance. Preserve provenance, history, + sensitivity, paths, and resource contexts as records survive, move, or disappear. +- **Lifecycle and validation:** move CLI application after inheritance and typed + defaults/decoding, preserve explicit-presence and source metadata for application + resolution, reject CLI structural directives, and add final application identity + validation through the shared publication path. Keep YAML-only enforcement + within structural operations. +- **Documentation and verification:** exercise ordinary and layered loading, + imported and inline collections, item-local scopes, version-header consumption, + selected overlays, inheritance, typed defaults, CLI, callbacks, and inspection. + Update the public contract and consumer examples when behavior lands. + +Before implementation, record the durable architectural changes in a superseding +decision and reconcile affected architecture policy. The feature changes identity +ownership and precedence replacement in +[ADR 0014](../adr/0014-sequence-placement-and-composition.md), and sequence +combination and CLI-before-inheritance ordering in +[ADR 0009](../adr/0009-sibling-inheritance.md). Existing ADRs and contracts remain +authoritative until that architectural and implementation work lands. This roadmap +does not select migration machinery. + +No open feature decisions remain. The implementation plan should translate this +end state into reviewable stages without adding new guarantees, restrictions, or +combination semantics. diff --git a/docs/roadmap/sequences.md b/docs/roadmap/sequences.md deleted file mode 100644 index 86beef9..0000000 --- a/docs/roadmap/sequences.md +++ /dev/null @@ -1,904 +0,0 @@ -# Sequence Identity And Combination - -Status: feature roadmap with OD-1 through OD-12 settled and ready for a staged -implementation plan. No open design decisions remain in this roadmap. This is -not an implementation plan or an implemented contract. - -## Motivation - -Allow configuration authors to assemble collections from independently authored -records, declare their identity in configuration, and progressively extend a -baseline collection through higher layers. Authors should also be able to -replace the accumulated collection or replace individual records by identity. - -The motivating example is a prompt collection whose defaults supply some -records and whose application or installation layers add or supersede records. -Across tiers, authors need three operations: whole-sequence replacement, -duplicate-rejecting append, and whole-record replacement by identity. The latter -retains unmatched lower-tier records and adds higher-tier records with new -identities. Matching records are replaced in full. Sibling inheritance through -`extends` uses the same sequence operations and effective-identity defaults. - -## Current Baseline - -The [import contract](../config.md#explicit-imports) owns current selector, -destination, placement, and ordering behavior. An import with -`placement: sequence-item` places each selected source mapping as one item at -its effective destination. Item-local helper imports can contribute fields -before placement. - -The application currently registers sequence identity through -`Definition.SequenceCompositions`, using one literal string field at each -registered mapping path. Within a strict tier, distinct identities append and -duplicate identities fail. Repeated unregistered sequences conflict. Across -precedence tiers, a higher sequence replaces the lower sequence wholesale. -The [composition contract](../config.md#strict-document-composition) owns these -rules; [layered loading](../config.md#layered-loading) owns tier precedence. - -Current [sibling inheritance](../config.md#sibling-inheritance) recursively -overlays an inheriting mapping onto its resolved parent and replaces supplied -sequences wholesale. The proposed inheritance extension is settled in OD-7. - -Source-backed underlays and `Definition.Defaults` have different lifecycle -roles. Underlays participate in source assembly; typed defaults are applied -later. Extending one does not automatically imply extending the other. - -## Requested Capabilities - -The proposed `configkit.sequences` block introduces a `key` declaration that -accepts either one field or a list of fields for a compound identity. -Fields are treated as a set; identity components may be strings, numbers, -booleans, or explicitly supplied null values. -Applications retain an optional interface for registering a collection -and imposing its identity definition. Configuration declarations must conform -to an application-owned constraint when one exists. The exact Go interface -remains to be designed. - -```yaml -configkit: - sequences: - - destination: [prompts] - key: [namespace, id] - mode: append - imports: - - dir: prompts - pattern: '*.yaml' - destination: [prompts] - placement: sequence-item -``` - -Each selected file would contribute a record such as: - -```yaml -namespace: support -id: welcome -text: Welcome -``` - -Here `mode: append` illustrates retaining a lower-layer collection and adding -distinct records from the higher layer. The `mode` field expresses the operation; -OD-5 defines its names and effective-identity default. Identity and combination -controls belong to the collection declaration; import shorthand is deferred. - -Letters below denote record identities; `B*` is a higher-tier record with the same -identity as `B` and a different payload. The outcomes and ordering shown are -settled below. - -| Lower-tier sequence | Higher-tier sequence | Operation | Proposed result | -| --- | --- | --- | --- | -| `[A, B]` | `[C]` | `append` | `[A, B, C]` | -| `[A, B]` | `[C]` | `replace` | `[C]` | -| `[A, B]` | `[B*]` | `append` | Duplicate-identity error | -| `[A, B]` | `[B*]` | `replace` | `[B*]` | -| `[A, B]` | `[B*]` | `upsert` | `[A, B*]` | -| `[A, B]` | `[B*, C]` | `upsert` | `[A, B*, C]` | - -Whole-sequence replacement discards the lower sequence. A repeated identity in the -retained combined sequence is an error under the proposed append semantics, -including when the records have identical or disjoint payloads. - -## Settled Design Choices - -### Collection Declarations - -**OD-2: settled.** - -Use a root `configkit.sequences` block containing a sequence of collection -declarations. Each entry identifies a `destination` and may declare `key`, -`mode`, or both. Application sequences remain ordinary YAML data. This declaration -form serves imported items, inline records, and sequences supplied by ordinary -mapping imports, including inline sequences in selected overlays. - -`destination` uses the existing sequence of literal mapping keys. Resolve it -relative to the declaring source's placement: a fragment mounted under -`catalog` can declare `[prompts]` locally. A sequence-item source can declare a -nested collection within its own item without referring to an eventual numeric -index. Preserve those source and item-local scope boundaries. -OD-11 settles policy ownership when enclosing records are replaced or inherited. - -Identity and mode can be supplied independently. Omitted identity can use an -application registration; an explicit matching declaration is accepted, and a -conflicting declaration is invalid. Whole-sequence replacement does not itself -require identity matching, but application-owned constraints still apply. -Identity-policy lifetime across tiers is settled in OD-3 below. - -Mode describes the collection's combination instruction at source-precedence -and inheritance boundaries, as detailed in OD-7. An explicit mode persists at -the same applicable destination until a higher-tier declaration changes it. -Gather collection declarations before composing their relevant scope so their -effect does not depend on which contribution is assembled first. Identity agreement is settled -in OD-3; operation agreement and omission among peers are settled in OD-4. -OD-9 settles how operation controls survive source precedence for later inheritance. - -Permit sequence declarations in selected overlays and make their use independent -of filesystem import enablement. This deliberately changes the current blanket -prohibition on overlay `configkit` declarations. Enabling imports in overlays -is not part of this decision. Consume sequence controls before application -decoding, as with other structural directives. - -For example, an overlay can replace a complete record by identity: - -```yaml -configkit: - sequences: - - destination: [prompts] - key: id - mode: upsert - -prompts: - - id: foo - text: Revised definition -``` - -The explicit `mode: upsert` line can be omitted here when no different explicit -mode applies from a lower tier: `key: id` then supplies the identity for inferred -upsert. An overlay can instead select duplicate-rejecting append: - -```yaml -configkit: - sequences: - - destination: [prompts] - key: id - mode: append - -prompts: - - id: baz - text: Another prompt -``` - -Or it can explicitly select whole-sequence replacement: - -```yaml -configkit: - sequences: - - destination: [prompts] - mode: replace - -prompts: - - id: bar - text: Bar definition - - id: baz - text: Baz definition -``` - -Explicitly selecting the default operation is valid. With no applicable explicit -mode, removing the last example's `configkit` block selects replace only when no -effective identity is available; an identified collection instead selects upsert. -A retained explicit mode also affects omission. Explicit `mode: replace` overrides -either another retained mode or inferred upsert. OD-5 defines the selection rule -for `replace`, `append`, and `upsert`. - -Start with this common collection declaration and defer `key` and `mode` -shorthand on imports. A single declaration form covers inline overlays and mixed -inline/imported collections without duplicate authoring surfaces. Local wrappers -around sequence values would require distinguishing application data from -control mappings; the existing reserved namespace keeps that distinction clear. - -### Identity Definitions And Lifetime - -**OD-3: settled.** - -#### Field Sets - -`key` accepts a string or a nonempty list of strings naming direct, literal -fields in each record. `key: id` and `key: [id]` are equivalent. A name such as -`metadata.id` denotes that literal field, not a nested lookup. - -Treat the selected fields as a set: remove repeated names and disregard authored -field order when comparing definitions. For example, `[namespace, id]`, -`[id, namespace]`, and `[namespace, id, id]` define the same identity. An empty -list is invalid because it supplies no identity fields. These equivalence rules -also apply when comparing configuration declarations with application -registrations. - -Use a consistent field order internally to construct compound identity tuples. -Preserve component types and tuple boundaries; concatenating unstructured values -must not cause otherwise distinct identities to collide. - -#### Scalar Components And Equality - -Each identified record must be a mapping, and every designated identity field -must be present. Components may be strings, numbers, booleans, or explicit null. -Missing fields and mapping or sequence components are invalid. A present null is -a valid component and is distinct from an absent field. Two single-field null -identities are equal; in a compound identity, all components must compare equal. - -Strings compare exactly, with no trimming, case folding, or normalization. -Empty strings remain valid. Scalar categories remain distinct: the string -`"1"`, the number `1`, the boolean `true`, and null are different identities. - -Numbers compare by mathematical value across integer, fractional, and exponent -spellings: `1`, `1.0`, and `1e0` are equal, as are `0` and `-0`. Use exact numeric -normalization for identity comparison without rounding through a floating-point -conversion. This comparison does not rewrite authored values or alter existing -typed decoding and projection semantics. The existing parser's rejection of -non-finite numeric forms remains unchanged. - -#### Agreement And Introduction - -Within a strict tier, identity declarations for the same collection must agree. -Equivalent field sets coalesce; different sets produce an error. Across tiers, -append and replacement by identity use one agreed definition for the records -being combined: - -| Lower collection identity | Higher declaration | Result when combining records | -| --- | --- | --- | -| `id` | Omitted | Use `id` | -| `id` | `id`, `[id]`, or `[id, id]` | Accept the equivalent definition | -| `id` | `[namespace, id]` | Error: different identity definitions | -| None | `id` | Establish `id` and validate all participating records under it | - -OD-10 makes applicable lower-tier YAML identity available during the higher -tier's strict composition, before this precedence combination occurs. - -An application registration supplies identity when configuration omits it and -remains authoritative over explicit declarations. If neither the application nor -either participating collection supplies an identity, append and replacement by -identity fail; ConfigKit does not infer an `id` convention. Introducing identity -for a previously unkeyed lower sequence requires checking both lower and incoming -records, including uniqueness within each contribution. - -#### Replacement And Policy Lifetime - -YAML-owned identity follows the collection that survives an operation. OD-12 -also permits declarations pending at absent or non-sequence destinations. - -| Operation | Effective YAML-owned identity | -| --- | --- | -| Append | Retain or establish the agreed identity | -| Replacement by identity | Retain or establish the agreed identity | -| Whole-sequence replacement | Use the replacing tier's declaration and discard the lower collection's YAML-only identity | -| Replacement by null or another non-sequence value | Discard the lower YAML-only identity; retain any identity explicitly declared by the replacing tier under OD-12 | - -Without an application constraint, whole-sequence replacement may introduce a -different identity, such as replacing an `id`-keyed collection with a `name`-keyed -collection. A whole-sequence replacement with no new identity declaration and no -application registration produces an unkeyed sequence. No separate identity -removal directive is needed. - -Application registrations continue to constrain the replacement collection. -If the application requires `id`, declaring `name` is invalid under every mode. -The policy reset above applies when the collection value is actually replaced. -OD-6 defines the value effects of absence, null, and kind mismatches; OD-12 -defines the corresponding declaration-only and identity-lifetime rules. -YAML-only identity declarations receive assembly-time enforcement, as settled -in OD-1 below. - -### Combination Scope, Declarations, And Ordering - -**OD-4: settled.** - -#### One Operation Per Collection Per Tier - -Resolve applicable collection policy, including available lower-tier identity -under OD-10, before strictly composing a tier's contributions. Then apply one -selected operation per collection against the accumulated lower sequence. A collection declaration -governs all contributions to that collection in its tier, including imported and -inline records; it does not govern only records in the declaring file. - -Collection scope is the resolved destination within its composition scope. -Resolve relative destinations first and preserve separate item-local scopes for -collections inside different imported items. Apply the same combination -semantics to selected overlay contributions. - -For example, with an agreed identity, higher-tier fragments contributing `[C]` -and `[D]` assemble `[C, D]`. Against lower `[A, B]`, append or replacement by -identity produces `[A, B, C, D]`; whole-sequence replacement produces `[C, D]`. -The tier operation does not sequentially replace peer source contributions. - -#### Operation Agreement And Omission - -Coalesce agreeing explicit modes for the same collection within a tier and -reject conflicting explicit modes. Omission expresses no preference; apply the -effective-identity default only after reconciling declarations and checking for -an applicable explicit mode retained from a lower tier. - -| Modes declared within one tier | Effective operation | -| --- | --- | -| `append` and `append` | `append` | -| `append` and omitted | `append` | -| `replace` and omitted | `replace` | -| `append` and `replace` | Error | -| All omitted | Retain an applicable lower explicit mode; otherwise infer from effective identity | - -Explicitly selecting the default is a real instruction and can conflict with -another explicit selection. Different modes in different tiers are valid; the -higher tier's explicit mode replaces the lower mode, while omission retains it. -Neither a root declaration nor a later declaration wins over conflicting peers. -Diagnostics identify the collection -destination and conflicting declaration locations. OD-5 defines mode names and -the default-selection rule. - -#### Strict Duplicate Handling - -Duplicate identities among peer contributions or within one contributed list -remain strict-composition errors under all three modes. For example, peer -fragments contributing `[B]` and `[B*]` conflict even under replacement by -identity: neither has higher precedence. Lower `[B]` and higher `[B*]` can -instead produce `[B*]` at a precedence boundary. Whole-sequence replacement -does not permit duplicate identities within an identified higher-tier sequence. - -#### Ordering - -Retain existing dependency-ready source scheduling within each tier and authored -item order within each list. Gathering collection declarations does not change -assembly order, and identity values do not sort records. - -Across tiers: - -- Append retains lower records in order, followed by higher records in assembled - order. -- Whole-sequence replacement retains only the higher sequence, in its assembled - order. -- Replacement by identity replaces matched records in their existing lower - positions and appends new identities in higher-tier order. - -For example, lower `[A, B, C]` and higher `[C*, A*, D]` produce -`[A*, B, C*, D]` under replacement by identity. Updating a record's contents -preserves its position; whole-sequence replacement gives the higher tier control -of the complete sequence order. - -OD-5 settles default selection and operation naming. OD-6 settles absent -contributions, empty sequences, null, and kind mismatches. - -### Mode Names And Effective-Identity Defaults - -**OD-5: settled.** - -Use the mode names `replace`, `append`, and `upsert`, with one spelling per -operation: - -| Mode | Behavior at source-precedence and inheritance boundaries | -| --- | --- | -| `replace` | Replace the complete lower sequence with the assembled higher sequence | -| `append` | Retain lower records and append higher records; duplicate identities fail | -| `upsert` | Replace matching records in full, retain unmatched lower records, and append new identities | - -`upsert` preserves the existing positions of matching lower records and adds new -identities in higher-tier order, as settled in OD-4. It does not recursively merge -fields within matching records. - -Select the operation at each collection's precedence boundary in this order: - -1. Reconcile explicit modes within the current tier under OD-4; conflicting - explicit modes fail. -2. If the current tier supplies an explicit mode, use it in place of a lower mode. -3. Otherwise, retain an applicable explicit mode from a lower tier. -4. If no explicit mode applies, select `upsert` when effective identity is available. -5. Otherwise, select `replace`. - -Effective identity may come from an application registration, a current-tier -declaration, or the lower collection, subject to OD-3's agreement rules. -Selection depends on identity availability, not on whether the current tier -explicitly repeats a `key` declaration. With no applicable explicit mode, a plain -overlay sequence can therefore default to `upsert` when identity was established -elsewhere. New sequence values do not reset an existing explicit mode. Inferred -defaults do not become persistent explicit instructions: when no explicit mode -applies, determine the default from the identity available at that boundary. - -Determine the inferred mode using the available identity before applying the -operation. `replace`, including a retained explicit mode, follows OD-3: discard -the lower collection's YAML-only identity with the lower sequence, use the -replacing tier's identity declaration if supplied, and preserve application-owned -constraints. Selecting -`append` or `upsert` explicitly still requires an effective identity. - -This default governs combination between tiers and during sibling inheritance -under OD-7. Within a tier, identified -sequences append distinct records and reject duplicate identities regardless of -mode. Unidentified sequences retain their existing strict-composition conflict -behavior; this decision does not introduce unkeyed concatenation. - -### Whole-Record Replacement By Identity - -The `upsert` operation replaces each matching lower record with the -complete higher record, retains unmatched lower records, and adds new identities. -For example, given these lower and higher records: - -```yaml -# Lower record -id: B -text: Original -temperature: 0.5 -``` - -```yaml -# Higher record -id: B -text: Revised -``` - -The resulting record at this precedence boundary contains `id` and `text`. -The lower record's `temperature` is discarded. Subsequent defaulting or other -lifecycle operations retain their separately defined semantics. Recursive -field-by-field overlay of matching records is outside this selected operation. - -### Selected Overlay Control - -Selected overlay authors must be able to choose any of the three operations -for a sequence supplied by the overlay, including records authored inline. -The operation applies against the existing effective sequence at that path. -For an existing sequence `[foo, bar]`: - -| Overlay contribution | Selected operation | Result | -| --- | --- | --- | -| `[foo*]` | Replacement by identity | `[foo*, bar]` | -| `[baz]` | Append | `[foo, bar, baz]` | -| `[bar, baz]` | Whole-sequence replacement | `[bar, baz]` | - -Here `foo*` has the same identity as `foo` and supplies its complete replacement -record. Append retains its duplicate-identity error semantics. The retained -records, replacement behavior, and ordering shown are settled, including the -OD-4 rule that matched records retain their lower positions. - -The settled OD-2 collection declaration supplies the overlay authoring surface. -OD-5 supplies operation names and effective-identity default selection. The current -[selected-overlay contract](../config.md#selected-overlays) remains the baseline -until the new controls are implemented. - -### Optional Application-Owned Identity Constraints - -Retain an interface through which applications may register a collection and -impose its unique-key definition. If the application requires `key: id`, a -configuration declaration of `key: [id, name]` is invalid: the compound key -would permit repeated `id` values with different names. The application identity -constraint applies whichever cross-tier operation configuration selects. - -This is an authoritative constraint, distinct from an application default that -only fills an omission. Configuration can define identity for collections -without an application-owned constraint. Omitted configuration identity uses a -registered definition when present, and an explicit matching declaration is -accepted, as settled in OD-2. OD-1 below distinguishes the final-result guarantee -for application-owned constraints from assembly-time enforcement for YAML-only -declarations. - -### Final-Result Guarantee For Application Registrations - -**OD-1: settled, including the YAML-only follow-up below.** - -> On successful loading, the final effective collections covered by application -> registrations satisfy their declared identity constraints. - -Retain assembly-time identity checks and validate the final effective -collections before publication. This covers the loaded state after defaults, -inheritance, overrides, and application resolution have taken effect. Those -operations retain their separately defined combination semantics. The guarantee -does not require every intermediate callback state to satisfy the constraint -and does not extend to caller mutations after loading returns. - -Keep the promise limited to the declared identity constraints. It does not by -itself require registered paths to exist or be non-null. Identity validation checks -actual sequences; the application schema and validation determine whether absence, -null, or another shape is acceptable, as settled in OD-6. Final validation examines -effective typed values, while assembly-time checks retain their role in -validating authored contributions. A final scalar value alone does not prove -that the field was explicitly supplied in a source. - -The existing canonical final-value representation and shared publication path -provide the implementation foundation. Diagnostics should identify the affected -collection and item paths, using available origin information without exposing -identity values. The guarantee requires neither a new mutation-history system -nor exact causal reconstruction for records supplied by defaults or callbacks. - -### Assembly-Time Enforcement For YAML-Only Identities - -Enforce identities declared only in YAML during structural composition, -precedence assembly, and identified collection operations during sibling -inheritance. They do not carry a final-result guarantee after defaulting, CLI -overrides, or application resolution. Those operations may produce a collection -that no longer satisfies a YAML-only identity, including duplicate identities. -OD-7 defines the combination and validation boundaries. - -Applications requiring a final identity constraint register it through the -application-owned interface. This keeps YAML declarations aligned with the -structural operations they control and avoids extending their policy lifetime -through later replacements, callbacks, and nested-record changes solely for -final validation. - -### Absence, Empty Sequences, Null, And Kind Mismatches - -**OD-6: settled.** - -At a collection's precedence boundary: - -| Higher-tier contribution | `replace` | `append` / `upsert` | -| --- | --- | --- | -| Absent | Leave the lower value unchanged | Leave the lower value unchanged | -| Explicit `[]` | Clear the sequence | Contribute no records | -| Explicit null or another non-sequence value | Ordinary replacement, subject to application schema | Error: the operation requires a sequence | - -A declaration alone does not clear a collection. OD-12 defines how a key-only -declaration validates a retained sequence or remains pending when no sequence -exists. An optional import selecting nothing contributes nothing; it does not -erase records supplied elsewhere. -An absent lower collection can be initialized by an incoming sequence under -any mode, subject to identity requirements. An explicitly null or other -non-sequence lower value is distinct from absence: append and upsert against it -fail, while replacement remains available. An empty incoming sequence does not -bypass these kind checks or the operation's identity requirements. - -With effective identity available and no applicable explicit mode, omission -selects upsert. Consequently, `prompts: []` alone does not clear such a collection. -Clearing it requires replace, selected explicitly in the current tier or retained -from a lower declaration. Replacing an identified collection with null likewise -requires replacement and an application schema that permits null. - -An application identity registration does not itself require the collection path -to exist or be non-null, and does not independently reject non-sequence values. -Identity validation checks actual sequences; application schema and validation -own the permitted shape. Null as a collection is distinct from a present null -identity component inside a record, which is valid under OD-3. - -### Lifecycle Coverage And Sibling Inheritance - -**OD-7: settled.** - -Apply the sequence operations during source composition and precedence assembly, -selected overlays, and sibling inheritance through `extends`. Preserve the -existing combination semantics of `Definition.Defaults`, CLI overrides, and -application resolution callbacks. - -#### Inheritance Operations And Controls - -Complete source-tier assembly before resolving inheritance, retaining the -existing dependency-ready inheritance algorithm and scope rules. At each -inheritance combination, the resolved parent's collection is the lower value -and the inheriting mapping's local collection is the higher value. Use the same -operation selection as source precedence: the effective explicit mode wins, -including one retained under OD-9; otherwise use `upsert` with effective identity -and `replace` without it. There is no separate -default-append rule for inheritance. - -Use the inheriting destination collection's controls. Its selected mode governs -both source-precedence and inheritance combinations at that destination, without -separate per-phase mode settings. A mode governing the parent's source collection -does not automatically become an instruction at the inheriting destination merely -because records are inherited. Retain the relevant controls until inheritance -uses them, then keep them out of application decoding. Identity agreement, -application constraints, and replacement lifetime follow OD-3. - -For example: - -```yaml -configkit: - sequences: - - destination: [parent, items] - key: id - -child: - items: - - id: foo - value: A - - id: bar - value: B -parent: - extends: child - items: - - id: baz - value: C -``` - -Despite the mapping names, `parent` inherits from `child` in this example. -The effective identity at `[parent, items]` selects `upsert`, yielding records -`[foo, bar, baz]`. If the local contribution instead supplies a revised `foo`, -the result is `[foo*, bar]`: replace the complete record in its existing position. -Adding `mode: append` would reject that repeated identity; `mode: replace` would -retain only the local sequence. - -Mappings continue to combine recursively. Identified sequence records remain -whole-record replacements, not recursive field merges. Identity supplies the -matching key; ConfigKit does not infer identity from a field named `id` or from -record position. Append and upsert continue to require effective identity, and -this extension does not introduce unkeyed sequence concatenation. - -Apply OD-4 ordering and OD-6 presence and kind rules at inheritance boundaries. -An omitted local sequence retains the inherited sequence. An explicit empty -sequence clears it under replace, while append and upsert contribute no records. - -#### Other Lifecycle Stages - -Source-backed underlays participate in the new combination model, including -application-supplied mapping sources. Typed `Definition.Defaults` remain the -initial decoded state: supplied sequences replace default sequences. Source -`mode: append` does not append to a typed default list. - -CLI overrides retain explicit replacement for supplied sequences. OD-8 moves -their application after inheritance and typed defaulting, before application -resolution, secret lookup, and final validation. -Application resolution retains its existing assignment and recorded-transformation -semantics; ConfigKit does not automatically combine a callback's assigned sequence -with its previous value. Application-registered identities still receive final -validation under OD-1, independently of how these stages produce their values. - -#### Validation Timing - -Allow item-local helper imports to complete, and consume applicable non-main -source-version headers, before checking the application payload's identity for -collection assembly. A helper may supply an identity field; a consumed header -cannot supply a component after removal. Validate each identified contribution -before combination and uniqueness in the resulting identified collection, -including inheritance operations. An upsert match across a precedence boundary -is a replacement, not a duplicate error; duplicates within a contribution remain -errors. - -Later inheritance, typed defaults, or callbacks do not repair an identity missing -at an earlier assembly boundary. Final application-registered validation examines -the final effective typed collection, while YAML-only enforcement remains limited -to the structural operations described above. - -### CLI Precedence And Application Resolution - -**OD-8: settled.** - -CLI overrides are the highest-precedence input to application resolution. Use -this lifecycle order: - -1. Assemble source tiers and selected overlays. -2. Resolve sibling inheritance, then apply typed defaults and decode the assembled - configuration. -3. Apply explicit CLI overrides to that configuration and check supplied values - against the application schema. -4. Run application resolution against the overridden configuration. -5. Resolve effective secret references, validate the final configuration including - application-registered identities, and publish it. - -Sequence values supplied by the CLI replace whole sequences, including sequences -inside supplied mappings. Mappings retain ordinary recursive override behavior; -empty sequences and permitted nulls retain their explicit replacement meaning. -Configuration-selected append or upsert does not reinterpret CLI assignments. -Inheritance and defaulting do not run again after CLI application. For inherited -and assembled `[A, B]` overridden by CLI `[C]`, application resolution receives -`[C]`; inheritance cannot reintroduce `[A, B]` afterward. - -Application resolution sees CLI values and their explicit-presence and source -metadata. Existing `Resolution.Default` behavior preserves explicit CLI values, -while normalization and recorded application transformations may still change -them. CLI input is authoritative over sources, inheritance, and typed defaults; -it is not immutable during application-owned resolution. For example, a derived -URL can use a CLI-supplied hostname, while an explicitly supplied CLI URL can be -preserved by an absence-based default. Retain existing recording and provenance -requirements for those transformations. - -Secret lookup uses the effective references after both overrides and resolution. -Final validation remains after these stages; neither CLI replacement nor -application resolution bypasses identity constraints or semantic validation. - -Reject structural directives such as `extends` and `configkit` in CLI override -paths or supplied values. CLI overrides supply application values after structural -processing; they do not initiate a second inheritance pass. This deliberately -changes the current ability to supply or cancel `extends` through CLI input. -CLI changes cannot prevent an earlier inheritance failure or change which -relationships that completed pass resolved. Likewise, because typed decoding -precedes CLI application, an invalid source value that fails decoding cannot be -repaired by a CLI replacement. For example, a string supplied for an integer -field fails before a valid integer override can apply. This differs from the -current override-before-decoding behavior. Semantic validation remains after -overrides and application resolution, as specified above. - -This ordering supersedes the current CLI-before-inheritance lifecycle described -in [ADR 0009](../adr/0009-sibling-inheritance.md). It avoids special replacement -markers carried through inheritance. Current contracts remain the implemented -baseline until the architectural and implementation work lands. - -### Persistent Operation Policy - -**OD-9: settled; this revises the earlier tier-local mode rule in OD-4 and OD-5.** - -An explicit mode is collection policy that persists at its applicable destination -until a higher-priority declaration changes it. Reconcile peers strictly within -each tier, then let a higher explicit mode replace a lower one. Omission retains -an applicable explicit mode regardless of whether the higher tier supplies new -sequence values. Source precedence and subsequent inheritance use the same -effective operation policy. - -| Lower explicit mode | Higher declaration | Effective mode | -| --- | --- | --- | -| `append` | Mode omitted | `append` | -| `append` | `upsert` | `upsert` | -| `upsert` | `replace` | `replace` | -| None | Mode omitted, effective identity available | Inferred `upsert` | -| None | Mode omitted, no effective identity | Inferred `replace` | - -An unrelated overlay leaves the policy intact. A mode-only declaration may change -the policy without supplying records; it neither clears the collection nor -retroactively recombines earlier tiers. A later source contribution or inheritance -operation uses the resulting policy. Supplying values alone does not reset it. - -Only explicit modes persist. Inferred defaults are recomputed from effective -identity when needed. Thus explicitly selecting replace for an identified -collection continues to select replace in subsequent tiers until another mode is -declared; an author wanting upsert afterward must select it explicitly. Identity -availability and constraints still follow OD-3, including YAML-only identity -removal on whole-sequence replacement. Persisting a mode does not preserve an -identity that OD-3 discards. - -This persistence is scoped to the applicable collection. It does not transfer a -parent mapping's mode automatically to another destination through `extends` or -resurrect policies owned by discarded records. OD-11 addresses nested policy -ownership and transfer, including the distinction between surviving records and -inheritance copies. CLI overrides remain outside sequence mode selection -under OD-8. Strict peer composition and all ordering rules in OD-4 are unchanged. - -### Lower-Tier Identity During Strict Composition - -**OD-10: settled.** - -Allow identity established in accumulated lower tiers to inform strict composition -of a higher tier. Resolve applicable collection identity before combining that -tier's peer contributions, using application registrations, available lower-tier -identity, and declarations gathered within the higher tier's relevant scope. -Authors need not repeat a baseline identity declaration solely because the higher -tier contributes the collection through multiple fragments. - -For example, an underlay declares `key: id` for `prompts`; two main-tier fragments -contribute records `[foo]` and `[bar]` without redeclaring a key. The inherited -identity enables those fragments to compose as `[foo, bar]` before the main tier -combines with the accumulated lower collection. If both fragments instead -contribute `foo`, strict composition fails. A selected upsert mode never grants -precedence to peers within the tier. - -This does not turn lower YAML identity into an application-owned constraint. -Determine applicability using OD-3's agreement and replacement rules: append and -upsert require agreement, while whole-sequence replacement may change or discard -YAML-only identity. A replacing collection must not be constrained by an obsolete -lower identity merely because policy resolution precedes strict composition. -Application registrations remain authoritative. Under OD-11, a replacement -record also cannot borrow item-local identity owned by the discarded lower -record, even if both records have the same outer identity or numeric position. -Higher-tier declarations do not retroactively repair invalid lower-tier -composition. - -Separate source acquisition from policy-aware composition, or otherwise revise -assembly sequencing, so accumulated lower policy is available before higher-tier -strict composition. The current main-before-underlay composition order is not a -constraint on this design. Preserve the distinction between source acquisition -order, strict within-tier dependency-ready scheduling, and precedence order. -The implementation plan must account explicitly for source records, callbacks, -main-version checks, and failure diagnostics when changing the handoffs. Retain -version ownership and validation semantics; do not use higher-tier replacement -to hide lower-tier structural failures. - -### Nested Policy Ownership And Transfer - -**OD-11: settled.** - -Attach item-local collection policies to their owning contributions. Policies -follow surviving records independently of numeric indexes. Whole-record upsert -replacement discards the old record's local policies along with its fields; -whole-sequence replacement likewise discards policies owned by removed records. -A replacement record supplies its own local declarations. Sharing an outer -identity or occupying the same final index does not transfer the discarded -record's policies to its replacement. - -For example, a source imported as one record into `prompts` can declare: - -```yaml -configkit: - sequences: - - destination: [steps] - key: id - mode: append -id: foo -steps: - - id: prepare -``` - -The declaration governs `steps` inside that particular `foo` record. Retaining -`foo` retains its local policy, including its explicit mode under OD-9. Upserting -a new `foo` discards the old `steps` and its declaration; the replacement does not -inherit either merely because upsert retains the outer record's position. - -Distinguish inherited collection identity from operation instructions. YAML -identity follows a collection copied through sibling inheritance, subject to -OD-3's agreement and replacement rules. The inheriting destination selects its -operation under OD-7 and OD-9; the source collection's mode does not automatically -become an instruction at the new destination. - -For example, a source mapping's `steps` collection has `key: id` and -`mode: append`. An inheriting mapping supplies local `steps` but no applicable -explicit mode. The copied identity selects upsert at the inheriting destination; -the source's append mode remains applicable at its own destination. An explicit -append declaration at the inheriting destination instead selects append there. -A destination declaration of the same key agrees; a different key conflicts when -combining records, while explicit replacement can establish a new YAML identity -under OD-3. - -Application registrations remain constraints at their registered destinations; -copying a collection does not create another application registration. Policies -that independently apply at the destination are not discarded merely because -an individual record is replaced. Ownership determines which declarations are -removed. This preserves OD-9's persistence rule for applicable explicit modes -without retaining declarations owned by discarded records. The implementation -plan should choose the bookkeeping needed to preserve these distinctions. - -### Identity Declarations Without Sequence Values - -**OD-12: settled.** - -Allow identity declarations without accompanying records. Apply a declaration to -an existing retained sequence immediately, checking identity agreement and -validating its records. Do not recombine earlier tiers. For example, if a lower -tier supplies an unkeyed `prompts` sequence and a higher tier only declares -`destination: [prompts], key: id`, the retained records must satisfy that identity -at this boundary. - -If the collection is absent, retain the declaration in its owning scope for later -contributions without requiring the collection to exist. An otherwise empty -underlay can therefore establish identity for main-tier fragments under OD-10. -Item-local declarations remain subject to OD-11's ownership rules. - -Merely declaring replace without supplying a replacement value does not authorize -changing a retained collection's identity. Actual replacement of the collection -by null or another non-sequence value discards the lower YAML-only identity, -retains any identity explicitly declared by the replacing tier, and preserves -application constraints. This extends OD-3's replacement rule to non-sequence -values. - -An explicit identity declaration can remain pending at an absent or non-sequence -destination without independently imposing a shape requirement. OD-6 still -governs whether an attempted operation on a non-sequence value fails. These rules -allow baseline policy declarations while keeping identity removal tied to actual -replacement. Explicit mode persistence remains governed by OD-9. - -## Architectural Direction - -The settled collection declaration gives imported and inline records at the same -destination a shared identity policy. Declarations are gathered before composing -the relevant scope. - -OD-3 settles identity agreement and lifetime; OD-4 settles tier-wide operation -selection, strict peer composition, and ordering at precedence boundaries. -OD-10 requires resolving applicable lower-tier policy before higher-tier strict -composition, independently of source acquisition order. - -This requires collection policy to survive import expansion through precedence -assembly and the applicable inheritance operations. Correctness includes final -values, projected item paths, source destinations, provenance, history, -sensitivity, and passive resource contexts. -Appending changes final indexes; replacing changes which contributions survive. -The implementation must account for both through the existing loading lifecycle. - -## Scope And Decision Ownership - -This is narrower than the deferred -[identified-record sequences proposal](future.md#1-identified-record-sequences), -which also considers recursive merging of matching records and record-specific -inheritance. Those capabilities remain separate candidates. Compound keys, -duplicate-rejecting append, and whole-record replacement by identity do not -require adopting them. - -The proposal would extend identity declaration ownership and revise the -cross-tier replacement decision in -[ADR 0014](../adr/0014-sequence-placement-and-composition.md), as well as sequence -replacement during inheritance in -[ADR 0009](../adr/0009-sibling-inheritance.md), including its CLI-before-inheritance -lifecycle ordering. Record -the architectural change in a superseding decision and reconcile affected -architecture policy before implementation. The current ADR and contracts remain -authoritative until that decision is made. This draft neither changes them nor -selects migration machinery. - -The settled roadmap is ready for a staged implementation plan. -The plan should cover ordinary and layered loading, item-local scopes, source-version -processing, identity validation, precedence, inheritance, and inspection together. -Public API design and test sequencing belong to that later plan.