Files
ConfigKit/docs/roadmap/sequences.md

563 lines
26 KiB
Markdown

# Sequence Identity And Cross-Tier Combination
Status: draft proposal for discussion. This document captures the requested
capabilities, settled design choices, and initial recommendations. Choices under
Open Decisions remain unsettled.
It is not an implementation plan or an implemented contract. The declaration
structure is settled under OD-2; operation names in examples remain illustrative
pending OD-5.
## Motivation
Allow configuration authors to assemble collections from independently authored
records, declare their identity in configuration, and progressively extend a
baseline collection through higher layers. Authors should also be able to
replace the accumulated collection or replace individual records by identity.
The motivating example is a prompt collection whose defaults supply some
records and whose application or installation layers add or supersede records.
Across tiers, authors need three operations: whole-sequence replacement,
duplicate-rejecting append, and whole-record replacement by identity. The latter
retains unmatched lower-tier records and adds higher-tier records with new
identities. Matching records are replaced in full.
## Current Baseline
The [import contract](../config.md#explicit-imports) owns current selector,
destination, placement, and ordering behavior. An import with
`placement: sequence-item` places each selected source mapping as one item at
its effective destination. Item-local helper imports can contribute fields
before placement.
The application currently registers sequence identity through
`Definition.SequenceCompositions`, using one literal string field at each
registered mapping path. Within a strict tier, distinct identities append and
duplicate identities fail. Repeated unregistered sequences conflict. Across
precedence tiers, a higher sequence replaces the lower sequence wholesale.
The [composition contract](../config.md#strict-document-composition) owns these
rules; [layered loading](../config.md#layered-loading) owns tier precedence.
Source-backed underlays and `Definition.Defaults` have different lifecycle
roles. Underlays participate in source assembly; typed defaults are applied
later. Extending one does not automatically imply extending the other.
## Requested Capabilities
The proposed `configkit.sequences` block introduces a `key` declaration that
accepts either one field or a list of fields for a compound identity.
Fields are treated as a set; identity components may be strings, numbers,
booleans, or explicitly supplied null values.
Applications retain an optional interface for registering a collection
and imposing its identity definition. Configuration declarations must conform
to an application-owned constraint when one exists. The exact Go interface
remains to be designed.
```yaml
configkit:
sequences:
- destination: [prompts]
key: [namespace, id]
mode: append
imports:
- dir: prompts
pattern: '*.yaml'
destination: [prompts]
placement: sequence-item
```
Each selected file would contribute a record such as:
```yaml
namespace: support
id: welcome
text: Welcome
```
Here `mode: append` illustrates retaining a lower-layer collection and adding
distinct records from the higher layer. The `mode` field expresses the operation;
operation names and default behavior remain under OD-5. Identity and combination
controls belong to the collection declaration; import shorthand is deferred.
The intended results can be expressed without settling operation names. Letters
below denote record identities; `B*` is a higher-tier record with the same
identity as `B` and a different payload. The outcomes and ordering shown are
settled below.
| Lower-tier sequence | Higher-tier sequence | Operation | Proposed result |
| --- | --- | --- | --- |
| `[A, B]` | `[C]` | Append | `[A, B, C]` |
| `[A, B]` | `[C]` | Whole-sequence replacement | `[C]` |
| `[A, B]` | `[B*]` | Append | Duplicate-identity error |
| `[A, B]` | `[B*]` | Whole-sequence replacement | `[B*]` |
| `[A, B]` | `[B*]` | Replacement by identity | `[A, B*]` |
| `[A, B]` | `[B*, C]` | Replacement by identity | `[A, B*, C]` |
Whole-sequence replacement discards the lower sequence. A repeated identity in the
retained combined sequence is an error under the proposed append semantics,
including when the records have identical or disjoint payloads.
## Settled Design Choices
### Collection Declarations
**OD-2: settled.**
Use a root `configkit.sequences` block containing a sequence of collection
declarations. Each entry identifies a `destination` and may declare `key`,
`mode`, or both. Application sequences remain ordinary YAML data. This declaration
form serves imported items, inline records, and sequences supplied by ordinary
mapping imports, including inline sequences in selected overlays.
`destination` uses the existing sequence of literal mapping keys. Resolve it
relative to the declaring source's placement: a fragment mounted under
`catalog` can declare `[prompts]` locally. A sequence-item source can declare a
nested collection within its own item without referring to an eventual numeric
index. Preserve those source and item-local scope boundaries.
Identity and mode can be supplied independently. Omitted identity can use an
application registration; an explicit matching declaration is accepted, and a
conflicting declaration is invalid. Whole-sequence replacement does not itself
require identity matching, but application-owned constraints still apply.
Identity-policy lifetime across tiers is settled in OD-3 below.
Mode describes the current tier's combination instruction. A lower tier's
`append` does not silently select append for a later overlay. Gather collection
declarations before composing their relevant scope so their effect does not
depend on which contribution is assembled first. Identity agreement is settled
in OD-3; operation agreement and omission among peers are settled in OD-4.
Permit sequence declarations in selected overlays and make their use independent
of filesystem import enablement. This deliberately changes the current blanket
prohibition on overlay `configkit` declarations. Enabling imports in overlays
is not part of this decision. Consume sequence controls before application
decoding, as with other structural directives.
For example, an overlay can replace a complete record by identity:
```yaml
configkit:
sequences:
- destination: [prompts]
key: id
mode: replace-by-key
prompts:
- id: foo
text: Revised definition
```
An overlay can instead add a distinct record:
```yaml
configkit:
sequences:
- destination: [prompts]
key: id
mode: append
prompts:
- id: baz
text: Another prompt
```
Or it can explicitly select whole-sequence replacement:
```yaml
configkit:
sequences:
- destination: [prompts]
mode: replace
prompts:
- id: bar
text: Bar definition
- id: baz
text: Baz definition
```
Explicitly selecting the default operation is valid. Under the recommended
whole-sequence replacement default, removing the last example's `configkit`
block produces the same result. An author may keep it to make intent explicit.
The names `replace`, `append`, and `replace-by-key` illustrate the three
operations; exact names and the default remain under OD-5.
Start with this common collection declaration and defer `key` and `mode`
shorthand on imports. A single declaration form covers inline overlays and mixed
inline/imported collections without duplicate authoring surfaces. Local wrappers
around sequence values would require distinguishing application data from
control mappings; the existing reserved namespace keeps that distinction clear.
### Identity Definitions And Lifetime
**OD-3: settled.**
#### Field Sets
`key` accepts a string or a nonempty list of strings naming direct, literal
fields in each record. `key: id` and `key: [id]` are equivalent. A name such as
`metadata.id` denotes that literal field, not a nested lookup.
Treat the selected fields as a set: remove repeated names and disregard authored
field order when comparing definitions. For example, `[namespace, id]`,
`[id, namespace]`, and `[namespace, id, id]` define the same identity. An empty
list is invalid because it supplies no identity fields. These equivalence rules
also apply when comparing configuration declarations with application
registrations.
Use a consistent field order internally to construct compound identity tuples.
Preserve component types and tuple boundaries; concatenating unstructured values
must not cause otherwise distinct identities to collide.
#### Scalar Components And Equality
Each identified record must be a mapping, and every designated identity field
must be present. Components may be strings, numbers, booleans, or explicit null.
Missing fields and mapping or sequence components are invalid. A present null is
a valid component and is distinct from an absent field. Two single-field null
identities are equal; in a compound identity, all components must compare equal.
Strings compare exactly, with no trimming, case folding, or normalization.
Empty strings remain valid. Scalar categories remain distinct: the string
`"1"`, the number `1`, the boolean `true`, and null are different identities.
Numbers compare by mathematical value across integer, fractional, and exponent
spellings: `1`, `1.0`, and `1e0` are equal, as are `0` and `-0`. Use exact numeric
normalization for identity comparison without rounding through a floating-point
conversion. This comparison does not rewrite authored values or alter existing
typed decoding and projection semantics. The existing parser's rejection of
non-finite numeric forms remains unchanged.
#### Agreement And Introduction
Within a strict tier, identity declarations for the same collection must agree.
Equivalent field sets coalesce; different sets produce an error. Across tiers,
append and replacement by identity use one agreed definition for the records
being combined:
| Lower collection identity | Higher declaration | Result when combining records |
| --- | --- | --- |
| `id` | Omitted | Use `id` |
| `id` | `id`, `[id]`, or `[id, id]` | Accept the equivalent definition |
| `id` | `[namespace, id]` | Error: different identity definitions |
| None | `id` | Establish `id` and validate all participating records under it |
An application registration supplies identity when configuration omits it and
remains authoritative over explicit declarations. If neither the application nor
either participating collection supplies an identity, append and replacement by
identity fail; ConfigKit does not infer an `id` convention. Introducing identity
for a previously unkeyed lower sequence requires checking both lower and incoming
records, including uniqueness within each contribution.
#### Replacement And Policy Lifetime
YAML-owned identity follows the collection that survives an operation:
| Operation | Effective YAML-owned identity |
| --- | --- |
| Append | Retain or establish the agreed identity |
| Replacement by identity | Retain or establish the agreed identity |
| Whole-sequence replacement | Use the replacing tier's declaration and discard the lower collection's YAML-only identity |
Without an application constraint, whole-sequence replacement may introduce a
different identity, such as replacing an `id`-keyed collection with a `name`-keyed
collection. A whole-sequence replacement with no new identity declaration and no
application registration produces an unkeyed sequence. No separate identity
removal directive is needed.
Application registrations continue to constrain the replacement collection.
If the application requires `id`, declaring `name` is invalid under every mode.
The policy reset above applies when a sequence is actually replaced; effects of
absent contributions, null collections, and kind mismatches remain under OD-6.
These structural rules do not settle final-result enforcement for YAML-only
identity declarations, which remains the OD-1 follow-up.
### Combination Scope, Declarations, And Ordering
**OD-4: settled.**
#### One Operation Per Collection Per Tier
Strictly compose a tier's contributions first, then apply one selected operation
per collection against the accumulated lower sequence. A collection declaration
governs all contributions to that collection in its tier, including imported and
inline records; it does not govern only records in the declaring file.
Collection scope is the resolved destination within its composition scope.
Resolve relative destinations first and preserve separate item-local scopes for
collections inside different imported items. Apply the same combination
semantics to selected overlay contributions.
For example, with an agreed identity, higher-tier fragments contributing `[C]`
and `[D]` assemble `[C, D]`. Against lower `[A, B]`, append or replacement by
identity produces `[A, B, C, D]`; whole-sequence replacement produces `[C, D]`.
The tier operation does not sequentially replace peer source contributions.
#### Operation Agreement And Omission
Coalesce agreeing explicit modes for the same collection within a tier and
reject conflicting explicit modes. Omission expresses no preference; apply the
default only after reconciling declarations.
| Modes declared within one tier | Effective operation |
| --- | --- |
| `append` and `append` | `append` |
| `append` and omitted | `append` |
| `replace` and omitted | `replace` |
| `append` and `replace` | Error |
| All omitted | Default selected under OD-5 |
Explicitly selecting the default is a real instruction and can conflict with
another explicit selection. Different modes in different tiers are valid; a
lower tier's mode does not carry forward. Neither a root declaration nor a later
declaration wins over conflicting peers. Diagnostics identify the collection
destination and conflicting declaration locations. Mode names remain
illustrative pending OD-5.
#### Strict Duplicate Handling
Duplicate identities among peer contributions or within one contributed list
remain strict-composition errors under all three modes. For example, peer
fragments contributing `[B]` and `[B*]` conflict even under replacement by
identity: neither has higher precedence. Lower `[B]` and higher `[B*]` can
instead produce `[B*]` at a precedence boundary. Whole-sequence replacement
does not permit duplicate identities within an identified higher-tier sequence.
#### Ordering
Retain existing dependency-ready source scheduling within each tier and authored
item order within each list. Gathering collection declarations does not change
assembly order, and identity values do not sort records.
Across tiers:
- Append retains lower records in order, followed by higher records in assembled
order.
- Whole-sequence replacement retains only the higher sequence, in its assembled
order.
- Replacement by identity replaces matched records in their existing lower
positions and appends new identities in higher-tier order.
For example, lower `[A, B, C]` and higher `[C*, A*, D]` produce
`[A*, B, C*, D]` under replacement by identity. Updating a record's contents
preserves its position; whole-sequence replacement gives the higher tier control
of the complete sequence order.
OD-5 retains default selection and operation naming. Absent contributions, empty
sequences, null, and kind mismatches remain under OD-6.
### Whole-Record Replacement By Identity
The third cross-tier operation replaces each matching lower record with the
complete higher record, retains unmatched lower records, and adds new identities.
For example, given these lower and higher records:
```yaml
# Lower record
id: B
text: Original
temperature: 0.5
```
```yaml
# Higher record
id: B
text: Revised
```
The resulting record at this precedence boundary contains `id` and `text`.
The lower record's `temperature` is discarded. Subsequent defaulting or other
lifecycle operations retain their separately defined semantics. Recursive
field-by-field overlay of matching records is outside this selected operation.
### Selected Overlay Control
Selected overlay authors must be able to choose any of the three operations
for a sequence supplied by the overlay, including records authored inline.
The operation applies against the existing effective sequence at that path.
For an existing sequence `[foo, bar]`:
| Overlay contribution | Selected operation | Result |
| --- | --- | --- |
| `[foo*]` | Replacement by identity | `[foo*, bar]` |
| `[baz]` | Append | `[foo, bar, baz]` |
| `[bar, baz]` | Whole-sequence replacement | `[bar, baz]` |
Here `foo*` has the same identity as `foo` and supplies its complete replacement
record. Append retains its duplicate-identity error semantics. The retained
records, replacement behavior, and ordering shown are settled, including the
OD-4 rule that matched records retain their lower positions.
The settled OD-2 collection declaration supplies the overlay authoring surface.
OD-5 retains operation naming and default selection. The current
[selected-overlay contract](../config.md#selected-overlays) remains the baseline
until the new controls are implemented.
### Optional Application-Owned Identity Constraints
Retain an interface through which applications may register a collection and
impose its unique-key definition. If the application requires `key: id`, a
configuration declaration of `key: [id, name]` is invalid: the compound key
would permit repeated `id` values with different names. The application identity
constraint applies whichever cross-tier operation configuration selects.
This is an authoritative constraint, distinct from an application default that
only fills an omission. Configuration can define identity for collections
without an application-owned constraint. Omitted configuration identity uses a
registered definition when present, and an explicit matching declaration is
accepted, as settled in OD-2. The final-result guarantee for
application-owned constraints is settled below; YAML-only enforcement remains
an open follow-up to OD-1.
### Final-Result Guarantee For Application Registrations
**OD-1: settled for application-owned constraints.**
> On successful loading, the final effective collections covered by application
> registrations satisfy their declared identity constraints.
Retain assembly-time identity checks and validate the final effective
collections before publication. This covers the loaded state after defaults,
inheritance, overrides, and application resolution have taken effect. Those
operations retain their separately defined combination semantics. The guarantee
does not require every intermediate callback state to satisfy the constraint
and does not extend to caller mutations after loading returns.
Keep the promise limited to the declared identity constraints. It does not by
itself require registered paths to exist or settle null and non-sequence
handling; those collection boundaries remain under OD-6. Final validation examines
effective typed values, while assembly-time checks retain their role in
validating authored contributions. A final scalar value alone does not prove
that the field was explicitly supplied in a source.
The existing canonical final-value representation and shared publication path
provide the implementation foundation. Diagnostics should identify the affected
collection and item paths, using available origin information without exposing
identity values. The guarantee requires neither a new mutation-history system
nor exact causal reconstruction for records supplied by defaults or callbacks.
Final enforcement for identities declared only in YAML is not settled here.
## Architectural Direction
The settled collection declaration gives imported and inline records at the same
destination a shared identity policy. Declarations are gathered before composing
the relevant scope.
OD-3 settles identity agreement and lifetime; OD-4 settles tier-wide operation
selection, strict peer composition, and ordering at precedence boundaries.
This requires collection policy to survive import expansion through precedence
assembly. Correctness includes final values, projected item paths, source
destinations, provenance, history, sensitivity, and passive resource contexts.
Appending changes final indexes; replacing changes which contributions survive.
The implementation must account for both through the existing loading lifecycle.
## Scope And Decision Ownership
This is narrower than the deferred
[identified-record sequences proposal](future.md#1-identified-record-sequences),
which also considers recursive merging of matching records and record-specific
inheritance. Those capabilities remain separate candidates. Compound keys,
duplicate-rejecting append, and whole-record replacement by identity do not
require adopting them.
The proposal would extend identity declaration ownership and revise the
cross-tier replacement decision in
[ADR 0014](../adr/0014-sequence-placement-and-composition.md). If adopted, record
the architectural change in a superseding decision and reconcile affected
architecture policy before implementation. The current ADR and contracts remain
authoritative until that decision is made. This draft neither changes them nor
selects migration machinery.
A staged implementation plan should follow resolution of the questions below.
It should cover ordinary and layered loading, item-local scopes, source-version
processing, identity validation, precedence, and inspection together. Public API
design and test sequencing belong to that later plan.
## Open Decisions
### OD-1 Follow-Up. Final Validation For YAML-Only Identity Declarations
Application-registered collections have the settled final-result guarantee
above. Should a configuration-only identity declaration carry the same
guarantee, or govern structural assembly alone?
**Discussion basis:** Declaration scope and identity-policy lifetime are settled
in OD-2 and OD-3. Revisit final enforcement using the effective identity of each
retained collection. The application-owned guarantee remains independent of
this choice.
**Alternatives:** Applying the same final check to YAML-declared identities
would give both declaration forms consistent enforcement, but requires carrying
the effective policy through later lifecycle stages, including nested records.
Limiting YAML-only identity to assembly keeps policy lifetime narrower, but later
operations could produce a final collection that no longer satisfies that
identity declaration.
### OD-5. What Is The Default Operation And Its Syntax?
Choose names for the three `mode` values and decide the default operation.
OD-2 settles the declaration structure and accepts explicit selection of the
default. OD-4 settles omission as no preference among peer declarations, with the
default applied after reconciliation.
**Initial recommendation:** Keep whole-sequence replacement as the cross-tier
default and make append and replacement by identity explicit. Lists can represent
complete ordered instructions as well as extensible catalogs; an explicit append
communicates the intent to retain
lower-layer records. Use names that distinguish whole-sequence from whole-record
replacement; the examples use `replace`, `append`, and `replace-by-key`.
**Alternative:** Default identified collections to append or replacement by
identity and require explicit whole-sequence replacement. These defaults make
progressive collections concise, but retain lower-layer records unless an author
deliberately replaces the collection. For naming, `upsert` is a concise alternative
to `replace-by-key`, but requires explaining its whole-record replacement behavior
to authors unfamiliar with that term.
### OD-6. What Do Absence, Empty Sequences, And Null Mean?
Can a declaration selecting nothing affect a lower collection? How do explicit
empty sequences, null, and non-sequence values interact with all three operations?
**Initial recommendation:** An optional import selecting nothing contributes
nothing and does not clear lower values. An explicitly supplied empty sequence
with whole-sequence replacement can clear a collection; append or replacement
by identity with an empty sequence contributes no records. Preserve the
distinction between absence and supplied values. Decide
null and kind-mismatch handling explicitly rather than silently treating them
as empty sequences.
**Alternative:** Treat an explicitly declared replacement operation as a reset
even when its import selects nothing. This provides a compact clearing mechanism,
but makes an optional missing file or empty search destructive. Keeping clearing
as an explicit value avoids that coupling.
### OD-7. Which Lifecycle Stages Participate?
Do the new cross-tier operations extend only source-backed tiers, or also
`Definition.Defaults`, inheritance, CLI overrides, and application-created records?
At which boundaries
must identity be validated, including helper-supplied fields and consumed source
version headers?
**Initial recommendation:** Start with source-backed strict composition and
precedence assembly. Validate records at the relevant collection assembly
boundaries and uniqueness across each combined result. Preserve existing
typed-default and application-resolution semantics unless their inclusion is
explicitly chosen. Selected-overlay combination is included in the settled scope;
its declaration mechanism is settled in OD-2. Account for item-local helpers and
source-version consumption before deciding identity is available, consistent
with the current lifecycle.
The settled OD-1 guarantee covers final validation of application constraints.
Validating the final collection does not itself change how defaults, inheritance,
CLI overrides, or callbacks combine values.
**Alternative:** Extend the combination operations through defaulting,
inheritance, overrides, and application resolution. This would support
progressively extending collections across more of the lifecycle, but requires
additional combination, callback, and inspection rules. Supporting typed default
lists is a separate scope choice from extending source-backed baseline
collections; the application-owned final-result guarantee remains independent
of this combination-scope decision.