From e01aeca76118ded08e3f619f28ebabb284976e60 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 23 Oct 2026 21:57:35 +0000 Subject: [PATCH] Reconcile sequence identity documentation --- README.md | 26 +-- docs/adr/0017-collection-sequence-policy.md | 6 +- .../0018-cli-after-structural-resolution.md | 4 +- docs/config.md | 199 +++++++++++------- docs/consumers/README.md | 64 +++--- docs/development.md | 14 +- docs/internal/imports.md | 17 +- docs/internal/overview.md | 24 ++- docs/policy/architecture.md | 14 +- docs/roadmap/future.md | 115 +--------- docs/roadmap/implementation.md | 11 +- docs/roadmap/sequences.md | 5 +- 12 files changed, 239 insertions(+), 260 deletions(-) diff --git a/README.md b/README.md index 641c6a0..e0ef9a4 100644 --- a/README.md +++ b/README.md @@ -27,12 +27,14 @@ the [configuration contract](docs/config.md#effective-configuration-diffs) and [consumer guide](docs/consumers/README.md#effective-configuration-diffs) describe its safe changed-region records and maintained workflows. -Sources can also be explicitly placed as sequence items, and definitions can -register selected sequences to append records with distinct string IDs during -strict composition. Higher-priority sequences still replace lower-priority -sequences as whole values. See the [sequence contract](docs/config.md#strict-document-composition), -[maintained prompt example](docs/consumers/README.md#identified-prompt-sequences), -and [ADR 0014](docs/adr/0014-sequence-placement-and-composition.md). +Sources can also be explicitly placed as sequence items. Configuration +declarations and application registrations can give selected sequences scalar +identities and choose how they combine across precedence and inheritance. +Strict peers combine distinct records, while higher-priority collections can +replace, append, or upsert whole records. See the +[sequence contract](docs/config.md#strict-document-composition), +[maintained catalog examples](docs/consumers/README.md#identified-prompt-sequences), +and [ADR 0017](docs/adr/0017-collection-sequence-policy.md). Imports are also explicit: a source may name files, search declared directories, or select files with path globs. Applications can set load-wide import defaults @@ -78,12 +80,12 @@ precedence tiers, mappings combine recursively, retaining omitted lower-priority children; other values replace lower-priority values. The main document and its imports form one strictly composed tier. Explicit underlay groups sit below it and above typed defaults; a selected overlay applies before inheritance, while -explicit CLI overrides apply to the decoded configuration before application -resolution. The [configuration contract](docs/config.md#strict-document-composition) -owns the detailed combination rules and input-specific constraints. At -definition-registered paths only, strict composition can append records with -distinct explicit IDs; ordinary and unregistered sequences keep their conflict -or whole-value replacement behavior. +explicit CLI overrides apply after decoding and before application resolution. +The [configuration contract](docs/config.md#strict-document-composition) owns +the detailed combination rules and input-specific constraints. Declared or +registered collections combine distinct identified records within strict tiers +and use their selected operation at precedence boundaries; other sequences keep +their strict-conflict and whole-value-replacement behavior. The [architecture goals](docs/policy/architecture.md#success-criteria) explain these priorities and how genuine new needs can justify extending the shared diff --git a/docs/adr/0017-collection-sequence-policy.md b/docs/adr/0017-collection-sequence-policy.md index 8ddc063..676ffb2 100644 --- a/docs/adr/0017-collection-sequence-policy.md +++ b/docs/adr/0017-collection-sequence-policy.md @@ -138,6 +138,6 @@ Record inheritance also requires parent lookup by identity and checking for missing parents and cycles. Copied settings retain the same source, history, and sensitivity information as mapping-sibling inheritance. -The configuration contract will define declaration syntax, identity comparison, -omission and empty-value behavior, and diagnostics when the feature is -implemented. +The [configuration contract](../config.md#strict-document-composition) defines +declaration syntax, identity comparison, omission and empty-value behavior, and +diagnostics for the implemented feature. diff --git a/docs/adr/0018-cli-after-structural-resolution.md b/docs/adr/0018-cli-after-structural-resolution.md index 3a9530a..4c6d3ba 100644 --- a/docs/adr/0018-cli-after-structural-resolution.md +++ b/docs/adr/0018-cli-after-structural-resolution.md @@ -80,5 +80,5 @@ must remain consistent throughout loading. CLI input can no longer add or cancel inheritance relationships or bypass an earlier error in source assembly or inheritance. It also cannot fix a value that fails the earlier decoding step. Semantic validation still runs after overrides -and application resolution. The configuration contract and consumer guidance -must explain these behavior changes when they are implemented. +and application resolution. The [configuration contract](../config.md#explicit-cli-overrides) +and [consumer guidance](../consumers/README.md) describe this behavior. diff --git a/docs/config.md b/docs/config.md index ea4999b..85c0466 100644 --- a/docs/config.md +++ b/docs/config.md @@ -814,9 +814,10 @@ validated, but there are no imported peers to combine. | Any other pair of present nodes | Any other pair of present nodes | `compose.conflict` | Every non-mapping overlap is a conflict, even when values are equal, except for -sequence/sequence contributions at a registered sequence identity destination -or a source-declared collection destination. Nulls remain atomic conflicts. -Unregistered and undeclared sequences conflict when repeated and are never joined +sequence/sequence contributions at a destination with effective identity from an +application registration, a current YAML `key` declaration, or retained lower +policy. Nulls remain atomic conflicts. +Sequences without effective identity conflict when repeated and are never joined by index. Disjoint entries under shared mappings compose, making stable keyed mappings the natural representation for independently contributed entries. @@ -827,44 +828,49 @@ segments created by `SequenceLiteralKey` and `SequenceAnyMappingKey`. A literal `"*"` key is distinct from the wildcard constructor. Parsed numeric path tokens remain literal mapping keys; explicit index segments and reserved `configkit` or mapping-valued `extends` segments are invalid. The root selector is harmless under -the mapping-root schema. Each registration requires one or more literal field -names; compound field sets are normalized as sets, and field names are not -otherwise normalized. +the mapping-root schema. Selector segments consume one mapping level and match a +complete destination; an any-mapping-key segment matches exactly one key, not an +arbitrary prefix or suffix. Each registration requires one or more literal, +direct field names; compound field sets are normalized as sets, and field names +are not otherwise normalized. The same field-set and scalar-equality rules apply +to YAML `key` declarations. -At each selected path, records must provide every registered field as a scalar -string, number, boolean, or explicit null. The type-tagged tuple preserves -category distinctions, while equivalent finite decimal spellings such as `1`, -`1.0`, and `1e0` compare as the same number. Distinct tuples append in source and -list order; duplicate tuples conflict even when records are otherwise equal or -disjoint. Malformed records produce `compose.invalid`; duplicate identities +At each selected path, records must provide every effective identity field as a +scalar string, number, boolean, or explicit null. Field names form a set, so +order and repetition do not affect compound identity. The type-tagged tuple +preserves category distinctions: strings compare exactly, including empty +strings; numbers, booleans, and null remain distinct categories. Finite numbers +compare by exact mathematical value, so `1`, `1.0`, and `1e0` are equal, as are +`0` and `-0`, without floating-point rounding. Distinct tuples append in source +and list order; duplicate tuples conflict even when records are otherwise equal +or disjoint. Malformed records produce `compose.invalid`; duplicate identities produce `compose.conflict`, with paths and source locations identifying a duplicate without exposing identity values. Missing paths and non-sequence values -do not fail by themselves. Equivalent overlapping registrations coalesce; -incompatible overlaps and invalid registrations produce `options.invalid` before +do not fail by themselves. Equivalent overlapping registrations coalesce. +Incompatible overlaps and invalid registrations produce `options.invalid` before source acquisition. Unused and root registrations are allowed and have no effect. -Registration controls strict composition and supplies identity fields for parent -lookup when surviving records in that sequence use `extends`. Dynamic selectors -apply at strict-composition and inheritance destinations without requiring the -application to enumerate mapping keys. Each underlay group and the main tier has -independent duplicate-identity state. Precedence assembly is unchanged: a present -higher-priority sequence replaces the lower sequence as one value, including at a -registered path. +Registrations provide identity for strict composition, sequence operations, +inheritance lookup, and final validation. Dynamic selectors apply at concrete +strict-composition, inheritance, and final-result destinations without requiring +the application to enumerate mapping keys. Each strict tier has independent +duplicate-identity state among its peers; layered precedence and inheritance +then apply the collection operation described below. -After effective secret lookup, the shared load lifecycle validates every matching -sequence in the final typed projection before application semantic validation. -This catches identities introduced by typed defaults, CLI assignment, or recorded -resolution, in both ordinary and layered loads. Only actual sequences are checked; -absent paths and other shapes remain the schema or application validator's concern. -Identity fields are read from effective typed payload regardless of authored -presence. Diagnostics use concrete collection and item paths and available source -locations, and never include identity or credential values. The guarantee ends at -successful publication; later caller mutations are outside it. The accepted -rationale is recorded in -[ADR 0014](adr/0014-sequence-placement-and-composition.md). -The [maintained prompt catalog](consumers/README.md#identified-prompt-sequences) -shows item path and glob imports, helper-supplied identity, and main-tier -replacement of a lower-priority sequence. +Final registration checks run in both ordinary and layered loads after effective +secret lookup and before application semantic validation. They examine actual +final typed sequences, including values introduced by defaults, CLI, or recorded +resolution; they do not require path existence or sequence shape. Diagnostics +use concrete collection and item paths and available source locations, without +exposing identity or credential values. The guarantee ends when loading publishes +successfully and does not cover later caller mutations. +YAML-only identity is enforced during structural composition, precedence, and +identified inheritance; it carries no final-result guarantee after typed +defaults, CLI, or application resolution. Applications that need that guarantee +register the identity constraint. +The [maintained catalog examples](consumers/README.md#identified-prompt-sequences) +show strict peers, precedence operations, inheritance, dynamic destinations, and +the distinction between YAML-only declarations and application registrations. When an enabled source-version mode is selected, declarations are checked in each source's own coordinates before placement and composition. Accepted non-main headers are consumed before strict composition and cannot supply a registered @@ -872,34 +878,74 @@ record identity. Final typed decoding and application validation still run on th completed configuration. Ordinary sources, layered inputs, and selected overlays may declare collection -behavior under `configkit.sequences` at the document root. Each entry has a required literal -`destination` and may specify `key` as one string or a nonempty list of strings, -plus a `mode` of `replace`, `append`, or `upsert`. These controls are removed -before application decoding. A declaration destination is relative to that -source's placement, so a mounted fragment's declaration is mounted with its -values. Declarations from -strict peers must agree on identity fields and on any explicitly supplied mode; -they apply even when the declaring source has no value at that destination. -Singleton sequence contributions are validated as well as peers. A key supplies -the identity used by `append` and `upsert`; without an explicit or retained mode, -a keyed collection uses `upsert`, while an unkeyed collection uses `replace`. -`append` keeps contribution order and rejects repeated identities. `upsert` -replaces a complete matched record at its existing position and appends new -identities in incoming order; it does not merge record fields. +behavior under `configkit.sequences` at the document root. Each entry has a +required literal `destination` and may specify `key` as one string or a nonempty +list of strings, plus a `mode` of `replace`, `append`, or `upsert`. These +controls are removed before application decoding. A declaration destination is +relative to that source's placement, so a mounted fragment's declaration is +mounted with its values. Declarations from strict peers must agree on identity +fields and on any explicitly supplied mode; omission expresses no mode +preference. They apply even when the declaring source has no value at that +destination: a key validates an existing retained sequence or remains pending +until records arrive, while a mode-only declaration affects later combinations +without recombining earlier tiers. A key supplies the identity used by `append` +and `upsert` and is checked against any application registration. + +Strict peers at an identified destination first combine distinct records in +source and list order. Duplicate identities among peers or within one +contribution fail under every mode. After strict composition, precedence and +inheritance use one operation for the assembled incoming collection: + +| Mode | Behavior at source-precedence and inheritance boundaries | +| --- | --- | +| `replace` | Replace the complete lower sequence with the assembled higher sequence. | +| `append` | Retain lower records and append higher records; duplicate identities fail. | +| `upsert` | Replace matching records in full at their lower positions, retain unmatched lower records, and append new identities. | + +Upsert replaces the complete matched record; it does not recursively merge +record fields. Append and upsert require identity. With no explicit mode in the +current tier, an explicit lower mode persists; if none exists, effective identity +selects upsert and its absence selects replace. Inferred modes do not persist. +Explicit modes persist at their destination until changed by a higher explicit +mode, independently of YAML identity lifetime. A declaration alone does not +clear or recombine a collection. + +The presence and value kind at a precedence or inheritance boundary determine +whether a collection changes: + +| Higher contribution | `replace` | `append` / `upsert` | +| --- | --- | --- | +| Absent | Leave the lower value unchanged. | Leave the lower value unchanged. | +| Explicit `[]` | Clear the sequence. | Contribute no records. | +| Explicit null or another non-sequence value | Replace the lower value, subject to application schema. | Error; the operation requires a sequence. | + +An absent lower value can be initialized by an incoming sequence under any mode. +A present lower null or other non-sequence value is not absence: append and +upsert cannot combine with it. An empty sequence still requires a valid mode and +identity where applicable. Null used as an identity component inside a record is +distinct from null used as the collection value. Declarations govern ordinary strict source composition, including loads with -imports disabled, and each strict scope in layered loading. Layered groups and -main apply their resolved sequence operation between precedence tiers; a higher -strict scope inherits the lower identity and explicit mode before peer -composition. Overlay declarations apply after the base source graph is composed -and must agree with retained collection identity and any application -registration. During mapping-sibling inheritance, a parent's YAML identity can -follow its collection to the child destination, while the parent's explicit -operation stays at the parent destination. A child declaration selects the -operation at its destination; absent one, keyed collections use upsert and -unkeyed collections use replace. These policies do not change CLI behavior. A root -`configkit.imports` declaration still requires `Options.Imports`; `NoSource` has -no declaration scope. +imports disabled, and each strict scope in layered loading. Before composing a +higher tier's peers, ConfigKit makes applicable lower identity and explicit mode +available; after peer composition, the tier applies its resolved operation +against the lower collection. Selected overlays use the same collection policy +after the base source graph is composed. During mapping-sibling inheritance, a +parent's YAML identity can follow its collection to the child destination, while +the parent's explicit operation stays at the parent destination. The child +destination selects its operation; absent one, effective identity implies +upsert and its absence implies replace. Whole-sequence replacement discards +lower YAML identity and uses the replacing scope's key declaration, if any; +without one, the collection is unkeyed unless an application registration still +supplies identity. Replacement discards policies owned by removed records, while +an independently declared explicit mode and application registration remain +applicable. Item-local +policies follow their owning records: retaining a record retains its rules, while +whole-record upsert replacement discards the displaced record's rules. Matching +identity or reusing an index does not transfer them. These policies do not change +CLI behavior: CLI sequence assignments replace whole sequences after decoding. +A root `configkit.imports` declaration still requires `Options.Imports`; +`NoSource` has no declaration scope. An empty mapping contributes no child definitions. Two empty mappings compose, and an empty mapping neither clears nor conflicts with another mapping's children. @@ -1076,6 +1122,12 @@ filenames unrestricted; a non-nil empty slice selects none. Defaults do not make a field applicable to selectors that do not support it, and load calls snapshot the supplied pointers and slice before source acquisition. +A source's `configkit.sequences` declarations apply independently of import +enablement; a `configkit.imports` declaration still requires `Options.Imports`. +Sequence identities can be supplied either by these YAML declarations or by +application registrations. The [strict composition contract](#strict-document-composition) +defines their agreement, operation, and lifetime rules. + Acquisition visits declarations in authored order and each declaration's selected paths in namespace-path lexical order, preserving source-record and version-callback order. Ordinary mapping contributions in a main tier are assembled dependency-first: @@ -1087,9 +1139,10 @@ mapping scope, with helpers before the bodies that import them. Those local scop do not use application collection policies and are not flattened into the outer schedule. Item positions and diagnostic paths follow the resulting assembly order. Each underlay group likewise acquires its inputs before dependency-ready assembly -in one strict scope. Independent groups remain separate precedence tiers; a higher -tier still replaces a lower tier's sequence values wholesale. Assembly order never -permits replacement. +in one strict scope. Independent groups remain separate precedence tiers; at a +declared or registered destination, the assembled higher sequence uses the +resolved collection operation against the lower sequence. Other sequences retain +whole-value replacement between tiers. Assembly order never permits replacement. Acquisition order and resolution order serve different purposes: acquisition assigns source records and callback order, while the shared scheduler determines @@ -1120,13 +1173,17 @@ Each declaration may include `placement: mapping` or including every match of a glob; placement is not inherited. Sequence-item placement requires a nonempty effective destination and wraps the imported mapping as one element of a sequence there. It does not split source sequences -or infer identity fields. Repeated sequences still conflict unless their path is -registered through `Definition.SequenceIdentities`; registration does not -change placement or precedence behavior. Sequence records retain source -locations through placement and history; indexes describe their assembled -positions while source coordinates continue to identify the declaring files. -Header/version policy and final application validation retain their existing -owners and timing; sequence registration is a strict-composition rule only. +or infer identity fields. Repeated sequences combine only when the destination +has an applicable YAML identity declaration or application registration; source +placement remains independent of collection identity and operation. Sequence +records retain source locations through placement and history; indexes describe +their assembled positions while source coordinates continue to identify the +declaring files. +Header/version policy retains its existing owner and timing. Application +registrations also enforce identity on the final effective typed configuration; +YAML-only declarations are enforced at structural combination and inheritance +boundaries. The [configuration contract](#strict-document-composition) defines +these validation boundaries. An item import first opens a local mapping scope. Its body and ordinary helper imports compose within that record. An omitted helper destination or diff --git a/docs/consumers/README.md b/docs/consumers/README.md index ff7f325..d4766fe 100644 --- a/docs/consumers/README.md +++ b/docs/consumers/README.md @@ -21,7 +21,7 @@ sanitized shapes using only ConfigKit and the Go standard library: | Named presets | One-load sibling inheritance, explicit false, nested destination resolution, and source/derivation inspection | Choosing and executing a preset | | Effective configuration diff | Same-type frozen effective comparisons for two selected files or base-versus-overlay impact | Source/options selection and consumer-owned safe presentation | | Shared source-version checks | One `Definition.Version` policy for accepted declarations across participating source documents, with optional or required declarations and sanitized evidence | Choosing which declarations the application accepts; final assembled schema and domain validation | -| Identified prompt sequences | Explicit item path/glob imports, helper-supplied IDs, reusable single-field registration, same-sequence record inheritance, and main-tier replacement of a lower-priority collection | Prompt meaning and final domain validation | +| Sequence catalogs | Declared collection operations, record inheritance, typed CLI and resolution, inspection, exact and wildcard application registrations | Catalog meaning and final domain validation | The YAML under [`testdata`](../../examples/consumers/testdata) is loaded by the executable examples and ordinary tests. It contains no private application data, @@ -184,43 +184,35 @@ import, source-location, and ownership behavior. ## Identified prompt sequences -[`LoadPromptCatalog`](../../examples/consumers/prompt_catalog.go) uses one -definition registration for the literal `prompts` path and `id` field. Its -load-wide filename defaults accept two patterns, an explicit directory -declaration overrides those patterns, and an exclusion omits one matching -candidate. A nested ordinary helper import supplies an item's ID. A separate -empty optional directory search is read from `SourceInfo.ImportSelections`. -The executable example prints both the scheduled prompt order and the single -empty-search outcome; the behavior test also follows a mounted resource through -its effective metadata origin to the declaring source context. The completed -main sequence replaces the lower-priority default collection as one value. -The maintained [executable example and behavior tests](../../examples/consumers/prompt_catalog_test.go) -show the scheduled result, empty-search inspection, excluded candidate, and -resource lookup without a consumer-side discovery pass. The corresponding -[complex fixture](../../examples/consumers/testdata/prompt-catalog/config.yml) -and [simple required-file fixture](../../examples/consumers/testdata/prompt-catalog/simple.yml) -are complete local examples. +[`LoadPromptCatalog`](../../examples/consumers/prompt_catalog.go) shows the +smaller import-focused adoption path: a registered `prompts.id` identity, +directory and glob selection, a helper import that supplies an item's ID, and +resource lookup through the effective source context. Its executable examples +and tests are in +[`prompt_catalog_test.go`](../../examples/consumers/prompt_catalog_test.go), with +complete [catalog](../../examples/consumers/testdata/prompt-catalog/config.yml) +and [required-file](../../examples/consumers/testdata/prompt-catalog/simple.yml) +fixtures. -[`LoadRequiredPromptCatalog`](../../examples/consumers/prompt_catalog.go) and -[`Example_requiredPromptCatalog`](../../examples/consumers/prompt_catalog_test.go) -keep the smaller adoption path visible: one explicitly required file import, -the same identity registration, and the same application validation. +[`LoadSequenceCatalog`](../../examples/consumers/sequence_catalog.go) integrates +an underlay declaration, imported main fragments, identified-record inheritance, +selected overlay `append`, `upsert`, and `replace`, a typed CLI override, and +recorded application resolution. Its tests and runnable example demonstrate +stable output order, whole-record upsert, provenance/history inspection, +resource lookup for inherited records, and secret-safe explanation in +[`sequence_catalog_test.go`](../../examples/consumers/sequence_catalog_test.go). +[`LoadNamespacedMapCatalog`](../../examples/consumers/sequence_catalog.go) adds +an exact root registration and a wildcard namespace registration, showing that +each concrete sequence has an independent identity scope. The same tests show +that a YAML identity declaration without an application registration does not +create final-result identity enforcement. -The registered `id` also identifies surviving prompt records that use -`extends: {id: "base-id"}`. ConfigKit resolves these references within the -completed prompt sequence after tier assembly, preserves the child's own ID, and -records inherited values at the child's paths. The -[configuration contract](../config.md#sibling-inheritance) defines the exact -reference and nested collection behavior. - -The example's `validatePromptCatalog` remains application-owned. It checks that -prompt names and message lists satisfy this consumer's requirements after -ConfigKit has composed, inherited, projected, and decoded the configuration. -ConfigKit does not apply identity-based overlays or enforce final uniqueness -after defaults and resolution. See the [strict composition contract](../config.md#strict-document-composition), -[import syntax](../config.md#explicit-imports), and -[ADR 0014](../adr/0014-sequence-placement-and-composition.md) for the exact -boundaries. +The examples keep prompt meaning and domain checks in the consuming application. +The [configuration contract](../config.md#strict-document-composition) owns +declaration, identity, operation, ownership, and final-validation semantics; +[sibling inheritance](../config.md#sibling-inheritance) defines record +relationships, and the [CLI contract](../config.md#explicit-cli-overrides) owns +override ordering. Known module catalogs should use canonical object entries with a module identifier and typed options. A small union struct is sufficient when the available option diff --git a/docs/development.md b/docs/development.md index f1d9ce2..e7064ae 100644 --- a/docs/development.md +++ b/docs/development.md @@ -135,12 +135,13 @@ Current behavior belongs to the configuration contract's [input destinations](config.md#input-destinations), [imports](config.md#explicit-imports), and [shared source-version checks](config.md#shared-source-version-checks) sections. -Explicit sequence-item placement and registered strict composition are owned by -the [strict composition](config.md#strict-document-composition) and -[imports](config.md#explicit-imports) sections. The -[maintained prompt catalog](consumers/README.md#identified-prompt-sequences) -shows the current consumer boundary, and [ADR 0014](adr/0014-sequence-placement-and-composition.md) -records the decision. +Explicit sequence-item placement, collection operations, identity registration, +and final registered-identity validation are owned by the +[strict composition](config.md#strict-document-composition) section; +[imports](config.md#explicit-imports) owns acquisition and scheduling. The +[maintained sequence catalog](consumers/README.md#identified-prompt-sequences) +shows the consumer boundary, and [ADR 0017](adr/0017-collection-sequence-policy.md) +records the collection decision. The completed sequence-composition roadmap, implementation plan, and resolved remediation record are preserved in Git history at `008d2ba` under @@ -218,6 +219,7 @@ code changes. Use the component overview to find the corresponding implementatio | Import defaults, empty-selection evidence, and dependency-ready ordering | [Defaults and facade evidence](../facade_import_defaults_test.go), [engine import evidence](../internal/engine/import_evidence_test.go), [scheduler](../internal/engine/dependency_order_test.go), [scheduling integrations](../internal/engine/imports_test.go) | `go test ./internal/engine -run 'Import|ScheduleDependencies|Inheritance'`, `go test . -run 'ImportDefaults|FacadeImport'`, and `go test ./examples/consumers` | | Native filesystem and mapping sources | [Native acquisition](../internal/engine/native_filesystem_test.go), [filesystem facade](../facade_filesystem_test.go), [mapping facade](../facade_mapping_test.go) | `go test ./internal/engine -run 'NativeFilesystem'` and `go test . -run 'TestFacade'` | | Composition, selected overlays, CLI precedence | [Composition](../internal/engine/compose_test.go), [overlay](../internal/engine/overlay_test.go), [selected overlays](../internal/engine/selected_overlay_test.go), [overrides](../internal/engine/override_test.go) | `go test ./internal/engine -run 'Compose\|Overlay\|Override'` | +| Sequence identity, declarations, and operations | [Identity](../internal/engine/sequence_identity_test.go), [registrations](../internal/engine/sequence_registration_test.go), [declarations](../internal/engine/sequence_declaration_test.go), [combination](../internal/engine/sequence_combine_test.go), [final facade validation](../facade_final_sequence_identity_test.go), and [consumer catalog](../examples/consumers/sequence_catalog_test.go) | `go test ./internal/engine -run 'Sequence\|Collection'`, `go test . -run 'SequenceIdentity\|SequenceRegistration'`, and `go test ./examples/consumers` | | Sibling inheritance, provenance, history, sensitivity | [Expansion](../internal/engine/inheritance_test.go), [destination scopes](../internal/engine/inheritance_walk_test.go), [origins](../internal/engine/inheritance_origin_test.go), [history](../internal/engine/inheritance_history_test.go), [secrets](../internal/engine/inheritance_secret_test.go), [public loading](../facade_inheritance_test.go) | `go test ./internal/engine -run 'Inheritance\|Inherited'` and `go test . -run '^TestFacadeInheritance\|^TestFacadeInheritedSecrets'` | | Layered loading and source equivalence | [Grouped lifecycle](../internal/engine/load_request_test.go), [layered facade](../facade_layered_test.go), [equivalence](../facade_equivalence_test.go) | `go test ./internal/engine -run '^TestGrouped'`, `go test . -run '^TestFacadeLayered'`, and `go test . -run '^TestEquivalentSourceForms'` | | Input and relative-import destinations, mounted sources, and source ownership | [Destination engine tests](../internal/engine/destination_test.go), [tier tests](../internal/engine/destination_tiers_test.go), [import traversal and version tests](../internal/engine/destination_import_test.go), [accounting tests](../internal/engine/destination_accounting_test.go), [lifecycle inspection tests](../internal/engine/destination_import_lifecycle_test.go), [public source/error tests](../facade_destination_test.go), [resource contexts](../facade_resource_context_test.go), and [consumer examples](../examples/consumers/destinations_test.go) | `go test ./internal/engine -run 'Destination|Import|Glob|Mounted'`, `go test . -run 'Destination|Import|SourceContext|FacadeMounted'`, and `go test ./examples/consumers` | diff --git a/docs/internal/imports.md b/docs/internal/imports.md index e73d349..8424c23 100644 --- a/docs/internal/imports.md +++ b/docs/internal/imports.md @@ -65,11 +65,18 @@ returns a closed cycle rotated to its smallest key; blocked dependents are not reported as cycle members. Import composition schedules a main tier or one underlay group as a single -strict-scope graph. Item contributions own nested local strict-mapping scopes; -their helpers complete before the item is placed in its outer scope. Inheritance -uses the same ready-node rule inside each sibling mapping scope, while preserving -the authored mapping order in its output. These scopes do not flatten into one -global graph, and independent precedence tiers remain separate. +strict-scope graph. Applicable collection declarations are gathered for the +scope before peer composition, including identity inherited from lower tiers. +Identified peer sequences then combine distinct records in assembled source and +list order; after the strict scope completes, layered assembly applies the +resolved collection operation against the lower value, as defined by the +[configuration contract](../config.md#strict-document-composition). Item +contributions own nested local strict-mapping scopes; their helpers complete +before the item is placed in its outer scope. Inheritance uses the same +ready-node rule inside each sibling mapping scope, while preserving the authored +mapping order in its output. +These scopes do not flatten into one global graph, and independent precedence +tiers remain separate. The main owners are [`import_selection.go`](../../internal/engine/import_selection.go), [`directory_walk.go`](../../internal/engine/directory_walk.go), diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 3dcbf9f..8d93eac 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -9,12 +9,13 @@ Implemented component owners: - `internal/engine` owns host and native-filesystem source acquisition, grouped strict composition, recursive precedence assembly, per-input mapping - and sequence-item placement, definition-registered distinct-ID sequence - composition within strict tiers, and immediate-parent-relative import - placement within that assembly. Higher-priority sequences replace lower - sequences as whole values. The engine also owns dependency-ready, - destination-local sibling inheritance before typed decoding, plus shared - source-version checking with sanitized declaration evidence. + and sequence-item placement, YAML collection declarations and operations for + identified sequences, and application identity registrations for strict + composition, inheritance, and final validation. It applies the selected + collection operation at precedence boundaries and owns dependency-ready, + destination-local inheritance before typed decoding. CLI assignment follows + inheritance and decoding, before application resolution. The engine also owns + shared source-version checking with sanitized declaration evidence. Explicit import discovery supports declared paths, directory searches, and globs; load-wide defaults and source-local empty-selection evidence cross the root facade without changing acquisition order. Import and inheritance @@ -56,9 +57,10 @@ semantics, including [input destinations](../config.md#input-destinations), [shared source-version checks](../config.md#shared-source-version-checks), and [effective configuration diffs](../config.md#effective-configuration-diffs). The [strict composition contract](../config.md#strict-document-composition) -owns sequence placement, registration, replacement, and history behavior; the -[consumer guide](../consumers/README.md#identified-prompt-sequences) shows the -maintained example. +owns sequence declarations, identity, operations, ownership, and final +validation; [CLI overrides](../config.md#explicit-cli-overrides) own override +ordering. The [consumer guide](../consumers/README.md#identified-prompt-sequences) +links maintained workflow examples. The [consumer adoption guide](../consumers/README.md) owns application guidance. The [architecture policy](../policy/architecture.md) owns contributor boundaries; [ADR 0006](../adr/0006-public-facade.md) records the facade decision @@ -74,5 +76,7 @@ records the source-version decision. sequence placement and composition. [ADR 0015](../adr/0015-shared-dependency-ready-ordering.md) records the shared dependency-ready ordering decision; [ADR 0016](../adr/0016-independent-import-controls.md) records the independent -selection controls. The [import internals](imports.md) document candidate and +selection controls. [ADR 0017](../adr/0017-collection-sequence-policy.md) and +[ADR 0018](../adr/0018-cli-after-structural-resolution.md) record collection +operations and CLI ordering. The [import internals](imports.md) document candidate and namespace identities, discovery and scheduling, and the bounded algorithms. diff --git a/docs/policy/architecture.md b/docs/policy/architecture.md index e95a740..926ef91 100644 --- a/docs/policy/architecture.md +++ b/docs/policy/architecture.md @@ -32,11 +32,11 @@ structure as implemented when its changes land. Constraints must reflect deliberately selected requirements and decisions. Do not promote incidental implementation choices into architectural invariants. -The collection-policy and CLI-ordering requirements below reflect accepted -[ADR 0017](../adr/0017-collection-sequence-policy.md) and -[ADR 0018](../adr/0018-cli-after-structural-resolution.md). The -[sequence roadmap](../roadmap/sequences.md) tracks the work to implement them; -the configuration contract continues to describe implemented behavior. +The implemented collection-policy and CLI-ordering constraints below reflect +accepted [ADR 0017](../adr/0017-collection-sequence-policy.md) and +[ADR 0018](../adr/0018-cli-after-structural-resolution.md). The configuration +contract owns their current behavior; these architectural constraints describe +the system boundaries and guarantees that guide changes to that behavior. ## Success Criteria @@ -373,8 +373,8 @@ as records survive, move, or are replaced. Identity diagnostics use paths and available origin information without exposing identity values. The guarantee requires no new mutation-history system or exact reconstruction of causes for records supplied by defaults or callbacks. Exact identity equality, declaration -syntax, presence cases, and operation tables belong in the configuration contract -when implemented; the roadmap owns those details until then. +syntax, presence cases, and operation tables belong in the +[configuration contract](../config.md#strict-document-composition). ## Ownership And Confidentiality diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index aa88101..bc37848 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -6,112 +6,19 @@ commitments. Adopting a candidate requires an explicit design decision and a separate plan. Significant architectural exclusions and their rationale belong in ADRs; this document does not maintain a list of rejected proposals. -## 1. Identified record sequences +## 1. Recursive body merging for identified records -Status: deferred. PromptKit proposed this capability from its retired proposal 2, -**Namespaced Collections As Identified Records**. The original proposal is in the -PromptKit repository at -`b5866dd:docs/roadmap/configuration-collections-identified-records.md`. -PromptKit selected mounted mappings instead; this feature is not a dependency -of that design or a request to restore multiple installed prompt versions. +Status: deferred. The implemented +[collection policy](../config.md#strict-document-composition) uses `upsert` to +replace a complete matched record. Recursive merging of fields inside matching +records remains a separate candidate and would revisit the whole-record update +boundary selected by [ADR 0017](../adr/0017-collection-sequence-policy.md). -A narrower [sequence placement and composition feature](../config.md#strict-document-composition) -is implemented: consumers can place imported mappings as sequence items and -register a single literal string identity field for duplicate-rejecting strict -composition. Precedence still replaces sequences as whole values. This entry -tracks broader identified-record behavior that remains deferred and is not part -of that implemented contract. - -The active [sequence roadmap](sequences.md) now selects a narrower design for -collection policy, compound scalar identities, final application-registered -identity validation, and inheritance within identified sequences. It owns those -selected behaviors and their remaining decisions. The original broader proposal -below remains background for further consideration, particularly recursive -combination of matching record bodies; it does not override the active roadmap. - -### Motivation and proposed behavior - -Allow applications to declare that selected sequences contain records identified -by explicit fields, while retaining sequence-shaped output for slice decoding -and effective YAML. This could make composite identities and independently -authored catalogs more convenient than encoding identity in nested mapping keys. -Current [composition](../config.md#strict-document-composition) appends distinct -records only at explicitly registered paths; other sequences remain atomic. -Selected [overlays](../config.md#selected-overlays) always treat sequences -atomically. - -The proposed broader opt-in registration would extend the current reusable -application definition from one identity field to a composite identity made of -an ordered, nonempty set of distinct direct string-field names. For example, -records at `profiles` could be identified by `(namespace, id)`. Records -`(provider, brief)` -and `(application, brief)` would coexist, and a later layer could update the -provider record's temperature while retaining its backend and model. - -The requested semantics, all still subject to design review, are: - -- Match exact string tuples, preserving component boundaries and spelling, - including empty strings. Every contribution supplies its complete identity; - defaults, inheritance, filenames, and normalization do not supply it. -- Retain distinct identities. Matching records from strict peers recursively - compose disjoint body fields and conflict on overlapping non-mapping body - values, even equal values. Equal identity fields act as structural keys rather - than conflicts and retain their contributing source evidence. -- Higher layers recursively overlay matching bodies. A changed identity denotes - a different record, not a rename. Proposed ordering is first contribution in - precedence-assembly order; matching records keep position and new ones append. -- Reject malformed identities, non-mapping records, non-null non-sequence values - at registered paths, and duplicate tuples within one source sequence. Later - layers cannot repair malformed structural identity in an acquired contribution. -- Treat an empty registered sequence as contributing no records; whole-path null - retains ordinary conflict/replacement semantics. Unregistered sequences, - including nested message lists, retain existing atomic behavior. -- Support record inheritance within the same registered sequence using an - `extends` mapping containing the complete parent tuple. Resolve only final - active relationships after precedence, preserve the child's identity, overlay - its body on the parent's body, and consume the directive before decoding. - Preserve cancellation, missing-parent/cycle errors, and declaring resource - contexts. Existing string-based mapping-sibling inheritance stays unchanged. - -Integrate ordinary and layered loads, imports, mounted paths, selected overlays, -and whole-sequence CLI overrides. Keep origins, presence, history, sensitivity, -and resource contexts attached to the correct record despite different source -and final indexes. Diagnostics must retain source locations without exposing -sensitive identity values. Final inspection may use snapshot-specific sequence -indexes; existing atomic sequence diffs need not become identity-aware. - -Successful results would have valid, unique identities, including after defaults -and application resolution, without a second merge or inheritance pass. Existing -source-version checks remain independent; consumed headers cannot supply missing -record identities. No identity-selector path syntax, arbitrary merge callbacks, -wildcard registration, per-record deletion/rename operator, multiple inheritance, -or cross-path parent references are requested. - -### Questions to resolve before implementation - -- **Need and delivery scope:** does an active consumer benefit enough to justify - the added semantics over keyed mappings? Are composite identities required? - Staging composition before inheritance is possible but would leave part of - the request incomplete. -- **Defaults and callbacks:** do default records merge by identity or retain - existing sequence-replacement behavior? How are callback-created duplicates, - identity edits, and reordering validated and reflected in provenance? -- **Null, empty, and order:** what happens when null clears a collection and a - later layer reintroduces records? How do absent and explicitly empty sequences - interact with defaults and first-contribution order? -- **Inheritance references:** how do peers compose relationship declarations, - and how are complete references enforced without assembling partial tuples - accidentally? What happens if identity fields overlap structural controls? -- **Registration paths:** how do overlapping registrations and paths beneath - sequence indexes behave when assembly changes those indexes? -- **Evidence and lifecycle:** how are record identities mapped between source - trees, history, inheritance, callback snapshots, and final indexes without - mixing records' provenance or sensitivity? Where do identity checks run relative - to header consumption and other structural processing? - -Keep the feature deferred until its consumer value and full lifecycle design -justify reconsidering the sequence-merging exclusion. A correct final sequence -alone would not establish correct inspection or disclosure behavior. +Consider this only in response to a concrete consumer need. A proposal would need +to define conflicts for overlapping body fields, removal of omitted fields, +presence and ordering, and how provenance, history, sensitivity, and resource +context follow partially retained records. Any change requires a new decision and +implementation plan; this entry does not alter current upsert behavior. ## 2. Required version declarations for selected sources diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 21a140c..d8763f5 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,6 +1,6 @@ # Sequence Identity And Combination Implementation Plan -Status: implementation in progress; Stages 1–28 complete. +Status: implementation in progress; Stages 1–29 complete. Audience: a Luna coding agent implementing one numbered stage per prompt, in order. This plan starts at commit @@ -993,6 +993,15 @@ Validation: local documentation-link checks and whitespace review; run relevant example tests if executable examples changed. Reuse Stage 28's successful code/race validation when its covered content is unchanged. +**Completion:** Reconciled the configuration contract for identity equality, +selectors, declarations, operation selection and presence, ownership, +inheritance, final registered-identity validation, and post-decode CLI ordering. +Updated consumer, root, internal, development, and architecture guidance; removed +implemented behavior from the deferred feature list while retaining recursive +body merging as future work. Updated ADR implementation references and preserved +intentional historical references. Verified maintained prose/example searches, +`go test . -run '^TestLocalDocumentationLinks$' -count=1`, and `git diff --check`. + ## Stage 30 — Audit Scaling And Verify The Completed Target Scope: a bounded implementation review and final validation, not another broad diff --git a/docs/roadmap/sequences.md b/docs/roadmap/sequences.md index 52d1e77..93f7135 100644 --- a/docs/roadmap/sequences.md +++ b/docs/roadmap/sequences.md @@ -28,9 +28,8 @@ sequences, selected overlays, and sibling inheritance. Typed defaults, CLI assignments, and application resolution retain their own combination semantics; CLI application moves later in the loading lifecycle. -This scope is narrower than the deferred -[identified-record sequences proposal](future.md#1-identified-record-sequences). -Automatic recursive merging of matching records remains outside this scope; +Automatic recursive merging of matching records remains deferred +(see [future work](future.md#1-recursive-body-merging-for-identified-records)); explicit inheritance between distinct records is included. The initial declaration surface is `configkit.sequences`; `key` and `mode` shorthand on imports is deferred. Application lists remain