Add dynamic identity constraints to the sequence roadmap

docs/roadmap/roadmap.md

@@ -1,9 +1,10 @@
 # Sequence Identity And Combination Roadmap
 
-Status: settled feature roadmap, ready for a staged implementation plan. This
+Status: feature roadmap with dynamic application identity constraints included.
-is the targeted end state, not an implemented contract. The subsequent plan will
+The remaining registration API decision is recorded under
-specify execution order, the exact Go API, internal bookkeeping, and validation
+[Open Decisions](#open-decisions). This is the targeted end state, not an
-stages.
+implemented contract. The subsequent plan will specify execution order, exact
+Go names and signatures, internal bookkeeping, and validation stages.
 
 ## Purpose And Scope
 
@@ -19,7 +20,9 @@ identity.
 The motivating use case is a prompt catalog whose source-backed baseline supplies
 records and whose application or installation layers add or supersede records.
 Applications can optionally impose identity constraints that configuration must
-respect and that hold in the final loaded configuration.
+respect and that hold in the final loaded configuration. These constraints can
+cover exact destinations or collections beneath dynamically named mapping keys,
+without requiring the application to discover those names before loading.
 
 The feature covers ordinary and layered loading, imported items, inline
 sequences, selected overlays, and sibling inheritance. Typed defaults, CLI
@@ -50,6 +53,7 @@ baseline:
 | Area | Current behavior | Target behavior |
 | --- | --- | --- |
 | Identity declaration | `Definition.SequenceCompositions` registers one literal string field at each registered mapping path. | YAML collection declarations and optional authoritative application registrations support field sets and scalar identity components. |
+| Application registration destinations | Registrations select exact literal mapping paths. | Exact and structural selectors can impose identity beneath arbitrary mapping keys, independently at each matched destination. |
 | Strict composition | Registered sequences append distinct string identities and reject duplicates; repeated unregistered sequences conflict. | Collections with effective identity append distinct records and reject duplicates. Applicable lower-tier identity is available before composing higher-tier peers. Unidentified sequences retain strict-conflict behavior. |
 | Source precedence and selected overlays | A present higher sequence replaces the lower sequence wholesale. | Collection policy selects `replace`, `append`, or `upsert`; overlays can declare policy for inline sequences. |
 | Sibling inheritance | An inheriting mapping recursively overlays its resolved parent; supplied sequences replace wholesale. | Mappings retain recursive combination, while sequences use destination policy and the same three operations. |
@@ -223,6 +227,104 @@ a path to exist, be non-null, or have sequence shape. The application schema and
 validation own those requirements. Final-result enforcement for application
 registrations is defined under [Validation And Diagnostics](#validation-and-diagnostics).
 
+### Dynamic Application Registration Destinations
+
+Allow applications to register identity for collections whose containing mapping
+keys become known only during loading. For example:
+
+```yaml
+collections:
+  weatherreporter:
+    prompts:
+      - id: foo.bar
+        text: Report the weather.
+  homewatch:
+    prompts:
+      - id: foo.bar
+        text: Report the household status.
+```
+
+One application registration can require `id` for every namespace's `prompts`
+list. The equal IDs above are valid because each list has its own uniqueness
+scope. An optional root `prompts` list can have a separate exact registration.
+
+Use an explicit application-side structural selector composed of literal mapping
+keys and an arbitrary-mapping-key segment. Conceptually, the namespace rule is:
+
+```text
+Literal("collections") -> AnyMappingKey -> Literal("prompts")
+```
+
+This notation explains the selector; it is not a Go signature or YAML syntax.
+Each segment consumes exactly one mapping level. Match the complete destination,
+not an arbitrary prefix or suffix. Literal keys remain exact, including empty
+strings and keys containing punctuation such as `*`, `/`, or `.`. An arbitrary-key
+segment is distinct from any literal spelling and does not match a sequence
+index. Existing `Path` values retain their literal meaning.
+
+This capability belongs to the reusable application definition and applies to
+ordinary and layered loads. YAML collection destinations and import destinations
+retain their existing literal syntax. Recursive search, regular expressions,
+filesystem glob rules, sequence-index selectors, and custom matching callbacks
+are outside this addition.
+
+At each concrete matched destination, apply the same authoritative identity
+constraint as an exact registration. Use the settled field-set and scalar-equality
+rules. Do not pool identities across destinations or require records to repeat
+the containing mapping key as an identity component. Applications retain ownership
+of namespace interpretation, qualified runtime IDs, and domain validation.
+
+Matching supplies effective identity before relevant strict composition and
+precedence operations. Without an applicable explicit mode, this identity selects
+upsert under the existing default rule. It also supplies identity for inheritance
+within the matched sequence. YAML declarations must agree with it under every
+mode, including replace. These are ordinary application constraints with a broader
+destination selector, not a separate final-validation facility.
+
+### Registration Overlap
+
+Equivalent application constraints coalesce at any shared destination. Equivalence
+uses the identity field-set rules: for example, `id` and `[id, id]` agree.
+Incompatible constraints fail; exact registrations do not override broader
+selectors, and registration order grants no precedence.
+
+Detect incompatible overlapping registrations while validating the application
+definition, before source acquisition. For these fixed-depth mapping selectors,
+overlap exists when the selectors have the same number of segments and every
+corresponding pair can match a common key. Unequal literal keys at any position
+make the selectors disjoint; an arbitrary-key segment can overlap a literal or
+another arbitrary-key segment. Compare segment structure rather than rendered
+path strings. No configuration pre-read is needed.
+
+Thus an `id` rule for all namespace prompt lists conflicts with a `[id, name]`
+rule for one named namespace, even if that namespace is absent from a particular
+load. This is an invalid application definition. A valid selector matching no
+collections is harmless: registration does not require a path to exist, create
+collections, or impose shape and nullability requirements.
+
+### Matching Across The Loading Lifecycle
+
+Match application selectors against effective configuration destinations after
+accounting for input placement. Physical filenames and source-local paths do not
+determine applicability. Make constraints available before composing contributions
+at those destinations, including namespaces first introduced by imports, mounted
+inputs, or higher tiers. Apply matching constraints to key-only YAML declarations
+as well, even before their sequence exists. Consumers need neither a preliminary
+load nor source inspection to enumerate namespaces.
+
+During inheritance, apply registrations at the inheriting destination, including
+newly created matching destinations. Copying a collection does not transfer its
+application registrations to a different path. The existing YAML-policy ownership
+and transfer rules remain separate from application selector matching.
+
+During final validation, find matching actual sequences in the final effective
+configuration. This includes namespaces introduced or replaced by typed defaults,
+CLI overrides, and application resolution. Reusing only destinations found during
+source assembly would miss these values. Use the same selector meaning and
+identity validator in both phases; later assignments do not trigger composition
+or inheritance again. YAML-only identities retain their existing enforcement
+boundary.
+
 ## Strict Composition And Precedence Operations
 
 ### Compose Peers Before Applying Precedence
@@ -699,6 +801,11 @@ registrations satisfy their declared identity constraints. Retain assembly-time
 checks and validate final effective sequences before publication, after defaults,
 inheritance, overrides, application resolution, and secret lookup.
 
+This covers both exact and dynamic application registrations. Evaluate dynamic
+selectors against the final effective configuration, with independent validation
+for each concrete matched sequence as described under
+[Matching Across The Loading Lifecycle](#matching-across-the-loading-lifecycle).
+
 ConfigKit performs these identity checks; consumers register the constraint
 without needing to repeat duplicate detection in `Definition.Validate`.
 Application-owned semantic validation continues to cover domain-specific rules.
@@ -720,6 +827,11 @@ Identity diagnostics identify collection and item paths using available origin
 information without exposing identity values. Conflicting declarations identify
 the collection destination and declaration locations.
 
+For dynamic registrations, use concrete matched collection and item paths for
+value diagnostics, with available origins and existing sensitivity rules.
+Definition-time overlap errors identify the conflicting registrations; they do
+not require or invent a concrete destination from a loaded document.
+
 Correctness includes final values, projected item paths, source destinations,
 provenance, history, sensitivity, and passive resource contexts. Appending can
 change final indexes; replacing changes which contributions survive. Preserve
@@ -732,13 +844,19 @@ The implementation plan should cover these connected changes, with execution
 order and test sequencing defined there:
 
 - **Declarations and public API:** extend application registration to the settled
-  identity model; parse and consume `configkit.sequences` across supported source
+  identity model and explicit structural selectors while preserving literal
+  `Path` semantics. Parse and consume `configkit.sequences` across supported source
   forms and selected overlays; preserve placement-relative and item-local scopes.
   Keep sequence declarations independent of filesystem import enablement.
 - **Identity and policy representation:** implement field-set comparison, typed
   scalar identity tuples with exact numeric equality, declaration agreement,
   pending declarations, explicit-mode persistence, replacement lifetime, and
   record-owned policy transfer. Choose bookkeeping proportional to these semantics.
+- **Registration matching:** support literal and arbitrary-mapping-key segments,
+  detect incompatible selector overlap during definition validation, and resolve
+  applicable constraints at concrete placed destinations. Reuse the identity
+  rules and validator for exact and dynamic registrations. Keep matching separate
+  from source discovery and from YAML declaration ownership.
 - **Assembly handoffs:** make accumulated lower policy available before higher-tier
   strict composition. Separate source acquisition from policy-aware composition
   or otherwise revise assembly sequencing; current main-before-underlay composition
@@ -759,7 +877,9 @@ order and test sequencing defined there:
   defaults/decoding, preserve explicit-presence and source metadata for application
   resolution, reject CLI structural directives, and add final application identity
   validation through the shared publication path. Keep YAML-only enforcement
-  within structural operations.
+  within structural operations. Match dynamic registrations during assembly and
+  inheritance, and against the final effective configuration rather than a saved
+  list of source-backed destinations.
 - **Documentation and verification:** exercise ordinary and layered loading,
   imported and inline collections, item-local scopes, version-header consumption,
   selected overlays, inheritance, typed defaults, CLI, callbacks, and inspection.
@@ -768,6 +888,13 @@ order and test sequencing defined there:
   preservation, nested destination policy, cancellation, and stable output order.
   Verify helper-composed references, overlapping reference fields, missing or extra
   fields, and replacement of references together with their containing records.
+  Verify independent namespace uniqueness, root and namespaced registrations,
+  literal punctuation keys, equivalent and incompatible overlaps, and conflicts
+  detected even without a matching input destination. Cover namespaces introduced
+  by imports, placement, higher tiers, inheritance, defaults, CLI, and callbacks;
+  preserve ordinary strict-peer and upsert behavior. Check YAML disagreement under
+  replace, harmless absent or empty collections, schema-owned shape requirements,
+  unrelated sequences remaining unselected, and concrete-path diagnostics.
   Update the public contract and consumer examples when behavior lands.
 
 The collection-policy and CLI-ordering decisions are accepted in
@@ -779,12 +906,35 @@ These govern the target architecture; the configuration contract continues to
 describe implemented behavior until the feature lands. This roadmap does not
 select migration machinery.
 
-Before implementation, finalize the ADR 0017 revision for inheritance within
+ADR 0017 now includes inheritance within identified sequences;
-identified sequences, update the supersession status of
+[ADR 0009](../adr/0009-sibling-inheritance.md) records that scope extension in its
-[ADR 0009](../adr/0009-sibling-inheritance.md), and align the architecture policy.
+status while preserving its decision content. The architecture policy is aligned.
-Preserve ADR 0009's decision content and the final application-registration
+Dynamic selectors extend the destinations covered by application registration;
-guarantee. These architectural revisions remain subject to review.
+they retain the accepted application authority and final-validation boundary.
 
-No open feature decisions remain. The implementation plan should translate this
+After settling the API decision below, the implementation plan should translate
-end state into reviewable stages without adding guarantees, restrictions, or
+this end state into reviewable stages without adding guarantees, restrictions,
-combination semantics.
+or combination semantics.
+
+## Open Decisions
+
+### Registration API Shape
+
+The selector's behavior and definition-time overlap validation are settled. How
+should the public registration represent exact destinations alongside selectors
+containing arbitrary-mapping-key segments?
+
+**Recommendation:** use one registration model with an explicit selector type
+whose segments distinguish literal keys from arbitrary mapping keys. Provide a
+straightforward way to construct an exact selector from a literal `Path`.
+Both forms then share identity fields, validation, and lifecycle behavior, while
+ordinary `Path` remains unchanged. This requires a small new public type, but
+keeps one registration surface and avoids mutually exclusive destination fields.
+Exact Go names and constructor signatures belong in the implementation plan.
+
+**Alternative:** retain exact-path registrations and add a separate dynamic
+registration form, normalizing both into the same internal selector model.
+This keeps exact registration familiar for simple consumers, but introduces two
+public forms with the same identity rules and requires overlap validation across
+both. Both choices must preserve the settled matching and enforcement behavior;
+neither introduces selector syntax into ordinary paths or YAML destinations.