Plan identity-based sequence composition
This commit is contained in:
288
docs/roadmap/sequence-composition.md
Normal file
288
docs/roadmap/sequence-composition.md
Normal file
@@ -0,0 +1,288 @@
|
||||
# Identity-Based Sequence Composition
|
||||
|
||||
Status: agreed feature direction; not implemented. Resolve the open decisions
|
||||
below before implementation, then execute the plan in order.
|
||||
|
||||
## Goal and current baseline
|
||||
|
||||
Allow consumers to opt selected sequences into strict composition by one explicit
|
||||
string identity field. Independent fragments can contribute distinct records to
|
||||
the same list; repeated identities fail composition. Keep records as sequences
|
||||
for typed decoding, inspection, and effective YAML.
|
||||
|
||||
Current [strict composition](../config.md#strict-document-composition) recursively
|
||||
combines mappings and rejects every other overlap. Current
|
||||
[precedence assembly](../config.md#layered-loading) and
|
||||
[selected overlays](../config.md#selected-overlays) replace sequences wholesale.
|
||||
There is no sequence-composition registration in `Definition` today.
|
||||
|
||||
This is a narrower feature than the
|
||||
[identified record sequences candidate](future.md#1-identified-record-sequences).
|
||||
That candidate's composite identities, recursive merging of matching records,
|
||||
identity-based overlays, and record inheritance are not part of this plan.
|
||||
|
||||
## Agreed behavior
|
||||
|
||||
### Registration and identities
|
||||
|
||||
- Consumers explicitly designate each sequence path and one direct identity
|
||||
field, such as `prompts` and `id`. Unregistered sequences retain current rules.
|
||||
- Records in a participating sequence must be mappings containing the designated
|
||||
field as a string. Missing, null, and non-string identities are invalid;
|
||||
identities are not coerced from numbers or supplied by defaults or resolution.
|
||||
- Compare strings exactly, without trimming, case folding, or normalization.
|
||||
Whether the empty string is accepted is an open decision below.
|
||||
- Retain identity fields as ordinary record content in the resulting sequence.
|
||||
They are not consumed as metadata or converted into public mapping keys.
|
||||
- This is a source-composition rule, not a guarantee of identity uniqueness after
|
||||
application defaults and callbacks. Those retain their existing behavior;
|
||||
applications can enforce final domain constraints through validation.
|
||||
|
||||
### Strict composition
|
||||
|
||||
For a registered `prompts` sequence with identity field `id`, two separate
|
||||
source fragments could contribute:
|
||||
|
||||
```yaml
|
||||
prompts:
|
||||
- id: prompt_one
|
||||
text: "foo"
|
||||
```
|
||||
|
||||
```yaml
|
||||
prompts:
|
||||
- id: prompt_two
|
||||
text: "bar"
|
||||
```
|
||||
|
||||
The composed value is:
|
||||
|
||||
```yaml
|
||||
prompts:
|
||||
- id: prompt_one
|
||||
text: "foo"
|
||||
- id: prompt_two
|
||||
text: "bar"
|
||||
```
|
||||
|
||||
- Append distinct records in contribution order, preserving each source list's
|
||||
order. Use existing deterministic input/import traversal within a tier;
|
||||
identity values do not sort the result.
|
||||
- Reject repeated identities both across contributions and within a single
|
||||
contributing sequence, including a sequence that has no peer contribution.
|
||||
Equal bodies and disjoint bodies both conflict. Never recursively combine
|
||||
records with the same identity.
|
||||
- An empty sequence contributes no records. Composing empty sequences succeeds;
|
||||
an explicitly contributed empty sequence remains present, not absent.
|
||||
- Apply these rules to main inputs and their imports, and to inputs and imports
|
||||
composing each underlay group. Honor input and relative-import destinations.
|
||||
Source acquisition kind must not change composition behavior.
|
||||
- Invalid contributions and duplicate identities in a tier cannot be repaired
|
||||
by a higher-precedence replacement. Report a normal load error and return no
|
||||
usable partial result.
|
||||
|
||||
The analogy with keyed mappings is limited: distinct keys combine, but mappings
|
||||
at a shared key currently recurse. A shared record identity always conflicts
|
||||
under this feature.
|
||||
|
||||
### Precedence and other operations
|
||||
|
||||
Sequence replacement remains the rule between underlay tiers, between underlays
|
||||
and the main tier, and for the selected overlay. An upper sequence replaces the
|
||||
complete lower sequence, even at a registered path; `[]` clears it. Identities
|
||||
may recur across those tiers without a cross-tier duplicate error. Null and
|
||||
cross-kind precedence replacements retain current semantics.
|
||||
|
||||
Whole-sequence CLI overrides, defaults, application resolution, and built-in
|
||||
sibling inheritance retain their existing semantics. This feature adds no
|
||||
record inheritance, identity selector syntax, record update/delete/rename
|
||||
operators, or identity-based behavior to effective-configuration diffs. Existing
|
||||
atomic sequence diffs continue to compare final frozen values.
|
||||
|
||||
### Inspection and diagnostics
|
||||
|
||||
Appending records changes their effective indexes. Preserve the distinction
|
||||
between an original source position and its position in the assembled sequence.
|
||||
Effective presence, origins, resource contexts, history, and explanations must
|
||||
refer to the correct record and its original physical source locations.
|
||||
|
||||
Composition history must describe the accumulated sequence correctly, rather
|
||||
than replay each fragment as a whole-sequence replacement. Later actual
|
||||
replacements must still retain displaced history. Equal numeric indexes in
|
||||
different source lists must not make unrelated records share provenance.
|
||||
|
||||
Duplicate diagnostics must identify the conflicting source locations in
|
||||
deterministic order. Avoid including identity values in diagnostic text where
|
||||
sensitivity is not yet known; identify the sequence, identity field, and source
|
||||
locations instead. Apply existing sensitivity and disclosure rules to successful
|
||||
and failed inspection, including records whose effective indexes changed.
|
||||
Retain detached public records and input/result ownership guarantees.
|
||||
|
||||
## Implementation plan
|
||||
|
||||
### 1. Settle the remaining contract choices
|
||||
|
||||
- Resolve the Open Decisions section and update this document with the choices.
|
||||
- Select the public registration names, path interpretation, and diagnostic codes.
|
||||
Duplicate claims should use the existing composition-conflict category where
|
||||
appropriate; malformed registration and malformed records need distinct,
|
||||
actionable diagnostics.
|
||||
- Record the significant composition extension in an ADR during implementation,
|
||||
under the architecture policy. No ADR is created by this roadmap-writing pass.
|
||||
|
||||
Completion: the public behavior and validation boundary are unambiguous, with
|
||||
no accidental commitment to the broader identified-records proposal.
|
||||
|
||||
### 2. Add registration and pass it through the loading paths
|
||||
|
||||
- Add root-owned public registration types and fields using the selected API.
|
||||
Convert them explicitly into private engine state; preserve the root facade
|
||||
boundary and reusable definition ownership.
|
||||
- Validate registrations before source processing and pass the same policy
|
||||
through ordinary and layered loading, import assembly, and mounted inputs.
|
||||
- Compile path lookup once per load if useful. No registrations must preserve
|
||||
current behavior without introducing a new global cache or merge framework.
|
||||
|
||||
Completion: public callers can select sequence paths consistently across load
|
||||
forms, and invalid options fail through the normal diagnostics boundary.
|
||||
|
||||
### 3. Extend strict composition and its retained evidence together
|
||||
|
||||
The current `composeDocuments` implementation in
|
||||
`internal/engine/compose.go` collects claims by path, rejects non-mapping
|
||||
overlaps, merges disjoint mappings, and retains origins and history applications.
|
||||
Changing only the final tree merge would leave these other representations wrong.
|
||||
|
||||
- Extend claim validation and tree assembly for registered sequences. Validate
|
||||
single contributions as well as overlaps. Track identities per sequence per
|
||||
strict tier, and append cloned records in stable order. Keep duplicate lookup
|
||||
proportional to record count rather than repeatedly scanning prior records.
|
||||
- Keep original source locations while assigning effective indexes. Carry the
|
||||
necessary placement information into claims, origins, resource contexts, and
|
||||
history; do not expose a new public identity-addressing model.
|
||||
- Update history/explanation replay to represent sequence accumulation. Ensure
|
||||
snapshots, historical paths, sensitivity classification, and subsequent
|
||||
whole-sequence replacement agree with the assembled tree.
|
||||
- Integrate with ordinary input assembly and grouped assembly in
|
||||
`internal/engine/load_input.go` and `internal/engine/load_request.go`.
|
||||
Keep tier boundaries explicit so uniqueness is never enforced across
|
||||
precedence layers accidentally.
|
||||
- Preserve parser/source limits and retained-node accounting. Keep source-header
|
||||
processing and structural-directive handling in their existing lifecycle
|
||||
positions; do not restore consumed headers to supply record identities.
|
||||
|
||||
Completion: final values and their inspection evidence agree across composition
|
||||
and subsequent replacement. No input mutation, source-context mix-up, or
|
||||
sensitive-value disclosure is introduced.
|
||||
|
||||
### 4. Verify behavior at its owning boundaries
|
||||
|
||||
- Composition tests: distinct and repeated identities, duplicates in one list,
|
||||
malformed records and identity types, empty strings according to the chosen
|
||||
decision, empty lists, ordering, missing paths, and null/cross-kind cases.
|
||||
- Loading tests: imports, underlay groups, mounted and relative destinations,
|
||||
equivalent source forms, and conflicts hidden by later layers. Demonstrate
|
||||
that separate tiers may reuse identities and still replace whole lists.
|
||||
- Metadata and inspection tests: records appended from the same source index
|
||||
acquire distinct final indexes while keeping correct locations, history,
|
||||
resource contexts, and sensitivity. Cover later replacement and historical
|
||||
redaction, including a failure before final classification.
|
||||
- Public integration tests: registration conversion and reuse, normal load-error
|
||||
behavior, unchanged unregistered sequences, selected-overlay and CLI
|
||||
replacement, defaults/resolution behavior, and atomic effective diffs.
|
||||
- Add a small maintained consumer example showing two fragments composing and
|
||||
a selected overlay replacing the result. Include only enough cases at each
|
||||
layer to catch distinct failure modes.
|
||||
|
||||
Run focused tests during development, then the everyday validation prescribed
|
||||
by the [development guide](../development.md#baseline-validation). Run ownership
|
||||
or concurrency checks where required by the change. No independent performance
|
||||
benchmark campaign is part of this plan; keep measurement separate from the
|
||||
busy-host environment discussed in the performance audit.
|
||||
|
||||
Completion: new behavior is covered without weakening the existing atomic
|
||||
replacement and default-behavior regressions.
|
||||
|
||||
### 5. Publish the implemented contract in repository documentation
|
||||
|
||||
- Document the public API beside its declarations. Update the configuration
|
||||
contract's composition, inspection/history, and precedence sections, and the
|
||||
consumer guide and maintained example together when implementation lands.
|
||||
- Update the architecture policy and component overview only where the
|
||||
implemented boundary changes. Keep the broader candidate in `future.md`
|
||||
clearly distinct from this delivered subset.
|
||||
- Retire this completed roadmap through the usual documentation lifecycle.
|
||||
Release preparation and publication remain separate work.
|
||||
|
||||
Completion: current documentation describes the delivered feature, and no
|
||||
future proposal is presented as implemented behavior.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
### 1. Public registration ownership
|
||||
|
||||
**Recommendation:** place a collection of path/identity-field registrations on
|
||||
`Definition`, shared by ordinary and layered loading. This describes how the
|
||||
application's configuration is composed and avoids diverging registrations
|
||||
between load forms. Choose final type and field names before coding.
|
||||
|
||||
**Alternative:** place registrations on load options. That allows one definition
|
||||
to be composed differently per invocation, but duplicates configuration across
|
||||
ordinary/layered options and makes composition semantics less visibly part of
|
||||
the application's reusable definition.
|
||||
|
||||
### 2. Path matching, overlapping registrations, and unused rules
|
||||
|
||||
**Recommendation:** interpret paths in effective configuration coordinates after
|
||||
input/import placement, using literal mapping-key segments initially. Reject
|
||||
duplicate registrations for the same path; allow a registration to match no
|
||||
source value. These rules fit mounted collections without requiring discovery
|
||||
of which optional sources will be present. Restricting paths to mapping segments
|
||||
would be an explicit initial scope choice, not an existing library constraint.
|
||||
|
||||
**Alternative:** support exact sequence-index segments too, including registered
|
||||
sequences nested inside appended records. This is more expressive, but requires
|
||||
choosing whether a rule matches source indexes or composed indexes, and how
|
||||
matching behaves as earlier lists contribute records. Settle that interpretation
|
||||
and overlapping registration behavior before supporting these paths. Wildcard
|
||||
or identity-selector matching is not needed for the agreed use case.
|
||||
|
||||
### 3. Empty identity strings
|
||||
|
||||
**Recommendation:** accept the empty string as an exact identity; two empty
|
||||
identities conflict. Presence and string type are sufficient for the composition
|
||||
algorithm, and applications can require nonempty domain identifiers themselves.
|
||||
|
||||
**Alternative:** require nonempty identities. This catches likely authoring
|
||||
mistakes earlier, but adds a domain restriction beyond uniqueness and string
|
||||
type. Whitespace trimming or normalization would still remain out of scope.
|
||||
|
||||
### 4. Null and non-sequence values at registered paths
|
||||
|
||||
**Recommendation:** treat registration as selecting how actual sequences
|
||||
compose, not as a general schema assertion. Preserve existing rules for absent,
|
||||
null, scalar, and mapping values: non-sequence overlaps retain ordinary strict
|
||||
composition behavior, and final decoding/validation checks application types.
|
||||
Two sequence values use the new rule; sequence/null and sequence/mapping overlaps
|
||||
remain conflicts. No later layer can hide an actual strict-composition conflict.
|
||||
|
||||
**Alternative:** require every present non-null source value at the path to be a
|
||||
sequence, even when a later tier would replace it. This catches misplaced
|
||||
configuration earlier but makes registration an additional source-shape rule.
|
||||
If selected, define null handling explicitly rather than treating it as empty.
|
||||
|
||||
### 5. Validation of replacement-only source sequences
|
||||
|
||||
**Recommendation:** require well-formed, internally unique identities for actual
|
||||
source sequences at registered paths, including selected-overlay sequences.
|
||||
This makes source authoring consistent while preserving whole-sequence
|
||||
replacement and permitting identities to recur across tiers. Do not extend the
|
||||
rule to synthetic CLI replacements, defaults, or callback-generated values;
|
||||
final domain validation remains application-owned.
|
||||
|
||||
**Alternative:** validate only source sequences participating in a strict tier,
|
||||
leaving selected-overlay sequences subject solely to existing decoding and
|
||||
application validation. This keeps the check strictly tied to the composition
|
||||
operation, but allows the same list to succeed as an overlay and fail as a base
|
||||
contribution. Choose this boundary explicitly; replacement semantics alone do
|
||||
not settle whether an overlay's own duplicate identities are invalid.
|
||||
Reference in New Issue
Block a user