From 103e10639a30a596a0846c0491f6c53f502ffbfd Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 16 Oct 2026 14:05:43 +0000 Subject: [PATCH] Plan identity-based sequence composition --- docs/roadmap/future.md | 5 + docs/roadmap/sequence-composition.md | 288 +++++++++++++++++++++++++++ 2 files changed, 293 insertions(+) create mode 100644 docs/roadmap/sequence-composition.md diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index 0b22f89..9e8ee14 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -15,6 +15,11 @@ PromptKit repository at PromptKit selected mounted mappings instead; this feature is not a dependency of that design or a request to restore multiple installed prompt versions. +A narrower [identity-based sequence composition roadmap](sequence-composition.md) +now covers single-field identities, duplicate rejection during strict composition, +and whole-sequence replacement between precedence layers. The additional +capabilities below remain deferred and are not requirements of that roadmap. + ### Motivation and proposed behavior Allow applications to declare that selected sequences contain records identified diff --git a/docs/roadmap/sequence-composition.md b/docs/roadmap/sequence-composition.md new file mode 100644 index 0000000..9ef5e77 --- /dev/null +++ b/docs/roadmap/sequence-composition.md @@ -0,0 +1,288 @@ +# Identity-Based Sequence Composition + +Status: agreed feature direction; not implemented. Resolve the open decisions +below before implementation, then execute the plan in order. + +## Goal and current baseline + +Allow consumers to opt selected sequences into strict composition by one explicit +string identity field. Independent fragments can contribute distinct records to +the same list; repeated identities fail composition. Keep records as sequences +for typed decoding, inspection, and effective YAML. + +Current [strict composition](../config.md#strict-document-composition) recursively +combines mappings and rejects every other overlap. Current +[precedence assembly](../config.md#layered-loading) and +[selected overlays](../config.md#selected-overlays) replace sequences wholesale. +There is no sequence-composition registration in `Definition` today. + +This is a narrower feature than the +[identified record sequences candidate](future.md#1-identified-record-sequences). +That candidate's composite identities, recursive merging of matching records, +identity-based overlays, and record inheritance are not part of this plan. + +## Agreed behavior + +### Registration and identities + +- Consumers explicitly designate each sequence path and one direct identity + field, such as `prompts` and `id`. Unregistered sequences retain current rules. +- Records in a participating sequence must be mappings containing the designated + field as a string. Missing, null, and non-string identities are invalid; + identities are not coerced from numbers or supplied by defaults or resolution. +- Compare strings exactly, without trimming, case folding, or normalization. + Whether the empty string is accepted is an open decision below. +- Retain identity fields as ordinary record content in the resulting sequence. + They are not consumed as metadata or converted into public mapping keys. +- This is a source-composition rule, not a guarantee of identity uniqueness after + application defaults and callbacks. Those retain their existing behavior; + applications can enforce final domain constraints through validation. + +### Strict composition + +For a registered `prompts` sequence with identity field `id`, two separate +source fragments could contribute: + +```yaml +prompts: + - id: prompt_one + text: "foo" +``` + +```yaml +prompts: + - id: prompt_two + text: "bar" +``` + +The composed value is: + +```yaml +prompts: + - id: prompt_one + text: "foo" + - id: prompt_two + text: "bar" +``` + +- Append distinct records in contribution order, preserving each source list's + order. Use existing deterministic input/import traversal within a tier; + identity values do not sort the result. +- Reject repeated identities both across contributions and within a single + contributing sequence, including a sequence that has no peer contribution. + Equal bodies and disjoint bodies both conflict. Never recursively combine + records with the same identity. +- An empty sequence contributes no records. Composing empty sequences succeeds; + an explicitly contributed empty sequence remains present, not absent. +- Apply these rules to main inputs and their imports, and to inputs and imports + composing each underlay group. Honor input and relative-import destinations. + Source acquisition kind must not change composition behavior. +- Invalid contributions and duplicate identities in a tier cannot be repaired + by a higher-precedence replacement. Report a normal load error and return no + usable partial result. + +The analogy with keyed mappings is limited: distinct keys combine, but mappings +at a shared key currently recurse. A shared record identity always conflicts +under this feature. + +### Precedence and other operations + +Sequence replacement remains the rule between underlay tiers, between underlays +and the main tier, and for the selected overlay. An upper sequence replaces the +complete lower sequence, even at a registered path; `[]` clears it. Identities +may recur across those tiers without a cross-tier duplicate error. Null and +cross-kind precedence replacements retain current semantics. + +Whole-sequence CLI overrides, defaults, application resolution, and built-in +sibling inheritance retain their existing semantics. This feature adds no +record inheritance, identity selector syntax, record update/delete/rename +operators, or identity-based behavior to effective-configuration diffs. Existing +atomic sequence diffs continue to compare final frozen values. + +### Inspection and diagnostics + +Appending records changes their effective indexes. Preserve the distinction +between an original source position and its position in the assembled sequence. +Effective presence, origins, resource contexts, history, and explanations must +refer to the correct record and its original physical source locations. + +Composition history must describe the accumulated sequence correctly, rather +than replay each fragment as a whole-sequence replacement. Later actual +replacements must still retain displaced history. Equal numeric indexes in +different source lists must not make unrelated records share provenance. + +Duplicate diagnostics must identify the conflicting source locations in +deterministic order. Avoid including identity values in diagnostic text where +sensitivity is not yet known; identify the sequence, identity field, and source +locations instead. Apply existing sensitivity and disclosure rules to successful +and failed inspection, including records whose effective indexes changed. +Retain detached public records and input/result ownership guarantees. + +## Implementation plan + +### 1. Settle the remaining contract choices + +- Resolve the Open Decisions section and update this document with the choices. +- Select the public registration names, path interpretation, and diagnostic codes. + Duplicate claims should use the existing composition-conflict category where + appropriate; malformed registration and malformed records need distinct, + actionable diagnostics. +- Record the significant composition extension in an ADR during implementation, + under the architecture policy. No ADR is created by this roadmap-writing pass. + +Completion: the public behavior and validation boundary are unambiguous, with +no accidental commitment to the broader identified-records proposal. + +### 2. Add registration and pass it through the loading paths + +- Add root-owned public registration types and fields using the selected API. + Convert them explicitly into private engine state; preserve the root facade + boundary and reusable definition ownership. +- Validate registrations before source processing and pass the same policy + through ordinary and layered loading, import assembly, and mounted inputs. +- Compile path lookup once per load if useful. No registrations must preserve + current behavior without introducing a new global cache or merge framework. + +Completion: public callers can select sequence paths consistently across load +forms, and invalid options fail through the normal diagnostics boundary. + +### 3. Extend strict composition and its retained evidence together + +The current `composeDocuments` implementation in +`internal/engine/compose.go` collects claims by path, rejects non-mapping +overlaps, merges disjoint mappings, and retains origins and history applications. +Changing only the final tree merge would leave these other representations wrong. + +- Extend claim validation and tree assembly for registered sequences. Validate + single contributions as well as overlaps. Track identities per sequence per + strict tier, and append cloned records in stable order. Keep duplicate lookup + proportional to record count rather than repeatedly scanning prior records. +- Keep original source locations while assigning effective indexes. Carry the + necessary placement information into claims, origins, resource contexts, and + history; do not expose a new public identity-addressing model. +- Update history/explanation replay to represent sequence accumulation. Ensure + snapshots, historical paths, sensitivity classification, and subsequent + whole-sequence replacement agree with the assembled tree. +- Integrate with ordinary input assembly and grouped assembly in + `internal/engine/load_input.go` and `internal/engine/load_request.go`. + Keep tier boundaries explicit so uniqueness is never enforced across + precedence layers accidentally. +- Preserve parser/source limits and retained-node accounting. Keep source-header + processing and structural-directive handling in their existing lifecycle + positions; do not restore consumed headers to supply record identities. + +Completion: final values and their inspection evidence agree across composition +and subsequent replacement. No input mutation, source-context mix-up, or +sensitive-value disclosure is introduced. + +### 4. Verify behavior at its owning boundaries + +- Composition tests: distinct and repeated identities, duplicates in one list, + malformed records and identity types, empty strings according to the chosen + decision, empty lists, ordering, missing paths, and null/cross-kind cases. +- Loading tests: imports, underlay groups, mounted and relative destinations, + equivalent source forms, and conflicts hidden by later layers. Demonstrate + that separate tiers may reuse identities and still replace whole lists. +- Metadata and inspection tests: records appended from the same source index + acquire distinct final indexes while keeping correct locations, history, + resource contexts, and sensitivity. Cover later replacement and historical + redaction, including a failure before final classification. +- Public integration tests: registration conversion and reuse, normal load-error + behavior, unchanged unregistered sequences, selected-overlay and CLI + replacement, defaults/resolution behavior, and atomic effective diffs. +- Add a small maintained consumer example showing two fragments composing and + a selected overlay replacing the result. Include only enough cases at each + layer to catch distinct failure modes. + +Run focused tests during development, then the everyday validation prescribed +by the [development guide](../development.md#baseline-validation). Run ownership +or concurrency checks where required by the change. No independent performance +benchmark campaign is part of this plan; keep measurement separate from the +busy-host environment discussed in the performance audit. + +Completion: new behavior is covered without weakening the existing atomic +replacement and default-behavior regressions. + +### 5. Publish the implemented contract in repository documentation + +- Document the public API beside its declarations. Update the configuration + contract's composition, inspection/history, and precedence sections, and the + consumer guide and maintained example together when implementation lands. +- Update the architecture policy and component overview only where the + implemented boundary changes. Keep the broader candidate in `future.md` + clearly distinct from this delivered subset. +- Retire this completed roadmap through the usual documentation lifecycle. + Release preparation and publication remain separate work. + +Completion: current documentation describes the delivered feature, and no +future proposal is presented as implemented behavior. + +## Open Decisions + +### 1. Public registration ownership + +**Recommendation:** place a collection of path/identity-field registrations on +`Definition`, shared by ordinary and layered loading. This describes how the +application's configuration is composed and avoids diverging registrations +between load forms. Choose final type and field names before coding. + +**Alternative:** place registrations on load options. That allows one definition +to be composed differently per invocation, but duplicates configuration across +ordinary/layered options and makes composition semantics less visibly part of +the application's reusable definition. + +### 2. Path matching, overlapping registrations, and unused rules + +**Recommendation:** interpret paths in effective configuration coordinates after +input/import placement, using literal mapping-key segments initially. Reject +duplicate registrations for the same path; allow a registration to match no +source value. These rules fit mounted collections without requiring discovery +of which optional sources will be present. Restricting paths to mapping segments +would be an explicit initial scope choice, not an existing library constraint. + +**Alternative:** support exact sequence-index segments too, including registered +sequences nested inside appended records. This is more expressive, but requires +choosing whether a rule matches source indexes or composed indexes, and how +matching behaves as earlier lists contribute records. Settle that interpretation +and overlapping registration behavior before supporting these paths. Wildcard +or identity-selector matching is not needed for the agreed use case. + +### 3. Empty identity strings + +**Recommendation:** accept the empty string as an exact identity; two empty +identities conflict. Presence and string type are sufficient for the composition +algorithm, and applications can require nonempty domain identifiers themselves. + +**Alternative:** require nonempty identities. This catches likely authoring +mistakes earlier, but adds a domain restriction beyond uniqueness and string +type. Whitespace trimming or normalization would still remain out of scope. + +### 4. Null and non-sequence values at registered paths + +**Recommendation:** treat registration as selecting how actual sequences +compose, not as a general schema assertion. Preserve existing rules for absent, +null, scalar, and mapping values: non-sequence overlaps retain ordinary strict +composition behavior, and final decoding/validation checks application types. +Two sequence values use the new rule; sequence/null and sequence/mapping overlaps +remain conflicts. No later layer can hide an actual strict-composition conflict. + +**Alternative:** require every present non-null source value at the path to be a +sequence, even when a later tier would replace it. This catches misplaced +configuration earlier but makes registration an additional source-shape rule. +If selected, define null handling explicitly rather than treating it as empty. + +### 5. Validation of replacement-only source sequences + +**Recommendation:** require well-formed, internally unique identities for actual +source sequences at registered paths, including selected-overlay sequences. +This makes source authoring consistent while preserving whole-sequence +replacement and permitting identities to recur across tiers. Do not extend the +rule to synthetic CLI replacements, defaults, or callback-generated values; +final domain validation remains application-owned. + +**Alternative:** validate only source sequences participating in a strict tier, +leaving selected-overlay sequences subject solely to existing decoding and +application validation. This keeps the check strictly tied to the composition +operation, but allows the same list to succeed as an overlay and fail as a base +contribution. Choose this boundary explicitly; replacement semantics alone do +not settle whether an overlay's own duplicate identities are invalid.