diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..3d1d25a --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,693 @@ +# Sequence Identity And Combination Implementation Plan + +Status: ready for implementation; no stages completed. + +Audience: a Luna coding agent implementing one numbered stage per prompt, in +order. This plan starts at commit +`4f941f8ba4c9a9ed0afa87b07fe6ba29f131d1c6`. It implements the settled +[sequence roadmap](sequences.md), [ADR 0017](../adr/0017-collection-sequence-policy.md), +and [ADR 0018](../adr/0018-cli-after-structural-resolution.md). + +## Working Rules + +- Read [development guidance](../development.md) and the applicable policies, + especially [agents](../policy/agents.md), [architecture](../policy/architecture.md), + [testing](../policy/testing.md), and [documentation](../policy/documentation.md). + Prefer the repository knowledge graph for code discovery. +- The roadmap owns the selected feature semantics. This plan supplies execution + order and implementation choices, not additional product requirements. Preserve + unchanged source, parser, version, schema, ownership, and inspection contracts. +- Complete each stage's implementation and behavioral tests together. Record its + completion and validation here, then commit the nonempty unit with a short, + plain-English message. Preserve unrelated changes. Do not publish intermediate + stages or a release, update consumers' dependencies, or wait for CI. +- Run focused tests while iterating and `./scripts/check.sh` before handing off + each code stage. Use the installed compatible Go toolchain with dependencies + already acquired, as the development guide describes. Add race coverage at the + ownership milestones below. These are implementation checks, not new release + gates. Reuse successful checks for unchanged content rather than repeating them + at a documentation-only milestone. Do not require human platform validation. +- Update affected declaration comments and canonical current-behavior prose when + behavior lands. Keep incomplete target behavior in this roadmap/plan until it + works. The final documentation stage reconciles the cross-feature narrative; + it is not permission to leave misleading current contracts in earlier stages. +- Transitional internal adapters are acceptable where specified. Remove them by + completion. Do not add public compatibility aliases, permanent feature flags, + alternate semantics, new dependencies, new resource caps, or an extension/plugin + framework for this work. + +## Implementation Decisions + +### Public Registration Surface + +Replace `SequenceComposition` and `Definition.SequenceCompositions` with: + +```go +type SequenceIdentity struct { /* private, root-owned registration state */ } +type SequenceSelector []SequenceSelectorSegment +type SequenceSelectorSegment struct { /* private literal-or-any-mapping-key state */ } + +func SequenceLiteralKey(key string) SequenceSelectorSegment +func SequenceAnyMappingKey() SequenceSelectorSegment +func SequenceIdentityAt(path Path, fields ...string) SequenceIdentity +func SequenceIdentityMatching(selector SequenceSelector, fields ...string) SequenceIdentity + +// Field on Definition[T]: +// SequenceIdentities []SequenceIdentity +``` + +For example, the reusable definition can contain both registrations: + +```go +SequenceIdentities: []configkit.SequenceIdentity{ + configkit.SequenceIdentityAt(configkit.Keys("prompts"), "id"), + configkit.SequenceIdentityMatching(configkit.SequenceSelector{ + configkit.SequenceLiteralKey("collections"), + configkit.SequenceAnyMappingKey(), + configkit.SequenceLiteralKey("prompts"), + }, "id"), +}, +``` + +The exact constructor converts mapping-only `Path` segments into literal selector +segments. Preserve invalid index segments long enough to report an ordinary +definition error; do not panic, silently drop them, or change `Path` semantics. +The zero selector is the exact root selector, harmless under the existing +mapping-root schema. A zero selector segment denotes the literal empty key, +not a wildcard. Empty field names are valid literal field names; an empty field +set is invalid. Constructors snapshot slices; load-entry conversion also takes +an owned snapshot. Definition validation, not construction, reports errors. +Root types and functions must not expose internal types or bridge methods. + +### Identity Representation And Matching + +- Normalize identity field sets once: deduplicate and sort exact field names. + Use the same normalizer for YAML and application registration. +- Encode a record's identity as a hashable, type-tagged, length-delimited tuple in + normalized field order. Mapping lookup indexes avoid scanning record fields + repeatedly. Distinguish structural-payload lookup from final typed-payload and + parent-reference lookup: only fields acting as directives are excluded, not + every occurrence of the same literal spelling. Never include tuple encodings + in diagnostics. +- Normalize finite JSON numbers to sign, significant decimal digits, and a signed + base-10 exponent. Strip insignificant zeroes and give all zero spellings one + representation. Use `math/big.Int` for exponent arithmetic, not a power-of-ten + expansion or a float conversion. Thus large exponent spellings do not allocate + storage proportional to their mathematical magnitude. This comparison-only + representation does not rewrite document nodes or typed projection. +- Compile selectors into a mapping-segment trie with literal and wildcard edges. + Carry applicable trie states through traversal; stop at sequence-index edges. + Index concrete matches within one assembly/inheritance scope where useful, but + do not save an assembly-only destination list for final validation. +- Check registration overlap pairwise once during definition validation using + equal length and compatible corresponding segments. This straightforward + `O(R² × D)` check, for R registrations and maximum selector depth D, is + proportionate for registration definitions. Equivalent constraints coalesce; + conflicting overlaps fail even with no matching input. + Do not impose a registration-count limit to simplify this operation. + +### Policy And Evidence Storage + +Keep application constraints separate from YAML policy. Add private engine +collection-policy records containing normalized identity, explicit mode, and +declaration locations. Track whether identity is pending or belongs to a retained +collection. Retain declaration presence separately from inferred defaults. + +Use the existing acquisition/item scopes as ownership anchors, with an opaque +per-load owner token when a retained record needs one. Store item-local policies +relative to their owning record, and project concrete paths when that record is +placed. Do not key ownership solely by numeric index, identity tuple, or source +filename: all three can coincide for a replacement that must lose old policies. +Inheritance clones the relevant scoped state; it does not create registrations. + +Resolve policies through shared engine helpers used by strict composition, +precedence, and inheritance. Do not attach engine policy to public values or add +policy fields to the parser-independent `document.Node`. The sidecar can be +extended with concrete-path indexes; it need not become a general event store. + +Sequence combination returns the resulting list plus explicit input-to-output +item projections and retained/replaced contribution information. Use that same +result for values, policy transfer, origins, source destinations, sensitivity, +and historical replay. Extend the existing sequence-history descriptors for +noncontiguous upsert positions rather than inferring them from equal values. +Sharing immutable internal evidence is acceptable under existing ownership rules; +independently returned mutable records must remain detached. + +### Diagnostics + +Use existing `options.invalid` for invalid application definitions and existing +`compose.invalid` / `compose.conflict` for malformed or duplicate strict-peer +contributions. Add root-owned and engine diagnostic constants +`CodeSequenceInvalid` (`sequence.invalid`) and `CodeSequenceConflict` +(`sequence.conflict`) for malformed collection declarations/operations or final +identities, and conflicting policy / precedence duplicates respectively. +Final duplicate identities use `sequence.conflict`. Keep existing inheritance +invalid/missing/cycle codes for record relationships. Use existing override and +schema/decode codes at the CLI boundary; do not invent a new error taxonomy there. + +Report collection/item paths and available locations, not identity values, +normalized tuple strings, raw parent references, or raw hook errors. Declaration +conflicts include both sites. Definition overlap errors identify registration +positions without fabricating a concrete matched configuration path. Exact +English diagnostic wording is not a new guarantee. + +## Stage 1 — Implement Shared Scalar Identity Primitives + +Scope: new private identity helpers and their focused tests; no load-path change. + +1. Add `internal/engine/sequence_identity.go`. Implement field-set normalization, + scalar tuple construction, and exact decimal normalization as chosen above. + Accept string, number, boolean, and explicit null nodes; distinguish categories + and absence. Exclude fields acting as structural directives from payload lookup. +2. Validate mapping records, complete direct-field sets, and uniqueness using one + tuple-to-first-record index. Return paths and source locations, not values. + Support caller-selected diagnostic families for strict and other boundaries. +3. Cover equivalent field sets; empty strings and null; missing/non-scalar fields; + literal punctuation; compound tuple boundaries; `1`/`1.0`/`1e0`; signed zero; + close unequal decimals; and large exponents without expansion. Verify that + node text remains unchanged. Include `extends: null` as a structural field + that cannot supply source identity. + +Validation: `go test ./internal/engine -run 'SequenceIdentity|SequenceNumber'` +and the everyday check. Retain the old registration path until its adapters are +replaced; do not generalize unrelated YAML scalar decoding. + +## Stage 2 — Compile Exact And Dynamic Application Constraints + +Scope: private registration model, selector matching, overlap validation. + +1. Extend `sequence_registration.go` with the common registration representation + and compiled selector trie. Represent literal and arbitrary-mapping-key segments + structurally, not with a magic string. Snapshot registration inputs. +2. Validate field sets and mapping-only selectors before source acquisition. + Preserve existing destination validation without adding restrictions on identity + field spelling. Root and unused selectors remain harmless. +3. Implement full-destination matching, equivalent overlap coalescing, and + incompatible-overlap errors. Exact rules have no priority over dynamic rules. + Keep a temporary conversion from the current single-field registration so + existing callers still use their current behavior. +4. Test exact and wildcard matches, multiple wildcard levels, empty/punctuation + keys, sequence-index nonmatches, disjoint lengths/literals, equivalent sets, + incompatible overlaps with absent input, and independent uniqueness scopes. + +Validation: focused registration/selector tests and the everyday check. This +stage does not export the replacement API or add final-result validation yet. + +## Stage 3 — Parse Collection Declarations Without Filesystem Coupling + +Scope: private parser and directive extraction helpers, initially unit-tested +without activating collection combination in public loads. + +1. Add `sequence_declaration.go` for source-root `configkit.sequences`. Each entry + has a required existing literal `destination` form and optional `key` and + `mode`. `key` is a string or nonempty string list; `mode` is exactly replace, + append, or upsert. Use shared identity normalization. A destination-only entry + can be a harmless no-op; do not add an unrequested ban on empty declarations. +2. Split common root-directive extraction from import-specific parsing in + `import_spec.go` / `parse.go`. Extract imports and sequences together, retaining + source-local locations and explicit presence. Reject unrecognized directive + fields and wrong supplied field types using existing strict parsing style. +3. Keep declaration destinations relative to source placement, with item-local + coordinates retained until their scope is assembled. Do not project them + prematurely to a guessed numeric index. +4. Give extraction an explicit overlay context: sequences are allowed, imports + are not. Import enablement remains separately enforced; do not turn on + filesystem access because a source declares sequences. Keep enabled + source-version checks in raw source coordinates before consuming controls; + extend existing actual header/control-overlap handling to sequence directives + without treating an absent header beneath a control as an actual overlap. +5. Test key-only, mode-only, combined, repeated-field-set equivalence, no-op, + malformed types/modes/destinations, mixed imports/sequences, and overlay rules. + Do not add import-level key/mode shorthand or sequence wrappers. + +Validation: focused declaration/parser tests, existing import-spec tests, and the +everyday check. Public activation occurs with the complete operations in Stage 9. + +## Stage 4 — Model Scoped Policy Agreement And Lifetime + +Scope: private policy resolution/state transformations, independent of loading. + +1. Add `collection_policy.go`. Gather the current scope's declarations before + combining peers. Reconcile equivalent keys and agreeing explicit modes; + conflicting peers fail. Omitted fields have no preference. Apply authoritative + application constraints even for key-only declarations without a value. +2. Implement shared operation selection: current explicit mode, retained explicit + mode, inferred upsert with effective identity, otherwise replace. Persist only + explicit modes. Policy-only declarations must not replay earlier combinations. +3. Implement key agreement, replacement reset, pending key declarations, and + immediate validation when a key-only declaration identifies retained records. + Changing mode to replace without a replacement value does not reset identity. +4. Implement scope/record ownership transfer and retirement hooks. Actual ancestor + replacement removes descendant collection identity and removed-record policy; + independently owned modes and application constraints survive as specified in + the roadmap. Do not repeatedly reapply old declarations and resurrect reset + identity. Pending declarations retain their owning-scope rules. +5. Use table-driven tests derived from the roadmap's agreement, persistence, and + replacement tables, including absent/non-sequence destinations, equal outer + identities with distinct record owners, and independent root versus item policy. + +Validation: focused collection-policy tests and the everyday check. This stage +must not add path-existence, shape, nullability, or global YAML final-check rules. + +## Stage 5 — Implement The Shared Sequence Operations + +Scope: a pure document-level combination helper, not separate implementations +for overlays and inheritance. + +1. Add `sequence_combine.go`, accepting lower/higher sequence nodes, resolved + policy, and combination context. Reuse Stage 1 identity extraction and indexes. +2. Implement replace, duplicate-rejecting append, and whole-record upsert. Upsert + retains matched lower positions and appends new higher identities in order. + Return item projections and replacement/retention evidence with the new value. +3. Keep peer validation distinct: duplicate identities within any contribution or + among peers fail under every mode. Do not merge record payloads or sort by ID. +4. Cover absent versus null/non-sequence lower values; higher omission, `[]`, null, + and non-sequence values; missing identity under append/upsert; empty inputs; + stable ordering; equivalent/different keys; and explicit replacement identity + reset. An empty input does not bypass kind checks or identity requirements. +5. Verify clones do not alias mutable input nodes or policy slices. Build indexes + once per operation rather than scanning the accumulated list for every record. + +Validation: focused combination tests and the everyday check. Do not change the +ordinary recursive mapping rule or add unkeyed append. + +## Stage 6 — Extend Sequence Evidence And Historical Replay + +Scope: teach existing provenance/history projection machinery to represent the +operations before wiring them into public precedence. + +1. Extend `sequenceHistoryContribution`, sequence replay in `history.go`, and + projection helpers to consume Stage 5's explicit item mappings. Existing append + ranges may remain a compact case; upsert cannot be represented by one offset. +2. Preserve authored raw contribution snapshots and displaced values. A replaced + record's omitted fields must appear as removed, not as retained lower payload. + Preserve equal-valued replacement events. Keep existing index-based query and + atomic sequence-diff APIs; add no public identity-selector path syntax. +3. Project composition origins, nested source destinations, historical paths, and + sensitivity through the same item mapping. Physical source IDs/locations and + resource contexts continue to identify the declaring source. +4. Exercise synthetic append/upsert/replace histories, noncontiguous matches, + nested paths, discarded descendants, and secret-marked replaced records. + Compare replayed final values with the actual combination result, including + first/last and unchanged neighboring records. Reuse existing escaped-path and + detached-query tests rather than duplicating them. + +Validation: metadata/history/sequence/provenance/sensitivity tests, +`go test -race ./internal/engine`, and the everyday check. + +## Stage 7 — Separate Acquisition From Policy-Aware Tier Composition + +Scope: refactor the assembly handoff, initially preserving existing behavior. + +1. In `load_input.go`, `imports.go`, `load_request.go`, and related graph helpers, + retain acquisition scopes, raw contributions, inclusion edges, source records, + contexts, and extracted declarations without irrevocably composing them. + Reuse existing `deferAssembly` and acquired-scope mechanisms instead of adding + a second loader. Keep already acquired document handles closed as before. +2. Separate acquiring an underlay group from composing its scopes. Keep authored + acquisition order: main and depth-first imports, each underlay/input, overlay, + then synthetic CLI records. Retain repeated-source, namespace, traversal-cycle, + accounting, and stable scheduler behavior. Do not reacquire sources to obtain + lower policy. +3. Expose one internal composition entry that accepts accumulated lower policy and + compiled application constraints. Complete item-local helper scopes before + item placement; gather declarations before composing each relevant scope. +4. Preserve existing version checks for this refactor. Retain a small adapter + invoking the current composition semantics until the next integrations. + Preserve source-version callbacks at acquisition, including header consumption. +5. Run existing acquisition, dependency scheduling, item placement, source-version, + ordinary/layered equivalence, resource-lifetime, and source-context tests. Add + only targeted regressions for any newly split handoff not already covered. + +Validation: the focused suites above and the everyday check. This refactor must +not change source IDs or turn acquisition order into precedence order. + +## Stage 8 — Compose Strict Scopes With Resolved Collection Identity + +Scope: implement the policy-aware strict composition entry, including nested +item scopes, before selecting it in all load paths. + +1. Extend `composeDocumentsWithPolicy` and `validateRegisteredSequences` to use + normalized field sets and scalar tuples. Resolve current declarations, + applicable application constraints, and eligible lower identity before peer + composition, using the Stage 4 helpers. +2. Use the existing dependency-ready order for contributions and authored list + order. Validate every identified sequence, including singleton contributions; + strictly append distinct peers and reject duplicates under all modes. +3. Complete helper imports and consume applicable non-main version headers before + checking record payload identity. Retained structural directives cannot serve + as identity fields. Do not validate an incomplete helper-built compound parent + reference here; validate that completed reference during inheritance. +4. For explicit replacement, do not use obsolete lower YAML identity to authorize + otherwise unkeyed peer composition. Likewise, a new outer record cannot borrow + nested identity from the displaced lower record. Still apply independently + registered application constraints and genuinely applicable declarations. +5. Attach Stage 6 evidence and Stage 4 scoped policy to composed results. Test + lower-established identity, peer declaration conflicts, compound/scalar IDs, + helper-supplied fields, consumed headers, and replaced record-local ownership. + +Validation: composition, sequence identity, destination, import, and version +tests plus the everyday check. Keep unregistered non-mapping conflicts unchanged. + +## Stage 9 — Activate Ordinary Loading And Selected Overlay Policies + +Scope: ordinary `Load`, source declarations, and its selected overlay. + +1. Wire common directive extraction into every ordinary source form: file, + native filesystem, bytes, reader, mapping, imported mapping, and sequence item. + Declarations work without `Imports`; disabled imports retain their existing + rejection semantics. `NoSource` remains absence rather than an empty document. +2. Use Stage 8 composition, followed by Stage 5 precedence operations in the + assembly accumulator. Reuse the Stage 4 policy resolver; mappings still recurse + and unrelated sequences still replace. Preserve mode-only/key-only effects. +3. Permit `configkit.sequences` in selected overlays while continuing to reject + overlay imports. Strip structural controls from application decoding, but + retain their locations/evidence for their structural lifetime and diagnostics. +4. Apply projection and policy-retirement information as part of each operation, + not in a later best-effort repair pass. Cover inline/imported equivalence, + placement-relative declarations, append/upsert/replace, empty lists, retained + modes, and declarations without accompanying values. +5. Update current sequence/overlay contract sections and tests affected by this + activation. CLI and inheritance retain their explicitly incomplete milestones + until their later stages; do not claim those parts of the roadmap implemented. + +Validation: ordinary-load, overlay, import, composition, and metadata tests plus +the everyday check. Include one public ordinary-load declaration example test. + +## Stage 10 — Integrate Lower Policy Across Layered Assembly + +Scope: source-backed underlays and the layered selected-overlay path. + +1. Acquire main first, but assemble underlays from lowest to highest, strictly + composing each group with applicable accumulated lower policy before applying + its one precedence operation. Then compose/apply main and the selected overlay. + Do not sequentially upsert peer sources as if they were different tiers. +2. Pass lower identity into higher strict scopes before peer composition. Cover an + otherwise empty underlay establishing identity, inherited explicit mode, and + replacement that deliberately discards the old YAML key. Invalid lower tiers + still fail even if a higher replacement would hide their values. +3. Keep main-only version ownership and every enabled per-source version rule. + In default source-version mode, run the main checker once on the composed + main tier before overlay/inheritance/defaults, never on underlay-augmented + values. Its former main-before-underlay composition timing need not persist: + main composition now depends on lower policy. Enabled-mode source checks keep + acquisition order and exact source-local header handling. No extra callbacks + or second acquisition pass are permitted. +4. Keep histories in precedence order, sources in acquisition order, underlay + indexes stable across empty groups, and shared budgets intact. Update the + layered contract's acquisition-versus-composition description accordingly. +5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main, + explicit empty groups, layered overlays, version protection and diagnostics, + and equivalence with ordinary loading where the inputs express the same model. + +Validation: layered/grouped, source-version, dependency scheduling, destination, +and metadata tests plus the everyday check. + +## Stage 11 — Verify Replacement And Nested Policy Ownership End To End + +Scope: the consequential lifetime cases across the now-active source assembly. + +1. Exercise an item-local nested collection through retaining the outer record, + upserting the same outer ID, and replacing the entire outer sequence. Retained + items keep their local key/mode; replacements do not inherit discarded policy + merely because identity, source label, or final index is equal. +2. Exercise ancestor replacement and later reintroduction. Removed collection + identity does not reappear; independently declared append mode can survive + and require renewed identity. Cover renewed YAML key, application constraint, + explicit replace, and the intended failure with no available identity. +3. Cover key-only validation of retained records, pending declarations, a no-value + replace declaration that cannot authorize key changes, and independent policy + whose owning scope survives a record replacement. +4. Correct any integration defects in the Stage 4 ownership transformations; + consolidate tests with their mechanism owners. Do not introduce special-case + ownership by numeric path or a historical identity cache to make cases pass. +5. Verify frozen metadata, origin/resource context, sensitivity of discarded + records, and mutable independence across repeated loads for these scenarios. + +Validation: focused policy/sequence/layered/ownership tests, +`go test -race ./internal/engine`, and the everyday check. + +## Stage 12 — Apply Collection Policy During Mapping-Sibling Inheritance + +Scope: retain existing mapping-sibling relationships, extend their collection +combination and policy transfer. + +1. Extend inheritance templates/expansion sidecars to carry collection identity + and record-local policy alongside nodes and origins. Keep existing scheduler + keys, cancellation, version protection, missing-parent/cycle diagnostics, and + destination-local evaluation of nested templates. +2. Use the shared sequence combiner at the inheriting destination: parent lower, + child higher, destination explicit mode or identity-based default. Copy YAML + identity where applicable, but do not automatically copy a parent collection's + explicit mode to a different destination. Match application constraints at the + destination, including newly created matching paths. +3. Transfer record-owned state only with the records that survive. Preserve + original nested instructions for each destination's later resolution rather + than copying only a parent's already-expanded output. +4. Extend inheritance evidence using the same projections as source operations. + Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key + with inferred upsert, destination explicit mode, incompatible keys, application + constraints, cancellation, and policy at an inherited new namespace. + +Validation: inheritance, history, provenance, sensitivity, and resource-context +tests plus the everyday check. No identified-record parent lookup is active yet. + +## Stage 13 — Build Identified-Record Inheritance Graphs + +Scope: relationship extraction/validation and shared scheduling for sequence +scopes; test these helpers before activating payload expansion. + +1. Extend sibling-scope discovery to identified sequences. Build the tuple-to-item + lookup once using effective identity. Every record supplies its own complete + identity before inheritance; neither parent lookup nor typed defaults supplies + a missing child field. +2. Parse a surviving record's `extends` as null cancellation or a mapping with + exactly the normalized identity fields and supported scalar components. + Mapping-sibling references keep their existing string form. Do not accept a + scalar shorthand, multiple parents, positional or cross-collection lookup. +3. Parent-reference mappings have already used ordinary strict mapping + composition: disjoint helper fields can complete one reference, while equal + overlapping fields conflict. Validate completeness/extra fields only now on + the surviving reference. Discarded references remain historical evidence. +4. Build one-parent dependency edges and reuse `scheduleDependencies` with stable + semantic-pointer keys. Missing parents, self-reference, and longer cycles + use existing inheritance diagnostics without identity values. Keep authored + output positions independent from graph schedule. +5. Cover compound and null references, numeric equality, forward edges, chains, + unrelated equal identities in different lists, malformed refs, missing parents, + cycles, helper-built references, and overwritten/cancelled relationships. + +Validation: focused inheritance graph/reference tests and the everyday check. + +## Stage 14 — Resolve Record Inheritance With Ownership And Evidence + +Scope: integrate Stage 13 graphs into the existing structural expansion phase. + +1. Expand parent payload into the child while preserving the child's own identity + fields and their origin/presence. Exclude parent identity fields from payload + inheritance. Recursively combine mappings and use destination collection policy + for nested sequences; retain whole-record upsert at the earlier boundary. +2. Consume the child's directive before decoding. Keep parents as ordinary records + subject to the same schema/semantic validation; create no abstract templates or + acquired source records. +3. Resolve only records surviving tier assembly. Cover parent upsert/removal, + replacement of a child without `extends`, self-reference not exposing a lower + predecessor, and nested destination-local relationships. Enclosing scopes + finish before nested scopes; assembled list order stays unchanged. +4. Carry policy, origin/history, sensitivity, resource context, and existing + expansion/retained-history accounting through copies. Do not widen budgets or + require a new inspection-path API. +5. Add public-load regressions for complete inheritance chains and nested + collections, including secret canaries on success and failure. Update the + implemented inheritance contract and consumer guidance for this scope. + +Validation: all inheritance and cross-feature inspection tests, +`go test -race ./internal/engine`, and the everyday check. + +## Stage 15 — Expose The Root-Owned Registration API + +Scope: the settled facade surface and migration of in-repository callers. + +1. Implement the exact public names/signatures above in a focused root file, and + add `Definition.SequenceIdentities`. Convert root records/selectors explicitly + into the engine model. Snapshot fields, selectors, and paths on construction + and loading, preserving immutable definition reuse. +2. Remove `SequenceComposition`, `Definition.SequenceCompositions`, and the private + old-shape adapters. Migrate all maintained code/tests/examples; do not edit + historical release notes solely to erase the old name or add deprecated aliases. +3. Cover mixed exact/dynamic constructors, compound scalar fields, literal `*` + versus wildcard segments, root/unused selectors, invalid index paths and empty + field sets, overlap failure before source access, and independent loads. +4. Prove dynamic constraints apply before strict composition and at inherited + destinations, without consumer namespace enumeration. Keep final-result + enforcement described as pending until Stage 18 actually adds it. +5. Verify the public boundary and declaration documentation. Do not expose engine + enums, policy stores, parser nodes, or internal methods. + +Validation: root sequence/facade, registration, public/dependency-boundary, and +consumer example tests plus the everyday check. + +## Stage 16 — Prepare Schema-Aware Post-Decode CLI Application + +Scope: a private assignment helper and structural-directive validation, without +switching lifecycle order until Stage 17. + +1. Refactor override preparation so labels, paths, snapshots, type support, + disjointness, version protection, and existing budgets remain reusable outside + the source assembly accumulator. Allocate synthetic sources once in the same + order; do not reopen or reinterpret CLI values as YAML. +2. Extend structural rejection to `extends` as well as `configkit` in supplied + paths and nested values, using their established structural roles. This is + post-structural application data, not another instruction channel. Do not add + unrelated name restrictions. +3. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind + only supplied override subtrees against a detached copy of the decoded state. + Mappings recursively overlay, sequences replace, and explicit zero/empty/null + retain normal schema rules. Current typed values supply retained children, + not new defaults for replaced list items. Do not call `Definition.Defaults` + again or re-decode the whole projected configuration as newly authored input. +4. Return the updated typed value, canonical post-CLI projection, presence/origin + changes, and existing raw CLI history contributions. Preserve a pre-CLI typed + projection for decoding reconciliation/history. No resolver runs on a failed + override; no partially applied result is returned. +5. Test unknown/type-invalid fields, mapping retention, sequence replacement, + nested maps/lists, named/text/duration/secret schema fields, explicit null/false/ + zero, discarded children, structural controls, and mutable ownership. Reuse + existing label/path/cycle/budget tests without recopying their full inventory. + +Validation: override, decode, schema, projection, and ownership tests plus the +everyday check. Only the helpers are added here; lifecycle changes next. + +## Stage 17 — Move CLI After Decoding And Preserve Inspection Order + +Scope: activate ADR 0018 consistently through both public loading paths. + +1. Remove CLI assignment from ordinary/layered structural assembly. Invoke + Stage 16 after inheritance and successful typed-default creation/decoding, + immediately before application resolution. Keep the single shared lifecycle; + do not add separate ordinary and layered CLI implementations. +2. Seed resolution with the post-CLI configuration, explicit-presence/origin + metadata, source records, and contexts. Ensure `Resolution.Default` preserves + explicit CLI values while recorded transformations can change them normally. +3. Extend existing replay/reconciliation so evidence follows the real order: + source tiers, inheritance, typed decoding/defaults, CLI, application resolution. + Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and + post-CLI projection distinct where needed. Do not report CLI changes as + defaults, decoding conversions, or inheritance effects. +4. Feed effective post-override/post-resolution reference values and their + locations into secret lookup and final validation. Do not use stale structural + input to determine effective CLI secret references. Preserve no-credential-byte + history and exactly-once default-factory behavior. +5. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an + identified list despite append/upsert mode, derived values seeing CLI input, + equal-value provenance, and resolver changes after CLI. Add explicit regressions + that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a + relationship. Update obsolete early-CLI tests deliberately. +6. Update the lifecycle, override, history, and secret-reference contracts and + affected examples to describe the now-implemented order. + +Validation: override/lifecycle/resolution/inheritance/history/secret tests, +`go test -race ./...`, and the everyday check. + +## Stage 18 — Enforce Final Application Identity Constraints + +Scope: the final-result guarantee, not another combination or inheritance phase. + +1. Add shared final identity validation after effective secret lookup and before + application semantic validation/publication in the common lifecycle. Use the + canonical effective typed projection, which retains secret references rather + than credential bytes. Preserve the existing validated publication snapshot + and detachment behavior. +2. Traverse the actual final tree with the compiled selector trie, validating each + matching sequence independently with Stage 1 primitives. This must discover + paths introduced by typed defaults, CLI, and callbacks, not only assembly paths. + Support root/namespaced rules and unrelated unselected sequences. +3. Validate actual sequences only. Missing paths, null, and other shapes remain + the schema/application validator's responsibility. Identity extraction here + examines effective typed payload, not whether a field was authored. Do not + recheck YAML-only policies or every intermediate callback state. +4. Cover duplicate/missing/non-scalar identities produced at each late phase, + valid null/scalar identities, numeric equality after typed conversion, dynamic + namespaces created by each late phase, harmless unmatched rules, concrete-path + diagnostics with available origins, and secret-value non-disclosure. +5. Prove both `Load` and `LoadLayered` use this guarantee and return no usable + result on failure. Update registration comments and canonical contract with + this final boundary, excluding later caller mutations from the guarantee. + +Validation: final identity, selector, facade, defaults/resolution/override, and +secret tests plus the everyday check. + +## Stage 19 — Exercise Representative Consumer Workflows + +Scope: a small number of meaningful integrations, not a Cartesian test matrix. + +1. Extend `examples/consumers` with an executable source-backed catalog: underlay + records, main fragments, selected overlay append/upsert/replace, and explicit + record inheritance. Demonstrate complete replacement of a matched record, + stable order, and final inspection without application merge code. +2. Add a namespaced typed-map example using one wildcard registration plus an + exact root registration, with independent uniqueness. Show YAML-declared + identity without registration separately so the final guarantee is not implied. +3. Include one integrated CLI/resolution case and one representative provenance/ + resource-context/secret-history case where those interactions add protection + beyond mechanism tests. Keep application domain validation recognizable. +4. Audit coverage against the roadmap's work inventory, especially helper-built + parent references, version-header consumption, pending declarations, lower + identity before higher peers, replacement ownership, and dynamic destinations + introduced through inheritance and late typed changes. Add only missing, + consequential regressions at their narrowest stable owners. +5. Consolidate obsolete/brittle tests. Do not restore old replacement semantics + merely to keep fixtures unchanged, and do not preserve duplicates of the same + semantic table across every layer. + +Validation: `go test ./examples/consumers`, focused root integration tests, +`go test -race ./...`, and the everyday check. + +## Stage 20 — Reconcile Documentation And Complete The Review + +Scope: finish the current-contract narrative and verify the full target state. + +1. Reconcile `docs/config.md` for declarations, identity equality, selectors, + operations/persistence, ownership, inheritance, final validation, and CLI order. + Keep its operation and presence tables authoritative; Go comments own exported + symbols and consumer guidance links rather than copying full contracts. +2. Update `docs/consumers/README.md`, relevant root examples, `README.md`, + `docs/internal/overview.md`, `docs/internal/imports.md`, and + `docs/development.md` where the implemented structure or reading/test map has + changed. Remove stale claims about string-only registration, universal sequence + replacement, early CLI, and overlay prohibition of all directives. +3. Remove implemented portions from the deferred identified-record discussion in + `future.md`, retaining genuinely deferred recursive body merging and other + still-relevant ideas. Preserve accepted ADR decision content; update only status + or implementation references where appropriate. Architecture already adopts + the decisions: remove pending-implementation wording when true, without adding + new policy. Leave release version selection/publication to the release task. +4. Inspect the completed code for duplicate combination/identity algorithms, + per-record scans of whole collections, repeated declaration gathering, numeric + magnitude expansion, stale ownership/path caches, and unnecessary copies. + Prefer shared indexed mechanisms without pooling/global caches. Reuse existing + composition/underlay/inheritance benchmarks and add a focused mixed-upsert or + wildcard-collection benchmark only where existing fixtures cannot exercise the + new scaling risk. Record commands and qualitative findings here; timing or + allocation thresholds and a quiet-host benchmark campaign are not gates. +5. Remove transitional adapters and dead early-CLI paths. Search current code and + maintained examples for `SequenceCompositions` / `SequenceComposition`; only + intentional historical references may remain. Confirm no new dependencies, + public internal-type exposure, background work, release machinery, or additional + limits were introduced. +6. Run `./scripts/check.sh`, `go test -race ./...`, and the development guide's + local documentation/public-boundary checks, reusing Stage 19's successful code + checks if code is unchanged and rerunning affected documentation checks. Review + the full diff against the roadmap, not merely passing tests. Record completed + scope, validation, and any real remaining issues accurately; do not mark this + plan complete with an unimplemented target behavior. + +Acceptance: all roadmap operations and lifecycle boundaries work through ordinary +and layered loads; values and inspection agree; application constraints hold on +successful final results; deferred behavior and YAML-only guarantees have not +expanded. All twenty stages are complete and their nonempty units committed. + +## Open Questions + +None. Feature choices are settled in the roadmap and accepted ADRs; this plan +selects the remaining API, representation, validation, and execution decisions. diff --git a/docs/roadmap/sequences.md b/docs/roadmap/sequences.md index 231f494..52d1e77 100644 --- a/docs/roadmap/sequences.md +++ b/docs/roadmap/sequences.md @@ -1,9 +1,9 @@ # Sequence Identity And Combination Roadmap -Status: settled feature roadmap, ready for a staged implementation plan. This -is the targeted end state, not an implemented contract. The subsequent plan will -specify execution order, exact Go names and signatures, internal bookkeeping, -and validation stages. +Status: settled feature roadmap; implementation is specified in the +[staged implementation plan](implementation.md). This document defines the +targeted end state, not an implemented contract. The implementation plan owns +execution order, exact Go names and signatures, bookkeeping, and validation. ## Purpose And Scope @@ -906,8 +906,9 @@ policy ownership and inheritance transfer. ## Work Needed From The Baseline -The implementation plan should cover these connected changes, with execution -order and test sequencing defined there: +The baseline needs the following connected capabilities to reach the target +state. Execution order and test sequencing belong to the +[implementation plan](implementation.md): - **Declarations and public API:** extend application registration to the settled identity model with one public registration type and definition field, an @@ -993,6 +994,6 @@ status while preserving its decision content. The architecture policy is aligned Dynamic selectors extend the destinations covered by application registration; they retain the accepted application authority and final-validation boundary. -No open feature decisions remain. The implementation plan should translate this -end state into reviewable stages without adding guarantees, restrictions, or +No open feature decisions remain. The implementation plan translates this end +state into reviewable stages without adding guarantees, restrictions, or combination semantics.