From bc6b977585896aaa5c24f6d0dfdec0b3fbffbe51 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 23 Oct 2026 13:40:29 +0000 Subject: [PATCH] Split the sequence implementation into smaller stages --- docs/roadmap/implementation.md | 473 +++++++++++++++++++++++---------- 1 file changed, 337 insertions(+), 136 deletions(-) diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 3d1d25a..ba39da8 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -35,6 +35,9 @@ and [ADR 0018](../adr/0018-cli-after-structural-resolution.md). completion. Do not add public compatibility aliases, permanent feature flags, alternate semantics, new dependencies, new resource caps, or an extension/plugin framework for this work. +- Preparation stages keep existing public behavior working and test new machinery + privately. Activation stages integrate values, policy, metadata, and relevant + failure handling together; do not defer correct inspection to a later stage. ## Implementation Decisions @@ -196,7 +199,9 @@ stage does not export the replacement API or add final-result validation yet. ## Stage 3 — Parse Collection Declarations Without Filesystem Coupling Scope: private parser and directive extraction helpers, initially unit-tested -without activating collection combination in public loads. +without activating collection combination in public loads. Keep source-form +activation out of this stage; ordinary sources and selected overlays are wired +in Stages 13 and 14 respectively. 1. Add `sequence_declaration.go` for source-root `configkit.sequences`. Each entry has a required existing literal `destination` form and optional `key` and @@ -221,11 +226,12 @@ without activating collection combination in public loads. Do not add import-level key/mode shorthand or sequence wrappers. Validation: focused declaration/parser tests, existing import-spec tests, and the -everyday check. Public activation occurs with the complete operations in Stage 9. +everyday check. Existing import-only behavior remains intact during preparation. -## Stage 4 — Model Scoped Policy Agreement And Lifetime +## Stage 4 — Reconcile Declarations And Select Collection Operations -Scope: private policy resolution/state transformations, independent of loading. +Scope: pure agreement and operation-selection helpers, independent of loading +and of the ownership transformations introduced next. 1. Add `collection_policy.go`. Gather the current scope's declarations before combining peers. Reconcile equivalent keys and agreeing explicit modes; @@ -234,22 +240,42 @@ Scope: private policy resolution/state transformations, independent of loading. 2. Implement shared operation selection: current explicit mode, retained explicit mode, inferred upsert with effective identity, otherwise replace. Persist only explicit modes. Policy-only declarations must not replay earlier combinations. -3. Implement key agreement, replacement reset, pending key declarations, and - immediate validation when a key-only declaration identifies retained records. +3. Accept eligible lower policy as an input; do not decide record survival or + ancestor removal inside declaration reconciliation. Retain declaration + locations/presence separately from inferred mode and application constraints. +4. Test equivalent/different field sets, agreeing/conflicting peer modes, + omission, explicit default modes, and operation inference. Include key-only + declarations agreeing or conflicting with application constraints before a + sequence exists. + +Validation: focused policy-agreement/selection tests and the everyday check. +Do not activate collection combination or implement the ownership layer here. + +## Stage 5 — Track Policy Lifetime And Record Ownership + +Scope: private scoped-state transformations built on Stage 4's resolution helpers. + +1. Implement key agreement across retained values, replacement reset, pending + key declarations, and immediate validation when a key-only declaration + identifies retained records. Changing mode to replace without a replacement value does not reset identity. -4. Implement scope/record ownership transfer and retirement hooks. Actual ancestor +2. Implement scope/record ownership transfer and retirement hooks. Actual ancestor replacement removes descendant collection identity and removed-record policy; independently owned modes and application constraints survive as specified in the roadmap. Do not repeatedly reapply old declarations and resurrect reset identity. Pending declarations retain their owning-scope rules. -5. Use table-driven tests derived from the roadmap's agreement, persistence, and +3. Use item/acquisition scope anchors and record-owned relative policy as selected + above. Provide transformations for retention, whole-record replacement, + whole-sequence replacement, and inheritance copies without wiring them into + public loading yet. Application registrations remain destination constraints. +4. Use table-driven tests derived from the roadmap's lifetime, persistence, and replacement tables, including absent/non-sequence destinations, equal outer identities with distinct record owners, and independent root versus item policy. Validation: focused collection-policy tests and the everyday check. This stage must not add path-existence, shape, nullability, or global YAML final-check rules. -## Stage 5 — Implement The Shared Sequence Operations +## Stage 6 — Implement The Shared Sequence Operations Scope: a pure document-level combination helper, not separate implementations for overlays and inheritance. @@ -271,112 +297,189 @@ for overlays and inheritance. Validation: focused combination tests and the everyday check. Do not change the ordinary recursive mapping rule or add unkeyed append. -## Stage 6 — Extend Sequence Evidence And Historical Replay +## Stage 7 — Extend Sequence History Representation And Replay -Scope: teach existing provenance/history projection machinery to represent the -operations before wiring them into public precedence. +Scope: represent and replay sequence operations without yet changing all metadata +projection consumers or public precedence. 1. Extend `sequenceHistoryContribution`, sequence replay in `history.go`, and - projection helpers to consume Stage 5's explicit item mappings. Existing append - ranges may remain a compact case; upsert cannot be represented by one offset. + its contribution indexes to consume Stage 6's explicit item mappings. Existing + append ranges may remain a compact case; upsert cannot be represented by one + offset. 2. Preserve authored raw contribution snapshots and displaced values. A replaced record's omitted fields must appear as removed, not as retained lower payload. Preserve equal-valued replacement events. Keep existing index-based query and atomic sequence-diff APIs; add no public identity-selector path syntax. -3. Project composition origins, nested source destinations, historical paths, and - sensitivity through the same item mapping. Physical source IDs/locations and - resource contexts continue to identify the declaring source. -4. Exercise synthetic append/upsert/replace histories, noncontiguous matches, - nested paths, discarded descendants, and secret-marked replaced records. - Compare replayed final values with the actual combination result, including - first/last and unchanged neighboring records. Reuse existing escaped-path and - detached-query tests rather than duplicating them. +3. Exercise synthetic append/upsert/replace histories, noncontiguous matches, + nested paths, and discarded descendants. Compare replayed final values with + the actual combination result, including first/last and unchanged neighboring + records. Reuse existing escaped-path and detached-query tests rather than + duplicating them. + +Validation: sequence/history/replay tests and the everyday check. Keep current +append/replace callers working through the extended descriptors. + +## Stage 8 — Project Sequence Origins And Inspection Evidence + +Scope: use the prepared item mappings consistently in metadata consumers. + +1. Project composition origins, nested source destinations, historical paths, and + sensitivity through Stage 6's item mapping and Stage 7's history descriptors. + Physical source IDs/locations and resource contexts continue to identify the + declaring source. Reuse one projection mechanism rather than searching again + by identity or equal payload. +2. Test synthetic noncontiguous upserts, replaced descendants, nested source + destinations, unchanged neighbors, and secret-marked replaced records. Verify + projected queries agree with replayed and actual final values, retaining + displaced evidence and detached results. +3. Keep the new precedence operations unactivated until their value, policy, and + projection machinery are integrated in the later load-path stages. Validation: metadata/history/sequence/provenance/sensitivity tests, `go test -race ./internal/engine`, and the everyday check. -## Stage 7 — Separate Acquisition From Policy-Aware Tier Composition +## Stage 9 — Separate Per-Input Acquisition From Composition -Scope: refactor the assembly handoff, initially preserving existing behavior. +Scope: a behavior-preserving acquired-input handoff; grouped underlay coordination +is the next stage. -1. In `load_input.go`, `imports.go`, `load_request.go`, and related graph helpers, - retain acquisition scopes, raw contributions, inclusion edges, source records, +1. In `load_input.go`, `imports.go`, and related graph helpers, retain acquisition + scopes, raw contributions, inclusion edges, source records, contexts, and extracted declarations without irrevocably composing them. Reuse existing `deferAssembly` and acquired-scope mechanisms instead of adding a second loader. Keep already acquired document handles closed as before. -2. Separate acquiring an underlay group from composing its scopes. Keep authored - acquisition order: main and depth-first imports, each underlay/input, overlay, - then synthetic CLI records. Retain repeated-source, namespace, traversal-cycle, - accounting, and stable scheduler behavior. Do not reacquire sources to obtain - lower policy. -3. Expose one internal composition entry that accepts accumulated lower policy and - compiled application constraints. Complete item-local helper scopes before - item placement; gather declarations before composing each relevant scope. -4. Preserve existing version checks for this refactor. Retain a small adapter - invoking the current composition semantics until the next integrations. +2. Separate acquisition from the per-input composition entry. Retain pending + item/helper scope boundaries, namespace identities, inclusion/repetition checks, + source records, accounting, and scheduler inputs. Do not reacquire sources to + compose them later or change grouped coordination yet. +3. Preserve existing version checks for this refactor. Retain a small adapter + invoking current composition semantics at the existing lifecycle point. Preserve source-version callbacks at acquisition, including header consumption. -5. Run existing acquisition, dependency scheduling, item placement, source-version, +4. Run existing acquisition, dependency scheduling, item placement, source-version, ordinary/layered equivalence, resource-lifetime, and source-context tests. Add only targeted regressions for any newly split handoff not already covered. Validation: the focused suites above and the everyday check. This refactor must not change source IDs or turn acquisition order into precedence order. -## Stage 8 — Compose Strict Scopes With Resolved Collection Identity +## Stage 10 — Separate Underlay-Group Acquisition From Composition -Scope: implement the policy-aware strict composition entry, including nested -item scopes, before selecting it in all load paths. +Scope: behavior-preserving group coordination in `load_request.go`, using the +per-input handoff already prepared in Stage 9. + +1. Separate acquiring an underlay group from scheduling/composing its acquired + scopes. Preserve namespace reanchoring, shared inclusion/repetition state, + item-local helper scopes, source/context aggregation, and group accounting. +2. Keep authored acquisition order: main and depth-first imports, each underlay + and input, overlay, then synthetic CLI records. Preserve underlay indexes and + current version-check timing in this refactor; policy-dependent layered + composition is activated only in Stage 15. +3. Expose the common scope-composition handoff with inputs for accumulated lower + policy and compiled application constraints. Initially adapt it to the current + semantics. Do not add another loader or repeat acquisition/version callbacks. +4. Test the split with existing grouped/dependency/version/destination coverage, + including empty inputs/groups and repeated sources within versus across tiers. + Confirm unchanged values, source ordering, contexts, and failure evidence. + +Validation: grouped/layered, import scheduling, version, destination, and +resource-context tests plus the everyday check. + +## Stage 11 — Compose Strict Collections With Resolved Identity + +Scope: the policy-aware strict composition core. Nested pending-item scope +integration is deferred to Stage 12, not absorbed into this prompt. 1. Extend `composeDocumentsWithPolicy` and `validateRegisteredSequences` to use normalized field sets and scalar tuples. Resolve current declarations, applicable application constraints, and eligible lower identity before peer - composition, using the Stage 4 helpers. + composition, using the Stage 4 agreement and Stage 5 lifetime helpers. 2. Use the existing dependency-ready order for contributions and authored list order. Validate every identified sequence, including singleton contributions; strictly append distinct peers and reject duplicates under all modes. -3. Complete helper imports and consume applicable non-main version headers before - checking record payload identity. Retained structural directives cannot serve - as identity fields. Do not validate an incomplete helper-built compound parent - reference here; validate that completed reference during inheritance. -4. For explicit replacement, do not use obsolete lower YAML identity to authorize - otherwise unkeyed peer composition. Likewise, a new outer record cannot borrow - nested identity from the displaced lower record. Still apply independently - registered application constraints and genuinely applicable declarations. -5. Attach Stage 6 evidence and Stage 4 scoped policy to composed results. Test - lower-established identity, peer declaration conflicts, compound/scalar IDs, - helper-supplied fields, consumed headers, and replaced record-local ownership. +3. For explicit replacement, do not use obsolete lower YAML identity to authorize + otherwise unkeyed peer composition. Still apply independently registered + application constraints and genuinely applicable declarations. +4. Attach Stages 7–8 evidence and Stage 5 scoped policy to composed results. Test + completed-document contributions with lower-established identity, declaration + conflicts, compound/scalar IDs, singleton validation, and replacement resets. -Validation: composition, sequence identity, destination, import, and version -tests plus the everyday check. Keep unregistered non-mapping conflicts unchanged. +Validation: composition and sequence identity tests plus the everyday check. +Keep unregistered non-mapping conflicts unchanged. Public load activation remains +in the later integration stages. -## Stage 9 — Activate Ordinary Loading And Selected Overlay Policies +## Stage 12 — Integrate Policy-Aware Item And Helper Scopes -Scope: ordinary `Load`, source declarations, and its selected overlay. +Scope: adapt pending-item/helper composition to Stage 11's core using the acquired +scope boundaries from Stages 9–10. + +1. Gather declarations before composing each relevant scope, complete item-local + helper imports before item placement, and consume applicable non-main version + headers before checking record payload identity. Keep relative policy with its + owner until concrete placement is known. +2. Apply eligible lower identity at each scope, but prevent a replacement outer + record from borrowing nested identity owned by the discarded lower record. + Preserve independently applicable policy and application constraints. +3. Retained structural directives cannot serve as identity fields. A helper may + supply payload identity components. Do not require a helper-built compound + parent reference to be complete at this point; validate the completed surviving + reference during inheritance. +4. Project item evidence and owned policies through Stages 7–8 machinery. Test + nested items, helper-supplied identity, header consumption, declaration scope, + discarded owner identity, and exact diagnostic paths with no value disclosure. + +Validation: item/destination/import/composition/source-version tests and the +everyday check. Keep current public behavior until load-path activation. + +## Stage 13 — Activate Ordinary Source Collection Declarations + +Scope: ordinary `Load` base-source assembly, excluding activation of the new +selected-overlay policy surface until Stage 14. 1. Wire common directive extraction into every ordinary source form: file, native filesystem, bytes, reader, mapping, imported mapping, and sequence item. Declarations work without `Imports`; disabled imports retain their existing rejection semantics. `NoSource` remains absence rather than an empty document. -2. Use Stage 8 composition, followed by Stage 5 precedence operations in the - assembly accumulator. Reuse the Stage 4 policy resolver; mappings still recurse - and unrelated sequences still replace. Preserve mode-only/key-only effects. -3. Permit `configkit.sequences` in selected overlays while continuing to reject - overlay imports. Strip structural controls from application decoding, but - retain their locations/evidence for their structural lifetime and diagnostics. -4. Apply projection and policy-retirement information as part of each operation, - not in a later best-effort repair pass. Cover inline/imported equivalence, - placement-relative declarations, append/upsert/replace, empty lists, retained - modes, and declarations without accompanying values. -5. Update current sequence/overlay contract sections and tests affected by this - activation. CLI and inheritance retain their explicitly incomplete milestones - until their later stages; do not claim those parts of the roadmap implemented. +2. Use Stages 11–12 composition and Stage 5 scoped policy. Strip collection + controls from application decoding, but retain their locations/evidence for + their structural lifetime and diagnostics. +3. Integrate policy/projections together with the values. Cover inline/imported + equivalence, placement-relative declarations, singleton and peer validation, + and declarations without accompanying values. Maintain ordinary loading's + existing non-collection overlay behavior while preparing its policy handoff. +4. Update the implemented declaration/composition contract sections. Collection + precedence, inheritance, and later CLI behavior remain incomplete until their + integration stages; do not claim them implemented here. -Validation: ordinary-load, overlay, import, composition, and metadata tests plus +Validation: ordinary-load, import, composition, and metadata tests plus the everyday check. Include one public ordinary-load declaration example test. -## Stage 10 — Integrate Lower Policy Across Layered Assembly +## Stage 14 — Activate Selected Overlay Collection Policies -Scope: source-backed underlays and the layered selected-overlay path. +Scope: ordinary loading's selected overlay and shared precedence integration, +using the source/declaration handoff already active in Stage 13. + +1. Permit `configkit.sequences` in selected overlays while continuing to reject + overlay imports. Preserve source-version header/control handling and ordinary + overlay namespace selection. Consume controls before application decoding. +2. Use Stage 6's sequence operations and Stages 4–5 policy helpers in the assembly + accumulator. Mappings still recurse and unrelated sequences still replace. + Wire value, policy retirement/transfer, and Stages 7–8 inspection projections + together, not as a later best-effort repair pass. +3. Cover append/upsert/replace, empty/null/kind cases, key-only/mode-only changes, + retained explicit modes, overlay identity agreement, and independently owned + policies. Check values, history, origins, and discarded descendants together. +4. Update the selected-overlay and collection-operation contract sections. Make + the shared precedence entry reusable by layered loading without adding another + operation implementation or turning overlay imports on. + +Validation: ordinary-load, overlay, composition, policy, and metadata tests plus +the everyday check. + +## Stage 15 — Integrate Lower Policy Across Layered Assembly + +Scope: source-backed underlay coordination and the layered selected-overlay path, +using the prepared acquisition, ownership, strict-scope, and precedence machinery. +Do not absorb unfinished work from Stages 5–14 into this stage. 1. Acquire main first, but assemble underlays from lowest to highest, strictly composing each group with applicable accumulated lower policy before applying @@ -396,6 +499,8 @@ Scope: source-backed underlays and the layered selected-overlay path. 4. Keep histories in precedence order, sources in acquisition order, underlay indexes stable across empty groups, and shared budgets intact. Update the layered contract's acquisition-versus-composition description accordingly. + Remove the acquisition/composition adapters superseded by this integration; + the old public-registration adapter remains until Stage 22. 5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main, explicit empty groups, layered overlays, version protection and diagnostics, and equivalence with ordinary loading where the inputs express the same model. @@ -403,7 +508,7 @@ Scope: source-backed underlays and the layered selected-overlay path. Validation: layered/grouped, source-version, dependency scheduling, destination, and metadata tests plus the everyday check. -## Stage 11 — Verify Replacement And Nested Policy Ownership End To End +## Stage 16 — Verify Replacement And Nested Policy Ownership End To End Scope: the consequential lifetime cases across the now-active source assembly. @@ -418,7 +523,7 @@ Scope: the consequential lifetime cases across the now-active source assembly. 3. Cover key-only validation of retained records, pending declarations, a no-value replace declaration that cannot authorize key changes, and independent policy whose owning scope survives a record replacement. -4. Correct any integration defects in the Stage 4 ownership transformations; +4. Correct any integration defects in the Stage 5 ownership transformations; consolidate tests with their mechanism owners. Do not introduce special-case ownership by numeric path or a historical identity cache to make cases pass. 5. Verify frozen metadata, origin/resource context, sensitivity of discarded @@ -427,32 +532,51 @@ Scope: the consequential lifetime cases across the now-active source assembly. Validation: focused policy/sequence/layered/ownership tests, `go test -race ./internal/engine`, and the everyday check. -## Stage 12 — Apply Collection Policy During Mapping-Sibling Inheritance +## Stage 17 — Prepare Inheritance Policy And Template Transport -Scope: retain existing mapping-sibling relationships, extend their collection -combination and policy transfer. +Scope: behavior-preserving inheritance state transport, before activating +collection-aware inheritance operations in Stage 18. 1. Extend inheritance templates/expansion sidecars to carry collection identity and record-local policy alongside nodes and origins. Keep existing scheduler keys, cancellation, version protection, missing-parent/cycle diagnostics, and destination-local evaluation of nested templates. -2. Use the shared sequence combiner at the inheriting destination: parent lower, +2. Provide owned-state copy/transfer helpers from Stage 5 and path projections from + Stage 8. Preserve original nested instructions for each destination's later + evaluation rather than retaining only a parent's expanded output. Keep + application registration matching separate from copied YAML policy. +3. Test template reuse at multiple destinations, independent scoped state, and + retained instruction locations/resource context. Preserve current inheritance + values/history while these new helpers remain preparatory; do not activate a + value change without its corresponding policy/evidence handling. + +Validation: existing inheritance, ownership, history, and resource-context tests, +focused transport tests, and the everyday check. + +## Stage 18 — Apply Collection Policy During Mapping-Sibling Inheritance + +Scope: activate shared collection operations using the prepared template/state +handoff; existing mapping-sibling relationship selection remains unchanged. + +1. Use the shared sequence combiner at the inheriting destination: parent lower, child higher, destination explicit mode or identity-based default. Copy YAML identity where applicable, but do not automatically copy a parent collection's explicit mode to a different destination. Match application constraints at the destination, including newly created matching paths. -3. Transfer record-owned state only with the records that survive. Preserve - original nested instructions for each destination's later resolution rather - than copying only a parent's already-expanded output. -4. Extend inheritance evidence using the same projections as source operations. - Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key +2. Transfer record-owned state only with surviving records using Stage 17's + helpers. Integrate inheritance evidence with Stage 8's projections in the same + activation, preserving origins, historical sensitivity, resource context, + cancellation, and existing expansion/history accounting. +3. Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key with inferred upsert, destination explicit mode, incompatible keys, application constraints, cancellation, and policy at an inherited new namespace. +4. Update current mapping-sibling inheritance documentation for collection + operations. Identified-record parent lookup remains incomplete until Stage 21. Validation: inheritance, history, provenance, sensitivity, and resource-context tests plus the everyday check. No identified-record parent lookup is active yet. -## Stage 13 — Build Identified-Record Inheritance Graphs +## Stage 19 — Build Identified-Record Inheritance Graphs Scope: relationship extraction/validation and shared scheduling for sequence scopes; test these helpers before activating payload expansion. @@ -479,9 +603,33 @@ scopes; test these helpers before activating payload expansion. Validation: focused inheritance graph/reference tests and the everyday check. -## Stage 14 — Resolve Record Inheritance With Ownership And Evidence +## Stage 20 — Prepare Reusable Inheritance Scope Execution -Scope: integrate Stage 13 graphs into the existing structural expansion phase. +Scope: remove mapping-container assumptions from the private template executor +without activating identified-record inheritance in public loads. + +1. Refactor template execution to separate relationship scheduling/payload + expansion from reading and rebuilding its scope container. Preserve current + mapping-sibling output, locations, cancellation, evidence, and budgets. +2. Provide private mapping- and identified-sequence-scope adapters using Stage 19's + relationship graphs. The sequence adapter preserves output positions and + exposes child identity fields to the payload-expansion step. Keep this private; + do not add a public generic inheritance or merge callback API. +3. Reuse Stage 17's template/policy transport and Stage 18's collection-aware + payload operations. Existing origin/history/retained-node handling must remain + available to both adapters rather than duplicating an entire executor. +4. Run existing mapping-inheritance regressions and privately exercise a small + sequence graph through the adapter. Check output order, independent templates, + and operation/evidence paths before public activation in Stage 21. + +Validation: inheritance execution, scheduling, history, and budget tests plus +the everyday check. This refactor does not introduce new parent-reference forms +or change public loading semantics yet. + +## Stage 21 — Resolve Record Inheritance With Ownership And Evidence + +Scope: integrate Stage 19's graphs and Stage 20's executor into the structural +expansion phase, with values and inspection working together. 1. Expand parent payload into the child while preserving the child's own identity fields and their origin/presence. Exclude parent identity fields from payload @@ -504,7 +652,7 @@ Scope: integrate Stage 13 graphs into the existing structural expansion phase. Validation: all inheritance and cross-feature inspection tests, `go test -race ./internal/engine`, and the everyday check. -## Stage 15 — Expose The Root-Owned Registration API +## Stage 22 — Expose The Root-Owned Registration API Scope: the settled facade surface and migration of in-repository callers. @@ -520,76 +668,111 @@ Scope: the settled facade surface and migration of in-repository callers. field sets, overlap failure before source access, and independent loads. 4. Prove dynamic constraints apply before strict composition and at inherited destinations, without consumer namespace enumeration. Keep final-result - enforcement described as pending until Stage 18 actually adds it. + enforcement described as pending until Stage 27 actually adds it. 5. Verify the public boundary and declaration documentation. Do not expose engine enums, policy stores, parser nodes, or internal methods. Validation: root sequence/facade, registration, public/dependency-boundary, and consumer example tests plus the everyday check. -## Stage 16 — Prepare Schema-Aware Post-Decode CLI Application +## Stage 23 — Separate Reusable CLI Override Preparation -Scope: a private assignment helper and structural-directive validation, without -switching lifecycle order until Stage 17. +Scope: mechanical preparation refactor and a privately tested post-structural +validation entry, without changing current CLI processing order. 1. Refactor override preparation so labels, paths, snapshots, type support, disjointness, version protection, and existing budgets remain reusable outside the source assembly accumulator. Allocate synthetic sources once in the same order; do not reopen or reinterpret CLI values as YAML. -2. Extend structural rejection to `extends` as well as `configkit` in supplied - paths and nested values, using their established structural roles. This is - post-structural application data, not another instruction channel. Do not add - unrelated name restrictions. -3. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind +2. Prepare post-structural validation that rejects `extends` as well as + `configkit` in supplied paths and nested values, using their established + structural roles. Test this entry privately; retain the existing entry's + behavior until the lifecycle switches in Stage 26. Do not reject previously + valid early-CLI instructions before that switch or add unrelated restrictions. +3. Test detached prepared batches, synthetic source ordering, and the new + directive validation. Reuse existing label/path/cycle/type-support/budget tests + rather than adding another inventory of the same preparation cases. + +Validation: override preparation/source tests and the everyday check. Keep +ordinary/layered behavior unchanged; typed assignment is the next stage. + +## Stage 24 — Implement Schema-Aware Post-Decode CLI Assignment + +Scope: a private typed assignment helper using Stage 23's prepared overrides, +without activating the later lifecycle order yet. + +1. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind only supplied override subtrees against a detached copy of the decoded state. Mappings recursively overlay, sequences replace, and explicit zero/empty/null retain normal schema rules. Current typed values supply retained children, not new defaults for replaced list items. Do not call `Definition.Defaults` again or re-decode the whole projected configuration as newly authored input. -4. Return the updated typed value, canonical post-CLI projection, presence/origin +2. Return the updated typed value, canonical post-CLI projection, presence/origin changes, and existing raw CLI history contributions. Preserve a pre-CLI typed projection for decoding reconciliation/history. No resolver runs on a failed override; no partially applied result is returned. -5. Test unknown/type-invalid fields, mapping retention, sequence replacement, +3. Test unknown/type-invalid fields, mapping retention, sequence replacement, nested maps/lists, named/text/duration/secret schema fields, explicit null/false/ - zero, discarded children, structural controls, and mutable ownership. Reuse - existing label/path/cycle/budget tests without recopying their full inventory. + zero, discarded children, structural controls, and mutable ownership. Validation: override, decode, schema, projection, and ownership tests plus the -everyday check. Only the helpers are added here; lifecycle changes next. +everyday check. Only the helper is added here; the shared lifecycle does not +switch until its inspection/context support is prepared in Stage 25. -## Stage 17 — Move CLI After Decoding And Preserve Inspection Order +## Stage 25 — Prepare Post-Decode CLI Inspection And Effective Context -Scope: activate ADR 0018 consistently through both public loading paths. +Scope: prepare history, resolution-input, and secret-lookup context handling; +retain the existing public CLI lifecycle until Stage 26. -1. Remove CLI assignment from ordinary/layered structural assembly. Invoke - Stage 16 after inheritance and successful typed-default creation/decoding, - immediately before application resolution. Keep the single shared lifecycle; - do not add separate ordinary and layered CLI implementations. -2. Seed resolution with the post-CLI configuration, explicit-presence/origin - metadata, source records, and contexts. Ensure `Resolution.Default` preserves - explicit CLI values while recorded transformations can change them normally. -3. Extend existing replay/reconciliation so evidence follows the real order: +1. Extend existing replay/reconciliation to represent the target order: source tiers, inheritance, typed decoding/defaults, CLI, application resolution. Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and - post-CLI projection distinct where needed. Do not report CLI changes as - defaults, decoding conversions, or inheritance effects. -4. Feed effective post-override/post-resolution reference values and their - locations into secret lookup and final validation. Do not use stale structural - input to determine effective CLI secret references. Preserve no-credential-byte - history and exactly-once default-factory behavior. -5. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an + post-CLI projection distinct where needed. Use the existing history machinery, + not a second replay framework, and retain legacy input compatibility until the + lifecycle switch. Do not mislabel CLI changes as defaults or inheritance. +2. Prepare resolution-entry metadata from Stage 24's result so explicit presence, + origins, source records, and resource contexts describe the overridden state. +3. Prepare the effective reference/location handoff for secret lookup and final + validation using the post-override/post-resolution values, not stale structural + input. Preserve existing no-credential-byte and ownership behavior. +4. Test the target pipeline with privately assembled snapshots: equal-value CLI + replacement, replaced descendants, decoding reconciliation before CLI, later + recorded resolver transformations, explicit-presence defaults, and an overridden + secret reference. Verify current public-load regressions still pass unchanged. + +Validation: history/metadata/resolution/secret tests and the everyday check. +Public activation must not require a later stage to repair these mechanisms. + +## Stage 26 — Activate CLI After Decoding In The Shared Lifecycle + +Scope: activate ADR 0018 through both loading paths using Stages 23–25's complete +assignment, inspection, and effective-value handoffs. + +1. Remove CLI assignment from ordinary/layered structural assembly. Invoke + Stage 24 after inheritance and successful typed-default creation/decoding, + immediately before application resolution. Activate Stage 23's post-structural + validation. Keep the single shared lifecycle, not separate CLI implementations. +2. Wire Stage 25's history and context handoffs in the same change. Resolution + receives explicit-presence/origin metadata and effective CLI values; + `Resolution.Default` preserves them while recorded transformations can change + them normally. Secret lookup uses effective references and locations. Preserve + exactly-once default-factory behavior and synthetic source ordering. +3. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an identified list despite append/upsert mode, derived values seeing CLI input, equal-value provenance, and resolver changes after CLI. Add explicit regressions that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a relationship. Update obsolete early-CLI tests deliberately. -6. Update the lifecycle, override, history, and secret-reference contracts and +4. Update the lifecycle, override, history, and secret-reference contracts and affected examples to describe the now-implemented order. +5. Remove the superseded early-CLI application paths, old preparation entry, and + temporary replay compatibility paths. Keep their still-relevant behavioral + tests under the new lifecycle rather than postponing code cleanup to the + documentation stage. Validation: override/lifecycle/resolution/inheritance/history/secret tests, `go test -race ./...`, and the everyday check. -## Stage 18 — Enforce Final Application Identity Constraints +## Stage 27 — Enforce Final Application Identity Constraints Scope: the final-result guarantee, not another combination or inheritance phase. @@ -617,7 +800,7 @@ Scope: the final-result guarantee, not another combination or inheritance phase. Validation: final identity, selector, facade, defaults/resolution/override, and secret tests plus the everyday check. -## Stage 19 — Exercise Representative Consumer Workflows +## Stage 28 — Exercise Representative Consumer Workflows Scope: a small number of meaningful integrations, not a Cartesian test matrix. @@ -643,9 +826,10 @@ Scope: a small number of meaningful integrations, not a Cartesian test matrix. Validation: `go test ./examples/consumers`, focused root integration tests, `go test -race ./...`, and the everyday check. -## Stage 20 — Reconcile Documentation And Complete The Review +## Stage 29 — Reconcile Documentation -Scope: finish the current-contract narrative and verify the full target state. +Scope: finish the current-contract narrative. Code cleanup belongs with the +integrations that supersede each adapter, not in this documentation prompt. 1. Reconcile `docs/config.md` for declarations, identity equality, selectors, operations/persistence, ownership, inheritance, final validation, and CLI order. @@ -662,7 +846,20 @@ Scope: finish the current-contract narrative and verify the full target state. or implementation references where appropriate. Architecture already adopts the decisions: remove pending-implementation wording when true, without adding new policy. Leave release version selection/publication to the release task. -4. Inspect the completed code for duplicate combination/identity algorithms, +4. Search maintained prose/examples for obsolete registration names and lifecycle + claims; retain intentional historical references. Follow canonical ownership + rather than copying the complete contract into each guide. + +Validation: local documentation-link checks and whitespace review; run relevant +example tests if executable examples changed. Reuse Stage 28's successful +code/race validation when its covered content is unchanged. + +## Stage 30 — Audit Scaling And Verify The Completed Target + +Scope: a bounded implementation review and final validation, not another broad +refactor or expansion of the feature/performance scope. + +1. Inspect the completed code for duplicate combination/identity algorithms, per-record scans of whole collections, repeated declaration gathering, numeric magnitude expansion, stale ownership/path caches, and unnecessary copies. Prefer shared indexed mechanisms without pooling/global caches. Reuse existing @@ -670,22 +867,26 @@ Scope: finish the current-contract narrative and verify the full target state. wildcard-collection benchmark only where existing fixtures cannot exercise the new scaling risk. Record commands and qualitative findings here; timing or allocation thresholds and a quiet-host benchmark campaign are not gates. -5. Remove transitional adapters and dead early-CLI paths. Search current code and - maintained examples for `SequenceCompositions` / `SequenceComposition`; only - intentional historical references may remain. Confirm no new dependencies, - public internal-type exposure, background work, release machinery, or additional - limits were introduced. -6. Run `./scripts/check.sh`, `go test -race ./...`, and the development guide's - local documentation/public-boundary checks, reusing Stage 19's successful code - checks if code is unchanged and rerunning affected documentation checks. Review - the full diff against the roadmap, not merely passing tests. Record completed - scope, validation, and any real remaining issues accurately; do not mark this - plan complete with an unimplemented target behavior. +2. Confirm transitional adapters/dead early-CLI paths have been removed. Search + current code and maintained examples for `SequenceCompositions` and + `SequenceComposition`; only intentional historical references may remain. + Confirm no new dependencies, public internal-type exposure, background work, + release machinery, or additional limits were introduced. Review the full diff + against the roadmap, not merely passing tests. Record bounded review findings + here; significant corrective + work requires its own focused unit rather than being hidden inside this stage. + Do not declare completion while a target behavior remains incorrect or missing. + +Validation: `./scripts/check.sh`, `go test -race ./...`, and the development +guide's local documentation/public-boundary checks, reusing successful checks +from Stages 28–29 if their covered content is unchanged. Record completed scope, +validation, benchmark commands/qualitative findings, and any real remaining +issues accurately. No new timing thresholds or release gates are introduced. Acceptance: all roadmap operations and lifecycle boundaries work through ordinary and layered loads; values and inspection agree; application constraints hold on successful final results; deferred behavior and YAML-only guarantees have not -expanded. All twenty stages are complete and their nonempty units committed. +expanded. All thirty stages are complete and their nonempty units committed. ## Open Questions