eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Split the sequence implementation into smaller stages

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-23 13:40:29 +00:00
parent d06aae5461
commit bc6b977585
1 changed files with 337 additions and 136 deletions
Download Patch File Download Diff File Expand all files Collapse all files

473
docs/roadmap/implementation.md
View File

@@ -35,6 +35,9 @@ and [ADR 0018](../adr/0018-cli-after-structural-resolution.md).
completion. Do not add public compatibility aliases, permanent feature flags, completion. Do not add public compatibility aliases, permanent feature flags,
alternate semantics, new dependencies, new resource caps, or an extension/plugin alternate semantics, new dependencies, new resource caps, or an extension/plugin
framework for this work. framework for this work.
- Preparation stages keep existing public behavior working and test new machinery
privately. Activation stages integrate values, policy, metadata, and relevant
failure handling together; do not defer correct inspection to a later stage.
## Implementation Decisions ## Implementation Decisions
@@ -196,7 +199,9 @@ stage does not export the replacement API or add final-result validation yet.
## Stage 3 — Parse Collection Declarations Without Filesystem Coupling ## Stage 3 — Parse Collection Declarations Without Filesystem Coupling
Scope: private parser and directive extraction helpers, initially unit-tested Scope: private parser and directive extraction helpers, initially unit-tested
without activating collection combination in public loads. without activating collection combination in public loads. Keep source-form
activation out of this stage; ordinary sources and selected overlays are wired
in Stages 13 and 14 respectively.
1. Add `sequence_declaration.go` for source-root `configkit.sequences`. Each entry 1. Add `sequence_declaration.go` for source-root `configkit.sequences`. Each entry
has a required existing literal `destination` form and optional `key` and has a required existing literal `destination` form and optional `key` and
@@ -221,11 +226,12 @@ without activating collection combination in public loads.
Do not add import-level key/mode shorthand or sequence wrappers. Do not add import-level key/mode shorthand or sequence wrappers.
Validation: focused declaration/parser tests, existing import-spec tests, and the Validation: focused declaration/parser tests, existing import-spec tests, and the
everyday check. Public activation occurs with the complete operations in Stage 9. everyday check. Existing import-only behavior remains intact during preparation.
## Stage 4 — Model Scoped Policy Agreement And Lifetime ## Stage 4 — Reconcile Declarations And Select Collection Operations
Scope: private policy resolution/state transformations, independent of loading. Scope: pure agreement and operation-selection helpers, independent of loading
and of the ownership transformations introduced next.
1. Add `collection_policy.go`. Gather the current scope's declarations before 1. Add `collection_policy.go`. Gather the current scope's declarations before
combining peers. Reconcile equivalent keys and agreeing explicit modes; combining peers. Reconcile equivalent keys and agreeing explicit modes;
@@ -234,22 +240,42 @@ Scope: private policy resolution/state transformations, independent of loading.
2. Implement shared operation selection: current explicit mode, retained explicit 2. Implement shared operation selection: current explicit mode, retained explicit
mode, inferred upsert with effective identity, otherwise replace. Persist only mode, inferred upsert with effective identity, otherwise replace. Persist only
explicit modes. Policy-only declarations must not replay earlier combinations. explicit modes. Policy-only declarations must not replay earlier combinations.
3. Implement key agreement, replacement reset, pending key declarations, and 3. Accept eligible lower policy as an input; do not decide record survival or
immediate validation when a key-only declaration identifies retained records. ancestor removal inside declaration reconciliation. Retain declaration
locations/presence separately from inferred mode and application constraints.
4. Test equivalent/different field sets, agreeing/conflicting peer modes,
omission, explicit default modes, and operation inference. Include key-only
declarations agreeing or conflicting with application constraints before a
sequence exists.
Validation: focused policy-agreement/selection tests and the everyday check.
Do not activate collection combination or implement the ownership layer here.
## Stage 5 — Track Policy Lifetime And Record Ownership
Scope: private scoped-state transformations built on Stage 4's resolution helpers.
1. Implement key agreement across retained values, replacement reset, pending
key declarations, and immediate validation when a key-only declaration
identifies retained records.
Changing mode to replace without a replacement value does not reset identity. Changing mode to replace without a replacement value does not reset identity.
4. Implement scope/record ownership transfer and retirement hooks. Actual ancestor 2. Implement scope/record ownership transfer and retirement hooks. Actual ancestor
replacement removes descendant collection identity and removed-record policy; replacement removes descendant collection identity and removed-record policy;
independently owned modes and application constraints survive as specified in independently owned modes and application constraints survive as specified in
the roadmap. Do not repeatedly reapply old declarations and resurrect reset the roadmap. Do not repeatedly reapply old declarations and resurrect reset
identity. Pending declarations retain their owning-scope rules. identity. Pending declarations retain their owning-scope rules.
5. Use table-driven tests derived from the roadmap's agreement, persistence, and 3. Use item/acquisition scope anchors and record-owned relative policy as selected
above. Provide transformations for retention, whole-record replacement,
whole-sequence replacement, and inheritance copies without wiring them into
public loading yet. Application registrations remain destination constraints.
4. Use table-driven tests derived from the roadmap's lifetime, persistence, and
replacement tables, including absent/non-sequence destinations, equal outer replacement tables, including absent/non-sequence destinations, equal outer
identities with distinct record owners, and independent root versus item policy. identities with distinct record owners, and independent root versus item policy.
Validation: focused collection-policy tests and the everyday check. This stage Validation: focused collection-policy tests and the everyday check. This stage
must not add path-existence, shape, nullability, or global YAML final-check rules. must not add path-existence, shape, nullability, or global YAML final-check rules.
## Stage 5 — Implement The Shared Sequence Operations ## Stage 6 — Implement The Shared Sequence Operations
Scope: a pure document-level combination helper, not separate implementations Scope: a pure document-level combination helper, not separate implementations
for overlays and inheritance. for overlays and inheritance.
@@ -271,112 +297,189 @@ for overlays and inheritance.
Validation: focused combination tests and the everyday check. Do not change the Validation: focused combination tests and the everyday check. Do not change the
ordinary recursive mapping rule or add unkeyed append. ordinary recursive mapping rule or add unkeyed append.
## Stage 6 — Extend Sequence Evidence And Historical Replay ## Stage 7 — Extend Sequence History Representation And Replay
Scope: teach existing provenance/history projection machinery to represent the Scope: represent and replay sequence operations without yet changing all metadata
operations before wiring them into public precedence. projection consumers or public precedence.
1. Extend `sequenceHistoryContribution`, sequence replay in `history.go`, and 1. Extend `sequenceHistoryContribution`, sequence replay in `history.go`, and
projection helpers to consume Stage 5's explicit item mappings. Existing append its contribution indexes to consume Stage 6's explicit item mappings. Existing
ranges may remain a compact case; upsert cannot be represented by one offset. append ranges may remain a compact case; upsert cannot be represented by one
offset.
2. Preserve authored raw contribution snapshots and displaced values. A replaced 2. Preserve authored raw contribution snapshots and displaced values. A replaced
record's omitted fields must appear as removed, not as retained lower payload. record's omitted fields must appear as removed, not as retained lower payload.
Preserve equal-valued replacement events. Keep existing index-based query and Preserve equal-valued replacement events. Keep existing index-based query and
atomic sequence-diff APIs; add no public identity-selector path syntax. atomic sequence-diff APIs; add no public identity-selector path syntax.
3. Project composition origins, nested source destinations, historical paths, and 3. Exercise synthetic append/upsert/replace histories, noncontiguous matches,
sensitivity through the same item mapping. Physical source IDs/locations and nested paths, and discarded descendants. Compare replayed final values with
resource contexts continue to identify the declaring source. the actual combination result, including first/last and unchanged neighboring
4. Exercise synthetic append/upsert/replace histories, noncontiguous matches, records. Reuse existing escaped-path and detached-query tests rather than
nested paths, discarded descendants, and secret-marked replaced records. duplicating them.
Compare replayed final values with the actual combination result, including
first/last and unchanged neighboring records. Reuse existing escaped-path and Validation: sequence/history/replay tests and the everyday check. Keep current
detached-query tests rather than duplicating them. append/replace callers working through the extended descriptors.
## Stage 8 — Project Sequence Origins And Inspection Evidence
Scope: use the prepared item mappings consistently in metadata consumers.
1. Project composition origins, nested source destinations, historical paths, and
sensitivity through Stage 6's item mapping and Stage 7's history descriptors.
Physical source IDs/locations and resource contexts continue to identify the
declaring source. Reuse one projection mechanism rather than searching again
by identity or equal payload.
2. Test synthetic noncontiguous upserts, replaced descendants, nested source
destinations, unchanged neighbors, and secret-marked replaced records. Verify
projected queries agree with replayed and actual final values, retaining
displaced evidence and detached results.
3. Keep the new precedence operations unactivated until their value, policy, and
projection machinery are integrated in the later load-path stages.
Validation: metadata/history/sequence/provenance/sensitivity tests, Validation: metadata/history/sequence/provenance/sensitivity tests,
`go test -race ./internal/engine`, and the everyday check. `go test -race ./internal/engine`, and the everyday check.
## Stage 7 — Separate Acquisition From Policy-Aware Tier Composition ## Stage 9 — Separate Per-Input Acquisition From Composition
Scope: refactor the assembly handoff, initially preserving existing behavior. Scope: a behavior-preserving acquired-input handoff; grouped underlay coordination
is the next stage.
1. In `load_input.go`, `imports.go`, `load_request.go`, and related graph helpers, 1. In `load_input.go`, `imports.go`, and related graph helpers, retain acquisition
retain acquisition scopes, raw contributions, inclusion edges, source records, scopes, raw contributions, inclusion edges, source records,
contexts, and extracted declarations without irrevocably composing them. contexts, and extracted declarations without irrevocably composing them.
Reuse existing `deferAssembly` and acquired-scope mechanisms instead of adding Reuse existing `deferAssembly` and acquired-scope mechanisms instead of adding
a second loader. Keep already acquired document handles closed as before. a second loader. Keep already acquired document handles closed as before.
2. Separate acquiring an underlay group from composing its scopes. Keep authored 2. Separate acquisition from the per-input composition entry. Retain pending
acquisition order: main and depth-first imports, each underlay/input, overlay, item/helper scope boundaries, namespace identities, inclusion/repetition checks,
then synthetic CLI records. Retain repeated-source, namespace, traversal-cycle, source records, accounting, and scheduler inputs. Do not reacquire sources to
accounting, and stable scheduler behavior. Do not reacquire sources to obtain compose them later or change grouped coordination yet.
lower policy. 3. Preserve existing version checks for this refactor. Retain a small adapter
3. Expose one internal composition entry that accepts accumulated lower policy and invoking current composition semantics at the existing lifecycle point.
compiled application constraints. Complete item-local helper scopes before
item placement; gather declarations before composing each relevant scope.
4. Preserve existing version checks for this refactor. Retain a small adapter
invoking the current composition semantics until the next integrations.
Preserve source-version callbacks at acquisition, including header consumption. Preserve source-version callbacks at acquisition, including header consumption.
5. Run existing acquisition, dependency scheduling, item placement, source-version, 4. Run existing acquisition, dependency scheduling, item placement, source-version,
ordinary/layered equivalence, resource-lifetime, and source-context tests. Add ordinary/layered equivalence, resource-lifetime, and source-context tests. Add
only targeted regressions for any newly split handoff not already covered. only targeted regressions for any newly split handoff not already covered.
Validation: the focused suites above and the everyday check. This refactor must Validation: the focused suites above and the everyday check. This refactor must
not change source IDs or turn acquisition order into precedence order. not change source IDs or turn acquisition order into precedence order.
## Stage 8 — Compose Strict Scopes With Resolved Collection Identity ## Stage 10 — Separate Underlay-Group Acquisition From Composition
Scope: implement the policy-aware strict composition entry, including nested Scope: behavior-preserving group coordination in `load_request.go`, using the
item scopes, before selecting it in all load paths. per-input handoff already prepared in Stage 9.
1. Separate acquiring an underlay group from scheduling/composing its acquired
scopes. Preserve namespace reanchoring, shared inclusion/repetition state,
item-local helper scopes, source/context aggregation, and group accounting.
2. Keep authored acquisition order: main and depth-first imports, each underlay
and input, overlay, then synthetic CLI records. Preserve underlay indexes and
current version-check timing in this refactor; policy-dependent layered
composition is activated only in Stage 15.
3. Expose the common scope-composition handoff with inputs for accumulated lower
policy and compiled application constraints. Initially adapt it to the current
semantics. Do not add another loader or repeat acquisition/version callbacks.
4. Test the split with existing grouped/dependency/version/destination coverage,
including empty inputs/groups and repeated sources within versus across tiers.
Confirm unchanged values, source ordering, contexts, and failure evidence.
Validation: grouped/layered, import scheduling, version, destination, and
resource-context tests plus the everyday check.
## Stage 11 — Compose Strict Collections With Resolved Identity
Scope: the policy-aware strict composition core. Nested pending-item scope
integration is deferred to Stage 12, not absorbed into this prompt.
1. Extend `composeDocumentsWithPolicy` and `validateRegisteredSequences` to use 1. Extend `composeDocumentsWithPolicy` and `validateRegisteredSequences` to use
normalized field sets and scalar tuples. Resolve current declarations, normalized field sets and scalar tuples. Resolve current declarations,
applicable application constraints, and eligible lower identity before peer applicable application constraints, and eligible lower identity before peer
composition, using the Stage 4 helpers. composition, using the Stage 4 agreement and Stage 5 lifetime helpers.
2. Use the existing dependency-ready order for contributions and authored list 2. Use the existing dependency-ready order for contributions and authored list
order. Validate every identified sequence, including singleton contributions; order. Validate every identified sequence, including singleton contributions;
strictly append distinct peers and reject duplicates under all modes. strictly append distinct peers and reject duplicates under all modes.
3. Complete helper imports and consume applicable non-main version headers before 3. For explicit replacement, do not use obsolete lower YAML identity to authorize
checking record payload identity. Retained structural directives cannot serve otherwise unkeyed peer composition. Still apply independently registered
as identity fields. Do not validate an incomplete helper-built compound parent application constraints and genuinely applicable declarations.
reference here; validate that completed reference during inheritance. 4. Attach Stages 7–8 evidence and Stage 5 scoped policy to composed results. Test
4. For explicit replacement, do not use obsolete lower YAML identity to authorize completed-document contributions with lower-established identity, declaration
otherwise unkeyed peer composition. Likewise, a new outer record cannot borrow conflicts, compound/scalar IDs, singleton validation, and replacement resets.
nested identity from the displaced lower record. Still apply independently
registered application constraints and genuinely applicable declarations.
5. Attach Stage 6 evidence and Stage 4 scoped policy to composed results. Test
lower-established identity, peer declaration conflicts, compound/scalar IDs,
helper-supplied fields, consumed headers, and replaced record-local ownership.
Validation: composition, sequence identity, destination, import, and version Validation: composition and sequence identity tests plus the everyday check.
tests plus the everyday check. Keep unregistered non-mapping conflicts unchanged. Keep unregistered non-mapping conflicts unchanged. Public load activation remains
in the later integration stages.
## Stage 9 — Activate Ordinary Loading And Selected Overlay Policies ## Stage 12 — Integrate Policy-Aware Item And Helper Scopes
Scope: ordinary `Load`, source declarations, and its selected overlay. Scope: adapt pending-item/helper composition to Stage 11's core using the acquired
scope boundaries from Stages 9–10.
1. Gather declarations before composing each relevant scope, complete item-local
helper imports before item placement, and consume applicable non-main version
headers before checking record payload identity. Keep relative policy with its
owner until concrete placement is known.
2. Apply eligible lower identity at each scope, but prevent a replacement outer
record from borrowing nested identity owned by the discarded lower record.
Preserve independently applicable policy and application constraints.
3. Retained structural directives cannot serve as identity fields. A helper may
supply payload identity components. Do not require a helper-built compound
parent reference to be complete at this point; validate the completed surviving
reference during inheritance.
4. Project item evidence and owned policies through Stages 7–8 machinery. Test
nested items, helper-supplied identity, header consumption, declaration scope,
discarded owner identity, and exact diagnostic paths with no value disclosure.
Validation: item/destination/import/composition/source-version tests and the
everyday check. Keep current public behavior until load-path activation.
## Stage 13 — Activate Ordinary Source Collection Declarations
Scope: ordinary `Load` base-source assembly, excluding activation of the new
selected-overlay policy surface until Stage 14.
1. Wire common directive extraction into every ordinary source form: file, 1. Wire common directive extraction into every ordinary source form: file,
native filesystem, bytes, reader, mapping, imported mapping, and sequence item. native filesystem, bytes, reader, mapping, imported mapping, and sequence item.
Declarations work without `Imports`; disabled imports retain their existing Declarations work without `Imports`; disabled imports retain their existing
rejection semantics. `NoSource` remains absence rather than an empty document. rejection semantics. `NoSource` remains absence rather than an empty document.
2. Use Stage 8 composition, followed by Stage 5 precedence operations in the 2. Use Stages 11–12 composition and Stage 5 scoped policy. Strip collection
assembly accumulator. Reuse the Stage 4 policy resolver; mappings still recurse controls from application decoding, but retain their locations/evidence for
and unrelated sequences still replace. Preserve mode-only/key-only effects. their structural lifetime and diagnostics.
3. Permit `configkit.sequences` in selected overlays while continuing to reject 3. Integrate policy/projections together with the values. Cover inline/imported
overlay imports. Strip structural controls from application decoding, but equivalence, placement-relative declarations, singleton and peer validation,
retain their locations/evidence for their structural lifetime and diagnostics. and declarations without accompanying values. Maintain ordinary loading's
4. Apply projection and policy-retirement information as part of each operation, existing non-collection overlay behavior while preparing its policy handoff.
not in a later best-effort repair pass. Cover inline/imported equivalence, 4. Update the implemented declaration/composition contract sections. Collection
placement-relative declarations, append/upsert/replace, empty lists, retained precedence, inheritance, and later CLI behavior remain incomplete until their
modes, and declarations without accompanying values. integration stages; do not claim them implemented here.
5. Update current sequence/overlay contract sections and tests affected by this
activation. CLI and inheritance retain their explicitly incomplete milestones
until their later stages; do not claim those parts of the roadmap implemented.
Validation: ordinary-load, overlay, import, composition, and metadata tests plus Validation: ordinary-load, import, composition, and metadata tests plus
the everyday check. Include one public ordinary-load declaration example test. the everyday check. Include one public ordinary-load declaration example test.
## Stage 10 — Integrate Lower Policy Across Layered Assembly ## Stage 14 — Activate Selected Overlay Collection Policies
Scope: source-backed underlays and the layered selected-overlay path. Scope: ordinary loading's selected overlay and shared precedence integration,
using the source/declaration handoff already active in Stage 13.
1. Permit `configkit.sequences` in selected overlays while continuing to reject
overlay imports. Preserve source-version header/control handling and ordinary
overlay namespace selection. Consume controls before application decoding.
2. Use Stage 6's sequence operations and Stages 4–5 policy helpers in the assembly
accumulator. Mappings still recurse and unrelated sequences still replace.
Wire value, policy retirement/transfer, and Stages 7–8 inspection projections
together, not as a later best-effort repair pass.
3. Cover append/upsert/replace, empty/null/kind cases, key-only/mode-only changes,
retained explicit modes, overlay identity agreement, and independently owned
policies. Check values, history, origins, and discarded descendants together.
4. Update the selected-overlay and collection-operation contract sections. Make
the shared precedence entry reusable by layered loading without adding another
operation implementation or turning overlay imports on.
Validation: ordinary-load, overlay, composition, policy, and metadata tests plus
the everyday check.
## Stage 15 — Integrate Lower Policy Across Layered Assembly
Scope: source-backed underlay coordination and the layered selected-overlay path,
using the prepared acquisition, ownership, strict-scope, and precedence machinery.
Do not absorb unfinished work from Stages 5–14 into this stage.
1. Acquire main first, but assemble underlays from lowest to highest, strictly 1. Acquire main first, but assemble underlays from lowest to highest, strictly
composing each group with applicable accumulated lower policy before applying composing each group with applicable accumulated lower policy before applying
@@ -396,6 +499,8 @@ Scope: source-backed underlays and the layered selected-overlay path.
4. Keep histories in precedence order, sources in acquisition order, underlay 4. Keep histories in precedence order, sources in acquisition order, underlay
indexes stable across empty groups, and shared budgets intact. Update the indexes stable across empty groups, and shared budgets intact. Update the
layered contract's acquisition-versus-composition description accordingly. layered contract's acquisition-versus-composition description accordingly.
Remove the acquisition/composition adapters superseded by this integration;
the old public-registration adapter remains until Stage 22.
5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main, 5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main,
explicit empty groups, layered overlays, version protection and diagnostics, explicit empty groups, layered overlays, version protection and diagnostics,
and equivalence with ordinary loading where the inputs express the same model. and equivalence with ordinary loading where the inputs express the same model.
@@ -403,7 +508,7 @@ Scope: source-backed underlays and the layered selected-overlay path.
Validation: layered/grouped, source-version, dependency scheduling, destination, Validation: layered/grouped, source-version, dependency scheduling, destination,
and metadata tests plus the everyday check. and metadata tests plus the everyday check.
## Stage 11 — Verify Replacement And Nested Policy Ownership End To End ## Stage 16 — Verify Replacement And Nested Policy Ownership End To End
Scope: the consequential lifetime cases across the now-active source assembly. Scope: the consequential lifetime cases across the now-active source assembly.
@@ -418,7 +523,7 @@ Scope: the consequential lifetime cases across the now-active source assembly.
3. Cover key-only validation of retained records, pending declarations, a no-value 3. Cover key-only validation of retained records, pending declarations, a no-value
replace declaration that cannot authorize key changes, and independent policy replace declaration that cannot authorize key changes, and independent policy
whose owning scope survives a record replacement. whose owning scope survives a record replacement.
4. Correct any integration defects in the Stage 4 ownership transformations; 4. Correct any integration defects in the Stage 5 ownership transformations;
consolidate tests with their mechanism owners. Do not introduce special-case consolidate tests with their mechanism owners. Do not introduce special-case
ownership by numeric path or a historical identity cache to make cases pass. ownership by numeric path or a historical identity cache to make cases pass.
5. Verify frozen metadata, origin/resource context, sensitivity of discarded 5. Verify frozen metadata, origin/resource context, sensitivity of discarded
@@ -427,32 +532,51 @@ Scope: the consequential lifetime cases across the now-active source assembly.
Validation: focused policy/sequence/layered/ownership tests, Validation: focused policy/sequence/layered/ownership tests,
`go test -race ./internal/engine`, and the everyday check. `go test -race ./internal/engine`, and the everyday check.
## Stage 12 — Apply Collection Policy During Mapping-Sibling Inheritance ## Stage 17 — Prepare Inheritance Policy And Template Transport
Scope: retain existing mapping-sibling relationships, extend their collection Scope: behavior-preserving inheritance state transport, before activating
combination and policy transfer. collection-aware inheritance operations in Stage 18.
1. Extend inheritance templates/expansion sidecars to carry collection identity 1. Extend inheritance templates/expansion sidecars to carry collection identity
and record-local policy alongside nodes and origins. Keep existing scheduler and record-local policy alongside nodes and origins. Keep existing scheduler
keys, cancellation, version protection, missing-parent/cycle diagnostics, and keys, cancellation, version protection, missing-parent/cycle diagnostics, and
destination-local evaluation of nested templates. destination-local evaluation of nested templates.
2. Use the shared sequence combiner at the inheriting destination: parent lower, 2. Provide owned-state copy/transfer helpers from Stage 5 and path projections from
Stage 8. Preserve original nested instructions for each destination's later
evaluation rather than retaining only a parent's expanded output. Keep
application registration matching separate from copied YAML policy.
3. Test template reuse at multiple destinations, independent scoped state, and
retained instruction locations/resource context. Preserve current inheritance
values/history while these new helpers remain preparatory; do not activate a
value change without its corresponding policy/evidence handling.
Validation: existing inheritance, ownership, history, and resource-context tests,
focused transport tests, and the everyday check.
## Stage 18 — Apply Collection Policy During Mapping-Sibling Inheritance
Scope: activate shared collection operations using the prepared template/state
handoff; existing mapping-sibling relationship selection remains unchanged.
1. Use the shared sequence combiner at the inheriting destination: parent lower,
child higher, destination explicit mode or identity-based default. Copy YAML child higher, destination explicit mode or identity-based default. Copy YAML
identity where applicable, but do not automatically copy a parent collection's identity where applicable, but do not automatically copy a parent collection's
explicit mode to a different destination. Match application constraints at the explicit mode to a different destination. Match application constraints at the
destination, including newly created matching paths. destination, including newly created matching paths.
3. Transfer record-owned state only with the records that survive. Preserve 2. Transfer record-owned state only with surviving records using Stage 17's
original nested instructions for each destination's later resolution rather helpers. Integrate inheritance evidence with Stage 8's projections in the same
than copying only a parent's already-expanded output. activation, preserving origins, historical sensitivity, resource context,
4. Extend inheritance evidence using the same projections as source operations. cancellation, and existing expansion/history accounting.
Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key 3. Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key
with inferred upsert, destination explicit mode, incompatible keys, application with inferred upsert, destination explicit mode, incompatible keys, application
constraints, cancellation, and policy at an inherited new namespace. constraints, cancellation, and policy at an inherited new namespace.
4. Update current mapping-sibling inheritance documentation for collection
operations. Identified-record parent lookup remains incomplete until Stage 21.
Validation: inheritance, history, provenance, sensitivity, and resource-context Validation: inheritance, history, provenance, sensitivity, and resource-context
tests plus the everyday check. No identified-record parent lookup is active yet. tests plus the everyday check. No identified-record parent lookup is active yet.
## Stage 13 — Build Identified-Record Inheritance Graphs ## Stage 19 — Build Identified-Record Inheritance Graphs
Scope: relationship extraction/validation and shared scheduling for sequence Scope: relationship extraction/validation and shared scheduling for sequence
scopes; test these helpers before activating payload expansion. scopes; test these helpers before activating payload expansion.
@@ -479,9 +603,33 @@ scopes; test these helpers before activating payload expansion.
Validation: focused inheritance graph/reference tests and the everyday check. Validation: focused inheritance graph/reference tests and the everyday check.
## Stage 14 — Resolve Record Inheritance With Ownership And Evidence ## Stage 20 — Prepare Reusable Inheritance Scope Execution
Scope: integrate Stage 13 graphs into the existing structural expansion phase. Scope: remove mapping-container assumptions from the private template executor
without activating identified-record inheritance in public loads.
1. Refactor template execution to separate relationship scheduling/payload
expansion from reading and rebuilding its scope container. Preserve current
mapping-sibling output, locations, cancellation, evidence, and budgets.
2. Provide private mapping- and identified-sequence-scope adapters using Stage 19's
relationship graphs. The sequence adapter preserves output positions and
exposes child identity fields to the payload-expansion step. Keep this private;
do not add a public generic inheritance or merge callback API.
3. Reuse Stage 17's template/policy transport and Stage 18's collection-aware
payload operations. Existing origin/history/retained-node handling must remain
available to both adapters rather than duplicating an entire executor.
4. Run existing mapping-inheritance regressions and privately exercise a small
sequence graph through the adapter. Check output order, independent templates,
and operation/evidence paths before public activation in Stage 21.
Validation: inheritance execution, scheduling, history, and budget tests plus
the everyday check. This refactor does not introduce new parent-reference forms
or change public loading semantics yet.
## Stage 21 — Resolve Record Inheritance With Ownership And Evidence
Scope: integrate Stage 19's graphs and Stage 20's executor into the structural
expansion phase, with values and inspection working together.
1. Expand parent payload into the child while preserving the child's own identity 1. Expand parent payload into the child while preserving the child's own identity
fields and their origin/presence. Exclude parent identity fields from payload fields and their origin/presence. Exclude parent identity fields from payload
@@ -504,7 +652,7 @@ Scope: integrate Stage 13 graphs into the existing structural expansion phase.
Validation: all inheritance and cross-feature inspection tests, Validation: all inheritance and cross-feature inspection tests,
`go test -race ./internal/engine`, and the everyday check. `go test -race ./internal/engine`, and the everyday check.
## Stage 15 — Expose The Root-Owned Registration API ## Stage 22 — Expose The Root-Owned Registration API
Scope: the settled facade surface and migration of in-repository callers. Scope: the settled facade surface and migration of in-repository callers.
@@ -520,76 +668,111 @@ Scope: the settled facade surface and migration of in-repository callers.
field sets, overlap failure before source access, and independent loads. field sets, overlap failure before source access, and independent loads.
4. Prove dynamic constraints apply before strict composition and at inherited 4. Prove dynamic constraints apply before strict composition and at inherited
destinations, without consumer namespace enumeration. Keep final-result destinations, without consumer namespace enumeration. Keep final-result
enforcement described as pending until Stage 18 actually adds it. enforcement described as pending until Stage 27 actually adds it.
5. Verify the public boundary and declaration documentation. Do not expose engine 5. Verify the public boundary and declaration documentation. Do not expose engine
enums, policy stores, parser nodes, or internal methods. enums, policy stores, parser nodes, or internal methods.
Validation: root sequence/facade, registration, public/dependency-boundary, and Validation: root sequence/facade, registration, public/dependency-boundary, and
consumer example tests plus the everyday check. consumer example tests plus the everyday check.
## Stage 16 — Prepare Schema-Aware Post-Decode CLI Application ## Stage 23 — Separate Reusable CLI Override Preparation
Scope: a private assignment helper and structural-directive validation, without Scope: mechanical preparation refactor and a privately tested post-structural
switching lifecycle order until Stage 17. validation entry, without changing current CLI processing order.
1. Refactor override preparation so labels, paths, snapshots, type support, 1. Refactor override preparation so labels, paths, snapshots, type support,
disjointness, version protection, and existing budgets remain reusable outside disjointness, version protection, and existing budgets remain reusable outside
the source assembly accumulator. Allocate synthetic sources once in the same the source assembly accumulator. Allocate synthetic sources once in the same
order; do not reopen or reinterpret CLI values as YAML. order; do not reopen or reinterpret CLI values as YAML.
2. Extend structural rejection to `extends` as well as `configkit` in supplied 2. Prepare post-structural validation that rejects `extends` as well as
paths and nested values, using their established structural roles. This is `configkit` in supplied paths and nested values, using their established
post-structural application data, not another instruction channel. Do not add structural roles. Test this entry privately; retain the existing entry's
unrelated name restrictions. behavior until the lifecycle switches in Stage 26. Do not reject previously
3. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind valid early-CLI instructions before that switch or add unrelated restrictions.
3. Test detached prepared batches, synthetic source ordering, and the new
directive validation. Reuse existing label/path/cycle/type-support/budget tests
rather than adding another inventory of the same preparation cases.
Validation: override preparation/source tests and the everyday check. Keep
ordinary/layered behavior unchanged; typed assignment is the next stage.
## Stage 24 — Implement Schema-Aware Post-Decode CLI Assignment
Scope: a private typed assignment helper using Stage 23's prepared overrides,
without activating the later lifecycle order yet.
1. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind
only supplied override subtrees against a detached copy of the decoded state. only supplied override subtrees against a detached copy of the decoded state.
Mappings recursively overlay, sequences replace, and explicit zero/empty/null Mappings recursively overlay, sequences replace, and explicit zero/empty/null
retain normal schema rules. Current typed values supply retained children, retain normal schema rules. Current typed values supply retained children,
not new defaults for replaced list items. Do not call `Definition.Defaults` not new defaults for replaced list items. Do not call `Definition.Defaults`
again or re-decode the whole projected configuration as newly authored input. again or re-decode the whole projected configuration as newly authored input.
4. Return the updated typed value, canonical post-CLI projection, presence/origin 2. Return the updated typed value, canonical post-CLI projection, presence/origin
changes, and existing raw CLI history contributions. Preserve a pre-CLI typed changes, and existing raw CLI history contributions. Preserve a pre-CLI typed
projection for decoding reconciliation/history. No resolver runs on a failed projection for decoding reconciliation/history. No resolver runs on a failed
override; no partially applied result is returned. override; no partially applied result is returned.
5. Test unknown/type-invalid fields, mapping retention, sequence replacement, 3. Test unknown/type-invalid fields, mapping retention, sequence replacement,
nested maps/lists, named/text/duration/secret schema fields, explicit null/false/ nested maps/lists, named/text/duration/secret schema fields, explicit null/false/
zero, discarded children, structural controls, and mutable ownership. Reuse zero, discarded children, structural controls, and mutable ownership.
existing label/path/cycle/budget tests without recopying their full inventory.
Validation: override, decode, schema, projection, and ownership tests plus the Validation: override, decode, schema, projection, and ownership tests plus the
everyday check. Only the helpers are added here; lifecycle changes next. everyday check. Only the helper is added here; the shared lifecycle does not
switch until its inspection/context support is prepared in Stage 25.
## Stage 17 — Move CLI After Decoding And Preserve Inspection Order ## Stage 25 — Prepare Post-Decode CLI Inspection And Effective Context
Scope: activate ADR 0018 consistently through both public loading paths. Scope: prepare history, resolution-input, and secret-lookup context handling;
retain the existing public CLI lifecycle until Stage 26.
1. Remove CLI assignment from ordinary/layered structural assembly. Invoke 1. Extend existing replay/reconciliation to represent the target order:
Stage 16 after inheritance and successful typed-default creation/decoding,
immediately before application resolution. Keep the single shared lifecycle;
do not add separate ordinary and layered CLI implementations.
2. Seed resolution with the post-CLI configuration, explicit-presence/origin
metadata, source records, and contexts. Ensure `Resolution.Default` preserves
explicit CLI values while recorded transformations can change them normally.
3. Extend existing replay/reconciliation so evidence follows the real order:
source tiers, inheritance, typed decoding/defaults, CLI, application resolution. source tiers, inheritance, typed decoding/defaults, CLI, application resolution.
Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and
post-CLI projection distinct where needed. Do not report CLI changes as post-CLI projection distinct where needed. Use the existing history machinery,
defaults, decoding conversions, or inheritance effects. not a second replay framework, and retain legacy input compatibility until the
4. Feed effective post-override/post-resolution reference values and their lifecycle switch. Do not mislabel CLI changes as defaults or inheritance.
locations into secret lookup and final validation. Do not use stale structural 2. Prepare resolution-entry metadata from Stage 24's result so explicit presence,
input to determine effective CLI secret references. Preserve no-credential-byte origins, source records, and resource contexts describe the overridden state.
history and exactly-once default-factory behavior. 3. Prepare the effective reference/location handoff for secret lookup and final
5. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an validation using the post-override/post-resolution values, not stale structural
input. Preserve existing no-credential-byte and ownership behavior.
4. Test the target pipeline with privately assembled snapshots: equal-value CLI
replacement, replaced descendants, decoding reconciliation before CLI, later
recorded resolver transformations, explicit-presence defaults, and an overridden
secret reference. Verify current public-load regressions still pass unchanged.
Validation: history/metadata/resolution/secret tests and the everyday check.
Public activation must not require a later stage to repair these mechanisms.
## Stage 26 — Activate CLI After Decoding In The Shared Lifecycle
Scope: activate ADR 0018 through both loading paths using Stages 23–25's complete
assignment, inspection, and effective-value handoffs.
1. Remove CLI assignment from ordinary/layered structural assembly. Invoke
Stage 24 after inheritance and successful typed-default creation/decoding,
immediately before application resolution. Activate Stage 23's post-structural
validation. Keep the single shared lifecycle, not separate CLI implementations.
2. Wire Stage 25's history and context handoffs in the same change. Resolution
receives explicit-presence/origin metadata and effective CLI values;
`Resolution.Default` preserves them while recorded transformations can change
them normally. Secret lookup uses effective references and locations. Preserve
exactly-once default-factory behavior and synthetic source ordering.
3. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an
identified list despite append/upsert mode, derived values seeing CLI input, identified list despite append/upsert mode, derived values seeing CLI input,
equal-value provenance, and resolver changes after CLI. Add explicit regressions equal-value provenance, and resolver changes after CLI. Add explicit regressions
that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a
relationship. Update obsolete early-CLI tests deliberately. relationship. Update obsolete early-CLI tests deliberately.
6. Update the lifecycle, override, history, and secret-reference contracts and 4. Update the lifecycle, override, history, and secret-reference contracts and
affected examples to describe the now-implemented order. affected examples to describe the now-implemented order.
5. Remove the superseded early-CLI application paths, old preparation entry, and
temporary replay compatibility paths. Keep their still-relevant behavioral
tests under the new lifecycle rather than postponing code cleanup to the
documentation stage.
Validation: override/lifecycle/resolution/inheritance/history/secret tests, Validation: override/lifecycle/resolution/inheritance/history/secret tests,
`go test -race ./...`, and the everyday check. `go test -race ./...`, and the everyday check.
## Stage 18 — Enforce Final Application Identity Constraints ## Stage 27 — Enforce Final Application Identity Constraints
Scope: the final-result guarantee, not another combination or inheritance phase. Scope: the final-result guarantee, not another combination or inheritance phase.
@@ -617,7 +800,7 @@ Scope: the final-result guarantee, not another combination or inheritance phase.
Validation: final identity, selector, facade, defaults/resolution/override, and Validation: final identity, selector, facade, defaults/resolution/override, and
secret tests plus the everyday check. secret tests plus the everyday check.
## Stage 19 — Exercise Representative Consumer Workflows ## Stage 28 — Exercise Representative Consumer Workflows
Scope: a small number of meaningful integrations, not a Cartesian test matrix. Scope: a small number of meaningful integrations, not a Cartesian test matrix.
@@ -643,9 +826,10 @@ Scope: a small number of meaningful integrations, not a Cartesian test matrix.
Validation: `go test ./examples/consumers`, focused root integration tests, Validation: `go test ./examples/consumers`, focused root integration tests,
`go test -race ./...`, and the everyday check. `go test -race ./...`, and the everyday check.
## Stage 20 — Reconcile Documentation And Complete The Review ## Stage 29 — Reconcile Documentation
Scope: finish the current-contract narrative and verify the full target state. Scope: finish the current-contract narrative. Code cleanup belongs with the
integrations that supersede each adapter, not in this documentation prompt.
1. Reconcile `docs/config.md` for declarations, identity equality, selectors, 1. Reconcile `docs/config.md` for declarations, identity equality, selectors,
operations/persistence, ownership, inheritance, final validation, and CLI order. operations/persistence, ownership, inheritance, final validation, and CLI order.
@@ -662,7 +846,20 @@ Scope: finish the current-contract narrative and verify the full target state.
or implementation references where appropriate. Architecture already adopts or implementation references where appropriate. Architecture already adopts
the decisions: remove pending-implementation wording when true, without adding the decisions: remove pending-implementation wording when true, without adding
new policy. Leave release version selection/publication to the release task. new policy. Leave release version selection/publication to the release task.
4. Inspect the completed code for duplicate combination/identity algorithms, 4. Search maintained prose/examples for obsolete registration names and lifecycle
claims; retain intentional historical references. Follow canonical ownership
rather than copying the complete contract into each guide.
Validation: local documentation-link checks and whitespace review; run relevant
example tests if executable examples changed. Reuse Stage 28's successful
code/race validation when its covered content is unchanged.
## Stage 30 — Audit Scaling And Verify The Completed Target
Scope: a bounded implementation review and final validation, not another broad
refactor or expansion of the feature/performance scope.
1. Inspect the completed code for duplicate combination/identity algorithms,
per-record scans of whole collections, repeated declaration gathering, numeric per-record scans of whole collections, repeated declaration gathering, numeric
magnitude expansion, stale ownership/path caches, and unnecessary copies. magnitude expansion, stale ownership/path caches, and unnecessary copies.
Prefer shared indexed mechanisms without pooling/global caches. Reuse existing Prefer shared indexed mechanisms without pooling/global caches. Reuse existing
@@ -670,22 +867,26 @@ Scope: finish the current-contract narrative and verify the full target state.
wildcard-collection benchmark only where existing fixtures cannot exercise the wildcard-collection benchmark only where existing fixtures cannot exercise the
new scaling risk. Record commands and qualitative findings here; timing or new scaling risk. Record commands and qualitative findings here; timing or
allocation thresholds and a quiet-host benchmark campaign are not gates. allocation thresholds and a quiet-host benchmark campaign are not gates.
5. Remove transitional adapters and dead early-CLI paths. Search current code and 2. Confirm transitional adapters/dead early-CLI paths have been removed. Search
maintained examples for `SequenceCompositions` / `SequenceComposition`; only current code and maintained examples for `SequenceCompositions` and
intentional historical references may remain. Confirm no new dependencies, `SequenceComposition`; only intentional historical references may remain.
public internal-type exposure, background work, release machinery, or additional Confirm no new dependencies, public internal-type exposure, background work,
limits were introduced. release machinery, or additional limits were introduced. Review the full diff
6. Run `./scripts/check.sh`, `go test -race ./...`, and the development guide's against the roadmap, not merely passing tests. Record bounded review findings
local documentation/public-boundary checks, reusing Stage 19's successful code here; significant corrective
checks if code is unchanged and rerunning affected documentation checks. Review work requires its own focused unit rather than being hidden inside this stage.
the full diff against the roadmap, not merely passing tests. Record completed Do not declare completion while a target behavior remains incorrect or missing.
scope, validation, and any real remaining issues accurately; do not mark this
plan complete with an unimplemented target behavior. Validation: `./scripts/check.sh`, `go test -race ./...`, and the development
guide's local documentation/public-boundary checks, reusing successful checks
from Stages 28–29 if their covered content is unchanged. Record completed scope,
validation, benchmark commands/qualitative findings, and any real remaining
issues accurately. No new timing thresholds or release gates are introduced.
Acceptance: all roadmap operations and lifecycle boundaries work through ordinary Acceptance: all roadmap operations and lifecycle boundaries work through ordinary
and layered loads; values and inspection agree; application constraints hold on and layered loads; values and inspection agree; application constraints hold on
successful final results; deferred behavior and YAML-only guarantees have not successful final results; deferred behavior and YAML-only guarantees have not
expanded. All twenty stages are complete and their nonempty units committed. expanded. All thirty stages are complete and their nonempty units committed.
## Open Questions ## Open Questions