Plan sequence identity and combination implementation
This commit is contained in:
693
docs/roadmap/implementation.md
Normal file
693
docs/roadmap/implementation.md
Normal file
@@ -0,0 +1,693 @@
|
||||
# Sequence Identity And Combination Implementation Plan
|
||||
|
||||
Status: ready for implementation; no stages completed.
|
||||
|
||||
Audience: a Luna coding agent implementing one numbered stage per prompt, in
|
||||
order. This plan starts at commit
|
||||
`4f941f8ba4c9a9ed0afa87b07fe6ba29f131d1c6`. It implements the settled
|
||||
[sequence roadmap](sequences.md), [ADR 0017](../adr/0017-collection-sequence-policy.md),
|
||||
and [ADR 0018](../adr/0018-cli-after-structural-resolution.md).
|
||||
|
||||
## Working Rules
|
||||
|
||||
- Read [development guidance](../development.md) and the applicable policies,
|
||||
especially [agents](../policy/agents.md), [architecture](../policy/architecture.md),
|
||||
[testing](../policy/testing.md), and [documentation](../policy/documentation.md).
|
||||
Prefer the repository knowledge graph for code discovery.
|
||||
- The roadmap owns the selected feature semantics. This plan supplies execution
|
||||
order and implementation choices, not additional product requirements. Preserve
|
||||
unchanged source, parser, version, schema, ownership, and inspection contracts.
|
||||
- Complete each stage's implementation and behavioral tests together. Record its
|
||||
completion and validation here, then commit the nonempty unit with a short,
|
||||
plain-English message. Preserve unrelated changes. Do not publish intermediate
|
||||
stages or a release, update consumers' dependencies, or wait for CI.
|
||||
- Run focused tests while iterating and `./scripts/check.sh` before handing off
|
||||
each code stage. Use the installed compatible Go toolchain with dependencies
|
||||
already acquired, as the development guide describes. Add race coverage at the
|
||||
ownership milestones below. These are implementation checks, not new release
|
||||
gates. Reuse successful checks for unchanged content rather than repeating them
|
||||
at a documentation-only milestone. Do not require human platform validation.
|
||||
- Update affected declaration comments and canonical current-behavior prose when
|
||||
behavior lands. Keep incomplete target behavior in this roadmap/plan until it
|
||||
works. The final documentation stage reconciles the cross-feature narrative;
|
||||
it is not permission to leave misleading current contracts in earlier stages.
|
||||
- Transitional internal adapters are acceptable where specified. Remove them by
|
||||
completion. Do not add public compatibility aliases, permanent feature flags,
|
||||
alternate semantics, new dependencies, new resource caps, or an extension/plugin
|
||||
framework for this work.
|
||||
|
||||
## Implementation Decisions
|
||||
|
||||
### Public Registration Surface
|
||||
|
||||
Replace `SequenceComposition` and `Definition.SequenceCompositions` with:
|
||||
|
||||
```go
|
||||
type SequenceIdentity struct { /* private, root-owned registration state */ }
|
||||
type SequenceSelector []SequenceSelectorSegment
|
||||
type SequenceSelectorSegment struct { /* private literal-or-any-mapping-key state */ }
|
||||
|
||||
func SequenceLiteralKey(key string) SequenceSelectorSegment
|
||||
func SequenceAnyMappingKey() SequenceSelectorSegment
|
||||
func SequenceIdentityAt(path Path, fields ...string) SequenceIdentity
|
||||
func SequenceIdentityMatching(selector SequenceSelector, fields ...string) SequenceIdentity
|
||||
|
||||
// Field on Definition[T]:
|
||||
// SequenceIdentities []SequenceIdentity
|
||||
```
|
||||
|
||||
For example, the reusable definition can contain both registrations:
|
||||
|
||||
```go
|
||||
SequenceIdentities: []configkit.SequenceIdentity{
|
||||
configkit.SequenceIdentityAt(configkit.Keys("prompts"), "id"),
|
||||
configkit.SequenceIdentityMatching(configkit.SequenceSelector{
|
||||
configkit.SequenceLiteralKey("collections"),
|
||||
configkit.SequenceAnyMappingKey(),
|
||||
configkit.SequenceLiteralKey("prompts"),
|
||||
}, "id"),
|
||||
},
|
||||
```
|
||||
|
||||
The exact constructor converts mapping-only `Path` segments into literal selector
|
||||
segments. Preserve invalid index segments long enough to report an ordinary
|
||||
definition error; do not panic, silently drop them, or change `Path` semantics.
|
||||
The zero selector is the exact root selector, harmless under the existing
|
||||
mapping-root schema. A zero selector segment denotes the literal empty key,
|
||||
not a wildcard. Empty field names are valid literal field names; an empty field
|
||||
set is invalid. Constructors snapshot slices; load-entry conversion also takes
|
||||
an owned snapshot. Definition validation, not construction, reports errors.
|
||||
Root types and functions must not expose internal types or bridge methods.
|
||||
|
||||
### Identity Representation And Matching
|
||||
|
||||
- Normalize identity field sets once: deduplicate and sort exact field names.
|
||||
Use the same normalizer for YAML and application registration.
|
||||
- Encode a record's identity as a hashable, type-tagged, length-delimited tuple in
|
||||
normalized field order. Mapping lookup indexes avoid scanning record fields
|
||||
repeatedly. Distinguish structural-payload lookup from final typed-payload and
|
||||
parent-reference lookup: only fields acting as directives are excluded, not
|
||||
every occurrence of the same literal spelling. Never include tuple encodings
|
||||
in diagnostics.
|
||||
- Normalize finite JSON numbers to sign, significant decimal digits, and a signed
|
||||
base-10 exponent. Strip insignificant zeroes and give all zero spellings one
|
||||
representation. Use `math/big.Int` for exponent arithmetic, not a power-of-ten
|
||||
expansion or a float conversion. Thus large exponent spellings do not allocate
|
||||
storage proportional to their mathematical magnitude. This comparison-only
|
||||
representation does not rewrite document nodes or typed projection.
|
||||
- Compile selectors into a mapping-segment trie with literal and wildcard edges.
|
||||
Carry applicable trie states through traversal; stop at sequence-index edges.
|
||||
Index concrete matches within one assembly/inheritance scope where useful, but
|
||||
do not save an assembly-only destination list for final validation.
|
||||
- Check registration overlap pairwise once during definition validation using
|
||||
equal length and compatible corresponding segments. This straightforward
|
||||
`O(R² × D)` check, for R registrations and maximum selector depth D, is
|
||||
proportionate for registration definitions. Equivalent constraints coalesce;
|
||||
conflicting overlaps fail even with no matching input.
|
||||
Do not impose a registration-count limit to simplify this operation.
|
||||
|
||||
### Policy And Evidence Storage
|
||||
|
||||
Keep application constraints separate from YAML policy. Add private engine
|
||||
collection-policy records containing normalized identity, explicit mode, and
|
||||
declaration locations. Track whether identity is pending or belongs to a retained
|
||||
collection. Retain declaration presence separately from inferred defaults.
|
||||
|
||||
Use the existing acquisition/item scopes as ownership anchors, with an opaque
|
||||
per-load owner token when a retained record needs one. Store item-local policies
|
||||
relative to their owning record, and project concrete paths when that record is
|
||||
placed. Do not key ownership solely by numeric index, identity tuple, or source
|
||||
filename: all three can coincide for a replacement that must lose old policies.
|
||||
Inheritance clones the relevant scoped state; it does not create registrations.
|
||||
|
||||
Resolve policies through shared engine helpers used by strict composition,
|
||||
precedence, and inheritance. Do not attach engine policy to public values or add
|
||||
policy fields to the parser-independent `document.Node`. The sidecar can be
|
||||
extended with concrete-path indexes; it need not become a general event store.
|
||||
|
||||
Sequence combination returns the resulting list plus explicit input-to-output
|
||||
item projections and retained/replaced contribution information. Use that same
|
||||
result for values, policy transfer, origins, source destinations, sensitivity,
|
||||
and historical replay. Extend the existing sequence-history descriptors for
|
||||
noncontiguous upsert positions rather than inferring them from equal values.
|
||||
Sharing immutable internal evidence is acceptable under existing ownership rules;
|
||||
independently returned mutable records must remain detached.
|
||||
|
||||
### Diagnostics
|
||||
|
||||
Use existing `options.invalid` for invalid application definitions and existing
|
||||
`compose.invalid` / `compose.conflict` for malformed or duplicate strict-peer
|
||||
contributions. Add root-owned and engine diagnostic constants
|
||||
`CodeSequenceInvalid` (`sequence.invalid`) and `CodeSequenceConflict`
|
||||
(`sequence.conflict`) for malformed collection declarations/operations or final
|
||||
identities, and conflicting policy / precedence duplicates respectively.
|
||||
Final duplicate identities use `sequence.conflict`. Keep existing inheritance
|
||||
invalid/missing/cycle codes for record relationships. Use existing override and
|
||||
schema/decode codes at the CLI boundary; do not invent a new error taxonomy there.
|
||||
|
||||
Report collection/item paths and available locations, not identity values,
|
||||
normalized tuple strings, raw parent references, or raw hook errors. Declaration
|
||||
conflicts include both sites. Definition overlap errors identify registration
|
||||
positions without fabricating a concrete matched configuration path. Exact
|
||||
English diagnostic wording is not a new guarantee.
|
||||
|
||||
## Stage 1 — Implement Shared Scalar Identity Primitives
|
||||
|
||||
Scope: new private identity helpers and their focused tests; no load-path change.
|
||||
|
||||
1. Add `internal/engine/sequence_identity.go`. Implement field-set normalization,
|
||||
scalar tuple construction, and exact decimal normalization as chosen above.
|
||||
Accept string, number, boolean, and explicit null nodes; distinguish categories
|
||||
and absence. Exclude fields acting as structural directives from payload lookup.
|
||||
2. Validate mapping records, complete direct-field sets, and uniqueness using one
|
||||
tuple-to-first-record index. Return paths and source locations, not values.
|
||||
Support caller-selected diagnostic families for strict and other boundaries.
|
||||
3. Cover equivalent field sets; empty strings and null; missing/non-scalar fields;
|
||||
literal punctuation; compound tuple boundaries; `1`/`1.0`/`1e0`; signed zero;
|
||||
close unequal decimals; and large exponents without expansion. Verify that
|
||||
node text remains unchanged. Include `extends: null` as a structural field
|
||||
that cannot supply source identity.
|
||||
|
||||
Validation: `go test ./internal/engine -run 'SequenceIdentity|SequenceNumber'`
|
||||
and the everyday check. Retain the old registration path until its adapters are
|
||||
replaced; do not generalize unrelated YAML scalar decoding.
|
||||
|
||||
## Stage 2 — Compile Exact And Dynamic Application Constraints
|
||||
|
||||
Scope: private registration model, selector matching, overlap validation.
|
||||
|
||||
1. Extend `sequence_registration.go` with the common registration representation
|
||||
and compiled selector trie. Represent literal and arbitrary-mapping-key segments
|
||||
structurally, not with a magic string. Snapshot registration inputs.
|
||||
2. Validate field sets and mapping-only selectors before source acquisition.
|
||||
Preserve existing destination validation without adding restrictions on identity
|
||||
field spelling. Root and unused selectors remain harmless.
|
||||
3. Implement full-destination matching, equivalent overlap coalescing, and
|
||||
incompatible-overlap errors. Exact rules have no priority over dynamic rules.
|
||||
Keep a temporary conversion from the current single-field registration so
|
||||
existing callers still use their current behavior.
|
||||
4. Test exact and wildcard matches, multiple wildcard levels, empty/punctuation
|
||||
keys, sequence-index nonmatches, disjoint lengths/literals, equivalent sets,
|
||||
incompatible overlaps with absent input, and independent uniqueness scopes.
|
||||
|
||||
Validation: focused registration/selector tests and the everyday check. This
|
||||
stage does not export the replacement API or add final-result validation yet.
|
||||
|
||||
## Stage 3 — Parse Collection Declarations Without Filesystem Coupling
|
||||
|
||||
Scope: private parser and directive extraction helpers, initially unit-tested
|
||||
without activating collection combination in public loads.
|
||||
|
||||
1. Add `sequence_declaration.go` for source-root `configkit.sequences`. Each entry
|
||||
has a required existing literal `destination` form and optional `key` and
|
||||
`mode`. `key` is a string or nonempty string list; `mode` is exactly replace,
|
||||
append, or upsert. Use shared identity normalization. A destination-only entry
|
||||
can be a harmless no-op; do not add an unrequested ban on empty declarations.
|
||||
2. Split common root-directive extraction from import-specific parsing in
|
||||
`import_spec.go` / `parse.go`. Extract imports and sequences together, retaining
|
||||
source-local locations and explicit presence. Reject unrecognized directive
|
||||
fields and wrong supplied field types using existing strict parsing style.
|
||||
3. Keep declaration destinations relative to source placement, with item-local
|
||||
coordinates retained until their scope is assembled. Do not project them
|
||||
prematurely to a guessed numeric index.
|
||||
4. Give extraction an explicit overlay context: sequences are allowed, imports
|
||||
are not. Import enablement remains separately enforced; do not turn on
|
||||
filesystem access because a source declares sequences. Keep enabled
|
||||
source-version checks in raw source coordinates before consuming controls;
|
||||
extend existing actual header/control-overlap handling to sequence directives
|
||||
without treating an absent header beneath a control as an actual overlap.
|
||||
5. Test key-only, mode-only, combined, repeated-field-set equivalence, no-op,
|
||||
malformed types/modes/destinations, mixed imports/sequences, and overlay rules.
|
||||
Do not add import-level key/mode shorthand or sequence wrappers.
|
||||
|
||||
Validation: focused declaration/parser tests, existing import-spec tests, and the
|
||||
everyday check. Public activation occurs with the complete operations in Stage 9.
|
||||
|
||||
## Stage 4 — Model Scoped Policy Agreement And Lifetime
|
||||
|
||||
Scope: private policy resolution/state transformations, independent of loading.
|
||||
|
||||
1. Add `collection_policy.go`. Gather the current scope's declarations before
|
||||
combining peers. Reconcile equivalent keys and agreeing explicit modes;
|
||||
conflicting peers fail. Omitted fields have no preference. Apply authoritative
|
||||
application constraints even for key-only declarations without a value.
|
||||
2. Implement shared operation selection: current explicit mode, retained explicit
|
||||
mode, inferred upsert with effective identity, otherwise replace. Persist only
|
||||
explicit modes. Policy-only declarations must not replay earlier combinations.
|
||||
3. Implement key agreement, replacement reset, pending key declarations, and
|
||||
immediate validation when a key-only declaration identifies retained records.
|
||||
Changing mode to replace without a replacement value does not reset identity.
|
||||
4. Implement scope/record ownership transfer and retirement hooks. Actual ancestor
|
||||
replacement removes descendant collection identity and removed-record policy;
|
||||
independently owned modes and application constraints survive as specified in
|
||||
the roadmap. Do not repeatedly reapply old declarations and resurrect reset
|
||||
identity. Pending declarations retain their owning-scope rules.
|
||||
5. Use table-driven tests derived from the roadmap's agreement, persistence, and
|
||||
replacement tables, including absent/non-sequence destinations, equal outer
|
||||
identities with distinct record owners, and independent root versus item policy.
|
||||
|
||||
Validation: focused collection-policy tests and the everyday check. This stage
|
||||
must not add path-existence, shape, nullability, or global YAML final-check rules.
|
||||
|
||||
## Stage 5 — Implement The Shared Sequence Operations
|
||||
|
||||
Scope: a pure document-level combination helper, not separate implementations
|
||||
for overlays and inheritance.
|
||||
|
||||
1. Add `sequence_combine.go`, accepting lower/higher sequence nodes, resolved
|
||||
policy, and combination context. Reuse Stage 1 identity extraction and indexes.
|
||||
2. Implement replace, duplicate-rejecting append, and whole-record upsert. Upsert
|
||||
retains matched lower positions and appends new higher identities in order.
|
||||
Return item projections and replacement/retention evidence with the new value.
|
||||
3. Keep peer validation distinct: duplicate identities within any contribution or
|
||||
among peers fail under every mode. Do not merge record payloads or sort by ID.
|
||||
4. Cover absent versus null/non-sequence lower values; higher omission, `[]`, null,
|
||||
and non-sequence values; missing identity under append/upsert; empty inputs;
|
||||
stable ordering; equivalent/different keys; and explicit replacement identity
|
||||
reset. An empty input does not bypass kind checks or identity requirements.
|
||||
5. Verify clones do not alias mutable input nodes or policy slices. Build indexes
|
||||
once per operation rather than scanning the accumulated list for every record.
|
||||
|
||||
Validation: focused combination tests and the everyday check. Do not change the
|
||||
ordinary recursive mapping rule or add unkeyed append.
|
||||
|
||||
## Stage 6 — Extend Sequence Evidence And Historical Replay
|
||||
|
||||
Scope: teach existing provenance/history projection machinery to represent the
|
||||
operations before wiring them into public precedence.
|
||||
|
||||
1. Extend `sequenceHistoryContribution`, sequence replay in `history.go`, and
|
||||
projection helpers to consume Stage 5's explicit item mappings. Existing append
|
||||
ranges may remain a compact case; upsert cannot be represented by one offset.
|
||||
2. Preserve authored raw contribution snapshots and displaced values. A replaced
|
||||
record's omitted fields must appear as removed, not as retained lower payload.
|
||||
Preserve equal-valued replacement events. Keep existing index-based query and
|
||||
atomic sequence-diff APIs; add no public identity-selector path syntax.
|
||||
3. Project composition origins, nested source destinations, historical paths, and
|
||||
sensitivity through the same item mapping. Physical source IDs/locations and
|
||||
resource contexts continue to identify the declaring source.
|
||||
4. Exercise synthetic append/upsert/replace histories, noncontiguous matches,
|
||||
nested paths, discarded descendants, and secret-marked replaced records.
|
||||
Compare replayed final values with the actual combination result, including
|
||||
first/last and unchanged neighboring records. Reuse existing escaped-path and
|
||||
detached-query tests rather than duplicating them.
|
||||
|
||||
Validation: metadata/history/sequence/provenance/sensitivity tests,
|
||||
`go test -race ./internal/engine`, and the everyday check.
|
||||
|
||||
## Stage 7 — Separate Acquisition From Policy-Aware Tier Composition
|
||||
|
||||
Scope: refactor the assembly handoff, initially preserving existing behavior.
|
||||
|
||||
1. In `load_input.go`, `imports.go`, `load_request.go`, and related graph helpers,
|
||||
retain acquisition scopes, raw contributions, inclusion edges, source records,
|
||||
contexts, and extracted declarations without irrevocably composing them.
|
||||
Reuse existing `deferAssembly` and acquired-scope mechanisms instead of adding
|
||||
a second loader. Keep already acquired document handles closed as before.
|
||||
2. Separate acquiring an underlay group from composing its scopes. Keep authored
|
||||
acquisition order: main and depth-first imports, each underlay/input, overlay,
|
||||
then synthetic CLI records. Retain repeated-source, namespace, traversal-cycle,
|
||||
accounting, and stable scheduler behavior. Do not reacquire sources to obtain
|
||||
lower policy.
|
||||
3. Expose one internal composition entry that accepts accumulated lower policy and
|
||||
compiled application constraints. Complete item-local helper scopes before
|
||||
item placement; gather declarations before composing each relevant scope.
|
||||
4. Preserve existing version checks for this refactor. Retain a small adapter
|
||||
invoking the current composition semantics until the next integrations.
|
||||
Preserve source-version callbacks at acquisition, including header consumption.
|
||||
5. Run existing acquisition, dependency scheduling, item placement, source-version,
|
||||
ordinary/layered equivalence, resource-lifetime, and source-context tests. Add
|
||||
only targeted regressions for any newly split handoff not already covered.
|
||||
|
||||
Validation: the focused suites above and the everyday check. This refactor must
|
||||
not change source IDs or turn acquisition order into precedence order.
|
||||
|
||||
## Stage 8 — Compose Strict Scopes With Resolved Collection Identity
|
||||
|
||||
Scope: implement the policy-aware strict composition entry, including nested
|
||||
item scopes, before selecting it in all load paths.
|
||||
|
||||
1. Extend `composeDocumentsWithPolicy` and `validateRegisteredSequences` to use
|
||||
normalized field sets and scalar tuples. Resolve current declarations,
|
||||
applicable application constraints, and eligible lower identity before peer
|
||||
composition, using the Stage 4 helpers.
|
||||
2. Use the existing dependency-ready order for contributions and authored list
|
||||
order. Validate every identified sequence, including singleton contributions;
|
||||
strictly append distinct peers and reject duplicates under all modes.
|
||||
3. Complete helper imports and consume applicable non-main version headers before
|
||||
checking record payload identity. Retained structural directives cannot serve
|
||||
as identity fields. Do not validate an incomplete helper-built compound parent
|
||||
reference here; validate that completed reference during inheritance.
|
||||
4. For explicit replacement, do not use obsolete lower YAML identity to authorize
|
||||
otherwise unkeyed peer composition. Likewise, a new outer record cannot borrow
|
||||
nested identity from the displaced lower record. Still apply independently
|
||||
registered application constraints and genuinely applicable declarations.
|
||||
5. Attach Stage 6 evidence and Stage 4 scoped policy to composed results. Test
|
||||
lower-established identity, peer declaration conflicts, compound/scalar IDs,
|
||||
helper-supplied fields, consumed headers, and replaced record-local ownership.
|
||||
|
||||
Validation: composition, sequence identity, destination, import, and version
|
||||
tests plus the everyday check. Keep unregistered non-mapping conflicts unchanged.
|
||||
|
||||
## Stage 9 — Activate Ordinary Loading And Selected Overlay Policies
|
||||
|
||||
Scope: ordinary `Load`, source declarations, and its selected overlay.
|
||||
|
||||
1. Wire common directive extraction into every ordinary source form: file,
|
||||
native filesystem, bytes, reader, mapping, imported mapping, and sequence item.
|
||||
Declarations work without `Imports`; disabled imports retain their existing
|
||||
rejection semantics. `NoSource` remains absence rather than an empty document.
|
||||
2. Use Stage 8 composition, followed by Stage 5 precedence operations in the
|
||||
assembly accumulator. Reuse the Stage 4 policy resolver; mappings still recurse
|
||||
and unrelated sequences still replace. Preserve mode-only/key-only effects.
|
||||
3. Permit `configkit.sequences` in selected overlays while continuing to reject
|
||||
overlay imports. Strip structural controls from application decoding, but
|
||||
retain their locations/evidence for their structural lifetime and diagnostics.
|
||||
4. Apply projection and policy-retirement information as part of each operation,
|
||||
not in a later best-effort repair pass. Cover inline/imported equivalence,
|
||||
placement-relative declarations, append/upsert/replace, empty lists, retained
|
||||
modes, and declarations without accompanying values.
|
||||
5. Update current sequence/overlay contract sections and tests affected by this
|
||||
activation. CLI and inheritance retain their explicitly incomplete milestones
|
||||
until their later stages; do not claim those parts of the roadmap implemented.
|
||||
|
||||
Validation: ordinary-load, overlay, import, composition, and metadata tests plus
|
||||
the everyday check. Include one public ordinary-load declaration example test.
|
||||
|
||||
## Stage 10 — Integrate Lower Policy Across Layered Assembly
|
||||
|
||||
Scope: source-backed underlays and the layered selected-overlay path.
|
||||
|
||||
1. Acquire main first, but assemble underlays from lowest to highest, strictly
|
||||
composing each group with applicable accumulated lower policy before applying
|
||||
its one precedence operation. Then compose/apply main and the selected overlay.
|
||||
Do not sequentially upsert peer sources as if they were different tiers.
|
||||
2. Pass lower identity into higher strict scopes before peer composition. Cover an
|
||||
otherwise empty underlay establishing identity, inherited explicit mode, and
|
||||
replacement that deliberately discards the old YAML key. Invalid lower tiers
|
||||
still fail even if a higher replacement would hide their values.
|
||||
3. Keep main-only version ownership and every enabled per-source version rule.
|
||||
In default source-version mode, run the main checker once on the composed
|
||||
main tier before overlay/inheritance/defaults, never on underlay-augmented
|
||||
values. Its former main-before-underlay composition timing need not persist:
|
||||
main composition now depends on lower policy. Enabled-mode source checks keep
|
||||
acquisition order and exact source-local header handling. No extra callbacks
|
||||
or second acquisition pass are permitted.
|
||||
4. Keep histories in precedence order, sources in acquisition order, underlay
|
||||
indexes stable across empty groups, and shared budgets intact. Update the
|
||||
layered contract's acquisition-versus-composition description accordingly.
|
||||
5. Cover multiple inputs/groups, mounted main inputs, item scopes, empty main,
|
||||
explicit empty groups, layered overlays, version protection and diagnostics,
|
||||
and equivalence with ordinary loading where the inputs express the same model.
|
||||
|
||||
Validation: layered/grouped, source-version, dependency scheduling, destination,
|
||||
and metadata tests plus the everyday check.
|
||||
|
||||
## Stage 11 — Verify Replacement And Nested Policy Ownership End To End
|
||||
|
||||
Scope: the consequential lifetime cases across the now-active source assembly.
|
||||
|
||||
1. Exercise an item-local nested collection through retaining the outer record,
|
||||
upserting the same outer ID, and replacing the entire outer sequence. Retained
|
||||
items keep their local key/mode; replacements do not inherit discarded policy
|
||||
merely because identity, source label, or final index is equal.
|
||||
2. Exercise ancestor replacement and later reintroduction. Removed collection
|
||||
identity does not reappear; independently declared append mode can survive
|
||||
and require renewed identity. Cover renewed YAML key, application constraint,
|
||||
explicit replace, and the intended failure with no available identity.
|
||||
3. Cover key-only validation of retained records, pending declarations, a no-value
|
||||
replace declaration that cannot authorize key changes, and independent policy
|
||||
whose owning scope survives a record replacement.
|
||||
4. Correct any integration defects in the Stage 4 ownership transformations;
|
||||
consolidate tests with their mechanism owners. Do not introduce special-case
|
||||
ownership by numeric path or a historical identity cache to make cases pass.
|
||||
5. Verify frozen metadata, origin/resource context, sensitivity of discarded
|
||||
records, and mutable independence across repeated loads for these scenarios.
|
||||
|
||||
Validation: focused policy/sequence/layered/ownership tests,
|
||||
`go test -race ./internal/engine`, and the everyday check.
|
||||
|
||||
## Stage 12 — Apply Collection Policy During Mapping-Sibling Inheritance
|
||||
|
||||
Scope: retain existing mapping-sibling relationships, extend their collection
|
||||
combination and policy transfer.
|
||||
|
||||
1. Extend inheritance templates/expansion sidecars to carry collection identity
|
||||
and record-local policy alongside nodes and origins. Keep existing scheduler
|
||||
keys, cancellation, version protection, missing-parent/cycle diagnostics, and
|
||||
destination-local evaluation of nested templates.
|
||||
2. Use the shared sequence combiner at the inheriting destination: parent lower,
|
||||
child higher, destination explicit mode or identity-based default. Copy YAML
|
||||
identity where applicable, but do not automatically copy a parent collection's
|
||||
explicit mode to a different destination. Match application constraints at the
|
||||
destination, including newly created matching paths.
|
||||
3. Transfer record-owned state only with the records that survive. Preserve
|
||||
original nested instructions for each destination's later resolution rather
|
||||
than copying only a parent's already-expanded output.
|
||||
4. Extend inheritance evidence using the same projections as source operations.
|
||||
Cover nested append/upsert/replace, explicit empty/null/kind cases, copied key
|
||||
with inferred upsert, destination explicit mode, incompatible keys, application
|
||||
constraints, cancellation, and policy at an inherited new namespace.
|
||||
|
||||
Validation: inheritance, history, provenance, sensitivity, and resource-context
|
||||
tests plus the everyday check. No identified-record parent lookup is active yet.
|
||||
|
||||
## Stage 13 — Build Identified-Record Inheritance Graphs
|
||||
|
||||
Scope: relationship extraction/validation and shared scheduling for sequence
|
||||
scopes; test these helpers before activating payload expansion.
|
||||
|
||||
1. Extend sibling-scope discovery to identified sequences. Build the tuple-to-item
|
||||
lookup once using effective identity. Every record supplies its own complete
|
||||
identity before inheritance; neither parent lookup nor typed defaults supplies
|
||||
a missing child field.
|
||||
2. Parse a surviving record's `extends` as null cancellation or a mapping with
|
||||
exactly the normalized identity fields and supported scalar components.
|
||||
Mapping-sibling references keep their existing string form. Do not accept a
|
||||
scalar shorthand, multiple parents, positional or cross-collection lookup.
|
||||
3. Parent-reference mappings have already used ordinary strict mapping
|
||||
composition: disjoint helper fields can complete one reference, while equal
|
||||
overlapping fields conflict. Validate completeness/extra fields only now on
|
||||
the surviving reference. Discarded references remain historical evidence.
|
||||
4. Build one-parent dependency edges and reuse `scheduleDependencies` with stable
|
||||
semantic-pointer keys. Missing parents, self-reference, and longer cycles
|
||||
use existing inheritance diagnostics without identity values. Keep authored
|
||||
output positions independent from graph schedule.
|
||||
5. Cover compound and null references, numeric equality, forward edges, chains,
|
||||
unrelated equal identities in different lists, malformed refs, missing parents,
|
||||
cycles, helper-built references, and overwritten/cancelled relationships.
|
||||
|
||||
Validation: focused inheritance graph/reference tests and the everyday check.
|
||||
|
||||
## Stage 14 — Resolve Record Inheritance With Ownership And Evidence
|
||||
|
||||
Scope: integrate Stage 13 graphs into the existing structural expansion phase.
|
||||
|
||||
1. Expand parent payload into the child while preserving the child's own identity
|
||||
fields and their origin/presence. Exclude parent identity fields from payload
|
||||
inheritance. Recursively combine mappings and use destination collection policy
|
||||
for nested sequences; retain whole-record upsert at the earlier boundary.
|
||||
2. Consume the child's directive before decoding. Keep parents as ordinary records
|
||||
subject to the same schema/semantic validation; create no abstract templates or
|
||||
acquired source records.
|
||||
3. Resolve only records surviving tier assembly. Cover parent upsert/removal,
|
||||
replacement of a child without `extends`, self-reference not exposing a lower
|
||||
predecessor, and nested destination-local relationships. Enclosing scopes
|
||||
finish before nested scopes; assembled list order stays unchanged.
|
||||
4. Carry policy, origin/history, sensitivity, resource context, and existing
|
||||
expansion/retained-history accounting through copies. Do not widen budgets or
|
||||
require a new inspection-path API.
|
||||
5. Add public-load regressions for complete inheritance chains and nested
|
||||
collections, including secret canaries on success and failure. Update the
|
||||
implemented inheritance contract and consumer guidance for this scope.
|
||||
|
||||
Validation: all inheritance and cross-feature inspection tests,
|
||||
`go test -race ./internal/engine`, and the everyday check.
|
||||
|
||||
## Stage 15 — Expose The Root-Owned Registration API
|
||||
|
||||
Scope: the settled facade surface and migration of in-repository callers.
|
||||
|
||||
1. Implement the exact public names/signatures above in a focused root file, and
|
||||
add `Definition.SequenceIdentities`. Convert root records/selectors explicitly
|
||||
into the engine model. Snapshot fields, selectors, and paths on construction
|
||||
and loading, preserving immutable definition reuse.
|
||||
2. Remove `SequenceComposition`, `Definition.SequenceCompositions`, and the private
|
||||
old-shape adapters. Migrate all maintained code/tests/examples; do not edit
|
||||
historical release notes solely to erase the old name or add deprecated aliases.
|
||||
3. Cover mixed exact/dynamic constructors, compound scalar fields, literal `*`
|
||||
versus wildcard segments, root/unused selectors, invalid index paths and empty
|
||||
field sets, overlap failure before source access, and independent loads.
|
||||
4. Prove dynamic constraints apply before strict composition and at inherited
|
||||
destinations, without consumer namespace enumeration. Keep final-result
|
||||
enforcement described as pending until Stage 18 actually adds it.
|
||||
5. Verify the public boundary and declaration documentation. Do not expose engine
|
||||
enums, policy stores, parser nodes, or internal methods.
|
||||
|
||||
Validation: root sequence/facade, registration, public/dependency-boundary, and
|
||||
consumer example tests plus the everyday check.
|
||||
|
||||
## Stage 16 — Prepare Schema-Aware Post-Decode CLI Application
|
||||
|
||||
Scope: a private assignment helper and structural-directive validation, without
|
||||
switching lifecycle order until Stage 17.
|
||||
|
||||
1. Refactor override preparation so labels, paths, snapshots, type support,
|
||||
disjointness, version protection, and existing budgets remain reusable outside
|
||||
the source assembly accumulator. Allocate synthetic sources once in the same
|
||||
order; do not reopen or reinterpret CLI values as YAML.
|
||||
2. Extend structural rejection to `extends` as well as `configkit` in supplied
|
||||
paths and nested values, using their established structural roles. This is
|
||||
post-structural application data, not another instruction channel. Do not add
|
||||
unrelated name restrictions.
|
||||
3. Add `applyDecodedOverrides`, using existing schema/decode mechanisms to bind
|
||||
only supplied override subtrees against a detached copy of the decoded state.
|
||||
Mappings recursively overlay, sequences replace, and explicit zero/empty/null
|
||||
retain normal schema rules. Current typed values supply retained children,
|
||||
not new defaults for replaced list items. Do not call `Definition.Defaults`
|
||||
again or re-decode the whole projected configuration as newly authored input.
|
||||
4. Return the updated typed value, canonical post-CLI projection, presence/origin
|
||||
changes, and existing raw CLI history contributions. Preserve a pre-CLI typed
|
||||
projection for decoding reconciliation/history. No resolver runs on a failed
|
||||
override; no partially applied result is returned.
|
||||
5. Test unknown/type-invalid fields, mapping retention, sequence replacement,
|
||||
nested maps/lists, named/text/duration/secret schema fields, explicit null/false/
|
||||
zero, discarded children, structural controls, and mutable ownership. Reuse
|
||||
existing label/path/cycle/budget tests without recopying their full inventory.
|
||||
|
||||
Validation: override, decode, schema, projection, and ownership tests plus the
|
||||
everyday check. Only the helpers are added here; lifecycle changes next.
|
||||
|
||||
## Stage 17 — Move CLI After Decoding And Preserve Inspection Order
|
||||
|
||||
Scope: activate ADR 0018 consistently through both public loading paths.
|
||||
|
||||
1. Remove CLI assignment from ordinary/layered structural assembly. Invoke
|
||||
Stage 16 after inheritance and successful typed-default creation/decoding,
|
||||
immediately before application resolution. Keep the single shared lifecycle;
|
||||
do not add separate ordinary and layered CLI implementations.
|
||||
2. Seed resolution with the post-CLI configuration, explicit-presence/origin
|
||||
metadata, source records, and contexts. Ensure `Resolution.Default` preserves
|
||||
explicit CLI values while recorded transformations can change them normally.
|
||||
3. Extend existing replay/reconciliation so evidence follows the real order:
|
||||
source tiers, inheritance, typed decoding/defaults, CLI, application resolution.
|
||||
Keep the structural input, pre-CLI typed snapshot, raw CLI contributions, and
|
||||
post-CLI projection distinct where needed. Do not report CLI changes as
|
||||
defaults, decoding conversions, or inheritance effects.
|
||||
4. Feed effective post-override/post-resolution reference values and their
|
||||
locations into secret lookup and final validation. Do not use stale structural
|
||||
input to determine effective CLI secret references. Preserve no-credential-byte
|
||||
history and exactly-once default-factory behavior.
|
||||
5. Cover `NoSource`, ordinary/layered equivalence, CLI clearing/replacing an
|
||||
identified list despite append/upsert mode, derived values seeing CLI input,
|
||||
equal-value provenance, and resolver changes after CLI. Add explicit regressions
|
||||
that CLI cannot fix earlier decode/inheritance errors or introduce/cancel a
|
||||
relationship. Update obsolete early-CLI tests deliberately.
|
||||
6. Update the lifecycle, override, history, and secret-reference contracts and
|
||||
affected examples to describe the now-implemented order.
|
||||
|
||||
Validation: override/lifecycle/resolution/inheritance/history/secret tests,
|
||||
`go test -race ./...`, and the everyday check.
|
||||
|
||||
## Stage 18 — Enforce Final Application Identity Constraints
|
||||
|
||||
Scope: the final-result guarantee, not another combination or inheritance phase.
|
||||
|
||||
1. Add shared final identity validation after effective secret lookup and before
|
||||
application semantic validation/publication in the common lifecycle. Use the
|
||||
canonical effective typed projection, which retains secret references rather
|
||||
than credential bytes. Preserve the existing validated publication snapshot
|
||||
and detachment behavior.
|
||||
2. Traverse the actual final tree with the compiled selector trie, validating each
|
||||
matching sequence independently with Stage 1 primitives. This must discover
|
||||
paths introduced by typed defaults, CLI, and callbacks, not only assembly paths.
|
||||
Support root/namespaced rules and unrelated unselected sequences.
|
||||
3. Validate actual sequences only. Missing paths, null, and other shapes remain
|
||||
the schema/application validator's responsibility. Identity extraction here
|
||||
examines effective typed payload, not whether a field was authored. Do not
|
||||
recheck YAML-only policies or every intermediate callback state.
|
||||
4. Cover duplicate/missing/non-scalar identities produced at each late phase,
|
||||
valid null/scalar identities, numeric equality after typed conversion, dynamic
|
||||
namespaces created by each late phase, harmless unmatched rules, concrete-path
|
||||
diagnostics with available origins, and secret-value non-disclosure.
|
||||
5. Prove both `Load` and `LoadLayered` use this guarantee and return no usable
|
||||
result on failure. Update registration comments and canonical contract with
|
||||
this final boundary, excluding later caller mutations from the guarantee.
|
||||
|
||||
Validation: final identity, selector, facade, defaults/resolution/override, and
|
||||
secret tests plus the everyday check.
|
||||
|
||||
## Stage 19 — Exercise Representative Consumer Workflows
|
||||
|
||||
Scope: a small number of meaningful integrations, not a Cartesian test matrix.
|
||||
|
||||
1. Extend `examples/consumers` with an executable source-backed catalog: underlay
|
||||
records, main fragments, selected overlay append/upsert/replace, and explicit
|
||||
record inheritance. Demonstrate complete replacement of a matched record,
|
||||
stable order, and final inspection without application merge code.
|
||||
2. Add a namespaced typed-map example using one wildcard registration plus an
|
||||
exact root registration, with independent uniqueness. Show YAML-declared
|
||||
identity without registration separately so the final guarantee is not implied.
|
||||
3. Include one integrated CLI/resolution case and one representative provenance/
|
||||
resource-context/secret-history case where those interactions add protection
|
||||
beyond mechanism tests. Keep application domain validation recognizable.
|
||||
4. Audit coverage against the roadmap's work inventory, especially helper-built
|
||||
parent references, version-header consumption, pending declarations, lower
|
||||
identity before higher peers, replacement ownership, and dynamic destinations
|
||||
introduced through inheritance and late typed changes. Add only missing,
|
||||
consequential regressions at their narrowest stable owners.
|
||||
5. Consolidate obsolete/brittle tests. Do not restore old replacement semantics
|
||||
merely to keep fixtures unchanged, and do not preserve duplicates of the same
|
||||
semantic table across every layer.
|
||||
|
||||
Validation: `go test ./examples/consumers`, focused root integration tests,
|
||||
`go test -race ./...`, and the everyday check.
|
||||
|
||||
## Stage 20 — Reconcile Documentation And Complete The Review
|
||||
|
||||
Scope: finish the current-contract narrative and verify the full target state.
|
||||
|
||||
1. Reconcile `docs/config.md` for declarations, identity equality, selectors,
|
||||
operations/persistence, ownership, inheritance, final validation, and CLI order.
|
||||
Keep its operation and presence tables authoritative; Go comments own exported
|
||||
symbols and consumer guidance links rather than copying full contracts.
|
||||
2. Update `docs/consumers/README.md`, relevant root examples, `README.md`,
|
||||
`docs/internal/overview.md`, `docs/internal/imports.md`, and
|
||||
`docs/development.md` where the implemented structure or reading/test map has
|
||||
changed. Remove stale claims about string-only registration, universal sequence
|
||||
replacement, early CLI, and overlay prohibition of all directives.
|
||||
3. Remove implemented portions from the deferred identified-record discussion in
|
||||
`future.md`, retaining genuinely deferred recursive body merging and other
|
||||
still-relevant ideas. Preserve accepted ADR decision content; update only status
|
||||
or implementation references where appropriate. Architecture already adopts
|
||||
the decisions: remove pending-implementation wording when true, without adding
|
||||
new policy. Leave release version selection/publication to the release task.
|
||||
4. Inspect the completed code for duplicate combination/identity algorithms,
|
||||
per-record scans of whole collections, repeated declaration gathering, numeric
|
||||
magnitude expansion, stale ownership/path caches, and unnecessary copies.
|
||||
Prefer shared indexed mechanisms without pooling/global caches. Reuse existing
|
||||
composition/underlay/inheritance benchmarks and add a focused mixed-upsert or
|
||||
wildcard-collection benchmark only where existing fixtures cannot exercise the
|
||||
new scaling risk. Record commands and qualitative findings here; timing or
|
||||
allocation thresholds and a quiet-host benchmark campaign are not gates.
|
||||
5. Remove transitional adapters and dead early-CLI paths. Search current code and
|
||||
maintained examples for `SequenceCompositions` / `SequenceComposition`; only
|
||||
intentional historical references may remain. Confirm no new dependencies,
|
||||
public internal-type exposure, background work, release machinery, or additional
|
||||
limits were introduced.
|
||||
6. Run `./scripts/check.sh`, `go test -race ./...`, and the development guide's
|
||||
local documentation/public-boundary checks, reusing Stage 19's successful code
|
||||
checks if code is unchanged and rerunning affected documentation checks. Review
|
||||
the full diff against the roadmap, not merely passing tests. Record completed
|
||||
scope, validation, and any real remaining issues accurately; do not mark this
|
||||
plan complete with an unimplemented target behavior.
|
||||
|
||||
Acceptance: all roadmap operations and lifecycle boundaries work through ordinary
|
||||
and layered loads; values and inspection agree; application constraints hold on
|
||||
successful final results; deferred behavior and YAML-only guarantees have not
|
||||
expanded. All twenty stages are complete and their nonempty units committed.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None. Feature choices are settled in the roadmap and accepted ADRs; this plan
|
||||
selects the remaining API, representation, validation, and execution decisions.
|
||||
@@ -1,9 +1,9 @@
|
||||
# Sequence Identity And Combination Roadmap
|
||||
|
||||
Status: settled feature roadmap, ready for a staged implementation plan. This
|
||||
is the targeted end state, not an implemented contract. The subsequent plan will
|
||||
specify execution order, exact Go names and signatures, internal bookkeeping,
|
||||
and validation stages.
|
||||
Status: settled feature roadmap; implementation is specified in the
|
||||
[staged implementation plan](implementation.md). This document defines the
|
||||
targeted end state, not an implemented contract. The implementation plan owns
|
||||
execution order, exact Go names and signatures, bookkeeping, and validation.
|
||||
|
||||
## Purpose And Scope
|
||||
|
||||
@@ -906,8 +906,9 @@ policy ownership and inheritance transfer.
|
||||
|
||||
## Work Needed From The Baseline
|
||||
|
||||
The implementation plan should cover these connected changes, with execution
|
||||
order and test sequencing defined there:
|
||||
The baseline needs the following connected capabilities to reach the target
|
||||
state. Execution order and test sequencing belong to the
|
||||
[implementation plan](implementation.md):
|
||||
|
||||
- **Declarations and public API:** extend application registration to the settled
|
||||
identity model with one public registration type and definition field, an
|
||||
@@ -993,6 +994,6 @@ status while preserving its decision content. The architecture policy is aligned
|
||||
Dynamic selectors extend the destinations covered by application registration;
|
||||
they retain the accepted application authority and final-validation boundary.
|
||||
|
||||
No open feature decisions remain. The implementation plan should translate this
|
||||
end state into reviewable stages without adding guarantees, restrictions, or
|
||||
No open feature decisions remain. The implementation plan translates this end
|
||||
state into reviewable stages without adding guarantees, restrictions, or
|
||||
combination semantics.
|
||||
|
||||
Reference in New Issue
Block a user