45 KiB
Implementation Plan: Sequence Placement and Composition
Status: implementation in progress; stages 1–10 completed.
Purpose and authority
Implement the accepted feature roadmap, starting from
2051177 (Settle sequence placement and composition roadmap decisions). This
plan owns execution order and progress; the roadmap owns intent, scope, and the
ten settled behavior decisions. The target implementer is a Luna coding agent.
Complete the numbered stages in order. Each stage is a separate focused prompt
and validated commit, not a request to implement the whole feature at once.
Read development, agents, architecture, testing, and documentation before starting. Use the code graph for discovery, then inspect the current owning implementation and tests. Preserve unrelated work. Follow the existing commit policy after each nonempty completed unit; record completion and validation in this plan.
This planning change implements nothing. Later stages introduce private engine capabilities before exposing them through the root facade. Until a capability lands, current consumer documentation must continue to describe implemented behavior only. Accepted ADRs may describe the decision before implementation. When a stage makes behavior reachable by consumers, update its owning declaration comments and configuration contract in that same unit, to the implemented boundary. Stage 13 consolidates guidance and examples; it does not postpone those contract updates.
Fixed design for implementation
Public API and YAML spelling
Use these root-owned declarations and fields. Their zero values preserve current
behavior; do not change the signatures of Load or LoadLayered.
type PlacementMode uint8
const (
PlacementModeMapping PlacementMode = iota
PlacementModeSequenceItem
)
type SequenceComposition struct {
Path Path
IdentityField string
}
- Add
SequenceCompositions []SequenceCompositiontoDefinition[T]. - Add
PlacementMode PlacementModetoInputandSourceInfo. - Add
DestinationUnresolved booltoSourceInfo. False preserves the existing exact-destination interpretation; true explicitly marks the enclosing-container fallback described below. This is output evidence, not an input option. - Add both
Destination PathandPlacementMode PlacementModetoOptions[T]. OrdinaryLoadneeds the same per-source controls as the layered entry point; its overlay and CLI coordinates remain application-rooted. - YAML import declarations accept
placement: mappingorplacement: sequence-item. Omission meansmapping, independently for every declaration. Retaindestination: [literal, keys],path,glob, andoptionalwith their existing meanings. Do not add aliases or inferred modes. - Root fields and types stay root-owned. Use explicit conversions into engine records and a private engine mode, following the source-version-mode pattern. Preserve unknown numeric modes until engine preflight rejects them.
Validate options before acquisition. Unknown placement modes, item placement at
an empty destination, invalid destinations/registration paths, and conflicting
registrations use CodeOptionsInvalid. Registration paths use existing literal
mapping-key destination normalization, including its current reserved-key rules;
do not add more reserved names. Parsed numeric tokens become literal keys;
explicit indexes and invalid paths fail. Identical registrations coalesce, unused
registrations are permitted, and an empty/root registration is harmless and
unused under the mapping-root model. Do not reject ancestor/descendant pairs.
IdentityField is one literal string, including an empty string, not a path.
No field-name or identity-value normalization is introduced.
Use CodeImportInvalid for malformed import placement syntax, including an item
import whose effective destination is empty in its current mapping scope. An
omitted item-import destination is valid when inherited placement already names
a non-root collection. At an item root, an omitted destination names the item
mapping itself and cannot turn that mapping into a sequence. A nested item import
needs a nonempty effective destination within that item scope; this can come from
its relative destination or an ordinary helper's existing placement prefix.
Use CodeComposeInvalid for malformed registered-sequence records/identities and
CodeComposeConflict for duplicate identities or ordinary strict conflicts.
Malformed-record paths identify the effective item or its identity field;
duplicate paths identify the sequence container, with the first and conflicting
physical locations in Primary/Related. Error messages must not interpolate
identity strings or record bodies. Preserve deterministic reporting without
inventing an all-errors guarantee or freezing incidental message wording.
Composition, scopes, and evidence
- Compile registrations once per load into an owned exact-path lookup. A strict tier has a separate identity table for each encountered registered sequence. Scan every actual sequence contribution, even one with no peer. Require map items and a present string identity after source-header processing and helper composition. Compare exact strings; accept empty strings. Non-sequences retain existing behavior. Distinct records append in contribution order; repeated identities conflict even when bodies are equal or disjoint.
- Keep every precedence boundary atomic for sequences. Each underlay group and the main tier starts new identity state. Selected overlays, CLI, defaults, inheritance, and application resolution do not run this identity validator. Registrations apply at effective literal mapping paths, never through record indexes. They are composition policy, not final configuration schema.
- Source acquisition remains depth-first and keeps current source IDs, physical resource identity, cycle/repetition detection, and shared budgets. Model an item source plus its ordinary helper imports as a local strict mapping scope. Ordinary imports share that scope and append relative mapping destinations; an explicit item import opens a child item scope. Complete each item scope before wrapping/validating its record in the enclosing composition. Keep its contribution at the declaring source's original traversal position.
- Carry a small private placement projection with each contribution: its scope,
local destination, and any sequence range assigned by strict assembly. Use it
to translate source paths into effective paths without reordering sources or
mutating acquired trees. Authored sequence index
jbecomesoffset+j; an item source's local root becomes the assigned item index. Nested projections compose. Do not locate records again by rescanning IDs during inspection. - Keep completed trees distinct from source/history applications. Extend private history operations to distinguish a sequence contribution from recursive mapping merge: the first contribution to a collection in a tier replaces the previous tier/default value; later contributions append. An empty first list clears the prior list; an empty later list adds no items. Store only incoming fragments and offsets, not a cumulative list snapshot after every append. For item scopes, retain the initial item body and helper contributions at their projected item paths, so helpers retain their own history and origins.
- Reuse that projection in origins, history replay/query relevance, historical path enumeration, sensitivity, and version evidence. Public source records describe containers, not a new permanent record-identity address. An item source reports its sequence container; a mapping helper inside an item reports its effective mapping destination, which can include an enclosing item index. Output evidence may contain such indexes even though input destinations and registrations accept only mapping-key segments.
Unresolved destination evidence
SourceInfo.DestinationUnresolved describes whether that snapshot's Destination
is the source's exact effective destination container:
- When false,
Destinationis exact. This says nothing about whether the load succeeded or the source's values survived later replacement. An item source can have an exact sequence-container destination before its own item index is assigned, because that index is not part of its reported container path. - When true, an enclosing item index needed for the destination has not been
assigned.
Destinationis only the nearest resolved enclosing collection container, not the source's actual mapping destination or nested sequence destination. Do not append the unresolved local suffix as though the missing index were a mapping key, or use a fabricated index. - For example, a mapping helper destined for
prompts[1].metadatareports the containerpromptswithDestinationUnresolved: trueuntil the outer item index is assigned. Later snapshots reportprompts[1].metadatawith false. The item-root source itself reportspromptswith false when that container is known. The placement mode continues to describe each source's own mode. - Apply this distinction to every source-record exposure, including source records available during callbacks and failed-load records. Once projection resolves the container, subsequent snapshots report the exact path with false. Previously returned detached snapshots retain their original path and flag. Completed successful loads report resolved containers; early failures may contain both resolved and unresolved source records.
This flag makes the existing planned fallback observable without adding a public item-index API, promising permanent indexes, or reconstructing unknown placement after a failure. Physical source locations, source IDs, and source-local version evidence retain their existing meaning regardless of the flag.
Choose private helper names and file splits to fit existing responsibilities. Do not introduce a provider framework, generic merge callbacks, a second loading lifecycle, or a public identity-selector API to implement these mechanics.
Efficiency and preserved limits
Use one string-keyed identity map per registered sequence per strict tier, with ordered appends and existing indexed mapping lookup. Clone incoming records into owned assembly storage without repeatedly cloning the accumulated sequence. Registration lookup, validation, and sequence construction should be linear in the visited data apart from existing path/depth costs. Complete each item scope once; do not recursively recompose the entire ordinary-helper subtree at every import edge. Keep retained history fragments immutable and independently owned from the working tree. These are implementation choices, not a new public complexity, allocation, or latency guarantee.
Preserve current per-source, aggregate, traversal, projection, and retained-history limits and error families. Count acquired content once and synthetic mapping and sequence wrappers where introduced; include the extra sequence level in depth checks. Do not charge an already-accounted item/helper body again merely because it moves to a completed scope. Keep library-owned handles closed before returning on success or failure. Do not add new limits or change their existing values.
Stage 1 — Record the decision and prepare private registration controls
Scope: one ADR and engine option normalization; no new public fields yet.
- Write
docs/adr/0014-sequence-placement-and-composition.mdas an Accepted ADR using the established format. Explain explicit item placement and registered duplicate-rejecting strict composition, unchanged precedence replacement, and their rationale in ConfigKit terms. Discuss alternatives only in Alternatives Considered; do not center a downstream application's semantics. Link the roadmap for implementation status. Do not rewrite accepted earlier decisions. - Add engine-owned registration records/state and normalization, reachable from
engine
Definition. Validate/coalesce using the fixed design above; detach registration slices and paths at the load boundary. Keep compiled policy per-load, not mutable shared Definition state or a global cache. - Wire preflight into both entry points without consuming a reader or opening a source for invalid options. Thread compiled policy through the request rather than rebuilding it or invoking application hooks for internal handoffs. Keep the existing shared source-version checker lifecycle intact.
- Add focused normalization tests for duplicate/conflicting registrations, empty identity-field names, numeric/escaped literal keys, invalid index paths, unused/root registrations, and preserved zero-option behavior.
Validation: focused option tests and ./scripts/check.sh. Completion: existing
loads remain unchanged, with only private registered-policy input newly accepted.
Completion evidence: added Accepted ADR 0014 and private engine registration
normalization. Ordinary and layered entry points compile/coalesce policy and
reject invalid registrations before source reads, then carry the compiled policy
through the load request while clearing raw registration slices from internal
handoffs. No public API or composition behavior was added. Focused coverage
checks duplicate/conflicting definitions, empty identity names, numeric and
escaped literal keys, invalid explicit indexes, unused/root registrations,
zero-option behavior, and reader preservation on preflight failure. These checks
passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SequenceRegistration' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
The compiled registration policy is intentionally not consumed by composition until the next implementation unit. No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 2 — Implement registered strict sequence validation and assembly
Scope: internal/engine/compose.go and focused composition tests.
- Add a policy-aware internal composer while preserving an unregistered adapter for existing callers/tests. Collect/check ordinary claims with current strict semantics; only sequence/sequence claims at registered effective paths use the new rule. Mapping/sequence, null/sequence, and scalar overlaps still conflict.
- Validate every registered sequence as it is encountered, not only overlapping claims. Missing paths and non-sequences do not independently fail registration. Validate the completed contribution, not the unfinished raw item source.
- Build identity tables once, reject duplicates within/across lists, and append records into owned assembly storage. Preserve mapping traversal and list order, explicit empty sequences, exact identity spelling, and existing error safety.
- Return the assigned source-to-effective sequence ranges alongside the composed result for the following stages. Do not change overlay merging or diff logic.
- Test distinct/mixed lists, singleton malformed records, missing/null/non-string fields, empty/whitespace-distinct identities, duplicate empty identities, identical/disjoint-body duplicates, empty lists, missing/non-sequence values, unused/ancestor registrations, and unchanged unregistered conflicts. Use a compact table of meaningful categories, not a full cross-product.
Validation: composition tests and ./scripts/check.sh. Completion: the private
composer returns the correct tree and range projection without aliasing inputs.
Completion evidence: added a policy-aware private composer while retaining the
unregistered adapter. Registered sequences validate every actual contribution,
including singleton and empty lists; distinct records append in source/list
order, duplicate exact string identities fail, and other overlap kinds retain
strict-conflict behavior. The result carries per-source sequence ranges for
later projection. Tests cover malformed identity records, within/across-list
duplicates including empty IDs and differing bodies, absent/non-sequence paths,
empty lists, unregistered conflicts, and input/result ownership. These checks
passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Compose' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 3 — Project composition origins and claims onto appended indexes
Scope: composition evidence, not import acquisition.
- Use Stage 2's assigned ranges to project each sequence record and descendant onto its actual effective index. Retain physical locations and contributor ordering for mapping and sequence containers. Preserve field-specific origins; do not attribute every descendant to the sequence's first source.
- Keep source-local claims useful for conflict reporting while effective origins and presence describe assembled paths. Projection must handle already-authored lists, mounted mapping prefixes, escaped/numeric keys, and more than one registered collection without using textual pointer replacement.
- Add composition-origin tests in which a later list's local index zero becomes an effective nonzero index. Mutate returned/copied evidence and verify input and separately retained paths remain independent.
Validation: composition/provenance/path tests and ./scripts/check.sh.
Completion: correct trees and effective origins share the same record ordering.
Completion evidence: composition origins now traverse each source document with
separate local and effective semantic paths. Stage 2's source ranges shift list
items and descendants to assigned indexes while retaining each field's physical
location and contributor order at sequence containers; conflict claims remain
source-local. Tests cover a later source's local index zero becoming effective
index one, mounted and escaped/numeric-key paths, multiple collections, and
detached origins, claims, ranges, and input trees. These checks passed with Go
go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Compose|Provenance|Path' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 4 — Preserve append history and whole-list tier replacement
Scope: private history operations, replay, and precedence handoff.
- Extend
historyApplicationand its cloning/query/replay helpers with explicit sequence-contribution information from the fixed design. Keep ordinary merge behavior as the default. Support semantic indexed prefixes for helper events; currentmergeHistoryAtPathsupports mappings only and cannot be reused unchanged for them. - At every strict-tier boundary mark the first contribution at a sequence path
as replacement and subsequent contributions as append. This boundary cannot
be inferred from
LayerUnderlayalone: different underlay groups share that layer name. Keep this information whenassemblyAccumulator.applyOverlayimports a group's history applications. - Append replay must affect the collection and newly added record paths, not falsely attribute unchanged earlier records to a later source. Translate historical queries to the incoming fragment's local indexes for location/value lookup. Preserve source-level applications rather than replacing them with one opaque completed-tier snapshot.
- Test before/after replay for append, explicit empty lists, independent lists, two underlay tiers with the same identity, main replacement, and a later whole list replacement/removal. Verify origin attribution and history agree with Stage 3's composed tree.
Validation: history/explanation/overlay tests and ./scripts/check.sh.
Completion: strict accumulation is distinguishable from precedence replacement.
Completion evidence: history applications now carry owned per-collection
contribution ranges and distinguish each strict tier's first sequence
contribution (replacement) from later appends. Replay preserves source
applications through underlay handoff, maps appended record queries back to
source-local indexes, and supports semantic indexed helper paths. Tests cover
appended and independent collections, origin attribution, detached paths,
explicit empty-list replacement, ordinary whole-list replacement, and tier
handoff. These checks passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'History|Explain|Overlay' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 5 — Integrate projected paths with inspection and sensitivity
Scope: metadata, retained history traversal, and safe inspection.
- Adapt
finalizeMetadataPaths, application-value traversal, explanation queries, and secret classification to Stage 4's projected operations. Do not coerce indexed application prefixes into mapping keys. Reuse a small common projected traversal/path helper where these consumers need the same translation. - Preserve independent effective and historical indexes, numeric key/index shape distinctions, removed paths, detached query results, and current sensitivity propagation. Classify original historical shapes before aggregating sensitivity by pointer; do not repeat the previously fixed shape-collision defect.
- Ensure sensitive fields on appended records are classified at their assigned indexes and stay safely handled after replacement. Apply the same projection to source-version sensitivity evidence when that evidence is integrated in Stage 10. Do not classify a fragment solely at its unshifted source index.
- Add focused tests for shifted-index metadata/history/rendering agreement and a replaced sensitive record. Reuse existing numeric-shape, removed-descendant, ownership, inheritance, and redaction cases where they already protect behavior.
Validation: metadata/provenance/history/explanation/secret tests,
go test -race ./internal/engine, and ./scripts/check.sh.
Completion: the private registered composer can feed existing inspection safely.
Completion evidence: history applications now expose one source-local/effective
path traversal used by metadata enumeration, explanation replay, origin lookup,
and secret classification. Appended records and nested descendants use assigned
indexes while retained nodes and source locations remain local. Secret
classification evaluates projected historical shapes before folding sensitivity
by pointer, preserving numeric key/index distinctions and redacting replaced
records. Tests cover shifted metadata/history/rendering, origin attribution, and
redaction after replacement. These checks passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Metadata|Provenance|History|Explain|Secret' -count=1GOTOOLCHAIN=local GOPROXY=off go test -race ./internal/engineGOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 6 — Refactor acquisition into composition scopes without new behavior
Scope: imports.go, load_input.go, load_request.go, and their handoffs.
- Replace the assumption that traversal yields only a flat slice of already placed documents with ordered contribution/scope records. At this stage all scopes are ordinary mappings; preserve current placement and traversal exactly.
- Separate acquisition facts from completed composition and retained history. Keep source-local payloads, physical locations, resource contexts, relative destinations, processing results, and source order available to later stages. Preserve input-specific filesystem contexts, tier-local repetition tracking, active import stacks, and load-wide shared accounting.
- Stop reconstructing underlay contributions by treating every history
application's
valueas a complete root document. That current shortcut loses projected paths and append semantics. Compose acquired contributions at the group boundary while preserving their source/history evidence. - Avoid recursive public
Loadcalls, reopening files, a second parse pass, per-helper defaults/hooks, or resetting import limits and active/visited state. Preserve acquisition-first main behavior and all close/error paths. - Prefer existing ordinary, relative-destination, underlay, namespace, repetition, budget, and handle-ownership tests as the regression oracle. Add a test only for a real handoff risk not already covered.
Validation: import/destination/grouped/source-version/resource-context tests and
./scripts/check.sh. Completion: this is a behavior-preserving private refactor.
Completion evidence: acquisition now retains ordered ordinary-mapping scopes
with source-local payloads, placed contributions, destinations, source records,
and source-version processing results. Underlay groups compose those acquired
contributions at their boundary instead of treating retained history values as
complete roots. Existing filesystem contexts, import traversal/repetition
state, load-wide accounting, hooks, and lifecycle handling remain in their
existing paths. These checks passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Destination|Import|Grouped|SourceVersion|Resource' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 7 — Add the private item-placement operation and accounting
Scope: placement primitives and engine per-input options; not public exposure.
- Add the private placement mode to engine input/request/options/source records, with preflight validation. Keep mapping placement unchanged. A valid item mode needs a nonempty mapping-key destination relative to its current mapping scope.
- Carry the destination-resolution state in engine source records so later scoped acquisition can distinguish exact containers from unresolved fallbacks.
- Extend placement to wrap one completed source mapping in a singleton sequence
at that destination. Preserve mapping-root input validation; do not accept raw
source sequence/scalar roots. A nil payload stays absent, and
{}contributes an actual empty item. Do not require a composition registration here. - Account for the sequence wrapper plus existing mapping wrappers and full mounted depth. Check limits before allocating oversized wrappers. Carry node accounting through completed scopes without charging acquired content twice.
- Test mapping default equivalence, explicit item placement, nested literal destinations, absent versus empty payloads, invalid/root placement, and exact versus exceeded private depth/node allowances. Use test-controlled limits.
Validation: placement/accounting/option tests and ./scripts/check.sh.
Completion: the private operation places a mapping once with correct ownership.
Completion evidence: added private placement modes to engine options, load
requests, and source records, with preflight rejection of unknown modes and
item placement at an empty destination. A completed mapping can now be wrapped
in one singleton sequence at its destination; nil payloads remain absent and
empty mappings remain real items. Wrapper accounting includes both destination
mapping nodes and the added sequence depth/node, with limits checked before
allocation. Focused tests cover mapping zero-value equivalence, nested paths,
empty and absent payloads, invalid modes/destinations, and exact/exceeded
allowances. These checks passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Destination|Import|Mounted|PlaceDocument|Options|Layered' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 8 — Assemble item scopes and YAML path/glob imports
Scope: directive parsing and scoped acquisition/composition.
- Parse the fixed
placementsyntax and report malformed values safely. Each declaration's default is mapping mode; item mode is never inherited. Apply the same mode independently to every glob match in existing sorted traversal order. - An item source opens a local mapping scope. Its body and ordinary helpers compose strictly within that mapping, with omitted/empty helper destinations at the item root and other destinations relative to the immediate importer. A helper can supply the identity; a duplicate identity field in the item still fails ordinary mapping composition.
- Complete child item scopes before wrapping them into their enclosing scope. Compose each scope once, preserving declaration/contribution order and source history. Explicit nested item imports create nested sequences; registrations do not descend through the enclosing record index, so repeated contributions to those unregistered nested sequences conflict atomically.
- Pass completed records and their relative history/origins to the parent composer. Assign outer ranges there, then project the retained item body and helper events through those ranges. Do not emit each ordinary helper as a separate record or validate its identity independently.
- Maintain source destination-resolution state as enclosing ranges are assigned. Before assignment, source snapshots use the explicitly marked fallback from the fixed design; after assignment, project the exact container. Preserve already-returned snapshots rather than updating them in place.
- Cover path and glob item imports, a helper-provided ID, relative and transitive helper destinations, mixed authored lists/items, optional unmatched imports, empty items, nested placement, default-mode independence, and physical paths relative to the declaring helper. Exercise native filesystem and host imports through existing shared-mechanism coverage rather than duplicating every case.
- If the new syntax is now reachable through ordinary public import loading,
document that implemented placement subset in
docs/config.mdin this unit; do not advertise still-private registration controls yet.
Validation: import/directive/glob/destination/composition tests and
./scripts/check.sh. Completion: private item imports construct the agreed
records and preserve physical acquisition evidence.
Completion evidence: import declarations now parse per-declaration placement,
and item imports compose their body and ordinary helpers in a local mapping
scope before wrapping the completed record. Nested item scopes, glob matches,
relative and transitive helper destinations, source origins/history, and
resolved underlay destinations are retained. Configuration documentation
describes only the implemented import-placement subset; private registration
controls remain unadvertised. These checks passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Import|Directive|Glob|Destination|Compose|NativeFilesystem' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 9 — Wire strict policy through the complete loading lifecycle
Scope: ordinary/grouped engine loading and every strict-tier entry point.
- Pass compiled registrations and placement mode through main acquisition and each underlay group. Validate an authored registered list in an ordinary one-source load without imports, overlays, or CLI; no composition shortcut may bypass it. Skip only genuinely absent contributions.
- Assemble and validate each complete strict tier before precedence can replace it. Reset identity maps between tiers and preserve current acquisition order. A malformed lower-tier list must fail even when main or overlay would replace it. Avoid running the checker a second time on the overlaid final tree.
- Keep unregistered item lists atomic. Selected overlays, CLI replacements, defaults, callbacks, and sibling inheritance retain their current behavior. Do not require the final list to retain unique or even present identities.
- Add compact engine lifecycle cases for main, an underlay group, separate tiers
sharing IDs,
[]clearing, unregistered conflict, a hidden lower-tier error, and duplicate IDs introduced by a replacement overlay/CLI/callback. Verify normal application validation can reject such final values when configured.
Validation: load/grouped/overlay/override/resolution/default tests and
./scripts/check.sh. Completion: private engine policy is applied at strict
composition boundaries only, including the one-source case.
Completion evidence: compiled sequence policy now reaches ordinary input
composition, imported scopes, and each underlay group. Registered sequences
are checked even for one-source loads and before tier replacement; identity
state resets at tier boundaries, while replacement overlays, CLI values, and
recorded resolver results retain their existing behavior. Lifecycle tests cover
grouped composition, empty-list replacement, unregistered conflicts, malformed
lower tiers hidden by main or overlay replacement, duplicate replacements, and
application validation. These checks passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SequenceRegistration|Import|Grouped|Overlay|Override|Resolution|Default' -count=1GOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 10 — Preserve source-version and structural ownership under placement
Scope: version path projection and existing protected-path checks.
- Keep source-version checking on each original source before placement, exactly once using the current local rule and callback lifecycle. Required/if-present modes retain main absence and helper-document behavior. Accepted non-main headers are consumed before helper/record identity validation; main's header stays application content. Helpers do not inherit their parent's header.
- Default mode continues checking the configured effective path without prefix reinterpretation. Enabled modes derive main's protected effective path from its actual projection, including item indexes. Update both the request and version state's effective rule before later protection checks need it.
- Audit
effectiveSourceVersionRule,fragmentVersionClaim, import/underlay/ overlay checks, override checks, inheritance checks, and resolution auditing. Mapping-only navigation or ancestor-sequence shortcuts cannot protect the wrong record or reject an unrelated strict peer merely because it appends to the same collection. Preserve protections of the actual main path and its destructive ancestor replacements; do not add an exception to version policy. - Retain source-local public VersionCheck paths, physical locations, full absent paths, and conservative failure disclosure. Use assigned projections for sensitivity checks of headers consumed before the final tree exists. Keep original semantic key/index shapes and remap nested item/helper prefixes once.
- Keep import opt-in checks on original sources before consumed headers can hide a directive. Identity lookup must not count consumed headers or structural controls as application fields. Do not add a blacklist of identity-field names or use later defaults/inheritance/resolution to supply an absent identity.
- Test main item placement in both entry points, default versus enabled version coordinates, consumed helper/import headers, protected-main versus unrelated peer paths, safe shifted evidence, optional absence below controls, and normal inheritance inside item mappings. Reuse existing source-version and inheritance regressions rather than rewriting their selected contracts.
Validation: source-version/version/inheritance/override/resolution tests,
go test -race ./internal/engine, and ./scripts/check.sh.
Completion: item placement changes coordinates, not version or structural policy.
Completion evidence: enabled source-version rules now derive their effective
paths through the main item's assigned index, while default-mode coordinates
remain unchanged. Structural import and underlay checks use assigned sequence
ranges so they protect the actual main item without rejecting unrelated peers;
source-local checking still runs once per original source, and projected header
evidence is used for sensitivity. Tests cover ordinary and layered placement,
protected versus peer indexes, nested projections, and redacted version
evidence. These checks passed with Go go1.26.9 linux/amd64:
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SourceVersion|Version|Inheritance|Override|Resolution' -count=1GOTOOLCHAIN=local GOPROXY=off go test -race ./internal/engineGOTOOLCHAIN=local ./scripts/check.shgit diff --check
No environmental limitation affected validation; checks ran on Linux/amd64.
Stage 11 — Expose root-owned controls and detached placement evidence
Scope: root facade declarations, conversions, and public integration tests.
- Add the exact public types/fields specified above, with declaration comments
describing defaults, option failures, and ownership. Convert Definition
registrations and paths, ordinary Options destinations/modes, layered Inputs,
and successful/failed SourceInfo records in both directions as applicable.
Convert
DestinationUnresolvedalongsideDestinationandPlacementMode. Update callback-entry source conversions too, not onlyResult.Sources. - Preserve immutable source snapshots, reader single-use behavior, callback lifetime, independent loads, and no public exposure of engine/parser types. SourceInfo reports its own selected mode; CLI/overlay sources use mapping mode unless an actual supported item placement was applied.
- For a completed load, resolve nested source containers through their owning
item projection. Apply the fixed unresolved-destination contract to callbacks
and early failures: a fallback container always has
DestinationUnresolvedset to true; an exact container has false. Failed-load records describe acquisition context, not a usable partial configuration. Do not add a public index or identity-address API or promise resolution of every failed source's final placement. - Test an unresolved helper record exposed during a callback and on early failure, its exact projected destination after assignment, and a failure after assignment that retains the exact destination. Distinguish an exact item-root container from a helper fallback with the same path. Verify nested-item fallback, preserved snapshot ownership, and false for unchanged mapping-only sources. Reuse fixtures where these scenarios share the same lifecycle path.
- Public tests demonstrate ordinary and layered opt-in, an item-import glob, authored-list/item interoperability, one-source validation, helper origins and resource contexts at shifted indexes, both option-error entry points, detached source/registration paths, and unchanged zero-value callers. Keep mechanism edge cases in engine tests.
- Update
docs/config.mdfor the now-public registration controls, ordinary destination options, complete placement/evidence behavior, and strict-versus- replacement validation boundary. DocumentDestinationUnresolvedand its fallback meaning in bothSourceInfocomments and the owning configuration contract, including callback and failure snapshots. Update the architecture policy's composition summary to recognize the registered-sequence rule and link ADR 0014. Stage 13 reviews and completes the surrounding guidance, not these canonical contracts.
Validation: focused facade tests,
go test . -run 'TestPublicFacadeBoundary|TestDependencyImportBoundary',
go test -race ./..., and ./scripts/check.sh.
Completion: both public entry points expose the complete feature with the same
composition rules and independently owned evidence.
Stage 12 — Close cross-cutting regression and limit gaps
Scope: risks that require the assembled feature, not duplicate mechanism tests.
- Review Stage 1–11 coverage against the roadmap's ten settled decisions. Add only missing consequential cases. Use a representative completed load to check typed output, explicit zero/false/null/empty versus omission, effective origins, callback-entry metadata, resource contexts, history, explanation, and rendering agree after an append followed by a replacement.
- Verify item/helper source canaries do not leak through duplicate diagnostics, errors, explanations, effective rendering, or consumed version evidence under existing sensitivity rules. Preserve historical redaction after index shifts and later removal/replacement, without imposing identity-based final masking rules that the existing model does not have.
- Exercise shared aggregate accounting across item scopes/globs/underlays and retained-history accounting of the new operations. Confirm immutable fragments are counted according to existing shared-node rules and no retained cumulative sequence snapshots multiply history storage. Use existing close/reader seams for one representative nested-item failure, not a second filesystem framework.
- Confirm effective Diff still reports atomic sequence changes and unchanged defaults/resolution/inheritance semantics. Review the implementation for repeated ID scans, growing-list clones, and helper-subtree recomposition. No timing or allocation threshold assertions, benchmark campaign, or new release gate is required; add local performance evidence only if a concrete concern needs it.
Validation: targeted new regressions, go test -race ./..., and
./scripts/check.sh. Completion: no known evidence, ownership, or accounting gap
remains hidden behind a correct final tree.
Stage 13 — Document implemented contracts and provide a maintained example
Scope: consumer documentation and a tested in-repository adoption fixture.
- Update
docs/config.mdas the canonical owner for import syntax, placement, registered strict composition, validation timing, replacement boundaries, structural/version processing, source containers, and history semantics. Link declaration comments for API details rather than duplicating them everywhere. - Update
docs/policy/architecture.md's composition summary so its unconditional non-mapping-conflict description recognizes the explicitly registered sequence rule. Preserve consistency, ownership, and precedence principles; this feature does not authorize arbitrary configurable merge strategies. Link the new ADR and current contract. Preserve upstream provenance and locally significant policy decisions. - Update
README.md,docs/consumers/README.md,docs/internal/overview.md, anddocs/development.mdonly to summarize/navigate the implemented feature at their respective ownership boundaries. Updatefuture.mdto distinguish the implemented subset from still-potential broader identified-record features; do not promote their deferred semantics into this feature's contract. - Add a maintained fixture/test under
examples/consumersshowing independently authored prompt-shaped records, explicit item path/glob imports, reusable registration, and a lower-priority default collection replaced by the main list. Show that a helper can supply an ID and that final domain validation is consumer-owned. Use in-repository disposable data, not a live PromptKit dependency or a second application-side merge. - Keep ADR 0014 decision text immutable after acceptance; update status/navigation to implemented when appropriate. Do not rewrite earlier ADR decision text to make it appear they anticipated the extension.
Validation: executable example/consumer tests, local documentation-link tests,
whitespace review, and ./scripts/check.sh.
Completion: current docs and tested public examples describe what actually lands.
Stage 14 — Review the completed unit and hand off
- Read the complete implementation diff against
2051177and the feature roadmap. Check all ten settled decisions and the two independent opt-ins. Look especially for validation accidentally extended into replacement-only layers, final uniqueness/schema promises, new field restrictions, wrong source indexes, and stale acquisition/history shortcuts. - Run
go test -race ./...and./scripts/check.shon the completed candidate. Reuse unchanged successful checks where appropriate; document actual results and any unresolved issue rather than claiming an unrun consumer verification. - Record stage completion and a concise final validation summary in this plan, then commit the completed nonempty unit under the normal commit policy. Do not create an empty completion commit or delete unfinished plans.
- Report the new public controls, preserved precedence/ownership behavior, validation, and any remaining work. This sprint does not select/tag/push a release, alter release gates, update Go/dependencies, or update downstream repositories. Separately authorized release preparation can assess the public field additions and feature under the existing release policy.
Completion: the repository is clean, the feature and its evidence agree, and no known in-scope implementation work remains. No product decisions remain open; private helper names and equivalent mechanics are ordinary implementation work.