Retire completed sequence composition plans

This commit is contained in:
2026-10-17 04:56:51 +00:00
parent 008d2ba9c8
commit ef28eabaf8
5 changed files with 15 additions and 1488 deletions

View File

@@ -4,8 +4,10 @@
Accepted and implemented. The [configuration contract](../config.md#strict-document-composition) Accepted and implemented. The [configuration contract](../config.md#strict-document-composition)
defines current behavior, and the [consumer guide](../consumers/README.md#identified-prompt-sequences) defines current behavior, and the [consumer guide](../consumers/README.md#identified-prompt-sequences)
shows the maintained example. Implementation status and delivery sequencing shows the maintained example. The completed roadmap, implementation plan, and
are tracked in the [implementation plan](../roadmap/implementation.md). resolved remediation record are preserved in Git history at `008d2ba` under
`docs/roadmap/`; see the [development guide](../development.md) for retrieval
commands.
## Date ## Date

View File

@@ -137,6 +137,17 @@ the [strict composition](config.md#strict-document-composition) and
[maintained prompt catalog](consumers/README.md#identified-prompt-sequences) [maintained prompt catalog](consumers/README.md#identified-prompt-sequences)
shows the current consumer boundary, and [ADR 0014](adr/0014-sequence-placement-and-composition.md) shows the current consumer boundary, and [ADR 0014](adr/0014-sequence-placement-and-composition.md)
records the decision. records the decision.
The completed sequence-composition roadmap, implementation plan, and resolved
remediation record are preserved in Git history at `008d2ba` under
`docs/roadmap/`. For example:
```sh
git show 008d2ba:docs/roadmap/sequence-composition.md
git show 008d2ba:docs/roadmap/implementation.md
git show 008d2ba:docs/roadmap/remediation.md
```
Potential enhancements are tracked in [future features](roadmap/future.md). Potential enhancements are tracked in [future features](roadmap/future.md).
Install the Go version required by `go.mod` or a compatible newer toolchain. Install the Go version required by `go.mod` or a compatible newer toolchain.

View File

@@ -1,815 +0,0 @@
# Implementation Plan: Sequence Placement and Composition
Status: implementation complete; all 14 stages completed.
## Purpose and authority
Implement the accepted [feature roadmap](sequence-composition.md), starting from
`2051177` (`Settle sequence placement and composition roadmap decisions`). This
plan owns execution order and progress; the roadmap owns intent, scope, and the
ten settled behavior decisions. The target implementer is a Luna coding agent.
Complete the numbered stages in order. Each stage is a separate focused prompt
and validated commit, not a request to implement the whole feature at once.
Read [development](../development.md), [agents](../policy/agents.md),
[architecture](../policy/architecture.md), [testing](../policy/testing.md), and
[documentation](../policy/documentation.md) before starting. Use the code graph
for discovery, then inspect the current owning implementation and tests. Preserve
unrelated work. Follow the existing commit policy after each nonempty completed
unit; record completion and validation in this plan.
This planning change implements nothing. Later stages introduce private engine
capabilities before exposing them through the root facade. Until a capability
lands, current consumer documentation must continue to describe implemented
behavior only. Accepted ADRs may describe the decision before implementation.
When a stage makes behavior reachable by consumers, update its owning declaration
comments and configuration contract in that same unit, to the implemented
boundary. Stage 13 consolidates guidance and examples; it does not postpone those
contract updates.
## Fixed design for implementation
### Public API and YAML spelling
Use these root-owned declarations and fields. Their zero values preserve current
behavior; do not change the signatures of `Load` or `LoadLayered`.
```go
type PlacementMode uint8
const (
PlacementModeMapping PlacementMode = iota
PlacementModeSequenceItem
)
type SequenceComposition struct {
Path Path
IdentityField string
}
```
- Add `SequenceCompositions []SequenceComposition` to `Definition[T]`.
- Add `PlacementMode PlacementMode` to `Input` and `SourceInfo`.
- Add `DestinationUnresolved bool` to `SourceInfo`. False preserves the existing
exact-destination interpretation; true explicitly marks the enclosing-container
fallback described below. This is output evidence, not an input option.
- Add both `Destination Path` and `PlacementMode PlacementMode` to `Options[T]`.
Ordinary `Load` needs the same per-source controls as the layered entry point;
its overlay and CLI coordinates remain application-rooted.
- YAML import declarations accept `placement: mapping` or
`placement: sequence-item`. Omission means `mapping`, independently for every
declaration. Retain `destination: [literal, keys]`, `path`, `glob`, and
`optional` with their existing meanings. Do not add aliases or inferred modes.
- Root fields and types stay root-owned. Use explicit conversions into engine
records and a private engine mode, following the source-version-mode pattern.
Preserve unknown numeric modes until engine preflight rejects them.
Validate options before acquisition. Unknown placement modes, item placement at
an empty destination, invalid destinations/registration paths, and conflicting
registrations use `CodeOptionsInvalid`. Registration paths use existing literal
mapping-key destination normalization, including its current reserved-key rules;
do not add more reserved names. Parsed numeric tokens become literal keys;
explicit indexes and invalid paths fail. Identical registrations coalesce, unused
registrations are permitted, and an empty/root registration is harmless and
unused under the mapping-root model. Do not reject ancestor/descendant pairs.
`IdentityField` is one literal string, including an empty string, not a path.
No field-name or identity-value normalization is introduced.
Use `CodeImportInvalid` for malformed import placement syntax, including an item
import whose effective destination is empty in its current mapping scope. An
omitted item-import destination is valid when inherited placement already names
a non-root collection. At an item root, an omitted destination names the item
mapping itself and cannot turn that mapping into a sequence. A nested item import
needs a nonempty effective destination within that item scope; this can come from
its relative destination or an ordinary helper's existing placement prefix.
Use `CodeComposeInvalid` for malformed registered-sequence records/identities and
`CodeComposeConflict` for duplicate identities or ordinary strict conflicts.
Malformed-record paths identify the effective item or its identity field;
duplicate paths identify the sequence container, with the first and conflicting
physical locations in `Primary`/`Related`. Error messages must not interpolate
identity strings or record bodies. Preserve deterministic reporting without
inventing an all-errors guarantee or freezing incidental message wording.
### Composition, scopes, and evidence
1. Compile registrations once per load into an owned exact-path lookup. A strict
tier has a separate identity table for each encountered registered sequence.
Scan every actual sequence contribution, even one with no peer. Require map
items and a present string identity after source-header processing and helper
composition. Compare exact strings; accept empty strings. Non-sequences retain
existing behavior. Distinct records append in contribution order; repeated
identities conflict even when bodies are equal or disjoint.
2. Keep every precedence boundary atomic for sequences. Each underlay group and
the main tier starts new identity state. Selected overlays, CLI, defaults,
inheritance, and application resolution do not run this identity validator.
Registrations apply at effective literal mapping paths, never through record
indexes. They are composition policy, not final configuration schema.
3. Source acquisition remains depth-first and keeps current source IDs, physical
resource identity, cycle/repetition detection, and shared budgets. Model an
item source plus its ordinary helper imports as a local strict mapping scope.
Ordinary imports share that scope and append relative mapping destinations;
an explicit item import opens a child item scope. Complete each item scope
before wrapping/validating its record in the enclosing composition. Keep its
contribution at the declaring source's original traversal position.
4. Carry a small private placement projection with each contribution: its scope,
local destination, and any sequence range assigned by strict assembly. Use it
to translate source paths into effective paths without reordering sources or
mutating acquired trees. Authored sequence index `j` becomes `offset+j`;
an item source's local root becomes the assigned item index. Nested projections
compose. Do not locate records again by rescanning IDs during inspection.
5. Keep completed trees distinct from source/history applications. Extend private
history operations to distinguish a sequence contribution from recursive
mapping merge: the first contribution to a collection in a tier replaces the
previous tier/default value; later contributions append. An empty first list
clears the prior list; an empty later list adds no items. Store only incoming
fragments and offsets, not a cumulative list snapshot after every append.
For item scopes, retain the initial item body and helper contributions at their
projected item paths, so helpers retain their own history and origins.
6. Reuse that projection in origins, history replay/query relevance, historical
path enumeration, sensitivity, and version evidence. Public source records
describe containers, not a new permanent record-identity address. An item
source reports its sequence container; a mapping helper inside an item reports
its effective mapping destination, which can include an enclosing item index.
Output evidence may contain such indexes even though input destinations and
registrations accept only mapping-key segments.
### Unresolved destination evidence
`SourceInfo.DestinationUnresolved` describes whether that snapshot's `Destination`
is the source's exact effective destination container:
- When false, `Destination` is exact. This says nothing about whether the load
succeeded or the source's values survived later replacement. An item source
can have an exact sequence-container destination before its own item index is
assigned, because that index is not part of its reported container path.
- When true, an enclosing item index needed for the destination has not been
assigned. `Destination` is only the nearest resolved enclosing collection
container, not the source's actual mapping destination or nested sequence
destination. Do not append the unresolved local suffix as though the missing
index were a mapping key, or use a fabricated index.
- For example, a mapping helper destined for `prompts[1].metadata` reports the
container `prompts` with `DestinationUnresolved: true` until the outer item
index is assigned. Later snapshots report `prompts[1].metadata` with false.
The item-root source itself reports `prompts` with false when that container
is known. The placement mode continues to describe each source's own mode.
- Apply this distinction to every source-record exposure, including source
records available during callbacks and failed-load records. Once projection
resolves the container, subsequent snapshots report the exact path with false.
Previously returned detached snapshots retain their original path and flag.
Completed successful loads report resolved containers; early failures may
contain both resolved and unresolved source records.
This flag makes the existing planned fallback observable without adding a public
item-index API, promising permanent indexes, or reconstructing unknown placement
after a failure. Physical source locations, source IDs, and source-local version
evidence retain their existing meaning regardless of the flag.
Choose private helper names and file splits to fit existing responsibilities.
Do not introduce a provider framework, generic merge callbacks, a second loading
lifecycle, or a public identity-selector API to implement these mechanics.
### Efficiency and preserved limits
Use one string-keyed identity map per registered sequence per strict tier, with
ordered appends and existing indexed mapping lookup. Clone incoming records into
owned assembly storage without repeatedly cloning the accumulated sequence.
Registration lookup, validation, and sequence construction should be linear in
the visited data apart from existing path/depth costs. Complete each item scope
once; do not recursively recompose the entire ordinary-helper subtree at every
import edge. Keep retained history fragments immutable and independently owned
from the working tree. These are implementation choices, not a new public
complexity, allocation, or latency guarantee.
Preserve current per-source, aggregate, traversal, projection, and retained-history
limits and error families. Count acquired content once and synthetic mapping and
sequence wrappers where introduced; include the extra sequence level in depth
checks. Do not charge an already-accounted item/helper body again merely because
it moves to a completed scope. Keep library-owned handles closed before returning
on success or failure. Do not add new limits or change their existing values.
## Stage 1 — Record the decision and prepare private registration controls
Scope: one ADR and engine option normalization; no new public fields yet.
- Write `docs/adr/0014-sequence-placement-and-composition.md` as an Accepted ADR
using the established format. Explain explicit item placement and registered
duplicate-rejecting strict composition, unchanged precedence replacement, and
their rationale in ConfigKit terms. Discuss alternatives only in Alternatives
Considered; do not center a downstream application's semantics. Link the
roadmap for implementation status. Do not rewrite accepted earlier decisions.
- Add engine-owned registration records/state and normalization, reachable from
engine `Definition`. Validate/coalesce using the fixed design above; detach
registration slices and paths at the load boundary. Keep compiled policy
per-load, not mutable shared Definition state or a global cache.
- Wire preflight into both entry points without consuming a reader or opening a
source for invalid options. Thread compiled policy through the request rather
than rebuilding it or invoking application hooks for internal handoffs. Keep
the existing shared source-version checker lifecycle intact.
- Add focused normalization tests for duplicate/conflicting registrations, empty
identity-field names, numeric/escaped literal keys, invalid index paths,
unused/root registrations, and preserved zero-option behavior.
Validation: focused option tests and `./scripts/check.sh`. Completion: existing
loads remain unchanged, with only private registered-policy input newly accepted.
Completion evidence: added Accepted ADR 0014 and private engine registration
normalization. Ordinary and layered entry points compile/coalesce policy and
reject invalid registrations before source reads, then carry the compiled policy
through the load request while clearing raw registration slices from internal
handoffs. No public API or composition behavior was added. Focused coverage
checks duplicate/conflicting definitions, empty identity names, numeric and
escaped literal keys, invalid explicit indexes, unused/root registrations,
zero-option behavior, and reader preservation on preflight failure. These checks
passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SequenceRegistration' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
The compiled registration policy is intentionally not consumed by composition
until the next implementation unit. No environmental limitation affected
validation; checks ran on Linux/amd64.
## Stage 2 — Implement registered strict sequence validation and assembly
Scope: `internal/engine/compose.go` and focused composition tests.
- Add a policy-aware internal composer while preserving an unregistered adapter
for existing callers/tests. Collect/check ordinary claims with current strict
semantics; only sequence/sequence claims at registered effective paths use the
new rule. Mapping/sequence, null/sequence, and scalar overlaps still conflict.
- Validate every registered sequence as it is encountered, not only overlapping
claims. Missing paths and non-sequences do not independently fail registration.
Validate the completed contribution, not the unfinished raw item source.
- Build identity tables once, reject duplicates within/across lists, and append
records into owned assembly storage. Preserve mapping traversal and list order,
explicit empty sequences, exact identity spelling, and existing error safety.
- Return the assigned source-to-effective sequence ranges alongside the composed
result for the following stages. Do not change overlay merging or diff logic.
- Test distinct/mixed lists, singleton malformed records, missing/null/non-string
fields, empty/whitespace-distinct identities, duplicate empty identities,
identical/disjoint-body duplicates, empty lists, missing/non-sequence values,
unused/ancestor registrations, and unchanged unregistered conflicts. Use a
compact table of meaningful categories, not a full cross-product.
Validation: composition tests and `./scripts/check.sh`. Completion: the private
composer returns the correct tree and range projection without aliasing inputs.
Completion evidence: added a policy-aware private composer while retaining the
unregistered adapter. Registered sequences validate every actual contribution,
including singleton and empty lists; distinct records append in source/list
order, duplicate exact string identities fail, and other overlap kinds retain
strict-conflict behavior. The result carries per-source sequence ranges for
later projection. Tests cover malformed identity records, within/across-list
duplicates including empty IDs and differing bodies, absent/non-sequence paths,
empty lists, unregistered conflicts, and input/result ownership. These checks
passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Compose' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 3 — Project composition origins and claims onto appended indexes
Scope: composition evidence, not import acquisition.
- Use Stage 2's assigned ranges to project each sequence record and descendant
onto its actual effective index. Retain physical locations and contributor
ordering for mapping and sequence containers. Preserve field-specific origins;
do not attribute every descendant to the sequence's first source.
- Keep source-local claims useful for conflict reporting while effective origins
and presence describe assembled paths. Projection must handle already-authored
lists, mounted mapping prefixes, escaped/numeric keys, and more than one
registered collection without using textual pointer replacement.
- Add composition-origin tests in which a later list's local index zero becomes
an effective nonzero index. Mutate returned/copied evidence and verify input
and separately retained paths remain independent.
Validation: composition/provenance/path tests and `./scripts/check.sh`.
Completion: correct trees and effective origins share the same record ordering.
Completion evidence: composition origins now traverse each source document with
separate local and effective semantic paths. Stage 2's source ranges shift list
items and descendants to assigned indexes while retaining each field's physical
location and contributor order at sequence containers; conflict claims remain
source-local. Tests cover a later source's local index zero becoming effective
index one, mounted and escaped/numeric-key paths, multiple collections, and
detached origins, claims, ranges, and input trees. These checks passed with Go
`go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Compose|Provenance|Path' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 4 — Preserve append history and whole-list tier replacement
Scope: private history operations, replay, and precedence handoff.
- Extend `historyApplication` and its cloning/query/replay helpers with explicit
sequence-contribution information from the fixed design. Keep ordinary merge
behavior as the default. Support semantic indexed prefixes for helper events;
current `mergeHistoryAtPath` supports mappings only and cannot be reused
unchanged for them.
- At every strict-tier boundary mark the first contribution at a sequence path
as replacement and subsequent contributions as append. This boundary cannot
be inferred from `LayerUnderlay` alone: different underlay groups share that
layer name. Keep this information when `assemblyAccumulator.applyOverlay`
imports a group's history applications.
- Append replay must affect the collection and newly added record paths, not
falsely attribute unchanged earlier records to a later source. Translate
historical queries to the incoming fragment's local indexes for location/value
lookup. Preserve source-level applications rather than replacing them with one
opaque completed-tier snapshot.
- Test before/after replay for append, explicit empty lists, independent lists,
two underlay tiers with the same identity, main replacement, and a later whole
list replacement/removal. Verify origin attribution and history agree with
Stage 3's composed tree.
Validation: history/explanation/overlay tests and `./scripts/check.sh`.
Completion: strict accumulation is distinguishable from precedence replacement.
Completion evidence: history applications now carry owned per-collection
contribution ranges and distinguish each strict tier's first sequence
contribution (replacement) from later appends. Replay preserves source
applications through underlay handoff, maps appended record queries back to
source-local indexes, and supports semantic indexed helper paths. Tests cover
appended and independent collections, origin attribution, detached paths,
explicit empty-list replacement, ordinary whole-list replacement, and tier
handoff. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'History|Explain|Overlay' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 5 — Integrate projected paths with inspection and sensitivity
Scope: metadata, retained history traversal, and safe inspection.
- Adapt `finalizeMetadataPaths`, application-value traversal, explanation queries,
and secret classification to Stage 4's projected operations. Do not coerce
indexed application prefixes into mapping keys. Reuse a small common projected
traversal/path helper where these consumers need the same translation.
- Preserve independent effective and historical indexes, numeric key/index shape
distinctions, removed paths, detached query results, and current sensitivity
propagation. Classify original historical shapes before aggregating sensitivity
by pointer; do not repeat the previously fixed shape-collision defect.
- Ensure sensitive fields on appended records are classified at their assigned
indexes and stay safely handled after replacement. Apply the same projection
to source-version sensitivity evidence when that evidence is integrated in
Stage 10. Do not classify a fragment solely at its unshifted source index.
- Add focused tests for shifted-index metadata/history/rendering agreement and
a replaced sensitive record. Reuse existing numeric-shape, removed-descendant,
ownership, inheritance, and redaction cases where they already protect behavior.
Validation: metadata/provenance/history/explanation/secret tests,
`go test -race ./internal/engine`, and `./scripts/check.sh`.
Completion: the private registered composer can feed existing inspection safely.
Completion evidence: history applications now expose one source-local/effective
path traversal used by metadata enumeration, explanation replay, origin lookup,
and secret classification. Appended records and nested descendants use assigned
indexes while retained nodes and source locations remain local. Secret
classification evaluates projected historical shapes before folding sensitivity
by pointer, preserving numeric key/index distinctions and redacting replaced
records. Tests cover shifted metadata/history/rendering, origin attribution, and
redaction after replacement. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Metadata|Provenance|History|Explain|Secret' -count=1`
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./internal/engine`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 6 — Refactor acquisition into composition scopes without new behavior
Scope: `imports.go`, `load_input.go`, `load_request.go`, and their handoffs.
- Replace the assumption that traversal yields only a flat slice of already
placed documents with ordered contribution/scope records. At this stage all
scopes are ordinary mappings; preserve current placement and traversal exactly.
- Separate acquisition facts from completed composition and retained history.
Keep source-local payloads, physical locations, resource contexts, relative
destinations, processing results, and source order available to later stages.
Preserve input-specific filesystem contexts, tier-local repetition tracking,
active import stacks, and load-wide shared accounting.
- Stop reconstructing underlay contributions by treating every history
application's `value` as a complete root document. That current shortcut loses
projected paths and append semantics. Compose acquired contributions at the
group boundary while preserving their source/history evidence.
- Avoid recursive public `Load` calls, reopening files, a second parse pass,
per-helper defaults/hooks, or resetting import limits and active/visited state.
Preserve acquisition-first main behavior and all close/error paths.
- Prefer existing ordinary, relative-destination, underlay, namespace, repetition,
budget, and handle-ownership tests as the regression oracle. Add a test only for
a real handoff risk not already covered.
Validation: import/destination/grouped/source-version/resource-context tests and
`./scripts/check.sh`. Completion: this is a behavior-preserving private refactor.
Completion evidence: acquisition now retains ordered ordinary-mapping scopes
with source-local payloads, placed contributions, destinations, source records,
and source-version processing results. Underlay groups compose those acquired
contributions at their boundary instead of treating retained history values as
complete roots. Existing filesystem contexts, import traversal/repetition
state, load-wide accounting, hooks, and lifecycle handling remain in their
existing paths. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Destination|Import|Grouped|SourceVersion|Resource' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 7 — Add the private item-placement operation and accounting
Scope: placement primitives and engine per-input options; not public exposure.
- Add the private placement mode to engine input/request/options/source records,
with preflight validation. Keep mapping placement unchanged. A valid item mode
needs a nonempty mapping-key destination relative to its current mapping scope.
- Carry the destination-resolution state in engine source records so later
scoped acquisition can distinguish exact containers from unresolved fallbacks.
- Extend placement to wrap one completed source mapping in a singleton sequence
at that destination. Preserve mapping-root input validation; do not accept raw
source sequence/scalar roots. A nil payload stays absent, and `{}` contributes
an actual empty item. Do not require a composition registration here.
- Account for the sequence wrapper plus existing mapping wrappers and full
mounted depth. Check limits before allocating oversized wrappers. Carry node
accounting through completed scopes without charging acquired content twice.
- Test mapping default equivalence, explicit item placement, nested literal
destinations, absent versus empty payloads, invalid/root placement, and exact
versus exceeded private depth/node allowances. Use test-controlled limits.
Validation: placement/accounting/option tests and `./scripts/check.sh`.
Completion: the private operation places a mapping once with correct ownership.
Completion evidence: added private placement modes to engine options, load
requests, and source records, with preflight rejection of unknown modes and
item placement at an empty destination. A completed mapping can now be wrapped
in one singleton sequence at its destination; nil payloads remain absent and
empty mappings remain real items. Wrapper accounting includes both destination
mapping nodes and the added sequence depth/node, with limits checked before
allocation. Focused tests cover mapping zero-value equivalence, nested paths,
empty and absent payloads, invalid modes/destinations, and exact/exceeded
allowances. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Destination|Import|Mounted|PlaceDocument|Options|Layered' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 8 — Assemble item scopes and YAML path/glob imports
Scope: directive parsing and scoped acquisition/composition.
- Parse the fixed `placement` syntax and report malformed values safely. Each
declaration's default is mapping mode; item mode is never inherited. Apply the
same mode independently to every glob match in existing sorted traversal order.
- An item source opens a local mapping scope. Its body and ordinary helpers
compose strictly within that mapping, with omitted/empty helper destinations
at the item root and other destinations relative to the immediate importer.
A helper can supply the identity; a duplicate identity field in the item still
fails ordinary mapping composition.
- Complete child item scopes before wrapping them into their enclosing scope.
Compose each scope once, preserving declaration/contribution order and source
history. Explicit nested item imports create nested sequences; registrations
do not descend through the enclosing record index, so repeated contributions
to those unregistered nested sequences conflict atomically.
- Pass completed records and their relative history/origins to the parent
composer. Assign outer ranges there, then project the retained item body and
helper events through those ranges. Do not emit each ordinary helper as a
separate record or validate its identity independently.
- Maintain source destination-resolution state as enclosing ranges are assigned.
Before assignment, source snapshots use the explicitly marked fallback from
the fixed design; after assignment, project the exact container. Preserve
already-returned snapshots rather than updating them in place.
- Cover path and glob item imports, a helper-provided ID, relative and transitive
helper destinations, mixed authored lists/items, optional unmatched imports,
empty items, nested placement, default-mode independence, and physical paths
relative to the declaring helper. Exercise native filesystem and host imports
through existing shared-mechanism coverage rather than duplicating every case.
- If the new syntax is now reachable through ordinary public import loading,
document that implemented placement subset in `docs/config.md` in this unit;
do not advertise still-private registration controls yet.
Validation: import/directive/glob/destination/composition tests and
`./scripts/check.sh`. Completion: private item imports construct the agreed
records and preserve physical acquisition evidence.
Completion evidence: import declarations now parse per-declaration placement,
and item imports compose their body and ordinary helpers in a local mapping
scope before wrapping the completed record. Nested item scopes, glob matches,
relative and transitive helper destinations, source origins/history, and
resolved underlay destinations are retained. Configuration documentation
describes only the implemented import-placement subset; private registration
controls remain unadvertised. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Import|Directive|Glob|Destination|Compose|NativeFilesystem' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 9 — Wire strict policy through the complete loading lifecycle
Scope: ordinary/grouped engine loading and every strict-tier entry point.
- Pass compiled registrations and placement mode through main acquisition and
each underlay group. Validate an authored registered list in an ordinary
one-source load without imports, overlays, or CLI; no composition shortcut may
bypass it. Skip only genuinely absent contributions.
- Assemble and validate each complete strict tier before precedence can replace
it. Reset identity maps between tiers and preserve current acquisition order.
A malformed lower-tier list must fail even when main or overlay would replace
it. Avoid running the checker a second time on the overlaid final tree.
- Keep unregistered item lists atomic. Selected overlays, CLI replacements,
defaults, callbacks, and sibling inheritance retain their current behavior.
Do not require the final list to retain unique or even present identities.
- Add compact engine lifecycle cases for main, an underlay group, separate tiers
sharing IDs, `[]` clearing, unregistered conflict, a hidden lower-tier error,
and duplicate IDs introduced by a replacement overlay/CLI/callback. Verify
normal application validation can reject such final values when configured.
Validation: load/grouped/overlay/override/resolution/default tests and
`./scripts/check.sh`. Completion: private engine policy is applied at strict
composition boundaries only, including the one-source case.
Completion evidence: compiled sequence policy now reaches ordinary input
composition, imported scopes, and each underlay group. Registered sequences
are checked even for one-source loads and before tier replacement; identity
state resets at tier boundaries, while replacement overlays, CLI values, and
recorded resolver results retain their existing behavior. Lifecycle tests cover
grouped composition, empty-list replacement, unregistered conflicts, malformed
lower tiers hidden by main or overlay replacement, duplicate replacements, and
application validation. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SequenceRegistration|Import|Grouped|Overlay|Override|Resolution|Default' -count=1`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 10 — Preserve source-version and structural ownership under placement
Scope: version path projection and existing protected-path checks.
- Keep source-version checking on each original source before placement, exactly
once using the current local rule and callback lifecycle. Required/if-present
modes retain main absence and helper-document behavior. Accepted non-main
headers are consumed before helper/record identity validation; main's header
stays application content. Helpers do not inherit their parent's header.
- Default mode continues checking the configured effective path without prefix
reinterpretation. Enabled modes derive main's protected effective path from
its actual projection, including item indexes. Update both the request and
version state's effective rule before later protection checks need it.
- Audit `effectiveSourceVersionRule`, `fragmentVersionClaim`, import/underlay/
overlay checks, override checks, inheritance checks, and resolution auditing.
Mapping-only navigation or ancestor-sequence shortcuts cannot protect the
wrong record or reject an unrelated strict peer merely because it appends to
the same collection. Preserve protections of the actual main path and its
destructive ancestor replacements; do not add an exception to version policy.
- Retain source-local public VersionCheck paths, physical locations, full absent
paths, and conservative failure disclosure. Use assigned projections for
sensitivity checks of headers consumed before the final tree exists. Keep
original semantic key/index shapes and remap nested item/helper prefixes once.
- Keep import opt-in checks on original sources before consumed headers can hide
a directive. Identity lookup must not count consumed headers or structural
controls as application fields. Do not add a blacklist of identity-field names
or use later defaults/inheritance/resolution to supply an absent identity.
- Test main item placement in both entry points, default versus enabled version
coordinates, consumed helper/import headers, protected-main versus unrelated
peer paths, safe shifted evidence, optional absence below controls, and normal
inheritance inside item mappings. Reuse existing source-version and inheritance
regressions rather than rewriting their selected contracts.
Validation: source-version/version/inheritance/override/resolution tests,
`go test -race ./internal/engine`, and `./scripts/check.sh`.
Completion: item placement changes coordinates, not version or structural policy.
Completion evidence: enabled source-version rules now derive their effective
paths through the main item's assigned index, while default-mode coordinates
remain unchanged. Structural import and underlay checks use assigned sequence
ranges so they protect the actual main item without rejecting unrelated peers;
source-local checking still runs once per original source, and projected header
evidence is used for sensitivity. Tests cover ordinary and layered placement,
protected versus peer indexes, nested projections, and redacted version
evidence. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SourceVersion|Version|Inheritance|Override|Resolution' -count=1`
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./internal/engine`
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 11 — Expose root-owned controls and detached placement evidence
Scope: root facade declarations, conversions, and public integration tests.
- Add the exact public types/fields specified above, with declaration comments
describing defaults, option failures, and ownership. Convert Definition
registrations and paths, ordinary Options destinations/modes, layered Inputs,
and successful/failed SourceInfo records in both directions as applicable.
Convert `DestinationUnresolved` alongside `Destination` and `PlacementMode`.
Update callback-entry source conversions too, not only `Result.Sources`.
- Preserve immutable source snapshots, reader single-use behavior, callback
lifetime, independent loads, and no public exposure of engine/parser types.
SourceInfo reports its own selected mode; CLI/overlay sources use mapping mode
unless an actual supported item placement was applied.
- For a completed load, resolve nested source containers through their owning
item projection. Apply the fixed unresolved-destination contract to callbacks
and early failures: a fallback container always has `DestinationUnresolved`
set to true; an exact container has false. Failed-load records describe
acquisition context, not a usable partial configuration. Do not add a public
index or identity-address API or promise resolution of every failed source's
final placement.
- Test an unresolved helper record exposed during a callback and on early
failure, its exact projected destination after assignment, and a failure after
assignment that retains the exact destination. Distinguish an exact item-root
container from a helper fallback with the same path. Verify nested-item
fallback, preserved snapshot ownership, and false for unchanged mapping-only
sources. Reuse fixtures where these scenarios share the same lifecycle path.
- Public tests demonstrate ordinary and layered opt-in, an item-import glob,
authored-list/item interoperability, one-source validation, helper origins and
resource contexts at shifted indexes, both option-error entry points, detached
source/registration paths, and unchanged zero-value callers. Keep mechanism
edge cases in engine tests.
- Update `docs/config.md` for the now-public registration controls, ordinary
destination options, complete placement/evidence behavior, and strict-versus-
replacement validation boundary. Document `DestinationUnresolved` and its
fallback meaning in both `SourceInfo` comments and the owning configuration
contract, including callback and failure snapshots. Update the architecture
policy's composition summary to recognize the registered-sequence rule and
link ADR 0014. Stage 13 reviews and completes the surrounding guidance, not
these canonical contracts.
Validation: focused facade tests,
`go test . -run 'TestPublicFacadeBoundary|TestDependencyImportBoundary'`,
`go test -race ./...`, and `./scripts/check.sh`.
Completion: both public entry points expose the complete feature with the same
composition rules and independently owned evidence.
Completion evidence: the root facade now exposes sequence registrations,
ordinary and layered placement controls, and detached placement-mode and
destination-resolution source evidence. Public integration coverage exercises
ordinary/layered placement, strict registration with authored and glob-imported
items, shifted origins and resource contexts, option preflight, early and
post-placement failure snapshots, nested fallback containers, registration and
snapshot ownership, and zero-value mapping behavior. The existing resolver
callback runs after item projection, so its source snapshots are exact; the
unresolved helper fallback is observable on early failure. No acquisition-time
callback exposes `SourceInfo`, and no additional callback lifecycle was added.
These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test . -run 'TestPublicFacadeBoundary|TestDependencyImportBoundary' -count=1`
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`
- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 12 — Close cross-cutting regression and limit gaps
Scope: risks that require the assembled feature, not duplicate mechanism tests.
- Review Stage 1–11 coverage against the roadmap's ten settled decisions. Add
only missing consequential cases. Use a representative completed load to check
typed output, explicit zero/false/null/empty versus omission, effective origins,
callback-entry metadata, resource contexts, history, explanation, and rendering
agree after an append followed by a replacement.
- Verify item/helper source canaries do not leak through duplicate diagnostics,
errors, explanations, effective rendering, or consumed version evidence under
existing sensitivity rules. Preserve historical redaction after index shifts
and later removal/replacement, without imposing identity-based final masking
rules that the existing model does not have.
- Exercise shared aggregate accounting across item scopes/globs/underlays and
retained-history accounting of the new operations. Confirm immutable fragments
are counted according to existing shared-node rules and no retained cumulative
sequence snapshots multiply history storage. Use existing close/reader seams
for one representative nested-item failure, not a second filesystem framework.
- Confirm effective Diff still reports atomic sequence changes and unchanged
defaults/resolution/inheritance semantics. Review the implementation for repeated
ID scans, growing-list clones, and helper-subtree recomposition. No timing or
allocation threshold assertions, benchmark campaign, or new release gate is
required; add local performance evidence only if a concrete concern needs it.
Validation: targeted new regressions, `go test -race ./...`, and
`./scripts/check.sh`. Completion: no known evidence, ownership, or accounting gap
remains hidden behind a correct final tree.
Completion evidence: cross-cutting regressions now cover a typed main-tier
sequence replacement after underlay item appends, including presence and
zero/null distinctions, callback metadata and resource context, final origins,
removed-item explanation, and historical redaction in explanation and effective
rendering. Additional tests verify shared aggregate node accounting across a
nested item glob and sibling contribution, native file closure after nested item
composition failure, and retained sequence-history applications storing source
fragments rather than cumulative snapshots. Existing sequence placement, strict
diagnostic, version-evidence, and rendering tests continue to cover the other
settled risks. These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`
- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 13 — Document implemented contracts and provide a maintained example
Scope: consumer documentation and a tested in-repository adoption fixture.
- Update `docs/config.md` as the canonical owner for import syntax, placement,
registered strict composition, validation timing, replacement boundaries,
structural/version processing, source containers, and history semantics. Link
declaration comments for API details rather than duplicating them everywhere.
- Update `docs/policy/architecture.md`'s composition summary so its unconditional
non-mapping-conflict description recognizes the explicitly registered sequence
rule. Preserve consistency, ownership, and precedence principles; this feature
does not authorize arbitrary configurable merge strategies. Link the new ADR
and current contract. Preserve upstream provenance and locally significant
policy decisions.
- Update `README.md`, `docs/consumers/README.md`, `docs/internal/overview.md`, and
`docs/development.md` only to summarize/navigate the implemented feature at
their respective ownership boundaries. Update `future.md` to distinguish the
implemented subset from still-potential broader identified-record features;
do not promote their deferred semantics into this feature's contract.
- Add a maintained fixture/test under `examples/consumers` showing independently
authored prompt-shaped records, explicit item path/glob imports, reusable
registration, and a lower-priority default collection replaced by the main
list. Show that a helper can supply an ID and that final domain validation is
consumer-owned. Use in-repository disposable data, not a live PromptKit
dependency or a second application-side merge.
- Keep ADR 0014 decision text immutable after acceptance; update status/navigation
to implemented when appropriate. Do not rewrite earlier ADR decision text to
make it appear they anticipated the extension.
Validation: executable example/consumer tests, local documentation-link tests,
whitespace review, and `./scripts/check.sh`.
Completion: current docs and tested public examples describe what actually lands.
Completion evidence: the canonical configuration contract and architecture
policy now describe placement, registered strict composition, version/header
boundaries, source containers, history, and unchanged precedence. README,
consumer, development, and internal guides link to those contracts and the
maintained prompt-catalog example. The example uses disposable in-repository
fixtures to demonstrate path/glob item imports, a helper-supplied identity,
strict composition, whole-sequence replacement of defaults, and consumer-owned
validation. The feature roadmap and ADR status point to the implemented contract
and delivery record; broader composite-identity behavior remains deferred.
These checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./examples/consumers`
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`
- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh`
- `git diff --check`
No environmental limitation affected validation; checks ran on Linux/amd64.
## Stage 14 — Review the completed unit and hand off
- Read the complete implementation diff against `2051177` and the feature
roadmap. Check all ten settled decisions and the two independent opt-ins.
Look especially for validation accidentally extended into replacement-only
layers, final uniqueness/schema promises, new field restrictions, wrong
source indexes, and stale acquisition/history shortcuts.
- Run `go test -race ./...` and `./scripts/check.sh` on the completed candidate.
Reuse unchanged successful checks where appropriate; document actual results
and any unresolved issue rather than claiming an unrun consumer verification.
- Record stage completion and a concise final validation summary in this plan,
then commit the completed nonempty unit under the normal commit policy.
Do not create an empty completion commit or delete unfinished plans.
- Report the new public controls, preserved precedence/ownership behavior,
validation, and any remaining work. This sprint does not select/tag/push a
release, alter release gates, update Go/dependencies, or update downstream
repositories. Separately authorized release preparation can assess the public
field additions and feature under the existing release policy.
Completion: the repository is clean, the feature and its evidence agree, and no
known in-scope implementation work remains. No product decisions remain open;
private helper names and equivalent mechanics are ordinary implementation work.
Completion evidence: final review covered the complete implementation diff from
baseline `2051177` against the settled behavior and public-contract boundaries.
It found one malformed-record diagnostic using a contribution-local rather than
assigned sequence index. The composer now reports the effective assigned item
path, with a regression for a malformed record in the second contribution at
`/records/1`. Review also confirmed registered identity checks remain limited to
strict composition, precedence still replaces complete sequences, both opt-ins
are explicit, and source ownership, physical locations, contexts, and detached
evidence remain intact. Final checks passed with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'ComposeDocuments(ReportsAssignedPathForMalformedRegisteredSequenceContribution|ValidatesRegisteredSequenceSingletons|RejectsRegisteredSequence)' -count=1`
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`
- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh`
- `git diff --check 2051177`
No environmental limitation affected validation; checks ran on Linux/amd64.

View File

@@ -1,353 +0,0 @@
# Sequence Composition Review Remediation
Status: complete; all nine findings addressed in separate validated commits.
Reviewed implementation `2051177..0858af1`
(`0858af1029d07d93a5aa2e4f26cbec61fcfb3a51`) against the
[feature roadmap](sequence-composition.md), [implementation plan](implementation.md),
and accepted [ADR 0014](../adr/0014-sequence-placement-and-composition.md).
The root-owned facade, explicit placement, definition-owned registrations,
item/helper scopes, and strict-versus-precedence distinction are implemented.
The findings below concern correctness and evidence at their boundaries, plus
targeted scaling and documentation gaps. They do not call for new features,
stricter final identity validation, broader version policy, or new release gates.
## 1. Project consumed version evidence through authored sequence appends — P1
- [x] Addressed and regression-tested.
Completion: retained sensitivity paths now project declarations and ancestors
through assigned authored-list ranges without changing source-local evidence.
Public regressions cover imports and grouped underlays, leaf/container masking,
and unclassified declarations. Focused source-version tests, the everyday check,
and `go test -race ./internal/engine` passed.
**Owners:** `projectAcquiredDestinations` / `projectSourceVersionEvidence` in
[imports.go](../../internal/engine/imports.go), and evidence classification in
[source_version.go](../../internal/engine/source_version.go) and
[secret.go](../../internal/engine/secret.go).
Version evidence is projected for explicitly item-placed sources and their
helpers, but not for an ordinary source containing an authored registered list.
Its retained source-local path can contain an index shifted by composition.
Concatenating only `evidence.destination` and that path classifies the wrong
effective record and can publish sensitive declaration text.
**Confirmed reproduction:** main contains `records: [{id: main, version: main}]`
and imports a mapping containing
`records: [{id: child, version: private-version-canary}]`. Register `/records`
by `id`, select required source-version checks at
`Keys("records").Index(0).Key("version")`, and classify
`/records/1/version` as sensitive. Loading succeeds and consumes the child's
header, but its successful `SourceInfo.VersionCheck` exposes the canary with
`Redacted: false`. Its record is at index 1; evidence was classified at index 0.
**Required change:** project internal evidence paths, including applicable
ancestors, through assigned authored-sequence ranges as well as item scopes.
Keep public VersionCheck paths and physical locations source-local. Preserve
original semantic shapes and existing conservative failure/callback disclosure;
do not solve this by masking every declaration or changing source-version policy.
**Acceptance:** cover an imported authored list and the corresponding grouped
underlay handoff. Sensitive shifted evidence is redacted; an unclassified
declaration remains inspectable. Existing item/helper evidence tests still pass.
## 2. Preserve source chronology when expanding item history — P2
- [x] Addressed and regression-tested.
Completion: structural fragments use their local singleton index; item body and
helper events are expanded beside their declaring contribution. Mapping history
queries ignore unrelated sibling fields. A public regression covers preceding
authored records, two item sources, nested helpers, field origins, and root
traversal order. Focused facade/history tests and the everyday check passed.
**Owner:** `projectNestedEvidence` in
[imports.go](../../internal/engine/imports.go), especially lines 592–608.
The structural source fragment is indexed with the effective append offset:
`node.Sequence[start]`. That fragment is still a source-local singleton, so a
later item's `start > 0` prevents clearing its completed body. History therefore
retains helper values in the outer application and subsequently replays the same
item body and helper values again. Nested applications are also appended after
all outer applications rather than beside their declaring contribution.
**Confirmed reproduction:** a manifest imports `one.yml` and `two.yml` as items
at registered `/records`. Each item declares its own ordinary helper supplying
only `text`. The second item's `Explain(/records/1/text)` reports three events,
including an unchanged-value event attributed to the item root that never
authored `text`; the first item's explanation has only its helper contribution.
Root history visits source IDs `1,2,4,2,3,4,5`, placing the second item ahead of
the first item's helper despite depth-first acquisition `1,2,3,4,5`.
**Required change:** distinguish local indexes from effective offsets when
constructing structural fragments. Expand each item's retained applications in
source traversal order, preserving physical field origins without replaying a
completed body as an additional source contribution. Do not replace individual
source history with an opaque completed-scope snapshot.
**Acceptance:** use at least two item sources with helpers and a preceding
authored-list contribution. Helpers alone define their fields in explanations;
root/container history is chronological, final values/origins agree, and nested
items retain the same guarantees.
## 3. Record removals caused by registered-sequence replacement — P2
- [x] Addressed and regression-tested.
Completion: replacement queries include missing descendants and use the replacing
container's origin. Public regressions verify shorter lists, empty lists, and
removed fields; existing append-relevance tests remain green. Focused facade
tests and the everyday check passed.
**Owner:** `applicationOperationPath` in
[history.go](../../internal/engine/history.go), lines 134–149.
Query relevance filters every sequence operation to the incoming assigned range,
including `sequenceHistoryReplace`. This is appropriate for an append but not a
replacement: replacement also removes records outside the incoming range.
Replay removes them correctly while Explain silently omits the removal event.
**Confirmed reproduction:** a strictly composed underlay supplies records
`[{id: one}, {id: two}]`; main supplies `[{id: main}]` at the same registered
path. `Explain(/records/1/id)` says the value is absent, but its only history
event defines `two` in the underlay. No main-tier removal appears. Replacing with
`[]` has the same problem. Existing cross-cutting tests check absence/redaction
but do not assert that the removing operation is actually recorded.
**Required change:** make replacement relevant to removed descendants and
attribute those transitions to the replacing container/source when an incoming
leaf no longer exists. Keep append relevance limited to newly contributed items.
**Acceptance:** explanations show present-to-absent transitions and the correct
replacing origin for shorter/empty lists and removed fields; ordinary same-tier
appends do not falsely claim to change earlier records.
## 4. Honor contextual parsed version-path indexes — P2
- [x] Addressed and regression-tested.
Completion: protected-path sequence traversal uses the existing canonical
contextual index interpreter. Public import and grouped-underlay parity tests
cover parsed and explicitly indexed rules; existing version-path tests preserve
numeric-key intent and invalid index behavior. Focused tests and the everyday
check passed.
**Owner:** `fragmentVersionClaimProjected` in
[directives.go](../../internal/engine/directives.go), lines 190–194.
Sequence traversal accepts only `pathSegmentIndex`. Canonical numeric tokens
from public `ParsePath` are contextual parsed segments, accepted by the existing
version lookup and other semantic-path operations. The new protection code
instead treats them as a claim on the entire ancestor sequence.
**Confirmed reproduction:** use the two-source required-version fixture from
finding 1, without the sensitivity predicate. A rule built with
`Keys("records").Index(0).Key("version")` succeeds. The equivalent
`ParsePath("/records/0/version")` fails with `import.version`, even though the
import appends to index 1 and does not claim main's index 0 header.
**Required change:** use existing contextual sequence-index interpretation while
preserving explicit mapping-key versus index intent and canonical token rules.
Do not narrow VersionRule to programmatically indexed paths or relax real
ancestor-replacement protection.
**Acceptance:** equivalent parsed/indexed rules give equivalent ownership
results across imports and grouped underlays, with existing literal numeric-key,
wrong-container, and noncanonical-index cases preserved. The current divergence
is an unintended restriction beyond the requested feature.
## 5. Keep default-mode main version ownership inside item scopes — P2
- [x] Addressed and regression-tested.
Completion: default-mode effective protection is translated into the main item's
local scope, preserving segment intent and effective diagnostic paths. Both
public entry points reject helper-supplied main declarations and permit unrelated
version fields. Focused source-version tests and the everyday check passed.
**Owner:** the `mainIsItem` branch of
[imports.go](../../internal/engine/imports.go), lines 228–235.
The helper ownership check applies `versionState.localRule` directly to
item-local documents. In default mode that rule is an effective application
path, not a local item path, so the check misses a helper's protected contribution.
**Confirmed reproduction:** load main as an item at `/records`, enable imports,
and configure the default-mode rule at
`Keys("records").Index(0).Key("version")`. Main contains only `id` plus an
ordinary helper import. The helper supplies `version: imported-version`.
Loading succeeds and the main checker receives the helper's version, although
imported fragments are not allowed to supply the protected main version path.
**Required change:** translate the protected effective path into the item scope
before checking its individual helper contributions, or check correctly
projected contributions at the enclosing boundary. Retain default effective-path
interpretation and enabled-mode source-local checking/header consumption.
**Acceptance:** the fixture fails with `import.version` in both public entry
points. Helpers outside the actual protected path remain permitted, and enabled
source-version-mode behavior is unchanged. Do not add a general ban on fields
named `version`.
## 6. Preserve absence for ordinary defaults-only loads — P2
- [x] Addressed and regression-tested.
Completion: absent ordinary input stays absent until an existing explicit layer
needs assembly. Public parity tests cover values, root presence/origin, optional
registrations/defaults, and real empty mappings. Existing defaults-only override
coverage and the everyday check passed.
**Owner:** `executeLoadRequest` in
[load.go](../../internal/engine/load.go), lines 210–216, and
`compositionForInputWithPolicy` in [load_input.go](../../internal/engine/load_input.go).
Ordinary loads now unconditionally initialize a composition, including when
`NoSource()` supplied no document. Its synthetic empty mapping is assigned back
to `input.document` and decoded as explicit input. This changes callers that
have not opted into either feature.
**Confirmed reproduction:** `Load[map[string]any](NoSource())` reports
`Metadata().IsExplicit(Keys()) == true`, while the equivalent `LoadLayered` call
reports false. No source contributed a root definition. This violates the
existing NoSource absence distinction rather than adding useful feature behavior.
**Required change:** retain the absent-input handoff when no real contribution
exists, while still validating actual singleton registered sequences. Preserve
synthetic assembly only where existing overlay/CLI behavior needs it.
**Acceptance:** ordinary/layered defaults-only loads agree on root presence and
origin, with and without registrations/default factories; an actual empty source
mapping is still explicit. Defaults-only overrides keep their existing behavior.
## 7. Remove new quadratic scans from sequence construction — P2
- [x] Addressed and behaviorally validated.
Completion: claims retain encountered immutable nodes; history construction
groups ranges by source; scope/version projection uses source/path range
lookups; retained-history traversal and replay use per-application path indexes.
Ordered slices and detached clone ownership remain intact. Static review confirms
the listed nested scans are removed. A wide-collection replay/origin/ownership
regression, the everyday check, and `go test -race ./...` passed. No timing
threshold or release gate was added.
**Owners:** [compose.go](../../internal/engine/compose.go), especially
`validateRegisteredSequences` and application construction; range lookups in
[imports.go](../../internal/engine/imports.go); `sequenceHistoryAt` in
[history.go](../../internal/engine/history.go).
Identity comparison uses a map, but surrounding new machinery reintroduces
avoidable nonlinear work:
- Validation looks each sequence claim up again with unindexed `documentAtPath`.
One source with N sibling registered sequences performs N linear root-mapping
searches: O(N²) at fixed depth and constant record size.
- Application construction scans all ranges for each source: O(S × R). Item
evidence/source projection repeats full range scans for each contribution.
- `sequenceHistoryAt` linearly scans the application's sequence descriptors
repeatedly during retained-tree traversal and replay. A wide document with
many registered sequence paths similarly accumulates quadratic work.
These are structural findings; no timing benchmark claim is made. They diverge
from the plan's near-linear construction objective and the user's explicit
efficiency priority, without requiring that public behavior change.
**Required change:** retain encountered node references or reuse existing indexed
document lookup; group/index ranges by source and path once; use a small owned
per-application path lookup where repeatedly needed. Keep order in slices and
ownership unchanged. Avoid a new global cache/index framework or unrelated
performance refactoring.
**Acceptance:** static review no longer finds these nested whole-set scans on
the new construction path. Existing ordering, evidence, cloning, and behavior
tests pass; optional scaling measurements may help but are not a new gate or
timing-threshold test requirement.
## 8. Do not fabricate index zero for item-local conflict diagnostics — P2
- [x] Addressed and regression-tested.
Completion: failed item composition is projected through each enclosing scope
using preceding contributions, including prior grouped inputs. Only composition
diagnostics are rebased; physical locations and source-local access/syntax
diagnostics are retained. Regressions cover authored/imported predecessors,
nested items, item mains, grouped item sources, and failure evidence. Focused
facade/engine tests and the everyday check passed.
**Owner:** `visitItemSource` in
[imports.go](../../internal/engine/imports.go), lines 519–525.
Helper composition errors are rebased with `destination.Index(0)` before the
item's actual range is assigned. Nested rebasing also omits an enclosing item
index. Physical locations are retained, but the application path points at a
different record.
**Confirmed reproduction:** main authors `records: [{id: first}]` and imports
an item with `id: second` whose ordinary helper also defines `id`. The conflict
is reported at `/records/0/id`, incorrectly naming the existing first record;
the imported contribution would be at index 1.
**Required change:** carry scoped diagnostic paths until they can be projected
through the enclosing assignment, rather than injecting a fabricated index.
Preserve safe failure source evidence and actual physical conflicting locations;
do not require publishing a usable partial result to compute a diagnostic.
**Acceptance:** a helper conflict after preceding authored/imported records names
the correct item, and nested item conflicts retain enclosing semantic indexes.
The recently added malformed-registered-record regression must continue to pass.
## 9. Correct the public identity-validation phase statement — P3
- [x] Documentation corrected.
Completion: the public declaration distinguishes registration preflight from
input identity validation during strict composition, matching the existing
configuration contract. The final everyday check and full race suite passed.
**Owner:** `Definition.SequenceCompositions` comment in
[lifecycle.go](../../lifecycle.go), line 20.
The comment says repeated identities and conflicting registrations both fail
option validation. Only invalid/conflicting registrations are preflight errors;
duplicate identities in input are `compose.conflict` after source acquisition
and applicable helper/header processing. The declaration comment inadvertently
promises an earlier failure phase than the approved design or implementation.
**Required change:** distinguish registration preflight from record validation
during strict composition. Correct the comment rather than moving data checks
earlier or adding a new pre-acquisition guarantee. Keep the canonical
configuration contract and declaration comments consistent.
## Review validation
Passed on the reviewed tree with Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`
- `git diff --check 2051177..HEAD`
The behavioral findings were independently reproduced through public API calls
using disposable in-memory filesystems in a temporary Go program outside the
repository. The scaling finding is based on the inspected loops and lookup
implementation, not host timing. No production code, tests, dependencies,
release machinery, or downstream repositories were modified during this review.
## Remediation validation
All nine findings have completion evidence above. Final validation passed with
Go `go1.26.9 linux/amd64`:
- `GOTOOLCHAIN=local ./scripts/check.sh`
- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`
- Local documentation-link and whitespace checks.
The original external public-API reproductions were rerun against the completed
changes: consumed sensitive evidence is redacted, helper events are chronological
and source-specific, replacements explain removals, parsed/indexed version rules
agree, imported main-version claims fail, NoSource stays absent, and item conflict
paths identify the correct records. Static review and the wide-collection
regression verify the targeted lookup changes without adding performance
thresholds. No dependencies, public controls, release gates, or downstream
repositories were changed by remediation.

View File

@@ -1,318 +0,0 @@
# Sequence-Item Placement and Identity-Based Composition
Status: implemented. This roadmap records the target state and accepted behavior.
The [configuration contract](../config.md#strict-document-composition) defines
the current behavior, and the [implementation plan](implementation.md) records
delivery sequencing and validation.
## Goal and current baseline
Let consumers assemble a list of records from independently authored mapping
files. Placement makes a source mapping one sequence item; identity-based strict
composition combines distinct items and rejects repeated identities. These are
complementary operations, with explicit consumer opt-in.
Sources can explicitly place mappings as sequence items at non-root destinations,
and definitions can register literal sequence paths with one literal string
identity field for duplicate-rejecting strict composition. Ordinary unregistered
sequences retain strict conflict behavior. [Precedence assembly](../config.md#layered-loading)
and [selected overlays](../config.md#selected-overlays) continue to replace
sequences wholesale. The canonical contract documents the exact rules and limits.
This roadmap covers a narrower feature than the
[identified record sequences candidate](future.md#1-identified-record-sequences).
Composite identities, recursive merging of matching records, identity-based
overlays, and record inheritance are not requested here. That scope statement
does not establish permanent architectural prohibitions.
## Targeted behavior
### PromptKit use case
Each file defines one prompt directly.
`prompt_one.yml`:
```yaml
id: prompt.one
text: "foo"
```
`prompt_two.yml`:
```yaml
id: prompt.two
text: "bar"
```
The consumer selects item placement at `prompts` for both sources and registers
`id` as that sequence's identity field. Within one strict composition tier, the
result is:
```yaml
prompts:
- id: prompt.one
text: "foo"
- id: prompt.two
text: "bar"
```
The files need neither a filename-derived wrapper key nor their own list wrapper.
Existing sources that already contain `prompts: [...]` can contribute to the same
registered sequence under the same composition rule.
### Explicit sequence-item placement
- The destination identifies where content belongs; an explicit placement mode
selects ordinary mapping placement or placement as one sequence item. Preserve
existing behavior when the new mode is not selected. Do not infer the mode
from document shape, filename, or the presence of an `id` field.
- Support item placement for application-supplied inputs and YAML path/glob
imports. Each matched file of an item-placement glob contributes one item,
using existing match and traversal ordering. Physical import-path resolution
remains relative to the declaring file.
- Wrap the source's application mapping as one item at the destination sequence;
the new operation is not flattening a source sequence or splitting a mapping
into several items. Existing supported source forms should receive equivalent
placement behavior; this does not expand accepted source-root shapes.
- Item placement supplies structure, not identity values. Keep authored `id` and
other fields in the item. Source directives and version headers retain their
own processing rules, with the item-placement interpretation recorded under
Settled Decisions.
- Preserve existing absence distinctions: `NoSource()` and an unmatched optional
import contribute nothing. An actual empty mapping contributes an empty item,
which fails the agreed identity requirement when that item participates in a
registered composition sequence.
Item placement is independent of composition registration. Ordinary imports
inside an item compose into its mapping; public source evidence reports the
placement mode and destination container. The settled decisions below define
these behaviors in detail.
### Identity-based strict composition
- Consumers designate sequence paths and one direct string identity field for
each. Unregistered sequences retain their current composition behavior.
- Each participating record is a mapping with the designated string field.
Missing, null, and non-string identities are invalid. Compare exact strings
without trimming, case folding, normalization, or numeric coercion. Empty
strings are valid identities; repeated empty identities conflict.
- Append distinct records in contribution order, preserving each source list's
order. Use existing deterministic traversal within a strict tier rather than
sorting by identity.
- Repeated identities conflict across contributions and within one contributing
list, including a list with no peer contribution. Equal bodies and disjoint
bodies both conflict; matching records do not recursively merge.
- An empty sequence contributes no records. Two empty sequences compose, and an
explicitly contributed empty sequence remains present.
- Apply composition to main inputs and their imports and within each underlay
group, including placed item contributions and already-authored sequences.
Identity uniqueness is scoped to the sequence in that strict tier.
- A composition error cannot be hidden by a higher-precedence replacement. Use
normal load errors and the existing failed-load result contract.
This differs from mapping composition at a shared key: mapping values can recurse,
whereas a repeated record identity conflicts. It does not establish final
uniqueness after defaults or application resolution. Identity values are not
supplied by those later phases; final domain validation remains application-owned.
### Precedence and other operations
Between underlay tiers, between underlays and the main tier, and in selected
overlays, sequences continue to replace lower sequences wholesale. `[]` clears
the lower list. The same identity can occur in different tiers without a
cross-tier duplicate error. Item placement does not turn precedence assembly
into append semantics.
Whole-sequence CLI overrides, defaults, application resolution, and existing
sibling inheritance retain their behavior. This feature does not make records
inherit from one another by identity. Effective-configuration diffs continue to
compare final sequences atomically. Replacement-only selected overlays and
synthetic CLI overrides do not acquire identity validation from composition
registration; existing decoding and application validation still apply.
### Existing inspection and ownership obligations
A field at source path `text` can become `prompts[1].text`; a record originally
at source index zero can acquire a different effective index after composition.
Keep original physical locations and resource contexts associated with the right
record while reflecting effective placement in presence and origin queries.
History and explanations must represent accumulation within a tier and actual
replacement between tiers. Replaying each contribution as a replacement would
lose earlier records. Apply existing historical sensitivity and disclosure rules
when indexes change or sequences are replaced, and preserve detached public
records and existing input/result ownership. These are applications of current
contracts, not a new promise of persistent identity-based paths across loads.
## Work needed
The target state requires changes in these areas; execution order belongs to the
[implementation plan](implementation.md):
- **Public API and import syntax:** introduce the selected composition
registration and placement controls at the existing root-owned boundaries;
convert them into private engine state. Define invalid-option handling and
retain current defaults for callers that do not opt in.
- **Acquisition and placement:** support item contributions in ordinary and
layered loading, input/import destinations, and glob expansion. Implement the
selected nested-import and source-version interpretation without changing
acquisition identity or physical resource resolution.
- **Composition and retained evidence:** extend claim validation, tree assembly,
origins, and history together. The current composer in
`internal/engine/compose.go` collects claims by path and stores history
applications as well as the final tree. Updating only the tree merge is
insufficient. Keep tier boundaries explicit and preserve the existing limits
and accounting through the new wrapper and accumulation operations.
- **Inspection integration:** carry source-to-effective placement through history,
explanations, sensitivity, resource contexts, and public source records. Use
the smallest internal representation that preserves those existing contracts;
this roadmap does not prescribe a new indexing or caching subsystem.
- **Behavioral verification:** cover direct and glob-imported item sources,
already-authored lists, mixed contributions, duplicate/malformed identities,
empty and absent contributions, ordering, mounted paths, and nested imports.
Verify whole-list replacement, errors hidden by later tiers, version and
structural processing, remapped indexes, resource contexts, historical
redaction, and detached records. Protect current unregistered behavior,
defaults/resolution, and atomic diffs with proportionate regression coverage.
- **Documentation and adoption:** provide a maintained PromptKit-shaped example
and update public declaration comments, configuration/inspection contracts,
consumer guidance, and relevant internal documentation when behavior lands.
Record the significant design decision under the architecture policy and keep
the broader candidate in `future.md` distinct from the implemented subset.
Completion means the PromptKit example works through public inputs and imports,
composition and replacement behave as agreed, and configuration values agree
with their inspection evidence. Validation follows the
[development guide](../development.md#baseline-validation).
## Settled Decisions
The following choices are accepted for this feature. No substantive design
choices remain open in this roadmap. The implementation plan selects exact
public names, YAML option names, and existing diagnostic-code mappings.
Internal representations remain implementation choices, subject to the existing
contracts and the behavior recorded here.
### 1. Public controls and ownership
Put sequence path/identity-field registrations on `Definition`. Put an explicit
placement mode beside input destination controls, with equivalent YAML import
syntax and support in ordinary loading. This separates reusable composition
policy from placement of individual sources. Default placement remains unchanged.
### 2. Destination and registration paths
Use literal mapping-key paths to identify destination sequences and match
registrations at their effective location after placement. For example, a source
mounted beneath `application` uses a registration at `application.prompts`.
Preserve existing destination validation without adding new reserved names.
Permit unused registrations, coalesce identical registrations, and reject
conflicting identity fields for the same path. Ancestor/descendant registrations
apply only where their literal paths reach sequences; they do not implicitly
traverse every record.
Sequence-index selectors and wildcard matching are outside this initial feature.
Item placement identifies a sequence beneath the configuration root; a root
sequence would expand the existing mapping-root model and is not included.
These are the selected feature boundaries, not permanent architectural exclusions.
Nested placement follows decision 4 without expanding registration addressing.
### 3. Item placement without identity registration
Allow item placement independently. It contributes a singleton sequence without
requiring an identity registration. Existing rules apply at unregistered paths:
two present sequences conflict in one strict tier, and an upper-tier sequence
replaces a lower one. Do not infer append semantics merely from item placement.
### 4. Imports declared inside an item-placed source
Compose ordinary imports into the declaring item's mapping. For an import
declared at the item root, an omitted destination or `destination: []` means that
item context; `[metadata]` places the helper inside the item's `metadata`.
Further relative destinations continue from the immediately importing document's
placement within that context. Physical import paths still resolve from the
declaring file.
Item mode is explicit on each import, not inherited automatically. A collection
manifest explicitly imports peer prompt files as items at `prompts`. A helper
import supplies fields of the declaring item and need not declare its own ID.
An explicitly item-placed import inside a prompt can contribute to a nested
sequence. Registrations do not reach through outer sequence indexes in this
initial feature, so those nested sequences retain atomic composition. Multiple
contributions to such a nested sequence therefore follow ordinary strict
conflict rules. This permits nested placement without introducing index selectors
or wildcard matching.
### 5. Identity-validation boundary and structural processing
Validate each completed record after applicable source-header processing and
ordinary helper-import composition, before combining records in the collection
and before later precedence replacement. A helper may supply the identity field;
it need not be authored in the item file itself. Duplicate fields within the
item still follow ordinary strict mapping composition. Defaults and application
resolution cannot supply missing identities because they run later.
Preserve existing structural semantics. An item has no mapping-sibling namespace
for record inheritance, while mappings inside it can use existing inheritance.
Consumed structural fields and headers do not become application identity
values. Existing processing rules govern overlap with those controls; no new
identity-field-name blacklist is introduced.
### 6. Empty identities and literal field names
Accept empty string identities; repeated empty identities conflict. Interpret
the designated direct field name literally, including unusual or empty YAML
mapping keys where existing processing permits them. Presence and string type
suffice for identity comparison. Nonempty-ID requirements, naming conventions,
and whitespace restrictions remain application-owned validation choices.
### 7. Non-sequence values at registered paths
Registration selects how actual sequences compose; it is not an additional
schema declaration. Missing and non-sequence values retain existing composition
and decoding behavior. Sequence/null and sequence/mapping overlaps still
conflict in strict composition. Item placement contributes an actual sequence
and follows these rules. Registration does not independently reject every
non-sequence source value that a later tier might replace.
### 8. Validation of replacement-only sequences
Apply identity validation to registered sequences participating in strict
composition, including singleton contributions. Selected-overlay sequences and
synthetic CLI replacements retain their existing validation. Defaults and
application callbacks also retain their behavior.
An overlay can therefore replace a list with duplicate IDs unless the consuming
application's validator rejects them. The identity rule governs composition,
not final configuration validity.
### 9. Source-version checks for item-placed main inputs
Support item placement for main inputs as well as imported and underlay sources.
Enabled source-version modes inspect declarations in source-local coordinates
before placement. Default mode retains its existing effective-path interpretation.
Accepted non-main headers are consumed; the main header remains application
content, following existing rules.
The retained main header's effective location includes its actual item index.
Existing ownership protection follows that location and must not accidentally
protect another record because its source index differs from its effective
index. Preserve source-local version evidence and diagnostics where the existing
contract uses them. This feature introduces neither a new version policy nor
an exception to existing version-path protections.
### 10. Public placement evidence and diagnostics
Keep `SourceInfo.Destination` describing the effective destination container and
expose the placement mode. Use existing effective-path metadata to describe
resulting item paths, retaining physical source locations and resource contexts.
Do not promise one permanent item index per source: replacement can remove its
contribution, and contribution order can differ between loads.
Duplicate diagnostics identify conflicting locations and the sequence path
without exposing potentially sensitive identity values. Reuse existing diagnostic
categories where appropriate. Resolved historical item paths and persistent
identity-addressed paths in source records are not part of this feature.