From ef28eabaf8eb85eddac0e7057fea2ba9c5c4304a Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 17 Oct 2026 04:56:51 +0000 Subject: [PATCH] Retire completed sequence composition plans --- ...0014-sequence-placement-and-composition.md | 6 +- docs/development.md | 11 + docs/roadmap/implementation.md | 815 ------------------ docs/roadmap/remediation.md | 353 -------- docs/roadmap/sequence-composition.md | 318 ------- 5 files changed, 15 insertions(+), 1488 deletions(-) delete mode 100644 docs/roadmap/implementation.md delete mode 100644 docs/roadmap/remediation.md delete mode 100644 docs/roadmap/sequence-composition.md diff --git a/docs/adr/0014-sequence-placement-and-composition.md b/docs/adr/0014-sequence-placement-and-composition.md index 25991cf..b78ebfd 100644 --- a/docs/adr/0014-sequence-placement-and-composition.md +++ b/docs/adr/0014-sequence-placement-and-composition.md @@ -4,8 +4,10 @@ Accepted and implemented. The [configuration contract](../config.md#strict-document-composition) defines current behavior, and the [consumer guide](../consumers/README.md#identified-prompt-sequences) -shows the maintained example. Implementation status and delivery sequencing -are tracked in the [implementation plan](../roadmap/implementation.md). +shows the maintained example. The completed roadmap, implementation plan, and +resolved remediation record are preserved in Git history at `008d2ba` under +`docs/roadmap/`; see the [development guide](../development.md) for retrieval +commands. ## Date diff --git a/docs/development.md b/docs/development.md index 103dc46..7d9a77f 100644 --- a/docs/development.md +++ b/docs/development.md @@ -137,6 +137,17 @@ the [strict composition](config.md#strict-document-composition) and [maintained prompt catalog](consumers/README.md#identified-prompt-sequences) shows the current consumer boundary, and [ADR 0014](adr/0014-sequence-placement-and-composition.md) records the decision. + +The completed sequence-composition roadmap, implementation plan, and resolved +remediation record are preserved in Git history at `008d2ba` under +`docs/roadmap/`. For example: + +```sh +git show 008d2ba:docs/roadmap/sequence-composition.md +git show 008d2ba:docs/roadmap/implementation.md +git show 008d2ba:docs/roadmap/remediation.md +``` + Potential enhancements are tracked in [future features](roadmap/future.md). Install the Go version required by `go.mod` or a compatible newer toolchain. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index abdb6d7..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,815 +0,0 @@ -# Implementation Plan: Sequence Placement and Composition - -Status: implementation complete; all 14 stages completed. - -## Purpose and authority - -Implement the accepted [feature roadmap](sequence-composition.md), starting from -`2051177` (`Settle sequence placement and composition roadmap decisions`). This -plan owns execution order and progress; the roadmap owns intent, scope, and the -ten settled behavior decisions. The target implementer is a Luna coding agent. -Complete the numbered stages in order. Each stage is a separate focused prompt -and validated commit, not a request to implement the whole feature at once. - -Read [development](../development.md), [agents](../policy/agents.md), -[architecture](../policy/architecture.md), [testing](../policy/testing.md), and -[documentation](../policy/documentation.md) before starting. Use the code graph -for discovery, then inspect the current owning implementation and tests. Preserve -unrelated work. Follow the existing commit policy after each nonempty completed -unit; record completion and validation in this plan. - -This planning change implements nothing. Later stages introduce private engine -capabilities before exposing them through the root facade. Until a capability -lands, current consumer documentation must continue to describe implemented -behavior only. Accepted ADRs may describe the decision before implementation. -When a stage makes behavior reachable by consumers, update its owning declaration -comments and configuration contract in that same unit, to the implemented -boundary. Stage 13 consolidates guidance and examples; it does not postpone those -contract updates. - -## Fixed design for implementation - -### Public API and YAML spelling - -Use these root-owned declarations and fields. Their zero values preserve current -behavior; do not change the signatures of `Load` or `LoadLayered`. - -```go -type PlacementMode uint8 - -const ( - PlacementModeMapping PlacementMode = iota - PlacementModeSequenceItem -) - -type SequenceComposition struct { - Path Path - IdentityField string -} -``` - -- Add `SequenceCompositions []SequenceComposition` to `Definition[T]`. -- Add `PlacementMode PlacementMode` to `Input` and `SourceInfo`. -- Add `DestinationUnresolved bool` to `SourceInfo`. False preserves the existing - exact-destination interpretation; true explicitly marks the enclosing-container - fallback described below. This is output evidence, not an input option. -- Add both `Destination Path` and `PlacementMode PlacementMode` to `Options[T]`. - Ordinary `Load` needs the same per-source controls as the layered entry point; - its overlay and CLI coordinates remain application-rooted. -- YAML import declarations accept `placement: mapping` or - `placement: sequence-item`. Omission means `mapping`, independently for every - declaration. Retain `destination: [literal, keys]`, `path`, `glob`, and - `optional` with their existing meanings. Do not add aliases or inferred modes. -- Root fields and types stay root-owned. Use explicit conversions into engine - records and a private engine mode, following the source-version-mode pattern. - Preserve unknown numeric modes until engine preflight rejects them. - -Validate options before acquisition. Unknown placement modes, item placement at -an empty destination, invalid destinations/registration paths, and conflicting -registrations use `CodeOptionsInvalid`. Registration paths use existing literal -mapping-key destination normalization, including its current reserved-key rules; -do not add more reserved names. Parsed numeric tokens become literal keys; -explicit indexes and invalid paths fail. Identical registrations coalesce, unused -registrations are permitted, and an empty/root registration is harmless and -unused under the mapping-root model. Do not reject ancestor/descendant pairs. -`IdentityField` is one literal string, including an empty string, not a path. -No field-name or identity-value normalization is introduced. - -Use `CodeImportInvalid` for malformed import placement syntax, including an item -import whose effective destination is empty in its current mapping scope. An -omitted item-import destination is valid when inherited placement already names -a non-root collection. At an item root, an omitted destination names the item -mapping itself and cannot turn that mapping into a sequence. A nested item import -needs a nonempty effective destination within that item scope; this can come from -its relative destination or an ordinary helper's existing placement prefix. - -Use `CodeComposeInvalid` for malformed registered-sequence records/identities and -`CodeComposeConflict` for duplicate identities or ordinary strict conflicts. -Malformed-record paths identify the effective item or its identity field; -duplicate paths identify the sequence container, with the first and conflicting -physical locations in `Primary`/`Related`. Error messages must not interpolate -identity strings or record bodies. Preserve deterministic reporting without -inventing an all-errors guarantee or freezing incidental message wording. - -### Composition, scopes, and evidence - -1. Compile registrations once per load into an owned exact-path lookup. A strict - tier has a separate identity table for each encountered registered sequence. - Scan every actual sequence contribution, even one with no peer. Require map - items and a present string identity after source-header processing and helper - composition. Compare exact strings; accept empty strings. Non-sequences retain - existing behavior. Distinct records append in contribution order; repeated - identities conflict even when bodies are equal or disjoint. -2. Keep every precedence boundary atomic for sequences. Each underlay group and - the main tier starts new identity state. Selected overlays, CLI, defaults, - inheritance, and application resolution do not run this identity validator. - Registrations apply at effective literal mapping paths, never through record - indexes. They are composition policy, not final configuration schema. -3. Source acquisition remains depth-first and keeps current source IDs, physical - resource identity, cycle/repetition detection, and shared budgets. Model an - item source plus its ordinary helper imports as a local strict mapping scope. - Ordinary imports share that scope and append relative mapping destinations; - an explicit item import opens a child item scope. Complete each item scope - before wrapping/validating its record in the enclosing composition. Keep its - contribution at the declaring source's original traversal position. -4. Carry a small private placement projection with each contribution: its scope, - local destination, and any sequence range assigned by strict assembly. Use it - to translate source paths into effective paths without reordering sources or - mutating acquired trees. Authored sequence index `j` becomes `offset+j`; - an item source's local root becomes the assigned item index. Nested projections - compose. Do not locate records again by rescanning IDs during inspection. -5. Keep completed trees distinct from source/history applications. Extend private - history operations to distinguish a sequence contribution from recursive - mapping merge: the first contribution to a collection in a tier replaces the - previous tier/default value; later contributions append. An empty first list - clears the prior list; an empty later list adds no items. Store only incoming - fragments and offsets, not a cumulative list snapshot after every append. - For item scopes, retain the initial item body and helper contributions at their - projected item paths, so helpers retain their own history and origins. -6. Reuse that projection in origins, history replay/query relevance, historical - path enumeration, sensitivity, and version evidence. Public source records - describe containers, not a new permanent record-identity address. An item - source reports its sequence container; a mapping helper inside an item reports - its effective mapping destination, which can include an enclosing item index. - Output evidence may contain such indexes even though input destinations and - registrations accept only mapping-key segments. - -### Unresolved destination evidence - -`SourceInfo.DestinationUnresolved` describes whether that snapshot's `Destination` -is the source's exact effective destination container: - -- When false, `Destination` is exact. This says nothing about whether the load - succeeded or the source's values survived later replacement. An item source - can have an exact sequence-container destination before its own item index is - assigned, because that index is not part of its reported container path. -- When true, an enclosing item index needed for the destination has not been - assigned. `Destination` is only the nearest resolved enclosing collection - container, not the source's actual mapping destination or nested sequence - destination. Do not append the unresolved local suffix as though the missing - index were a mapping key, or use a fabricated index. -- For example, a mapping helper destined for `prompts[1].metadata` reports the - container `prompts` with `DestinationUnresolved: true` until the outer item - index is assigned. Later snapshots report `prompts[1].metadata` with false. - The item-root source itself reports `prompts` with false when that container - is known. The placement mode continues to describe each source's own mode. -- Apply this distinction to every source-record exposure, including source - records available during callbacks and failed-load records. Once projection - resolves the container, subsequent snapshots report the exact path with false. - Previously returned detached snapshots retain their original path and flag. - Completed successful loads report resolved containers; early failures may - contain both resolved and unresolved source records. - -This flag makes the existing planned fallback observable without adding a public -item-index API, promising permanent indexes, or reconstructing unknown placement -after a failure. Physical source locations, source IDs, and source-local version -evidence retain their existing meaning regardless of the flag. - -Choose private helper names and file splits to fit existing responsibilities. -Do not introduce a provider framework, generic merge callbacks, a second loading -lifecycle, or a public identity-selector API to implement these mechanics. - -### Efficiency and preserved limits - -Use one string-keyed identity map per registered sequence per strict tier, with -ordered appends and existing indexed mapping lookup. Clone incoming records into -owned assembly storage without repeatedly cloning the accumulated sequence. -Registration lookup, validation, and sequence construction should be linear in -the visited data apart from existing path/depth costs. Complete each item scope -once; do not recursively recompose the entire ordinary-helper subtree at every -import edge. Keep retained history fragments immutable and independently owned -from the working tree. These are implementation choices, not a new public -complexity, allocation, or latency guarantee. - -Preserve current per-source, aggregate, traversal, projection, and retained-history -limits and error families. Count acquired content once and synthetic mapping and -sequence wrappers where introduced; include the extra sequence level in depth -checks. Do not charge an already-accounted item/helper body again merely because -it moves to a completed scope. Keep library-owned handles closed before returning -on success or failure. Do not add new limits or change their existing values. - -## Stage 1 — Record the decision and prepare private registration controls - -Scope: one ADR and engine option normalization; no new public fields yet. - -- Write `docs/adr/0014-sequence-placement-and-composition.md` as an Accepted ADR - using the established format. Explain explicit item placement and registered - duplicate-rejecting strict composition, unchanged precedence replacement, and - their rationale in ConfigKit terms. Discuss alternatives only in Alternatives - Considered; do not center a downstream application's semantics. Link the - roadmap for implementation status. Do not rewrite accepted earlier decisions. -- Add engine-owned registration records/state and normalization, reachable from - engine `Definition`. Validate/coalesce using the fixed design above; detach - registration slices and paths at the load boundary. Keep compiled policy - per-load, not mutable shared Definition state or a global cache. -- Wire preflight into both entry points without consuming a reader or opening a - source for invalid options. Thread compiled policy through the request rather - than rebuilding it or invoking application hooks for internal handoffs. Keep - the existing shared source-version checker lifecycle intact. -- Add focused normalization tests for duplicate/conflicting registrations, empty - identity-field names, numeric/escaped literal keys, invalid index paths, - unused/root registrations, and preserved zero-option behavior. - -Validation: focused option tests and `./scripts/check.sh`. Completion: existing -loads remain unchanged, with only private registered-policy input newly accepted. - -Completion evidence: added Accepted ADR 0014 and private engine registration -normalization. Ordinary and layered entry points compile/coalesce policy and -reject invalid registrations before source reads, then carry the compiled policy -through the load request while clearing raw registration slices from internal -handoffs. No public API or composition behavior was added. Focused coverage -checks duplicate/conflicting definitions, empty identity names, numeric and -escaped literal keys, invalid explicit indexes, unused/root registrations, -zero-option behavior, and reader preservation on preflight failure. These checks -passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SequenceRegistration' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -The compiled registration policy is intentionally not consumed by composition -until the next implementation unit. No environmental limitation affected -validation; checks ran on Linux/amd64. - -## Stage 2 — Implement registered strict sequence validation and assembly - -Scope: `internal/engine/compose.go` and focused composition tests. - -- Add a policy-aware internal composer while preserving an unregistered adapter - for existing callers/tests. Collect/check ordinary claims with current strict - semantics; only sequence/sequence claims at registered effective paths use the - new rule. Mapping/sequence, null/sequence, and scalar overlaps still conflict. -- Validate every registered sequence as it is encountered, not only overlapping - claims. Missing paths and non-sequences do not independently fail registration. - Validate the completed contribution, not the unfinished raw item source. -- Build identity tables once, reject duplicates within/across lists, and append - records into owned assembly storage. Preserve mapping traversal and list order, - explicit empty sequences, exact identity spelling, and existing error safety. -- Return the assigned source-to-effective sequence ranges alongside the composed - result for the following stages. Do not change overlay merging or diff logic. -- Test distinct/mixed lists, singleton malformed records, missing/null/non-string - fields, empty/whitespace-distinct identities, duplicate empty identities, - identical/disjoint-body duplicates, empty lists, missing/non-sequence values, - unused/ancestor registrations, and unchanged unregistered conflicts. Use a - compact table of meaningful categories, not a full cross-product. - -Validation: composition tests and `./scripts/check.sh`. Completion: the private -composer returns the correct tree and range projection without aliasing inputs. - -Completion evidence: added a policy-aware private composer while retaining the -unregistered adapter. Registered sequences validate every actual contribution, -including singleton and empty lists; distinct records append in source/list -order, duplicate exact string identities fail, and other overlap kinds retain -strict-conflict behavior. The result carries per-source sequence ranges for -later projection. Tests cover malformed identity records, within/across-list -duplicates including empty IDs and differing bodies, absent/non-sequence paths, -empty lists, unregistered conflicts, and input/result ownership. These checks -passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Compose' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 3 — Project composition origins and claims onto appended indexes - -Scope: composition evidence, not import acquisition. - -- Use Stage 2's assigned ranges to project each sequence record and descendant - onto its actual effective index. Retain physical locations and contributor - ordering for mapping and sequence containers. Preserve field-specific origins; - do not attribute every descendant to the sequence's first source. -- Keep source-local claims useful for conflict reporting while effective origins - and presence describe assembled paths. Projection must handle already-authored - lists, mounted mapping prefixes, escaped/numeric keys, and more than one - registered collection without using textual pointer replacement. -- Add composition-origin tests in which a later list's local index zero becomes - an effective nonzero index. Mutate returned/copied evidence and verify input - and separately retained paths remain independent. - -Validation: composition/provenance/path tests and `./scripts/check.sh`. -Completion: correct trees and effective origins share the same record ordering. - -Completion evidence: composition origins now traverse each source document with -separate local and effective semantic paths. Stage 2's source ranges shift list -items and descendants to assigned indexes while retaining each field's physical -location and contributor order at sequence containers; conflict claims remain -source-local. Tests cover a later source's local index zero becoming effective -index one, mounted and escaped/numeric-key paths, multiple collections, and -detached origins, claims, ranges, and input trees. These checks passed with Go -`go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Compose|Provenance|Path' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 4 — Preserve append history and whole-list tier replacement - -Scope: private history operations, replay, and precedence handoff. - -- Extend `historyApplication` and its cloning/query/replay helpers with explicit - sequence-contribution information from the fixed design. Keep ordinary merge - behavior as the default. Support semantic indexed prefixes for helper events; - current `mergeHistoryAtPath` supports mappings only and cannot be reused - unchanged for them. -- At every strict-tier boundary mark the first contribution at a sequence path - as replacement and subsequent contributions as append. This boundary cannot - be inferred from `LayerUnderlay` alone: different underlay groups share that - layer name. Keep this information when `assemblyAccumulator.applyOverlay` - imports a group's history applications. -- Append replay must affect the collection and newly added record paths, not - falsely attribute unchanged earlier records to a later source. Translate - historical queries to the incoming fragment's local indexes for location/value - lookup. Preserve source-level applications rather than replacing them with one - opaque completed-tier snapshot. -- Test before/after replay for append, explicit empty lists, independent lists, - two underlay tiers with the same identity, main replacement, and a later whole - list replacement/removal. Verify origin attribution and history agree with - Stage 3's composed tree. - -Validation: history/explanation/overlay tests and `./scripts/check.sh`. -Completion: strict accumulation is distinguishable from precedence replacement. - -Completion evidence: history applications now carry owned per-collection -contribution ranges and distinguish each strict tier's first sequence -contribution (replacement) from later appends. Replay preserves source -applications through underlay handoff, maps appended record queries back to -source-local indexes, and supports semantic indexed helper paths. Tests cover -appended and independent collections, origin attribution, detached paths, -explicit empty-list replacement, ordinary whole-list replacement, and tier -handoff. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'History|Explain|Overlay' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 5 — Integrate projected paths with inspection and sensitivity - -Scope: metadata, retained history traversal, and safe inspection. - -- Adapt `finalizeMetadataPaths`, application-value traversal, explanation queries, - and secret classification to Stage 4's projected operations. Do not coerce - indexed application prefixes into mapping keys. Reuse a small common projected - traversal/path helper where these consumers need the same translation. -- Preserve independent effective and historical indexes, numeric key/index shape - distinctions, removed paths, detached query results, and current sensitivity - propagation. Classify original historical shapes before aggregating sensitivity - by pointer; do not repeat the previously fixed shape-collision defect. -- Ensure sensitive fields on appended records are classified at their assigned - indexes and stay safely handled after replacement. Apply the same projection - to source-version sensitivity evidence when that evidence is integrated in - Stage 10. Do not classify a fragment solely at its unshifted source index. -- Add focused tests for shifted-index metadata/history/rendering agreement and - a replaced sensitive record. Reuse existing numeric-shape, removed-descendant, - ownership, inheritance, and redaction cases where they already protect behavior. - -Validation: metadata/provenance/history/explanation/secret tests, -`go test -race ./internal/engine`, and `./scripts/check.sh`. -Completion: the private registered composer can feed existing inspection safely. - -Completion evidence: history applications now expose one source-local/effective -path traversal used by metadata enumeration, explanation replay, origin lookup, -and secret classification. Appended records and nested descendants use assigned -indexes while retained nodes and source locations remain local. Secret -classification evaluates projected historical shapes before folding sensitivity -by pointer, preserving numeric key/index distinctions and redacting replaced -records. Tests cover shifted metadata/history/rendering, origin attribution, and -redaction after replacement. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Metadata|Provenance|History|Explain|Secret' -count=1` -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./internal/engine` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 6 — Refactor acquisition into composition scopes without new behavior - -Scope: `imports.go`, `load_input.go`, `load_request.go`, and their handoffs. - -- Replace the assumption that traversal yields only a flat slice of already - placed documents with ordered contribution/scope records. At this stage all - scopes are ordinary mappings; preserve current placement and traversal exactly. -- Separate acquisition facts from completed composition and retained history. - Keep source-local payloads, physical locations, resource contexts, relative - destinations, processing results, and source order available to later stages. - Preserve input-specific filesystem contexts, tier-local repetition tracking, - active import stacks, and load-wide shared accounting. -- Stop reconstructing underlay contributions by treating every history - application's `value` as a complete root document. That current shortcut loses - projected paths and append semantics. Compose acquired contributions at the - group boundary while preserving their source/history evidence. -- Avoid recursive public `Load` calls, reopening files, a second parse pass, - per-helper defaults/hooks, or resetting import limits and active/visited state. - Preserve acquisition-first main behavior and all close/error paths. -- Prefer existing ordinary, relative-destination, underlay, namespace, repetition, - budget, and handle-ownership tests as the regression oracle. Add a test only for - a real handoff risk not already covered. - -Validation: import/destination/grouped/source-version/resource-context tests and -`./scripts/check.sh`. Completion: this is a behavior-preserving private refactor. - -Completion evidence: acquisition now retains ordered ordinary-mapping scopes -with source-local payloads, placed contributions, destinations, source records, -and source-version processing results. Underlay groups compose those acquired -contributions at their boundary instead of treating retained history values as -complete roots. Existing filesystem contexts, import traversal/repetition -state, load-wide accounting, hooks, and lifecycle handling remain in their -existing paths. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Destination|Import|Grouped|SourceVersion|Resource' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 7 — Add the private item-placement operation and accounting - -Scope: placement primitives and engine per-input options; not public exposure. - -- Add the private placement mode to engine input/request/options/source records, - with preflight validation. Keep mapping placement unchanged. A valid item mode - needs a nonempty mapping-key destination relative to its current mapping scope. -- Carry the destination-resolution state in engine source records so later - scoped acquisition can distinguish exact containers from unresolved fallbacks. -- Extend placement to wrap one completed source mapping in a singleton sequence - at that destination. Preserve mapping-root input validation; do not accept raw - source sequence/scalar roots. A nil payload stays absent, and `{}` contributes - an actual empty item. Do not require a composition registration here. -- Account for the sequence wrapper plus existing mapping wrappers and full - mounted depth. Check limits before allocating oversized wrappers. Carry node - accounting through completed scopes without charging acquired content twice. -- Test mapping default equivalence, explicit item placement, nested literal - destinations, absent versus empty payloads, invalid/root placement, and exact - versus exceeded private depth/node allowances. Use test-controlled limits. - -Validation: placement/accounting/option tests and `./scripts/check.sh`. -Completion: the private operation places a mapping once with correct ownership. - -Completion evidence: added private placement modes to engine options, load -requests, and source records, with preflight rejection of unknown modes and -item placement at an empty destination. A completed mapping can now be wrapped -in one singleton sequence at its destination; nil payloads remain absent and -empty mappings remain real items. Wrapper accounting includes both destination -mapping nodes and the added sequence depth/node, with limits checked before -allocation. Focused tests cover mapping zero-value equivalence, nested paths, -empty and absent payloads, invalid modes/destinations, and exact/exceeded -allowances. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Destination|Import|Mounted|PlaceDocument|Options|Layered' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 8 — Assemble item scopes and YAML path/glob imports - -Scope: directive parsing and scoped acquisition/composition. - -- Parse the fixed `placement` syntax and report malformed values safely. Each - declaration's default is mapping mode; item mode is never inherited. Apply the - same mode independently to every glob match in existing sorted traversal order. -- An item source opens a local mapping scope. Its body and ordinary helpers - compose strictly within that mapping, with omitted/empty helper destinations - at the item root and other destinations relative to the immediate importer. - A helper can supply the identity; a duplicate identity field in the item still - fails ordinary mapping composition. -- Complete child item scopes before wrapping them into their enclosing scope. - Compose each scope once, preserving declaration/contribution order and source - history. Explicit nested item imports create nested sequences; registrations - do not descend through the enclosing record index, so repeated contributions - to those unregistered nested sequences conflict atomically. -- Pass completed records and their relative history/origins to the parent - composer. Assign outer ranges there, then project the retained item body and - helper events through those ranges. Do not emit each ordinary helper as a - separate record or validate its identity independently. -- Maintain source destination-resolution state as enclosing ranges are assigned. - Before assignment, source snapshots use the explicitly marked fallback from - the fixed design; after assignment, project the exact container. Preserve - already-returned snapshots rather than updating them in place. -- Cover path and glob item imports, a helper-provided ID, relative and transitive - helper destinations, mixed authored lists/items, optional unmatched imports, - empty items, nested placement, default-mode independence, and physical paths - relative to the declaring helper. Exercise native filesystem and host imports - through existing shared-mechanism coverage rather than duplicating every case. -- If the new syntax is now reachable through ordinary public import loading, - document that implemented placement subset in `docs/config.md` in this unit; - do not advertise still-private registration controls yet. - -Validation: import/directive/glob/destination/composition tests and -`./scripts/check.sh`. Completion: private item imports construct the agreed -records and preserve physical acquisition evidence. - -Completion evidence: import declarations now parse per-declaration placement, -and item imports compose their body and ordinary helpers in a local mapping -scope before wrapping the completed record. Nested item scopes, glob matches, -relative and transitive helper destinations, source origins/history, and -resolved underlay destinations are retained. Configuration documentation -describes only the implemented import-placement subset; private registration -controls remain unadvertised. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'Import|Directive|Glob|Destination|Compose|NativeFilesystem' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 9 — Wire strict policy through the complete loading lifecycle - -Scope: ordinary/grouped engine loading and every strict-tier entry point. - -- Pass compiled registrations and placement mode through main acquisition and - each underlay group. Validate an authored registered list in an ordinary - one-source load without imports, overlays, or CLI; no composition shortcut may - bypass it. Skip only genuinely absent contributions. -- Assemble and validate each complete strict tier before precedence can replace - it. Reset identity maps between tiers and preserve current acquisition order. - A malformed lower-tier list must fail even when main or overlay would replace - it. Avoid running the checker a second time on the overlaid final tree. -- Keep unregistered item lists atomic. Selected overlays, CLI replacements, - defaults, callbacks, and sibling inheritance retain their current behavior. - Do not require the final list to retain unique or even present identities. -- Add compact engine lifecycle cases for main, an underlay group, separate tiers - sharing IDs, `[]` clearing, unregistered conflict, a hidden lower-tier error, - and duplicate IDs introduced by a replacement overlay/CLI/callback. Verify - normal application validation can reject such final values when configured. - -Validation: load/grouped/overlay/override/resolution/default tests and -`./scripts/check.sh`. Completion: private engine policy is applied at strict -composition boundaries only, including the one-source case. - -Completion evidence: compiled sequence policy now reaches ordinary input -composition, imported scopes, and each underlay group. Registered sequences -are checked even for one-source loads and before tier replacement; identity -state resets at tier boundaries, while replacement overlays, CLI values, and -recorded resolver results retain their existing behavior. Lifecycle tests cover -grouped composition, empty-list replacement, unregistered conflicts, malformed -lower tiers hidden by main or overlay replacement, duplicate replacements, and -application validation. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SequenceRegistration|Import|Grouped|Overlay|Override|Resolution|Default' -count=1` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 10 — Preserve source-version and structural ownership under placement - -Scope: version path projection and existing protected-path checks. - -- Keep source-version checking on each original source before placement, exactly - once using the current local rule and callback lifecycle. Required/if-present - modes retain main absence and helper-document behavior. Accepted non-main - headers are consumed before helper/record identity validation; main's header - stays application content. Helpers do not inherit their parent's header. -- Default mode continues checking the configured effective path without prefix - reinterpretation. Enabled modes derive main's protected effective path from - its actual projection, including item indexes. Update both the request and - version state's effective rule before later protection checks need it. -- Audit `effectiveSourceVersionRule`, `fragmentVersionClaim`, import/underlay/ - overlay checks, override checks, inheritance checks, and resolution auditing. - Mapping-only navigation or ancestor-sequence shortcuts cannot protect the - wrong record or reject an unrelated strict peer merely because it appends to - the same collection. Preserve protections of the actual main path and its - destructive ancestor replacements; do not add an exception to version policy. -- Retain source-local public VersionCheck paths, physical locations, full absent - paths, and conservative failure disclosure. Use assigned projections for - sensitivity checks of headers consumed before the final tree exists. Keep - original semantic key/index shapes and remap nested item/helper prefixes once. -- Keep import opt-in checks on original sources before consumed headers can hide - a directive. Identity lookup must not count consumed headers or structural - controls as application fields. Do not add a blacklist of identity-field names - or use later defaults/inheritance/resolution to supply an absent identity. -- Test main item placement in both entry points, default versus enabled version - coordinates, consumed helper/import headers, protected-main versus unrelated - peer paths, safe shifted evidence, optional absence below controls, and normal - inheritance inside item mappings. Reuse existing source-version and inheritance - regressions rather than rewriting their selected contracts. - -Validation: source-version/version/inheritance/override/resolution tests, -`go test -race ./internal/engine`, and `./scripts/check.sh`. -Completion: item placement changes coordinates, not version or structural policy. - -Completion evidence: enabled source-version rules now derive their effective -paths through the main item's assigned index, while default-mode coordinates -remain unchanged. Structural import and underlay checks use assigned sequence -ranges so they protect the actual main item without rejecting unrelated peers; -source-local checking still runs once per original source, and projected header -evidence is used for sensitivity. Tests cover ordinary and layered placement, -protected versus peer indexes, nested projections, and redacted version -evidence. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SourceVersion|Version|Inheritance|Override|Resolution' -count=1` -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./internal/engine` -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 11 — Expose root-owned controls and detached placement evidence - -Scope: root facade declarations, conversions, and public integration tests. - -- Add the exact public types/fields specified above, with declaration comments - describing defaults, option failures, and ownership. Convert Definition - registrations and paths, ordinary Options destinations/modes, layered Inputs, - and successful/failed SourceInfo records in both directions as applicable. - Convert `DestinationUnresolved` alongside `Destination` and `PlacementMode`. - Update callback-entry source conversions too, not only `Result.Sources`. -- Preserve immutable source snapshots, reader single-use behavior, callback - lifetime, independent loads, and no public exposure of engine/parser types. - SourceInfo reports its own selected mode; CLI/overlay sources use mapping mode - unless an actual supported item placement was applied. -- For a completed load, resolve nested source containers through their owning - item projection. Apply the fixed unresolved-destination contract to callbacks - and early failures: a fallback container always has `DestinationUnresolved` - set to true; an exact container has false. Failed-load records describe - acquisition context, not a usable partial configuration. Do not add a public - index or identity-address API or promise resolution of every failed source's - final placement. -- Test an unresolved helper record exposed during a callback and on early - failure, its exact projected destination after assignment, and a failure after - assignment that retains the exact destination. Distinguish an exact item-root - container from a helper fallback with the same path. Verify nested-item - fallback, preserved snapshot ownership, and false for unchanged mapping-only - sources. Reuse fixtures where these scenarios share the same lifecycle path. -- Public tests demonstrate ordinary and layered opt-in, an item-import glob, - authored-list/item interoperability, one-source validation, helper origins and - resource contexts at shifted indexes, both option-error entry points, detached - source/registration paths, and unchanged zero-value callers. Keep mechanism - edge cases in engine tests. -- Update `docs/config.md` for the now-public registration controls, ordinary - destination options, complete placement/evidence behavior, and strict-versus- - replacement validation boundary. Document `DestinationUnresolved` and its - fallback meaning in both `SourceInfo` comments and the owning configuration - contract, including callback and failure snapshots. Update the architecture - policy's composition summary to recognize the registered-sequence rule and - link ADR 0014. Stage 13 reviews and completes the surrounding guidance, not - these canonical contracts. - -Validation: focused facade tests, -`go test . -run 'TestPublicFacadeBoundary|TestDependencyImportBoundary'`, -`go test -race ./...`, and `./scripts/check.sh`. -Completion: both public entry points expose the complete feature with the same -composition rules and independently owned evidence. - -Completion evidence: the root facade now exposes sequence registrations, -ordinary and layered placement controls, and detached placement-mode and -destination-resolution source evidence. Public integration coverage exercises -ordinary/layered placement, strict registration with authored and glob-imported -items, shifted origins and resource contexts, option preflight, early and -post-placement failure snapshots, nested fallback containers, registration and -snapshot ownership, and zero-value mapping behavior. The existing resolver -callback runs after item projection, so its source snapshots are exact; the -unresolved helper fallback is observable on early failure. No acquisition-time -callback exposes `SourceInfo`, and no additional callback lifecycle was added. -These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test . -run 'TestPublicFacadeBoundary|TestDependencyImportBoundary' -count=1` -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...` -- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 12 — Close cross-cutting regression and limit gaps - -Scope: risks that require the assembled feature, not duplicate mechanism tests. - -- Review Stage 1–11 coverage against the roadmap's ten settled decisions. Add - only missing consequential cases. Use a representative completed load to check - typed output, explicit zero/false/null/empty versus omission, effective origins, - callback-entry metadata, resource contexts, history, explanation, and rendering - agree after an append followed by a replacement. -- Verify item/helper source canaries do not leak through duplicate diagnostics, - errors, explanations, effective rendering, or consumed version evidence under - existing sensitivity rules. Preserve historical redaction after index shifts - and later removal/replacement, without imposing identity-based final masking - rules that the existing model does not have. -- Exercise shared aggregate accounting across item scopes/globs/underlays and - retained-history accounting of the new operations. Confirm immutable fragments - are counted according to existing shared-node rules and no retained cumulative - sequence snapshots multiply history storage. Use existing close/reader seams - for one representative nested-item failure, not a second filesystem framework. -- Confirm effective Diff still reports atomic sequence changes and unchanged - defaults/resolution/inheritance semantics. Review the implementation for repeated - ID scans, growing-list clones, and helper-subtree recomposition. No timing or - allocation threshold assertions, benchmark campaign, or new release gate is - required; add local performance evidence only if a concrete concern needs it. - -Validation: targeted new regressions, `go test -race ./...`, and -`./scripts/check.sh`. Completion: no known evidence, ownership, or accounting gap -remains hidden behind a correct final tree. - -Completion evidence: cross-cutting regressions now cover a typed main-tier -sequence replacement after underlay item appends, including presence and -zero/null distinctions, callback metadata and resource context, final origins, -removed-item explanation, and historical redaction in explanation and effective -rendering. Additional tests verify shared aggregate node accounting across a -nested item glob and sibling contribution, native file closure after nested item -composition failure, and retained sequence-history applications storing source -fragments rather than cumulative snapshots. Existing sequence placement, strict -diagnostic, version-evidence, and rendering tests continue to cover the other -settled risks. These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...` -- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 13 — Document implemented contracts and provide a maintained example - -Scope: consumer documentation and a tested in-repository adoption fixture. - -- Update `docs/config.md` as the canonical owner for import syntax, placement, - registered strict composition, validation timing, replacement boundaries, - structural/version processing, source containers, and history semantics. Link - declaration comments for API details rather than duplicating them everywhere. -- Update `docs/policy/architecture.md`'s composition summary so its unconditional - non-mapping-conflict description recognizes the explicitly registered sequence - rule. Preserve consistency, ownership, and precedence principles; this feature - does not authorize arbitrary configurable merge strategies. Link the new ADR - and current contract. Preserve upstream provenance and locally significant - policy decisions. -- Update `README.md`, `docs/consumers/README.md`, `docs/internal/overview.md`, and - `docs/development.md` only to summarize/navigate the implemented feature at - their respective ownership boundaries. Update `future.md` to distinguish the - implemented subset from still-potential broader identified-record features; - do not promote their deferred semantics into this feature's contract. -- Add a maintained fixture/test under `examples/consumers` showing independently - authored prompt-shaped records, explicit item path/glob imports, reusable - registration, and a lower-priority default collection replaced by the main - list. Show that a helper can supply an ID and that final domain validation is - consumer-owned. Use in-repository disposable data, not a live PromptKit - dependency or a second application-side merge. -- Keep ADR 0014 decision text immutable after acceptance; update status/navigation - to implemented when appropriate. Do not rewrite earlier ADR decision text to - make it appear they anticipated the extension. - -Validation: executable example/consumer tests, local documentation-link tests, -whitespace review, and `./scripts/check.sh`. -Completion: current docs and tested public examples describe what actually lands. - -Completion evidence: the canonical configuration contract and architecture -policy now describe placement, registered strict composition, version/header -boundaries, source containers, history, and unchanged precedence. README, -consumer, development, and internal guides link to those contracts and the -maintained prompt-catalog example. The example uses disposable in-repository -fixtures to demonstrate path/glob item imports, a helper-supplied identity, -strict composition, whole-sequence replacement of defaults, and consumer-owned -validation. The feature roadmap and ADR status point to the implemented contract -and delivery record; broader composite-identity behavior remains deferred. -These checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./examples/consumers` -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...` -- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh` -- `git diff --check` - -No environmental limitation affected validation; checks ran on Linux/amd64. - -## Stage 14 — Review the completed unit and hand off - -- Read the complete implementation diff against `2051177` and the feature - roadmap. Check all ten settled decisions and the two independent opt-ins. - Look especially for validation accidentally extended into replacement-only - layers, final uniqueness/schema promises, new field restrictions, wrong - source indexes, and stale acquisition/history shortcuts. -- Run `go test -race ./...` and `./scripts/check.sh` on the completed candidate. - Reuse unchanged successful checks where appropriate; document actual results - and any unresolved issue rather than claiming an unrun consumer verification. -- Record stage completion and a concise final validation summary in this plan, - then commit the completed nonempty unit under the normal commit policy. - Do not create an empty completion commit or delete unfinished plans. -- Report the new public controls, preserved precedence/ownership behavior, - validation, and any remaining work. This sprint does not select/tag/push a - release, alter release gates, update Go/dependencies, or update downstream - repositories. Separately authorized release preparation can assess the public - field additions and feature under the existing release policy. - -Completion: the repository is clean, the feature and its evidence agree, and no -known in-scope implementation work remains. No product decisions remain open; -private helper names and equivalent mechanics are ordinary implementation work. - -Completion evidence: final review covered the complete implementation diff from -baseline `2051177` against the settled behavior and public-contract boundaries. -It found one malformed-record diagnostic using a contribution-local rather than -assigned sequence index. The composer now reports the effective assigned item -path, with a regression for a malformed record in the second contribution at -`/records/1`. Review also confirmed registered identity checks remain limited to -strict composition, precedence still replaces complete sequences, both opt-ins -are explicit, and source ownership, physical locations, contexts, and detached -evidence remain intact. Final checks passed with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'ComposeDocuments(ReportsAssignedPathForMalformedRegisteredSequenceContribution|ValidatesRegisteredSequenceSingletons|RejectsRegisteredSequence)' -count=1` -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...` -- `GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh` -- `git diff --check 2051177` - -No environmental limitation affected validation; checks ran on Linux/amd64. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index fd2e77f..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,353 +0,0 @@ -# Sequence Composition Review Remediation - -Status: complete; all nine findings addressed in separate validated commits. -Reviewed implementation `2051177..0858af1` -(`0858af1029d07d93a5aa2e4f26cbec61fcfb3a51`) against the -[feature roadmap](sequence-composition.md), [implementation plan](implementation.md), -and accepted [ADR 0014](../adr/0014-sequence-placement-and-composition.md). - -The root-owned facade, explicit placement, definition-owned registrations, -item/helper scopes, and strict-versus-precedence distinction are implemented. -The findings below concern correctness and evidence at their boundaries, plus -targeted scaling and documentation gaps. They do not call for new features, -stricter final identity validation, broader version policy, or new release gates. - -## 1. Project consumed version evidence through authored sequence appends — P1 - -- [x] Addressed and regression-tested. - -Completion: retained sensitivity paths now project declarations and ancestors -through assigned authored-list ranges without changing source-local evidence. -Public regressions cover imports and grouped underlays, leaf/container masking, -and unclassified declarations. Focused source-version tests, the everyday check, -and `go test -race ./internal/engine` passed. - -**Owners:** `projectAcquiredDestinations` / `projectSourceVersionEvidence` in -[imports.go](../../internal/engine/imports.go), and evidence classification in -[source_version.go](../../internal/engine/source_version.go) and -[secret.go](../../internal/engine/secret.go). - -Version evidence is projected for explicitly item-placed sources and their -helpers, but not for an ordinary source containing an authored registered list. -Its retained source-local path can contain an index shifted by composition. -Concatenating only `evidence.destination` and that path classifies the wrong -effective record and can publish sensitive declaration text. - -**Confirmed reproduction:** main contains `records: [{id: main, version: main}]` -and imports a mapping containing -`records: [{id: child, version: private-version-canary}]`. Register `/records` -by `id`, select required source-version checks at -`Keys("records").Index(0).Key("version")`, and classify -`/records/1/version` as sensitive. Loading succeeds and consumes the child's -header, but its successful `SourceInfo.VersionCheck` exposes the canary with -`Redacted: false`. Its record is at index 1; evidence was classified at index 0. - -**Required change:** project internal evidence paths, including applicable -ancestors, through assigned authored-sequence ranges as well as item scopes. -Keep public VersionCheck paths and physical locations source-local. Preserve -original semantic shapes and existing conservative failure/callback disclosure; -do not solve this by masking every declaration or changing source-version policy. - -**Acceptance:** cover an imported authored list and the corresponding grouped -underlay handoff. Sensitive shifted evidence is redacted; an unclassified -declaration remains inspectable. Existing item/helper evidence tests still pass. - -## 2. Preserve source chronology when expanding item history — P2 - -- [x] Addressed and regression-tested. - -Completion: structural fragments use their local singleton index; item body and -helper events are expanded beside their declaring contribution. Mapping history -queries ignore unrelated sibling fields. A public regression covers preceding -authored records, two item sources, nested helpers, field origins, and root -traversal order. Focused facade/history tests and the everyday check passed. - -**Owner:** `projectNestedEvidence` in -[imports.go](../../internal/engine/imports.go), especially lines 592–608. - -The structural source fragment is indexed with the effective append offset: -`node.Sequence[start]`. That fragment is still a source-local singleton, so a -later item's `start > 0` prevents clearing its completed body. History therefore -retains helper values in the outer application and subsequently replays the same -item body and helper values again. Nested applications are also appended after -all outer applications rather than beside their declaring contribution. - -**Confirmed reproduction:** a manifest imports `one.yml` and `two.yml` as items -at registered `/records`. Each item declares its own ordinary helper supplying -only `text`. The second item's `Explain(/records/1/text)` reports three events, -including an unchanged-value event attributed to the item root that never -authored `text`; the first item's explanation has only its helper contribution. -Root history visits source IDs `1,2,4,2,3,4,5`, placing the second item ahead of -the first item's helper despite depth-first acquisition `1,2,3,4,5`. - -**Required change:** distinguish local indexes from effective offsets when -constructing structural fragments. Expand each item's retained applications in -source traversal order, preserving physical field origins without replaying a -completed body as an additional source contribution. Do not replace individual -source history with an opaque completed-scope snapshot. - -**Acceptance:** use at least two item sources with helpers and a preceding -authored-list contribution. Helpers alone define their fields in explanations; -root/container history is chronological, final values/origins agree, and nested -items retain the same guarantees. - -## 3. Record removals caused by registered-sequence replacement — P2 - -- [x] Addressed and regression-tested. - -Completion: replacement queries include missing descendants and use the replacing -container's origin. Public regressions verify shorter lists, empty lists, and -removed fields; existing append-relevance tests remain green. Focused facade -tests and the everyday check passed. - -**Owner:** `applicationOperationPath` in -[history.go](../../internal/engine/history.go), lines 134–149. - -Query relevance filters every sequence operation to the incoming assigned range, -including `sequenceHistoryReplace`. This is appropriate for an append but not a -replacement: replacement also removes records outside the incoming range. -Replay removes them correctly while Explain silently omits the removal event. - -**Confirmed reproduction:** a strictly composed underlay supplies records -`[{id: one}, {id: two}]`; main supplies `[{id: main}]` at the same registered -path. `Explain(/records/1/id)` says the value is absent, but its only history -event defines `two` in the underlay. No main-tier removal appears. Replacing with -`[]` has the same problem. Existing cross-cutting tests check absence/redaction -but do not assert that the removing operation is actually recorded. - -**Required change:** make replacement relevant to removed descendants and -attribute those transitions to the replacing container/source when an incoming -leaf no longer exists. Keep append relevance limited to newly contributed items. - -**Acceptance:** explanations show present-to-absent transitions and the correct -replacing origin for shorter/empty lists and removed fields; ordinary same-tier -appends do not falsely claim to change earlier records. - -## 4. Honor contextual parsed version-path indexes — P2 - -- [x] Addressed and regression-tested. - -Completion: protected-path sequence traversal uses the existing canonical -contextual index interpreter. Public import and grouped-underlay parity tests -cover parsed and explicitly indexed rules; existing version-path tests preserve -numeric-key intent and invalid index behavior. Focused tests and the everyday -check passed. - -**Owner:** `fragmentVersionClaimProjected` in -[directives.go](../../internal/engine/directives.go), lines 190–194. - -Sequence traversal accepts only `pathSegmentIndex`. Canonical numeric tokens -from public `ParsePath` are contextual parsed segments, accepted by the existing -version lookup and other semantic-path operations. The new protection code -instead treats them as a claim on the entire ancestor sequence. - -**Confirmed reproduction:** use the two-source required-version fixture from -finding 1, without the sensitivity predicate. A rule built with -`Keys("records").Index(0).Key("version")` succeeds. The equivalent -`ParsePath("/records/0/version")` fails with `import.version`, even though the -import appends to index 1 and does not claim main's index 0 header. - -**Required change:** use existing contextual sequence-index interpretation while -preserving explicit mapping-key versus index intent and canonical token rules. -Do not narrow VersionRule to programmatically indexed paths or relax real -ancestor-replacement protection. - -**Acceptance:** equivalent parsed/indexed rules give equivalent ownership -results across imports and grouped underlays, with existing literal numeric-key, -wrong-container, and noncanonical-index cases preserved. The current divergence -is an unintended restriction beyond the requested feature. - -## 5. Keep default-mode main version ownership inside item scopes — P2 - -- [x] Addressed and regression-tested. - -Completion: default-mode effective protection is translated into the main item's -local scope, preserving segment intent and effective diagnostic paths. Both -public entry points reject helper-supplied main declarations and permit unrelated -version fields. Focused source-version tests and the everyday check passed. - -**Owner:** the `mainIsItem` branch of -[imports.go](../../internal/engine/imports.go), lines 228–235. - -The helper ownership check applies `versionState.localRule` directly to -item-local documents. In default mode that rule is an effective application -path, not a local item path, so the check misses a helper's protected contribution. - -**Confirmed reproduction:** load main as an item at `/records`, enable imports, -and configure the default-mode rule at -`Keys("records").Index(0).Key("version")`. Main contains only `id` plus an -ordinary helper import. The helper supplies `version: imported-version`. -Loading succeeds and the main checker receives the helper's version, although -imported fragments are not allowed to supply the protected main version path. - -**Required change:** translate the protected effective path into the item scope -before checking its individual helper contributions, or check correctly -projected contributions at the enclosing boundary. Retain default effective-path -interpretation and enabled-mode source-local checking/header consumption. - -**Acceptance:** the fixture fails with `import.version` in both public entry -points. Helpers outside the actual protected path remain permitted, and enabled -source-version-mode behavior is unchanged. Do not add a general ban on fields -named `version`. - -## 6. Preserve absence for ordinary defaults-only loads — P2 - -- [x] Addressed and regression-tested. - -Completion: absent ordinary input stays absent until an existing explicit layer -needs assembly. Public parity tests cover values, root presence/origin, optional -registrations/defaults, and real empty mappings. Existing defaults-only override -coverage and the everyday check passed. - -**Owner:** `executeLoadRequest` in -[load.go](../../internal/engine/load.go), lines 210–216, and -`compositionForInputWithPolicy` in [load_input.go](../../internal/engine/load_input.go). - -Ordinary loads now unconditionally initialize a composition, including when -`NoSource()` supplied no document. Its synthetic empty mapping is assigned back -to `input.document` and decoded as explicit input. This changes callers that -have not opted into either feature. - -**Confirmed reproduction:** `Load[map[string]any](NoSource())` reports -`Metadata().IsExplicit(Keys()) == true`, while the equivalent `LoadLayered` call -reports false. No source contributed a root definition. This violates the -existing NoSource absence distinction rather than adding useful feature behavior. - -**Required change:** retain the absent-input handoff when no real contribution -exists, while still validating actual singleton registered sequences. Preserve -synthetic assembly only where existing overlay/CLI behavior needs it. - -**Acceptance:** ordinary/layered defaults-only loads agree on root presence and -origin, with and without registrations/default factories; an actual empty source -mapping is still explicit. Defaults-only overrides keep their existing behavior. - -## 7. Remove new quadratic scans from sequence construction — P2 - -- [x] Addressed and behaviorally validated. - -Completion: claims retain encountered immutable nodes; history construction -groups ranges by source; scope/version projection uses source/path range -lookups; retained-history traversal and replay use per-application path indexes. -Ordered slices and detached clone ownership remain intact. Static review confirms -the listed nested scans are removed. A wide-collection replay/origin/ownership -regression, the everyday check, and `go test -race ./...` passed. No timing -threshold or release gate was added. - -**Owners:** [compose.go](../../internal/engine/compose.go), especially -`validateRegisteredSequences` and application construction; range lookups in -[imports.go](../../internal/engine/imports.go); `sequenceHistoryAt` in -[history.go](../../internal/engine/history.go). - -Identity comparison uses a map, but surrounding new machinery reintroduces -avoidable nonlinear work: - -- Validation looks each sequence claim up again with unindexed `documentAtPath`. - One source with N sibling registered sequences performs N linear root-mapping - searches: O(N²) at fixed depth and constant record size. -- Application construction scans all ranges for each source: O(S × R). Item - evidence/source projection repeats full range scans for each contribution. -- `sequenceHistoryAt` linearly scans the application's sequence descriptors - repeatedly during retained-tree traversal and replay. A wide document with - many registered sequence paths similarly accumulates quadratic work. - -These are structural findings; no timing benchmark claim is made. They diverge -from the plan's near-linear construction objective and the user's explicit -efficiency priority, without requiring that public behavior change. - -**Required change:** retain encountered node references or reuse existing indexed -document lookup; group/index ranges by source and path once; use a small owned -per-application path lookup where repeatedly needed. Keep order in slices and -ownership unchanged. Avoid a new global cache/index framework or unrelated -performance refactoring. - -**Acceptance:** static review no longer finds these nested whole-set scans on -the new construction path. Existing ordering, evidence, cloning, and behavior -tests pass; optional scaling measurements may help but are not a new gate or -timing-threshold test requirement. - -## 8. Do not fabricate index zero for item-local conflict diagnostics — P2 - -- [x] Addressed and regression-tested. - -Completion: failed item composition is projected through each enclosing scope -using preceding contributions, including prior grouped inputs. Only composition -diagnostics are rebased; physical locations and source-local access/syntax -diagnostics are retained. Regressions cover authored/imported predecessors, -nested items, item mains, grouped item sources, and failure evidence. Focused -facade/engine tests and the everyday check passed. - -**Owner:** `visitItemSource` in -[imports.go](../../internal/engine/imports.go), lines 519–525. - -Helper composition errors are rebased with `destination.Index(0)` before the -item's actual range is assigned. Nested rebasing also omits an enclosing item -index. Physical locations are retained, but the application path points at a -different record. - -**Confirmed reproduction:** main authors `records: [{id: first}]` and imports -an item with `id: second` whose ordinary helper also defines `id`. The conflict -is reported at `/records/0/id`, incorrectly naming the existing first record; -the imported contribution would be at index 1. - -**Required change:** carry scoped diagnostic paths until they can be projected -through the enclosing assignment, rather than injecting a fabricated index. -Preserve safe failure source evidence and actual physical conflicting locations; -do not require publishing a usable partial result to compute a diagnostic. - -**Acceptance:** a helper conflict after preceding authored/imported records names -the correct item, and nested item conflicts retain enclosing semantic indexes. -The recently added malformed-registered-record regression must continue to pass. - -## 9. Correct the public identity-validation phase statement — P3 - -- [x] Documentation corrected. - -Completion: the public declaration distinguishes registration preflight from -input identity validation during strict composition, matching the existing -configuration contract. The final everyday check and full race suite passed. - -**Owner:** `Definition.SequenceCompositions` comment in -[lifecycle.go](../../lifecycle.go), line 20. - -The comment says repeated identities and conflicting registrations both fail -option validation. Only invalid/conflicting registrations are preflight errors; -duplicate identities in input are `compose.conflict` after source acquisition -and applicable helper/header processing. The declaration comment inadvertently -promises an earlier failure phase than the approved design or implementation. - -**Required change:** distinguish registration preflight from record validation -during strict composition. Correct the comment rather than moving data checks -earlier or adding a new pre-acquisition guarantee. Keep the canonical -configuration contract and declaration comments consistent. - -## Review validation - -Passed on the reviewed tree with Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...` -- `git diff --check 2051177..HEAD` - -The behavioral findings were independently reproduced through public API calls -using disposable in-memory filesystems in a temporary Go program outside the -repository. The scaling finding is based on the inspected loops and lookup -implementation, not host timing. No production code, tests, dependencies, -release machinery, or downstream repositories were modified during this review. - -## Remediation validation - -All nine findings have completion evidence above. Final validation passed with -Go `go1.26.9 linux/amd64`: - -- `GOTOOLCHAIN=local ./scripts/check.sh` -- `GOTOOLCHAIN=local GOPROXY=off go test -race ./...` -- Local documentation-link and whitespace checks. - -The original external public-API reproductions were rerun against the completed -changes: consumed sensitive evidence is redacted, helper events are chronological -and source-specific, replacements explain removals, parsed/indexed version rules -agree, imported main-version claims fail, NoSource stays absent, and item conflict -paths identify the correct records. Static review and the wide-collection -regression verify the targeted lookup changes without adding performance -thresholds. No dependencies, public controls, release gates, or downstream -repositories were changed by remediation. diff --git a/docs/roadmap/sequence-composition.md b/docs/roadmap/sequence-composition.md deleted file mode 100644 index 05b060c..0000000 --- a/docs/roadmap/sequence-composition.md +++ /dev/null @@ -1,318 +0,0 @@ -# Sequence-Item Placement and Identity-Based Composition - -Status: implemented. This roadmap records the target state and accepted behavior. -The [configuration contract](../config.md#strict-document-composition) defines -the current behavior, and the [implementation plan](implementation.md) records -delivery sequencing and validation. - -## Goal and current baseline - -Let consumers assemble a list of records from independently authored mapping -files. Placement makes a source mapping one sequence item; identity-based strict -composition combines distinct items and rejects repeated identities. These are -complementary operations, with explicit consumer opt-in. - -Sources can explicitly place mappings as sequence items at non-root destinations, -and definitions can register literal sequence paths with one literal string -identity field for duplicate-rejecting strict composition. Ordinary unregistered -sequences retain strict conflict behavior. [Precedence assembly](../config.md#layered-loading) -and [selected overlays](../config.md#selected-overlays) continue to replace -sequences wholesale. The canonical contract documents the exact rules and limits. - -This roadmap covers a narrower feature than the -[identified record sequences candidate](future.md#1-identified-record-sequences). -Composite identities, recursive merging of matching records, identity-based -overlays, and record inheritance are not requested here. That scope statement -does not establish permanent architectural prohibitions. - -## Targeted behavior - -### PromptKit use case - -Each file defines one prompt directly. - -`prompt_one.yml`: - -```yaml -id: prompt.one -text: "foo" -``` - -`prompt_two.yml`: - -```yaml -id: prompt.two -text: "bar" -``` - -The consumer selects item placement at `prompts` for both sources and registers -`id` as that sequence's identity field. Within one strict composition tier, the -result is: - -```yaml -prompts: - - id: prompt.one - text: "foo" - - id: prompt.two - text: "bar" -``` - -The files need neither a filename-derived wrapper key nor their own list wrapper. -Existing sources that already contain `prompts: [...]` can contribute to the same -registered sequence under the same composition rule. - -### Explicit sequence-item placement - -- The destination identifies where content belongs; an explicit placement mode - selects ordinary mapping placement or placement as one sequence item. Preserve - existing behavior when the new mode is not selected. Do not infer the mode - from document shape, filename, or the presence of an `id` field. -- Support item placement for application-supplied inputs and YAML path/glob - imports. Each matched file of an item-placement glob contributes one item, - using existing match and traversal ordering. Physical import-path resolution - remains relative to the declaring file. -- Wrap the source's application mapping as one item at the destination sequence; - the new operation is not flattening a source sequence or splitting a mapping - into several items. Existing supported source forms should receive equivalent - placement behavior; this does not expand accepted source-root shapes. -- Item placement supplies structure, not identity values. Keep authored `id` and - other fields in the item. Source directives and version headers retain their - own processing rules, with the item-placement interpretation recorded under - Settled Decisions. -- Preserve existing absence distinctions: `NoSource()` and an unmatched optional - import contribute nothing. An actual empty mapping contributes an empty item, - which fails the agreed identity requirement when that item participates in a - registered composition sequence. - -Item placement is independent of composition registration. Ordinary imports -inside an item compose into its mapping; public source evidence reports the -placement mode and destination container. The settled decisions below define -these behaviors in detail. - -### Identity-based strict composition - -- Consumers designate sequence paths and one direct string identity field for - each. Unregistered sequences retain their current composition behavior. -- Each participating record is a mapping with the designated string field. - Missing, null, and non-string identities are invalid. Compare exact strings - without trimming, case folding, normalization, or numeric coercion. Empty - strings are valid identities; repeated empty identities conflict. -- Append distinct records in contribution order, preserving each source list's - order. Use existing deterministic traversal within a strict tier rather than - sorting by identity. -- Repeated identities conflict across contributions and within one contributing - list, including a list with no peer contribution. Equal bodies and disjoint - bodies both conflict; matching records do not recursively merge. -- An empty sequence contributes no records. Two empty sequences compose, and an - explicitly contributed empty sequence remains present. -- Apply composition to main inputs and their imports and within each underlay - group, including placed item contributions and already-authored sequences. - Identity uniqueness is scoped to the sequence in that strict tier. -- A composition error cannot be hidden by a higher-precedence replacement. Use - normal load errors and the existing failed-load result contract. - -This differs from mapping composition at a shared key: mapping values can recurse, -whereas a repeated record identity conflicts. It does not establish final -uniqueness after defaults or application resolution. Identity values are not -supplied by those later phases; final domain validation remains application-owned. - -### Precedence and other operations - -Between underlay tiers, between underlays and the main tier, and in selected -overlays, sequences continue to replace lower sequences wholesale. `[]` clears -the lower list. The same identity can occur in different tiers without a -cross-tier duplicate error. Item placement does not turn precedence assembly -into append semantics. - -Whole-sequence CLI overrides, defaults, application resolution, and existing -sibling inheritance retain their behavior. This feature does not make records -inherit from one another by identity. Effective-configuration diffs continue to -compare final sequences atomically. Replacement-only selected overlays and -synthetic CLI overrides do not acquire identity validation from composition -registration; existing decoding and application validation still apply. - -### Existing inspection and ownership obligations - -A field at source path `text` can become `prompts[1].text`; a record originally -at source index zero can acquire a different effective index after composition. -Keep original physical locations and resource contexts associated with the right -record while reflecting effective placement in presence and origin queries. - -History and explanations must represent accumulation within a tier and actual -replacement between tiers. Replaying each contribution as a replacement would -lose earlier records. Apply existing historical sensitivity and disclosure rules -when indexes change or sequences are replaced, and preserve detached public -records and existing input/result ownership. These are applications of current -contracts, not a new promise of persistent identity-based paths across loads. - -## Work needed - -The target state requires changes in these areas; execution order belongs to the -[implementation plan](implementation.md): - -- **Public API and import syntax:** introduce the selected composition - registration and placement controls at the existing root-owned boundaries; - convert them into private engine state. Define invalid-option handling and - retain current defaults for callers that do not opt in. -- **Acquisition and placement:** support item contributions in ordinary and - layered loading, input/import destinations, and glob expansion. Implement the - selected nested-import and source-version interpretation without changing - acquisition identity or physical resource resolution. -- **Composition and retained evidence:** extend claim validation, tree assembly, - origins, and history together. The current composer in - `internal/engine/compose.go` collects claims by path and stores history - applications as well as the final tree. Updating only the tree merge is - insufficient. Keep tier boundaries explicit and preserve the existing limits - and accounting through the new wrapper and accumulation operations. -- **Inspection integration:** carry source-to-effective placement through history, - explanations, sensitivity, resource contexts, and public source records. Use - the smallest internal representation that preserves those existing contracts; - this roadmap does not prescribe a new indexing or caching subsystem. -- **Behavioral verification:** cover direct and glob-imported item sources, - already-authored lists, mixed contributions, duplicate/malformed identities, - empty and absent contributions, ordering, mounted paths, and nested imports. - Verify whole-list replacement, errors hidden by later tiers, version and - structural processing, remapped indexes, resource contexts, historical - redaction, and detached records. Protect current unregistered behavior, - defaults/resolution, and atomic diffs with proportionate regression coverage. -- **Documentation and adoption:** provide a maintained PromptKit-shaped example - and update public declaration comments, configuration/inspection contracts, - consumer guidance, and relevant internal documentation when behavior lands. - Record the significant design decision under the architecture policy and keep - the broader candidate in `future.md` distinct from the implemented subset. - -Completion means the PromptKit example works through public inputs and imports, -composition and replacement behave as agreed, and configuration values agree -with their inspection evidence. Validation follows the -[development guide](../development.md#baseline-validation). - -## Settled Decisions - -The following choices are accepted for this feature. No substantive design -choices remain open in this roadmap. The implementation plan selects exact -public names, YAML option names, and existing diagnostic-code mappings. -Internal representations remain implementation choices, subject to the existing -contracts and the behavior recorded here. - -### 1. Public controls and ownership - -Put sequence path/identity-field registrations on `Definition`. Put an explicit -placement mode beside input destination controls, with equivalent YAML import -syntax and support in ordinary loading. This separates reusable composition -policy from placement of individual sources. Default placement remains unchanged. - -### 2. Destination and registration paths - -Use literal mapping-key paths to identify destination sequences and match -registrations at their effective location after placement. For example, a source -mounted beneath `application` uses a registration at `application.prompts`. -Preserve existing destination validation without adding new reserved names. - -Permit unused registrations, coalesce identical registrations, and reject -conflicting identity fields for the same path. Ancestor/descendant registrations -apply only where their literal paths reach sequences; they do not implicitly -traverse every record. - -Sequence-index selectors and wildcard matching are outside this initial feature. -Item placement identifies a sequence beneath the configuration root; a root -sequence would expand the existing mapping-root model and is not included. -These are the selected feature boundaries, not permanent architectural exclusions. -Nested placement follows decision 4 without expanding registration addressing. - -### 3. Item placement without identity registration - -Allow item placement independently. It contributes a singleton sequence without -requiring an identity registration. Existing rules apply at unregistered paths: -two present sequences conflict in one strict tier, and an upper-tier sequence -replaces a lower one. Do not infer append semantics merely from item placement. - -### 4. Imports declared inside an item-placed source - -Compose ordinary imports into the declaring item's mapping. For an import -declared at the item root, an omitted destination or `destination: []` means that -item context; `[metadata]` places the helper inside the item's `metadata`. -Further relative destinations continue from the immediately importing document's -placement within that context. Physical import paths still resolve from the -declaring file. - -Item mode is explicit on each import, not inherited automatically. A collection -manifest explicitly imports peer prompt files as items at `prompts`. A helper -import supplies fields of the declaring item and need not declare its own ID. - -An explicitly item-placed import inside a prompt can contribute to a nested -sequence. Registrations do not reach through outer sequence indexes in this -initial feature, so those nested sequences retain atomic composition. Multiple -contributions to such a nested sequence therefore follow ordinary strict -conflict rules. This permits nested placement without introducing index selectors -or wildcard matching. - -### 5. Identity-validation boundary and structural processing - -Validate each completed record after applicable source-header processing and -ordinary helper-import composition, before combining records in the collection -and before later precedence replacement. A helper may supply the identity field; -it need not be authored in the item file itself. Duplicate fields within the -item still follow ordinary strict mapping composition. Defaults and application -resolution cannot supply missing identities because they run later. - -Preserve existing structural semantics. An item has no mapping-sibling namespace -for record inheritance, while mappings inside it can use existing inheritance. -Consumed structural fields and headers do not become application identity -values. Existing processing rules govern overlap with those controls; no new -identity-field-name blacklist is introduced. - -### 6. Empty identities and literal field names - -Accept empty string identities; repeated empty identities conflict. Interpret -the designated direct field name literally, including unusual or empty YAML -mapping keys where existing processing permits them. Presence and string type -suffice for identity comparison. Nonempty-ID requirements, naming conventions, -and whitespace restrictions remain application-owned validation choices. - -### 7. Non-sequence values at registered paths - -Registration selects how actual sequences compose; it is not an additional -schema declaration. Missing and non-sequence values retain existing composition -and decoding behavior. Sequence/null and sequence/mapping overlaps still -conflict in strict composition. Item placement contributes an actual sequence -and follows these rules. Registration does not independently reject every -non-sequence source value that a later tier might replace. - -### 8. Validation of replacement-only sequences - -Apply identity validation to registered sequences participating in strict -composition, including singleton contributions. Selected-overlay sequences and -synthetic CLI replacements retain their existing validation. Defaults and -application callbacks also retain their behavior. - -An overlay can therefore replace a list with duplicate IDs unless the consuming -application's validator rejects them. The identity rule governs composition, -not final configuration validity. - -### 9. Source-version checks for item-placed main inputs - -Support item placement for main inputs as well as imported and underlay sources. -Enabled source-version modes inspect declarations in source-local coordinates -before placement. Default mode retains its existing effective-path interpretation. -Accepted non-main headers are consumed; the main header remains application -content, following existing rules. - -The retained main header's effective location includes its actual item index. -Existing ownership protection follows that location and must not accidentally -protect another record because its source index differs from its effective -index. Preserve source-local version evidence and diagnostics where the existing -contract uses them. This feature introduces neither a new version policy nor -an exception to existing version-path protections. - -### 10. Public placement evidence and diagnostics - -Keep `SourceInfo.Destination` describing the effective destination container and -expose the placement mode. Use existing effective-path metadata to describe -resulting item paths, retaining physical source locations and resource contexts. -Do not promise one permanent item index per source: replacement can remove its -contribution, and contribution order can differ between loads. - -Duplicate diagnostics identify conflicting locations and the sequence path -without exposing potentially sensitive identity values. Reuse existing diagnostic -categories where appropriate. Resolved historical item paths and persistent -identity-addressed paths in source records are not part of this feature.