502 lines
30 KiB
Markdown
502 lines
30 KiB
Markdown
# Filesystem Sources and Underlays Implementation Plan
|
|
|
|
## Purpose and baseline
|
|
|
|
Implement the [feature roadmap](sources-and-layering.md) in the numbered order
|
|
below. Each stage is sized as one focused implementation assignment for a
|
|
gpt-5.6-terra coding agent. The baseline is
|
|
`36172ed20e832b371506d6aceb02dc6c8f50f71b`, plus the documentation changes committed
|
|
with this plan. No feature implementation is part of this planning pass.
|
|
|
|
Read `docs/development.md` and the policies it selects for the stage. The roadmap
|
|
owns target semantics; [ADR 0007](../adr/0007-source-backed-underlays.md) and
|
|
[ADR 0008](../adr/0008-complete-on-return-loading.md) record the proposed rationale.
|
|
Their draft status is independent of implementation progress: do not mark them
|
|
accepted automatically. The [configuration contract](../config.md) is the
|
|
compatibility baseline. Preserve the root-owned facade and private engine/tree
|
|
boundary.
|
|
|
|
At each stage, preserve unrelated work, add focused behavioral coverage with the
|
|
change, update declaration documentation for public additions, and run the checks
|
|
specified below. Keep intermediate commits building and tested. Do not expose
|
|
placeholder public APIs that silently ignore options. Internal scaffolding may
|
|
precede the public feature. Use plain-English commits under the agent policy.
|
|
|
|
## Settled implementation decisions
|
|
|
|
### Public input surface
|
|
|
|
Use these names and shapes; bodies and private fields belong to implementation:
|
|
|
|
```go
|
|
// Root-owned opaque filesystem namespace, copying preserves namespace identity.
|
|
type Filesystem struct { /* private engine state */ }
|
|
func NewFilesystem(label string, filesystem fs.FS) Filesystem
|
|
func (f Filesystem) File(name string) Source
|
|
func Mapping(label string, values map[string]any) Source
|
|
|
|
// Extend the existing record; nil FS selects host-filesystem context.
|
|
type FilesystemContext struct {
|
|
BaseDir string
|
|
Root string
|
|
FS *Filesystem
|
|
}
|
|
|
|
type Input struct {
|
|
Source Source
|
|
Imports bool
|
|
Filesystem *FilesystemContext
|
|
}
|
|
type Underlay struct { Inputs []Input }
|
|
type LayeredOptions[T any] struct {
|
|
Definition Definition[T]
|
|
Underlays []Underlay
|
|
Overlay string
|
|
OverlayFilesystem *FilesystemContext
|
|
Overrides []Override
|
|
}
|
|
func LoadLayered[T any](main Input, supplied ...LayeredOptions[T]) (*Result[T], error)
|
|
|
|
type ResourceContext struct {
|
|
FS fs.FS
|
|
BaseDir string
|
|
}
|
|
func (r *Result[T]) SourceContext(id string) (ResourceContext, bool)
|
|
func (r *Resolution) SourceContext(id string) (ResourceContext, bool)
|
|
```
|
|
|
|
`fs` above means the standard `io/fs` package. All named public types are defined
|
|
at the root, not aliases of engine types. Existing `Load` and `Options` stay
|
|
source-compatible apart from the additive field in `FilesystemContext`. Avoid a
|
|
parallel lifecycle implementation: adapt both entry points to one private load
|
|
request. No new dependency or minimum-Go-version change is needed.
|
|
|
|
- Like `Load`, `LoadLayered` accepts zero or one options value. A zero `Source`
|
|
stays invalid. Represent omitted main input as `Input{Source: NoSource()}`.
|
|
`NoSource` contributes no document, source ID, or context; preserve its existing
|
|
invalid import/context combinations. An empty underlay group is a no-op.
|
|
- Each underlay contains strictly composed inputs, including enabled imports.
|
|
Groups are ordered lowest to highest. Repeated explicit inputs in different
|
|
tiers are allowed subject to each source's reuse contract. Within a tier,
|
|
repeated file identities remain import/repetition errors. Equal scalar claims
|
|
still conflict. Independent synthetic sources are distinguished by occurrence.
|
|
- `Input.Imports` opts in per input and its import graph. The main source and its
|
|
imports form the main tier. There is no programmatic high-priority escape hatch:
|
|
a mapping source can be main input or any underlay contribution; CLI overrides
|
|
retain their explicit-flag contract.
|
|
- Active named overlays in `LoadLayered` require `OverlayFilesystem` explicitly.
|
|
Their names, root-relative lookup, directive prohibition, and version protection
|
|
follow existing rules. An inactive overlay needs no context; an unused explicit
|
|
context has no effect. Ordinary `Load` retains implicit selection of its source's
|
|
filesystem context and existing `NoSource` restrictions.
|
|
|
|
### Filesystem namespaces, paths, and acquisition
|
|
|
|
`NewFilesystem` allocates a reusable private namespace token and retains the
|
|
caller-owned filesystem without opening it. Copies share the token; separate
|
|
constructors are separate namespaces even for the same underlying filesystem.
|
|
Never compare or hash arbitrary `fs.FS` values. Validate nonempty safe labels and
|
|
nil filesystem arguments through deferred source diagnostics, following existing
|
|
source-constructor conventions. The zero `Filesystem` yields an invalid source.
|
|
|
|
`Filesystem.File(name)` uses a non-root `fs.ValidPath` file name relative to that
|
|
namespace. The default root is `.` and the base is the file's containing directory.
|
|
An optional filesystem context can narrow the root; file sources continue to
|
|
derive their own base directory. An explicit FS must identify the same namespace
|
|
as the file source; context overrides for native file sources supply that FS.
|
|
A host `File` cannot be rebound to a virtual namespace.
|
|
For byte, reader, and mapping inputs, an explicit context supplies their namespace
|
|
and base: with FS set, require valid slash-relative Root and BaseDir (use `.` for
|
|
the namespace root), with BaseDir inside Root. With FS nil, preserve the existing
|
|
absolute host-directory requirements. Labels never imply a context.
|
|
|
|
Resolve import paths relative to the declaring document, normalizing joined
|
|
paths and rejecting escape from the configured root. Preserve the existing
|
|
relative-path/glob language, including contained parent references, optional
|
|
missing behavior, sorted matches, bounded traversal, and cycle diagnostics.
|
|
Virtual paths use `path`, not host `filepath` rules. Imports stay in their namespace;
|
|
generic filesystems provide their own symlink behavior. Preserve canonicalization,
|
|
regular-file checks, and `os.Root` containment for host acquisition. Use ordinary
|
|
`fs.FS` operations and `fs.ReadDirFile` where available; support the standard
|
|
`fs.ReadDir` fallback without claiming control over its internal allocation.
|
|
|
|
Configuration reads close each file ConfigKit opens, including read/stat/parse
|
|
failures. Caller readers and caller filesystems are never closed by ConfigKit.
|
|
Namespace tokens live with source objects; deterministic load-local IDs are
|
|
assigned by acquisition, not addresses or global counters.
|
|
|
|
### Programmatic representation and diagnostics
|
|
|
|
`Mapping` snapshots its input at construction, matching `Bytes`. Convert directly
|
|
to the private document tree using the CLI converter's existing built-in scalar,
|
|
`[]any`, and `map[string]any` repertoire. Preserve integer precision, finite-float
|
|
validation, sorted keys, sequence order, explicit null, and cycle rejection.
|
|
Typed nil maps/slices mean null, as in existing overrides. A nil root mapping is
|
|
invalid because document roots are mappings; `{}` is a valid empty mapping.
|
|
Constructor failures are stored and surfaced on load. Detach once at construction;
|
|
do not retain caller maps or serialize them. Concurrent caller mutation during
|
|
construction is outside the supported ownership contract.
|
|
|
|
Generic conversion returns private issues; adapt them to existing `override.*`
|
|
codes for CLI input, and `source.invalid` or `source.limit_exceeded` for mapping
|
|
input. Do not leak raw values or conversion errors. Source/path locations use
|
|
the label and semantic path, with zero line/column for synthetic input. Apply
|
|
reserved `configkit` directive rules consistently: with imports enabled, a
|
|
mapping can contain the same directive structure as YAML. Without context,
|
|
relative imports fail under existing context rules.
|
|
|
|
### Ordering, lifecycle, budgets, and metadata
|
|
|
|
Acquire and strictly compose main input first, construct the schema, and check
|
|
the optional version. Only the main source owns the version; main imports and
|
|
all underlay documents must satisfy fragment-style version-path protections.
|
|
With `NoSource` main input, pass an absent version to the checker. Do not run the
|
|
checker once per tier or validate the final overlaid version instead.
|
|
|
|
Then acquire/compose underlays in listed order and combine them recursively,
|
|
apply the composed main tier, selected overlay, and CLI overrides. Execute typed
|
|
defaulting/decoding, recorded resolution, secret lookup, validation, and final
|
|
publication once. Syntax/composition/version failures cannot be masked by higher
|
|
precedence; effective typed validation sees the final assembled input only.
|
|
|
|
Keep source-record order separate from precedence: main depth-first sources,
|
|
underlay tiers in order with each input traversed depth-first, selected overlay,
|
|
then CLI. IDs remain `source-N`. Existing ordinary loads retain their source order.
|
|
The history application order follows precedence, not acquisition order.
|
|
|
|
Extend `Layer` with `LayerUnderlay = "underlay"`. Add to `SourceInfo` a `Kind
|
|
SourceKind` and `UnderlayIndex int` (one-based original tier position for underlay
|
|
sources, zero otherwise). Define `SourceKindFile`, `SourceKindFilesystem`,
|
|
`SourceKindBytes`, `SourceKindReader`, `SourceKindMapping`, and `SourceKindCLI`
|
|
with string values `file`, `filesystem`, `bytes`, `reader`, `mapping`, and `cli`.
|
|
Imported documents inherit tier role/index and have their actual acquisition kind.
|
|
Preserve existing host Path meaning; filesystem-source Path is namespace-relative;
|
|
synthetic Path stays empty. Labels for native files combine namespace label and
|
|
relative path for display, while identity uses the token and path.
|
|
|
|
Origins continue to refer to source IDs through locations. Composite origins
|
|
preserve all contributors; equal-value overrides and displaced descendants stay
|
|
inspectable. Carry source context separately from serializable source records.
|
|
Keep existing inspection schema and renderer structure, adding the new layer/kind
|
|
where appropriate rather than changing existing output gratuitously.
|
|
|
|
Use one load-wide accounting object for layered loads: existing 64 MiB aggregate
|
|
input bytes, 1,024 acquired sources, 1,000,000 input nodes, and glob-entry budget;
|
|
existing per-source 8 MiB, depth 256, active import depth 64, and retained-history
|
|
limit remain. Mapping key/string bytes use the existing CLI converter's byte
|
|
accounting. Charge each accepted occurrence once, including CLI contributions;
|
|
do not charge intermediate copies as new input. Explicit empty groups and
|
|
`NoSource` consume no source allowance. Ordinary loads retain their established
|
|
limit behavior. Use existing source/import/overlay/override limit codes according
|
|
to the acquisition that exceeds a budget. Add `CodeUnderlayVersion =
|
|
"underlay.version"` for version protection in underlay roots; imported fragments
|
|
retain `import.version`. Avoid a separate arbitrary tier-count limit.
|
|
|
|
### Passive resource contexts
|
|
|
|
`SourceContext` returns false for unknown IDs or sources without a context, and
|
|
performs no I/O. The result method is nil-safe; the resolution method follows
|
|
existing callback lifetime guards. Return a record copy containing a borrowed
|
|
filesystem and slash-relative source BaseDir within its configured root.
|
|
Consumers obtain source IDs from ordinary origins; there is no single-context
|
|
query that guesses among composite or derived origins. Dependencies continue to
|
|
identify the sources of application-derived values through existing metadata.
|
|
|
|
For native inputs, use a namespace-rooted view over the caller's filesystem and
|
|
retain its lifetime semantics. For host input, retain the canonical root path,
|
|
not an open `os.Root`. A private `fs.FS` adapter validates a relative resource
|
|
name, resolves it using existing host containment semantics, opens a fresh root,
|
|
and opens the resource. Return a private `fs.File` wrapper whose Close releases
|
|
both file and root, including failed-open cleanup. Expose `fs.ReadDirFile` when
|
|
the underlying directory supports it. Resources themselves are application data:
|
|
do not apply configuration YAML parsing or configuration-byte budgets to them.
|
|
Helpers such as `fs.ReadFile` then close the returned handle normally. Errors
|
|
follow filesystem access conventions, separately from load diagnostics.
|
|
|
|
Neither queries nor retained results keep library-owned filesystem handles open.
|
|
Files/resources can change or disappear after loading; configuration/metadata
|
|
remain usable. No background work, hidden reopening on metadata queries,
|
|
filesystem snapshot, universal sandbox, or transaction guarantee is added.
|
|
|
|
## Stage 1: Index mapping assembly and establish scaling baselines
|
|
|
|
**Completed 2026-09-29.** Private per-assembly mapping indexes now serve strict
|
|
composition and recursive overlay lookup while ordered entries and diagnostics
|
|
remain stable; conflict groups are checked once. Focused Compose/Overlay tests
|
|
and the everyday checker passed. On Linux amd64, the 512-entry benchmark
|
|
fixtures measured approximately 4.0 ms/op before and 3.3 ms/op after for wide
|
|
composition, and 12.9 ms/op before and 12.0 ms/op after for recursive overlay.
|
|
These are comparative evidence, not timing gates; provenance work still
|
|
dominates the latter and belongs to the next stage.
|
|
|
|
Scope: `internal/document` and engine composition/overlay lookup, with existing
|
|
behavior only.
|
|
|
|
1. Add parameterized benchmarks for wide disjoint composition and recursive
|
|
overlay, recording runtime and allocations at increasing sizes. Keep fixtures
|
|
within existing budgets and record the baseline in the stage handoff.
|
|
2. Replace repeated linear mapping-key lookup during bulk assembly with a private
|
|
indexed builder alongside ordered entries. Build each target index once per
|
|
assembly session and update it for appended keys. Keep tree traversal and
|
|
diagnostic order stable; avoid mutable indexes exposed through public APIs.
|
|
3. Check each strict-composition claim group once while preserving first-conflict
|
|
ordering and all relevant locations. Preserve original input ownership.
|
|
4. Extend existing composition tests only for uncovered ordering/conflict risks.
|
|
|
|
Acceptance: existing Compose/Overlay tests pass unchanged; broad keyed input
|
|
no longer incurs a full key scan for each insertion. Run the focused composition
|
|
tests and the everyday checker. Benchmarks are evidence, not timing gates.
|
|
|
|
## Stage 2: Prepare load-owned assembly and incremental provenance
|
|
|
|
**Completed 2026-09-29.** Ordinary loads now apply selected overlays and CLI
|
|
overrides to one owned working document, while copy-preserving helpers remain
|
|
available for direct use. Acquired history trees stay detached; composite
|
|
contributors survive overlay application. Origin deletion visits only replaced
|
|
subtrees, mapping contributors append without copying growing lists, and history
|
|
replay shares indexed lookup across applications. Focused behavior tests, the
|
|
everyday checker, and the engine race suite passed. On Linux amd64 at 128 small
|
|
applications, owned assembly measured about 0.30 ms/op versus 6.0 ms/op for
|
|
repeated copy-preserving application; alternating subtree/scalar replacement
|
|
measured about 0.39 versus 7.3 ms/op. These benchmarks are evidence, not gates.
|
|
|
|
Scope: `compose.go`, `overlay.go`, origin indexes, override application, and
|
|
history replay in `internal/engine`.
|
|
|
|
1. Introduce a private load-owned accumulator with explicit ownership. Existing
|
|
copy-preserving helpers may wrap it, but sequential assembly must mutate one
|
|
owned working tree rather than clone all accumulated state per tier.
|
|
2. Retain immutable acquired inputs for history, detached from mutable working
|
|
nodes. Parameterize internal overlay application by incoming layer and origins
|
|
so a composed tier's imported contributors are preserved.
|
|
3. Remove displaced origins by visiting the affected subtree. Accumulate mapping
|
|
contributors without copying an ever-growing ancestor list on each addition;
|
|
materialize defensive public records at the existing publication boundary.
|
|
4. Preserve equal-value replacement events, descendant displacement, redaction,
|
|
and typed-decoding reconciliation. Extend benchmarks for many small applications
|
|
and repeated subtree/scalar replacement. Do not add a persistent-tree framework.
|
|
|
|
Acceptance: existing override/history/rendering/ownership tests pass; benchmark
|
|
growth avoids accumulated full-tree copying. Run the everyday checker and
|
|
`GOPROXY=off go test -race ./internal/engine` for ownership changes.
|
|
|
|
## Stage 3: Unify private filesystem acquisition without public changes
|
|
|
|
**Completed 2026-09-29.** Imports, glob traversal, and selected-overlay loading
|
|
now use a private access boundary with the existing contained host backend and
|
|
a native `fs.FS` backend. Namespace tokens and occurrence identities are separate
|
|
from labels and paths; repetition tracking is composition-local and input/glob
|
|
accounting can be shared. Native tests cover slash-relative imports, sorted
|
|
globs, optional missing paths, root escape, overlays, repeated/cyclic imports,
|
|
non-comparable filesystems, and handle closure on success and representative
|
|
failures. Existing host tests, focused native tests, and the everyday checker
|
|
passed. No public constructor or context field was added.
|
|
|
|
Scope: engine source/file, filesystem, import, glob, and selected-overlay code.
|
|
|
|
1. Extract a small private namespace/access boundary used by imports and overlays.
|
|
Keep the host implementation's canonicalization, containment, regular-file
|
|
checks, safe diagnostics, deterministic traversal, and close behavior.
|
|
2. Separate namespace identity from source labels and resource paths. Make
|
|
repeated/cycle tracking belong to an input composition group, and accounting
|
|
capable of being shared across groups.
|
|
3. Add a native `fs.FS` backend internally. Resolve paths using slash semantics,
|
|
preserve glob grammar and optional-missing behavior, and avoid host fallback.
|
|
4. Use real host tests plus `fstest.MapFS` and a small tracking/failing filesystem
|
|
to cover imports, missing paths, directory/read failures, and handle closure.
|
|
Include a filesystem with a non-comparable dynamic value.
|
|
|
|
Acceptance: all existing host source/import/glob/overlay tests remain green;
|
|
native backend tests establish namespace isolation and supported paths. Run the
|
|
everyday checker. No public constructors land until the next stage.
|
|
|
|
## Stage 4: Expose native filesystem sources through the facade
|
|
|
|
**Completed 2026-09-29.** The root API now exposes reusable native filesystem
|
|
namespaces, namespace-aware contexts, and source acquisition kinds. Ordinary
|
|
loads support native files, opt-in imports, selected overlays, and byte/reader
|
|
contexts while retaining host semantics. Public tests cover embedded fixtures,
|
|
`fstest.MapFS`, same-label namespace isolation, wrapper copies and reuse,
|
|
constructor/context diagnostics, and host/CLI source kinds. The everyday checker
|
|
passed, including public-boundary and documentation checks.
|
|
|
|
1. Add `Filesystem`, `NewFilesystem`, `Filesystem.File`, and the additive FS
|
|
context field at root and engine boundaries. Constructors follow deferred
|
|
diagnostics; copying the wrapper preserves namespace identity.
|
|
2. Integrate native sources with ordinary `Load`, including explicitly enabled
|
|
imports and named overlays. Enforce the path/context rules settled above.
|
|
3. Add `SourceKind` and populate it for every current/native acquisition. Preserve
|
|
host paths and use namespace-relative virtual paths with accurate locations.
|
|
4. Add facade-level tests using embedded fixtures, `fstest.MapFS`, byte/reader
|
|
context, namespace-label collisions, and source reuse. Keep backend error cases
|
|
at their Stage 3 owner. Update current source/API docs to this implemented scope.
|
|
|
|
Acceptance: native `Load` works through the public API and existing sources retain
|
|
their contracts. Run the everyday checker, including public-boundary tests.
|
|
|
|
## Stage 5: Add snapshotted mapping sources
|
|
|
|
**Completed 2026-09-29.** `Mapping` now snapshots supported built-in values at
|
|
construction, retaining safe deferred diagnostics for invalid roots, values,
|
|
cycles, nonfinite floats, and limits. A shared private converter preserves CLI
|
|
override behavior through a separate adapter. Mapping inputs participate in
|
|
ordinary directives, imports, version checks, decoding, and inspection with
|
|
their own source kind and synthetic locations. Public tests cover YAML
|
|
equivalence, reuse after caller mutation, null/empty/zero distinctions, exact
|
|
integers, explicit filesystem context, and safe failures. The everyday checker
|
|
and engine race suite passed.
|
|
|
|
1. Extract reusable value conversion from `override_value.go`; keep CLI-specific
|
|
path, label, and diagnostic handling in its adapter.
|
|
2. Implement `Mapping` at root and engine boundaries with constructor-time
|
|
snapshotting and deferred safe failures. Source-ID assignment at acquisition
|
|
must not mutate a reusable stored tree shared by another load.
|
|
3. Route converted documents through the same directive processing, strict
|
|
composition, version checking, decode, and inspection path as YAML sources.
|
|
4. Test omission/zero/false/null, nil versus empty collections/root, exact integers,
|
|
invalid/cyclic values, imports with explicit context, and caller mutation after
|
|
construction. Verify mapping input does not gain CLI origins or restrictions.
|
|
|
|
Acceptance: equivalent byte/mapping inputs have equivalent semantic results and
|
|
origin meaning, with genuine location differences allowed. Existing override
|
|
diagnostics remain stable. Run the everyday checker and engine race tests.
|
|
|
|
## Stage 6: Implement private grouped-underlay orchestration
|
|
|
|
1. Add private request/input/group structures matching the settled facade design.
|
|
Adapt ordinary `Load` to the shared core while preserving its options and
|
|
source-order behavior. Keep its existing validation order where observable.
|
|
2. Strictly compose each group with group-scoped repetition tracking and shared
|
|
aggregate accounting. Apply the lifecycle order defined above, including the
|
|
single main-owned version check before underlay precedence assembly.
|
|
3. Add underlay layer/index metadata and version diagnostics. Keep history in
|
|
precedence order while source IDs follow acquisition order. Apply composed
|
|
main origins over underlays without relabeling main imports as overlays.
|
|
4. Reuse Stage 2 assembly and the existing decode/resolution/secret/publication
|
|
pipeline. Empty groups, omitted main, defaults-only loads, and underlay-only
|
|
loads must follow the chosen semantics.
|
|
5. Add engine coverage for two underlay tiers, multiple strict contributions,
|
|
main/import conflicts, version checks/absent main, limits across groups, and
|
|
one final decode/resolve/validate lifecycle. Test an invalid intermediate typed
|
|
value replaced by valid higher input, alongside unmaskable syntax conflicts.
|
|
|
|
Acceptance: engine layered requests produce correct values and full provenance;
|
|
ordinary-load tests remain green. Run the everyday checker and engine race tests.
|
|
|
|
## Stage 7: Expose layered loading and complete inspection integration
|
|
|
|
1. Add `Input`, `Underlay`, `LayeredOptions`, and `LoadLayered` root wrappers and
|
|
conversions. Copy option slices/context records into private load state while
|
|
preserving reusable namespace identities and single-consumption readers.
|
|
2. Wire explicit selected-overlay context for layered loads, including host and
|
|
native namespaces. The selected file may not duplicate an acquired file in
|
|
that namespace, preserving selected-overlay repetition protection.
|
|
3. Finish root Layer/SourceInfo conversions, history/explanation rendering,
|
|
composite mapping attribution, deleted descendants, default and derived
|
|
origins, and secret redaction. New records must remain detached.
|
|
4. Add representative public tests for full precedence and same-name recursive
|
|
mapping combination; retain zero/false/empty/null distinctions and atomic
|
|
sequence replacement. Test ordinary versus zero-underlay layered equivalence
|
|
using equivalent explicit overlay context when required.
|
|
|
|
Acceptance: callers can load native, host, and mapping tiers in one public call;
|
|
configuration, metadata, and safe rendering agree. Run the everyday checker and
|
|
`GOPROXY=off go test -race ./...`.
|
|
|
|
## Stage 8: Add handle-free source-context queries
|
|
|
|
1. Retain context descriptors keyed by acquired source ID, including imported
|
|
sources, in private inspection and live resolution state. Keep descriptors
|
|
separate from serializable metadata and preserve callback guards.
|
|
2. Implement the root query methods and `ResourceContext`. Native contexts expose
|
|
rooted borrowed access; host contexts use the passive adapter described above.
|
|
Include host main files even when imports are disabled, and explicit contexts
|
|
on byte/reader/mapping sources. Sources without contexts return false.
|
|
3. Verify each origin's BaseDir points to its actual source. Composite origins
|
|
require callers to follow individual locations; derived dependencies retain
|
|
their existing meaning. No arbitrary last-source context is selected.
|
|
4. Test that queries cause no filesystem operations; loaded values and metadata
|
|
survive source deletion; later resource access observes mutation/missing files;
|
|
and caller filesystem closure affects only later resource operations.
|
|
5. Verify owned files/root handles close on successful loading and representative
|
|
acquisition, parse, composition, version, and validation failures. Test resource
|
|
Open failure cleanup and returned-file Close ownership separately. Prefer
|
|
narrow tracking seams over timing-based goroutine/descriptor-count assertions.
|
|
|
|
Acceptance: a retained underlay resource can be opened relative to its true origin
|
|
after load, with no library-owned loading handles retained. Run the everyday
|
|
checker and full race suite.
|
|
|
|
## Stage 9: Validate cross-feature consistency and scaling
|
|
|
|
1. Consolidate a small semantic equivalence matrix across host, native, bytes,
|
|
readers, and mappings, using equivalent logical inputs and contexts. Cover
|
|
strict conflict versus intentional precedence and version-path preservation;
|
|
avoid duplicating every lower-level case at every source permutation.
|
|
2. Exercise load-wide limits across mixed sources/CLI, repeated explicit tiers,
|
|
shared namespace imports, and source IDs/history order. Protect against double
|
|
charging and reader re-consumption. Confirm secret canaries stay absent from
|
|
diagnostics and historical/effective output after replacements.
|
|
3. Run parameterized benchmarks for broad catalogs, many small tiers, subtree
|
|
replacement, imported native defaults, and retained history. Compare Stage 1
|
|
baselines with the final path. Address demonstrated avoidable quadratic scans
|
|
or accumulated copying within this feature's scope. Account for sorting,
|
|
filesystem operations, callbacks, and output/history size explicitly.
|
|
4. Review source/context ownership and success/failure cleanup against ADR 0008.
|
|
Keep performance evidence in the stage handoff, not permanent timing gates.
|
|
|
|
Acceptance: no semantic discrepancy remains among equivalent operations; scaling
|
|
results substantiate the engineering target without an unconditional O(n) claim.
|
|
Run the everyday checker and full race suite, reusing successful checks for an
|
|
unchanged final tree. Benchmarks run with `-benchmem` and parameterized sizes.
|
|
|
|
## Stage 10: Complete consumer examples and architecture documentation
|
|
|
|
1. Add a maintained simple native-filesystem example and a combined example with
|
|
embedded defaults, two underlay tiers, independently located main input,
|
|
programmatic contributions, and a relative resource from a retained origin.
|
|
Keep fixtures secret-free and offline; include zero/false presence and domain
|
|
validation. The consumer should perform no generic post-load merge/defaulting
|
|
or provenance reconstruction. Reuse existing example/test conventions.
|
|
2. Update `docs/config.md`, Go declarations, `docs/consumers/`, README,
|
|
`docs/internal/overview.md`, and applicable architecture/deferred-scope text
|
|
to describe implemented support. Preserve strict-within-tier and recursive-
|
|
between-tier terminology, distinguish source kinds from precedence roles,
|
|
and document context lifetimes and the explicit source-ID query workflow.
|
|
3. Add the relevant tests/commands to the development guide's existing task map.
|
|
Review ADR links and statuses, preserving draft status pending acceptance and
|
|
accepted ADR decision bodies. Keep named inheritance separately deferred.
|
|
4. Mark implementation progress accurately in this plan and the roadmap. Do not
|
|
retire plans, choose/tag a version, prepare release gates, or publish as part
|
|
of implementation; those belong to separate release preparation.
|
|
|
|
Acceptance: examples compile and execute, documentation describes only completed
|
|
capabilities, and simple `Load` usage remains simple. Run the everyday checker
|
|
(which includes examples, links, and boundary checks); run the full race suite
|
|
if code changed since its last successful run. Report completed stages, checks,
|
|
benchmark conclusions, and any remaining limitations honestly.
|
|
|
|
## Validation commands
|
|
|
|
Use the development guide's focused commands during each stage. The normal
|
|
stage-end check is `./scripts/check.sh`; it already includes tests, vet, build,
|
|
formatting, and documentation links. Ownership stages additionally use the race
|
|
commands above. Benchmarks may use:
|
|
|
|
```sh
|
|
GOPROXY=off go test ./internal/engine -run '^$' -bench 'Compose|Overlay|Underlay' -benchmem
|
|
```
|
|
|
|
Name the new benchmarks to match that expression. Select representative sizes
|
|
under existing limits. No CI wait, human macOS validation, external service,
|
|
credentials, or new release gate is part of this plan.
|
|
|
|
## Open Questions
|
|
|
|
None required to implement this plan. The remaining roadmap recommendations
|
|
have been selected above: labeled snapshotted mappings and separate source-context
|
|
queries. Private helper names and package-local factoring remain implementation
|
|
choices within these contracts. If implementation uncovers a genuine conflict
|
|
with existing behavior, report it rather than silently adding a restriction,
|
|
weakening compatibility, or broadening a guarantee.
|