Retire completed plans and outline named inheritance
This commit is contained in:
@@ -1,565 +0,0 @@
|
||||
# Filesystem Sources and Underlays Implementation Plan
|
||||
|
||||
## Purpose and baseline
|
||||
|
||||
Implement the [feature roadmap](sources-and-layering.md) in the numbered order
|
||||
below. Each stage is sized as one focused implementation assignment for a
|
||||
gpt-5.6-terra coding agent. The baseline is
|
||||
`36172ed20e832b371506d6aceb02dc6c8f50f71b`, plus the documentation changes committed
|
||||
with this plan. No feature implementation is part of this planning pass.
|
||||
|
||||
Read `docs/development.md` and the policies it selects for the stage. The roadmap
|
||||
owns target semantics; [ADR 0007](../adr/0007-source-backed-underlays.md) and
|
||||
[ADR 0008](../adr/0008-complete-on-return-loading.md) record the proposed rationale.
|
||||
Their draft status is independent of implementation progress: do not mark them
|
||||
accepted automatically. The [configuration contract](../config.md) is the
|
||||
compatibility baseline. Preserve the root-owned facade and private engine/tree
|
||||
boundary.
|
||||
|
||||
At each stage, preserve unrelated work, add focused behavioral coverage with the
|
||||
change, update declaration documentation for public additions, and run the checks
|
||||
specified below. Keep intermediate commits building and tested. Do not expose
|
||||
placeholder public APIs that silently ignore options. Internal scaffolding may
|
||||
precede the public feature. Use plain-English commits under the agent policy.
|
||||
|
||||
## Settled implementation decisions
|
||||
|
||||
### Public input surface
|
||||
|
||||
Use these names and shapes; bodies and private fields belong to implementation:
|
||||
|
||||
```go
|
||||
// Root-owned opaque filesystem namespace, copying preserves namespace identity.
|
||||
type Filesystem struct { /* private engine state */ }
|
||||
func NewFilesystem(label string, filesystem fs.FS) Filesystem
|
||||
func (f Filesystem) File(name string) Source
|
||||
func Mapping(label string, values map[string]any) Source
|
||||
|
||||
// Extend the existing record; nil FS selects host-filesystem context.
|
||||
type FilesystemContext struct {
|
||||
BaseDir string
|
||||
Root string
|
||||
FS *Filesystem
|
||||
}
|
||||
|
||||
type Input struct {
|
||||
Source Source
|
||||
Imports bool
|
||||
Filesystem *FilesystemContext
|
||||
}
|
||||
type Underlay struct { Inputs []Input }
|
||||
type LayeredOptions[T any] struct {
|
||||
Definition Definition[T]
|
||||
Underlays []Underlay
|
||||
Overlay string
|
||||
OverlayFilesystem *FilesystemContext
|
||||
Overrides []Override
|
||||
}
|
||||
func LoadLayered[T any](main Input, supplied ...LayeredOptions[T]) (*Result[T], error)
|
||||
|
||||
type ResourceContext struct {
|
||||
FS fs.FS
|
||||
BaseDir string
|
||||
}
|
||||
func (r *Result[T]) SourceContext(id string) (ResourceContext, bool)
|
||||
func (r *Resolution) SourceContext(id string) (ResourceContext, bool)
|
||||
```
|
||||
|
||||
`fs` above means the standard `io/fs` package. All named public types are defined
|
||||
at the root, not aliases of engine types. Existing `Load` and `Options` stay
|
||||
source-compatible apart from the additive field in `FilesystemContext`. Avoid a
|
||||
parallel lifecycle implementation: adapt both entry points to one private load
|
||||
request. No new dependency or minimum-Go-version change is needed.
|
||||
|
||||
- Like `Load`, `LoadLayered` accepts zero or one options value. A zero `Source`
|
||||
stays invalid. Represent omitted main input as `Input{Source: NoSource()}`.
|
||||
`NoSource` contributes no document, source ID, or context; preserve its existing
|
||||
invalid import/context combinations. An empty underlay group is a no-op.
|
||||
- Each underlay contains strictly composed inputs, including enabled imports.
|
||||
Groups are ordered lowest to highest. Repeated explicit inputs in different
|
||||
tiers are allowed subject to each source's reuse contract. Within a tier,
|
||||
repeated file identities remain import/repetition errors. Equal scalar claims
|
||||
still conflict. Independent synthetic sources are distinguished by occurrence.
|
||||
- `Input.Imports` opts in per input and its import graph. The main source and its
|
||||
imports form the main tier. There is no programmatic high-priority escape hatch:
|
||||
a mapping source can be main input or any underlay contribution; CLI overrides
|
||||
retain their explicit-flag contract.
|
||||
- Active named overlays in `LoadLayered` require `OverlayFilesystem` explicitly.
|
||||
Their names, root-relative lookup, directive prohibition, and version protection
|
||||
follow existing rules. An inactive overlay needs no context; an unused explicit
|
||||
context has no effect. Ordinary `Load` retains implicit selection of its source's
|
||||
filesystem context and existing `NoSource` restrictions.
|
||||
|
||||
### Filesystem namespaces, paths, and acquisition
|
||||
|
||||
`NewFilesystem` allocates a reusable private namespace token and retains the
|
||||
caller-owned filesystem without opening it. Copies share the token; separate
|
||||
constructors are separate namespaces even for the same underlying filesystem.
|
||||
Never compare or hash arbitrary `fs.FS` values. Validate nonempty safe labels and
|
||||
nil filesystem arguments through deferred source diagnostics, following existing
|
||||
source-constructor conventions. The zero `Filesystem` yields an invalid source.
|
||||
|
||||
`Filesystem.File(name)` uses a non-root `fs.ValidPath` file name relative to that
|
||||
namespace. The default root is `.` and the base is the file's containing directory.
|
||||
An optional filesystem context can narrow the root; file sources continue to
|
||||
derive their own base directory. An explicit FS must identify the same namespace
|
||||
as the file source; context overrides for native file sources supply that FS.
|
||||
A host `File` cannot be rebound to a virtual namespace.
|
||||
For byte, reader, and mapping inputs, an explicit context supplies their namespace
|
||||
and base: with FS set, require valid slash-relative Root and BaseDir (use `.` for
|
||||
the namespace root), with BaseDir inside Root. With FS nil, preserve the existing
|
||||
absolute host-directory requirements. Labels never imply a context.
|
||||
|
||||
Resolve import paths relative to the declaring document, normalizing joined
|
||||
paths and rejecting escape from the configured root. Preserve the existing
|
||||
relative-path/glob language, including contained parent references, optional
|
||||
missing behavior, sorted matches, bounded traversal, and cycle diagnostics.
|
||||
Virtual paths use `path`, not host `filepath` rules. Imports stay in their namespace;
|
||||
generic filesystems provide their own symlink behavior. Preserve canonicalization,
|
||||
regular-file checks, and `os.Root` containment for host acquisition. Use ordinary
|
||||
`fs.FS` operations and `fs.ReadDirFile` where available; support the standard
|
||||
`fs.ReadDir` fallback without claiming control over its internal allocation.
|
||||
|
||||
Configuration reads close each file ConfigKit opens, including read/stat/parse
|
||||
failures. Caller readers and caller filesystems are never closed by ConfigKit.
|
||||
Namespace tokens live with source objects; deterministic load-local IDs are
|
||||
assigned by acquisition, not addresses or global counters.
|
||||
|
||||
### Programmatic representation and diagnostics
|
||||
|
||||
`Mapping` snapshots its input at construction, matching `Bytes`. Convert directly
|
||||
to the private document tree using the CLI converter's existing built-in scalar,
|
||||
`[]any`, and `map[string]any` repertoire. Preserve integer precision, finite-float
|
||||
validation, sorted keys, sequence order, explicit null, and cycle rejection.
|
||||
Typed nil maps/slices mean null, as in existing overrides. A nil root mapping is
|
||||
invalid because document roots are mappings; `{}` is a valid empty mapping.
|
||||
Constructor failures are stored and surfaced on load. Detach once at construction;
|
||||
do not retain caller maps or serialize them. Concurrent caller mutation during
|
||||
construction is outside the supported ownership contract.
|
||||
|
||||
Generic conversion returns private issues; adapt them to existing `override.*`
|
||||
codes for CLI input, and `source.invalid` or `source.limit_exceeded` for mapping
|
||||
input. Do not leak raw values or conversion errors. Source/path locations use
|
||||
the label and semantic path, with zero line/column for synthetic input. Apply
|
||||
reserved `configkit` directive rules consistently: with imports enabled, a
|
||||
mapping can contain the same directive structure as YAML. Without context,
|
||||
relative imports fail under existing context rules.
|
||||
|
||||
### Ordering, lifecycle, budgets, and metadata
|
||||
|
||||
Acquire and strictly compose main input first, construct the schema, and check
|
||||
the optional version. Only the main source owns the version; main imports and
|
||||
all underlay documents must satisfy fragment-style version-path protections.
|
||||
With `NoSource` main input, pass an absent version to the checker. Do not run the
|
||||
checker once per tier or validate the final overlaid version instead.
|
||||
|
||||
Then acquire/compose underlays in listed order and combine them recursively,
|
||||
apply the composed main tier, selected overlay, and CLI overrides. Execute typed
|
||||
defaulting/decoding, recorded resolution, secret lookup, validation, and final
|
||||
publication once. Syntax/composition/version failures cannot be masked by higher
|
||||
precedence; effective typed validation sees the final assembled input only.
|
||||
|
||||
Keep source-record order separate from precedence: main depth-first sources,
|
||||
underlay tiers in order with each input traversed depth-first, selected overlay,
|
||||
then CLI. IDs remain `source-N`. Existing ordinary loads retain their source order.
|
||||
The history application order follows precedence, not acquisition order.
|
||||
|
||||
Extend `Layer` with `LayerUnderlay = "underlay"`. Add to `SourceInfo` a `Kind
|
||||
SourceKind` and `UnderlayIndex int` (one-based original tier position for underlay
|
||||
sources, zero otherwise). Define `SourceKindFile`, `SourceKindFilesystem`,
|
||||
`SourceKindBytes`, `SourceKindReader`, `SourceKindMapping`, and `SourceKindCLI`
|
||||
with string values `file`, `filesystem`, `bytes`, `reader`, `mapping`, and `cli`.
|
||||
Imported documents inherit tier role/index and have their actual acquisition kind.
|
||||
Preserve existing host Path meaning; filesystem-source Path is namespace-relative;
|
||||
synthetic Path stays empty. Labels for native files combine namespace label and
|
||||
relative path for display, while identity uses the token and path.
|
||||
|
||||
Origins continue to refer to source IDs through locations. Composite origins
|
||||
preserve all contributors; equal-value overrides and displaced descendants stay
|
||||
inspectable. Carry source context separately from serializable source records.
|
||||
Keep existing inspection schema and renderer structure, adding the new layer/kind
|
||||
where appropriate rather than changing existing output gratuitously.
|
||||
|
||||
Use one load-wide accounting object for layered loads: existing 64 MiB aggregate
|
||||
input bytes, 1,024 acquired sources, 1,000,000 input nodes, and glob-entry budget;
|
||||
existing per-source 8 MiB, depth 256, active import depth 64, and retained-history
|
||||
limit remain. Mapping key/string bytes use the existing CLI converter's byte
|
||||
accounting. Charge each accepted occurrence once, including CLI contributions;
|
||||
do not charge intermediate copies as new input. Explicit empty groups and
|
||||
`NoSource` consume no source allowance. Ordinary loads retain their established
|
||||
limit behavior. Use existing source/import/overlay/override limit codes according
|
||||
to the acquisition that exceeds a budget. Add `CodeUnderlayVersion =
|
||||
"underlay.version"` for version protection in underlay roots; imported fragments
|
||||
retain `import.version`. Avoid a separate arbitrary tier-count limit.
|
||||
|
||||
### Passive resource contexts
|
||||
|
||||
`SourceContext` returns false for unknown IDs or sources without a context, and
|
||||
performs no I/O. The result method is nil-safe; the resolution method follows
|
||||
existing callback lifetime guards. Return a record copy containing a borrowed
|
||||
filesystem and slash-relative source BaseDir within its configured root.
|
||||
Consumers obtain source IDs from ordinary origins; there is no single-context
|
||||
query that guesses among composite or derived origins. Dependencies continue to
|
||||
identify the sources of application-derived values through existing metadata.
|
||||
|
||||
For native inputs, use a namespace-rooted view over the caller's filesystem and
|
||||
retain its lifetime semantics. For host input, retain the canonical root path,
|
||||
not an open `os.Root`. A private `fs.FS` adapter validates a relative resource
|
||||
name, resolves it using existing host containment semantics, opens a fresh root,
|
||||
and opens the resource. Return a private `fs.File` wrapper whose Close releases
|
||||
both file and root, including failed-open cleanup. Expose `fs.ReadDirFile` when
|
||||
the underlying directory supports it. Resources themselves are application data:
|
||||
do not apply configuration YAML parsing or configuration-byte budgets to them.
|
||||
Helpers such as `fs.ReadFile` then close the returned handle normally. Errors
|
||||
follow filesystem access conventions, separately from load diagnostics.
|
||||
|
||||
Neither queries nor retained results keep library-owned filesystem handles open.
|
||||
Files/resources can change or disappear after loading; configuration/metadata
|
||||
remain usable. No background work, hidden reopening on metadata queries,
|
||||
filesystem snapshot, universal sandbox, or transaction guarantee is added.
|
||||
|
||||
## Stage 1: Index mapping assembly and establish scaling baselines
|
||||
|
||||
**Completed 2026-09-29.** Private per-assembly mapping indexes now serve strict
|
||||
composition and recursive overlay lookup while ordered entries and diagnostics
|
||||
remain stable; conflict groups are checked once. Focused Compose/Overlay tests
|
||||
and the everyday checker passed. On Linux amd64, the 512-entry benchmark
|
||||
fixtures measured approximately 4.0 ms/op before and 3.3 ms/op after for wide
|
||||
composition, and 12.9 ms/op before and 12.0 ms/op after for recursive overlay.
|
||||
These are comparative evidence, not timing gates; provenance work still
|
||||
dominates the latter and belongs to the next stage.
|
||||
|
||||
Scope: `internal/document` and engine composition/overlay lookup, with existing
|
||||
behavior only.
|
||||
|
||||
1. Add parameterized benchmarks for wide disjoint composition and recursive
|
||||
overlay, recording runtime and allocations at increasing sizes. Keep fixtures
|
||||
within existing budgets and record the baseline in the stage handoff.
|
||||
2. Replace repeated linear mapping-key lookup during bulk assembly with a private
|
||||
indexed builder alongside ordered entries. Build each target index once per
|
||||
assembly session and update it for appended keys. Keep tree traversal and
|
||||
diagnostic order stable; avoid mutable indexes exposed through public APIs.
|
||||
3. Check each strict-composition claim group once while preserving first-conflict
|
||||
ordering and all relevant locations. Preserve original input ownership.
|
||||
4. Extend existing composition tests only for uncovered ordering/conflict risks.
|
||||
|
||||
Acceptance: existing Compose/Overlay tests pass unchanged; broad keyed input
|
||||
no longer incurs a full key scan for each insertion. Run the focused composition
|
||||
tests and the everyday checker. Benchmarks are evidence, not timing gates.
|
||||
|
||||
## Stage 2: Prepare load-owned assembly and incremental provenance
|
||||
|
||||
**Completed 2026-09-29.** Ordinary loads now apply selected overlays and CLI
|
||||
overrides to one owned working document, while copy-preserving helpers remain
|
||||
available for direct use. Acquired history trees stay detached; composite
|
||||
contributors survive overlay application. Origin deletion visits only replaced
|
||||
subtrees, mapping contributors append without copying growing lists, and history
|
||||
replay shares indexed lookup across applications. Focused behavior tests, the
|
||||
everyday checker, and the engine race suite passed. On Linux amd64 at 128 small
|
||||
applications, owned assembly measured about 0.30 ms/op versus 6.0 ms/op for
|
||||
repeated copy-preserving application; alternating subtree/scalar replacement
|
||||
measured about 0.39 versus 7.3 ms/op. These benchmarks are evidence, not gates.
|
||||
|
||||
Scope: `compose.go`, `overlay.go`, origin indexes, override application, and
|
||||
history replay in `internal/engine`.
|
||||
|
||||
1. Introduce a private load-owned accumulator with explicit ownership. Existing
|
||||
copy-preserving helpers may wrap it, but sequential assembly must mutate one
|
||||
owned working tree rather than clone all accumulated state per tier.
|
||||
2. Retain immutable acquired inputs for history, detached from mutable working
|
||||
nodes. Parameterize internal overlay application by incoming layer and origins
|
||||
so a composed tier's imported contributors are preserved.
|
||||
3. Remove displaced origins by visiting the affected subtree. Accumulate mapping
|
||||
contributors without copying an ever-growing ancestor list on each addition;
|
||||
materialize defensive public records at the existing publication boundary.
|
||||
4. Preserve equal-value replacement events, descendant displacement, redaction,
|
||||
and typed-decoding reconciliation. Extend benchmarks for many small applications
|
||||
and repeated subtree/scalar replacement. Do not add a persistent-tree framework.
|
||||
|
||||
Acceptance: existing override/history/rendering/ownership tests pass; benchmark
|
||||
growth avoids accumulated full-tree copying. Run the everyday checker and
|
||||
`GOPROXY=off go test -race ./internal/engine` for ownership changes.
|
||||
|
||||
## Stage 3: Unify private filesystem acquisition without public changes
|
||||
|
||||
**Completed 2026-09-29.** Imports, glob traversal, and selected-overlay loading
|
||||
now use a private access boundary with the existing contained host backend and
|
||||
a native `fs.FS` backend. Namespace tokens and occurrence identities are separate
|
||||
from labels and paths; repetition tracking is composition-local and input/glob
|
||||
accounting can be shared. Native tests cover slash-relative imports, sorted
|
||||
globs, optional missing paths, root escape, overlays, repeated/cyclic imports,
|
||||
non-comparable filesystems, and handle closure on success and representative
|
||||
failures. Existing host tests, focused native tests, and the everyday checker
|
||||
passed. No public constructor or context field was added.
|
||||
|
||||
Scope: engine source/file, filesystem, import, glob, and selected-overlay code.
|
||||
|
||||
1. Extract a small private namespace/access boundary used by imports and overlays.
|
||||
Keep the host implementation's canonicalization, containment, regular-file
|
||||
checks, safe diagnostics, deterministic traversal, and close behavior.
|
||||
2. Separate namespace identity from source labels and resource paths. Make
|
||||
repeated/cycle tracking belong to an input composition group, and accounting
|
||||
capable of being shared across groups.
|
||||
3. Add a native `fs.FS` backend internally. Resolve paths using slash semantics,
|
||||
preserve glob grammar and optional-missing behavior, and avoid host fallback.
|
||||
4. Use real host tests plus `fstest.MapFS` and a small tracking/failing filesystem
|
||||
to cover imports, missing paths, directory/read failures, and handle closure.
|
||||
Include a filesystem with a non-comparable dynamic value.
|
||||
|
||||
Acceptance: all existing host source/import/glob/overlay tests remain green;
|
||||
native backend tests establish namespace isolation and supported paths. Run the
|
||||
everyday checker. No public constructors land until the next stage.
|
||||
|
||||
## Stage 4: Expose native filesystem sources through the facade
|
||||
|
||||
**Completed 2026-09-29.** The root API now exposes reusable native filesystem
|
||||
namespaces, namespace-aware contexts, and source acquisition kinds. Ordinary
|
||||
loads support native files, opt-in imports, selected overlays, and byte/reader
|
||||
contexts while retaining host semantics. Public tests cover embedded fixtures,
|
||||
`fstest.MapFS`, same-label namespace isolation, wrapper copies and reuse,
|
||||
constructor/context diagnostics, and host/CLI source kinds. The everyday checker
|
||||
passed, including public-boundary and documentation checks.
|
||||
|
||||
1. Add `Filesystem`, `NewFilesystem`, `Filesystem.File`, and the additive FS
|
||||
context field at root and engine boundaries. Constructors follow deferred
|
||||
diagnostics; copying the wrapper preserves namespace identity.
|
||||
2. Integrate native sources with ordinary `Load`, including explicitly enabled
|
||||
imports and named overlays. Enforce the path/context rules settled above.
|
||||
3. Add `SourceKind` and populate it for every current/native acquisition. Preserve
|
||||
host paths and use namespace-relative virtual paths with accurate locations.
|
||||
4. Add facade-level tests using embedded fixtures, `fstest.MapFS`, byte/reader
|
||||
context, namespace-label collisions, and source reuse. Keep backend error cases
|
||||
at their Stage 3 owner. Update current source/API docs to this implemented scope.
|
||||
|
||||
Acceptance: native `Load` works through the public API and existing sources retain
|
||||
their contracts. Run the everyday checker, including public-boundary tests.
|
||||
|
||||
## Stage 5: Add snapshotted mapping sources
|
||||
|
||||
**Completed 2026-09-29.** `Mapping` now snapshots supported built-in values at
|
||||
construction, retaining safe deferred diagnostics for invalid roots, values,
|
||||
cycles, nonfinite floats, and limits. A shared private converter preserves CLI
|
||||
override behavior through a separate adapter. Mapping inputs participate in
|
||||
ordinary directives, imports, version checks, decoding, and inspection with
|
||||
their own source kind and synthetic locations. Public tests cover YAML
|
||||
equivalence, reuse after caller mutation, null/empty/zero distinctions, exact
|
||||
integers, explicit filesystem context, and safe failures. The everyday checker
|
||||
and engine race suite passed.
|
||||
|
||||
1. Extract reusable value conversion from `override_value.go`; keep CLI-specific
|
||||
path, label, and diagnostic handling in its adapter.
|
||||
2. Implement `Mapping` at root and engine boundaries with constructor-time
|
||||
snapshotting and deferred safe failures. Source-ID assignment at acquisition
|
||||
must not mutate a reusable stored tree shared by another load.
|
||||
3. Route converted documents through the same directive processing, strict
|
||||
composition, version checking, decode, and inspection path as YAML sources.
|
||||
4. Test omission/zero/false/null, nil versus empty collections/root, exact integers,
|
||||
invalid/cyclic values, imports with explicit context, and caller mutation after
|
||||
construction. Verify mapping input does not gain CLI origins or restrictions.
|
||||
|
||||
Acceptance: equivalent byte/mapping inputs have equivalent semantic results and
|
||||
origin meaning, with genuine location differences allowed. Existing override
|
||||
diagnostics remain stable. Run the everyday checker and engine race tests.
|
||||
|
||||
## Stage 6: Implement private grouped-underlay orchestration
|
||||
|
||||
**Completed 2026-09-29.** Ordinary and grouped engine loads now use one private
|
||||
request lifecycle. Group-local strict composition and repetition tracking,
|
||||
load-wide source/byte/node/glob accounting, main-owned version checking,
|
||||
underlay layer/index metadata, and precedence-ordered assembly feed one final
|
||||
decode, resolution, validation, and publication. Engine tests cover two tiers,
|
||||
multiple contributions, source/history order, absent main/defaults-only loads,
|
||||
main/import and group conflicts, version protection, shared limits, reusable
|
||||
files across tiers, and replacement of invalid intermediate typed input. The
|
||||
everyday checker and engine race suite passed. No root layered API is exposed yet.
|
||||
|
||||
1. Add private request/input/group structures matching the settled facade design.
|
||||
Adapt ordinary `Load` to the shared core while preserving its options and
|
||||
source-order behavior. Keep its existing validation order where observable.
|
||||
2. Strictly compose each group with group-scoped repetition tracking and shared
|
||||
aggregate accounting. Apply the lifecycle order defined above, including the
|
||||
single main-owned version check before underlay precedence assembly.
|
||||
3. Add underlay layer/index metadata and version diagnostics. Keep history in
|
||||
precedence order while source IDs follow acquisition order. Apply composed
|
||||
main origins over underlays without relabeling main imports as overlays.
|
||||
4. Reuse Stage 2 assembly and the existing decode/resolution/secret/publication
|
||||
pipeline. Empty groups, omitted main, defaults-only loads, and underlay-only
|
||||
loads must follow the chosen semantics.
|
||||
5. Add engine coverage for two underlay tiers, multiple strict contributions,
|
||||
main/import conflicts, version checks/absent main, limits across groups, and
|
||||
one final decode/resolve/validate lifecycle. Test an invalid intermediate typed
|
||||
value replaced by valid higher input, alongside unmaskable syntax conflicts.
|
||||
|
||||
Acceptance: engine layered requests produce correct values and full provenance;
|
||||
ordinary-load tests remain green. Run the everyday checker and engine race tests.
|
||||
|
||||
## Stage 7: Expose layered loading and complete inspection integration
|
||||
|
||||
**Completed 2026-09-29.** The root facade now offers `Input`, `Underlay`,
|
||||
`LayeredOptions`, and `LoadLayered` with copied options/context records. Explicit
|
||||
host or native selected-overlay contexts work independently of the main source;
|
||||
repetition uses native namespace identity rather than a display path alone.
|
||||
Root layer/source conversions and explanation rendering include underlays.
|
||||
Public tests cover mixed host/native/mapping tiers, recursive precedence,
|
||||
empty/zero/false/null and sequence replacement, source order and tier indexes,
|
||||
composite and displaced origins, resolution, redaction, and zero-underlay
|
||||
equivalence. The everyday checker and full race suite passed.
|
||||
|
||||
1. Add `Input`, `Underlay`, `LayeredOptions`, and `LoadLayered` root wrappers and
|
||||
conversions. Copy option slices/context records into private load state while
|
||||
preserving reusable namespace identities and single-consumption readers.
|
||||
2. Wire explicit selected-overlay context for layered loads, including host and
|
||||
native namespaces. The selected file may not duplicate an acquired file in
|
||||
that namespace, preserving selected-overlay repetition protection.
|
||||
3. Finish root Layer/SourceInfo conversions, history/explanation rendering,
|
||||
composite mapping attribution, deleted descendants, default and derived
|
||||
origins, and secret redaction. New records must remain detached.
|
||||
4. Add representative public tests for full precedence and same-name recursive
|
||||
mapping combination; retain zero/false/empty/null distinctions and atomic
|
||||
sequence replacement. Test ordinary versus zero-underlay layered equivalence
|
||||
using equivalent explicit overlay context when required.
|
||||
|
||||
Acceptance: callers can load native, host, and mapping tiers in one public call;
|
||||
configuration, metadata, and safe rendering agree. Run the everyday checker and
|
||||
`GOPROXY=off go test -race ./...`.
|
||||
|
||||
## Stage 8: Add handle-free source-context queries
|
||||
|
||||
**Completed 2026-09-29.** Results and active resolution callbacks now expose
|
||||
passive per-source `ResourceContext` records. Native views borrow caller-owned
|
||||
filesystems under their configured roots; host views retain only canonical root
|
||||
paths and open/close fresh roots with each resource file. Main, imported,
|
||||
selected-overlay, and explicitly contextual synthetic sources have their own
|
||||
base directories; contextless and CLI sources do not. Public tests verify
|
||||
origin-based resource access, no I/O on queries, later mutation/deletion,
|
||||
borrowed-filesystem closure, and callback guards. Tracking tests cover resource
|
||||
Open failure cleanup, returned file/directory ownership, and loading-handle
|
||||
closure across success and representative failures. The everyday checker and
|
||||
full race suite passed.
|
||||
|
||||
1. Retain context descriptors keyed by acquired source ID, including imported
|
||||
sources, in private inspection and live resolution state. Keep descriptors
|
||||
separate from serializable metadata and preserve callback guards.
|
||||
2. Implement the root query methods and `ResourceContext`. Native contexts expose
|
||||
rooted borrowed access; host contexts use the passive adapter described above.
|
||||
Include host main files even when imports are disabled, and explicit contexts
|
||||
on byte/reader/mapping sources. Sources without contexts return false.
|
||||
3. Verify each origin's BaseDir points to its actual source. Composite origins
|
||||
require callers to follow individual locations; derived dependencies retain
|
||||
their existing meaning. No arbitrary last-source context is selected.
|
||||
4. Test that queries cause no filesystem operations; loaded values and metadata
|
||||
survive source deletion; later resource access observes mutation/missing files;
|
||||
and caller filesystem closure affects only later resource operations.
|
||||
5. Verify owned files/root handles close on successful loading and representative
|
||||
acquisition, parse, composition, version, and validation failures. Test resource
|
||||
Open failure cleanup and returned-file Close ownership separately. Prefer
|
||||
narrow tracking seams over timing-based goroutine/descriptor-count assertions.
|
||||
|
||||
Acceptance: a retained underlay resource can be opened relative to its true origin
|
||||
after load, with no library-owned loading handles retained. Run the everyday
|
||||
checker and full race suite.
|
||||
|
||||
## Stage 9: Validate cross-feature consistency and scaling
|
||||
|
||||
**Completed 2026-09-29.** A compact public matrix now checks equivalent host,
|
||||
native, byte, reader, and mapping inputs; tier-local repetition versus cross-tier
|
||||
reuse; reader consumption; and safe diagnostics/effective/history output after
|
||||
secret replacement. Engine tests verify exact occurrence accounting across
|
||||
imports, shared glob/source limits, and the mixed-source CLI ceiling. Rendering
|
||||
now indexes source labels once per explanation instead of scanning all records
|
||||
for every location. Loading and passive context ownership were reviewed against
|
||||
ADR 0008; no handles or background work are retained by queries.
|
||||
|
||||
On Linux amd64 with `-benchmem` and 100 ms samples, the 512-entry assembly-only
|
||||
composition measured about 3.5 ms/op (the earlier post-index baseline was about
|
||||
3.3 ms), and recursive overlay about 2.2 ms/op (about 12 ms at the first-stage
|
||||
handoff before subsequent provenance work). The 128-application owned assembly
|
||||
remained about 0.30 ms/op versus 6.35 ms/op for copy-preserving application.
|
||||
End-to-end layered measurements were about 7.15 ms/op for 512 broad catalog
|
||||
entries, 2.59 ms/op for 128 small tiers, 3.27 ms/op for 128 alternating subtree
|
||||
replacements, 1.86 ms/op for 64 imported native defaults, and 1.36 ms/op for
|
||||
64 retained-history applications. Allocations grew roughly with these fixture
|
||||
sizes. I/O, sorting, decoding, callbacks, and retained output contribute to
|
||||
full-path cost; these observations are not timing gates or an unconditional
|
||||
complexity guarantee. The everyday checker and full race suite passed.
|
||||
|
||||
1. Consolidate a small semantic equivalence matrix across host, native, bytes,
|
||||
readers, and mappings, using equivalent logical inputs and contexts. Cover
|
||||
strict conflict versus intentional precedence and version-path preservation;
|
||||
avoid duplicating every lower-level case at every source permutation.
|
||||
2. Exercise load-wide limits across mixed sources/CLI, repeated explicit tiers,
|
||||
shared namespace imports, and source IDs/history order. Protect against double
|
||||
charging and reader re-consumption. Confirm secret canaries stay absent from
|
||||
diagnostics and historical/effective output after replacements.
|
||||
3. Run parameterized benchmarks for broad catalogs, many small tiers, subtree
|
||||
replacement, imported native defaults, and retained history. Compare Stage 1
|
||||
baselines with the final path. Address demonstrated avoidable quadratic scans
|
||||
or accumulated copying within this feature's scope. Account for sorting,
|
||||
filesystem operations, callbacks, and output/history size explicitly.
|
||||
4. Review source/context ownership and success/failure cleanup against ADR 0008.
|
||||
Keep performance evidence in the stage handoff, not permanent timing gates.
|
||||
|
||||
Acceptance: no semantic discrepancy remains among equivalent operations; scaling
|
||||
results substantiate the engineering target without an unconditional O(n) claim.
|
||||
Run the everyday checker and full race suite, reusing successful checks for an
|
||||
unchanged final tree. Benchmarks run with `-benchmem` and parameterized sizes.
|
||||
|
||||
## Stage 10: Complete consumer examples and architecture documentation
|
||||
|
||||
**Completed 2026-09-29.** Maintained executable examples now cover a simple
|
||||
embedded filesystem source and a layered catalog with imported embedded defaults,
|
||||
a mapping underlay, an independent host main file, explicit false/zero values,
|
||||
domain validation, and resource access through the retained origin's source ID.
|
||||
The configuration contract, public declaration comments, consumer/development
|
||||
guides, README, component overview, architecture/deferred text, roadmap status,
|
||||
and draft ADR status links reflect implemented behavior. Named inheritance remains
|
||||
separate and deferred; the ADRs remain drafts. The everyday checker and full race
|
||||
suite passed. No release preparation or publication was performed.
|
||||
|
||||
1. Add a maintained simple native-filesystem example and a combined example with
|
||||
embedded defaults, two underlay tiers, independently located main input,
|
||||
programmatic contributions, and a relative resource from a retained origin.
|
||||
Keep fixtures secret-free and offline; include zero/false presence and domain
|
||||
validation. The consumer should perform no generic post-load merge/defaulting
|
||||
or provenance reconstruction. Reuse existing example/test conventions.
|
||||
2. Update `docs/config.md`, Go declarations, `docs/consumers/`, README,
|
||||
`docs/internal/overview.md`, and applicable architecture/deferred-scope text
|
||||
to describe implemented support. Preserve strict-within-tier and recursive-
|
||||
between-tier terminology, distinguish source kinds from precedence roles,
|
||||
and document context lifetimes and the explicit source-ID query workflow.
|
||||
3. Add the relevant tests/commands to the development guide's existing task map.
|
||||
Review ADR links and statuses, preserving draft status pending acceptance and
|
||||
accepted ADR decision bodies. Keep named inheritance separately deferred.
|
||||
4. Mark implementation progress accurately in this plan and the roadmap. Do not
|
||||
retire plans, choose/tag a version, prepare release gates, or publish as part
|
||||
of implementation; those belong to separate release preparation.
|
||||
|
||||
Acceptance: examples compile and execute, documentation describes only completed
|
||||
capabilities, and simple `Load` usage remains simple. Run the everyday checker
|
||||
(which includes examples, links, and boundary checks); run the full race suite
|
||||
if code changed since its last successful run. Report completed stages, checks,
|
||||
benchmark conclusions, and any remaining limitations honestly.
|
||||
|
||||
## Validation commands
|
||||
|
||||
Use the development guide's focused commands during each stage. The normal
|
||||
stage-end check is `./scripts/check.sh`; it already includes tests, vet, build,
|
||||
formatting, and documentation links. Ownership stages additionally use the race
|
||||
commands above. Benchmarks may use:
|
||||
|
||||
```sh
|
||||
GOPROXY=off go test ./internal/engine -run '^$' -bench 'Compose|Overlay|Underlay' -benchmem
|
||||
```
|
||||
|
||||
Name the new benchmarks to match that expression. Select representative sizes
|
||||
under existing limits. No CI wait, human macOS validation, external service,
|
||||
credentials, or new release gate is part of this plan.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None required to implement this plan. The remaining roadmap recommendations
|
||||
have been selected above: labeled snapshotted mappings and separate source-context
|
||||
queries. Private helper names and package-local factoring remain implementation
|
||||
choices within these contracts. If implementation uncovers a genuine conflict
|
||||
with existing behavior, report it rather than silently adding a restriction,
|
||||
weakening compatibility, or broadening a guarantee.
|
||||
Reference in New Issue
Block a user