eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Document and demonstrate native and layered configuration loading

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-29 18:22:45 +00:00
parent 34e18c84c1
commit cbf031aedc
21 changed files with 291 additions and 46 deletions
Download Patch File Download Diff File Expand all files Collapse all files

3
docs/adr/0007-source-backed-underlays.md
View File

@@ -4,7 +4,8 @@
Proposed draft for review. The
[filesystem sources and underlays roadmap](../roadmap/sources-and-layering.md)
owns implementation status, detailed target behavior, and remaining API choices.
records the target and implementation progress; the
[configuration contract](../config.md#layered-loading) owns current behavior.
This extends [ADR 0001](0001-separate-composition-and-overlay.md) and preserves
its distinction between strict composition and intentional overlay.

5
docs/adr/0008-complete-on-return-loading.md
View File

@@ -4,9 +4,8 @@
Proposed draft for review. The
[configuration contract](../config.md#completion-and-resource-lifetime) describes
the implemented loading lifecycle. The
[source-backed underlays roadmap](../roadmap/sources-and-layering.md) owns its
application to the planned source and resource-context APIs.
the implemented loading lifecycle and
[source resource contexts](../config.md#source-resource-contexts).
## Date

10
docs/config.md
View File

@@ -148,6 +148,10 @@ have index zero. Explicit empty groups and `NoSource` consume no source ID.
Byte, source, node, and glob-entry budgets are shared across the load, while
the existing per-source and import-depth bounds still apply.
The maintained [layered consumer example](consumers/README.md#native-filesystem-and-layered-input)
shows a main host file, two underlay tiers, embedded defaults, a mapping source,
and explicit resource access without a second configuration pass.
### Source resource contexts
`Result.SourceContext(id)` and callback-scoped `Resolution.SourceContext(id)`
@@ -165,6 +169,12 @@ contextless synthetic inputs have no resource context. To open a relative
resource, join `BaseDir` with the application resource name using slash-path
operations and open that path through `FS`.
For a resource-valued field, query `result.Metadata().Lookup(fieldPath)`, inspect
its origin locations, choose a source ID only when the application's policy can
identify the intended contributor, then call `result.SourceContext(id)`. The
context belongs to that source even when the field survived from a lower-priority
underlay. A composite origin has no automatically selected resource base.
Native contexts borrow the caller's filesystem within the configured root.
Host contexts retain a canonical root path, not an open root handle; each
resource `Open` checks containment and opens a fresh handle that the returned

26
docs/consumers/README.md
View File

@@ -7,7 +7,7 @@ validation, schema-version policy, CLI parsing, runtime matching and conversion,
resource checks, remote acquisition, and service initialization.
The maintained [`examples/consumers`](../../examples/consumers) package demonstrates
four sanitized shapes using only ConfigKit and the Go standard library:
sanitized shapes using only ConfigKit and the Go standard library:
| Shape | Demonstrated fit | Application code deliberately retained |
| --- | --- | --- |
@@ -15,6 +15,8 @@ four sanitized shapes using only ConfigKit and the Go standard library:
| Notarius-like | Canonical object bindings, a string-backed schema version accepting string or integer YAML, strict dynamic options, and `int64` dynamic integers | Catalog selection, allowed option keys, range-checked conversion to runtime `int` |
| WeatherReporter-like | Defaults-only loading, report maps, canonical module objects, typed option unions, explicit CLI records, and typed secret startup | Report/module compatibility, rejection of explicitly empty domain lists, CLI parsing, and startup ordering |
| Seriatim-like | Independent strict ordered speaker and autocorrect rule files | First-match and ordered replacement algorithms; no central application configuration file |
| Bundled file | Simple native `fs.FS` source through ordinary `Load` | Selection and lifetime of the application-owned embedded filesystem |
| Layered catalog | Embedded defaults with an import, a mapping underlay, an independent host main file, explicit false/zero input, and source-relative resource access | Domain validation and opening/interpreting the template resource |
The YAML under [`testdata`](../../examples/consumers/testdata) is loaded by the
executable examples and ordinary tests. It contains no private application data,
@@ -48,6 +50,27 @@ both without redefining their syntax. ConfigKit overlays are configuration overl
they do not select or replace an application's LLM, model, pipeline, report, or
execution profile concepts.
## Native filesystem and layered input
[`LoadBundled`](../../examples/consumers/layered.go) keeps the simple `Load` shape:
it selects a file in an application-owned embedded filesystem and needs no
underlay setup. [`LoadCatalog`](../../examples/consumers/layered.go) uses
`LoadLayered` with an imported embedded-defaults group, a separate
`Mapping` group, and a host main file supplied by its caller. The groups
compose strictly within themselves and recursively across precedence boundaries;
the main file's explicit `false` and `0` replace lower values without being
mistaken for omissions. Its validator owns allowed modes and resource-name policy.
[`ReadWelcome`](../../examples/consumers/layered.go) follows the effective field's
origin location to a source ID, queries `Result.SourceContext`, and opens the
resource relative to that context's `BaseDir`. It does not assume the main file
owns a retained underlay value. A composite origin needs an application choice;
ConfigKit does not guess a directory. The context query is passive, while later
resource reads observe the caller-owned filesystem or a freshly opened host root
and may fail independently of the completed configuration load. The
[configuration contract](../config.md#source-resource-contexts) owns exact
lifetime and precedence rules.
Known module catalogs should use canonical object entries with a module identifier
and typed options. A small union struct is sufficient when the available option
fields are known; validation selects which fields apply to each module. Runtime-
@@ -126,6 +149,7 @@ focused behavioral owner:
| Source/import/selected-overlay pipeline rules and artifact defaults | Narratio-shaped executable example and fixture |
| Canonical module objects and dynamic option handling | WeatherReporter- and Notarius-shaped examples and migration tests |
| `LoadSecretsDir`, bootstrap reporting, typed lookup, CLI explanation | `Example_weatherStartup` |
| Native filesystem, grouped underlays, programmatic mapping, retained resource context | `Example_nativeFilesystem`, `Example_layeredCatalog`, and their focused consumer tests |
| Results, metadata, diagnostics, effective output, and writer failures | Focused root-package load, provenance, rendering, and explanation tests |
The package-level import-boundary test covers the consumer examples too: supported

6
docs/development.md
View File

@@ -133,12 +133,16 @@ code changes. Use the component overview to find the corresponding implementatio
| --- | --- | --- |
| YAML syntax, diagnostics, parser limits | [Adapter tests](../internal/yaml/parse_test.go), [source translation](../internal/engine/parse_test.go) | `go test ./internal/yaml` and `go test ./internal/engine -run '^TestParseSource'` |
| File/reader access, imports, globs, containment | [Sources](../internal/engine/source_test.go), [FIFO](../internal/engine/source_fifo_test.go), [imports](../internal/engine/imports_test.go), [globs](../internal/engine/glob_test.go) | `go test ./internal/engine -run 'Source\|Import\|Glob\|Filesystem\|Nonregular\|RegularFile'` |
| Native filesystem and mapping sources | [Native acquisition](../internal/engine/native_filesystem_test.go), [filesystem facade](../facade_filesystem_test.go), [mapping facade](../facade_mapping_test.go) | `go test ./internal/engine -run 'NativeFilesystem'` and `go test . -run 'TestFacade'` |
| Composition, selected overlays, CLI precedence | [Composition](../internal/engine/compose_test.go), [overlay](../internal/engine/overlay_test.go), [selected overlays](../internal/engine/selected_overlay_test.go), [overrides](../internal/engine/override_test.go) | `go test ./internal/engine -run 'Compose\|Overlay\|Override'` |
| Layered loading and source equivalence | [Grouped lifecycle](../internal/engine/load_request_test.go), [layered facade](../facade_layered_test.go), [equivalence](../facade_equivalence_test.go) | `go test ./internal/engine -run '^TestGrouped'`, `go test . -run '^TestFacadeLayered'`, and `go test . -run '^TestEquivalentSourceForms'` |
| Schema, defaults, projection, cloning | [Schema](../internal/engine/schema_test.go), [decoding](../internal/engine/decode_test.go), [projection](../internal/engine/projection_test.go) | `go test ./internal/engine -run 'Schema\|Decode\|Projection\|Clone\|Inline\|CompareDocument'` |
| Lifecycle, resolution, validation, callback ownership | [Loading](../internal/engine/load_test.go), [resolution](../internal/engine/resolution_test.go), [validation](../internal/engine/validation_test.go), [ownership](../internal/engine/callback_ownership_test.go), [independence](../internal/engine/independence_test.go) | `go test ./internal/engine -run 'Load\|Resolution\|Validation\|Validator\|Hook\|Callback\|Parallel'` |
| Paths, metadata, provenance | [Paths](../internal/engine/path_test.go), [metadata](../internal/engine/metadata_test.go), [provenance](../internal/engine/provenance_test.go) | `go test ./internal/engine -run 'Path\|Metadata\|Provenance\|Queries'` |
| Passive resource contexts | [Engine contexts](../internal/engine/resource_context_test.go), [public contexts](../facade_resource_context_test.go) | `go test ./internal/engine -run 'Resource'` and `go test . -run 'SourceContext'` |
| Secrets, bootstrap, redaction, explanation | [References](../internal/engine/secret_reference_test.go), [bootstrap](../internal/engine/secrets_test.go), [rendering](../internal/engine/render_test.go), [explanation](../internal/engine/explain_test.go) | `go test ./internal/engine -run 'Secret\|Bootstrap\|Effective\|Explain\|Explanation\|History\|Sensitivity'` |
| Consumer examples and migration fixtures | [Consumer tests](../examples/consumers/consumers_test.go), [API examples](../example_test.go) | `go test ./examples/consumers` and `go test . -run '^Example'` |
| Consumer examples and migration fixtures | [Consumer tests](../examples/consumers/consumers_test.go), [native and layered examples](../examples/consumers/layered_test.go), [API examples](../example_test.go) | `go test ./examples/consumers` and `go test . -run '^Example'` |
| Assembly and layered scaling evidence | [Assembly benchmarks](../internal/engine/assembly_bench_test.go), [layered benchmarks](../internal/engine/underlay_bench_test.go) | `go test ./internal/engine -run '^$' -bench '^BenchmarkUnderlay' -benchmem`; repeat with `^BenchmarkCompose` and `^BenchmarkOverlay` |
| Documentation links and dependency isolation | [Link checks](../documentation_test.go), [import boundary](../import_boundary_test.go), [public boundary](../public_boundary_test.go) | `go test . -run 'TestLocalDocumentationLinks\|TestDependencyImportBoundary\|TestPublicFacadeBoundary'` |
| Development and release checker behavior | [Shell regression](../release_check_test.go) | `go test . -run 'TestCheckScripts\|TestDevelopmentCheck'` |
| Guarded release publication and recovery | [Publication regression](../release_publish_test.go) | `go test . -run '^TestReleasePublication'` |

25
docs/internal/overview.md
View File

@@ -7,23 +7,26 @@ private state; public signatures and fields do not expose engine types.
Implemented component owners:
- `internal/engine` owns source acquisition, composition, schema inspection,
decoding, lifecycle coordination, callback auditing, provenance/history,
secret resolution and bootstrap, and safe inspection rendering. It never
imports the root package.
- Root `Source`, `Path`, `Secret`, records, lifecycle callbacks, results, and
bootstrap reports are root-defined wrappers or records. Callback wrappers pass
only live engine context during a callback; `Result.Config` is detached and its
private inspection remains frozen independently.
- `internal/engine` owns host and native-filesystem source acquisition,
grouped strict composition and recursive precedence assembly, schema
inspection, decoding, lifecycle coordination, callback auditing,
provenance/history, passive source-context retention, secret resolution and
bootstrap, and safe inspection rendering. It never imports the root package.
- Root `Source`, `Filesystem`, `Input`, `Underlay`, `Path`, `Secret`, records,
lifecycle callbacks, results, and bootstrap reports are root-defined wrappers
or records. Callback wrappers pass only live engine context during a callback;
`Result.Config` is detached and its private inspection remains frozen
independently.
- Root conversion files explicitly cross the path, record, callback, secret, and
error boundaries. There are no public aliases, anonymous engine embeddings, or
public engine bridge methods.
- `internal/document` owns the parser-independent ordered tree and cloning.
`internal/yaml` is the sole upstream YAML adapter and parser/dumper boundary.
- Engine mechanism tests own source, composition, decoding, callback, provenance,
secret, history, rendering, and bootstrap behavior. Root tests own public API
compatibility, facade integration, documentation links, and dependency/public
boundary checks. Consumer examples remain under `examples/consumers`.
source-context, secret, history, rendering, and bootstrap behavior. Root tests
own public API compatibility, facade integration, documentation links, and
dependency/public boundary checks. Consumer examples remain under
`examples/consumers`.
Dependency direction is root to engine to document/YAML support. The root may
import only `internal/engine` among project internal production packages; engine

15
docs/policy/architecture.md
View File

@@ -161,9 +161,11 @@ Tier precedence is explicit; traversal order within a tier establishes stable
reporting, not permission to override. The distinction and its rationale are
recorded in [ADR 0001](../adr/0001-separate-composition-and-overlay.md).
The main document and its imports currently form the strictly composed base
tier. Input-specific constraints, including CLI path rules and opt-in version
ownership, remain part of the configuration contract. Consistent combination
The main document and its imports form a strictly composed tier. Explicit
underlay groups compose strictly within each group, then combine recursively
from lowest to highest beneath the main tier. Source acquisition kind does not
determine its precedence role. Input-specific constraints, including CLI path
rules and opt-in version ownership, remain part of the configuration contract. Consistent combination
semantics do not require identical acquisition or lifetime behavior from every
source kind.
@@ -172,9 +174,10 @@ parser defaults. Processing order and diagnostics must be deterministic for the
same inputs and options. Invalid input must not produce a successful usable
configuration. Do not run application hooks on invalid typed configuration.
User-controlled precedence is application defaults, composed base, selected
overlay, then explicitly supplied CLI overrides. Applications own flag parsing;
ConfigKit owns overlay semantics and provenance. Domain resolution follows these
User-controlled precedence is application defaults, ordered underlays, composed
main input, selected overlay, then explicitly supplied CLI overrides.
Applications own flag parsing; ConfigKit owns overlay semantics and provenance.
Domain resolution follows these
inputs and must remain recorded, not become a hidden override channel.
Preserve node-based composition until the effective input is ready for typed

6
docs/roadmap/deferred.md
View File

@@ -11,11 +11,13 @@ General environment-variable configuration overlays are not supported. Typed sec
references and explicit directory bootstrap are the environment convention. Literal
credential configuration and implicit secret-directory discovery are unsupported.
The initial release deliberately excludes:
The following remain deliberately excluded:
- automatic `conf.d` discovery;
- remote imports, arbitrary providers, and additional configuration formats;
- custom merge strategies, sequence merging, reset operators, and overlay inheritance;
- custom merge strategies, sequence merging, and reset operators;
- named overlay inheritance, tracked separately in the
[PromptKit inheritance request](feature-request-promptkit-inheritance.md);
- parser-specific or custom structural-decoding extension APIs;
- generic dynamic-subtree conversion and module-registry APIs;
- automatic schema migrations;

10
docs/roadmap/implementation.md
View File

@@ -506,6 +506,16 @@ unchanged final tree. Benchmarks run with `-benchmem` and parameterized sizes.
## Stage 10: Complete consumer examples and architecture documentation
**Completed 2026-09-29.** Maintained executable examples now cover a simple
embedded filesystem source and a layered catalog with imported embedded defaults,
a mapping underlay, an independent host main file, explicit false/zero values,
domain validation, and resource access through the retained origin's source ID.
The configuration contract, public declaration comments, consumer/development
guides, README, component overview, architecture/deferred text, roadmap status,
and draft ADR status links reflect implemented behavior. Named inheritance remains
separate and deferred; the ADRs remain drafts. The everyday checker and full race
suite passed. No release preparation or publication was performed.
1. Add a maintained simple native-filesystem example and a combined example with
embedded defaults, two underlay tiers, independently located main input,
programmatic contributions, and a relative resource from a retained origin.

9
docs/roadmap/sources-and-layering.md
View File

@@ -1,8 +1,9 @@
# Filesystem Sources and Source-Backed Underlays
Status: agreed target for implementation; not implemented. This document owns
the desired behavior and design rationale. The [implementation plan](implementation.md)
owns API signatures, sequential work, and validation commands.
Status: implemented; draft ADRs remain pending review. This document preserves
the design target and rationale, including its historical baseline. The
[configuration contract](../config.md) owns current behavior, and the
[implementation plan](implementation.md) records sequential work and validation.
## Purpose and priorities
@@ -36,7 +37,7 @@ The [named-inheritance request](feature-request-promptkit-inheritance.md) remain
separate and optional. This roadmap neither implements it nor adds speculative
infrastructure for it.
## Current baseline
## Planning baseline (historical)
Planning baseline: `ec26591b275bd74f89120bdea69c5f6459dfcafb`, following v0.4.2.
Current behavior is owned by the [configuration contract](../config.md), not by