eric/ConfigKit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Unify layered provenance history

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-06 21:09:52 +00:00
parent ef0e6078f7
commit 26cece6d53
9 changed files with 347 additions and 48 deletions
Download Patch File Download Diff File Expand all files Collapse all files

19
docs/config.md
View File

@@ -79,7 +79,7 @@ error without reporting one, ConfigKit emits a generic `version.hook_failed`
diagnostic and suppresses the returned error text. Warnings alone are retained on a
successful result. Callback panics remain programming failures.
### Layered input/default metadata
### Layered metadata
`Result.Metadata` describes every final canonical path. Exact surviving input paths
are explicit and have a base or profile origin; omitted paths have a default origin.
@@ -91,6 +91,23 @@ metadata while retaining their base definition and the parent replacement operat
in history.
All metadata queries and returned nested records are detached.
History is chronological: a path can have a synthetic default record, its exact base
definition, a profile replacement, and a recorded resolution operation. A descendant
query uses the ancestor's path for a whole-container or whole-sequence replacement;
the descendant can therefore retain history after `Lookup` stops reporting it as
effective. Equal-valued input is still an explicit base definition, while a derived
value for absent input remains non-explicit.
A mapping assembled from the selected file and imports has one composite base origin
whose locations retain source traversal order. A mapping also visited by a profile
has a profile-layer composite origin with the lower contributing locations followed
by the profile location; an empty mapping is still an explicit container even though
it contributes no children. Resolution origins are synthetic: they have layer
`resolution`, no fabricated file location, and a `derived_default` or `normalization`
operation. Reasons and dependency paths retain first occurrence order with duplicates
removed. Source records remain ordered as selected source, depth-first imports, then
the selected profile.
### Recorded application resolution
An optional `Definition.Resolve` callback runs after successful typed decoding. It

12
docs/internal/extraction.md
View File

@@ -188,6 +188,18 @@ Assigned to layered metadata, redaction, rendering, and explanation:
records, and redaction. Narratio commands, diff/digest APIs, generated artifact
attribution, campaign logic, and party expansion remain application-owned.
Layered-provenance implementation record (2026-09-06): ConfigKit integrated its
previously extracted source claims, overlay displacement records, typed baseline,
and audited resolution snapshots in `load_metadata.go`. The representative
`TestLayeredProvenanceAgreesWithResolvedSnapshot` adapts Narratio's stable ownership,
profile ownership, deterministic effective projection, and side-effect-free
inspection cases across one default/import/profile/resolution workflow. ConfigKit
adds chronological descendant history for parent and whole-sequence operations,
composite empty-mapping origins, canonical resolution reasons/dependencies, private
snapshot agreement, defensive nested copies, and concurrent load isolation. Value
rendering, redaction, explanation, commands, generated attribution, diffs, and
digests remain assigned to their later owning work.
### Defaults, presence, and domain resolution
Assigned across schema, decoding, resolution, validation, and consumer examples:

8
docs/internal/overview.md
View File

@@ -24,8 +24,8 @@ module and package documentation boundary. Its implemented public primitives are
baseline/final trees, and initial default/input origins; and
- public lifecycle coordination in `load.go`, bounded canonical import DFS in
`imports.go`, pattern-directed glob expansion in `glob.go`, version-report contexts
in `validation.go`, and detached
base/default metadata assembly in `load_metadata.go`; and
in `validation.go`, and detached layered metadata/history assembly in
`load_metadata.go`; and
- recorded application defaulting and normalization in `resolution.go`, including
explicitness queries, helper verification, structural mutation audit, coalesced
intent, private before/after snapshots, and resolution metadata integration; and
@@ -33,8 +33,8 @@ module and package documentation boundary. Its implemented public primitives are
`validation.go`, completing the public callback lifecycle after composed input.
The [configuration contract](../config.md) owns implemented input, schema, decoding,
loading lifecycle, path imports, composition, profile overlay, version checking, recorded resolution,
semantic validation, metadata, and ownership behavior.
loading lifecycle, path imports, composition, profile overlay, version checking,
recorded resolution, semantic validation, metadata, and ownership behavior.
The source-extraction inventory records the application foundations assigned to
future implementation work:

28
docs/roadmap/implementation.md
View File

@@ -2,7 +2,7 @@
## Status and ownership
Status: implementation in progress; Stages 1–13 are complete and later stages are
Status: implementation in progress; Stages 1–14 are complete and later stages are
pending.
Audience: a `gpt-5.6-terra` coding agent implementing one stage per prompt, in
@@ -1038,7 +1038,31 @@ Acceptance:
## Stage 14 — Verify layered provenance and final snapshot agreement
Status: pending.
Status: complete.
Completion evidence (2026-09-06):
- Unified default, composed base, profile displacement, and resolution records in
the metadata history index. Container resolution now appends its operation to each
changed, added, or removed descendant while preserving the ancestor operation path
and path-specific previous origin; effective metadata still contains only the final
projected paths.
- Added a representative public `Load` workflow spanning a fresh default, selected
source, depth-first import, profile, derived default, mapping deletion, and
whole-sequence normalization. It verifies source order, equal-to-default
explicitness, null state, composite empty-mapping origins, chronological parent
replacement history, deduplicated reasons/dependencies, typed/final snapshot
agreement, and unchanged siblings.
- Verified nested metadata, source import locations, diagnostics, caller-owned
configuration, and private snapshots are detached. Repeated concurrent selected-
profile loads retain independent typed values, metadata, sources, and histories;
the race baseline passes.
- Promoted layered metadata semantics and concrete composite/synthetic origin cases
to the [current metadata contract](../config.md#layered-metadata), updated Go
reference comments, and recorded the adapted Narratio ownership/inspection cases.
Rendering, redaction, value snapshots, and explanation remain within later stages.
Verified `go test ./...`, `go test -race ./...`, `go vet ./...`, `go build ./...`,
`go mod tidy -diff`, and `git diff --check` with Go 1.26.7 on linux/amd64.
Read: feature [presence](initial.md#paths-and-presence),
[history](initial.md#provenance-diagnostics-and-redaction), metadata API.

49
docs/roadmap/initial.md
View File

@@ -430,23 +430,12 @@ conversion through the current loading lifecycle.
## Paths and presence
Semantic path construction, JSON Pointer parsing, and base/default metadata
population are implemented. Their current contracts are owned by the Go
Semantic path construction, JSON Pointer parsing, and layered metadata population
are implemented. Their current contracts are owned by the Go
documentation in [`path.go`](../../path.go), [`metadata.go`](../../metadata.go),
and the [current configuration contract](../config.md#base-default-metadata).
Layered replacement and resolution behavior below remains target lifecycle work.
Distinguish absent, explicitly null, explicitly zero/false/empty, and other
explicit values. An explicit value equal to its default remains explicit.
Presence queries describe the surviving input contribution after overlays;
separate history queries describe earlier appearances. A parent mapping's
presence does not make all of its children explicit. Replacing a subtree removes
its descendants from effective presence and origin queries while retaining
historical replacement records.
Application normalization preserves the fact that a surviving input was
explicit. A derived value for an absent input remains non-explicit. Its final
origin is a resolution operation, with input origins retained separately.
and the [current metadata contract](../config.md#layered-metadata), including
presence, composite origins, parent replacements, removed-path history, and
synthetic resolution operations.
## Composition and overlay
@@ -481,8 +470,7 @@ The [current application schema contract](../config.md#application-schema-bindin
owns root/type support, field and tag binding, inlining, codec pairing, and
schema-aware path rules. The [current defaults and typed-decoding contract](../config.md#defaults-and-typed-decoding)
owns strict conversions, container application, presence, safe failures, and initial
origins. Public lifecycle integration, layered provenance, secret handling, and
rendering remain target work below.
origins. Secret handling and rendering remain target work below.
## Recorded resolution and validation
@@ -490,7 +478,8 @@ Application-owned recorded resolution is implemented. The
[current resolution contract](../config.md#recorded-application-resolution) owns
explicitness queries, helpers and manual records, structural mutation auditing,
coalescing, diagnostics, origins, history, and snapshot ownership. Layered input
history, redacted explanation, and secret dependency propagation remain target work.
history is owned by the [current metadata contract](../config.md#layered-metadata);
redacted explanation and secret dependency propagation remain target work.
Canonical typed projection, detached cloning, structural comparison, and publishing
independently captured resolved results are implemented. The
@@ -507,16 +496,10 @@ callback lifetime, and application/runtime responsibility boundaries.
Safe diagnostic/provenance value records, canonical diagnostic rendering, and
`LoadError` access are implemented. Their current contracts are owned by
[`records.go`](../../records.go) and [`diagnostic.go`](../../diagnostic.go).
Base/default and resolution records are attached to the current loading lifecycle.
Additional layers, safe historical values, redaction, and rendering remain target
work below.
Retain input sources, layer, line/column where available, import sites, effective
origins, overridden definition records, and resolution operations. Container
origins may be composite; never invent one file location for a mapping assembled
from several sources. Defaults and resolution have synthetic origins rather
than fabricated file coordinates. Preserve definition history without exposing
mutable parser nodes or unrestricted raw value references through the API.
Default, base, profile, and resolution records are attached to the loading lifecycle;
their source ordering, composite/synthetic origins, replacement history, and detached
queries are owned by the [current metadata contract](../config.md#layered-metadata).
Safe historical values, redaction, and rendering remain target work below.
Diagnostics have stable codes, severity, semantic path when available, a safe
message, primary location, and related locations. Sort deterministically by
@@ -697,11 +680,11 @@ func ReadWorkerDefaults() (*configkit.Result[Config], error) {
### Paths and metadata
The path API, detached metadata records, empty/unknown query behavior, and
base/default population are now implemented; [`path.go`](../../path.go),
The path API, detached metadata records, empty/unknown query behavior, and layered
population are implemented; [`path.go`](../../path.go),
[`records.go`](../../records.go), [`metadata.go`](../../metadata.go), and the
[metadata configuration contract](../config.md#base-default-metadata) own current
behavior. Layered callback history and safe historical values remain target behavior.
[metadata configuration contract](../config.md#layered-metadata) own current
behavior. Safe historical values remain target behavior for explanation.
### Resolution and validation callbacks

43
load_metadata.go
View File

@@ -1,6 +1,10 @@
package configkit
import "gitea.maximumdirect.net/eric/ConfigKit/internal/document"
import (
"slices"
"gitea.maximumdirect.net/eric/ConfigKit/internal/document"
)
func metadataFromDecoded(decoded decodedConfiguration) Metadata {
metadata := Metadata{
@@ -140,6 +144,7 @@ func applyResolutionMetadata(metadata *Metadata, resolved resolutionResult) {
}
metadata.history[pointer] = append(metadata.history[pointer], change)
}
appendResolutionDescendantHistory(metadata, resolved)
for _, delta := range resolved.deltas {
record := coveringResolutionSnapshot(resolved.snapshots, delta.path)
if record == nil {
@@ -157,6 +162,42 @@ func applyResolutionMetadata(metadata *Metadata, resolved resolutionResult) {
refreshResolutionFields(metadata.fields, resolved.final, resolved.snapshots)
}
func appendResolutionDescendantHistory(metadata *Metadata, resolved resolutionResult) {
finalPaths := make(map[string]*document.Node)
indexDocument(resolved.final, Path{}, finalPaths)
for _, delta := range resolved.deltas {
record := coveringResolutionSnapshot(resolved.snapshots, delta.path)
if record == nil {
continue
}
affected := make(map[string]bool)
for pointer := range metadata.history {
if path, err := ParsePath(pointer); err == nil && pathContains(delta.path, path) {
affected[pointer] = true
}
}
for pointer := range finalPaths {
if path, err := ParsePath(pointer); err == nil && pathContains(delta.path, path) {
affected[pointer] = true
}
}
pointers := make([]string, 0, len(affected))
for pointer := range affected {
if pointer != record.change.Path.String() {
pointers = append(pointers, pointer)
}
}
slices.Sort(pointers)
for _, pointer := range pointers {
change := cloneChange(record.change)
if info, ok := metadata.fields[pointer]; ok {
change.Previous = []Origin{cloneOrigin(info.Origin)}
}
metadata.history[pointer] = append(metadata.history[pointer], change)
}
}
}
func coveringResolutionSnapshot(snapshots []resolutionSnapshot, path Path) *resolutionSnapshot {
var covering *resolutionSnapshot
for index := range snapshots {

10
metadata.go
View File

@@ -1,7 +1,7 @@
package configkit
// Metadata provides detached presence, origin, and history queries for a load.
// Its zero value contains no paths.
// Metadata provides detached effective presence/origin and chronological history
// queries for a load. Its zero value contains no paths.
type Metadata struct {
fields map[string]FieldInfo
history map[string][]Change
@@ -27,8 +27,10 @@ func (m Metadata) IsExplicit(path Path) bool {
return ok && info.Explicit
}
// History returns detached definition and resolution records for path. Unknown and
// invalid paths return nil.
// History returns detached definition and resolution records for path. A record's
// Path can be an ancestor when that container operation replaced or removed path.
// Removed historical paths can have history even though Lookup returns false.
// Unknown and invalid paths return nil.
func (m Metadata) History(path Path) []Change {
if !path.valid() {
return nil

218
provenance_test.go Normal file
View File

@@ -0,0 +1,218 @@
package configkit
import (
"errors"
"reflect"
"sync"
"testing"
)
type layeredItem struct {
Value string `yaml:"value"`
}
type layeredConfig struct {
Shared struct {
Equal string `yaml:"equal"`
Imported string `yaml:"imported"`
Profile string `yaml:"profile"`
} `yaml:"shared"`
Values []int `yaml:"values"`
Items map[string]layeredItem `yaml:"items"`
Value any `yaml:"value"`
Nullable *string `yaml:"nullable"`
Derived int `yaml:"derived"`
Empty map[string]string `yaml:"empty"`
}
func TestLayeredProvenanceAgreesWithResolvedSnapshot(t *testing.T) {
directory := t.TempDir()
mainPath := writeImportFile(t, directory, "main.yml", `_configkit:
imports: [{path: imported.yml}]
shared: {equal: same}
values: [1, 2]
items:
keep: {value: 'yes'}
remove: {value: 'no'}
value: {child: base}
nullable: base
empty: {}
`)
importPath := writeImportFile(t, directory, "imported.yml", "shared: {imported: base}\n")
profilePath := writeImportFile(t, directory, "profiles/testing.yml", `shared: {profile: selected}
values: [3, 4]
value: selected
nullable: null
empty: {}
`)
defaultNullable := "default"
definition := Definition[layeredConfig]{
Defaults: func() layeredConfig {
defaults := layeredConfig{
Values: []int{0, 0}, Value: map[string]any{"child": "default"},
Nullable: &defaultNullable, Empty: map[string]string{"retained": "default"},
}
defaults.Shared.Equal = "same"
return defaults
},
Resolve: func(configuration *layeredConfig, resolution *Resolution) error {
Normalize(resolution, Keys("values"), &configuration.Values, []int{5},
"bounded values", Keys("shared", "imported"), Keys("shared", "imported"))
Normalize(resolution, Keys("values"), &configuration.Values, []int{5},
"bounded values", Keys("shared", "imported"))
delete(configuration.Items, "remove")
resolution.Record(Keys("items"), ChangeNormalization, "removed disabled items")
Default(resolution, Keys("derived"), &configuration.Derived, 7,
"derived from equal", Keys("shared", "equal"), Keys("shared", "equal"))
resolution.Warning(Keys("values"), "values were bounded", Keys("shared", "imported"))
return nil
},
}
result, err := Load[layeredConfig](File(mainPath), Options[layeredConfig]{
Imports: true, Profile: "testing", Definition: definition,
})
if err != nil {
var loadError *LoadError
_ = errors.As(err, &loadError)
t.Fatalf("%v: %#v", err, loadError.Diagnostics())
}
wantSources := []string{canonicalTestPath(t, mainPath), canonicalTestPath(t, importPath), canonicalTestPath(t, profilePath)}
sources := result.Sources()
gotSources := make([]string, len(sources))
for index, source := range sources {
gotSources[index] = source.Path
}
if !reflect.DeepEqual(gotSources, wantSources) || sources[2].Layer != LayerProfile {
t.Fatalf("sources = %#v", sources)
}
if result.Config.Shared.Equal != "same" || result.Config.Shared.Imported != "base" || result.Config.Shared.Profile != "selected" ||
!reflect.DeepEqual(result.Config.Values, []int{5}) || result.Config.Nullable != nil || result.Config.Derived != 7 ||
result.Config.Empty["retained"] != "default" {
t.Fatalf("configuration = %#v", result.Config)
}
metadata := result.Metadata()
equal, _ := metadata.Lookup(Keys("shared", "equal"))
derived, _ := metadata.Lookup(Keys("derived"))
nullable, _ := metadata.Lookup(Keys("nullable"))
empty, _ := metadata.Lookup(Keys("empty"))
firstValue, _ := metadata.Lookup(Keys("values").Index(0))
if !equal.Explicit || equal.Origin.Layer != LayerBase || derived.Explicit || derived.Origin.Operation != ChangeDerivedDefault ||
!nullable.Explicit || !nullable.Null || nullable.Origin.Layer != LayerProfile ||
!empty.Explicit || empty.Origin.Layer != LayerProfile || len(empty.Origin.Locations) != 2 ||
!firstValue.Explicit || firstValue.Origin.Operation != ChangeNormalization {
t.Fatalf("metadata = equal %#v derived %#v nullable %#v empty %#v value %#v", equal, derived, nullable, empty, firstValue)
}
if _, ok := metadata.Lookup(Keys("value", "child")); ok {
t.Fatal("profile-removed child remains effective")
}
if _, ok := metadata.Lookup(Keys("values").Index(1)); ok {
t.Fatal("resolution-removed sequence index remains effective")
}
if _, ok := metadata.Lookup(Keys("items", "remove", "value")); ok {
t.Fatal("resolution-removed mapping child remains effective")
}
assertHistoryLayersAndPaths(t, metadata.History(Keys("values").Index(1)),
[]Layer{LayerDefault, LayerBase, LayerProfile, LayerResolution},
[]string{"/values/1", "/values/1", "/values", "/values"})
assertHistoryLayersAndPaths(t, metadata.History(Keys("value", "child")),
[]Layer{LayerDefault, LayerBase, LayerProfile},
[]string{"/value/child", "/value/child", "/value"})
removedHistory := metadata.History(Keys("items", "remove", "value"))
assertHistoryLayersAndPaths(t, removedHistory,
[]Layer{LayerBase, LayerResolution},
[]string{"/items/remove/value", "/items"})
if !reflect.DeepEqual(removedHistory[1].Reasons, []string{"removed disabled items"}) {
t.Fatalf("removed history = %#v", removedHistory)
}
valuesHistory := metadata.History(Keys("values").Index(0))
lastValue := valuesHistory[len(valuesHistory)-1]
if !reflect.DeepEqual(lastValue.Reasons, []string{"bounded values"}) || len(lastValue.Dependencies) != 1 || lastValue.Dependencies[0].String() != "/shared/imported" {
t.Fatalf("canonical resolution history = %#v", lastValue)
}
schema, issue := buildSchema(reflect.TypeFor[layeredConfig]())
if issue != nil {
t.Fatal(issue)
}
projected, projectionIssue := projectConfiguration(schema, reflect.ValueOf(result.Config), defaultProjectionLimits())
if projectionIssue != nil || !documentNodesEqual(projected, result.final) {
t.Fatalf("typed projection and private snapshot disagree: %#v", projectionIssue)
}
metadata.history[Keys("values").Index(0).String()][0].Origin.Layer = "mutated"
sources[1].ImportedFrom.SourceID = "mutated"
diagnostics := result.Diagnostics()
diagnostics[0].Related[0].SourceID = "mutated"
result.Config.Values[0] = 99
freshHistory := result.Metadata().History(Keys("values").Index(0))
if freshHistory[0].Origin.Layer == "mutated" || result.Sources()[1].ImportedFrom.SourceID == "mutated" ||
result.Diagnostics()[0].Related[0].SourceID == "mutated" {
t.Fatal("returned inspection records alias private state")
}
snapshotValue, ok := documentAtPath(result.final, Keys("values").Index(0))
if !ok || snapshotValue.Text != "5" {
t.Fatal("caller configuration mutation changed the private final snapshot")
}
}
func TestSelectedProfileLoadsRemainIndependentConcurrently(t *testing.T) {
directory := t.TempDir()
mainPath := writeImportFile(t, directory, "main.yml", "values: [1, 2]\n")
writeImportFile(t, directory, "profiles/testing.yml", "values: [3, 4]\n")
options := Options[layeredConfig]{Profile: "testing"}
const loads = 16
results := make(chan *Result[layeredConfig], loads)
errors := make(chan error, loads)
var group sync.WaitGroup
for index := 0; index < loads; index++ {
group.Add(1)
go func() {
defer group.Done()
result, err := Load[layeredConfig](File(mainPath), options)
if err != nil {
errors <- err
return
}
results <- result
}()
}
group.Wait()
close(results)
close(errors)
for err := range errors {
t.Fatal(err)
}
var collected []*Result[layeredConfig]
for result := range results {
collected = append(collected, result)
}
if len(collected) != loads {
t.Fatalf("completed loads = %d", len(collected))
}
collected[0].Config.Values[0] = 99
collected[0].metadata.fields[Keys("values").String()] = FieldInfo{}
for _, result := range collected[1:] {
if !reflect.DeepEqual(result.Config.Values, []int{3, 4}) {
t.Fatalf("concurrent result contaminated = %#v", result.Config.Values)
}
if info, ok := result.Metadata().Lookup(Keys("values")); !ok || info.Origin.Layer != LayerProfile {
t.Fatalf("concurrent metadata contaminated = %#v, %v", info, ok)
}
}
}
func assertHistoryLayersAndPaths(t *testing.T, history []Change, layers []Layer, paths []string) {
t.Helper()
if len(history) != len(layers) || len(history) != len(paths) {
t.Fatalf("history = %#v", history)
}
for index, change := range history {
if change.Origin.Layer != layers[index] || change.Path.String() != paths[index] {
t.Fatalf("history[%d] = %#v", index, change)
}
}
}

8
records.go
View File

@@ -150,7 +150,8 @@ type Diagnostic struct {
}
// Origin describes where an effective definition came from. Multiple locations
// represent a composite origin.
// represent a composite origin; resolution origins are synthetic and identify their
// operation instead of a file location.
type Origin struct {
Layer Layer
Locations []Location
@@ -164,8 +165,9 @@ type FieldInfo struct {
Origin Origin
}
// Change describes a definition or application-resolution operation. Historical
// values are intentionally not exposed by this record.
// Change describes a definition or application-resolution operation. Path can name
// an ancestor operation affecting the queried descendant. Historical values are
// intentionally not exposed by this record.
type Change struct {
Path Path
Origin Origin