diff --git a/docs/config.md b/docs/config.md index b0e060e..39ebd52 100644 --- a/docs/config.md +++ b/docs/config.md @@ -79,7 +79,7 @@ error without reporting one, ConfigKit emits a generic `version.hook_failed` diagnostic and suppresses the returned error text. Warnings alone are retained on a successful result. Callback panics remain programming failures. -### Layered input/default metadata +### Layered metadata `Result.Metadata` describes every final canonical path. Exact surviving input paths are explicit and have a base or profile origin; omitted paths have a default origin. @@ -91,6 +91,23 @@ metadata while retaining their base definition and the parent replacement operat in history. All metadata queries and returned nested records are detached. +History is chronological: a path can have a synthetic default record, its exact base +definition, a profile replacement, and a recorded resolution operation. A descendant +query uses the ancestor's path for a whole-container or whole-sequence replacement; +the descendant can therefore retain history after `Lookup` stops reporting it as +effective. Equal-valued input is still an explicit base definition, while a derived +value for absent input remains non-explicit. + +A mapping assembled from the selected file and imports has one composite base origin +whose locations retain source traversal order. A mapping also visited by a profile +has a profile-layer composite origin with the lower contributing locations followed +by the profile location; an empty mapping is still an explicit container even though +it contributes no children. Resolution origins are synthetic: they have layer +`resolution`, no fabricated file location, and a `derived_default` or `normalization` +operation. Reasons and dependency paths retain first occurrence order with duplicates +removed. Source records remain ordered as selected source, depth-first imports, then +the selected profile. + ### Recorded application resolution An optional `Definition.Resolve` callback runs after successful typed decoding. It diff --git a/docs/internal/extraction.md b/docs/internal/extraction.md index d9395ef..d3952dd 100644 --- a/docs/internal/extraction.md +++ b/docs/internal/extraction.md @@ -188,6 +188,18 @@ Assigned to layered metadata, redaction, rendering, and explanation: records, and redaction. Narratio commands, diff/digest APIs, generated artifact attribution, campaign logic, and party expansion remain application-owned. +Layered-provenance implementation record (2026-09-06): ConfigKit integrated its +previously extracted source claims, overlay displacement records, typed baseline, +and audited resolution snapshots in `load_metadata.go`. The representative +`TestLayeredProvenanceAgreesWithResolvedSnapshot` adapts Narratio's stable ownership, +profile ownership, deterministic effective projection, and side-effect-free +inspection cases across one default/import/profile/resolution workflow. ConfigKit +adds chronological descendant history for parent and whole-sequence operations, +composite empty-mapping origins, canonical resolution reasons/dependencies, private +snapshot agreement, defensive nested copies, and concurrent load isolation. Value +rendering, redaction, explanation, commands, generated attribution, diffs, and +digests remain assigned to their later owning work. + ### Defaults, presence, and domain resolution Assigned across schema, decoding, resolution, validation, and consumer examples: diff --git a/docs/internal/overview.md b/docs/internal/overview.md index 3764b5f..f7aa4c9 100644 --- a/docs/internal/overview.md +++ b/docs/internal/overview.md @@ -24,8 +24,8 @@ module and package documentation boundary. Its implemented public primitives are baseline/final trees, and initial default/input origins; and - public lifecycle coordination in `load.go`, bounded canonical import DFS in `imports.go`, pattern-directed glob expansion in `glob.go`, version-report contexts - in `validation.go`, and detached - base/default metadata assembly in `load_metadata.go`; and + in `validation.go`, and detached layered metadata/history assembly in + `load_metadata.go`; and - recorded application defaulting and normalization in `resolution.go`, including explicitness queries, helper verification, structural mutation audit, coalesced intent, private before/after snapshots, and resolution metadata integration; and @@ -33,8 +33,8 @@ module and package documentation boundary. Its implemented public primitives are `validation.go`, completing the public callback lifecycle after composed input. The [configuration contract](../config.md) owns implemented input, schema, decoding, -loading lifecycle, path imports, composition, profile overlay, version checking, recorded resolution, -semantic validation, metadata, and ownership behavior. +loading lifecycle, path imports, composition, profile overlay, version checking, +recorded resolution, semantic validation, metadata, and ownership behavior. The source-extraction inventory records the application foundations assigned to future implementation work: diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index 64a9d03..9e6a1af 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -2,7 +2,7 @@ ## Status and ownership -Status: implementation in progress; Stages 1–13 are complete and later stages are +Status: implementation in progress; Stages 1–14 are complete and later stages are pending. Audience: a `gpt-5.6-terra` coding agent implementing one stage per prompt, in @@ -1038,7 +1038,31 @@ Acceptance: ## Stage 14 — Verify layered provenance and final snapshot agreement -Status: pending. +Status: complete. + +Completion evidence (2026-09-06): + +- Unified default, composed base, profile displacement, and resolution records in + the metadata history index. Container resolution now appends its operation to each + changed, added, or removed descendant while preserving the ancestor operation path + and path-specific previous origin; effective metadata still contains only the final + projected paths. +- Added a representative public `Load` workflow spanning a fresh default, selected + source, depth-first import, profile, derived default, mapping deletion, and + whole-sequence normalization. It verifies source order, equal-to-default + explicitness, null state, composite empty-mapping origins, chronological parent + replacement history, deduplicated reasons/dependencies, typed/final snapshot + agreement, and unchanged siblings. +- Verified nested metadata, source import locations, diagnostics, caller-owned + configuration, and private snapshots are detached. Repeated concurrent selected- + profile loads retain independent typed values, metadata, sources, and histories; + the race baseline passes. +- Promoted layered metadata semantics and concrete composite/synthetic origin cases + to the [current metadata contract](../config.md#layered-metadata), updated Go + reference comments, and recorded the adapted Narratio ownership/inspection cases. + Rendering, redaction, value snapshots, and explanation remain within later stages. + Verified `go test ./...`, `go test -race ./...`, `go vet ./...`, `go build ./...`, + `go mod tidy -diff`, and `git diff --check` with Go 1.26.7 on linux/amd64. Read: feature [presence](initial.md#paths-and-presence), [history](initial.md#provenance-diagnostics-and-redaction), metadata API. diff --git a/docs/roadmap/initial.md b/docs/roadmap/initial.md index d85dbb6..1f76a6b 100644 --- a/docs/roadmap/initial.md +++ b/docs/roadmap/initial.md @@ -430,23 +430,12 @@ conversion through the current loading lifecycle. ## Paths and presence -Semantic path construction, JSON Pointer parsing, and base/default metadata -population are implemented. Their current contracts are owned by the Go +Semantic path construction, JSON Pointer parsing, and layered metadata population +are implemented. Their current contracts are owned by the Go documentation in [`path.go`](../../path.go), [`metadata.go`](../../metadata.go), -and the [current configuration contract](../config.md#base-default-metadata). -Layered replacement and resolution behavior below remains target lifecycle work. - -Distinguish absent, explicitly null, explicitly zero/false/empty, and other -explicit values. An explicit value equal to its default remains explicit. -Presence queries describe the surviving input contribution after overlays; -separate history queries describe earlier appearances. A parent mapping's -presence does not make all of its children explicit. Replacing a subtree removes -its descendants from effective presence and origin queries while retaining -historical replacement records. - -Application normalization preserves the fact that a surviving input was -explicit. A derived value for an absent input remains non-explicit. Its final -origin is a resolution operation, with input origins retained separately. +and the [current metadata contract](../config.md#layered-metadata), including +presence, composite origins, parent replacements, removed-path history, and +synthetic resolution operations. ## Composition and overlay @@ -481,8 +470,7 @@ The [current application schema contract](../config.md#application-schema-bindin owns root/type support, field and tag binding, inlining, codec pairing, and schema-aware path rules. The [current defaults and typed-decoding contract](../config.md#defaults-and-typed-decoding) owns strict conversions, container application, presence, safe failures, and initial -origins. Public lifecycle integration, layered provenance, secret handling, and -rendering remain target work below. +origins. Secret handling and rendering remain target work below. ## Recorded resolution and validation @@ -490,7 +478,8 @@ Application-owned recorded resolution is implemented. The [current resolution contract](../config.md#recorded-application-resolution) owns explicitness queries, helpers and manual records, structural mutation auditing, coalescing, diagnostics, origins, history, and snapshot ownership. Layered input -history, redacted explanation, and secret dependency propagation remain target work. +history is owned by the [current metadata contract](../config.md#layered-metadata); +redacted explanation and secret dependency propagation remain target work. Canonical typed projection, detached cloning, structural comparison, and publishing independently captured resolved results are implemented. The @@ -507,16 +496,10 @@ callback lifetime, and application/runtime responsibility boundaries. Safe diagnostic/provenance value records, canonical diagnostic rendering, and `LoadError` access are implemented. Their current contracts are owned by [`records.go`](../../records.go) and [`diagnostic.go`](../../diagnostic.go). -Base/default and resolution records are attached to the current loading lifecycle. -Additional layers, safe historical values, redaction, and rendering remain target -work below. - -Retain input sources, layer, line/column where available, import sites, effective -origins, overridden definition records, and resolution operations. Container -origins may be composite; never invent one file location for a mapping assembled -from several sources. Defaults and resolution have synthetic origins rather -than fabricated file coordinates. Preserve definition history without exposing -mutable parser nodes or unrestricted raw value references through the API. +Default, base, profile, and resolution records are attached to the loading lifecycle; +their source ordering, composite/synthetic origins, replacement history, and detached +queries are owned by the [current metadata contract](../config.md#layered-metadata). +Safe historical values, redaction, and rendering remain target work below. Diagnostics have stable codes, severity, semantic path when available, a safe message, primary location, and related locations. Sort deterministically by @@ -697,11 +680,11 @@ func ReadWorkerDefaults() (*configkit.Result[Config], error) { ### Paths and metadata -The path API, detached metadata records, empty/unknown query behavior, and -base/default population are now implemented; [`path.go`](../../path.go), +The path API, detached metadata records, empty/unknown query behavior, and layered +population are implemented; [`path.go`](../../path.go), [`records.go`](../../records.go), [`metadata.go`](../../metadata.go), and the -[metadata configuration contract](../config.md#base-default-metadata) own current -behavior. Layered callback history and safe historical values remain target behavior. +[metadata configuration contract](../config.md#layered-metadata) own current +behavior. Safe historical values remain target behavior for explanation. ### Resolution and validation callbacks diff --git a/load_metadata.go b/load_metadata.go index bf55580..4d67966 100644 --- a/load_metadata.go +++ b/load_metadata.go @@ -1,6 +1,10 @@ package configkit -import "gitea.maximumdirect.net/eric/ConfigKit/internal/document" +import ( + "slices" + + "gitea.maximumdirect.net/eric/ConfigKit/internal/document" +) func metadataFromDecoded(decoded decodedConfiguration) Metadata { metadata := Metadata{ @@ -140,6 +144,7 @@ func applyResolutionMetadata(metadata *Metadata, resolved resolutionResult) { } metadata.history[pointer] = append(metadata.history[pointer], change) } + appendResolutionDescendantHistory(metadata, resolved) for _, delta := range resolved.deltas { record := coveringResolutionSnapshot(resolved.snapshots, delta.path) if record == nil { @@ -157,6 +162,42 @@ func applyResolutionMetadata(metadata *Metadata, resolved resolutionResult) { refreshResolutionFields(metadata.fields, resolved.final, resolved.snapshots) } +func appendResolutionDescendantHistory(metadata *Metadata, resolved resolutionResult) { + finalPaths := make(map[string]*document.Node) + indexDocument(resolved.final, Path{}, finalPaths) + for _, delta := range resolved.deltas { + record := coveringResolutionSnapshot(resolved.snapshots, delta.path) + if record == nil { + continue + } + affected := make(map[string]bool) + for pointer := range metadata.history { + if path, err := ParsePath(pointer); err == nil && pathContains(delta.path, path) { + affected[pointer] = true + } + } + for pointer := range finalPaths { + if path, err := ParsePath(pointer); err == nil && pathContains(delta.path, path) { + affected[pointer] = true + } + } + pointers := make([]string, 0, len(affected)) + for pointer := range affected { + if pointer != record.change.Path.String() { + pointers = append(pointers, pointer) + } + } + slices.Sort(pointers) + for _, pointer := range pointers { + change := cloneChange(record.change) + if info, ok := metadata.fields[pointer]; ok { + change.Previous = []Origin{cloneOrigin(info.Origin)} + } + metadata.history[pointer] = append(metadata.history[pointer], change) + } + } +} + func coveringResolutionSnapshot(snapshots []resolutionSnapshot, path Path) *resolutionSnapshot { var covering *resolutionSnapshot for index := range snapshots { diff --git a/metadata.go b/metadata.go index a1de205..22a4734 100644 --- a/metadata.go +++ b/metadata.go @@ -1,7 +1,7 @@ package configkit -// Metadata provides detached presence, origin, and history queries for a load. -// Its zero value contains no paths. +// Metadata provides detached effective presence/origin and chronological history +// queries for a load. Its zero value contains no paths. type Metadata struct { fields map[string]FieldInfo history map[string][]Change @@ -27,8 +27,10 @@ func (m Metadata) IsExplicit(path Path) bool { return ok && info.Explicit } -// History returns detached definition and resolution records for path. Unknown and -// invalid paths return nil. +// History returns detached definition and resolution records for path. A record's +// Path can be an ancestor when that container operation replaced or removed path. +// Removed historical paths can have history even though Lookup returns false. +// Unknown and invalid paths return nil. func (m Metadata) History(path Path) []Change { if !path.valid() { return nil diff --git a/provenance_test.go b/provenance_test.go new file mode 100644 index 0000000..f6e25ea --- /dev/null +++ b/provenance_test.go @@ -0,0 +1,218 @@ +package configkit + +import ( + "errors" + "reflect" + "sync" + "testing" +) + +type layeredItem struct { + Value string `yaml:"value"` +} + +type layeredConfig struct { + Shared struct { + Equal string `yaml:"equal"` + Imported string `yaml:"imported"` + Profile string `yaml:"profile"` + } `yaml:"shared"` + Values []int `yaml:"values"` + Items map[string]layeredItem `yaml:"items"` + Value any `yaml:"value"` + Nullable *string `yaml:"nullable"` + Derived int `yaml:"derived"` + Empty map[string]string `yaml:"empty"` +} + +func TestLayeredProvenanceAgreesWithResolvedSnapshot(t *testing.T) { + directory := t.TempDir() + mainPath := writeImportFile(t, directory, "main.yml", `_configkit: + imports: [{path: imported.yml}] +shared: {equal: same} +values: [1, 2] +items: + keep: {value: 'yes'} + remove: {value: 'no'} +value: {child: base} +nullable: base +empty: {} +`) + importPath := writeImportFile(t, directory, "imported.yml", "shared: {imported: base}\n") + profilePath := writeImportFile(t, directory, "profiles/testing.yml", `shared: {profile: selected} +values: [3, 4] +value: selected +nullable: null +empty: {} +`) + defaultNullable := "default" + definition := Definition[layeredConfig]{ + Defaults: func() layeredConfig { + defaults := layeredConfig{ + Values: []int{0, 0}, Value: map[string]any{"child": "default"}, + Nullable: &defaultNullable, Empty: map[string]string{"retained": "default"}, + } + defaults.Shared.Equal = "same" + return defaults + }, + Resolve: func(configuration *layeredConfig, resolution *Resolution) error { + Normalize(resolution, Keys("values"), &configuration.Values, []int{5}, + "bounded values", Keys("shared", "imported"), Keys("shared", "imported")) + Normalize(resolution, Keys("values"), &configuration.Values, []int{5}, + "bounded values", Keys("shared", "imported")) + delete(configuration.Items, "remove") + resolution.Record(Keys("items"), ChangeNormalization, "removed disabled items") + Default(resolution, Keys("derived"), &configuration.Derived, 7, + "derived from equal", Keys("shared", "equal"), Keys("shared", "equal")) + resolution.Warning(Keys("values"), "values were bounded", Keys("shared", "imported")) + return nil + }, + } + + result, err := Load[layeredConfig](File(mainPath), Options[layeredConfig]{ + Imports: true, Profile: "testing", Definition: definition, + }) + if err != nil { + var loadError *LoadError + _ = errors.As(err, &loadError) + t.Fatalf("%v: %#v", err, loadError.Diagnostics()) + } + wantSources := []string{canonicalTestPath(t, mainPath), canonicalTestPath(t, importPath), canonicalTestPath(t, profilePath)} + sources := result.Sources() + gotSources := make([]string, len(sources)) + for index, source := range sources { + gotSources[index] = source.Path + } + if !reflect.DeepEqual(gotSources, wantSources) || sources[2].Layer != LayerProfile { + t.Fatalf("sources = %#v", sources) + } + if result.Config.Shared.Equal != "same" || result.Config.Shared.Imported != "base" || result.Config.Shared.Profile != "selected" || + !reflect.DeepEqual(result.Config.Values, []int{5}) || result.Config.Nullable != nil || result.Config.Derived != 7 || + result.Config.Empty["retained"] != "default" { + t.Fatalf("configuration = %#v", result.Config) + } + + metadata := result.Metadata() + equal, _ := metadata.Lookup(Keys("shared", "equal")) + derived, _ := metadata.Lookup(Keys("derived")) + nullable, _ := metadata.Lookup(Keys("nullable")) + empty, _ := metadata.Lookup(Keys("empty")) + firstValue, _ := metadata.Lookup(Keys("values").Index(0)) + if !equal.Explicit || equal.Origin.Layer != LayerBase || derived.Explicit || derived.Origin.Operation != ChangeDerivedDefault || + !nullable.Explicit || !nullable.Null || nullable.Origin.Layer != LayerProfile || + !empty.Explicit || empty.Origin.Layer != LayerProfile || len(empty.Origin.Locations) != 2 || + !firstValue.Explicit || firstValue.Origin.Operation != ChangeNormalization { + t.Fatalf("metadata = equal %#v derived %#v nullable %#v empty %#v value %#v", equal, derived, nullable, empty, firstValue) + } + if _, ok := metadata.Lookup(Keys("value", "child")); ok { + t.Fatal("profile-removed child remains effective") + } + if _, ok := metadata.Lookup(Keys("values").Index(1)); ok { + t.Fatal("resolution-removed sequence index remains effective") + } + if _, ok := metadata.Lookup(Keys("items", "remove", "value")); ok { + t.Fatal("resolution-removed mapping child remains effective") + } + + assertHistoryLayersAndPaths(t, metadata.History(Keys("values").Index(1)), + []Layer{LayerDefault, LayerBase, LayerProfile, LayerResolution}, + []string{"/values/1", "/values/1", "/values", "/values"}) + assertHistoryLayersAndPaths(t, metadata.History(Keys("value", "child")), + []Layer{LayerDefault, LayerBase, LayerProfile}, + []string{"/value/child", "/value/child", "/value"}) + removedHistory := metadata.History(Keys("items", "remove", "value")) + assertHistoryLayersAndPaths(t, removedHistory, + []Layer{LayerBase, LayerResolution}, + []string{"/items/remove/value", "/items"}) + if !reflect.DeepEqual(removedHistory[1].Reasons, []string{"removed disabled items"}) { + t.Fatalf("removed history = %#v", removedHistory) + } + valuesHistory := metadata.History(Keys("values").Index(0)) + lastValue := valuesHistory[len(valuesHistory)-1] + if !reflect.DeepEqual(lastValue.Reasons, []string{"bounded values"}) || len(lastValue.Dependencies) != 1 || lastValue.Dependencies[0].String() != "/shared/imported" { + t.Fatalf("canonical resolution history = %#v", lastValue) + } + + schema, issue := buildSchema(reflect.TypeFor[layeredConfig]()) + if issue != nil { + t.Fatal(issue) + } + projected, projectionIssue := projectConfiguration(schema, reflect.ValueOf(result.Config), defaultProjectionLimits()) + if projectionIssue != nil || !documentNodesEqual(projected, result.final) { + t.Fatalf("typed projection and private snapshot disagree: %#v", projectionIssue) + } + + metadata.history[Keys("values").Index(0).String()][0].Origin.Layer = "mutated" + sources[1].ImportedFrom.SourceID = "mutated" + diagnostics := result.Diagnostics() + diagnostics[0].Related[0].SourceID = "mutated" + result.Config.Values[0] = 99 + freshHistory := result.Metadata().History(Keys("values").Index(0)) + if freshHistory[0].Origin.Layer == "mutated" || result.Sources()[1].ImportedFrom.SourceID == "mutated" || + result.Diagnostics()[0].Related[0].SourceID == "mutated" { + t.Fatal("returned inspection records alias private state") + } + snapshotValue, ok := documentAtPath(result.final, Keys("values").Index(0)) + if !ok || snapshotValue.Text != "5" { + t.Fatal("caller configuration mutation changed the private final snapshot") + } +} + +func TestSelectedProfileLoadsRemainIndependentConcurrently(t *testing.T) { + directory := t.TempDir() + mainPath := writeImportFile(t, directory, "main.yml", "values: [1, 2]\n") + writeImportFile(t, directory, "profiles/testing.yml", "values: [3, 4]\n") + options := Options[layeredConfig]{Profile: "testing"} + + const loads = 16 + results := make(chan *Result[layeredConfig], loads) + errors := make(chan error, loads) + var group sync.WaitGroup + for index := 0; index < loads; index++ { + group.Add(1) + go func() { + defer group.Done() + result, err := Load[layeredConfig](File(mainPath), options) + if err != nil { + errors <- err + return + } + results <- result + }() + } + group.Wait() + close(results) + close(errors) + for err := range errors { + t.Fatal(err) + } + var collected []*Result[layeredConfig] + for result := range results { + collected = append(collected, result) + } + if len(collected) != loads { + t.Fatalf("completed loads = %d", len(collected)) + } + collected[0].Config.Values[0] = 99 + collected[0].metadata.fields[Keys("values").String()] = FieldInfo{} + for _, result := range collected[1:] { + if !reflect.DeepEqual(result.Config.Values, []int{3, 4}) { + t.Fatalf("concurrent result contaminated = %#v", result.Config.Values) + } + if info, ok := result.Metadata().Lookup(Keys("values")); !ok || info.Origin.Layer != LayerProfile { + t.Fatalf("concurrent metadata contaminated = %#v, %v", info, ok) + } + } +} + +func assertHistoryLayersAndPaths(t *testing.T, history []Change, layers []Layer, paths []string) { + t.Helper() + if len(history) != len(layers) || len(history) != len(paths) { + t.Fatalf("history = %#v", history) + } + for index, change := range history { + if change.Origin.Layer != layers[index] || change.Path.String() != paths[index] { + t.Fatalf("history[%d] = %#v", index, change) + } + } +} diff --git a/records.go b/records.go index 7f33072..5b04d6b 100644 --- a/records.go +++ b/records.go @@ -150,7 +150,8 @@ type Diagnostic struct { } // Origin describes where an effective definition came from. Multiple locations -// represent a composite origin. +// represent a composite origin; resolution origins are synthetic and identify their +// operation instead of a file location. type Origin struct { Layer Layer Locations []Location @@ -164,8 +165,9 @@ type FieldInfo struct { Origin Origin } -// Change describes a definition or application-resolution operation. Historical -// values are intentionally not exposed by this record. +// Change describes a definition or application-resolution operation. Path can name +// an ancestor operation affecting the queried descendant. Historical values are +// intentionally not exposed by this record. type Change struct { Path Path Origin Origin