From 381430a24bdf01909842dccc4653de2b2737ab3d Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Sat, 12 Sep 2026 12:52:32 +0000 Subject: [PATCH] Plan source origins during resolution --- docs/development.md | 3 + docs/roadmap/implementation.md | 267 ++++++++++++++++++ .../source-origins-during-resolution.md | 229 +++++++++++++++ 3 files changed, 499 insertions(+) create mode 100644 docs/roadmap/implementation.md create mode 100644 docs/roadmap/source-origins-during-resolution.md diff --git a/docs/development.md b/docs/development.md index 7ff61ba..b33c7d4 100644 --- a/docs/development.md +++ b/docs/development.md @@ -46,6 +46,9 @@ The completed string-and-integer schema-version implementation and remediation records are preserved in Git history at `244104f`. The [configuration contract](config.md#application-schema-versions) owns current behavior. +Planned feature: [source origins during resolution](roadmap/source-origins-during-resolution.md). +This roadmap defines future behavior; its staged implementation plan is not yet written. + ## Baseline validation Install Go 1.25.0 or a supported patched/newer Go toolchain. Acquire declared diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..4ffb1de --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,267 @@ +# Source origins during resolution: implementation plan + +Status: not started. Implement stages in order, starting with Stage 1. +The [feature roadmap](source-origins-during-resolution.md) owns the target contract +and consumer intent. This document owns implementation steps, validation, and +handoffs. It is written for one gpt-5.6-terra coding agent per stage; no stage +requires parallel agents or changes to a downstream repository. + +## Required reading and working rules + +Before each stage, inspect the worktree and read the preceding stage's completion +record. Follow [development](../development.md) and all policies: +[architecture](../policy/architecture.md), [documentation](../policy/documentation.md), +[testing](../policy/testing.md), and [release](../policy/release.md). +Read the roadmap, the configuration contract's resolution and metadata sections, +and the relevant code before editing. Use the code graph for discovery and verify +current definitions rather than assuming line numbers or private signatures. + +Preserve unrelated work. Keep each stage independently buildable with passing +tests; do not introduce stub public methods or defer known correctness failures. +Run the focused commands listed for the stage and `./scripts/check.sh` from the +repository root. Acquire required toolchains/dependencies explicitly before offline +checks. Document unsupported filesystem fixtures rather than silently omitting them. + +Use real collaborators and temporary files for provenance. Keep default tests +offline, deterministic, and credential-free. Prefer codes, paths, layers, locations, +and observable snapshots over private call counts or incidental error messages. +Only use a narrow I/O or credential seam where it proves the no-extra-access rule. + +At each handoff, update that stage's status here with changed files, tests/results, +actual toolchains, deviations, and remaining risks. Keep public reference and current +contract documentation synchronized with behavior as it lands. Commit coherent +stage work when authorized, but do not push, tag, select a version, or prepare a +speculative release note as part of this implementation sprint. + +## Implementation decisions + +The roadmap's two method signatures are fixed for this sprint: + +```go +func (r *Resolution) InputInfo(path Path) (FieldInfo, bool) +func (r *Resolution) Source(id string) (SourceInfo, bool) +``` + +Use a private, immutable entry-state view, not live final metadata. Store only +effective field records, effective path shapes, and a source-ID index; do not build +or expose an additional historical view. Reuse `Metadata.Lookup` semantics by +representing the field/path part as a private `Metadata` value with only those +indexes populated. Never return that `Metadata` value publicly. + +Build the view after `metadataFromDecoded` and `applyOverlayMetadata` in +`resolveLoadInput`, immediately before `runResolution`, only when a resolver exists. +Its path shapes come from `decoded.final`, not the raw document or mutable Go +configuration. Clone field records and their nested origins into a separate map; +do not shallow-copy the mutable final metadata map. Build a map keyed by ID from +`assembledInput.sources`, cloning `SourceInfo` records with existing helpers. +Missing or duplicate internally generated IDs are implementation-invariant failures +to fix, not new consumer-facing diagnostic categories. + +Pass the private view as an explicit argument to `runResolution` and retain it in +`resolutionState`. Update all private callers/tests deliberately: construct their +entry view from their decoded fixtures and supplied sources where they test these +queries. Do not add optional/variadic arguments that silently omit provenance in +production. Private names are discretionary, but ownership and timing are not. + +Queries first check callback activity through the existing context guard. Only +then may invalid paths or unknown IDs return misses. `InputInfo` delegates to the +entry metadata lookup; `Source` clones the matched source on return. Preserve +existing `IsExplicit`, diagnostics, resolution recording, and secret timing. + +Do not clone input trees, re-project configuration, read files, resolve symlinks, +or access credentials during a query. Effective indexing is O(number of entry +paths), source indexing is O(number of sources), and each lookup must avoid a full +tree/source scan. Returned-copy cost is proportional to the returned record. + +## Stage 1 — Factor reusable effective-path indexing + +Status: not started. Dependencies: none. + +Scope: `metadata.go`, `load_metadata.go`, and their focused test owners. + +1. Extract the effective-path indexing portion of `finalizeMetadataPaths` into one + private helper using existing document traversal and detached path construction. + Keep historical indexing and result publication behavior unchanged. +2. Make result finalization use the extracted helper. Add the private entry-view + constructor described above, accepting the pre-resolution metadata, decoded + final tree, and acquired source records. Do not expose public methods yet. +3. Ensure the constructor copies field records and source records without copying + history or configuration values. Reuse `cloneFieldInfo`, `cloneSourceInfo`, and + existing shape matching instead of writing another path parser or numeric matcher. +4. Add focused coverage proving the private view recognizes defaulted fields, + missing descendants, root, escaped keys, explicit key/index mismatches, and + contextual canonical sequence indices. Prove later mutation of the original + metadata and source records cannot alter the view. Reuse existing shape tests + for exhaustive syntax cases rather than duplicating them. + +Acceptance: all existing final metadata/history behavior is unchanged; a complete +entry view can be constructed before final publication without a historical index. +No public API or configuration contract changes are claimed by this stage. + +Validation: `go test . -run 'Metadata|Path|Provenance|Historical'`, then the everyday +checker. Handoff should identify the helper and constructor signatures for Stage 2. + +## Stage 2 — Wire the callback API and enforce lifetime + +Status: not started. Dependencies: Stage 1. + +Scope: `load_result.go`, `resolution.go`, necessary private callers, reference +comments, and resolution/callback ownership tests. + +1. Build and pass the entry view at the existing pre-resolution boundary. Use + `input.sources`, which includes sources acquired by import, profile, and CLI + processing; do not reconstruct source identities from labels or field locations. +2. Implement both public methods with the existing activity guard and detached + returns. Keep lookup state frozen when recording changes or mutating configuration. +3. Add public-load tests for a file-authored scalar, a defaulted field, explicit + null, a missing path, empty/unknown source IDs, and `NoSource`. Assert exact + `InputInfo.Explicit` agreement with `IsExplicit` for found paths. +4. Exercise nil/zero contexts and retained contexts after successful and failed + callbacks. Both methods must use the existing fixed closed-context panic behavior. + Saved return records remain usable after callback exit. Do not broaden this work + into a new concurrency contract or a general callback framework refactor. +5. Add Go method documentation and update the configuration contract's resolution + section to describe the implemented entry-state semantics, misses, lifetime, + and source distinction. Fix the development guide's stale statement that this + implementation plan has not yet been written. + +Acceptance: both methods work during an ordinary public `Load`; calls cannot access +live mutable metadata or bypass callback scope. No-rule/no-resolver behavior and +existing mutation auditing remain unchanged. + +Validation: `go test . -run 'Resolution|Callback|Metadata|Load'`, then the everyday +checker. Do not mark complete with public methods that work only in private tests. + +## Stage 3 — Verify layered provenance and snapshot independence + +Status: not started. Dependencies: Stage 2. + +Scope: public lifecycle/provenance, import/profile/override, and ownership tests; +production fixes only for gaps exposed at those boundaries. + +1. Build a real temporary import graph with path-bearing scalars and a sequence + element. Include a composite mapping with contributions from distinct files. + During resolution, assert scalar ownership, all container contributors, and + successful source lookup for every origin location returned by the view. +2. Apply a profile replacing one scalar and an equal-text value; verify replacing + origins and untouched siblings. Exercise equal-text CLI replacement separately. + Keep source IDs derived from returned records, not inferred from value equality. +3. Exercise byte and reader roots with filesystem context and file imports: the + memory root has no file path while imported files do. Also cover non-file input + without imports, default-only fields, and unknown source IDs. Retain default + descendants reintroduced by normal decoding instead of treating them as raw + input declarations. +4. Extend shape coverage through inlined structs/maps, typed arrays, and dynamic + structures. Assert misses for invalid indices, signed index tokens in sequences, + and explicit key/index mismatches; signed mapping keys remain literal keys. +5. Normalize, delete, add, and move fields within a callback using valid records. + Queries for original paths must stay fixed; absent destinations stay absent, + and pre-existing destinations retain their entry information. Final metadata + must still reflect resolution rather than that fixed entry view. +6. Mutate returned `Origin.Locations` and `SourceInfo.ImportedFrom`; verify repeated + queries, saved records, final results, and another load are independent. Exercise + independent loads under the race detector without sharing an active context. + +Acceptance: roadmap criteria 2–8 and 11 are covered at their owning boundaries. +No duplicate provenance model, nearest-ancestor fallback, or source-order heuristic +is introduced. Any discovered defect is fixed before the stage is marked complete. + +Validation: `go test . -run 'Resolution|Provenance|Import|Profile|Override|Metadata|Path'`, +`go test -race ./...`, then the everyday checker. + +## Stage 4 — Demonstrate source-aware normalization and safe inspection + +Status: not started. Dependencies: Stage 3. + +Scope: maintained Notarius-shaped example code/fixtures, executable examples, +explanation/secret integration tests, and consumer guidance. + +1. Add a small application-owned path-bearing configuration example in + `examples/consumers`. Keep existing catalog/version examples working. Supply the + invocation directory explicitly to the example loader rather than calling + `os.Chdir` or adding process-global state to tests. +2. Implement the example's policy using the two public queries: absolute paths + unchanged; CLI-relative paths based on the supplied invocation directory; + default-relative paths based on the canonical main source; file-authored paths + based on the scalar's declaring source. Use the existing primary-source identity + convention for main-source lookup, not a guess about contributor ordering. +3. For this file-rooted example, require one unambiguous canonical declaring file + for file-relative values. Report a safe application resolution error if that + prerequisite is absent or ambiguous. Do not silently use a diagnostic label or + working directory. Non-file-root fallback remains an explicitly separate + application policy, not a new library option. +4. Record every changed path through `Normalize` or `Record`, including valid + dependencies where derived values depend on other configuration paths. Prove + runtime values, effective YAML, and explanation agree; retain the original + relative declaration and normalization reason in history. No existence or + confinement checks are part of the example. +5. Add a symlink fixture proving canonical target-directory semantics and an + invocation-directory mismatch proving resolution does not depend on process CWD. + Make unsupported symlink capability explicit under the testing policy. +6. Use synthetic sensitive fields to verify original/effective redaction remains + intact. With existing reader/secret lookup seams, prove queries do not reread + source data or resolve credentials during the resolver; normal later lookup + remains unchanged. Do not create credentials or environment-dependent tests. +7. Document the public workflow and canonical symlink choice in the consumer guide; + point to executable artifacts instead of duplicating large code examples. + +Acceptance: roadmap criteria 1, 9, 10, and 12 are demonstrated in one-load public +workflows. Filesystem policy stays in the example application, not the root library. + +Validation: `go test ./examples/consumers`, +`go test . -run 'Resolution|Explain|Secret|Example'`, then the everyday checker. + +## Stage 5 — Audit the complete contract and hand off release readiness + +Status: not started. Dependencies: Stage 4. + +Scope: integration review, canonical documentation, roadmap completion records, +and proportionate fixes for remaining acceptance failures. No feature expansion. + +1. Check all twelve roadmap acceptance criteria against named tests and current + implementation. Record the mapping in this stage's completion entry. Passing + broad suites alone is not evidence that a missing scenario was exercised. +2. Inspect every new query path for environment/filesystem access, raw values, + mutable aliases, accidental callback-lifetime extension, and per-query whole-tree + work. Confirm no history was duplicated solely for these queries and loads + without a resolver avoid constructing an unused entry snapshot. +3. Review public signatures and current configuration behavior against the previous + release. Confirm the capability is additive and no parser, path, composition, + credential, or existing callback semantics changed unintentionally. +4. Update the internal overview and finish canonical Go/configuration/consumer + documentation. Transfer implemented contract prose from the feature roadmap + to its proper owners and retain a brief completion record with links. Preserve + historical ADR rationale and published notes. No new ADR is required for this + information-access extension unless implementation actually changes a governing + decision; stop and obtain review rather than silently widening scope. +5. Run the full source checker with qualifying acquired toolchains, recording the + actual versions and host. The following are the repository's recorded versions; + follow the release runbook if selecting updated toolchains: + + ```sh + CONFIGKIT_BASELINE_TOOLCHAIN=go1.25.5 \ + CONFIGKIT_NEWER_TOOLCHAIN=go1.26.7 \ + ./scripts/release-check.sh + ``` + +6. Hand off remaining publication work separately: maintainer version choice, + dependency reassessment, version-matched release notes, completed-roadmap + retirement and link repair, exact-commit Linux/macOS validation, publication + guards, and clean consumer retrieval. Follow [the runbook](../release.md). + Historical one-release exceptions do not carry forward to this feature. + +Acceptance: all target requirements have concrete evidence; available automated +checks pass; documentation distinguishes implemented behavior from release status. +Implementation completion is not publication approval. Do not create a release +tag or mark unavailable native macOS evidence as passed. + +Validation: `./scripts/check.sh`, `go test -race ./...`, and the full checker above. + +## Open Questions + +None blocking implementation. The API, snapshot boundary, missing-value semantics, +source identity handling, ownership, and example fallback policy are specified by +the roadmap and decisions above. Release version and publication timing are later +maintainer decisions, not unresolved implementation choices. If new evidence +contradicts a target contract, document the conflict, a recommended correction and +its rationale, and at least one viable alternative before proceeding past it. diff --git a/docs/roadmap/source-origins-during-resolution.md b/docs/roadmap/source-origins-during-resolution.md new file mode 100644 index 0000000..a771c7f --- /dev/null +++ b/docs/roadmap/source-origins-during-resolution.md @@ -0,0 +1,229 @@ +# Source origins during resolution + +Status: planned; not implemented. This roadmap replaces the Notarius feature +request and owns the targeted end state. The [implementation plan](implementation.md) +owns execution order, progress, and handoffs. +This document does not authorize implementation or select a release version. + +Current behavior remains defined by the [configuration contract](../config.md). +The [architecture policy](../policy/architecture.md) governs responsibility +boundaries, and the [documentation policy](../policy/documentation.md) governs +contract ownership and retirement of completed plans. + +## Motivation + +Notarius needs source-aware normalization of configured filesystem paths. +For example, `../references/party.txt` declared in `pipelines/dnd.yml` should +resolve relative to that declaring file, rather than the process working directory. +A profile replacement should use the profile file's directory even when it supplies +identical text; an untouched sibling should retain its original declaration. + +Notarius separately chooses invocation-relative CLI paths, main-configuration- +relative defaults, and unchanged absolute paths. These are application policies, +not ConfigKit conventions. ConfigKit must provide the provenance needed to apply +them in one load without independently parsing YAML or reading files again. + +Today, `Definition.Resolve` can inspect explicitness and record normalization, but +source-origin queries are available only after loading. Changing `Result.Config` +then would not change its frozen effective output or explanation. The target is to +make provenance available before normalization, while keeping runtime values and +inspection consistent through existing change recording. + +## Target public surface + +Add the following methods to the existing callback-scoped `Resolution` context: + +```go +func (r *Resolution) InputInfo(path Path) (FieldInfo, bool) +func (r *Resolution) Source(id string) (SourceInfo, bool) +``` + +These signatures are the proposed target API, not callable APIs in the current +release. Reuse existing public `Path`, `FieldInfo`, `Origin`, `Location`, and +`SourceInfo` types. Do not introduce a second provenance record hierarchy, a +generic node API, or a new callback. + +### InputInfo: a fixed pre-resolution view + +Despite its short name, `InputInfo` describes the **decoded configuration at entry +to resolution**, including retained application defaults. It is not limited to +literal YAML nodes and is not a live view of callback mutations. + +Its snapshot is taken after imports/composition, selected-profile application, +CLI overrides, defaults, and typed decoding, but before application resolution +and typed credential lookup. The query returns the same presence/origin meaning +as effective metadata at that boundary: + +| Path state at callback entry | Result | +| --- | --- | +| Surviving file, profile, or CLI contribution | Found; existing explicit/null and origin information | +| Present default-only value | Found; non-explicit, default origin, no invented file location | +| Explicit null accepted by the schema | Found; explicit and null | +| Missing path or descendant removed before decoding | Not found unless ordinary decoding retained/reintroduced it through defaults | +| Invalid path or key/index shape mismatch | Not found | + +A Go zero-valued field that exists in the decoded projection is not a missing +path. Zero metadata fields alone must not be used to infer absence; callers use +the returned boolean. Invalid/missing queries add no diagnostics. + +Queries address exact semantic paths through nested structs, inlined fields, +mapping entries, arrays, and sequence elements. Use existing path rules, including +escaped keys, explicit key/index distinctions, and canonical contextual sequence +indices. Never substitute nearest-ancestor diagnostic lookup for exact field lookup. + +For a scalar, the origin identifies its surviving declaration, not the contributors +of its parent mapping. Equal-valued profile and CLI replacements retain the +replacing origin. Containers retain their existing possibly composite locations; +the API does not select a representative owner or single declaring directory. + +`InputInfo` and `IsExplicit` must agree for paths present in the entry snapshot. +The extension must not change existing `IsExplicit` behavior for paths outside +that view. Normalization records, additions, removals, and moves do not alter the +snapshot. A removed original path stays queryable; a new destination that did not +exist at entry remains absent from this view. If a destination already existed, +its query describes that original value, not the moved value. + +### Source: resolve identities without filesystem work + +`Source(id)` returns the existing source record for an ID belonging to this load. +Empty or unknown IDs return a zero record and false without adding diagnostics. +The source view contains the records already acquired before resolution, including +main input, imported files, the selected profile, and CLI records where applicable. +It does not discover files, reload input, or inspect unselected profiles. + +A returned `SourceInfo.Path` is the canonical local path only when the existing +source model supplies one. File paths retain current canonicalization and symlink +semantics. Labels remain diagnostic labels, never a substitute for paths. +Bytes/readers and CLI records have no fabricated file path, even if a filesystem +context was supplied for imports or profiles. Defaults have no source record or +synthetic file identity. + +Every source identity referenced by the entry view's origin locations must be +resolvable through this source view. Existing fields distinguish layers and import +relationships; do not require callers to guess scalar ownership from source order. +The existing main-source identity convention remains unchanged. Applications may +use the main source record for their own default-base policy; a non-file main +source still supplies no directory, and the application must choose its fallback. + +Multiple locations do not imply one declaring file. Consumers needing a single +file must inspect the exact value's origin and source records and handle ambiguity +explicitly. No new `DeclaringFile` shortcut should silently discard contributors. + +### Lifetime and detachment + +Both methods require an active resolution callback. Nil, zero, or retained contexts +used after callback closure follow the existing fixed, value-free panic convention, +including when a callback returns an error. These methods do not make the context +safe for concurrent use. + +Returned records are detached, including nested slices and `ImportedFrom` +locations. They may be retained after callback exit; modifying them must not affect +subsequent queries, other loads, final metadata, or explanations. The context's +lifetime restriction does not apply to those copied records. + +## Consumer normalization workflow + +The intended Notarius workflow is: + +1. Query a path's entry-state field information before moving or normalizing it. +2. Resolve its origin source IDs and select a base using application policy. +3. Compute the application path without asking ConfigKit to interpret it. +4. Assign and record the result using existing `Normalize` or `Resolution.Record` + semantics, including dependencies where appropriate. +5. Use the completed result for both execution and configuration inspection. + +Consumers may retain original origin records for later dependent work in the same +callback. ConfigKit does not infer ownership transfers, rewrite the input view, or +automatically record a move. Existing mutation auditing and record validation +continue to apply to all changes. + +For canonical file sources, a symlink refers to the resolved target file's +directory, not necessarily the directory containing the link used to open it. +The maintained example must make this explicit. Applications wanting another +path-base policy own that policy and the additional information it requires. + +## Intended internal architecture + +Keep one provenance model and one loading lifecycle. The existing +`resolveLoadInput` in `load_result.go` already assembles decoded metadata and +overlay history before calling `runResolution`. Pass a read-only entry-state +provenance view and the load's acquired source records into the private resolution +state. No public parser types or new external dependencies are needed. + +The view must have complete effective semantic-path indexing when the callback +starts. Current result metadata indexes are finalized later; exposing an unfinished +`Metadata` value would make legitimate lookups fail. Reuse or factor existing +effective-path traversal and shape matching so entry queries and final queries +cannot acquire different path semantics. A full historical index or public +`Metadata.History` interface during resolution is not required by this feature. + +Keep the view independent of metadata updates performed after resolution. +Implementations may share immutable backing data privately while returning detached +records; do not copy an entire configuration tree per query. Source lookup should +use a per-load index or equivalent bounded existing mechanism, rather than requiring +each consumer to rebuild source bookkeeping. + +The source and input views remain bounded by existing load limits. They contain +provenance records, not configuration values or resolved credentials. Do not read +the environment, resolve secret references, perform filesystem I/O, or invoke +application codecs in response to a query. + +Existing composition precedence, canonical decoding, normalization auditing, +resolution origins/history, redaction, and frozen result publication remain +authoritative. This is an information-access extension, not a new normalization +engine or permission to mutate post-load snapshots. + +## Acceptance criteria + +The completed feature must demonstrate the following through public API behavior, +using real temporary files where source identity matters: + +1. A main-file scalar can be normalized relative to its canonical declaring file + during resolution, independent of invocation directory. +2. An imported scalar, including a sequence element, identifies its own declaring + file rather than the main file or a parent container's contributors. +3. A profile replacement identifies the profile source while untouched siblings + retain their base/import origins. +4. Equal-text profile and CLI replacements still identify the replacing source. +5. Defaults, missing paths, explicit null, CLI input, and file input are distinct; + `NoSource` and non-file input introduce no fictional declaring paths. +6. Composite containers retain all contributing locations without selecting a file. + Every referenced source ID resolves, and unknown IDs are harmless misses. +7. Queries remain fixed across normalization, deletion, addition, and moves; + retained copies remain useful and detached. +8. Exact lookup works through typed and dynamic structures, escaped mapping keys, + and sequence indices; invalid paths and shape mismatches do not borrow ancestor + or sibling provenance. +9. A successful single load yields identical runtime and effective values, with + explanation preserving original declarations and recorded normalization reasons. +10. Querying origins performs no extra load or credential lookup, and existing + secret redaction, safe diagnostics, and callback-lifetime rules remain intact. +11. Mutating returned records cannot affect later queries or final results. +12. Maintained Notarius-shaped examples demonstrate the policy using only the + public API and standard library, including documented canonical/symlink behavior. + +Tests should extend existing provenance, resolution, ownership, and consumer +behavioral owners rather than duplicate every mechanism at every layer. Acceptance +does not require a live Notarius checkout, external credentials, or target-file +existence checks. + +## Explicit exclusions + +Do not add automatic path-string recognition, filesystem path normalization, +existence checks, path confinement, default/CLI base selection, source discovery, +new import/profile composition rules, historical value access during resolution, +automatic move tracking, post-load mutability, or a general provenance service. +Actual Notarius integration remains downstream work. + +## Compatibility and canonical ownership + +This is an additive public capability; existing callbacks, loading order, and +configuration semantics must remain unchanged. Release selection and validation +follow the [release policy](../policy/release.md) and +[release procedure](../release.md). No release gate is waived here. + +In the completed feature, Go reference comments own API and lifetime definitions, +the configuration contract owns shared resolution/provenance semantics, and the +adoption guide and executable examples own consumer usage. Component documentation +describes the resulting internal boundaries. Accepted ADR rationale remains +historical; changes to governing decisions follow the documented supersession policy.