ConfigKit/docs/roadmap/implementation.md

Source origins during resolution: implementation plan

Status: implementation complete (2026-09-12). All stages were completed in order. The feature roadmap owns the target contract and consumer intent. This document owns implementation steps, validation, and handoffs. It is written for one gpt-5.6-terra coding agent per stage; no stage requires parallel agents or changes to a downstream repository.

Required reading and working rules

Before each stage, inspect the worktree and read the preceding stage's completion record. Follow development and all policies: architecture, documentation, testing, and release. Read the roadmap, the configuration contract's resolution and metadata sections, and the relevant code before editing. Use the code graph for discovery and verify current definitions rather than assuming line numbers or private signatures.

Preserve unrelated work. Keep each stage independently buildable with passing tests; do not introduce stub public methods or defer known correctness failures. Run the focused commands listed for the stage and ./scripts/check.sh from the repository root. Acquire required toolchains/dependencies explicitly before offline checks. Document unsupported filesystem fixtures rather than silently omitting them.

Use real collaborators and temporary files for provenance. Keep default tests offline, deterministic, and credential-free. Prefer codes, paths, layers, locations, and observable snapshots over private call counts or incidental error messages. Only use a narrow I/O or credential seam where it proves the no-extra-access rule.

At each handoff, update that stage's status here with changed files, tests/results, actual toolchains, deviations, and remaining risks. Keep public reference and current contract documentation synchronized with behavior as it lands. Commit coherent stage work when authorized, but do not push, tag, select a version, or prepare a speculative release note as part of this implementation sprint.

Implementation decisions

The roadmap's two method signatures are fixed for this sprint:

func (r *Resolution) InputInfo(path Path) (FieldInfo, bool)
func (r *Resolution) Source(id string) (SourceInfo, bool)

The required behavior is a stable entry-state view and detached public return records, not a particular internal representation. The recommended implementation is a private Metadata value containing effective fields and path shapes, together with a source-ID index. Reuse Metadata.Lookup semantics. A historical view is not needed for this feature, and exposing one is outside this sprint's scope.

Build the view after metadataFromDecoded and applyOverlayMetadata in resolveLoadInput, immediately before runResolution, only when a resolver exists. Its path shapes come from decoded.final, not the raw document or mutable Go configuration. Separate copies of field/source records are one straightforward way to isolate the view; immutable sharing or ownership transfer is equally acceptable when it preserves query stability and detached returns. Source records come from assembledInput.sources. Avoid aliases that observable consumer operations or subsequent metadata updates could mutate while the view remains usable. Missing or duplicate internally generated IDs are implementation-invariant failures to fix, not new consumer-facing diagnostic categories.

Prefer passing the private view explicitly to runResolution and retaining it in resolutionState. Update all private callers/tests deliberately: construct their entry view from their decoded fixtures and supplied sources where they test these queries. Every production resolution callback must receive complete provenance. Private signatures and representation may change when an idiomatic alternative better serves that requirement; record the reasoning in the stage handoff.

Queries first check callback activity through the existing context guard. Only then may invalid paths or unknown IDs return misses. InputInfo delegates to the entry metadata lookup; Source clones the matched source on return. Preserve existing IsExplicit, diagnostics, resolution recording, and secret timing.

Queries must not reload input, read files, resolve symlinks, access credentials, or re-project mutable configuration instead of reading entry-state information. Prefer reusing existing indexes and avoiding unnecessary whole-tree copies. Per-load source indexing is a recommendation, not a complexity guarantee: a simple scan may be appropriate for the existing bounded source collection. Assess costs in context, including path lengths and nested record copies; this feature adds no performance guarantees, resource caps, or benchmark gates.

Stage 1 — Factor reusable effective-path indexing

Status: complete (2026-09-12). Dependencies: none.

Scope: metadata.go, load_metadata.go, and their focused test owners.

1. Prepare reusable effective-path lookup for the entry view. The recommended approach extracts indexing from finalizeMetadataPaths; an equivalent design may reuse existing traversal directly. Keep historical indexing and result publication behavior unchanged, and preserve one set of semantic-path rules.
2. Add the private entry-view construction described above using pre-resolution metadata, the decoded final tree, and acquired source records. Share extracted machinery with result finalization where useful. Do not expose public methods yet.
3. Establish and document internal ownership sufficient for stable queries and detached returns. Reuse cloneFieldInfo and cloneSourceInfo where copying is needed; do not require eager copies or duplicate history solely to match this plan's recommended representation.
4. Add focused coverage proving the private view recognizes defaulted fields, missing descendants, root, escaped keys, explicit key/index mismatches, and contextual canonical sequence indices. Test stability across actual lifecycle operations and mutation of returned records, not arbitrary mutation of backing data that the implementation owns as immutable. Reuse existing shape tests for exhaustive syntax cases rather than duplicating them.

Acceptance: all existing final metadata/history behavior is unchanged; a complete entry view can be constructed before final publication without a historical index. No public API or configuration contract changes are claimed by this stage.

Validation: go test . -run 'Metadata|Path|Provenance|Historical', then the everyday checker. Handoff should identify the helper and constructor signatures for Stage 2.

Completion record (2026-09-12): extracted effectivePathIndex(final) from final metadata publication and introduced newResolutionInput(metadata, final, sources). The private view owns cloned metadata/source records, indexes source IDs, and exposes private detached lookup helpers for the next step; it intentionally has no public API or load-lifecycle wiring yet. TestResolutionInputKeepsDetachedEntryMetadata covers root, defaults, explicit null, missing descendants, escaped keys, contextual parsed sequence indices, key/index mismatches, detached records, and stability after real recorded resolution. go test . -run 'Metadata|Path|Provenance|Historical' and ./scripts/check.sh passed with Go 1.26.7 on Linux. No deviations or remaining Stage 1 risks were identified.

Stage 2 — Wire the callback API and enforce lifetime

Status: complete (2026-09-12). Dependencies: Stage 1.

Scope: load_result.go, resolution.go, necessary private callers, reference comments, and resolution/callback ownership tests.

1. Build and pass the entry view at the existing pre-resolution boundary. Use input.sources, which includes sources acquired by import, profile, and CLI processing; do not reconstruct source identities from labels or field locations.
2. Implement both public methods with the existing activity guard and detached returns. Keep lookup state frozen when recording changes or mutating configuration.
3. Add public-load tests for a file-authored scalar, a defaulted field, explicit null, a missing path, empty/unknown source IDs, and NoSource. Assert exact InputInfo.Explicit agreement with IsExplicit for found paths.
4. Exercise nil/zero contexts and retained contexts after successful and failed callbacks. Both methods must use the existing fixed closed-context panic behavior. Saved return records remain usable after callback exit. Do not broaden this work into a new concurrency contract or a general callback framework refactor.
5. Add Go method documentation and update the configuration contract's resolution section to describe the implemented entry-state semantics, misses, lifetime, and source distinction. Fix the development guide's stale statement that this implementation plan has not yet been written.

Acceptance: both methods work during an ordinary public Load; calls cannot access live mutable metadata or bypass callback scope. No-rule/no-resolver behavior and existing mutation auditing remain unchanged.

Validation: go test . -run 'Resolution|Callback|Metadata|Load', then the everyday checker. Do not mark complete with public methods that work only in private tests.

Completion record (2026-09-12): resolveLoadInput now creates a resolutionInput after decoded/overlay metadata and passes it explicitly into runResolution; the private state retains it for the callback. Resolution.InputInfo and Resolution.Source use the existing activity guard before exact entry lookup or source lookup, and return detached records. Public load tests cover a canonical file-authored value, a retained default, explicit null, misses, empty/unknown IDs, and defaults-only input; they also cover nil, zero, successful-retained, and failed-retained contexts plus saved record independence. Go reference comments, the configuration contract, and the development guide now describe the active work accurately. go test . -run 'Resolution|Callback|Metadata|Load' and ./scripts/check.sh passed with Go 1.26.7 on Linux. No deviations or remaining Stage 2 risks were identified.

Stage 3 — Verify layered provenance and snapshot independence

Status: complete (2026-09-12). Dependencies: Stage 2.

Scope: public lifecycle/provenance, import/profile/override, and ownership tests; production fixes only for gaps exposed at those boundaries.

1. Build a real temporary import graph with path-bearing scalars and a sequence element. Include a composite mapping with contributions from distinct files. During resolution, assert scalar ownership, all container contributors, and successful source lookup for every origin location returned by the view.
2. Apply a profile replacing one scalar and an equal-text value; verify replacing origins and untouched siblings. Exercise equal-text CLI replacement separately. Keep source IDs derived from returned records, not inferred from value equality.
3. Exercise byte and reader roots with filesystem context and file imports: the memory root has no file path while imported files do. Also cover non-file input without imports, default-only fields, and unknown source IDs. Retain default descendants reintroduced by normal decoding instead of treating them as raw input declarations.
4. Extend shape coverage through inlined structs/maps, typed arrays, and dynamic structures. Assert misses for invalid indices, signed index tokens in sequences, and explicit key/index mismatches; signed mapping keys remain literal keys.
5. Normalize, delete, add, and move fields within a callback using valid records. Queries for original paths must stay fixed; absent destinations stay absent, and pre-existing destinations retain their entry information. Final metadata must still reflect resolution rather than that fixed entry view.
6. Mutate returned Origin.Locations and SourceInfo.ImportedFrom; verify repeated queries, saved records, final results, and another load are independent. Exercise independent loads under the race detector without sharing an active context.

Acceptance: roadmap criteria 2–8 and 11 are covered at their owning boundaries. No duplicate provenance model, nearest-ancestor fallback, or source-order heuristic is introduced. Any discovered defect is fixed before the stage is marked complete.

Validation: go test . -run 'Resolution|Provenance|Import|Profile|Override|Metadata|Path', go test -race ./..., then the everyday checker.

Completion record (2026-09-12): public lifecycle tests now build real temporary main/import/profile graphs and prove scalar, sequence, composite-container, profile, and equal-text CLI provenance through InputInfo/Source, deriving source IDs from returned locations. Coverage also exercises byte and reader roots with filesystem imports, a standalone non-file root, retained defaults, unknown IDs, typed/inlined/map/array/dynamic shape matching, signed mapping keys, and invalid sequence shapes. Recorded normalization, deletion, addition, and move operations leave entry queries fixed while final metadata reflects resolution. Returned records, completed results, another load, and concurrent independent loads remain detached. No production gap was exposed. go test . -run 'Resolution|Provenance|Import|Profile|Override|Metadata|Path', go test -race ./..., and ./scripts/check.sh passed with Go 1.26.7 on Linux. No deviations or remaining Stage 3 risks were identified.

Stage 4 — Demonstrate source-aware normalization and safe inspection

Status: complete (2026-09-12). Dependencies: Stage 3.

Scope: maintained Notarius-shaped example code/fixtures, executable examples, explanation/secret integration tests, and consumer guidance.

1. Add a small application-owned path-bearing configuration example in examples/consumers. Keep existing catalog/version examples working. Supply the invocation directory explicitly to the example loader rather than calling os.Chdir or adding process-global state to tests.
2. Implement the example's policy using the two public queries: absolute paths unchanged; CLI-relative paths based on the supplied invocation directory; default-relative paths based on the canonical main source; file-authored paths based on the scalar's declaring source. Use the existing primary-source identity convention for main-source lookup, not a guess about contributor ordering.
3. Demonstrate file-relative normalization with fixtures that have an unambiguous declaring file. If the example handles absent or ambiguous file provenance, make that behavior an explicit example policy, not a correctness requirement or prescribed Notarius policy. Rejection, a caller-supplied fallback directory, retaining a relative value, or warning are application choices. The example's selected behavior must be documented and tested; ConfigKit must continue to expose the distinction rather than fabricate a file from a diagnostic label.
4. Record every changed path through Normalize or Record, including valid dependencies where derived values depend on other configuration paths. Prove runtime values, effective YAML, and explanation agree; retain the original relative declaration and normalization reason in history. No existence or confinement checks are part of the example.
5. Add a symlink fixture proving canonical target-directory semantics and an invocation-directory mismatch proving resolution does not depend on process CWD. Make unsupported symlink capability explicit under the testing policy.
6. Use synthetic sensitive fields to verify original/effective redaction remains intact. With existing reader/secret lookup seams, prove queries do not reread source data or resolve credentials during the resolver; normal later lookup remains unchanged. Do not create credentials or environment-dependent tests.
7. Document the public workflow and canonical symlink choice in the consumer guide; point to executable artifacts instead of duplicating large code examples.

Acceptance: roadmap criteria 1, 9, 10, and 12 are demonstrated in one-load public workflows. Filesystem policy stays in the example application, not the root library.

Validation: go test ./examples/consumers, go test . -run 'Resolution|Explain|Secret|Example', then the everyday checker.

Completion record (2026-09-12): added the maintained NotariusPathConfig example and fixtures. Its application-owned resolver uses InputInfo/Source only: file values use their exact declaring file, defaults use source-1, CLI values use the explicit invocation directory, and absolute values remain unchanged. Missing CLI bases or missing/composite file provenance retain the relative value and report an application warning. Tests prove imported-file, default, CLI, absolute, and canonical symlink-target bases without changing process CWD; the symlink test explicitly skips when the platform cannot create it. They also prove effective/explanation agreement, retained original declaration and reason, sensitive redaction, and that provenance queries do not re-read a reader source or resolve credentials before normal lookup. The consumer guide links to the executable artifact. go test ./examples/consumers, go test . -run 'Resolution|Explain|Secret|Example', and ./scripts/check.sh passed with Go 1.26.7 on Linux. No deviations or remaining Stage 4 risks were identified.

Stage 5 — Audit the complete contract and hand off release readiness

Status: complete (2026-09-12). Dependencies: Stage 4.

Scope: integration review, canonical documentation, roadmap completion records, and proportionate fixes for remaining acceptance failures. No feature expansion.

1. Check all twelve roadmap acceptance criteria against named tests and current implementation. Record the mapping in this stage's completion entry. Passing broad suites alone is not evidence that a missing scenario was exercised.

2. Inspect every new query path for environment/filesystem access, raw values, caller-mutable aliases, and accidental callback-lifetime extension. Review allocation and traversal costs for proportionality without turning the chosen mechanism into a public guarantee or adding unsupported performance gates. Avoid unnecessary history duplication and unused work in no-resolver loads.

3. Review public signatures and current configuration behavior against the previous release. Confirm the capability is additive and no parser, path, composition, credential, or existing callback semantics changed unintentionally.

4. Update the internal overview and finish canonical Go/configuration/consumer documentation. Transfer implemented contract prose from the feature roadmap to its proper owners and retain a brief completion record with links. Preserve historical ADR rationale and published notes. No new ADR is required for this information-access extension unless implementation actually changes a governing decision; stop and obtain review rather than silently widening scope.

5. Run the full source checker with qualifying acquired toolchains, recording the actual versions and host. The following are the repository's recorded versions; follow the release runbook if selecting updated toolchains:

   CONFIGKIT_BASELINE_TOOLCHAIN=go1.25.5 \
   CONFIGKIT_NEWER_TOOLCHAIN=go1.26.7 \
   ./scripts/release-check.sh
   

6. Hand off remaining publication work separately: maintainer version choice, dependency reassessment, version-matched release notes, completed-roadmap retirement and link repair, exact-commit Linux/macOS validation, publication guards, and clean consumer retrieval. Follow the runbook. Historical one-release exceptions do not carry forward to this feature.

Acceptance: all target requirements have concrete evidence; available automated checks pass; documentation distinguishes implemented behavior from release status. Implementation completion is not publication approval. Do not create a release tag or mark unavailable native macOS evidence as passed.

Validation: ./scripts/check.sh, go test -race ./..., and the full checker above.

Completion record (2026-09-12): all acceptance criteria have named evidence:

1. TestNotariusPathExampleUsesEntryProvenance covers canonical main-file and symlink target bases without process-CWD dependence.
2. TestResolutionInputKeepsLayeredOriginsAndSources covers imported scalars and sequence elements.
3. The same layered test covers profile replacement and untouched base/import siblings.
4. The layered test covers equal-text profile and CLI replacement.
5. TestResolutionEntryQueriesExposeDetachedLoadInput and TestResolutionInputDistinguishesNonFileRootsAndImportedFiles cover defaults, nulls, misses, CLI/file identities, NoSource, bytes, and readers.
6. The layered test verifies every composite location has a source; TestResolutionEntryQueriesExposeDetachedLoadInput verifies unknown IDs miss safely.
7. TestResolutionInputRemainsFixedAcrossRecordedMutations covers normalization, deletion, addition, and move stability.
8. TestResolutionInputUsesEntryShapesAcrossTypedAndDynamicValues covers inline, map, array, dynamic, escaped/signed keys, and invalid shape cases.
9. TestNotariusPathExampleUsesEntryProvenance verifies runtime, effective YAML, and explanation history together.
10. TestResolutionInputQueriesDoNotReadOrResolveSecrets plus the Notarius test verify no extra reader/credential access and preserved redaction.
11. TestResolutionInputKeepsLayeredOriginsAndSources and TestResolutionEntryQueriesExposeDetachedLoadInput verify detached records.
12. NotariusPathConfig, its fixtures, executable example, and consumer-guide link provide the maintained public-API/standard-library workflow.

The final review found that queries use only the callback guard and detached metadata/source indexes; they do not extend callback lifetime, expose raw values, or perform environment/filesystem work. The view is created only for resolver loads and does not retain history. The comparison with v0.2.0 found only additive exported methods; loading, parser, path, composition, credential, and existing callback semantics remain unchanged. Canonical Go, configuration, consumer, internal, and roadmap documentation are synchronized; the feature roadmap is retained only until release preparation. ./scripts/check.sh, go test -race ./..., documentation and import-boundary checks, and CONFIGKIT_BASELINE_TOOLCHAIN=go1.25.5 CONFIGKIT_NEWER_TOOLCHAIN=go1.26.7 ./scripts/release-check.sh passed on Linux with Go 1.25.5 and Go 1.26.7. Publication remains unapproved: a maintainer must choose a pre-v1 minor version, write its note, retire completed roadmaps, obtain exact candidate native Linux/macOS evidence, and follow the release procedure. No version, release note, candidate, tag, or push was created.

Open Questions

None blocking implementation. The API, snapshot boundary, missing-value semantics, source identity handling, and ownership are specified by the roadmap and decisions above. Example fallback behavior is illustrative application policy, not a missing library contract; record any selected example behavior without prescribing it to consumers. Release version and publication timing are later maintainer decisions, not unresolved implementation choices. If new evidence contradicts a target contract, document the conflict, a recommended correction and its rationale, and at least one viable alternative before proceeding past it.