ConfigKit/docs/roadmap/source-origins-during-resolution.md

Source origins during resolution

Status: planned; not implemented. This roadmap replaces the Notarius feature request and owns the targeted end state. The implementation plan owns execution order, progress, and handoffs. This document does not authorize implementation or select a release version.

Current behavior remains defined by the configuration contract. The architecture policy governs responsibility boundaries, and the documentation policy governs contract ownership and retirement of completed plans.

Motivation

Notarius needs source-aware normalization of configured filesystem paths. For example, ../references/party.txt declared in pipelines/dnd.yml should resolve relative to that declaring file, rather than the process working directory. A profile replacement should use the profile file's directory even when it supplies identical text; an untouched sibling should retain its original declaration.

Notarius separately chooses invocation-relative CLI paths, main-configuration- relative defaults, and unchanged absolute paths. These are application policies, not ConfigKit conventions. ConfigKit must provide the provenance needed to apply them in one load without independently parsing YAML or reading files again.

Today, Definition.Resolve can inspect explicitness and record normalization, but source-origin queries are available only after loading. Changing Result.Config then would not change its frozen effective output or explanation. The target is to make provenance available before normalization, while keeping runtime values and inspection consistent through existing change recording.

Target public surface

Add the following methods to the existing callback-scoped Resolution context:

func (r *Resolution) InputInfo(path Path) (FieldInfo, bool)
func (r *Resolution) Source(id string) (SourceInfo, bool)

These signatures are the proposed target API, not callable APIs in the current release. Reuse existing public Path, FieldInfo, Origin, Location, and SourceInfo types. Do not introduce a second provenance record hierarchy, a generic node API, or a new callback.

InputInfo: a fixed pre-resolution view

Despite its short name, InputInfo describes the decoded configuration at entry to resolution, including retained application defaults. It is not limited to literal YAML nodes and is not a live view of callback mutations.

Its snapshot is taken after imports/composition, selected-profile application, CLI overrides, defaults, and typed decoding, but before application resolution and typed credential lookup. The query returns the same presence/origin meaning as effective metadata at that boundary:

Path state at callback entry	Result
Surviving file, profile, or CLI contribution	Found; existing explicit/null and origin information
Present default-only value	Found; non-explicit, default origin, no invented file location
Explicit null accepted by the schema	Found; explicit and null
Missing path or descendant removed before decoding	Not found unless ordinary decoding retained/reintroduced it through defaults
Invalid path or key/index shape mismatch	Not found

A Go zero-valued field that exists in the decoded projection is not a missing path. Zero metadata fields alone must not be used to infer absence; callers use the returned boolean. Invalid/missing queries add no diagnostics.

Queries address exact semantic paths through nested structs, inlined fields, mapping entries, arrays, and sequence elements. Use existing path rules, including escaped keys, explicit key/index distinctions, and canonical contextual sequence indices. Never substitute nearest-ancestor diagnostic lookup for exact field lookup.

For a scalar, the origin identifies its surviving declaration, not the contributors of its parent mapping. Equal-valued profile and CLI replacements retain the replacing origin. Containers retain their existing possibly composite locations; the API does not select a representative owner or single declaring directory.

InputInfo and IsExplicit must agree for paths present in the entry snapshot. The extension must not change existing IsExplicit behavior for paths outside that view. Normalization records, additions, removals, and moves do not alter the snapshot. A removed original path stays queryable; a new destination that did not exist at entry remains absent from this view. If a destination already existed, its query describes that original value, not the moved value.

Source: resolve identities without filesystem work

Source(id) returns the existing source record for an ID belonging to this load. Empty or unknown IDs return a zero record and false without adding diagnostics. The source view contains the records already acquired before resolution, including main input, imported files, the selected profile, and CLI records where applicable. It does not discover files, reload input, or inspect unselected profiles.

A returned SourceInfo.Path is the canonical local path only when the existing source model supplies one. File paths retain current canonicalization and symlink semantics. Labels remain diagnostic labels, never a substitute for paths. Bytes/readers and CLI records have no fabricated file path, even if a filesystem context was supplied for imports or profiles. Defaults have no source record or synthetic file identity.

Every source identity referenced by the entry view's origin locations must be resolvable through this source view. Existing fields distinguish layers and import relationships; do not require callers to guess scalar ownership from source order. The existing main-source identity convention remains unchanged. Applications may use the main source record for their own default-base policy; a non-file main source still supplies no directory, and the application must choose its fallback.

Multiple locations do not imply one declaring file. Consumers needing a single file must inspect the exact value's origin and source records and handle ambiguity explicitly. No new DeclaringFile shortcut should silently discard contributors.

Lifetime and detachment

Both methods require an active resolution callback. Nil, zero, or retained contexts used after callback closure follow the existing fixed, value-free panic convention, including when a callback returns an error. These methods do not make the context safe for concurrent use.

Returned records are detached, including nested slices and ImportedFrom locations. They may be retained after callback exit; modifying them must not affect subsequent queries, other loads, final metadata, or explanations. The context's lifetime restriction does not apply to those copied records.

Consumer normalization workflow

The intended Notarius workflow is:

1. Query a path's entry-state field information before moving or normalizing it.
2. Resolve its origin source IDs and select a base using application policy.
3. Compute the application path without asking ConfigKit to interpret it.
4. Assign and record the result using existing Normalize or Resolution.Record semantics, including dependencies where appropriate.
5. Use the completed result for both execution and configuration inspection.

Consumers may retain original origin records for later dependent work in the same callback. ConfigKit does not infer ownership transfers, rewrite the input view, or automatically record a move. Existing mutation auditing and record validation continue to apply to all changes.

For canonical file sources, a symlink refers to the resolved target file's directory, not necessarily the directory containing the link used to open it. The maintained example must make this explicit. Applications wanting another path-base policy own that policy and the additional information it requires.

Intended internal architecture

Keep one provenance model and one loading lifecycle. The existing resolveLoadInput in load_result.go already assembles decoded metadata and overlay history before calling runResolution. Pass a read-only entry-state provenance view and the load's acquired source records into the private resolution state. No public parser types or new external dependencies are needed.

The view must have complete effective semantic-path indexing when the callback starts. Current result metadata indexes are finalized later; exposing an unfinished Metadata value would make legitimate lookups fail. Reuse or factor existing effective-path traversal and shape matching so entry queries and final queries cannot acquire different path semantics. A full historical index or public Metadata.History interface during resolution is not required by this feature.

Keep the view independent of metadata updates performed after resolution. Implementations may share immutable backing data privately while returning detached records; do not copy an entire configuration tree per query. Source lookup should use a per-load index or equivalent bounded existing mechanism, rather than requiring each consumer to rebuild source bookkeeping.

The source and input views remain bounded by existing load limits. They contain provenance records, not configuration values or resolved credentials. Do not read the environment, resolve secret references, perform filesystem I/O, or invoke application codecs in response to a query.

Existing composition precedence, canonical decoding, normalization auditing, resolution origins/history, redaction, and frozen result publication remain authoritative. This is an information-access extension, not a new normalization engine or permission to mutate post-load snapshots.

Acceptance criteria

The completed feature must demonstrate the following through public API behavior, using real temporary files where source identity matters:

1. A main-file scalar can be normalized relative to its canonical declaring file during resolution, independent of invocation directory.
2. An imported scalar, including a sequence element, identifies its own declaring file rather than the main file or a parent container's contributors.
3. A profile replacement identifies the profile source while untouched siblings retain their base/import origins.
4. Equal-text profile and CLI replacements still identify the replacing source.
5. Defaults, missing paths, explicit null, CLI input, and file input are distinct; NoSource and non-file input introduce no fictional declaring paths.
6. Composite containers retain all contributing locations without selecting a file. Every referenced source ID resolves, and unknown IDs are harmless misses.
7. Queries remain fixed across normalization, deletion, addition, and moves; retained copies remain useful and detached.
8. Exact lookup works through typed and dynamic structures, escaped mapping keys, and sequence indices; invalid paths and shape mismatches do not borrow ancestor or sibling provenance.
9. A successful single load yields identical runtime and effective values, with explanation preserving original declarations and recorded normalization reasons.
10. Querying origins performs no extra load or credential lookup, and existing secret redaction, safe diagnostics, and callback-lifetime rules remain intact.
11. Mutating returned records cannot affect later queries or final results.
12. Maintained Notarius-shaped examples demonstrate the policy using only the public API and standard library, including documented canonical/symlink behavior.

Tests should extend existing provenance, resolution, ownership, and consumer behavioral owners rather than duplicate every mechanism at every layer. Acceptance does not require a live Notarius checkout, external credentials, or target-file existence checks.

Explicit exclusions

Do not add automatic path-string recognition, filesystem path normalization, existence checks, path confinement, default/CLI base selection, source discovery, new import/profile composition rules, historical value access during resolution, automatic move tracking, post-load mutability, or a general provenance service. Actual Notarius integration remains downstream work.

Compatibility and canonical ownership

This is an additive public capability; existing callbacks, loading order, and configuration semantics must remain unchanged. Release selection and validation follow the release policy and release procedure. No release gate is waived here.

In the completed feature, Go reference comments own API and lifetime definitions, the configuration contract owns shared resolution/provenance semantics, and the adoption guide and executable examples own consumer usage. Component documentation describes the resulting internal boundaries. Accepted ADR rationale remains historical; changes to governing decisions follow the documented supersession policy.