Align layering roadmap with consistent version semantics

docs/roadmap/sources-and-layering.md

@@ -24,6 +24,13 @@ Design priorities, in order:
 3. A small public surface with cohesive private implementation, usable by simple
    applications without requiring them to learn the layered-loading machinery.
 
+Consumers load once and receive a complete configuration and its inspection
+metadata. They should not need a second generic merge/defaulting pass, a separate
+presence tracker, or reconstructed provenance. Domain-specific resolution remains
+consumer-owned, but fits into the recorded loading lifecycle before the result
+is validated and returned. Subject to consistent semantics and correct presence
+and provenance, minimizing consumer integration code is an explicit design goal.
+
 The [named-inheritance request](feature-request-promptkit-inheritance.md) remains
 separate and optional. This roadmap neither implements it nor adds speculative
 infrastructure for it.
@@ -43,6 +50,10 @@ these proposals.
 - Composition rejects non-mapping overlaps, including equal values. Overlay
   recursively merges mappings and replaces other values. Empty mappings do not
   clear existing children; null is an explicit value, not omission.
+- An explicitly configured schema-version path currently belongs to the selected
+  source. Its checker runs before overlay/CLI input, which cannot alter that path;
+  imported fragments cannot contribute it. The target below deliberately revises
+  those ownership and timing rules.
 - CLI overrides already convert supported programmatic scalars, lists, and
   string-keyed maps directly to the private document representation. They have
   CLI-specific paths, labels, restrictions, and origins, not a general source API.
@@ -58,6 +69,8 @@ these proposals.
 Existing source/lifecycle contracts, ownership, secret handling, and diagnostics
 remain the compatibility baseline. This work is not permission to silently
 reinterpret current `Load`, imports, selected overlays, or CLI overrides.
+Effective-version validation is an explicit exception described below, not an
+unannounced consequence of refactoring.
 
 ## Target source model
 
@@ -119,6 +132,13 @@ Support native filesystem, host-file, byte, reader, and programmatic inputs.
 The order is supplied by the caller; ConfigKit does not discover priorities or
 know about PromptKit's catalog tiers.
 
+The selected operation determines how values combine, not their source format
+or loading entry point. Equivalent file, embedded, byte/reader, and programmatic
+input uses the same presence and combination rules. Explicitly choosing compose,
+overlay, or replacement expresses different intentions; merely changing the input
+medium must not switch between them. This applies to both ordinary `Load` and
+the new layered-loading interface wherever they offer the same operation.
+
 Keep strict composition and intentional overriding visibly distinct:
 
 | Operation | Desired semantics |
@@ -133,23 +153,68 @@ precedence is applied. An overlay must not hide malformed YAML or a composition
 conflict in an earlier contribution. There is no automatic matching by an `id`
 field, implicit sequence merge, universal profile registry, or merge callback.
 
-Assemble the effective input before applying the typed defaults/decoding,
+Assemble the effective input, including selected-overlay and CLI contributions,
+then check its configured schema version before typed defaults/decoding,
 application resolution, secret resolution, semantic validation, and final
-snapshot lifecycle. Those latter operations run once per load, not once per
+snapshotting. Each lifecycle phase runs once per load, not once per layer.
-layer. Higher layers may replace values that would not fit the final typed schema;
+Higher layers may replace values that would not fit the final typed schema;
 effective typed validation is not validation of each intermediate snapshot.
 Input syntax, source access, and composition remain validated at acquisition.
 
 The standard precedence relationship is typed defaults below assembled input,
 then the existing selected-overlay convenience and explicit CLI overrides where
-those are used, followed by recorded application resolution. The new API must
+those are used, followed by recorded application resolution. Named-overlay lookup
-make the selected overlay's owning filesystem context unambiguous; Decision 4
+has an unambiguous filesystem context, separate from version ownership: ordinary
-addresses the primary-root role alongside schema-version ownership.
+loading retains its source/root convention; a multi-context load explicitly
+identifies the lookup context when that convenience is used. Other sources retain
+their own import and resource contexts. Callers do not manufacture a primary
+document merely to combine layers, and no overlay lookup context is needed when
+no named overlay is selected.
 
 Replacing lower input does not mean resetting or deleting typed application
 defaults. Existing defaults and null rules still apply during final decoding.
 The distinction must be visible in examples, especially empty maps and partially
 specified definitions. No new reset/delete operator is implied by replacement.
+For example, replacing input `{timeout: 10, retries: 2}` with `{timeout: 20}`
+removes the lower input's `retries: 2`; a typed default of `retries: 3` can still
+supply the final value. The metadata then attributes `retries` to defaults, not
+to the replaced input. An empty replacement entry likewise removes lower-input
+children without suppressing applicable typed defaults.
+
+### Effective-configuration schema version
+
+An application opts into version checking by configuring a version path. A key
+spelled `version` has no automatic special status. The configured path follows
+the selected combination operation like any other setting:
+
+- Strict composition rejects conflicting scalar definitions, even if equal;
+  it is not last-wins.
+- An intentional overlay can replace an earlier version value. Imports may
+  contribute the version subject to ordinary composition rules; selected overlays
+  and CLI input are not excluded solely because a path is the version path.
+- Any selected subtree replacement has the same effect on version presence as
+  on other values in that subtree, including removing a lower-input version.
+
+After all input precedence operations, inspect the effective version and invoke
+the configured checker once, before typed decoding. Preserve the existing
+supported scalar forms, absent/null distinction, version diagnostics, and
+version-path integer-to-string decoding behavior. As today, an invalid effective
+version type produces a diagnostic rather than invoking the checker. Absent
+effective input is reported as absent, not manufactured from typed defaults.
+Version origins and replacement history follow the same rules as other fields.
+
+Do not check each source's version independently or fail merely because a
+superseded version value would fail effective-version validation. Invalid YAML,
+source failures, and strict-composition conflicts still fail at their owning
+phases; precedence does not conceal those errors. A later version value does not
+migrate earlier data. Consumers supply compatible inputs and own schema meaning;
+automatic migration and per-document format interpretation remain out of scope.
+
+Apply this model consistently to existing and new loading interfaces. It
+deliberately changes the current selected-source ownership restrictions and moves
+checking after overlay/CLI application. Document the resulting compatibility
+impact and callback timing in the feature's minor release rather than retaining
+different version semantics for old and new entry points.
 
 ### Programmatic presence
 
@@ -169,13 +234,20 @@ boundary. Results and inspection remain independently owned under existing
 contracts. This does not promise language-enforced immutability of Go values or
 an immutable filesystem.
 
+A final Go value alone need not encode presence: an omitted boolean and an
+explicit `false` can both decode to `false`. The returned configuration together
+with its presence/provenance metadata preserves that distinction through
+defaulting, layering, and recorded resolution. Null, empty values, zero, and
+absence must not be collapsed by source conversion or merging.
+
 ### Provenance and operational boundaries
 
 Origins, presence, history, explanation, and effective rendering agree with the
 final typed result. Overridden and replaced definitions remain explainable without
 retaining stale effective descendants. Replacements identify their scope and
 displaced sources; equal-value explicit overrides are still meaningful history.
-Retain redaction and callback-lifetime behavior.
+Include typed defaults and application-derived values, not only source-supplied
+values, in final-value provenance. Retain redaction and callback-lifetime behavior.
 
 Apply existing resource-accounting principles across the whole load rather than
 resetting allowances for every layer. Audit how bytes, programmatic nodes,
@@ -231,6 +303,9 @@ These are responsibility areas, not implementation stages:
   with ordinary loading rather than building a parallel configuration system.
 - Resolve the fixed operation and ownership decisions below, then extend source
   records, histories, diagnostics, and rendering consistently across input kinds.
+- Replace selected-source version ownership with the effective-version lifecycle
+  across both loading interfaces, including import, overlay, and CLI handling.
+  Keep filesystem-context selection independent of version checking.
 - Refactor mapping assembly and provenance updates where current algorithms
   would scale poorly under layered workloads. Keep optimization scope tied to
   the new workflow rather than rewriting unrelated library subsystems.
@@ -239,18 +314,26 @@ These are responsibility areas, not implementation stages:
   and null values, caller/result independence, and source-correct redaction and
   explanation. Add comparative benchmarks without making performance a brittle
   pass/fail contract.
+- Compare equivalent logical inputs across supported source kinds and loading
+  interfaces. Assert consistent values, presence, operation outcomes, and origin
+  meaning while allowing genuine location/label differences. Cover defaulted and
+  derived values, replacement followed by typed defaults, version conflicts under
+  composition, and effective-version checking after intentional overrides.
 - Provide maintained simple and PromptKit-shaped examples: embedded defaults,
   operator configuration from another context, and labeled programmatic input.
   Demonstrate exact operations and resource origins without provider calls or
-  recreating PromptKit itself.
+  recreating PromptKit itself. The consumer supplies sources, explicit operations,
+  types, and domain rules; it does not implement generic merging, presence
+  tracking, or origin reconstruction after loading.
 - Update Go declaration documentation, the configuration contract, consumer
   guidance, component overview, and relevant architecture/deferred-scope records
   as behavior lands. Record significant boundary changes in an ADR without
   rewriting accepted ADR history. Keep named inheritance explicitly separate.
 
-Plan an additive minor release for the new public capability. If any design choice
+Plan a minor release for the new public capability and the deliberate change to
-requires changing an existing documented behavior, identify it expressly and
+schema-version ownership and checking order. Explain the version-checking change
-provide migration guidance; do not describe it as an internal optimization.
+and provide migration guidance; this release is not wholly additive. Identify any
+other compatibility changes expressly rather than calling them optimizations.
 No release, implementation change, or current-policy revision occurs in this pass.
 
 ## Completion criteria
@@ -259,11 +342,16 @@ A consumer can load embedded configuration with relative imports, combine it
 with independently located operator configuration and explicit programmatic
 values, and obtain one typed result with accurate presence, sources, history,
 and resource context. No temporary configuration files or serialization round
-trip is needed. Simple `Load` usage retains its current behavior. Benchmarks
+trip or subsequent generic configuration processing is needed. Simple `Load`
+usage retains its behavior except for the explicitly selected version-policy
+change, which is shared by both entry points. Final-value provenance covers input,
+defaults, and recorded domain resolution, and metadata distinguishes explicit
+zero/false from absence even when the typed values are equal. Benchmarks
 demonstrate that adding inputs does not entail avoidable full-collection copying
 or quadratic key lookup. Public contracts and examples clearly distinguish
 composition, overlay, any selected replacement operation, and downstream domain
-resolution.
+resolution. Equivalent-input tests and the maintained consumer example establish
+semantic consistency and reduced integration code, not merely API availability.
 
 ## Open Decisions
 
@@ -312,21 +400,3 @@ registry mapping IDs to their filesystems. Metadata stays capability-free and
 serializable, but every consumer must maintain and correctly reconcile another
 registry. This is viable if limiting retained filesystem capabilities proves
 more important than the additional integration machinery.
-
-### 4. Primary-root ownership in multi-source loads
-
-**Recommendation:** retain an explicit primary source/root role for application
-schema version and the optional named-overlay convention. Additional inputs are
-fragments or explicit precedence contributions, not independent version owners.
-This best preserves current version ownership and prevents precedence from hiding
-a version failure. An in-memory primary mapping can carry the version without
-requiring a generated file. Other source contexts still govern their own imports
-and resource references.
-
-**Alternative:** permit independently versioned roots and validate each before
-assembly, with a separately selected context for named-overlay lookup. This fits
-standalone catalogs more naturally, but needs new rules for combining accepted
-versions, callback counts, and the effective version's provenance. It should not
-silently invoke the existing version callback once per layer or let the last
-version value win. Resolve this choice with a realistic mixed-catalog example
-before producing the staged implementation plan.