Plan interface contract remediation

docs/roadmap/interface-remediation.md

@@ -0,0 +1,239 @@
+# Interface Contract Remediation Roadmap
+
+Status: proposed target; implementation has not begun.
+Planning baseline: `1f64673` (v0.10.2 implementation plus documentation changes).
+
+This roadmap covers the four findings in the [interface audit](interfaces.md).
+The audit owns reproductions, evidence, and finding numbers. This document owns
+the combined target, required changes, and general dependency order. A subsequent
+implementation plan should own concrete stages, private implementation choices,
+validation commands, and progress.
+
+## Purpose And Scope
+
+Restore agreement between ConfigKit's presence, metadata, history, and diagnostic
+interfaces under their existing contracts. The first three findings concern
+semantic paths that share a JSON Pointer spelling; the fourth concerns a
+composition claimant order that is lost during diagnostic canonicalization.
+
+The [Path declarations](../../path.go), [metadata contract](../config.md#layered-metadata),
+[resolution contract](../config.md#recorded-application-resolution), and
+[strict-composition contract](../config.md#strict-document-composition) remain
+authoritative. This work changes the implementation of those contracts without
+selecting a new public API, path syntax, precedence rule, or configuration
+restriction. The existing root-owned facade and engine responsibilities remain.
+
+## Current Baseline And Targeted End State
+
+An explicit mapping-key path and an explicit sequence-index path can both render
+as `/shape/0`. `ParsePath` deliberately leaves numeric tokens contextual, whereas
+`Keys` and `Index` preserve the caller's explicit choice. The engine already has
+shape-aware effective/historical path indexes and a frozen resolution-entry view,
+but some queries and metadata updates still use pointer spelling alone.
+
+| Finding | Current discrepancy | Target |
+| --- | --- | --- |
+| 1: resolution presence | `IsExplicit` and `Default` consult string-keyed presence after schema validation, which cannot determine actual shape beneath dynamic values. | Presence decisions use the matching semantic path in the fixed resolution-entry state. |
+| 2: effective explicitness | Resolution retains old field information when pointer spelling survives a container-shape change. | Input explicitness carries forward for the same semantic path; a newly introduced key or index does not inherit another path's explicitness. |
+| 3: history attribution | A historical-shape check admits the query, then `History` returns the entire textual-pointer bucket. | Exact definitions and previous origins belong to the queried semantic path, with applicable ancestor operations retained. |
+| 4: claimant order | Composition constructs related locations in assembly order, then canonicalization sorts them by location. | Structured and rendered composition conflicts preserve the selected claimant order. |
+
+For example, if input contains `shape: {'0': old}` and recorded application
+resolution changes `shape` to a sequence, the old key remains an input path in
+the callback's entry view. The new index is effective in final metadata but was
+not explicitly supplied by that input. Its history must not present the old key's
+definition as its own. This distinction concerns container shape, not identity
+tracking for records that move within a sequence.
+
+## Required Changes
+
+### 1. Use Entry-State Shape For Resolution Presence
+
+In [resolution.go](../../internal/engine/resolution.go), align `IsExplicit` and
+the explicitness decision in `Default` with the frozen, shape-aware entry metadata
+already used by `InputInfo` in
+[load_metadata.go](../../internal/engine/load_metadata.go).
+
+Reuse the existing semantic-path matching machinery. A shared private lookup can
+avoid divergent presence decisions without requiring every public query to have
+identical error behavior. In particular:
+
+- Preserve `IsExplicit`'s schema-path validation and its existing diagnostics.
+  `InputInfo` remains a query that returns false for missing or mismatched paths
+  without those diagnostics.
+- Preserve `Default`'s helper-target validation, assignment, recording, and final
+  helper-value checks. Correct its presence gate; do not bypass the other checks.
+- Use entry-state shape even after application code changes the working value.
+  Explicit false, zero, empty, and null retain their existing presence meaning.
+- Keep callback lifetime and detached metadata behavior intact.
+
+A shape mismatch beneath a schema-valid dynamic path must not become explicit
+merely because its pointer spelling matches an input path. Contextually parsed
+numeric tokens continue to resolve against the entry state's actual containers.
+
+### 2. Preserve Explicitness By Semantic Path During Resolution
+
+Update resolution metadata application and field refresh in
+[load_metadata.go](../../internal/engine/load_metadata.go) to compare the relevant
+entry and final shapes before reusing presence information.
+
+The comparison must account for container kinds along the full path. A mapping
+key becoming a sequence index at an ancestor can change a descendant's semantic
+path even when its last segment and full pointer spelling stay unchanged.
+
+Keep explicitness for matching surviving paths, including descendants covered by
+an ancestor normalization record. New semantic paths introduced by resolution
+remain non-explicit, and retain the applicable recorded resolution origin.
+Removed paths leave effective metadata while remaining available to historical
+queries under the existing contract. Do not clear every descendant's presence
+just because its parent was normalized.
+
+Use the existing before/after projection and entry metadata where possible.
+This work does not require tracking every transient mutation inside a callback;
+the existing net-change and coalesced-record model continues to apply.
+
+### 3. Retain Semantic Attribution In History
+
+Repair history event construction and selection across
+[load_metadata.go](../../internal/engine/load_metadata.go),
+[metadata.go](../../internal/engine/metadata.go), and their inheritance-history
+integration. Finding 3's choice of internal evidence representation is discussed
+under [Open Decisions](#open-decisions).
+
+An explicit query should select definitions belonging to its semantic shape.
+Ancestor replacement or removal operations can still affect that path and must
+remain in its history where applicable. An ancestor operation's `Change.Path`
+need not equal the queried descendant; this is existing behavior.
+
+Retain enough evidence to distinguish the affected path from the operation's
+reported path. Merely filtering current `Change.Path` values is insufficient:
+some construction paths recreate parsed tokens from strings, and ancestor events
+intentionally name another path. Review event construction for defaults, input
+definitions, precedence operations, inheritance, CLI, and resolution to identify
+where these distinctions can be lost. Change the affected owners without
+duplicating the entire loading lifecycle.
+
+Select or reconstruct `Previous` origins using the relevant semantic history.
+Review `historyWithInheritance` as well as stored events: filtering out a false
+definition while retaining it as the previous origin would leave the attribution
+defect unresolved. Preserve chronology and genuine ancestor effects, including
+history of removed paths. A path that never existed does not acquire history
+solely because an ancestor changed.
+
+Retain the existing contextual behavior of `ParsePath` across historical shapes
+and the one-entry-per-canonical-pointer enumeration of `HistoricalPaths`. Neither
+needs a public redesign. `History` and `Explain` should agree on which path a
+definition belongs to and what affected it; their different record forms do not
+need to become identical lists.
+
+Reuse existing evidence where practical. Any additional retained evidence must
+respect existing ownership, confidentiality, and applicable accounting; avoid
+retaining extra document trees merely to identify a path's shape. This is not a
+new memory quota or a general history-storage redesign.
+
+### 4. Preserve Composition Claimant Order Through Diagnostics
+
+Keep the order produced by `compositionConflictDiagnostic` in
+[compose.go](../../internal/engine/compose.go) through accumulation, `LoadError`,
+facade conversion, and `WriteDiagnostics` in
+[diagnostic.go](../../internal/engine/diagnostic.go).
+
+Prefer a narrow internal canonicalization rule based on the diagnostic semantics
+already carried across these boundaries. Identify the composition diagnostic
+producers governed by the claimant-order contract rather than changing all
+related-location lists. Preserve canonical ordering of diagnostic records and
+the existing sorting of related locations for other diagnostics.
+
+The root renderer converts public diagnostic records back to engine records and
+canonicalizes them again. An engine-only marker discarded by facade conversion
+would therefore be insufficient. No public ordering option is needed for this
+correction. Keep primary selection, conflict detection, acquisition order, and
+dependency-ready scheduling unchanged.
+
+## General Implementation Order
+
+1. Resolve the history-evidence decision and prepare focused public regression
+   cases from the audit. Establish expected semantic-path behavior using existing
+   contracts and shared path-matching helpers.
+2. Repair entry-state presence queries and `Default` (finding 1). This is a local
+   correction that can proceed independently of the larger history work.
+3. Repair final explicitness preservation (finding 2), retaining the shape evidence
+   needed by metadata updates. Coordinate this work with history construction so
+   the two fixes do not introduce competing definitions of path identity.
+4. Repair historical definitions and previous-origin attribution (finding 3),
+   then check their interaction with final metadata and explanation. A successful
+   presence fix alone does not establish correct history.
+5. Complete composition diagnostic ordering (finding 4) as an independent change;
+   it may proceed alongside the path work. Check rendering as well as the
+   structured error before considering it complete.
+6. Review the integrated result, run applicable development checks, and update
+   affected implementation documentation and finding status.
+
+These are high-level dependencies, not a requirement to combine unrelated fixes
+in one commit. The implementation plan should select concrete edits and test
+commands while keeping the four findings individually traceable.
+
+## Validation And Completion
+
+Follow the [testing policy](../policy/testing.md) and use the
+[development guide](../development.md#baseline-validation) for checks. Extend
+existing test owners rather than recreating full internal matrices at the facade.
+The focused cases should cover:
+
+- Both mapping-key/index mismatch directions beneath dynamic values, including
+  explicit paths and contextual parsed pointers. Check callback presence and
+  `Default` behavior, including the fixed entry view after working-value changes.
+- Both container-shape transitions during recorded resolution, a descendant
+  affected by an ancestor shape change, and a same-shape normalization control.
+  Check explicitness and recorded origins through final metadata and explanation.
+- Shape-changing CLI and resolution history, checking exact definitions,
+  `Previous`, and genuine ancestor replacement/removal. Include contextual
+  enumeration/query coverage and preserve existing inheritance and overlay cases;
+  add another case only where those integrations expose an unprotected risk.
+- A composition conflict whose acquisition and assembly orders differ, using the
+  audit fixture. Check both published diagnostics and rendered related-location
+  order, while retaining coverage for ordinary diagnostic canonicalization.
+
+Retain existing checks for invalid schema paths, callback lifetime, detached
+records, safe historical rendering, and default/null presence semantics. Test
+observable behavior rather than a particular private index or number of copies.
+No benchmark campaign, broader audit, or new release gate is part of this roadmap.
+
+Completion requires all four findings to have implementation and regression
+evidence, agreement at the affected public boundaries, and appropriate development
+checks. The audit's previous passing suite is not validation of these future
+changes. Scope here does not promise identical output from different inspection
+interfaces or a universal guarantee about unrelated path-processing code.
+
+## Open Decisions
+
+### OD-1. Where Should Shape-Specific History Evidence Live?
+
+**Recommendation:** retain semantic-shape evidence with the internal history
+events as they are built, using existing concrete paths and document evidence.
+Select applicable events and previous origins for each query before producing
+detached public `Change` records. The exact representation belongs in the
+implementation plan.
+
+This fits the existing stored-history owner and keeps metadata queries independent
+of explanation rendering. It also permits a focused correction where event
+construction currently loses shape. The cost is carrying and preserving evidence
+across several event producers; the design must keep it consistent with replay
+and avoid redundant retained trees. The amount of new state should follow what
+existing evidence cannot supply, not an assumed new event framework.
+
+**Viable alternative:** derive the relevant metadata history from existing
+shape-aware replay evidence, sharing a private selection operation with
+explanation where their semantics agree. This can reduce duplicated attribution
+logic and reuse evidence that already handles the reported transition correctly.
+However, metadata history includes default/definition and inheritance records
+whose presentation differs from explanation. This option needs a clear account
+of those differences, query cost, and evidence ownership; routing `History`
+through the public explanation representation would introduce unnecessary
+coupling.
+
+Choose between these approaches after a focused comparison of the affected event
+producers and replay coverage, before finalizing the implementation plan. The
+decision concerns private ownership and implementation cost, not public path or
+history semantics. No other significant product or architectural decision is
+open for this scope.

docs/roadmap/interfaces.md

@@ -5,6 +5,10 @@ Reviewed baseline: `01b08e76a3fecc22c4414437bfeaa07b4b7a85aa`.
 Status: four open findings. This is an audit record, not an implementation plan.
 No production code or repository tests were changed.
 
+The [interface remediation roadmap](interface-remediation.md) describes the
+proposed target, required changes, dependencies, and remaining design decision.
+This audit retains the findings and reproduction evidence.
+
 ## Scope and evidence
 
 Reviewed the root-owned public facade and its conversions, engine loading and