Adopt shared history ownership in the interface roadmap

docs/roadmap/interface-remediation.md

@@ -1,6 +1,7 @@
 # Interface Contract Remediation Roadmap
 
-Status: proposed target; implementation has not begun.
+Status: target settled, including shared history ownership; implementation has
+not begun.
 Planning baseline: `1f64673` (v0.10.2 implementation plus documentation changes).
 
 This roadmap covers the four findings in the [interface audit](interfaces.md).
@@ -16,6 +17,11 @@ interfaces under their existing contracts. The first three findings concern
 semantic paths that share a JSON Pointer spelling; the fourth concerns a
 composition claimant order that is lost during diagnostic canonicalization.
 
+Build one shared internal history model from which metadata history and
+explanation are derived. It will own the historical evidence and semantic
+attribution used by both views, replacing their independent reconstruction of
+that attribution. This is the selected resolution of OD-1.
+
 The [Path declarations](../../path.go), [metadata contract](../config.md#layered-metadata),
 [resolution contract](../config.md#recorded-application-resolution), and
 [strict-composition contract](../config.md#strict-document-composition) remain
@@ -31,11 +37,17 @@ as `/shape/0`. `ParsePath` deliberately leaves numeric tokens contextual, wherea
 shape-aware effective/historical path indexes and a frozen resolution-entry view,
 but some queries and metadata updates still use pointer spelling alone.
 
+History currently reconstructs definition and previous-origin records, while
+explanation replays source contributions, inheritance, decoding checkpoints, CLI
+assignments, and grouped resolution snapshots. Neither public representation
+contains everything needed by the other. The target is shared underlying evidence
+with distinct projections for each interface, preserving their existing behavior.
+
 | Finding | Current discrepancy | Target |
 | --- | --- | --- |
 | 1: resolution presence | `IsExplicit` and `Default` consult string-keyed presence after schema validation, which cannot determine actual shape beneath dynamic values. | Presence decisions use the matching semantic path in the fixed resolution-entry state. |
 | 2: effective explicitness | Resolution retains old field information when pointer spelling survives a container-shape change. | Input explicitness carries forward for the same semantic path; a newly introduced key or index does not inherit another path's explicitness. |
-| 3: history attribution | A historical-shape check admits the query, then `History` returns the entire textual-pointer bucket. | Exact definitions and previous origins belong to the queried semantic path, with applicable ancestor operations retained. |
+| 3: history attribution | A historical-shape check admits the query, then `History` returns the entire textual-pointer bucket; explanation selects evidence separately. | Both interfaces derive attribution from one shared history model, selecting exact definitions, previous origins, and applicable ancestor operations by semantic path. |
 | 4: claimant order | Composition constructs related locations in assembly order, then canonicalization sorts them by location. | Structured and rendered composition conflicts preserve the selected claimant order. |
 
 For example, if input contains `shape: {'0': old}` and recorded application
@@ -92,44 +104,133 @@ Use the existing before/after projection and entry metadata where possible.
 This work does not require tracking every transient mutation inside a callback;
 the existing net-change and coalesced-record model continues to apply.
 
-### 3. Retain Semantic Attribution In History
+### 3. Build A Shared Internal History Model
 
-Repair history event construction and selection across
+#### Evidence And Ownership
-[load_metadata.go](../../internal/engine/load_metadata.go),
-[metadata.go](../../internal/engine/metadata.go), and their inheritance-history
-integration. Finding 3's choice of internal evidence representation is discussed
-under [Open Decisions](#open-decisions).
 
-An explicit query should select definitions belonging to its semantic shape.
+Give historical evidence one private owner in the engine. Composition,
-Ancestor replacement or removal operations can still affect that path and must
+inheritance, decoding, CLI assignment, and resolution continue to perform their
-remain in its history where applicable. An ancestor operation's `Change.Path`
+existing work and supply evidence to that owner. The model does not drive a
-need not equal the queried descendant; this is existing behavior.
+second configuration load or repeat application callbacks.
 
-Retain enough evidence to distinguish the affected path from the operation's
+Organize the evidence into ordered groups with the phase boundaries required by
-reported path. Merely filtering current `Change.Path` values is insufficient:
+the existing inspection contracts:
-some construction paths recreate parsed tokens from strings, and ancestor events
-intentionally name another path. Review event construction for defaults, input
-definitions, precedence operations, inheritance, CLI, and resolution to identify
-where these distinctions can be lost. Change the affected owners without
-duplicating the entire loading lifecycle.
 
-Select or reconstruct `Previous` origins using the relevant semantic history.
+| Evidence group | Information to preserve |
-Review `historyWithInheritance` as well as stored events: filtering out a false
+| --- | --- |
-definition while retaining it as the previous origin would leave the attribution
+| Default baseline | Canonical typed application defaults or Go-zero values and their default attribution. |
-defect unresolved. Preserve chronology and genuine ancestor effects, including
+| Source contributions | Underlay, main-tier, and selected-overlay contributions in their existing order; exact definitions, composite origins, displaced definitions, raw policy-validated values, and sequence-position projections. |
-history of removed paths. A path that never existed does not acquire history
+| Inheritance | Parent transfers, child contributions including equal-valued replacements, controlling directives and their consumption, identity-field preservation, and contribution origins/dependencies. |
-solely because an ancestor changed.
+| Typed checkpoints | Pre-CLI and post-CLI typed projections and attribution needed to reconcile retained defaults and raw contributions with decoded values. |
+| CLI contributions | Explicit assignments, including equal-valued assignments, raw supplied representation, source attribution, and affected paths. |
+| Resolution | Original recorded operations, reasons, dependencies, and before/after evidence, with the grouping needed to explain overlapping records as net subtree changes. |
 
-Retain the existing contextual behavior of `ParsePath` across historical shapes
+The logical order remains baseline, source contributions, inheritance, pre-CLI
-and the one-entry-per-canonical-pointer enumeration of `HistoricalPaths`. Neither
+decoding reconciliation, CLI contributions, post-CLI typed reconciliation, then
-needs a public redesign. `History` and `Explain` should agree on which path a
+recorded resolution. The two typed checkpoints belong on their respective sides
-definition belongs to and what affected it; their different record forms do not
+of the CLI contribution phase.
-need to become identical lists.
 
-Reuse existing evidence where practical. Any additional retained evidence must
+The groups describe logical inspection chronology, not a transcript of every
-respect existing ownership, confidentiality, and applicable accounting; avoid
+physical mutation. Retain the existing net-change and coalescing rules for
-retaining extra document trees merely to identify a path's shape. This is not a
+resolution. Preserve superseded raw input that need not decode into the final
-new memory quota or a general history-storage redesign.
+application type; typed checkpoints do not replace that evidence.
+
+Use a small set of ordinary private structures and existing document/path
+mechanisms. Exact types, indexes, and storage layout belong in the implementation
+plan. Complete evidence means enough for the existing history, explanation, and
+internal attribution consumers; it does not add persistence, a public event API,
+or recording of transient application effects.
+
+#### Shared Selection And Attribution
+
+Retain concrete semantic-path evidence when it is available. A query over the
+model must distinguish exact definitions from operations affecting an ancestor
+or descendant, determine the relevant previous origins, and apply sequence
+contribution coordinates at their assigned positions.
+
+The affected semantic path and the operation's reported path are distinct facts.
+For example, a replacement at `/shape` can remove a mapping key and introduce a
+sequence index with the same pointer spelling. Preserve enough before/after shape
+evidence to answer each explicit query correctly. Merely filtering `Change.Path`
+after concrete segment kinds have been discarded is insufficient.
+
+Make this selection and attribution common to both projections, including
+inheritance integration and `Previous` reconstruction. A false definition must
+not survive as a previous origin after it has been excluded from the query's
+history. Preserve genuine ancestor replacement/removal effects and chronology.
+A path that never existed does not acquire history solely because an ancestor
+changed.
+
+Keep the contextual meaning of parsed numeric tokens across historical states
+and the one-entry-per-canonical-pointer enumeration of `HistoricalPaths`.
+Indexes and compact derived records may remain, but their attribution must come
+from the shared evidence. Metadata queries should not require full configuration
+replay or YAML rendering to obtain definition and origin records.
+
+#### Distinct Public Projections
+
+`Metadata.History` produces detached definition, inheritance, and resolution
+records from the model. `Explain` reconstructs and safely renders values from
+the same evidence, retaining its existing decoding and grouping behavior. Neither
+public API should be implemented by translating the other's public result.
+
+Preserve these intentional differences:
+
+| Case | Metadata history | Explanation |
+| --- | --- | --- |
+| Several strict contributions to a mapping | Composite definition/origin information. | Each contribution's effect on the mapping. |
+| Raw input differs from canonical typed values | Definition and attribution records. | The applicable decoding reconciliation transition. |
+| Overlapping resolution records | Specific recorded attribution under the existing coverage rules. | One net subtree event with combined reasons and dependencies. |
+| Equal-valued input or inheritance operations | Explicit definition and operation attribution remains visible. | Relevant operations remain visible even when before and after values are equal. |
+
+Agreement means consistent semantic identity and attribution, not identical
+record lists. Retain original resolution records as evidence from which net
+replay groups are derived; a grouped explanation alone loses information needed
+by metadata.
+
+#### Replace Existing Attribution Machinery
+
+The common model must replace independent historical attribution, rather than
+become another representation alongside it. The implementation plan should map
+the following owners to their replacements:
+
+| Current owner | Required change |
+| --- | --- |
+| History-building portions of `metadataFromDecoded`, `applyOverlayMetadata`, and `appendPostDecodeCLIHistory` in [load_metadata.go](../../internal/engine/load_metadata.go) and [override_context.go](../../internal/engine/override_context.go) | Produce or project shared evidence instead of independently reconstructing pointer-keyed histories. Retain effective-field construction where it is still needed. |
+| Previous-origin reconstruction and inheritance insertion in [inheritance_history.go](../../internal/engine/inheritance_history.go) | Use common semantic selection and attribution while preserving the existing inheritance projection rules. |
+| Relevance and chronology orchestration in [explain.go](../../internal/engine/explain.go) | Consume the shared model; retain useful value reconstruction and redaction helpers. |
+| Separate published inheritance evidence for metadata and explanation in [load_result.go](../../internal/engine/load_result.go) | Consolidate under the frozen history owner where independent copies are unnecessary; keep required output detachment. |
+| Original and coalesced resolution evidence in [history.go](../../internal/engine/history.go) | Own original records in the model and derive the replay grouping without losing specific attribution or maintaining a second independently interpreted history. |
+
+Existing source, sequence-combination, and inheritance evidence producers may
+remain useful. Their algorithms need not be rewritten merely to contribute to
+the model. Likewise, final effective fields, the frozen resolution-entry view,
+source records, and final configuration snapshots retain their responsibilities.
+Findings 1 and 2 still require explicit presence corrections.
+
+#### Other Consumers, Lifetime, And Resource Accounting
+
+Move historical dependency reads in
+[secret classification](../../internal/engine/secret.go) and historical location
+fallbacks in [override_context.go](../../internal/engine/override_context.go) to
+the shared evidence. Do not retain the old history machinery solely for those
+readers. Preserve existing sensitivity propagation and historical classification;
+correcting attribution does not authorize narrowing redaction for paths that
+share a pointer spelling.
+
+Evidence must be available at the lifecycle stages where these internal readers
+need it, including diagnostic attribution before publication. Keep the callback's
+entry metadata fixed while later resolution evidence is added. Finalize the
+model for published queries without introducing lazy mutation of shared state.
+Private frozen evidence may be shared by the two views while their returned
+records remain detached under the existing ownership contract.
+
+Reuse retained trees and existing lookup machinery where practical. Review
+retained-history accounting against the resulting ownership graph, including
+typed checkpoints and inheritance/resolution evidence. The model does not require
+a full document copy per event, a new quota, or a new concurrency guarantee.
+Reduced duplication is a design benefit to evaluate, not a promised memory or
+speed improvement.
 
 ### 4. Preserve Composition Claimant Order Through Diagnostics
 
@@ -152,17 +253,22 @@ dependency-ready scheduling unchanged.
 
 ## General Implementation Order
 
-1. Resolve the history-evidence decision and prepare focused public regression
+1. Prepare the concrete shared-evidence design and focused regression cases.
-   cases from the audit. Establish expected semantic-path behavior using existing
+   Map existing producers and consumers to the new owner, distinguish common
-   contracts and shared path-matching helpers.
+   attribution from presentation rules, and identify the old paths to remove.
+   Select representations and indexes that preserve existing query and lifetime
+   needs without adding public contracts.
 2. Repair entry-state presence queries and `Default` (finding 1). This is a local
    correction that can proceed independently of the larger history work.
 3. Repair final explicitness preservation (finding 2), retaining the shape evidence
    needed by metadata updates. Coordinate this work with history construction so
    the two fixes do not introduce competing definitions of path identity.
-4. Repair historical definitions and previous-origin attribution (finding 3),
+4. Build the shared model and migrate metadata history, explanation, sensitivity
-   then check their interaction with final metadata and explanation. A successful
+   dependencies, and diagnostic-location readers. Preserve phase evidence and
-   presence fix alone does not establish correct history.
+   derive both public projections through common semantic attribution. Remove
+   the superseded reconstruction paths and reconcile retained-history accounting.
+   Verify finding 3 and its interaction with final presence metadata; repairing
+   pointer buckets alone does not complete the selected architecture.
 5. Complete composition diagnostic ordering (finding 4) as an independent change;
    it may proceed alongside the path work. Check rendering as well as the
    structured error before considering it complete.
@@ -190,6 +296,16 @@ The focused cases should cover:
   `Previous`, and genuine ancestor replacement/removal. Include contextual
   enumeration/query coverage and preserve existing inheritance and overlay cases;
   add another case only where those integrations expose an unprotected risk.
+- Shared-model projections across composite mapping origins, typed decoding and
+  default reconciliation, equal-valued writes, inheritance controls/transfers,
+  mapped sequence replacements and removed descendants, and overlapping
+  resolution records. Reuse existing cases where they protect those semantics;
+  check the intentional differences between the public projections as well as
+  agreement on attribution.
+- Historical sensitivity/dependency propagation, diagnostic source-location
+  fallback, detached returned records, and complete retained-history accounting
+  under the new ownership graph. Use small controlled budget cases and existing
+  disclosure fixtures rather than memory or timing thresholds.
 - A composition conflict whose acquisition and assembly orders differ, using the
   audit fixture. Check both published diagnostics and rendered related-location
   order, while retaining coverage for ordinary diagnostic canonicalization.
@@ -200,40 +316,31 @@ observable behavior rather than a particular private index or number of copies.
 No benchmark campaign, broader audit, or new release gate is part of this roadmap.
 
 Completion requires all four findings to have implementation and regression
-evidence, agreement at the affected public boundaries, and appropriate development
+evidence, appropriate development checks, and both history interfaces deriving
-checks. The audit's previous passing suite is not validation of these future
+semantic attribution from the shared model. Review the final producer/consumer
-changes. Scope here does not promise identical output from different inspection
+map to confirm superseded independent reconstruction has been removed and
-interfaces or a universal guarantee about unrelated path-processing code.
+internal attribution readers have migrated. Passing the four defect regressions
+while retaining the old parallel attribution machinery is not completion.
 
-## Open Decisions
+The audit's previous passing suite is not validation of these future changes.
+Scope here does not promise identical output from different inspection interfaces
+or a universal guarantee about unrelated path-processing code.
 
-### OD-1. Where Should Shape-Specific History Evidence Live?
+## Decision Status
 
-**Recommendation:** retain semantic-shape evidence with the internal history
+OD-1 is settled: build one shared internal history model with distinct metadata
-events as they are built, using existing concrete paths and document evidence.
+and explanation projections. The design investigation found that both views
-Select applicable events and previous origins for each query before producing
+require evidence the other's public representation does not preserve. Shared
-detached public `Change` records. The exact representation belongs in the
+ownership can replace independent attribution and redundant retained evidence,
-implementation plan.
+while keeping the current lifecycle and public contracts.
 
-This fits the existing stored-history owner and keeps metadata queries independent
+The narrower repair of stored history events would leave the separate replay
-of explanation rendering. It also permits a focused correction where event
+attribution owner in place. Promoting current explanation output to authoritative
-construction currently loses shape. The cost is carrying and preserving evidence
+history would lose composite-definition and specific resolution information.
-across several event producers; the design must keep it consistent with replay
+The selected common underlying model addresses both concerns. This accepts a
-and avoid redundant retained trees. The amount of new state should follow what
+larger internal refactor than a local shape-filtering fix.
-existing evidence cannot supply, not an assumed new event framework.
 
-**Viable alternative:** derive the relevant metadata history from existing
+No significant open decision remains at roadmap level. Private representation,
-shape-aware replay evidence, sharing a private selection operation with
+indexing, and helper choices belong in the implementation plan. This roadmap
-explanation where their semantics agree. This can reduce duplicated attribution
+records future implementation ownership; it does not revise accepted ADRs or
-logic and reuse evidence that already handles the reported transition correctly.
+declare the model implemented.
-However, metadata history includes default/definition and inheritance records
-whose presentation differs from explanation. This option needs a clear account
-of those differences, query cost, and evidence ownership; routing `History`
-through the public explanation representation would introduce unnecessary
-coupling.
-
-Choose between these approaches after a focused comparison of the affected event
-producers and replay coverage, before finalizing the implementation plan. The
-decision concerns private ownership and implementation cost, not public path or
-history semantics. No other significant product or architectural decision is
-open for this scope.

docs/roadmap/interfaces.md

@@ -6,7 +6,8 @@ Status: four open findings. This is an audit record, not an implementation plan.
 No production code or repository tests were changed.
 
 The [interface remediation roadmap](interface-remediation.md) describes the
-proposed target, required changes, dependencies, and remaining design decision.
+settled target, required changes, and dependencies, including a shared internal
+history model for metadata history and explanation. Implementation remains pending.
 This audit retains the findings and reproduction evidence.
 
 ## Scope and evidence