ConfigKit/docs/roadmap/interface-remediation.md

Interface Contract Remediation Roadmap

Status: target settled, including shared history ownership; implementation has not begun. Planning baseline: 1f64673 (v0.10.2 implementation plus documentation changes).

This roadmap covers the four findings in the interface audit. The audit owns reproductions, evidence, and finding numbers. This document owns the combined target, required changes, and general dependency order. A subsequent implementation plan should own concrete stages, private implementation choices, validation commands, and progress.

Purpose And Scope

Restore agreement between ConfigKit's presence, metadata, history, and diagnostic interfaces under their existing contracts. The first three findings concern semantic paths that share a JSON Pointer spelling; the fourth concerns a composition claimant order that is lost during diagnostic canonicalization.

Build one shared internal history model from which metadata history and explanation are derived. It will own the historical evidence and semantic attribution used by both views, replacing their independent reconstruction of that attribution. This is the selected resolution of OD-1.

The Path declarations, metadata contract, resolution contract, and strict-composition contract remain authoritative. This work changes the implementation of those contracts without selecting a new public API, path syntax, precedence rule, or configuration restriction. The existing root-owned facade and engine responsibilities remain.

Current Baseline And Targeted End State

An explicit mapping-key path and an explicit sequence-index path can both render as /shape/0. ParsePath deliberately leaves numeric tokens contextual, whereas Keys and Index preserve the caller's explicit choice. The engine already has shape-aware effective/historical path indexes and a frozen resolution-entry view, but some queries and metadata updates still use pointer spelling alone.

History currently reconstructs definition and previous-origin records, while explanation replays source contributions, inheritance, decoding checkpoints, CLI assignments, and grouped resolution snapshots. Neither public representation contains everything needed by the other. The target is shared underlying evidence with distinct projections for each interface, preserving their existing behavior.

Finding	Current discrepancy	Target
1: resolution presence	IsExplicit and Default consult string-keyed presence after schema validation, which cannot determine actual shape beneath dynamic values.	Presence decisions use the matching semantic path in the fixed resolution-entry state.
2: effective explicitness	Resolution retains old field information when pointer spelling survives a container-shape change.	Input explicitness carries forward for the same semantic path; a newly introduced key or index does not inherit another path's explicitness.
3: history attribution	A historical-shape check admits the query, then History returns the entire textual-pointer bucket; explanation selects evidence separately.	Both interfaces derive attribution from one shared history model, selecting exact definitions, previous origins, and applicable ancestor operations by semantic path.
4: claimant order	Composition constructs related locations in assembly order, then canonicalization sorts them by location.	Structured and rendered composition conflicts preserve the selected claimant order.

For example, if input contains shape: {'0': old} and recorded application resolution changes shape to a sequence, the old key remains an input path in the callback's entry view. The new index is effective in final metadata but was not explicitly supplied by that input. Its history must not present the old key's definition as its own. This distinction concerns container shape, not identity tracking for records that move within a sequence.

Required Changes

1. Use Entry-State Shape For Resolution Presence

In resolution.go, align IsExplicit and the explicitness decision in Default with the frozen, shape-aware entry metadata already used by InputInfo in load_metadata.go.

Reuse the existing semantic-path matching machinery. A shared private lookup can avoid divergent presence decisions without requiring every public query to have identical error behavior. In particular:

• Preserve IsExplicit's schema-path validation and its existing diagnostics. InputInfo remains a query that returns false for missing or mismatched paths without those diagnostics.
• Preserve Default's helper-target validation, assignment, recording, and final helper-value checks. Correct its presence gate; do not bypass the other checks.
• Use entry-state shape even after application code changes the working value. Explicit false, zero, empty, and null retain their existing presence meaning.
• Keep callback lifetime and detached metadata behavior intact.

A shape mismatch beneath a schema-valid dynamic path must not become explicit merely because its pointer spelling matches an input path. Contextually parsed numeric tokens continue to resolve against the entry state's actual containers.

2. Preserve Explicitness By Semantic Path During Resolution

Update resolution metadata application and field refresh in load_metadata.go to compare the relevant entry and final shapes before reusing presence information.

The comparison must account for container kinds along the full path. A mapping key becoming a sequence index at an ancestor can change a descendant's semantic path even when its last segment and full pointer spelling stay unchanged.

Keep explicitness for matching surviving paths, including descendants covered by an ancestor normalization record. New semantic paths introduced by resolution remain non-explicit, and retain the applicable recorded resolution origin. Removed paths leave effective metadata while remaining available to historical queries under the existing contract. Do not clear every descendant's presence just because its parent was normalized.

Use the existing before/after projection and entry metadata where possible. This work does not require tracking every transient mutation inside a callback; the existing net-change and coalesced-record model continues to apply.

3. Build A Shared Internal History Model

Evidence And Ownership

Give historical evidence one private owner in the engine. Composition, inheritance, decoding, CLI assignment, and resolution continue to perform their existing work and supply evidence to that owner. The model does not drive a second configuration load or repeat application callbacks.

Organize the evidence into ordered groups with the phase boundaries required by the existing inspection contracts:

Evidence group	Information to preserve
Default baseline	Canonical typed application defaults or Go-zero values and their default attribution.
Source contributions	Underlay, main-tier, and selected-overlay contributions in their existing order; exact definitions, composite origins, displaced definitions, raw policy-validated values, and sequence-position projections.
Inheritance	Parent transfers, child contributions including equal-valued replacements, controlling directives and their consumption, identity-field preservation, and contribution origins/dependencies.
Typed checkpoints	Pre-CLI and post-CLI typed projections and attribution needed to reconcile retained defaults and raw contributions with decoded values.
CLI contributions	Explicit assignments, including equal-valued assignments, raw supplied representation, source attribution, and affected paths.
Resolution	Original recorded operations, reasons, dependencies, and before/after evidence, with the grouping needed to explain overlapping records as net subtree changes.

The logical order remains baseline, source contributions, inheritance, pre-CLI decoding reconciliation, CLI contributions, post-CLI typed reconciliation, then recorded resolution. The two typed checkpoints belong on their respective sides of the CLI contribution phase.

The groups describe logical inspection chronology, not a transcript of every physical mutation. Retain the existing net-change and coalescing rules for resolution. Preserve superseded raw input that need not decode into the final application type; typed checkpoints do not replace that evidence.

Use a small set of ordinary private structures and existing document/path mechanisms. Exact types, indexes, and storage layout belong in the implementation plan. Complete evidence means enough for the existing history, explanation, and internal attribution consumers; it does not add persistence, a public event API, or recording of transient application effects.

Shared Selection And Attribution

Retain concrete semantic-path evidence when it is available. A query over the model must distinguish exact definitions from operations affecting an ancestor or descendant, determine the relevant previous origins, and apply sequence contribution coordinates at their assigned positions.

The affected semantic path and the operation's reported path are distinct facts. For example, a replacement at /shape can remove a mapping key and introduce a sequence index with the same pointer spelling. Preserve enough before/after shape evidence to answer each explicit query correctly. Merely filtering Change.Path after concrete segment kinds have been discarded is insufficient.

Make this selection and attribution common to both projections, including inheritance integration and Previous reconstruction. A false definition must not survive as a previous origin after it has been excluded from the query's history. Preserve genuine ancestor replacement/removal effects and chronology. A path that never existed does not acquire history solely because an ancestor changed.

Keep the contextual meaning of parsed numeric tokens across historical states and the one-entry-per-canonical-pointer enumeration of HistoricalPaths. Indexes and compact derived records may remain, but their attribution must come from the shared evidence. Metadata queries should not require full configuration replay or YAML rendering to obtain definition and origin records.

Distinct Public Projections

Metadata.History produces detached definition, inheritance, and resolution records from the model. Explain reconstructs and safely renders values from the same evidence, retaining its existing decoding and grouping behavior. Neither public API should be implemented by translating the other's public result.

Preserve these intentional differences:

Case	Metadata history	Explanation
Several strict contributions to a mapping	Composite definition/origin information.	Each contribution's effect on the mapping.
Raw input differs from canonical typed values	Definition and attribution records.	The applicable decoding reconciliation transition.
Overlapping resolution records	Specific recorded attribution under the existing coverage rules.	One net subtree event with combined reasons and dependencies.
Equal-valued input or inheritance operations	Explicit definition and operation attribution remains visible.	Relevant operations remain visible even when before and after values are equal.

Agreement means consistent semantic identity and attribution, not identical record lists. Retain original resolution records as evidence from which net replay groups are derived; a grouped explanation alone loses information needed by metadata.

Replace Existing Attribution Machinery

The common model must replace independent historical attribution, rather than become another representation alongside it. The implementation plan should map the following owners to their replacements:

Current owner	Required change
History-building portions of metadataFromDecoded, applyOverlayMetadata, and appendPostDecodeCLIHistory in load_metadata.go and override_context.go	Produce or project shared evidence instead of independently reconstructing pointer-keyed histories. Retain effective-field construction where it is still needed.
Previous-origin reconstruction and inheritance insertion in inheritance_history.go	Use common semantic selection and attribution while preserving the existing inheritance projection rules.
Relevance and chronology orchestration in explain.go	Consume the shared model; retain useful value reconstruction and redaction helpers.
Separate published inheritance evidence for metadata and explanation in load_result.go	Consolidate under the frozen history owner where independent copies are unnecessary; keep required output detachment.
Original and coalesced resolution evidence in history.go	Own original records in the model and derive the replay grouping without losing specific attribution or maintaining a second independently interpreted history.

Existing source, sequence-combination, and inheritance evidence producers may remain useful. Their algorithms need not be rewritten merely to contribute to the model. Likewise, final effective fields, the frozen resolution-entry view, source records, and final configuration snapshots retain their responsibilities. Findings 1 and 2 still require explicit presence corrections.

Other Consumers, Lifetime, And Resource Accounting

Move historical dependency reads in secret classification and historical location fallbacks in override_context.go to the shared evidence. Do not retain the old history machinery solely for those readers. Preserve existing sensitivity propagation and historical classification; correcting attribution does not authorize narrowing redaction for paths that share a pointer spelling.

Evidence must be available at the lifecycle stages where these internal readers need it, including diagnostic attribution before publication. Keep the callback's entry metadata fixed while later resolution evidence is added. Finalize the model for published queries without introducing lazy mutation of shared state. Private frozen evidence may be shared by the two views while their returned records remain detached under the existing ownership contract.

Reuse retained trees and existing lookup machinery where practical. Review retained-history accounting against the resulting ownership graph, including typed checkpoints and inheritance/resolution evidence. The model does not require a full document copy per event, a new quota, or a new concurrency guarantee. Reduced duplication is a design benefit to evaluate, not a promised memory or speed improvement.

4. Preserve Composition Claimant Order Through Diagnostics

Keep the order produced by compositionConflictDiagnostic in compose.go through accumulation, LoadError, facade conversion, and WriteDiagnostics in diagnostic.go.

Prefer a narrow internal canonicalization rule based on the diagnostic semantics already carried across these boundaries. Identify the composition diagnostic producers governed by the claimant-order contract rather than changing all related-location lists. Preserve canonical ordering of diagnostic records and the existing sorting of related locations for other diagnostics.

The root renderer converts public diagnostic records back to engine records and canonicalizes them again. An engine-only marker discarded by facade conversion would therefore be insufficient. No public ordering option is needed for this correction. Keep primary selection, conflict detection, acquisition order, and dependency-ready scheduling unchanged.

General Implementation Order

1. Prepare the concrete shared-evidence design and focused regression cases. Map existing producers and consumers to the new owner, distinguish common attribution from presentation rules, and identify the old paths to remove. Select representations and indexes that preserve existing query and lifetime needs without adding public contracts.
2. Repair entry-state presence queries and Default (finding 1). This is a local correction that can proceed independently of the larger history work.
3. Repair final explicitness preservation (finding 2), retaining the shape evidence needed by metadata updates. Coordinate this work with history construction so the two fixes do not introduce competing definitions of path identity.
4. Build the shared model and migrate metadata history, explanation, sensitivity dependencies, and diagnostic-location readers. Preserve phase evidence and derive both public projections through common semantic attribution. Remove the superseded reconstruction paths and reconcile retained-history accounting. Verify finding 3 and its interaction with final presence metadata; repairing pointer buckets alone does not complete the selected architecture.
5. Complete composition diagnostic ordering (finding 4) as an independent change; it may proceed alongside the path work. Check rendering as well as the structured error before considering it complete.
6. Review the integrated result, run applicable development checks, and update affected implementation documentation and finding status.

These are high-level dependencies, not a requirement to combine unrelated fixes in one commit. The implementation plan should select concrete edits and test commands while keeping the four findings individually traceable.

Validation And Completion

Follow the testing policy and use the development guide for checks. Extend existing test owners rather than recreating full internal matrices at the facade. The focused cases should cover:

• Both mapping-key/index mismatch directions beneath dynamic values, including explicit paths and contextual parsed pointers. Check callback presence and Default behavior, including the fixed entry view after working-value changes.
• Both container-shape transitions during recorded resolution, a descendant affected by an ancestor shape change, and a same-shape normalization control. Check explicitness and recorded origins through final metadata and explanation.
• Shape-changing CLI and resolution history, checking exact definitions, Previous, and genuine ancestor replacement/removal. Include contextual enumeration/query coverage and preserve existing inheritance and overlay cases; add another case only where those integrations expose an unprotected risk.
• Shared-model projections across composite mapping origins, typed decoding and default reconciliation, equal-valued writes, inheritance controls/transfers, mapped sequence replacements and removed descendants, and overlapping resolution records. Reuse existing cases where they protect those semantics; check the intentional differences between the public projections as well as agreement on attribution.
• Historical sensitivity/dependency propagation, diagnostic source-location fallback, detached returned records, and complete retained-history accounting under the new ownership graph. Use small controlled budget cases and existing disclosure fixtures rather than memory or timing thresholds.
• A composition conflict whose acquisition and assembly orders differ, using the audit fixture. Check both published diagnostics and rendered related-location order, while retaining coverage for ordinary diagnostic canonicalization.

Retain existing checks for invalid schema paths, callback lifetime, detached records, safe historical rendering, and default/null presence semantics. Test observable behavior rather than a particular private index or number of copies. No benchmark campaign, broader audit, or new release gate is part of this roadmap.

Completion requires all four findings to have implementation and regression evidence, appropriate development checks, and both history interfaces deriving semantic attribution from the shared model. Review the final producer/consumer map to confirm superseded independent reconstruction has been removed and internal attribution readers have migrated. Passing the four defect regressions while retaining the old parallel attribution machinery is not completion.

The audit's previous passing suite is not validation of these future changes. Scope here does not promise identical output from different inspection interfaces or a universal guarantee about unrelated path-processing code.

Decision Status

OD-1 is settled: build one shared internal history model with distinct metadata and explanation projections. The design investigation found that both views require evidence the other's public representation does not preserve. Shared ownership can replace independent attribution and redundant retained evidence, while keeping the current lifecycle and public contracts.

The narrower repair of stored history events would leave the separate replay attribution owner in place. Promoting current explanation output to authoritative history would lose composite-definition and specific resolution information. The selected common underlying model addresses both concerns. This accepts a larger internal refactor than a local shape-filtering fix.

No significant open decision remains at roadmap level. Private representation, indexing, and helper choices belong in the implementation plan. This roadmap records future implementation ownership; it does not revise accepted ADRs or declare the model implemented.