From f9ec66e8674545a0aac2e5456d5ade42376507b4 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 26 Oct 2026 12:36:17 +0000 Subject: [PATCH] Adopt shared history ownership in the interface roadmap --- docs/roadmap/interface-remediation.md | 251 ++++++++++++++++++-------- docs/roadmap/interfaces.md | 3 +- 2 files changed, 181 insertions(+), 73 deletions(-) diff --git a/docs/roadmap/interface-remediation.md b/docs/roadmap/interface-remediation.md index 1668b85..0617568 100644 --- a/docs/roadmap/interface-remediation.md +++ b/docs/roadmap/interface-remediation.md @@ -1,6 +1,7 @@ # Interface Contract Remediation Roadmap -Status: proposed target; implementation has not begun. +Status: target settled, including shared history ownership; implementation has +not begun. Planning baseline: `1f64673` (v0.10.2 implementation plus documentation changes). This roadmap covers the four findings in the [interface audit](interfaces.md). @@ -16,6 +17,11 @@ interfaces under their existing contracts. The first three findings concern semantic paths that share a JSON Pointer spelling; the fourth concerns a composition claimant order that is lost during diagnostic canonicalization. +Build one shared internal history model from which metadata history and +explanation are derived. It will own the historical evidence and semantic +attribution used by both views, replacing their independent reconstruction of +that attribution. This is the selected resolution of OD-1. + The [Path declarations](../../path.go), [metadata contract](../config.md#layered-metadata), [resolution contract](../config.md#recorded-application-resolution), and [strict-composition contract](../config.md#strict-document-composition) remain @@ -31,11 +37,17 @@ as `/shape/0`. `ParsePath` deliberately leaves numeric tokens contextual, wherea shape-aware effective/historical path indexes and a frozen resolution-entry view, but some queries and metadata updates still use pointer spelling alone. +History currently reconstructs definition and previous-origin records, while +explanation replays source contributions, inheritance, decoding checkpoints, CLI +assignments, and grouped resolution snapshots. Neither public representation +contains everything needed by the other. The target is shared underlying evidence +with distinct projections for each interface, preserving their existing behavior. + | Finding | Current discrepancy | Target | | --- | --- | --- | | 1: resolution presence | `IsExplicit` and `Default` consult string-keyed presence after schema validation, which cannot determine actual shape beneath dynamic values. | Presence decisions use the matching semantic path in the fixed resolution-entry state. | | 2: effective explicitness | Resolution retains old field information when pointer spelling survives a container-shape change. | Input explicitness carries forward for the same semantic path; a newly introduced key or index does not inherit another path's explicitness. | -| 3: history attribution | A historical-shape check admits the query, then `History` returns the entire textual-pointer bucket. | Exact definitions and previous origins belong to the queried semantic path, with applicable ancestor operations retained. | +| 3: history attribution | A historical-shape check admits the query, then `History` returns the entire textual-pointer bucket; explanation selects evidence separately. | Both interfaces derive attribution from one shared history model, selecting exact definitions, previous origins, and applicable ancestor operations by semantic path. | | 4: claimant order | Composition constructs related locations in assembly order, then canonicalization sorts them by location. | Structured and rendered composition conflicts preserve the selected claimant order. | For example, if input contains `shape: {'0': old}` and recorded application @@ -92,44 +104,133 @@ Use the existing before/after projection and entry metadata where possible. This work does not require tracking every transient mutation inside a callback; the existing net-change and coalesced-record model continues to apply. -### 3. Retain Semantic Attribution In History +### 3. Build A Shared Internal History Model -Repair history event construction and selection across -[load_metadata.go](../../internal/engine/load_metadata.go), -[metadata.go](../../internal/engine/metadata.go), and their inheritance-history -integration. Finding 3's choice of internal evidence representation is discussed -under [Open Decisions](#open-decisions). +#### Evidence And Ownership -An explicit query should select definitions belonging to its semantic shape. -Ancestor replacement or removal operations can still affect that path and must -remain in its history where applicable. An ancestor operation's `Change.Path` -need not equal the queried descendant; this is existing behavior. +Give historical evidence one private owner in the engine. Composition, +inheritance, decoding, CLI assignment, and resolution continue to perform their +existing work and supply evidence to that owner. The model does not drive a +second configuration load or repeat application callbacks. -Retain enough evidence to distinguish the affected path from the operation's -reported path. Merely filtering current `Change.Path` values is insufficient: -some construction paths recreate parsed tokens from strings, and ancestor events -intentionally name another path. Review event construction for defaults, input -definitions, precedence operations, inheritance, CLI, and resolution to identify -where these distinctions can be lost. Change the affected owners without -duplicating the entire loading lifecycle. +Organize the evidence into ordered groups with the phase boundaries required by +the existing inspection contracts: -Select or reconstruct `Previous` origins using the relevant semantic history. -Review `historyWithInheritance` as well as stored events: filtering out a false -definition while retaining it as the previous origin would leave the attribution -defect unresolved. Preserve chronology and genuine ancestor effects, including -history of removed paths. A path that never existed does not acquire history -solely because an ancestor changed. +| Evidence group | Information to preserve | +| --- | --- | +| Default baseline | Canonical typed application defaults or Go-zero values and their default attribution. | +| Source contributions | Underlay, main-tier, and selected-overlay contributions in their existing order; exact definitions, composite origins, displaced definitions, raw policy-validated values, and sequence-position projections. | +| Inheritance | Parent transfers, child contributions including equal-valued replacements, controlling directives and their consumption, identity-field preservation, and contribution origins/dependencies. | +| Typed checkpoints | Pre-CLI and post-CLI typed projections and attribution needed to reconcile retained defaults and raw contributions with decoded values. | +| CLI contributions | Explicit assignments, including equal-valued assignments, raw supplied representation, source attribution, and affected paths. | +| Resolution | Original recorded operations, reasons, dependencies, and before/after evidence, with the grouping needed to explain overlapping records as net subtree changes. | -Retain the existing contextual behavior of `ParsePath` across historical shapes -and the one-entry-per-canonical-pointer enumeration of `HistoricalPaths`. Neither -needs a public redesign. `History` and `Explain` should agree on which path a -definition belongs to and what affected it; their different record forms do not -need to become identical lists. +The logical order remains baseline, source contributions, inheritance, pre-CLI +decoding reconciliation, CLI contributions, post-CLI typed reconciliation, then +recorded resolution. The two typed checkpoints belong on their respective sides +of the CLI contribution phase. -Reuse existing evidence where practical. Any additional retained evidence must -respect existing ownership, confidentiality, and applicable accounting; avoid -retaining extra document trees merely to identify a path's shape. This is not a -new memory quota or a general history-storage redesign. +The groups describe logical inspection chronology, not a transcript of every +physical mutation. Retain the existing net-change and coalescing rules for +resolution. Preserve superseded raw input that need not decode into the final +application type; typed checkpoints do not replace that evidence. + +Use a small set of ordinary private structures and existing document/path +mechanisms. Exact types, indexes, and storage layout belong in the implementation +plan. Complete evidence means enough for the existing history, explanation, and +internal attribution consumers; it does not add persistence, a public event API, +or recording of transient application effects. + +#### Shared Selection And Attribution + +Retain concrete semantic-path evidence when it is available. A query over the +model must distinguish exact definitions from operations affecting an ancestor +or descendant, determine the relevant previous origins, and apply sequence +contribution coordinates at their assigned positions. + +The affected semantic path and the operation's reported path are distinct facts. +For example, a replacement at `/shape` can remove a mapping key and introduce a +sequence index with the same pointer spelling. Preserve enough before/after shape +evidence to answer each explicit query correctly. Merely filtering `Change.Path` +after concrete segment kinds have been discarded is insufficient. + +Make this selection and attribution common to both projections, including +inheritance integration and `Previous` reconstruction. A false definition must +not survive as a previous origin after it has been excluded from the query's +history. Preserve genuine ancestor replacement/removal effects and chronology. +A path that never existed does not acquire history solely because an ancestor +changed. + +Keep the contextual meaning of parsed numeric tokens across historical states +and the one-entry-per-canonical-pointer enumeration of `HistoricalPaths`. +Indexes and compact derived records may remain, but their attribution must come +from the shared evidence. Metadata queries should not require full configuration +replay or YAML rendering to obtain definition and origin records. + +#### Distinct Public Projections + +`Metadata.History` produces detached definition, inheritance, and resolution +records from the model. `Explain` reconstructs and safely renders values from +the same evidence, retaining its existing decoding and grouping behavior. Neither +public API should be implemented by translating the other's public result. + +Preserve these intentional differences: + +| Case | Metadata history | Explanation | +| --- | --- | --- | +| Several strict contributions to a mapping | Composite definition/origin information. | Each contribution's effect on the mapping. | +| Raw input differs from canonical typed values | Definition and attribution records. | The applicable decoding reconciliation transition. | +| Overlapping resolution records | Specific recorded attribution under the existing coverage rules. | One net subtree event with combined reasons and dependencies. | +| Equal-valued input or inheritance operations | Explicit definition and operation attribution remains visible. | Relevant operations remain visible even when before and after values are equal. | + +Agreement means consistent semantic identity and attribution, not identical +record lists. Retain original resolution records as evidence from which net +replay groups are derived; a grouped explanation alone loses information needed +by metadata. + +#### Replace Existing Attribution Machinery + +The common model must replace independent historical attribution, rather than +become another representation alongside it. The implementation plan should map +the following owners to their replacements: + +| Current owner | Required change | +| --- | --- | +| History-building portions of `metadataFromDecoded`, `applyOverlayMetadata`, and `appendPostDecodeCLIHistory` in [load_metadata.go](../../internal/engine/load_metadata.go) and [override_context.go](../../internal/engine/override_context.go) | Produce or project shared evidence instead of independently reconstructing pointer-keyed histories. Retain effective-field construction where it is still needed. | +| Previous-origin reconstruction and inheritance insertion in [inheritance_history.go](../../internal/engine/inheritance_history.go) | Use common semantic selection and attribution while preserving the existing inheritance projection rules. | +| Relevance and chronology orchestration in [explain.go](../../internal/engine/explain.go) | Consume the shared model; retain useful value reconstruction and redaction helpers. | +| Separate published inheritance evidence for metadata and explanation in [load_result.go](../../internal/engine/load_result.go) | Consolidate under the frozen history owner where independent copies are unnecessary; keep required output detachment. | +| Original and coalesced resolution evidence in [history.go](../../internal/engine/history.go) | Own original records in the model and derive the replay grouping without losing specific attribution or maintaining a second independently interpreted history. | + +Existing source, sequence-combination, and inheritance evidence producers may +remain useful. Their algorithms need not be rewritten merely to contribute to +the model. Likewise, final effective fields, the frozen resolution-entry view, +source records, and final configuration snapshots retain their responsibilities. +Findings 1 and 2 still require explicit presence corrections. + +#### Other Consumers, Lifetime, And Resource Accounting + +Move historical dependency reads in +[secret classification](../../internal/engine/secret.go) and historical location +fallbacks in [override_context.go](../../internal/engine/override_context.go) to +the shared evidence. Do not retain the old history machinery solely for those +readers. Preserve existing sensitivity propagation and historical classification; +correcting attribution does not authorize narrowing redaction for paths that +share a pointer spelling. + +Evidence must be available at the lifecycle stages where these internal readers +need it, including diagnostic attribution before publication. Keep the callback's +entry metadata fixed while later resolution evidence is added. Finalize the +model for published queries without introducing lazy mutation of shared state. +Private frozen evidence may be shared by the two views while their returned +records remain detached under the existing ownership contract. + +Reuse retained trees and existing lookup machinery where practical. Review +retained-history accounting against the resulting ownership graph, including +typed checkpoints and inheritance/resolution evidence. The model does not require +a full document copy per event, a new quota, or a new concurrency guarantee. +Reduced duplication is a design benefit to evaluate, not a promised memory or +speed improvement. ### 4. Preserve Composition Claimant Order Through Diagnostics @@ -152,17 +253,22 @@ dependency-ready scheduling unchanged. ## General Implementation Order -1. Resolve the history-evidence decision and prepare focused public regression - cases from the audit. Establish expected semantic-path behavior using existing - contracts and shared path-matching helpers. +1. Prepare the concrete shared-evidence design and focused regression cases. + Map existing producers and consumers to the new owner, distinguish common + attribution from presentation rules, and identify the old paths to remove. + Select representations and indexes that preserve existing query and lifetime + needs without adding public contracts. 2. Repair entry-state presence queries and `Default` (finding 1). This is a local correction that can proceed independently of the larger history work. 3. Repair final explicitness preservation (finding 2), retaining the shape evidence needed by metadata updates. Coordinate this work with history construction so the two fixes do not introduce competing definitions of path identity. -4. Repair historical definitions and previous-origin attribution (finding 3), - then check their interaction with final metadata and explanation. A successful - presence fix alone does not establish correct history. +4. Build the shared model and migrate metadata history, explanation, sensitivity + dependencies, and diagnostic-location readers. Preserve phase evidence and + derive both public projections through common semantic attribution. Remove + the superseded reconstruction paths and reconcile retained-history accounting. + Verify finding 3 and its interaction with final presence metadata; repairing + pointer buckets alone does not complete the selected architecture. 5. Complete composition diagnostic ordering (finding 4) as an independent change; it may proceed alongside the path work. Check rendering as well as the structured error before considering it complete. @@ -190,6 +296,16 @@ The focused cases should cover: `Previous`, and genuine ancestor replacement/removal. Include contextual enumeration/query coverage and preserve existing inheritance and overlay cases; add another case only where those integrations expose an unprotected risk. +- Shared-model projections across composite mapping origins, typed decoding and + default reconciliation, equal-valued writes, inheritance controls/transfers, + mapped sequence replacements and removed descendants, and overlapping + resolution records. Reuse existing cases where they protect those semantics; + check the intentional differences between the public projections as well as + agreement on attribution. +- Historical sensitivity/dependency propagation, diagnostic source-location + fallback, detached returned records, and complete retained-history accounting + under the new ownership graph. Use small controlled budget cases and existing + disclosure fixtures rather than memory or timing thresholds. - A composition conflict whose acquisition and assembly orders differ, using the audit fixture. Check both published diagnostics and rendered related-location order, while retaining coverage for ordinary diagnostic canonicalization. @@ -200,40 +316,31 @@ observable behavior rather than a particular private index or number of copies. No benchmark campaign, broader audit, or new release gate is part of this roadmap. Completion requires all four findings to have implementation and regression -evidence, agreement at the affected public boundaries, and appropriate development -checks. The audit's previous passing suite is not validation of these future -changes. Scope here does not promise identical output from different inspection -interfaces or a universal guarantee about unrelated path-processing code. +evidence, appropriate development checks, and both history interfaces deriving +semantic attribution from the shared model. Review the final producer/consumer +map to confirm superseded independent reconstruction has been removed and +internal attribution readers have migrated. Passing the four defect regressions +while retaining the old parallel attribution machinery is not completion. -## Open Decisions +The audit's previous passing suite is not validation of these future changes. +Scope here does not promise identical output from different inspection interfaces +or a universal guarantee about unrelated path-processing code. -### OD-1. Where Should Shape-Specific History Evidence Live? +## Decision Status -**Recommendation:** retain semantic-shape evidence with the internal history -events as they are built, using existing concrete paths and document evidence. -Select applicable events and previous origins for each query before producing -detached public `Change` records. The exact representation belongs in the -implementation plan. +OD-1 is settled: build one shared internal history model with distinct metadata +and explanation projections. The design investigation found that both views +require evidence the other's public representation does not preserve. Shared +ownership can replace independent attribution and redundant retained evidence, +while keeping the current lifecycle and public contracts. -This fits the existing stored-history owner and keeps metadata queries independent -of explanation rendering. It also permits a focused correction where event -construction currently loses shape. The cost is carrying and preserving evidence -across several event producers; the design must keep it consistent with replay -and avoid redundant retained trees. The amount of new state should follow what -existing evidence cannot supply, not an assumed new event framework. +The narrower repair of stored history events would leave the separate replay +attribution owner in place. Promoting current explanation output to authoritative +history would lose composite-definition and specific resolution information. +The selected common underlying model addresses both concerns. This accepts a +larger internal refactor than a local shape-filtering fix. -**Viable alternative:** derive the relevant metadata history from existing -shape-aware replay evidence, sharing a private selection operation with -explanation where their semantics agree. This can reduce duplicated attribution -logic and reuse evidence that already handles the reported transition correctly. -However, metadata history includes default/definition and inheritance records -whose presentation differs from explanation. This option needs a clear account -of those differences, query cost, and evidence ownership; routing `History` -through the public explanation representation would introduce unnecessary -coupling. - -Choose between these approaches after a focused comparison of the affected event -producers and replay coverage, before finalizing the implementation plan. The -decision concerns private ownership and implementation cost, not public path or -history semantics. No other significant product or architectural decision is -open for this scope. +No significant open decision remains at roadmap level. Private representation, +indexing, and helper choices belong in the implementation plan. This roadmap +records future implementation ownership; it does not revise accepted ADRs or +declare the model implemented. diff --git a/docs/roadmap/interfaces.md b/docs/roadmap/interfaces.md index 16a7166..d63c543 100644 --- a/docs/roadmap/interfaces.md +++ b/docs/roadmap/interfaces.md @@ -6,7 +6,8 @@ Status: four open findings. This is an audit record, not an implementation plan. No production code or repository tests were changed. The [interface remediation roadmap](interface-remediation.md) describes the -proposed target, required changes, dependencies, and remaining design decision. +settled target, required changes, and dependencies, including a shared internal +history model for metadata history and explanation. Implementation remains pending. This audit retains the findings and reproduction evidence. ## Scope and evidence