diff --git a/docs/development.md b/docs/development.md index d7c9b5c..f251ebd 100644 --- a/docs/development.md +++ b/docs/development.md @@ -43,7 +43,7 @@ public behavior. | Shared source-version checks | [Configuration contract](config.md#shared-source-version-checks), [ADR 0013](adr/0013-shared-source-version-checks.md), and the version/source-version tests below | | Documentation changes | The owning document | | Licensing, dependencies, attribution, or release-document changes | [Project grant](../COPYRIGHT) and [attribution](oss/attribution.md); release changes also follow the release procedure | -| Architecture decisions or future work | [Future features](roadmap/future.md), [consolidated audit review](roadmap/audit.md), and relevant ADRs listed below | +| Architecture decisions or future work | [Future features](roadmap/future.md), [consolidated audit review](roadmap/audit.md), [remediation roadmap](roadmap/roadmap.md), and relevant ADRs listed below | | Code and test changes | The owning contract and tests from the map below | | Examples or copyable assets | The owning implemented contract and [consumer adoption guide](consumers/README.md) | | Release preparation | [Release procedure](release.md) | @@ -200,8 +200,10 @@ git show 34cbc0d:docs/roadmap/remediation.md Potential enhancements are tracked in [future features](roadmap/future.md). The [consolidated audit review](roadmap/audit.md) owns the open findings from -the nine 2026-10-25 audits, their evidence, and scope questions. The original -reports are preserved at `ecd5d5d` under `docs/roadmap/`. +the nine 2026-10-25 audits, their evidence, and settled scope decisions. The +[remediation roadmap](roadmap/roadmap.md) owns the combined target state and +high-level dependency order. The original reports are preserved at `ecd5d5d` +under `docs/roadmap/`. Install the Go version required by `go.mod` or a compatible newer toolchain. Acquire declared dependencies explicitly before entering an offline diff --git a/docs/roadmap/audit.md b/docs/roadmap/audit.md index d127583..a143e39 100644 --- a/docs/roadmap/audit.md +++ b/docs/roadmap/audit.md @@ -6,8 +6,10 @@ Consolidated: 2026-10-25. Baseline: `ecd5d5d`. This document consolidates nine sequential audits for maintainer review. It owns their open findings, supporting evidence, proposed remedies, and relationships. -The settled decisions select the remediation roadmap's scope; this document is -not a staged implementation plan or a new set of release gates. The +The settled decisions select the [remediation roadmap's](roadmap.md) scope. +That document owns the combined target state and high-level dependency order; +this review owns detailed finding evidence and acceptance cases. Neither is a +staged implementation plan or a new set of release gates. The [configuration contract](../config.md), public Go declarations, and [architecture policy](../policy/architecture.md) continue to own behavior and guarantees. diff --git a/docs/roadmap/roadmap.md b/docs/roadmap/roadmap.md new file mode 100644 index 0000000..0880c0d --- /dev/null +++ b/docs/roadmap/roadmap.md @@ -0,0 +1,290 @@ +# Audit Remediation Roadmap + +Status: scope settled; implementation has not begun. +Prepared: 2026-10-25. Planning baseline: `e1e94ee`. + +This roadmap describes the target state and dependency order for the 29 entries +selected in the [consolidated audit review](audit.md#settled-scope-decisions). +It is intended for maintainers preparing the implementation plan. + +## Document Ownership + +- [audit.md](audit.md) owns finding IDs, original evidence, detailed remedies and + acceptance cases, and the settled scope decisions. Its evidence has not been + independently reproduced by writing this roadmap. +- This document owns the combined target state, work areas, and high-level + dependencies. Finding IDs below provide traceability without repeating the + full audit inventory. +- The subsequent implementation plan will own concrete stages, edits, validation + commands, and execution progress. It should carry the selected IDs through to + completion evidence. + +The [configuration contract](../config.md), public Go declarations, and +[architecture policy](../policy/architecture.md) remain the owners of behavior +and guarantees. Update their applicable implementation documentation when work +lands; this planning document does not redefine their contracts. + +## Current Baseline And Targeted End State + +The existing loading model remains the foundation. Public loading APIs delegate +to the shared engine lifecycle; acquisition retains source scopes for composition, +then precedence, inheritance, decoding, CLI overrides, application resolution, +secret lookup, and validation produce a detached result with inspection metadata. +The [component overview](../internal/overview.md) describes the current owners. + +The audits identify failures at particular traversal, lifetime, error, and +accounting boundaries, alongside redundant representations and repeated work. +The target is to repair those boundaries and simplify their implementation while +preserving ConfigKit's public API and configuration model. + +| Current gap | Targeted outcome | +| --- | --- | +| Recursive descriptors can fail to terminate, failed dynamic builds can leave partial state, and slice aliases can be mistaken for cycles. | Traversal terminates for the reported cases, failed construction leaves reusable state valid, and supported acyclic values remain accepted. | +| Callback and owned-root cleanup can be skipped during panic unwinding; directory errors lose their failure phase. | Cleanup follows the lifetime of each owned resource, and selected-directory failures retain the information needed for correct error handling. | +| Finite symlink dependencies can be expanded repeatedly. | Completed dependencies are reused within a resolution while active cycles remain detectable. | +| Projection reservations and published history can escape existing accounting. | Existing budgets govern the relevant allocations and complete retained document graph. | +| Collection-policy work repeatedly scans or copies unrelated state. | Policy preparation, application, and retained evidence have clear ownership, with work localized to affected policy. | +| Acquisition has redundant paths and difficult handoffs; several private representations have no production consumer. | Production paths and their tests share the active protocol, with cohesive handoffs and unused representations removed. | +| Some lookups repeat full scans, and some tests assert incidental representation or miss meaningful mutations. | Local indexing avoids the identified repeated work, and tests protect observable contracts at useful boundaries. | + +This is a bounded remediation effort, not a redesign of configuration semantics. +Correcting erroneous acceptance, rejection, cleanup, or diagnostics in the +reported cases restores existing contracts. It does not select new loading APIs, +configuration options, limits, background work, or persistent caches. + +## Selected Work + +The following areas account for all 29 selected entries. Detailed examples and +acceptance cases remain with their linked audit group. + +### 1. Safe Typed-Value Traversal And Descriptor Reuse + +Selected entries: **A01, A02, A03, A04, A05** in +[schema and typed values](audit.md#a-schema-and-typed-values). + +Make pointer-only descriptor traversal terminate in decoding and path resolution, +while preserving supported null values and ordinary recursive structures. Make +dynamic descriptor construction transactional: a failed build must not leave +partial descriptors available to subsequent operations. Staging successful +additions or rolling back the complete failed addition set are implementation +choices; both must handle recursive references and preserve prior valid entries. + +Correct active-path cycle detection to distinguish finite slice views sharing +storage from actual cycles. Keep detachment and nil/empty behavior intact across +the callers that share this traversal machinery. + +Enforce existing projection budgets before large container reservations. Include +map-key materialization and overflow-safe lower-bound checks in the review; a +check after the reservation does not address the reported failure. This work +does not introduce a general heap limit or reject large containers independently +of existing budgets. + +Once failed construction is safe, memoize completed successful dynamic-type +eligibility checks within the value operation. Keep failure diagnostics specific +to their current path and distinguish completed results from traversal visits. +This removes repeated descriptor-graph walks without a cross-load cache. + +### 2. Callback, Filesystem, And Dependency Lifetimes + +Selected entries: **B01, B02** in [callback lifetime](audit.md#b-callback-and-filesystem-lifetime) +and **C01, C02** in [filesystem traversal](audit.md#c-import-selection-and-link-resolution). + +Tie callback-context deactivation to the callback invocation with deferred cleanup +so it runs on normal return and panic unwinding. Independently give the owned +import root a deferred cleanup fallback. Preserve normal close-error handling, +caller ownership, and propagation of the original callback panic. Fixing either +state lifetime or handle lifetime does not fix the other. + +Preserve the phase of a directory failure. A genuinely absent directory before +successful opening remains subject to existing permissiveness rules; a later +read or close failure must not be treated as absence. Apply this distinction to +both supported directory-reading paths. Strengthen the shared filesystem tests +to check each handle's closure and structured, non-disclosing diagnostics, +including partial reads and failures masked by another successful directory. +Do not change regular-file close-error policy as a side effect. + +Avoid repeatedly expanding completed finite symlink dependencies in both host +and native filesystem resolution. Use operation-local memoization of completed +targets, or an equivalent dependency evaluation, with keys that respect the +resolution namespace. Keep active-path cycle detection separate from completed +work. Preserve containment, supported finite reuse, and existing host-path +semantics. The fix must work for directory and glob selection without a new +arbitrary traversal cap or changed symlink defaults. + +### 3. Retained History And Combination Evidence + +Selected entries: **D01, D02, D03, D04** in +[history and retained state](audit.md#d-retained-history-and-unused-evidence). + +Remove the unconsumed override-result pre-CLI copy and the unqueried published +structural-input copy. Preserve the separate pre-CLI state used by active replay +and the structural input still used before publication for metadata and secret +classification. Remove unused combination-evidence members and the work needed +solely to produce them, retaining projections, mappings, origins, and snapshots +that have active consumers. + +With retained roots clarified, enforce the existing history allowance against +the complete graph published through inspection, including replay and inheritance +state and publication-created copies. Count shared nodes once by pointer identity; +distinct retained copies still consume the allowance. Account for required copies +before creating them where practical. Removing an unused tree alone does not +resolve the accounting omissions. + +Keep required before/after evidence and existing inspection behavior. Validate +the complete publication path with small controlled allowances, including the +interaction of CLI, resolution replay, and inheritance, rather than relying only +on tests of an isolated counting helper. + +### 4. Collection-Policy Preparation And Application + +Selected entries: **E01, E02, E03** in +[collection policy and projection](audit.md#e-collection-processing-and-efficiency). + +Distinguish the existing policy state used for lookup, changes introduced by an +incoming tier, and evidence retained by the assembly. Prepare incoming claims and +declarations against read-only lower state, then apply the owned changes at the +appropriate combination boundary. Avoid copying every unrelated policy for +small tiers and scanning all policies for each independent replacement. + +Use a semantic-path index, localized updates, or equivalent bounded lookup to +find affected policy. Preserve retirement timing, policy established by the +incoming contribution, pending declarations, persistent explicit modes, and +nested record ownership. Copying remains appropriate wherever mutable ownership +requires it. Removing only one duplicate clone or shortening a function does not +address the identified growth patterns. + +Clarify preparation, evidence application, and finalization responsibilities. +Share straightforward history-descriptor translation where useful; evaluate +shared record-selection logic while keeping overlay and inheritance origin and +mode lifetimes distinct. Retain the common sequence-combination owner. This work +does not call for a generic merging framework. + +### 5. Active Acquisition Handoffs And Private Cleanup + +Selected entries: **F01, F02, F03, F04, F05, F06** in +[acquisition, cleanup, and test quality](audit.md#f-remaining-maintenance-and-test-improvements). + +Remove the test-only eager acquisition mode so acquisition retains scopes through +the active production protocol. Tests that need composition should invoke the +actual composition boundary explicitly. Retire the parallel test-only import +parser and route relevant fixtures through the active directive parser or the +appropriate declaration boundary, retaining malformed-input coverage. + +Then clarify active acquisition and coordinator handoffs around cohesive tasks: +parsed-input directive/version handling, standalone placement, and repeated tier +application/projection. Use explicit inputs and results consistent with existing +state owners. Extracting helpers that continue to share surrounding mutable state +does not establish a clearer handoff. Preserve ordinary/layered differences, +overlay and diff behavior, source identity, callback timing, diagnostics, and +accounting scopes. + +Remove the disconnected root record-clone family, uncalled engine helpers, and +unread private transport/result members identified in the audit. Reconfirm +references on the implementation baseline, including function values and relevant +build configurations. Preserve live conversion paths and validation side effects; +the audit's absence of field reads is not a general rule for deleting state. +Replace meaningful ownership coverage through the public boundary before removing +tests that only exercise disconnected helpers. + +### 6. Local Lookup Improvements + +Selected entries: **E04, E05** in +[collection policy and projection](audit.md#e-collection-processing-and-efficiency) +and **F08** in [secret-failure lookup](audit.md#f08-index-secret-failure-location-queries-lazily). + +Reuse a local document lookup or equivalent traversal for tier-history projection +instead of rescanning wide mappings for each sequence. Preserve authored +coordinates, sparse projections, and evidence ownership. + +Restore compound identity fields using indexed mapping positions and membership +checks rather than rescanning child and merged mappings for every field. Preserve +child-authored values and locations, mapping order, and detached nodes. + +Build secret-failure location lookup state lazily when failures need it. Preserve +all diagnostics, nearest-location fallbacks, and confidentiality; successful +loads should not incur an unnecessary index merely to support failure reporting. +These changes address specific repeated work and do not promise a measured +startup improvement. + +### 7. Tests That Protect The Intended Behavior + +Selected entries: **F09, F10, F11, F12** in +[acquisition, cleanup, and test quality](audit.md#f-remaining-maintenance-and-test-improvements), +in addition to the regression coverage attached to each reported defect. + +Cover historical-path enumeration through the public API, including removed +paths, queryability, ordering, and detached returned slices. Use those assertions +to replace private representation checks where they address the same risk. + +Strengthen fuzz detachment checks by mutating existing descendants or in-length +elements, so spare capacity cannot hide aliases. Replace exact filesystem stat +call sequences with observable candidate ordering and meaningful failure +selection, retaining relevant assertions such as not opening excluded inputs. + +Consolidate facade scaffolding around public construction, loading, errors, +ownership, and lifecycle behavior. Preserve distinct integration risks and +reader-byte snapshot coverage. Existing coverage percentages or a desired test +count are not acceptance criteria for this work. + +## General Implementation Order + +Begin with a focused implementation plan that maps each selected ID to its owning +change and validation. Establish regression evidence alongside fixes, using the +audit's bounded cases. Prioritize reported defects; small prerequisite cleanups +may accompany them, and broader refactoring must not hold up a consequential fix. + +| Order or dependency | Reason | +| --- | --- | +| Repair dynamic descriptor construction (A02) before successful eligibility reuse (A05). | Reuse must not amplify partial descriptor state. | +| Address callback state and owned-root cleanup (B01/B02) with separate acceptance checks. | They can proceed together or independently, but both lifetimes need protection. | +| Strengthen directory error/handle assertions with C01; remediate C02 independently of deferred glob extraction. | Tests must expose the lost failure phase, and the availability fix needs no broader traversal refactor. | +| Remove unused history copies (D02/D03) and clarify retained combination evidence (D04) before finalizing complete history accounting (D01). | Account for the graph actually retained, while keeping all required evidence. These changes may be delivered together. | +| Establish collection-policy state/evidence ownership (E03) before or together with localized retirement and copying changes (E01/E02). | Efficiency changes depend on knowing which state can be read, changed, and retained. Preserve separate behavioral and scaling acceptance cases. | +| Remove the eager test-only protocol and parallel parser (F01/F03) before or alongside active acquisition extraction (F02). | Refactor the production protocol and migrate tests to it without carrying redundant paths forward. | +| Establish replacement public coverage (F09 and relevant F12 cases) before deleting disconnected clone tests and facade scaffolding (F04/F12). | Cleanup should preserve useful protection while removing incidental assertions. | +| Complete integration review after the affected owners settle. | Check agreement among final values, metadata, replay, diagnostics, and resource accounting across the changed boundaries. | + +These are dependency constraints, not barriers requiring every work area to finish +before another starts. Independent traversal fixes, local lookup improvements, +and bounded helper/member cleanup can proceed as their owners are ready. Coordinate +changes that touch the same state: A01 with schema-path member cleanup, D04 with +history and collection evidence, and acquisition cleanup with version and resource +lifetime handling. + +## Validation And Completion + +Use the audit's detailed acceptance cases and the [testing policy](../policy/testing.md) +to prepare focused validation. For fatal or hanging regressions, use bounded +subprocesses; for budget cases, use small controlled limits. For symlink expansion +and repeated lookup work, prefer bounded operation evidence and algorithm review +over timing thresholds. Use real filesystem fixtures where filesystem semantics +matter and narrow fault injection for read/close failures. State platform limits. + +Run the applicable development checks from the [development guide](../development.md) +as implementation proceeds, including appropriate race coverage for changed +ownership and lifetime boundaries. Existing audit results are baseline evidence, +not validation of the eventual changes. No performance benchmark campaign is +required by this roadmap, and no new benchmark results are claimed. + +Completion means all 29 selected findings have implementation and acceptance +evidence, the affected public behavior remains consistent with its contracts, +and canonical internal documentation describes the resulting owners. Carry +status in the audit until the implementation plan takes over execution tracking; +avoid maintaining duplicate progress inventories. Retire completed planning +records under the normal documentation lifecycle. + +## Scope Boundaries And Decision Status + +All three audit scope questions are settled. No significant architectural or +product decision remains open for this roadmap. Private choices such as staging +versus rollback, index representation, or helper signatures belong in the +implementation plan and review, subject to the acceptance cases above. If that +work exposes a need to change a public contract, bring the specific decision back +for review rather than treating it as an implicit part of remediation. + +The glob preparation refactor (C03) and wider facade transport simplification +(F07) remain the first two entries in [future.md](future.md), for consideration +after this selected work is complete. Other deferred features, clone-traversal +redesign, and broader performance experiments remain outside this effort. Keep +existing ownership boundaries and loading APIs; no ADR or architecture-policy +change is proposed by this roadmap.