Document the internal complexity refactor roadmap

docs/roadmap/complexity.md

@@ -0,0 +1,221 @@
+# Internal Complexity Refactoring Roadmap
+
+Status: proposed target state. This roadmap defines a behavior-preserving
+refactor; a later implementation plan will define stages and exact private
+signatures and file names.
+
+## Purpose
+
+Make ConfigKit's import, inheritance, and projection implementation easier to
+understand, review, and change while preserving its public APIs and configuration
+behavior. Concentrate on functions that coordinate several responsibilities and
+carry substantial shared state.
+
+The target is clearer responsibility and ownership. Adding a few well-named
+helpers, private records, or focused files can serve that target even when total
+line count increases. Complexity measurements help locate candidates; they do
+not establish function-size limits, score thresholds, or release gates.
+
+The [architecture policy](../policy/architecture.md) owns package and ownership
+constraints. The [configuration contract](../config.md) and exported Go comments
+remain authoritative for consumer behavior.
+
+## Current Baseline
+
+Reviewed baseline: `7a83704f43ef9870db33661231cdff5d6ee502f7`.
+
+The root facade delegates to a cohesive private engine, supported by the ordered
+document representation and YAML adapter. Most functions are small. The
+complexity audit identified these concentrated maintenance costs:
+
+| Area | Current implementation | Required improvement |
+| --- | --- | --- |
+| Imports | `imports.go` spans acquisition, item-local composition, and evidence projection. Private loading variants pass long positional argument lists. Source admission and directory selection contain substantial mixed control flow. | Give import entry points explicit, cohesive inputs and separate the responsibilities of acquisition, scope assembly, and evidence projection. |
+| Inheritance | `applyInheritanceScopeTemplates` coordinates payload preparation, parent combination, policy ownership, origins, history, and accounting in one executor. | Keep one readable executor with focused operations and explicit state handoffs. |
+| Projection | `valueOperation.project` combines type dispatch with struct flattening, map and sequence traversal, scalar conversion, sensitivity, and accounting. | Retain clear dispatch while moving substantial value-kind handling into focused helpers. |
+
+The implemented inventory belongs to the
+[component overview](../internal/overview.md). Existing import flow and scheduling
+are described in [import internals](../internal/imports.md).
+
+## Target Architecture
+
+The refactor stays within `internal/engine`. The root facade, `internal/document`,
+and `internal/yaml` retain their existing responsibilities and dependency
+direction. Focused files and ordinary private functions express the new internal
+organization. Package extraction is outside this roadmap's selected design.
+
+`executeLoadRequest` remains the visible coordinator of the shared load lifecycle.
+Its phase order should remain easy to follow. Helpers expose meaningful work and
+results rather than distributing that order across a generic execution framework.
+
+Existing shared mechanisms remain shared: recursive mapping combination,
+identified-sequence operations, dependency-ready scheduling, schema descriptors,
+and projection/accounting state. New helper boundaries make their ownership and
+use clearer without creating parallel implementations.
+
+### Imports: Cohesive Inputs And Scope Ownership
+
+Import loading has a clear private entry surface using named request/context
+fields for related inputs. Reuse the existing load-request, acquisition-scope,
+and traversal models where they fit. Consolidate forwarding variants whose only
+purpose is passing another optional positional argument through the chain.
+Retain convenience helpers when they express a useful, distinct operation.
+
+Keep these state lifetimes identifiable:
+
+- Load-wide state: accounting, shared inclusion/repetition information,
+  namespaces, and source-version processing.
+- Acquisition requests: source and filesystem context, placement, declarations,
+  applicable policy, and the relevant assembly handoff.
+- Composition scopes: peer contributions, item-local helpers, record ownership,
+  and pending declarations.
+- Evidence: physical source records, effective destinations, origins, and retained
+  history projections.
+
+Group acquisition, pending-item assembly, and evidence projection in focused
+files. A caller should be able to identify which operation acquires resources,
+which composes already acquired content, and which projects retained evidence.
+Shared mutable state has a visible owner; a request record need not duplicate
+state already held by that owner.
+
+Within dense admission, directory-selection, and declaration-parsing routines,
+extract coherent operations where that reduces nested, mixed responsibilities.
+Preserve the existing selection algorithms and host/native filesystem behavior.
+Resource ownership and closure remain explicit across success and failure paths.
+
+The resulting flow preserves the distinction among acquisition order, strict
+scope scheduling, and precedence. Completing a helper scope before placing its
+item remains part of the same import flow. Ordinary and layered loading continue
+to share the applicable mechanisms.
+
+### Inheritance: One Executor With Explicit Phases
+
+Keep the mapping and identified-sequence adapters and their shared template
+executor. The executor coordinates focused operations for:
+
+- Preparing child payload, directives, and destination-local state.
+- Preparing the resolved parent payload and applicable inherited policy.
+- Combining values through the existing mapping and sequence mechanisms.
+- Producing policy, origin, replacement, and replay evidence from that combination.
+- Accounting for expansion and retained history under the existing budgets.
+
+These are responsibility boundaries, not a required helper count or a new
+scheduler. Small private result records are appropriate when related outputs
+otherwise travel through long argument lists or implicit mutations.
+
+Authored templates, resolved values, scoped policy, and inspection evidence remain
+distinguishable. Immutable borrowed inputs and owned mutable results have clear
+lifetimes. Nested instructions remain available for destination-local evaluation.
+For identified-record inheritance, the child's own identity fields and origins
+remain intact; the parent's identity fields are excluded from inherited payload
+and its history, as the current contract requires.
+
+One sequence-operation result continues to drive value construction, record
+ownership, origins, and historical position projection. Helpers consume that
+evidence directly. Scheduling remains dependency-ready, enclosing relationships
+resolve before nested ones, and assembled sequence order remains unchanged.
+
+### Projection: Explicit Value-Kind Handlers
+
+Keep `projectConfiguration` as the schema-checked entry and `valueOperation` as
+the owner of per-operation traversal and accounting state. The central dispatch
+clearly selects the appropriate handling for pointers, dynamic values, structs,
+maps, sequences, and scalars.
+
+Substantial struct, map, and sequence work lives in focused handlers. Struct
+projection retains declaration order, inline flattening, collision diagnostics,
+and sensitivity propagation. Map projection retains deterministic key ordering.
+Scalar conversion retains the existing number, duration, text-codec, and secret
+reference representations.
+
+Depth and node accounting retain their existing meaning, including flattened
+containers. Cycle tracking stays active through recursive child traversal and
+is balanced on error paths as well as success. Passing state through a helper
+does not introduce another full-tree projection or an unnecessary ownership copy.
+
+Cloning and source-value construction remain distinct operations with their
+existing semantics. Reuse small helpers when the semantics coincide; keep the
+projection refactor centered on making the existing traversal readable.
+
+## Behavior And Scope Preservation
+
+The implementation preserves:
+
+- Root-owned public types, signatures, supported import paths, and facade boundaries.
+- Supported source forms, opt-ins, import controls, placement, acquisition and
+  composition ordering, and source/version evidence.
+- Strict peer composition, precedence operations, identity agreement and lifetime,
+  record ownership, and destination-local inheritance.
+- The load lifecycle, including typed defaults, post-decode CLI assignment,
+  recorded application resolution, and final validation.
+- Explicit presence, provenance, history, sensitivity, resource contexts, and
+  detached publication/query results.
+- Established diagnostic codes, semantic paths, source coordinates, ordering,
+  resource lifetime, and existing limit behavior.
+
+The owning contracts define the extent of these obligations. Incidental private
+call sequences, exact helper names, and allocation counts do not acquire new
+guarantees through this roadmap. The work adds no consumer options, configuration
+syntax, dependencies, or complexity-based gates.
+
+## Relationship To Remediation And Other Work
+
+[Remediation](remediation.md) owns the identified correctness and algorithmic
+efficiency findings. Several overlap these files, particularly item-scope
+application matching and inheritance policy localization.
+
+Use the documented contract and completed remediation as the behavioral baseline.
+Keep corrective changes reviewable under their findings, and retain their
+regressions through this refactor. The later implementation plan should coordinate
+overlapping work and select its starting revision rather than duplicating the
+remediation plan here. These structural changes do not close a finding merely
+because its function moved or became smaller.
+
+The pending [discovery roadmap](discovery.md) remains separate feature work.
+Import cleanup should leave clear private boundaries for future use without
+implementing discovery behavior as part of this refactor.
+
+## Work Needed And Validation
+
+Implementation needs to consolidate import inputs and adapters, separate import
+responsibilities, extract inheritance operations with explicit outputs, and
+decompose projection handling. Remove superseded private plumbing as its callers
+migrate. Review the resulting call flow for repeated traversals, policy scans,
+copies, or hidden state transitions introduced by extraction.
+
+Reuse the existing behavioral suites under the
+[development guide](../development.md) and [testing policy](../policy/testing.md):
+
+- Imports: selection, scheduling, source/version evidence, placement, helper scopes,
+  ordinary/layered equivalence, filesystem failures, and resource lifetime.
+- Inheritance: mapping and record relationships, nested destination resolution,
+  cancellation and replacement, policy ownership, history, sensitivity, and
+  independent loads.
+- Projection: schema/decoding agreement, inline fields, text codecs, nil/empty
+  values, cycles and budgets, secrets, and detached results.
+
+Add focused regressions only where a consequential handoff risk lacks coverage.
+Tests should continue to describe behavior as private functions and files move.
+Use the existing everyday check and applicable ownership race checks during
+implementation. Complexity measurements can inform review; acceptance does not
+depend on numerical scores or a new benchmark campaign.
+
+Update the component overview and import internals to describe the implemented
+ownership and file responsibilities when the refactor lands. Update contributor
+navigation and test-owner references where moves affect them. Public behavior
+continues to be defined by its current canonical owners.
+
+## Completion Criteria
+
+- Import callers pass cohesive named inputs, with clearly owned load-wide and
+  scope-local state and fewer redundant forwarding variants.
+- Import acquisition, item assembly, and evidence projection are easy to locate
+  and follow across focused private files.
+- Inheritance orchestration exposes value, policy, evidence, and accounting work
+  without requiring readers to reconstruct their interaction from one large body.
+- Projection dispatch and substantial value-kind handlers have clear boundaries
+  and share the original operation state correctly.
+- Existing APIs and contracted behavior, including completed remediation, remain
+  covered by the relevant behavioral and ownership checks.
+- Internal documentation matches the resulting implementation.