Clarify ConfigKit prerequisites and efficiency goals

docs/roadmap/configkit-assessment.md

@@ -1,7 +1,8 @@
 # ConfigKit Adoption Assessment
 
-Status: proposed for discussion, not an approved feature specification or an
+Status: assessment with agreed design direction; upstream API details remain
-implementation plan. No runtime changes are part of this assessment.
+to be settled before the feature roadmap and implementation plan. No runtime
+changes are part of this assessment.
 
 ## Goal And Recommendation
 
@@ -24,6 +25,41 @@ lazy error arbitration, and repeated profile-chain resolution. Some of those
 savings come from the loading redesign rather than ConfigKit itself. ConfigKit
 adds ongoing value by owning the generic decoding behavior and its tests.
 
+## Prerequisite: Proposed Upstream ConfigKit Work
+
+Design, implement, and release these two ConfigKit capabilities before the
+dependent PromptKit refactor. The remaining recommendations assume this upstream
+work; finalize their integration details against its agreed public contracts.
+These are selected architectural prerequisites, not a claim that adoption is
+technically impossible using today's reader-based API.
+
+1. **Native `fs.FS` sources and source-relative context.** Support embedded and
+   caller-supplied filesystems as first-class configuration sources. Define source
+   identity, relative path resolution, ownership, and how enabled imports and
+   overlays operate within that filesystem. This is broadly useful for libraries
+   bundling default configuration. Passing an opened file to today's `Reader`
+   supplies bytes and a label, but does not make filesystem-relative loading
+   native to ConfigKit; that workaround is not the target architecture.
+2. **Public in-memory layered loading with origins.** Support explicitly supplied
+   layers without temporary files or YAML serialization round trips, retaining
+   presence and meaningful source provenance. The upstream design must settle
+   which inputs it accepts: filesystem/byte sources, typed values, or both; how
+   typed values express omission; and how strict composition differs from
+   intentional overriding. Recommend covering embedded defaults, caller sources,
+   and programmatic values, with explicit ordering and useful conflict diagnostics.
+
+Keep these APIs application-neutral. A list of sources alone does not implement
+PromptKit's whole-profile shadowing, named inheritance, or atomic provider maps.
+Define which layer operations ConfigKit owns and which identity/domain decisions
+remain in PromptKit; do not silently substitute recursive merging for an agreed
+replacement rule. Revisit ConfigKit's current single-overlay/deferred boundaries
+deliberately rather than working around them inside PromptKit.
+
+The upstream design should show how the two capabilities work together, including
+source origins and efficient ownership/copying. Resolve these API questions before
+turning the assessment into a decision-complete PromptKit implementation plan.
+This assessment does not authorize code changes in either repository.
+
 ## Evidence And Scope
 
 Reviewed local baselines on 2026-09-29:
@@ -80,7 +116,8 @@ dropping execution features to simplify the configuration layer.
 | YAML grammar, scalar interpretation, one-document enforcement, unknown fields, typed ranges, null legality | Delegate to ConfigKit; remove corresponding PromptKit parser logic and redundant parser tests |
 | Field presence and source coordinates | Delegate to ConfigKit; retain only the compact records needed for resolution and diagnostics |
 | File/directory/embedded source selection and definition ID indexing | Retain a small PromptKit collection loader; ConfigKit loads documents, not a prompt catalog |
-| Filesystem traversal and relative content references | Retain PromptKit's source-boundary adapters; use standard `io/fs` facilities where sufficient |
+| Configuration filesystem sources, relative imports, and generic layer origins | Delegate to the prerequisite ConfigKit APIs; do not maintain parallel reader-based loading machinery |
+| Definition discovery and relative message-content references | Retain PromptKit's domain resource adapters, using ConfigKit source context and standard `io/fs` facilities where sufficient |
 | Profile-source priority and duplicate definition policy | Retain explicit PromptKit collection assembly; eliminate error-driven repository fallback |
 | Named profile inheritance and backend selection | Retain a typed PromptKit resolver over the loaded collection, with explicit presence; consider generic inheritance upstream only with broader evidence |
 | LLM setting bounds, role/session normalization, output rules, reserved provider fields | Retain shared PromptKit domain owners; ConfigKit validation callbacks may invoke them |
@@ -106,10 +143,11 @@ belongs in this phase. Publish no partially initialized engine on failure.
 Use one typed document per prompt or profile initially. Existing files need not
 be converted into a monolithic application configuration merely to adopt
 ConfigKit. Support OS directories, individual files, embedded/custom `fs.FS`
-sources, and programmatic profiles. For `fs.FS`, a small adapter can open a file
+sources, and programmatic profiles. Use the prerequisite native ConfigKit
-and supply `configkit.Reader` with a safe source label; PromptKit retains root
+filesystem and layered-loading APIs for configuration acquisition and generic
-and path context for referenced message files. Close handles owned by the adapter.
+composition. PromptKit retains definition discovery and message-resource handling,
-ConfigKit's existing public API is sufficient for this design.
+using the source context supplied by ConfigKit. Exact integration is contingent
+on the upstream contracts; do not build a competing layer engine in PromptKit.
 
 Load and freeze referenced message text with definitions. Validate static prompt
 structure and template syntax at loading, using the same template function setup
@@ -147,10 +185,12 @@ requirements after runtime selection/overrides. Prompt default-profile reference
 can be checked against the assembled collection; a prompt with no default remains
 valid for callers that supply a profile explicitly.
 
-This is named-definition precedence, not ConfigKit's application overlay system.
+Named-definition precedence is distinct from recursive application overlays.
-Do not generate temporary overlay files or serialize maps to YAML to imitate a
+Use the upstream layered API where its operations fit; retain the small identity
-missing layer API. ConfigKit imports are also not directory discovery: their
+selection step in PromptKit where needed. The upstream design must make that
-strict composition has different semantics and need not be enabled here.
+boundary explicit. Do not generate temporary overlay files or serialize maps to
+YAML to imitate a missing operation. Imports are not definition discovery and
+need not be enabled merely because native filesystem sources support them.
 
 ### Inheritance And Presence
 
@@ -250,16 +290,44 @@ resolved values. ConfigKit also creates metadata and projections; keep only need
 records after loading. These are reasoned tradeoffs, not measured performance
 claims. Adoption is not justified by the decoder replacement alone.
 
-## 5. Upstream Opportunities
+### Algorithmic Efficiency
 
-No ConfigKit enhancement is a demonstrated prerequisite for the recommended
+Efficiency is an explicit design criterion for both the upstream work and this
-per-document snapshot design. Do not manufacture one to move a small adapter
+shared-library refactor. Prefer optimal or near-optimal algorithms relative to
-upstream. Reconsider the following if the agreed roadmap reveals concrete reuse:
+input and produced-data size, not a blanket O(n) claim with an undefined n.
+
+- Acquire and decode each definition once per snapshot; reuse captured content
+  rather than reopening or reparsing it for metadata, selection, and validation.
+- Build identity indexes in one pass, with expected constant-time map lookup.
+  Do not scan whole collections for every lookup or precedence decision.
+- Process inheritance graph structure in O(V + E), where V is the number of
+  profiles and E the inheritance edges. Resolve shared ancestors once. Account
+  separately for the size of materialized settings and retained origins; copying
+  large inherited maps to many independent results necessarily costs more than V.
+- Avoid whole-collection copying per layer, repeated serialization, and repeated
+  prefix/history traversal. Analyze layer processing against total input nodes
+  and produced values/provenance, including intentionally retained history.
+- Keep configuration discovery and decoding out of ordinary execution. Copy only
+  where ownership requires it; do not trade caller isolation for speed.
+
+These are design targets, not promised complexity for every operation. Sorting
+for deterministic discovery may require O(F log F) for F files, and processing
+message text costs work proportional to its bytes. Construction time, retained
+memory, and frequent-call costs all matter. Prefer straightforward indexes and
+clear immutable ownership over speculative global caches or intricate optimizers.
+The implementation plan should include focused benchmarks for consequential
+repeated-work risks, such as shared-ancestor profiles and many layers, rather than
+machine-dependent timing gates or benchmarks for every helper.
+
+## 5. Additional Upstream Opportunities
+
+Native filesystem sources and public layered loading with origins are the
+prerequisites described near the beginning, not optional follow-ups. The following
+remain optional and do not block adoption unless the upstream design establishes
+a concrete dependency:
 
 | Candidate | Recommendation and tradeoff |
 | --- | --- |
-| Native `fs.FS` sources with source-relative resource context | Most plausible optional enhancement: embedded configurations are useful across Go libraries. Reader sources suffice today. A fuller design must define imports/overlays in that filesystem rather than merely add a constructor or leak parser internals. |
-| Public in-memory layered loading with origins | Optional only if multiple consumers need application defaults and file/embedded layers composed together. It could remove adapters in those consumers, but is not necessary for whole-definition catalog shadowing. Review against ConfigKit's deliberate single-overlay scope. |
 | Generic named inheritance/presence-aware typed overlay | Consider an upstream proposal if a coherent application-neutral operation emerges; another concrete use case would strengthen the case but is not a prerequisite. PromptKit's named graph, atomic provider map, and routing rules alone do not justify a configurable general merge framework. Current fixed overlay semantics are not this API. |
 | Context-aware loading | Optional for reusable cancellable construction if large-source consumers need it. Promise cooperative cancellation only; no detached reads or instant interruption of arbitrary readers. |
 
@@ -272,12 +340,15 @@ ConfigKit and PromptKit both declare LGPL-3.0-or-later. Adoption must still upda
 dependency notices under the OSS policy and inspect the exact selected release
 and its transitive notices. Their current Go minimums do not require a PromptKit
 toolchain increase. Use a released module dependency, not a sibling-directory
-replacement. This review proposes no licensing change or upstream edits.
+replacement. This assessment proposes upstream work, but makes no upstream edits
+or licensing change.
 
 ## 6. Decisions And Migration
 
-The goal and permission to break compatibility are agreed. The following specific
+The goal, permission to break compatibility, and these four recommendations are
-choices are recommendations for discussion, not inferred approvals:
+the agreed working direction. Alternatives remain recorded for rationale, not as
+unresolved votes. The prerequisite upstream API questions remain open and must be
+settled before finalizing dependent implementation details.
 
 1. **Loading lifetime and failure scope.** Recommend eager construction, frozen
    definitions/content, and failure for malformed files anywhere in supplied
@@ -326,8 +397,9 @@ Expected consumer migration:
   literals as required. Keep representative preparation, execution, inspection,
   custom backend, fallback, and embedded workflows in the migration guidance.
 
-The subsequent feature roadmap should settle these choices and specify the target
+The subsequent feature roadmap should carry forward these choices, resolve the
-end state. The staged implementation plan can then sequence any accepted upstream
+upstream API details, and specify the target end state. The staged implementation
-work, PromptKit deletions/replacements, contract tests, canonical documentation,
+plan must sequence the prerequisite ConfigKit releases before dependent PromptKit
+work, then cover deletions/replacements, contract tests, canonical documentation,
 examples, attribution, and minor-release migration notes. Neither document should
 become a reason to preserve obsolete compatibility machinery.