promptkit/docs/roadmap/configkit-assessment.md

# ConfigKit Adoption Assessment

Status: assessment with agreed design direction; upstream API details remain
to be settled before the feature roadmap and implementation plan. No runtime
changes are part of this assessment.

## Goal And Recommendation

Use ConfigKit to standardize PromptKit's configuration semantics and simplify
its loading machinery, improving ConfigKit where a reusable capability is
justified. Preserve substantive consumer capabilities, not historical loading
sequences, file syntax, or Go signatures. Document intentional breaking changes
and their migration paths.

Recommend adoption through an eager, immutable definition-loading boundary.
ConfigKit should own YAML interpretation, strict typed decoding, presence, and
source diagnostics. PromptKit should own definition identity, named profile
relationships, LLM-specific rules, and execution. Do not retain the lazy loading
architecture merely to minimize migration, or introduce a second generic
configuration framework inside PromptKit.

The expected benefits are consistent configuration semantics for applications
using both libraries and removal of repeated discovery, partial metadata parsing,
lazy error arbitration, and repeated profile-chain resolution. Some of those
savings come from the loading redesign rather than ConfigKit itself. ConfigKit
adds ongoing value by owning the generic decoding behavior and its tests.

## Prerequisite: Proposed Upstream ConfigKit Work

Design, implement, and release these two ConfigKit capabilities before the
dependent PromptKit refactor. The remaining recommendations assume this upstream
work; finalize their integration details against its agreed public contracts.
These are selected architectural prerequisites, not a claim that adoption is
technically impossible using today's reader-based API.

1. **Native `fs.FS` sources and source-relative context.** Support embedded and
   caller-supplied filesystems as first-class configuration sources. Define source
   identity, relative path resolution, ownership, and how enabled imports and
   overlays operate within that filesystem. This is broadly useful for libraries
   bundling default configuration. Passing an opened file to today's `Reader`
   supplies bytes and a label, but does not make filesystem-relative loading
   native to ConfigKit; that workaround is not the target architecture.
2. **Public in-memory layered loading with origins.** Support explicitly supplied
   layers without temporary files or YAML serialization round trips, retaining
   presence and meaningful source provenance. The upstream design must settle
   which inputs it accepts: filesystem/byte sources, typed values, or both; how
   typed values express omission; and how strict composition differs from
   intentional overriding. Recommend covering embedded defaults, caller sources,
   and programmatic values, with explicit ordering and useful conflict diagnostics.

Keep these APIs application-neutral. A list of sources alone does not implement
PromptKit's whole-profile shadowing, named inheritance, or atomic provider maps.
Define which layer operations ConfigKit owns and which identity/domain decisions
remain in PromptKit; do not silently substitute recursive merging for an agreed
replacement rule. Revisit ConfigKit's current single-overlay/deferred boundaries
deliberately rather than working around them inside PromptKit.

The upstream design should show how the two capabilities work together, including
source origins and efficient ownership/copying. Resolve these API questions before
turning the assessment into a decision-complete PromptKit implementation plan.
This assessment does not authorize code changes in either repository.

## Evidence And Scope

Reviewed local baselines on 2026-09-29:

- PromptKit: `12d6d80024e80c552d122a076c22429bf6f2f6c2` (v0.11.2).
- ConfigKit: `67c03690181c5af8dde6be325abac241dae6732f` (v0.4.2).

PromptKit evidence: [consumer workflows](../consumers/pkg-promptkit.md),
[file formats](../formats.md), [source implementation](../internal/sources.md),
[runner](../internal/runner.md), and the implementation owners identified below.
ConfigKit evidence: its reviewed `docs/config.md`, `docs/roadmap/deferred.md`,
`lifecycle.go`, `source.go`, and ADR 0001 in the sibling checkout. That checkout
is review context, not a proposed build dependency or required directory layout.

ConfigKit currently exposes `Load[T]` over file, byte, reader, and defaults-only
sources. It supports strict decoding, presence/origin records, defaults,
recorded resolution, semantic validation, explicit imports, one selected
filesystem overlay, and secret handling. Its parser and generic conversion
machinery are private. It has no public arbitrary layer-composition, generic
inheritance, or `fs.FS` source API. Its deferred scope explicitly excludes several
of these generalizations; an upstream enhancement requires a deliberate decision.

This is a source-and-contract assessment, not a performance benchmark or full
correctness audit. It does not claim measured allocation, latency, or line-count
savings. Current code and canonical contracts remain authoritative until an
approved implementation changes them.

## 1. Capability Inventory

| Preserve the ability to… | Replaceable mechanics and representation |
| --- | --- |
| Define named, versioned prompts, inspect their declared inputs, and select a version explicitly | Directory scans on each operation, YAML metadata prepasses, filename layout, constructor signatures |
| Use inline and file-backed message templates, variables, input helpers, session templates, roles, and cache control | Timing of definition/content reads and private repository interfaces |
| Configure named profiles, alias/inherit another profile, and override selected settings | Zero-as-absence fields, per-call chain traversal, raw YAML decoding types |
| Supply embedded application defaults, operator definitions, maintained catalogs, and programmatic profiles | Repository-wrapper stacks and exact source-option API |
| Register custom/local backends, use built-ins or endpoint-only profiles, and control backend capacity | Configuration DTO shapes and registry assembly order |
| Prepare, inspect, execute, or freeze an execution for later single-claim execution | Source reloading during ordinary preparation; exact preparation error ordering |
| Override per-run settings, credentials, reasoning, session, and output contract; append rendered messages | Public value signatures if presence or clearer configuration separation requires revision |
| Supply inline/file artifacts and inject artifact readers, model clients, and HTTP clients | Assembly APIs; runtime collaborators need not be YAML-decodable |
| Validate output, use provider-native structured output, bounded repair, transport retries, cancellation, and observation | Internal wiring, not the substantive execution behaviors |
| Receive results, usage, identities, hashes, validation outcomes, and inspectable failures | Configuration error types and when definition failures appear |

Preserve independent profile and prompt inspection as useful operations. They
need not retain the old guarantee that engine construction has never examined
other definitions. Preserve custom backend identity and capacity behavior,
optional upstream credentials, and the distinction between failed generated
content and operational errors. This project does not authorize silently
dropping execution features to simplify the configuration layer.

## 2. Responsibility Mapping

| Concern | Recommended owner and disposition |
| --- | --- |
| YAML grammar, scalar interpretation, one-document enforcement, unknown fields, typed ranges, null legality | Delegate to ConfigKit; remove corresponding PromptKit parser logic and redundant parser tests |
| Field presence and source coordinates | Delegate to ConfigKit; retain only the compact records needed for resolution and diagnostics |
| File/directory/embedded source selection and definition ID indexing | Retain a small PromptKit collection loader; ConfigKit loads documents, not a prompt catalog |
| Configuration filesystem sources, relative imports, and generic layer origins | Delegate to the prerequisite ConfigKit APIs; do not maintain parallel reader-based loading machinery |
| Definition discovery and relative message-content references | Retain PromptKit's domain resource adapters, using ConfigKit source context and standard `io/fs` facilities where sufficient |
| Profile-source priority and duplicate definition policy | Retain explicit PromptKit collection assembly; eliminate error-driven repository fallback |
| Named profile inheritance and backend selection | Retain a typed PromptKit resolver over the loaded collection, with explicit presence; consider generic inheritance upstream only with broader evidence |
| LLM setting bounds, role/session normalization, output rules, reserved provider fields | Retain shared PromptKit domain owners; ConfigKit validation callbacks may invoke them |
| Definition defaults | Use ConfigKit defaults where they do not erase inheritance presence; apply effective execution defaults after inheritance |
| Programmatic configuration | Retain typed construction; normalize to the same presence-aware internal definitions without YAML round trips |
| Runtime clients, observers, artifact readers, request data | Retain normal Go injection and request APIs; do not load them through ConfigKit |
| Credentials | Retain optional environment-name and direct request-key semantics; do not replace these with ConfigKit's required typed secret resolution |
| Catalog manifest, ownership, reserved names, secret-safety checks | Retain the catalog adapter; reuse the common ConfigKit profile decoder |
| JSON Schema resources and generated-output validation | Retain JSON Schema implementation; these are not application configuration documents |
| Application imports, CLI overrides, deployment overlays, secrets-directory bootstrap | Leave with consuming applications unless a concrete PromptKit need is accepted |
| Generic source/decoding diagnostics | Reuse ConfigKit; add only PromptKit identity/context and domain relationship diagnostics |

## 3. Recommended Target Loading Model

### Definition Snapshot

Create one immutable definition snapshot during engine construction. Discover
the explicitly supplied sources, decode each definition through ConfigKit,
normalize domain values, assemble identity indexes, and resolve profile chains
before publishing the engine. No provider call or credential availability check
belongs in this phase. Publish no partially initialized engine on failure.

Use one typed document per prompt or profile initially. Existing files need not
be converted into a monolithic application configuration merely to adopt
ConfigKit. Support OS directories, individual files, embedded/custom `fs.FS`
sources, and programmatic profiles. Use the prerequisite native ConfigKit
filesystem and layered-loading APIs for configuration acquisition and generic
composition. PromptKit retains definition discovery and message-resource handling,
using the source context supplied by ConfigKit. Exact integration is contingent
on the upstream contracts; do not build a competing layer engine in PromptKit.

Load and freeze referenced message text with definitions. Validate static prompt
structure and template syntax at loading, using the same template function setup
as execution; render only when request variables and artifacts exist. Preserve
source-relative content resolution and containment without expanding the threat
model. A snapshot is a set of captured reads, not an atomic filesystem transaction.

Keep request artifacts and schema compilation at preparation. A request can
replace the output contract, so an unused default schema should not need eager
compilation. Prepared execution continues to freeze its effective resources.
The snapshot covers definitions/message text, not environment values, request
files, every schema file, or execution state.

### Composition And Validation

Use explicit source categories in the existing useful priority order:
programmatic profiles, operator source, application fallback, PromptKit-owned
profiles, then external catalogs. Within a category, duplicate profile IDs fail;
across categories, a higher-priority definition replaces the whole lower one.
Prompt ID/version pairs must be unique; different versions remain available.
An omitted version remains ambiguous when more than one version is present.

Strictly decode and locally validate every supplied definition, including
shadowed definitions. Thus an unused malformed file is a construction failure,
not a reason to perform a partial metadata scan. Resolve relationships for the
effective selected profile collection; shadowed definitions do not contribute
inheritance edges to that collection. Keep the catalog adapter's independent
validation of maintained data before consumer overrides.

Preserve profile templates that become usable through descendants or runtime
overrides: eager loading must not require every consumer profile to be an
independently runnable endpoint/model pair. Validate supplied fields and graph
integrity eagerly (including missing bases and cycles), and check final execution
requirements after runtime selection/overrides. Prompt default-profile references
can be checked against the assembled collection; a prompt with no default remains
valid for callers that supply a profile explicitly.

Named-definition precedence is distinct from recursive application overlays.
Use the upstream layered API where its operations fit; retain the small identity
selection step in PromptKit where needed. The upstream design must make that
boundary explicit. Do not generate temporary overlay files or serialize maps to
YAML to imitate a missing operation. Imports are not definition discovery and
need not be enabled merely because native filesystem sources support them.

### Inheritance And Presence

Recommend omission-based inheritance: absent fields inherit; explicit values
replace inherited settings, including zero, false, and empty strings where the
domain permits them. Preserve presence through inheritance, backend/framework
default application, request overrides, and outbound request construction.
Otherwise accepting `temperature: 0` during decoding would accomplish nothing.

Use ConfigKit metadata for file presence and presence-aware typed fields for
programmatic definitions. Both paths feed one typed resolver. Decode raw profiles
without effective execution defaults so defaults do not become child overrides.
Avoid a parallel reflection-based merge engine in PromptKit.

Keep provider `extra_params` replacement atomic: an explicitly supplied map
replaces the inherited map, including an empty map. This matches the useful
ability to choose a complete provider parameter set and avoids accidental mixing
of incompatible options. Do not generalize this domain rule into arbitrary
ConfigKit merge strategies.

Recommend optional scalar pointers where null means no value/inherit, while
explicit zero/false/empty is a value. For optional maps, null likewise means no
override and `{}` explicitly clears. Required identity fields remain required;
an explicitly empty model or endpoint cannot produce an executable target.
Publish these conventions deliberately rather than pretending ConfigKit's
presence metadata automatically determines domain inheritance semantics.

Resolve and memoize each effective chain once per snapshot. Retain source and
field-origin information sufficient to diagnose a bad base or inherited value;
do not claim that per-document ConfigKit metadata already represents the complete
cross-document inheritance history.

### Reloading, Errors, And Application Integration

Later definition edits take effect through a newly constructed engine. Do not add
watchers, mutable registry reloads, or a second lazy compatibility mode. Applications
choose how to swap engines and let in-flight work finish. Old prepared handles
remain associated with their original engine. Simultaneously active engines have
separate backend capacity pools; migration guidance must not imply a global limit
across a reload overlap.

Return configuration failures from construction with file/path/location and
useful definition identity where available. Recommend preserving ConfigKit's
public `LoadError` through wrapping for `errors.As`, together with a PromptKit
configuration category; do not copy its entire diagnostic API into the facade.
Domain-only relationship failures remain PromptKit errors with relevant source
locations. Start with deterministic fail-fast loading across files and ConfigKit's
existing multi-diagnostic behavior within a load, rather than inventing a batch
diagnostic subsystem. Do not expose raw credentials or add implicit logging.

Runtime failures continue to distinguish invalid requests, missing selections,
rendering/artifact/schema errors, provider/transport errors, capacity rejection,
and content-validation results. Cancellation remains effective at
PromptKit-controlled boundaries; ConfigKit's current synchronous `Load` has no
context argument, so do not promise interruption inside it or create goroutines
to simulate that guarantee.

Applications using ConfigKit can present the same underlying loading diagnostics
for their application settings and PromptKit files. They can also translate
application-owned typed settings into PromptKit profiles/backends. Do not require
applications to encode Go clients into YAML or adopt ConfigKit merely to call
PromptKit's runtime API. No full effective-configuration/explanation API is proposed
without an identified consumer requirement.

## 4. Simplification Accounting

| Current implementation | Expected disposition |
| --- | --- |
| `internal/profile/filesystem_repository.go`: metadata scans, selected strict decode, stream-end checks, malformed-file selection arbitration | Replace with eager document loads and indexed values; remove the metadata prepass and per-lookup filesystem path |
| `internal/promptdef/filesystem_repository.go`: metadata selection and repeated decoding/content opening | Replace with eager prompt loads and an ID/version index; keep domain normalization and contained content reading |
| Profile filesystem, `fs.FS`, eager, memory, and overlay repository variants | Collapse loading into source adapters plus immutable collections; preserve only interfaces that still serve a real boundary |
| `internal/profile/resolving_repository.go` | Replace per-operation traversal and zero-sentinel merging with once-per-snapshot presence-aware typed resolution; graph logic remains |
| `internal/engine` raw repository-stack assembly | Replace with explicit collection construction and precedence; engine capacity/client assembly remains |
| `internal/catalog` profile metadata extraction | Obtain explicit field information from ConfigKit; retain manifests, ownership, namespace, connection-metadata and secret checks |
| `internal/filecatalog` | Retain/reduce discovery and containment helpers actually needed by the eager loader; do not claim the entire package disappears |
| `internal/domain` source YAML tags and runtime/source coupling | Separate configuration DTOs from execution values where presence requires it; reuse domain validation rather than duplicating it |
| Direct `gopkg.in/yaml.v3` use for definition loading | Remove once all such consumers migrate; ConfigKit brings its own private YAML dependency |

New integration work is real: typed source DTOs, ConfigKit calls and error wrapping,
source-context records, collection indexing, and conversion to runtime definitions.
Some defensive copies remain necessary at public/injected-client boundaries;
ConfigKit's detached load result is not permission to expose shared engine maps.
`internal/jsonvalue` still serves programmatic values, request parameters, and
prepared schema ownership, so adoption does not eliminate it.

Maintenance savings depend on actually deleting the old lazy paths and their
special-case tests. Move generic YAML/scalar behavior coverage to ConfigKit's
ownership; PromptKit retains representative adapter tests plus domain, precedence,
snapshot, inheritance, embedded-source, error, and execution integration tests.
Do not duplicate ConfigKit's entire parser suite or discard execution regression
coverage because construction fixtures need migration.

Runtime benefit: ordinary runs no longer rediscover definition directories,
redecode definitions, reread template text, or traverse inheritance chains.
Cost: construction loads even unused definitions and retains their text and
resolved values. ConfigKit also creates metadata and projections; keep only needed
records after loading. These are reasoned tradeoffs, not measured performance
claims. Adoption is not justified by the decoder replacement alone.

### Algorithmic Efficiency

Efficiency is an explicit design criterion for both the upstream work and this
shared-library refactor. Prefer optimal or near-optimal algorithms relative to
input and produced-data size, not a blanket O(n) claim with an undefined n.

- Acquire and decode each definition once per snapshot; reuse captured content
  rather than reopening or reparsing it for metadata, selection, and validation.
- Build identity indexes in one pass, with expected constant-time map lookup.
  Do not scan whole collections for every lookup or precedence decision.
- Process inheritance graph structure in O(V + E), where V is the number of
  profiles and E the inheritance edges. Resolve shared ancestors once. Account
  separately for the size of materialized settings and retained origins; copying
  large inherited maps to many independent results necessarily costs more than V.
- Avoid whole-collection copying per layer, repeated serialization, and repeated
  prefix/history traversal. Analyze layer processing against total input nodes
  and produced values/provenance, including intentionally retained history.
- Keep configuration discovery and decoding out of ordinary execution. Copy only
  where ownership requires it; do not trade caller isolation for speed.

These are design targets, not promised complexity for every operation. Sorting
for deterministic discovery may require O(F log F) for F files, and processing
message text costs work proportional to its bytes. Construction time, retained
memory, and frequent-call costs all matter. Prefer straightforward indexes and
clear immutable ownership over speculative global caches or intricate optimizers.
The implementation plan should include focused benchmarks for consequential
repeated-work risks, such as shared-ancestor profiles and many layers, rather than
machine-dependent timing gates or benchmarks for every helper.

## 5. Additional Upstream Opportunities

Native filesystem sources and public layered loading with origins are the
prerequisites described near the beginning, not optional follow-ups. The following
remain optional and do not block adoption unless the upstream design establishes
a concrete dependency:

| Candidate | Recommendation and tradeoff |
| --- | --- |
| Generic named inheritance/presence-aware typed overlay | Consider an upstream proposal if a coherent application-neutral operation emerges; another concrete use case would strengthen the case but is not a prerequisite. PromptKit's named graph, atomic provider map, and routing rules alone do not justify a configurable general merge framework. Current fixed overlay semantics are not this API. |
| Context-aware loading | Optional for reusable cancellable construction if large-source consumers need it. Promise cooperative cancellation only; no detached reads or instant interruption of arbitrary readers. |

Do not propose a public raw AST, permissive identity-only decoder, or dynamic
subtree conversion solely to preserve PromptKit's current scanning design.
Do not upstream backend/model selection, credential timing for inference,
profile catalog reservations, or prompt-template functions.

ConfigKit and PromptKit both declare LGPL-3.0-or-later. Adoption must still update
dependency notices under the OSS policy and inspect the exact selected release
and its transitive notices. Their current Go minimums do not require a PromptKit
toolchain increase. Use a released module dependency, not a sibling-directory
replacement. This assessment proposes upstream work, but makes no upstream edits
or licensing change.

## 6. Decisions And Migration

The goal, permission to break compatibility, and these four recommendations are
the agreed working direction. Alternatives remain recorded for rationale, not as
unresolved votes. The prerequisite upstream API questions remain open and must be
settled before finalizing dependent implementation details.

1. **Loading lifetime and failure scope.** Recommend eager construction, frozen
   definitions/content, and failure for malformed files anywhere in supplied
   collections. This deletes lazy arbitration and repeated scans. Alternative:
   explicit caller-loaded reusable definition snapshots separate from engine
   construction; useful when multiple engines share data, but adds a public
   lifecycle. Retaining automatic per-run reload is possible but sacrifices much
   of the simplification and is not recommended.
2. **Source organization.** Recommend individual typed definition documents with
   explicit collection precedence. This preserves embedded catalogs and distributed
   authoring without copying application configuration machinery. Alternative:
   one keyed PromptKit bundle composed using ConfigKit imports and overlays;
   offers richer unified provenance but needs decisions about embedded sources,
   provider-map replacement, profile identity, and catalog conversion. Do not
   assume existing ConfigKit APIs already solve those issues.
3. **Presence and inheritance.** Recommend explicit-value replacement, null as
   unspecified for optional settings, and atomic provider maps as described above.
   Alternative: use ConfigKit-style recursive mapping overlays, where null clears;
   this is more uniform with application overlays but changes provider-parameter
   behavior and requires a supported cross-document mechanism. Historical
   zero-as-absence is another viable option but loses useful standardization.
4. **Diagnostic boundary.** Recommend wrapping ConfigKit's public load errors
   rather than mirroring them. Alternative: translate into a small PromptKit-owned
   diagnostic value; that limits public dependency coupling but introduces mapping
   code and risks inconsistent diagnostics. Either option must cover domain errors
   and preserve actionable source information without inventing an audit subsystem.

Expected consumer migration:

- Update definitions for ConfigKit's strict scalar rules, rejected YAML
  anchors/aliases/tags, input limits, and null/type checks. Quote prompt version
  strings rather than introducing a special numeric-version conversion merely
  for historical YAML coercion.
- Remove unintended explicit zero/false/empty values that formerly meant
  inheritance; express deliberate overrides using the new presence-aware Go
  fields and file semantics. Review provider maps for explicit replacement.
- Handle definition errors at construction, clean up unused invalid files, and
  deliberately rebuild/swap engines after definition changes.
- Keep runtime credential lookup optional, avoid baking resolved secrets into
  definition snapshots, and preserve request-scoped key overrides.
- Update maintained catalog data if strict parsing reveals incompatible syntax;
  coordinate catalog releases before the PromptKit dependency update. Preserve
  maintained profile names and catalog ownership rules, not every old scalar
  spelling. Do not validate live provider/model availability.
- Update error inspection, constructor/source options, and programmatic profile
  literals as required. Keep representative preparation, execution, inspection,
  custom backend, fallback, and embedded workflows in the migration guidance.

The subsequent feature roadmap should carry forward these choices, resolve the
upstream API details, and specify the target end state. The staged implementation
plan must sequence the prerequisite ConfigKit releases before dependent PromptKit
work, then cover deletions/replacements, contract tests, canonical documentation,
examples, attribution, and minor-release migration notes. Neither document should
become a reason to preserve obsolete compatibility machinery.