promptkit/docs/roadmap/configkit-assessment.md

ConfigKit Adoption Assessment

Status: assessment with agreed design direction; upstream API details remain to be settled before the feature roadmap and implementation plan. No runtime changes are part of this assessment.

Goal And Recommendation

Use ConfigKit to standardize PromptKit's configuration semantics and simplify its loading machinery, improving ConfigKit where a reusable capability is justified. Preserve substantive consumer capabilities, not historical loading sequences, file syntax, or Go signatures. Document intentional breaking changes and their migration paths.

Recommend adoption through an eager, immutable definition-loading boundary. ConfigKit should own YAML interpretation, strict typed decoding, presence, and source diagnostics. PromptKit should own definition identity, named profile relationships, LLM-specific rules, and execution. Do not retain the lazy loading architecture merely to minimize migration, or introduce a second generic configuration framework inside PromptKit.

The expected benefits are consistent configuration semantics for applications using both libraries and removal of repeated discovery, partial metadata parsing, lazy error arbitration, and repeated profile-chain resolution. Some of those savings come from the loading redesign rather than ConfigKit itself. ConfigKit adds ongoing value by owning the generic decoding behavior and its tests.

Prerequisite: Proposed Upstream ConfigKit Work

Design, implement, and release these two ConfigKit capabilities before the dependent PromptKit refactor. The remaining recommendations assume this upstream work; finalize their integration details against its agreed public contracts. These are selected architectural prerequisites, not a claim that adoption is technically impossible using today's reader-based API.

1. Native fs.FS sources and source-relative context. Support embedded and caller-supplied filesystems as first-class configuration sources. Define source identity, relative path resolution, ownership, and how enabled imports and overlays operate within that filesystem. This is broadly useful for libraries bundling default configuration. Passing an opened file to today's Reader supplies bytes and a label, but does not make filesystem-relative loading native to ConfigKit; that workaround is not the target architecture.
2. Public in-memory layered loading with origins. Support explicitly supplied layers without temporary files or YAML serialization round trips, retaining presence and meaningful source provenance. The upstream design must settle which inputs it accepts: filesystem/byte sources, typed values, or both; how typed values express omission; and how strict composition differs from intentional overriding. Recommend covering embedded defaults, caller sources, and programmatic values, with explicit ordering and useful conflict diagnostics.

Keep these APIs application-neutral. A list of sources alone does not implement PromptKit's whole-profile shadowing, named inheritance, or atomic provider maps. Define which layer operations ConfigKit owns and which identity/domain decisions remain in PromptKit; do not silently substitute recursive merging for an agreed replacement rule. Revisit ConfigKit's current single-overlay/deferred boundaries deliberately rather than working around them inside PromptKit.

The upstream design should show how the two capabilities work together, including source origins and efficient ownership/copying. Resolve these API questions before turning the assessment into a decision-complete PromptKit implementation plan. This assessment does not authorize code changes in either repository.

Evidence And Scope

Reviewed local baselines on 2026-09-29:

• PromptKit: 12d6d80024e80c552d122a076c22429bf6f2f6c2 (v0.11.2).
• ConfigKit: 67c03690181c5af8dde6be325abac241dae6732f (v0.4.2).

PromptKit evidence: consumer workflows, file formats, source implementation, runner, and the implementation owners identified below. ConfigKit evidence: its reviewed docs/config.md, docs/roadmap/deferred.md, lifecycle.go, source.go, and ADR 0001 in the sibling checkout. That checkout is review context, not a proposed build dependency or required directory layout.

ConfigKit currently exposes Load[T] over file, byte, reader, and defaults-only sources. It supports strict decoding, presence/origin records, defaults, recorded resolution, semantic validation, explicit imports, one selected filesystem overlay, and secret handling. Its parser and generic conversion machinery are private. It has no public arbitrary layer-composition, generic inheritance, or fs.FS source API. Its deferred scope explicitly excludes several of these generalizations; an upstream enhancement requires a deliberate decision.

This is a source-and-contract assessment, not a performance benchmark or full correctness audit. It does not claim measured allocation, latency, or line-count savings. Current code and canonical contracts remain authoritative until an approved implementation changes them.

1. Capability Inventory

Preserve the ability to…	Replaceable mechanics and representation
Define named, versioned prompts, inspect their declared inputs, and select a version explicitly	Directory scans on each operation, YAML metadata prepasses, filename layout, constructor signatures
Use inline and file-backed message templates, variables, input helpers, session templates, roles, and cache control	Timing of definition/content reads and private repository interfaces
Configure named profiles, alias/inherit another profile, and override selected settings	Zero-as-absence fields, per-call chain traversal, raw YAML decoding types
Supply embedded application defaults, operator definitions, maintained catalogs, and programmatic profiles	Repository-wrapper stacks and exact source-option API
Register custom/local backends, use built-ins or endpoint-only profiles, and control backend capacity	Configuration DTO shapes and registry assembly order
Prepare, inspect, execute, or freeze an execution for later single-claim execution	Source reloading during ordinary preparation; exact preparation error ordering
Override per-run settings, credentials, reasoning, session, and output contract; append rendered messages	Public value signatures if presence or clearer configuration separation requires revision
Supply inline/file artifacts and inject artifact readers, model clients, and HTTP clients	Assembly APIs; runtime collaborators need not be YAML-decodable
Validate output, use provider-native structured output, bounded repair, transport retries, cancellation, and observation	Internal wiring, not the substantive execution behaviors
Receive results, usage, identities, hashes, validation outcomes, and inspectable failures	Configuration error types and when definition failures appear

Preserve independent profile and prompt inspection as useful operations. They need not retain the old guarantee that engine construction has never examined other definitions. Preserve custom backend identity and capacity behavior, optional upstream credentials, and the distinction between failed generated content and operational errors. This project does not authorize silently dropping execution features to simplify the configuration layer.

2. Responsibility Mapping

Concern	Recommended owner and disposition
YAML grammar, scalar interpretation, one-document enforcement, unknown fields, typed ranges, null legality	Delegate to ConfigKit; remove corresponding PromptKit parser logic and redundant parser tests
Field presence and source coordinates	Delegate to ConfigKit; retain only the compact records needed for resolution and diagnostics
File/directory/embedded source selection and definition ID indexing	Retain a small PromptKit collection loader; ConfigKit loads documents, not a prompt catalog
Configuration filesystem sources, relative imports, and generic layer origins	Delegate to the prerequisite ConfigKit APIs; do not maintain parallel reader-based loading machinery
Definition discovery and relative message-content references	Retain PromptKit's domain resource adapters, using ConfigKit source context and standard io/fs facilities where sufficient
Profile-source priority and duplicate definition policy	Retain explicit PromptKit collection assembly; eliminate error-driven repository fallback
Named profile inheritance and backend selection	Retain a typed PromptKit resolver over the loaded collection, with explicit presence; consider generic inheritance upstream only with broader evidence
LLM setting bounds, role/session normalization, output rules, reserved provider fields	Retain shared PromptKit domain owners; ConfigKit validation callbacks may invoke them
Definition defaults	Use ConfigKit defaults where they do not erase inheritance presence; apply effective execution defaults after inheritance
Programmatic configuration	Retain typed construction; normalize to the same presence-aware internal definitions without YAML round trips
Runtime clients, observers, artifact readers, request data	Retain normal Go injection and request APIs; do not load them through ConfigKit
Credentials	Retain optional environment-name and direct request-key semantics; do not replace these with ConfigKit's required typed secret resolution
Catalog manifest, ownership, reserved names, secret-safety checks	Retain the catalog adapter; reuse the common ConfigKit profile decoder
JSON Schema resources and generated-output validation	Retain JSON Schema implementation; these are not application configuration documents
Application imports, CLI overrides, deployment overlays, secrets-directory bootstrap	Leave with consuming applications unless a concrete PromptKit need is accepted
Generic source/decoding diagnostics	Reuse ConfigKit; add only PromptKit identity/context and domain relationship diagnostics

3. Recommended Target Loading Model

Definition Snapshot

Create one immutable definition snapshot during engine construction. Discover the explicitly supplied sources, decode each definition through ConfigKit, normalize domain values, assemble identity indexes, and resolve profile chains before publishing the engine. No provider call or credential availability check belongs in this phase. Publish no partially initialized engine on failure.

Use one typed document per prompt or profile initially. Existing files need not be converted into a monolithic application configuration merely to adopt ConfigKit. Support OS directories, individual files, embedded/custom fs.FS sources, and programmatic profiles. Use the prerequisite native ConfigKit filesystem and layered-loading APIs for configuration acquisition and generic composition. PromptKit retains definition discovery and message-resource handling, using the source context supplied by ConfigKit. Exact integration is contingent on the upstream contracts; do not build a competing layer engine in PromptKit.

Load and freeze referenced message text with definitions. Validate static prompt structure and template syntax at loading, using the same template function setup as execution; render only when request variables and artifacts exist. Preserve source-relative content resolution and containment without expanding the threat model. A snapshot is a set of captured reads, not an atomic filesystem transaction.

Keep request artifacts and schema compilation at preparation. A request can replace the output contract, so an unused default schema should not need eager compilation. Prepared execution continues to freeze its effective resources. The snapshot covers definitions/message text, not environment values, request files, every schema file, or execution state.

Composition And Validation

Use explicit source categories in the existing useful priority order: programmatic profiles, operator source, application fallback, PromptKit-owned profiles, then external catalogs. Within a category, duplicate profile IDs fail; across categories, a higher-priority definition replaces the whole lower one. Prompt ID/version pairs must be unique; different versions remain available. An omitted version remains ambiguous when more than one version is present.

Strictly decode and locally validate every supplied definition, including shadowed definitions. Thus an unused malformed file is a construction failure, not a reason to perform a partial metadata scan. Resolve relationships for the effective selected profile collection; shadowed definitions do not contribute inheritance edges to that collection. Keep the catalog adapter's independent validation of maintained data before consumer overrides.

Preserve profile templates that become usable through descendants or runtime overrides: eager loading must not require every consumer profile to be an independently runnable endpoint/model pair. Validate supplied fields and graph integrity eagerly (including missing bases and cycles), and check final execution requirements after runtime selection/overrides. Prompt default-profile references can be checked against the assembled collection; a prompt with no default remains valid for callers that supply a profile explicitly.

Named-definition precedence is distinct from recursive application overlays. Use the upstream layered API where its operations fit; retain the small identity selection step in PromptKit where needed. The upstream design must make that boundary explicit. Do not generate temporary overlay files or serialize maps to YAML to imitate a missing operation. Imports are not definition discovery and need not be enabled merely because native filesystem sources support them.

Inheritance And Presence

Recommend omission-based inheritance: absent fields inherit; explicit values replace inherited settings, including zero, false, and empty strings where the domain permits them. Preserve presence through inheritance, backend/framework default application, request overrides, and outbound request construction. Otherwise accepting temperature: 0 during decoding would accomplish nothing.

Use ConfigKit metadata for file presence and presence-aware typed fields for programmatic definitions. Both paths feed one typed resolver. Decode raw profiles without effective execution defaults so defaults do not become child overrides. Avoid a parallel reflection-based merge engine in PromptKit.

Keep provider extra_params replacement atomic: an explicitly supplied map replaces the inherited map, including an empty map. This matches the useful ability to choose a complete provider parameter set and avoids accidental mixing of incompatible options. Do not generalize this domain rule into arbitrary ConfigKit merge strategies.

Recommend optional scalar pointers where null means no value/inherit, while explicit zero/false/empty is a value. For optional maps, null likewise means no override and {} explicitly clears. Required identity fields remain required; an explicitly empty model or endpoint cannot produce an executable target. Publish these conventions deliberately rather than pretending ConfigKit's presence metadata automatically determines domain inheritance semantics.

Resolve and memoize each effective chain once per snapshot. Retain source and field-origin information sufficient to diagnose a bad base or inherited value; do not claim that per-document ConfigKit metadata already represents the complete cross-document inheritance history.

Reloading, Errors, And Application Integration

Later definition edits take effect through a newly constructed engine. Do not add watchers, mutable registry reloads, or a second lazy compatibility mode. Applications choose how to swap engines and let in-flight work finish. Old prepared handles remain associated with their original engine. Simultaneously active engines have separate backend capacity pools; migration guidance must not imply a global limit across a reload overlap.

Return configuration failures from construction with file/path/location and useful definition identity where available. Recommend preserving ConfigKit's public LoadError through wrapping for errors.As, together with a PromptKit configuration category; do not copy its entire diagnostic API into the facade. Domain-only relationship failures remain PromptKit errors with relevant source locations. Start with deterministic fail-fast loading across files and ConfigKit's existing multi-diagnostic behavior within a load, rather than inventing a batch diagnostic subsystem. Do not expose raw credentials or add implicit logging.

Runtime failures continue to distinguish invalid requests, missing selections, rendering/artifact/schema errors, provider/transport errors, capacity rejection, and content-validation results. Cancellation remains effective at PromptKit-controlled boundaries; ConfigKit's current synchronous Load has no context argument, so do not promise interruption inside it or create goroutines to simulate that guarantee.

Applications using ConfigKit can present the same underlying loading diagnostics for their application settings and PromptKit files. They can also translate application-owned typed settings into PromptKit profiles/backends. Do not require applications to encode Go clients into YAML or adopt ConfigKit merely to call PromptKit's runtime API. No full effective-configuration/explanation API is proposed without an identified consumer requirement.

4. Simplification Accounting

Current implementation	Expected disposition
internal/profile/filesystem_repository.go: metadata scans, selected strict decode, stream-end checks, malformed-file selection arbitration	Replace with eager document loads and indexed values; remove the metadata prepass and per-lookup filesystem path
internal/promptdef/filesystem_repository.go: metadata selection and repeated decoding/content opening	Replace with eager prompt loads and an ID/version index; keep domain normalization and contained content reading
Profile filesystem, fs.FS, eager, memory, and overlay repository variants	Collapse loading into source adapters plus immutable collections; preserve only interfaces that still serve a real boundary
internal/profile/resolving_repository.go	Replace per-operation traversal and zero-sentinel merging with once-per-snapshot presence-aware typed resolution; graph logic remains
internal/engine raw repository-stack assembly	Replace with explicit collection construction and precedence; engine capacity/client assembly remains
internal/catalog profile metadata extraction	Obtain explicit field information from ConfigKit; retain manifests, ownership, namespace, connection-metadata and secret checks
internal/filecatalog	Retain/reduce discovery and containment helpers actually needed by the eager loader; do not claim the entire package disappears
internal/domain source YAML tags and runtime/source coupling	Separate configuration DTOs from execution values where presence requires it; reuse domain validation rather than duplicating it
Direct gopkg.in/yaml.v3 use for definition loading	Remove once all such consumers migrate; ConfigKit brings its own private YAML dependency

New integration work is real: typed source DTOs, ConfigKit calls and error wrapping, source-context records, collection indexing, and conversion to runtime definitions. Some defensive copies remain necessary at public/injected-client boundaries; ConfigKit's detached load result is not permission to expose shared engine maps. internal/jsonvalue still serves programmatic values, request parameters, and prepared schema ownership, so adoption does not eliminate it.

Maintenance savings depend on actually deleting the old lazy paths and their special-case tests. Move generic YAML/scalar behavior coverage to ConfigKit's ownership; PromptKit retains representative adapter tests plus domain, precedence, snapshot, inheritance, embedded-source, error, and execution integration tests. Do not duplicate ConfigKit's entire parser suite or discard execution regression coverage because construction fixtures need migration.

Runtime benefit: ordinary runs no longer rediscover definition directories, redecode definitions, reread template text, or traverse inheritance chains. Cost: construction loads even unused definitions and retains their text and resolved values. ConfigKit also creates metadata and projections; keep only needed records after loading. These are reasoned tradeoffs, not measured performance claims. Adoption is not justified by the decoder replacement alone.

Algorithmic Efficiency

Efficiency is an explicit design criterion for both the upstream work and this shared-library refactor. Prefer optimal or near-optimal algorithms relative to input and produced-data size, not a blanket O(n) claim with an undefined n.

• Acquire and decode each definition once per snapshot; reuse captured content rather than reopening or reparsing it for metadata, selection, and validation.
• Build identity indexes in one pass, with expected constant-time map lookup. Do not scan whole collections for every lookup or precedence decision.
• Process inheritance graph structure in O(V + E), where V is the number of profiles and E the inheritance edges. Resolve shared ancestors once. Account separately for the size of materialized settings and retained origins; copying large inherited maps to many independent results necessarily costs more than V.
• Avoid whole-collection copying per layer, repeated serialization, and repeated prefix/history traversal. Analyze layer processing against total input nodes and produced values/provenance, including intentionally retained history.
• Keep configuration discovery and decoding out of ordinary execution. Copy only where ownership requires it; do not trade caller isolation for speed.

These are design targets, not promised complexity for every operation. Sorting for deterministic discovery may require O(F log F) for F files, and processing message text costs work proportional to its bytes. Construction time, retained memory, and frequent-call costs all matter. Prefer straightforward indexes and clear immutable ownership over speculative global caches or intricate optimizers. The implementation plan should include focused benchmarks for consequential repeated-work risks, such as shared-ancestor profiles and many layers, rather than machine-dependent timing gates or benchmarks for every helper.

5. Additional Upstream Opportunities

Native filesystem sources and public layered loading with origins are the prerequisites described near the beginning, not optional follow-ups. The following remain optional and do not block adoption unless the upstream design establishes a concrete dependency:

Candidate	Recommendation and tradeoff
Generic named inheritance/presence-aware typed overlay	Consider an upstream proposal if a coherent application-neutral operation emerges; another concrete use case would strengthen the case but is not a prerequisite. PromptKit's named graph, atomic provider map, and routing rules alone do not justify a configurable general merge framework. Current fixed overlay semantics are not this API.
Context-aware loading	Optional for reusable cancellable construction if large-source consumers need it. Promise cooperative cancellation only; no detached reads or instant interruption of arbitrary readers.

Do not propose a public raw AST, permissive identity-only decoder, or dynamic subtree conversion solely to preserve PromptKit's current scanning design. Do not upstream backend/model selection, credential timing for inference, profile catalog reservations, or prompt-template functions.

ConfigKit and PromptKit both declare LGPL-3.0-or-later. Adoption must still update dependency notices under the OSS policy and inspect the exact selected release and its transitive notices. Their current Go minimums do not require a PromptKit toolchain increase. Use a released module dependency, not a sibling-directory replacement. This assessment proposes upstream work, but makes no upstream edits or licensing change.

6. Decisions And Migration

The goal, permission to break compatibility, and these four recommendations are the agreed working direction. Alternatives remain recorded for rationale, not as unresolved votes. The prerequisite upstream API questions remain open and must be settled before finalizing dependent implementation details.

1. Loading lifetime and failure scope. Recommend eager construction, frozen definitions/content, and failure for malformed files anywhere in supplied collections. This deletes lazy arbitration and repeated scans. Alternative: explicit caller-loaded reusable definition snapshots separate from engine construction; useful when multiple engines share data, but adds a public lifecycle. Retaining automatic per-run reload is possible but sacrifices much of the simplification and is not recommended.
2. Source organization. Recommend individual typed definition documents with explicit collection precedence. This preserves embedded catalogs and distributed authoring without copying application configuration machinery. Alternative: one keyed PromptKit bundle composed using ConfigKit imports and overlays; offers richer unified provenance but needs decisions about embedded sources, provider-map replacement, profile identity, and catalog conversion. Do not assume existing ConfigKit APIs already solve those issues.
3. Presence and inheritance. Recommend explicit-value replacement, null as unspecified for optional settings, and atomic provider maps as described above. Alternative: use ConfigKit-style recursive mapping overlays, where null clears; this is more uniform with application overlays but changes provider-parameter behavior and requires a supported cross-document mechanism. Historical zero-as-absence is another viable option but loses useful standardization.
4. Diagnostic boundary. Recommend wrapping ConfigKit's public load errors rather than mirroring them. Alternative: translate into a small PromptKit-owned diagnostic value; that limits public dependency coupling but introduces mapping code and risks inconsistent diagnostics. Either option must cover domain errors and preserve actionable source information without inventing an audit subsystem.

Expected consumer migration:

• Update definitions for ConfigKit's strict scalar rules, rejected YAML anchors/aliases/tags, input limits, and null/type checks. Quote prompt version strings rather than introducing a special numeric-version conversion merely for historical YAML coercion.
• Remove unintended explicit zero/false/empty values that formerly meant inheritance; express deliberate overrides using the new presence-aware Go fields and file semantics. Review provider maps for explicit replacement.
• Handle definition errors at construction, clean up unused invalid files, and deliberately rebuild/swap engines after definition changes.
• Keep runtime credential lookup optional, avoid baking resolved secrets into definition snapshots, and preserve request-scoped key overrides.
• Update maintained catalog data if strict parsing reveals incompatible syntax; coordinate catalog releases before the PromptKit dependency update. Preserve maintained profile names and catalog ownership rules, not every old scalar spelling. Do not validate live provider/model availability.
• Update error inspection, constructor/source options, and programmatic profile literals as required. Keep representative preparation, execution, inspection, custom backend, fallback, and embedded workflows in the migration guidance.

The subsequent feature roadmap should carry forward these choices, resolve the upstream API details, and specify the target end state. The staged implementation plan must sequence the prerequisite ConfigKit releases before dependent PromptKit work, then cover deletions/replacements, contract tests, canonical documentation, examples, attribution, and minor-release migration notes. Neither document should become a reason to preserve obsolete compatibility machinery.