From c7bdaa076b596cc812f3a747efa63a9f1e3a5f9d Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 29 Sep 2026 14:11:30 +0000 Subject: [PATCH] Assess ConfigKit adoption for PromptKit --- docs/roadmap/configkit-assessment.md | 333 +++++++++++++++++++++++++++ 1 file changed, 333 insertions(+) create mode 100644 docs/roadmap/configkit-assessment.md diff --git a/docs/roadmap/configkit-assessment.md b/docs/roadmap/configkit-assessment.md new file mode 100644 index 0000000..61ca449 --- /dev/null +++ b/docs/roadmap/configkit-assessment.md @@ -0,0 +1,333 @@ +# ConfigKit Adoption Assessment + +Status: proposed for discussion, not an approved feature specification or an +implementation plan. No runtime changes are part of this assessment. + +## Goal And Recommendation + +Use ConfigKit to standardize PromptKit's configuration semantics and simplify +its loading machinery, improving ConfigKit where a reusable capability is +justified. Preserve substantive consumer capabilities, not historical loading +sequences, file syntax, or Go signatures. Document intentional breaking changes +and their migration paths. + +Recommend adoption through an eager, immutable definition-loading boundary. +ConfigKit should own YAML interpretation, strict typed decoding, presence, and +source diagnostics. PromptKit should own definition identity, named profile +relationships, LLM-specific rules, and execution. Do not retain the lazy loading +architecture merely to minimize migration, or introduce a second generic +configuration framework inside PromptKit. + +The expected benefits are consistent configuration semantics for applications +using both libraries and removal of repeated discovery, partial metadata parsing, +lazy error arbitration, and repeated profile-chain resolution. Some of those +savings come from the loading redesign rather than ConfigKit itself. ConfigKit +adds ongoing value by owning the generic decoding behavior and its tests. + +## Evidence And Scope + +Reviewed local baselines on 2026-09-29: + +- PromptKit: `12d6d80024e80c552d122a076c22429bf6f2f6c2` (v0.11.2). +- ConfigKit: `67c03690181c5af8dde6be325abac241dae6732f` (v0.4.2). + +PromptKit evidence: [consumer workflows](../consumers/pkg-promptkit.md), +[file formats](../formats.md), [source implementation](../internal/sources.md), +[runner](../internal/runner.md), and the implementation owners identified below. +ConfigKit evidence: its reviewed `docs/config.md`, `docs/roadmap/deferred.md`, +`lifecycle.go`, `source.go`, and ADR 0001 in the sibling checkout. That checkout +is review context, not a proposed build dependency or required directory layout. + +ConfigKit currently exposes `Load[T]` over file, byte, reader, and defaults-only +sources. It supports strict decoding, presence/origin records, defaults, +recorded resolution, semantic validation, explicit imports, one selected +filesystem overlay, and secret handling. Its parser and generic conversion +machinery are private. It has no public arbitrary layer-composition, generic +inheritance, or `fs.FS` source API. Its deferred scope explicitly excludes several +of these generalizations; an upstream enhancement requires a deliberate decision. + +This is a source-and-contract assessment, not a performance benchmark or full +correctness audit. It does not claim measured allocation, latency, or line-count +savings. Current code and canonical contracts remain authoritative until an +approved implementation changes them. + +## 1. Capability Inventory + +| Preserve the ability to… | Replaceable mechanics and representation | +| --- | --- | +| Define named, versioned prompts, inspect their declared inputs, and select a version explicitly | Directory scans on each operation, YAML metadata prepasses, filename layout, constructor signatures | +| Use inline and file-backed message templates, variables, input helpers, session templates, roles, and cache control | Timing of definition/content reads and private repository interfaces | +| Configure named profiles, alias/inherit another profile, and override selected settings | Zero-as-absence fields, per-call chain traversal, raw YAML decoding types | +| Supply embedded application defaults, operator definitions, maintained catalogs, and programmatic profiles | Repository-wrapper stacks and exact source-option API | +| Register custom/local backends, use built-ins or endpoint-only profiles, and control backend capacity | Configuration DTO shapes and registry assembly order | +| Prepare, inspect, execute, or freeze an execution for later single-claim execution | Source reloading during ordinary preparation; exact preparation error ordering | +| Override per-run settings, credentials, reasoning, session, and output contract; append rendered messages | Public value signatures if presence or clearer configuration separation requires revision | +| Supply inline/file artifacts and inject artifact readers, model clients, and HTTP clients | Assembly APIs; runtime collaborators need not be YAML-decodable | +| Validate output, use provider-native structured output, bounded repair, transport retries, cancellation, and observation | Internal wiring, not the substantive execution behaviors | +| Receive results, usage, identities, hashes, validation outcomes, and inspectable failures | Configuration error types and when definition failures appear | + +Preserve independent profile and prompt inspection as useful operations. They +need not retain the old guarantee that engine construction has never examined +other definitions. Preserve custom backend identity and capacity behavior, +optional upstream credentials, and the distinction between failed generated +content and operational errors. This project does not authorize silently +dropping execution features to simplify the configuration layer. + +## 2. Responsibility Mapping + +| Concern | Recommended owner and disposition | +| --- | --- | +| YAML grammar, scalar interpretation, one-document enforcement, unknown fields, typed ranges, null legality | Delegate to ConfigKit; remove corresponding PromptKit parser logic and redundant parser tests | +| Field presence and source coordinates | Delegate to ConfigKit; retain only the compact records needed for resolution and diagnostics | +| File/directory/embedded source selection and definition ID indexing | Retain a small PromptKit collection loader; ConfigKit loads documents, not a prompt catalog | +| Filesystem traversal and relative content references | Retain PromptKit's source-boundary adapters; use standard `io/fs` facilities where sufficient | +| Profile-source priority and duplicate definition policy | Retain explicit PromptKit collection assembly; eliminate error-driven repository fallback | +| Named profile inheritance and backend selection | Retain a typed PromptKit resolver over the loaded collection, with explicit presence; consider generic inheritance upstream only with broader evidence | +| LLM setting bounds, role/session normalization, output rules, reserved provider fields | Retain shared PromptKit domain owners; ConfigKit validation callbacks may invoke them | +| Definition defaults | Use ConfigKit defaults where they do not erase inheritance presence; apply effective execution defaults after inheritance | +| Programmatic configuration | Retain typed construction; normalize to the same presence-aware internal definitions without YAML round trips | +| Runtime clients, observers, artifact readers, request data | Retain normal Go injection and request APIs; do not load them through ConfigKit | +| Credentials | Retain optional environment-name and direct request-key semantics; do not replace these with ConfigKit's required typed secret resolution | +| Catalog manifest, ownership, reserved names, secret-safety checks | Retain the catalog adapter; reuse the common ConfigKit profile decoder | +| JSON Schema resources and generated-output validation | Retain JSON Schema implementation; these are not application configuration documents | +| Application imports, CLI overrides, deployment overlays, secrets-directory bootstrap | Leave with consuming applications unless a concrete PromptKit need is accepted | +| Generic source/decoding diagnostics | Reuse ConfigKit; add only PromptKit identity/context and domain relationship diagnostics | + +## 3. Recommended Target Loading Model + +### Definition Snapshot + +Create one immutable definition snapshot during engine construction. Discover +the explicitly supplied sources, decode each definition through ConfigKit, +normalize domain values, assemble identity indexes, and resolve profile chains +before publishing the engine. No provider call or credential availability check +belongs in this phase. Publish no partially initialized engine on failure. + +Use one typed document per prompt or profile initially. Existing files need not +be converted into a monolithic application configuration merely to adopt +ConfigKit. Support OS directories, individual files, embedded/custom `fs.FS` +sources, and programmatic profiles. For `fs.FS`, a small adapter can open a file +and supply `configkit.Reader` with a safe source label; PromptKit retains root +and path context for referenced message files. Close handles owned by the adapter. +ConfigKit's existing public API is sufficient for this design. + +Load and freeze referenced message text with definitions. Validate static prompt +structure and template syntax at loading, using the same template function setup +as execution; render only when request variables and artifacts exist. Preserve +source-relative content resolution and containment without expanding the threat +model. A snapshot is a set of captured reads, not an atomic filesystem transaction. + +Keep request artifacts and schema compilation at preparation. A request can +replace the output contract, so an unused default schema should not need eager +compilation. Prepared execution continues to freeze its effective resources. +The snapshot covers definitions/message text, not environment values, request +files, every schema file, or execution state. + +### Composition And Validation + +Use explicit source categories in the existing useful priority order: +programmatic profiles, operator source, application fallback, PromptKit-owned +profiles, then external catalogs. Within a category, duplicate profile IDs fail; +across categories, a higher-priority definition replaces the whole lower one. +Prompt ID/version pairs must be unique; different versions remain available. +An omitted version remains ambiguous when more than one version is present. + +Strictly decode and locally validate every supplied definition, including +shadowed definitions. Thus an unused malformed file is a construction failure, +not a reason to perform a partial metadata scan. Resolve relationships for the +effective selected profile collection; shadowed definitions do not contribute +inheritance edges to that collection. Keep the catalog adapter's independent +validation of maintained data before consumer overrides. + +Preserve profile templates that become usable through descendants or runtime +overrides: eager loading must not require every consumer profile to be an +independently runnable endpoint/model pair. Validate supplied fields and graph +integrity eagerly (including missing bases and cycles), and check final execution +requirements after runtime selection/overrides. Prompt default-profile references +can be checked against the assembled collection; a prompt with no default remains +valid for callers that supply a profile explicitly. + +This is named-definition precedence, not ConfigKit's application overlay system. +Do not generate temporary overlay files or serialize maps to YAML to imitate a +missing layer API. ConfigKit imports are also not directory discovery: their +strict composition has different semantics and need not be enabled here. + +### Inheritance And Presence + +Recommend omission-based inheritance: absent fields inherit; explicit values +replace inherited settings, including zero, false, and empty strings where the +domain permits them. Preserve presence through inheritance, backend/framework +default application, request overrides, and outbound request construction. +Otherwise accepting `temperature: 0` during decoding would accomplish nothing. + +Use ConfigKit metadata for file presence and presence-aware typed fields for +programmatic definitions. Both paths feed one typed resolver. Decode raw profiles +without effective execution defaults so defaults do not become child overrides. +Avoid a parallel reflection-based merge engine in PromptKit. + +Keep provider `extra_params` replacement atomic: an explicitly supplied map +replaces the inherited map, including an empty map. This matches the useful +ability to choose a complete provider parameter set and avoids accidental mixing +of incompatible options. Do not generalize this domain rule into arbitrary +ConfigKit merge strategies. + +Recommend optional scalar pointers where null means no value/inherit, while +explicit zero/false/empty is a value. For optional maps, null likewise means no +override and `{}` explicitly clears. Required identity fields remain required; +an explicitly empty model or endpoint cannot produce an executable target. +Publish these conventions deliberately rather than pretending ConfigKit's +presence metadata automatically determines domain inheritance semantics. + +Resolve and memoize each effective chain once per snapshot. Retain source and +field-origin information sufficient to diagnose a bad base or inherited value; +do not claim that per-document ConfigKit metadata already represents the complete +cross-document inheritance history. + +### Reloading, Errors, And Application Integration + +Later definition edits take effect through a newly constructed engine. Do not add +watchers, mutable registry reloads, or a second lazy compatibility mode. Applications +choose how to swap engines and let in-flight work finish. Old prepared handles +remain associated with their original engine. Simultaneously active engines have +separate backend capacity pools; migration guidance must not imply a global limit +across a reload overlap. + +Return configuration failures from construction with file/path/location and +useful definition identity where available. Recommend preserving ConfigKit's +public `LoadError` through wrapping for `errors.As`, together with a PromptKit +configuration category; do not copy its entire diagnostic API into the facade. +Domain-only relationship failures remain PromptKit errors with relevant source +locations. Start with deterministic fail-fast loading across files and ConfigKit's +existing multi-diagnostic behavior within a load, rather than inventing a batch +diagnostic subsystem. Do not expose raw credentials or add implicit logging. + +Runtime failures continue to distinguish invalid requests, missing selections, +rendering/artifact/schema errors, provider/transport errors, capacity rejection, +and content-validation results. Cancellation remains effective at +PromptKit-controlled boundaries; ConfigKit's current synchronous `Load` has no +context argument, so do not promise interruption inside it or create goroutines +to simulate that guarantee. + +Applications using ConfigKit can present the same underlying loading diagnostics +for their application settings and PromptKit files. They can also translate +application-owned typed settings into PromptKit profiles/backends. Do not require +applications to encode Go clients into YAML or adopt ConfigKit merely to call +PromptKit's runtime API. No full effective-configuration/explanation API is proposed +without an identified consumer requirement. + +## 4. Simplification Accounting + +| Current implementation | Expected disposition | +| --- | --- | +| `internal/profile/filesystem_repository.go`: metadata scans, selected strict decode, stream-end checks, malformed-file selection arbitration | Replace with eager document loads and indexed values; remove the metadata prepass and per-lookup filesystem path | +| `internal/promptdef/filesystem_repository.go`: metadata selection and repeated decoding/content opening | Replace with eager prompt loads and an ID/version index; keep domain normalization and contained content reading | +| Profile filesystem, `fs.FS`, eager, memory, and overlay repository variants | Collapse loading into source adapters plus immutable collections; preserve only interfaces that still serve a real boundary | +| `internal/profile/resolving_repository.go` | Replace per-operation traversal and zero-sentinel merging with once-per-snapshot presence-aware typed resolution; graph logic remains | +| `internal/engine` raw repository-stack assembly | Replace with explicit collection construction and precedence; engine capacity/client assembly remains | +| `internal/catalog` profile metadata extraction | Obtain explicit field information from ConfigKit; retain manifests, ownership, namespace, connection-metadata and secret checks | +| `internal/filecatalog` | Retain/reduce discovery and containment helpers actually needed by the eager loader; do not claim the entire package disappears | +| `internal/domain` source YAML tags and runtime/source coupling | Separate configuration DTOs from execution values where presence requires it; reuse domain validation rather than duplicating it | +| Direct `gopkg.in/yaml.v3` use for definition loading | Remove once all such consumers migrate; ConfigKit brings its own private YAML dependency | + +New integration work is real: typed source DTOs, ConfigKit calls and error wrapping, +source-context records, collection indexing, and conversion to runtime definitions. +Some defensive copies remain necessary at public/injected-client boundaries; +ConfigKit's detached load result is not permission to expose shared engine maps. +`internal/jsonvalue` still serves programmatic values, request parameters, and +prepared schema ownership, so adoption does not eliminate it. + +Maintenance savings depend on actually deleting the old lazy paths and their +special-case tests. Move generic YAML/scalar behavior coverage to ConfigKit's +ownership; PromptKit retains representative adapter tests plus domain, precedence, +snapshot, inheritance, embedded-source, error, and execution integration tests. +Do not duplicate ConfigKit's entire parser suite or discard execution regression +coverage because construction fixtures need migration. + +Runtime benefit: ordinary runs no longer rediscover definition directories, +redecode definitions, reread template text, or traverse inheritance chains. +Cost: construction loads even unused definitions and retains their text and +resolved values. ConfigKit also creates metadata and projections; keep only needed +records after loading. These are reasoned tradeoffs, not measured performance +claims. Adoption is not justified by the decoder replacement alone. + +## 5. Upstream Opportunities + +No ConfigKit enhancement is a demonstrated prerequisite for the recommended +per-document snapshot design. Do not manufacture one to move a small adapter +upstream. Reconsider the following if the agreed roadmap reveals concrete reuse: + +| Candidate | Recommendation and tradeoff | +| --- | --- | +| Native `fs.FS` sources with source-relative resource context | Most plausible optional enhancement: embedded configurations are useful across Go libraries. Reader sources suffice today. A fuller design must define imports/overlays in that filesystem rather than merely add a constructor or leak parser internals. | +| Public in-memory layered loading with origins | Optional only if multiple consumers need application defaults and file/embedded layers composed together. It could remove adapters in those consumers, but is not necessary for whole-definition catalog shadowing. Review against ConfigKit's deliberate single-overlay scope. | +| Generic named inheritance/presence-aware typed overlay | Consider an upstream proposal if a coherent application-neutral operation emerges; another concrete use case would strengthen the case but is not a prerequisite. PromptKit's named graph, atomic provider map, and routing rules alone do not justify a configurable general merge framework. Current fixed overlay semantics are not this API. | +| Context-aware loading | Optional for reusable cancellable construction if large-source consumers need it. Promise cooperative cancellation only; no detached reads or instant interruption of arbitrary readers. | + +Do not propose a public raw AST, permissive identity-only decoder, or dynamic +subtree conversion solely to preserve PromptKit's current scanning design. +Do not upstream backend/model selection, credential timing for inference, +profile catalog reservations, or prompt-template functions. + +ConfigKit and PromptKit both declare LGPL-3.0-or-later. Adoption must still update +dependency notices under the OSS policy and inspect the exact selected release +and its transitive notices. Their current Go minimums do not require a PromptKit +toolchain increase. Use a released module dependency, not a sibling-directory +replacement. This review proposes no licensing change or upstream edits. + +## 6. Decisions And Migration + +The goal and permission to break compatibility are agreed. The following specific +choices are recommendations for discussion, not inferred approvals: + +1. **Loading lifetime and failure scope.** Recommend eager construction, frozen + definitions/content, and failure for malformed files anywhere in supplied + collections. This deletes lazy arbitration and repeated scans. Alternative: + explicit caller-loaded reusable definition snapshots separate from engine + construction; useful when multiple engines share data, but adds a public + lifecycle. Retaining automatic per-run reload is possible but sacrifices much + of the simplification and is not recommended. +2. **Source organization.** Recommend individual typed definition documents with + explicit collection precedence. This preserves embedded catalogs and distributed + authoring without copying application configuration machinery. Alternative: + one keyed PromptKit bundle composed using ConfigKit imports and overlays; + offers richer unified provenance but needs decisions about embedded sources, + provider-map replacement, profile identity, and catalog conversion. Do not + assume existing ConfigKit APIs already solve those issues. +3. **Presence and inheritance.** Recommend explicit-value replacement, null as + unspecified for optional settings, and atomic provider maps as described above. + Alternative: use ConfigKit-style recursive mapping overlays, where null clears; + this is more uniform with application overlays but changes provider-parameter + behavior and requires a supported cross-document mechanism. Historical + zero-as-absence is another viable option but loses useful standardization. +4. **Diagnostic boundary.** Recommend wrapping ConfigKit's public load errors + rather than mirroring them. Alternative: translate into a small PromptKit-owned + diagnostic value; that limits public dependency coupling but introduces mapping + code and risks inconsistent diagnostics. Either option must cover domain errors + and preserve actionable source information without inventing an audit subsystem. + +Expected consumer migration: + +- Update definitions for ConfigKit's strict scalar rules, rejected YAML + anchors/aliases/tags, input limits, and null/type checks. Quote prompt version + strings rather than introducing a special numeric-version conversion merely + for historical YAML coercion. +- Remove unintended explicit zero/false/empty values that formerly meant + inheritance; express deliberate overrides using the new presence-aware Go + fields and file semantics. Review provider maps for explicit replacement. +- Handle definition errors at construction, clean up unused invalid files, and + deliberately rebuild/swap engines after definition changes. +- Keep runtime credential lookup optional, avoid baking resolved secrets into + definition snapshots, and preserve request-scoped key overrides. +- Update maintained catalog data if strict parsing reveals incompatible syntax; + coordinate catalog releases before the PromptKit dependency update. Preserve + maintained profile names and catalog ownership rules, not every old scalar + spelling. Do not validate live provider/model availability. +- Update error inspection, constructor/source options, and programmatic profile + literals as required. Keep representative preparation, execution, inspection, + custom backend, fallback, and embedded workflows in the migration guidance. + +The subsequent feature roadmap should settle these choices and specify the target +end state. The staged implementation plan can then sequence any accepted upstream +work, PromptKit deletions/replacements, contract tests, canonical documentation, +examples, attribution, and minor-release migration notes. Neither document should +become a reason to preserve obsolete compatibility machinery.