Assess ConfigKit adoption for PromptKit

docs/roadmap/configkit-assessment.md

@@ -0,0 +1,333 @@
+# ConfigKit Adoption Assessment
+
+Status: proposed for discussion, not an approved feature specification or an
+implementation plan. No runtime changes are part of this assessment.
+
+## Goal And Recommendation
+
+Use ConfigKit to standardize PromptKit's configuration semantics and simplify
+its loading machinery, improving ConfigKit where a reusable capability is
+justified. Preserve substantive consumer capabilities, not historical loading
+sequences, file syntax, or Go signatures. Document intentional breaking changes
+and their migration paths.
+
+Recommend adoption through an eager, immutable definition-loading boundary.
+ConfigKit should own YAML interpretation, strict typed decoding, presence, and
+source diagnostics. PromptKit should own definition identity, named profile
+relationships, LLM-specific rules, and execution. Do not retain the lazy loading
+architecture merely to minimize migration, or introduce a second generic
+configuration framework inside PromptKit.
+
+The expected benefits are consistent configuration semantics for applications
+using both libraries and removal of repeated discovery, partial metadata parsing,
+lazy error arbitration, and repeated profile-chain resolution. Some of those
+savings come from the loading redesign rather than ConfigKit itself. ConfigKit
+adds ongoing value by owning the generic decoding behavior and its tests.
+
+## Evidence And Scope
+
+Reviewed local baselines on 2026-09-29:
+
+- PromptKit: `12d6d80024e80c552d122a076c22429bf6f2f6c2` (v0.11.2).
+- ConfigKit: `67c03690181c5af8dde6be325abac241dae6732f` (v0.4.2).
+
+PromptKit evidence: [consumer workflows](../consumers/pkg-promptkit.md),
+[file formats](../formats.md), [source implementation](../internal/sources.md),
+[runner](../internal/runner.md), and the implementation owners identified below.
+ConfigKit evidence: its reviewed `docs/config.md`, `docs/roadmap/deferred.md`,
+`lifecycle.go`, `source.go`, and ADR 0001 in the sibling checkout. That checkout
+is review context, not a proposed build dependency or required directory layout.
+
+ConfigKit currently exposes `Load[T]` over file, byte, reader, and defaults-only
+sources. It supports strict decoding, presence/origin records, defaults,
+recorded resolution, semantic validation, explicit imports, one selected
+filesystem overlay, and secret handling. Its parser and generic conversion
+machinery are private. It has no public arbitrary layer-composition, generic
+inheritance, or `fs.FS` source API. Its deferred scope explicitly excludes several
+of these generalizations; an upstream enhancement requires a deliberate decision.
+
+This is a source-and-contract assessment, not a performance benchmark or full
+correctness audit. It does not claim measured allocation, latency, or line-count
+savings. Current code and canonical contracts remain authoritative until an
+approved implementation changes them.
+
+## 1. Capability Inventory
+
+| Preserve the ability to… | Replaceable mechanics and representation |
+| --- | --- |
+| Define named, versioned prompts, inspect their declared inputs, and select a version explicitly | Directory scans on each operation, YAML metadata prepasses, filename layout, constructor signatures |
+| Use inline and file-backed message templates, variables, input helpers, session templates, roles, and cache control | Timing of definition/content reads and private repository interfaces |
+| Configure named profiles, alias/inherit another profile, and override selected settings | Zero-as-absence fields, per-call chain traversal, raw YAML decoding types |
+| Supply embedded application defaults, operator definitions, maintained catalogs, and programmatic profiles | Repository-wrapper stacks and exact source-option API |
+| Register custom/local backends, use built-ins or endpoint-only profiles, and control backend capacity | Configuration DTO shapes and registry assembly order |
+| Prepare, inspect, execute, or freeze an execution for later single-claim execution | Source reloading during ordinary preparation; exact preparation error ordering |
+| Override per-run settings, credentials, reasoning, session, and output contract; append rendered messages | Public value signatures if presence or clearer configuration separation requires revision |
+| Supply inline/file artifacts and inject artifact readers, model clients, and HTTP clients | Assembly APIs; runtime collaborators need not be YAML-decodable |
+| Validate output, use provider-native structured output, bounded repair, transport retries, cancellation, and observation | Internal wiring, not the substantive execution behaviors |
+| Receive results, usage, identities, hashes, validation outcomes, and inspectable failures | Configuration error types and when definition failures appear |
+
+Preserve independent profile and prompt inspection as useful operations. They
+need not retain the old guarantee that engine construction has never examined
+other definitions. Preserve custom backend identity and capacity behavior,
+optional upstream credentials, and the distinction between failed generated
+content and operational errors. This project does not authorize silently
+dropping execution features to simplify the configuration layer.
+
+## 2. Responsibility Mapping
+
+| Concern | Recommended owner and disposition |
+| --- | --- |
+| YAML grammar, scalar interpretation, one-document enforcement, unknown fields, typed ranges, null legality | Delegate to ConfigKit; remove corresponding PromptKit parser logic and redundant parser tests |
+| Field presence and source coordinates | Delegate to ConfigKit; retain only the compact records needed for resolution and diagnostics |
+| File/directory/embedded source selection and definition ID indexing | Retain a small PromptKit collection loader; ConfigKit loads documents, not a prompt catalog |
+| Filesystem traversal and relative content references | Retain PromptKit's source-boundary adapters; use standard `io/fs` facilities where sufficient |
+| Profile-source priority and duplicate definition policy | Retain explicit PromptKit collection assembly; eliminate error-driven repository fallback |
+| Named profile inheritance and backend selection | Retain a typed PromptKit resolver over the loaded collection, with explicit presence; consider generic inheritance upstream only with broader evidence |
+| LLM setting bounds, role/session normalization, output rules, reserved provider fields | Retain shared PromptKit domain owners; ConfigKit validation callbacks may invoke them |
+| Definition defaults | Use ConfigKit defaults where they do not erase inheritance presence; apply effective execution defaults after inheritance |
+| Programmatic configuration | Retain typed construction; normalize to the same presence-aware internal definitions without YAML round trips |
+| Runtime clients, observers, artifact readers, request data | Retain normal Go injection and request APIs; do not load them through ConfigKit |
+| Credentials | Retain optional environment-name and direct request-key semantics; do not replace these with ConfigKit's required typed secret resolution |
+| Catalog manifest, ownership, reserved names, secret-safety checks | Retain the catalog adapter; reuse the common ConfigKit profile decoder |
+| JSON Schema resources and generated-output validation | Retain JSON Schema implementation; these are not application configuration documents |
+| Application imports, CLI overrides, deployment overlays, secrets-directory bootstrap | Leave with consuming applications unless a concrete PromptKit need is accepted |
+| Generic source/decoding diagnostics | Reuse ConfigKit; add only PromptKit identity/context and domain relationship diagnostics |
+
+## 3. Recommended Target Loading Model
+
+### Definition Snapshot
+
+Create one immutable definition snapshot during engine construction. Discover
+the explicitly supplied sources, decode each definition through ConfigKit,
+normalize domain values, assemble identity indexes, and resolve profile chains
+before publishing the engine. No provider call or credential availability check
+belongs in this phase. Publish no partially initialized engine on failure.
+
+Use one typed document per prompt or profile initially. Existing files need not
+be converted into a monolithic application configuration merely to adopt
+ConfigKit. Support OS directories, individual files, embedded/custom `fs.FS`
+sources, and programmatic profiles. For `fs.FS`, a small adapter can open a file
+and supply `configkit.Reader` with a safe source label; PromptKit retains root
+and path context for referenced message files. Close handles owned by the adapter.
+ConfigKit's existing public API is sufficient for this design.
+
+Load and freeze referenced message text with definitions. Validate static prompt
+structure and template syntax at loading, using the same template function setup
+as execution; render only when request variables and artifacts exist. Preserve
+source-relative content resolution and containment without expanding the threat
+model. A snapshot is a set of captured reads, not an atomic filesystem transaction.
+
+Keep request artifacts and schema compilation at preparation. A request can
+replace the output contract, so an unused default schema should not need eager
+compilation. Prepared execution continues to freeze its effective resources.
+The snapshot covers definitions/message text, not environment values, request
+files, every schema file, or execution state.
+
+### Composition And Validation
+
+Use explicit source categories in the existing useful priority order:
+programmatic profiles, operator source, application fallback, PromptKit-owned
+profiles, then external catalogs. Within a category, duplicate profile IDs fail;
+across categories, a higher-priority definition replaces the whole lower one.
+Prompt ID/version pairs must be unique; different versions remain available.
+An omitted version remains ambiguous when more than one version is present.
+
+Strictly decode and locally validate every supplied definition, including
+shadowed definitions. Thus an unused malformed file is a construction failure,
+not a reason to perform a partial metadata scan. Resolve relationships for the
+effective selected profile collection; shadowed definitions do not contribute
+inheritance edges to that collection. Keep the catalog adapter's independent
+validation of maintained data before consumer overrides.
+
+Preserve profile templates that become usable through descendants or runtime
+overrides: eager loading must not require every consumer profile to be an
+independently runnable endpoint/model pair. Validate supplied fields and graph
+integrity eagerly (including missing bases and cycles), and check final execution
+requirements after runtime selection/overrides. Prompt default-profile references
+can be checked against the assembled collection; a prompt with no default remains
+valid for callers that supply a profile explicitly.
+
+This is named-definition precedence, not ConfigKit's application overlay system.
+Do not generate temporary overlay files or serialize maps to YAML to imitate a
+missing layer API. ConfigKit imports are also not directory discovery: their
+strict composition has different semantics and need not be enabled here.
+
+### Inheritance And Presence
+
+Recommend omission-based inheritance: absent fields inherit; explicit values
+replace inherited settings, including zero, false, and empty strings where the
+domain permits them. Preserve presence through inheritance, backend/framework
+default application, request overrides, and outbound request construction.
+Otherwise accepting `temperature: 0` during decoding would accomplish nothing.
+
+Use ConfigKit metadata for file presence and presence-aware typed fields for
+programmatic definitions. Both paths feed one typed resolver. Decode raw profiles
+without effective execution defaults so defaults do not become child overrides.
+Avoid a parallel reflection-based merge engine in PromptKit.
+
+Keep provider `extra_params` replacement atomic: an explicitly supplied map
+replaces the inherited map, including an empty map. This matches the useful
+ability to choose a complete provider parameter set and avoids accidental mixing
+of incompatible options. Do not generalize this domain rule into arbitrary
+ConfigKit merge strategies.
+
+Recommend optional scalar pointers where null means no value/inherit, while
+explicit zero/false/empty is a value. For optional maps, null likewise means no
+override and `{}` explicitly clears. Required identity fields remain required;
+an explicitly empty model or endpoint cannot produce an executable target.
+Publish these conventions deliberately rather than pretending ConfigKit's
+presence metadata automatically determines domain inheritance semantics.
+
+Resolve and memoize each effective chain once per snapshot. Retain source and
+field-origin information sufficient to diagnose a bad base or inherited value;
+do not claim that per-document ConfigKit metadata already represents the complete
+cross-document inheritance history.
+
+### Reloading, Errors, And Application Integration
+
+Later definition edits take effect through a newly constructed engine. Do not add
+watchers, mutable registry reloads, or a second lazy compatibility mode. Applications
+choose how to swap engines and let in-flight work finish. Old prepared handles
+remain associated with their original engine. Simultaneously active engines have
+separate backend capacity pools; migration guidance must not imply a global limit
+across a reload overlap.
+
+Return configuration failures from construction with file/path/location and
+useful definition identity where available. Recommend preserving ConfigKit's
+public `LoadError` through wrapping for `errors.As`, together with a PromptKit
+configuration category; do not copy its entire diagnostic API into the facade.
+Domain-only relationship failures remain PromptKit errors with relevant source
+locations. Start with deterministic fail-fast loading across files and ConfigKit's
+existing multi-diagnostic behavior within a load, rather than inventing a batch
+diagnostic subsystem. Do not expose raw credentials or add implicit logging.
+
+Runtime failures continue to distinguish invalid requests, missing selections,
+rendering/artifact/schema errors, provider/transport errors, capacity rejection,
+and content-validation results. Cancellation remains effective at
+PromptKit-controlled boundaries; ConfigKit's current synchronous `Load` has no
+context argument, so do not promise interruption inside it or create goroutines
+to simulate that guarantee.
+
+Applications using ConfigKit can present the same underlying loading diagnostics
+for their application settings and PromptKit files. They can also translate
+application-owned typed settings into PromptKit profiles/backends. Do not require
+applications to encode Go clients into YAML or adopt ConfigKit merely to call
+PromptKit's runtime API. No full effective-configuration/explanation API is proposed
+without an identified consumer requirement.
+
+## 4. Simplification Accounting
+
+| Current implementation | Expected disposition |
+| --- | --- |
+| `internal/profile/filesystem_repository.go`: metadata scans, selected strict decode, stream-end checks, malformed-file selection arbitration | Replace with eager document loads and indexed values; remove the metadata prepass and per-lookup filesystem path |
+| `internal/promptdef/filesystem_repository.go`: metadata selection and repeated decoding/content opening | Replace with eager prompt loads and an ID/version index; keep domain normalization and contained content reading |
+| Profile filesystem, `fs.FS`, eager, memory, and overlay repository variants | Collapse loading into source adapters plus immutable collections; preserve only interfaces that still serve a real boundary |
+| `internal/profile/resolving_repository.go` | Replace per-operation traversal and zero-sentinel merging with once-per-snapshot presence-aware typed resolution; graph logic remains |
+| `internal/engine` raw repository-stack assembly | Replace with explicit collection construction and precedence; engine capacity/client assembly remains |
+| `internal/catalog` profile metadata extraction | Obtain explicit field information from ConfigKit; retain manifests, ownership, namespace, connection-metadata and secret checks |
+| `internal/filecatalog` | Retain/reduce discovery and containment helpers actually needed by the eager loader; do not claim the entire package disappears |
+| `internal/domain` source YAML tags and runtime/source coupling | Separate configuration DTOs from execution values where presence requires it; reuse domain validation rather than duplicating it |
+| Direct `gopkg.in/yaml.v3` use for definition loading | Remove once all such consumers migrate; ConfigKit brings its own private YAML dependency |
+
+New integration work is real: typed source DTOs, ConfigKit calls and error wrapping,
+source-context records, collection indexing, and conversion to runtime definitions.
+Some defensive copies remain necessary at public/injected-client boundaries;
+ConfigKit's detached load result is not permission to expose shared engine maps.
+`internal/jsonvalue` still serves programmatic values, request parameters, and
+prepared schema ownership, so adoption does not eliminate it.
+
+Maintenance savings depend on actually deleting the old lazy paths and their
+special-case tests. Move generic YAML/scalar behavior coverage to ConfigKit's
+ownership; PromptKit retains representative adapter tests plus domain, precedence,
+snapshot, inheritance, embedded-source, error, and execution integration tests.
+Do not duplicate ConfigKit's entire parser suite or discard execution regression
+coverage because construction fixtures need migration.
+
+Runtime benefit: ordinary runs no longer rediscover definition directories,
+redecode definitions, reread template text, or traverse inheritance chains.
+Cost: construction loads even unused definitions and retains their text and
+resolved values. ConfigKit also creates metadata and projections; keep only needed
+records after loading. These are reasoned tradeoffs, not measured performance
+claims. Adoption is not justified by the decoder replacement alone.
+
+## 5. Upstream Opportunities
+
+No ConfigKit enhancement is a demonstrated prerequisite for the recommended
+per-document snapshot design. Do not manufacture one to move a small adapter
+upstream. Reconsider the following if the agreed roadmap reveals concrete reuse:
+
+| Candidate | Recommendation and tradeoff |
+| --- | --- |
+| Native `fs.FS` sources with source-relative resource context | Most plausible optional enhancement: embedded configurations are useful across Go libraries. Reader sources suffice today. A fuller design must define imports/overlays in that filesystem rather than merely add a constructor or leak parser internals. |
+| Public in-memory layered loading with origins | Optional only if multiple consumers need application defaults and file/embedded layers composed together. It could remove adapters in those consumers, but is not necessary for whole-definition catalog shadowing. Review against ConfigKit's deliberate single-overlay scope. |
+| Generic named inheritance/presence-aware typed overlay | Consider an upstream proposal if a coherent application-neutral operation emerges; another concrete use case would strengthen the case but is not a prerequisite. PromptKit's named graph, atomic provider map, and routing rules alone do not justify a configurable general merge framework. Current fixed overlay semantics are not this API. |
+| Context-aware loading | Optional for reusable cancellable construction if large-source consumers need it. Promise cooperative cancellation only; no detached reads or instant interruption of arbitrary readers. |
+
+Do not propose a public raw AST, permissive identity-only decoder, or dynamic
+subtree conversion solely to preserve PromptKit's current scanning design.
+Do not upstream backend/model selection, credential timing for inference,
+profile catalog reservations, or prompt-template functions.
+
+ConfigKit and PromptKit both declare LGPL-3.0-or-later. Adoption must still update
+dependency notices under the OSS policy and inspect the exact selected release
+and its transitive notices. Their current Go minimums do not require a PromptKit
+toolchain increase. Use a released module dependency, not a sibling-directory
+replacement. This review proposes no licensing change or upstream edits.
+
+## 6. Decisions And Migration
+
+The goal and permission to break compatibility are agreed. The following specific
+choices are recommendations for discussion, not inferred approvals:
+
+1. **Loading lifetime and failure scope.** Recommend eager construction, frozen
+   definitions/content, and failure for malformed files anywhere in supplied
+   collections. This deletes lazy arbitration and repeated scans. Alternative:
+   explicit caller-loaded reusable definition snapshots separate from engine
+   construction; useful when multiple engines share data, but adds a public
+   lifecycle. Retaining automatic per-run reload is possible but sacrifices much
+   of the simplification and is not recommended.
+2. **Source organization.** Recommend individual typed definition documents with
+   explicit collection precedence. This preserves embedded catalogs and distributed
+   authoring without copying application configuration machinery. Alternative:
+   one keyed PromptKit bundle composed using ConfigKit imports and overlays;
+   offers richer unified provenance but needs decisions about embedded sources,
+   provider-map replacement, profile identity, and catalog conversion. Do not
+   assume existing ConfigKit APIs already solve those issues.
+3. **Presence and inheritance.** Recommend explicit-value replacement, null as
+   unspecified for optional settings, and atomic provider maps as described above.
+   Alternative: use ConfigKit-style recursive mapping overlays, where null clears;
+   this is more uniform with application overlays but changes provider-parameter
+   behavior and requires a supported cross-document mechanism. Historical
+   zero-as-absence is another viable option but loses useful standardization.
+4. **Diagnostic boundary.** Recommend wrapping ConfigKit's public load errors
+   rather than mirroring them. Alternative: translate into a small PromptKit-owned
+   diagnostic value; that limits public dependency coupling but introduces mapping
+   code and risks inconsistent diagnostics. Either option must cover domain errors
+   and preserve actionable source information without inventing an audit subsystem.
+
+Expected consumer migration:
+
+- Update definitions for ConfigKit's strict scalar rules, rejected YAML
+  anchors/aliases/tags, input limits, and null/type checks. Quote prompt version
+  strings rather than introducing a special numeric-version conversion merely
+  for historical YAML coercion.
+- Remove unintended explicit zero/false/empty values that formerly meant
+  inheritance; express deliberate overrides using the new presence-aware Go
+  fields and file semantics. Review provider maps for explicit replacement.
+- Handle definition errors at construction, clean up unused invalid files, and
+  deliberately rebuild/swap engines after definition changes.
+- Keep runtime credential lookup optional, avoid baking resolved secrets into
+  definition snapshots, and preserve request-scoped key overrides.
+- Update maintained catalog data if strict parsing reveals incompatible syntax;
+  coordinate catalog releases before the PromptKit dependency update. Preserve
+  maintained profile names and catalog ownership rules, not every old scalar
+  spelling. Do not validate live provider/model availability.
+- Update error inspection, constructor/source options, and programmatic profile
+  literals as required. Keep representative preparation, execution, inspection,
+  custom backend, fallback, and embedded workflows in the migration guidance.
+
+The subsequent feature roadmap should settle these choices and specify the target
+end state. The staged implementation plan can then sequence any accepted upstream
+work, PromptKit deletions/replacements, contract tests, canonical documentation,
+examples, attribution, and minor-release migration notes. Neither document should
+become a reason to preserve obsolete compatibility machinery.