From 7b012033a604fd0e5d20a7710269c04c70fade33 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Tue, 29 Sep 2026 16:49:18 +0000 Subject: [PATCH] Align the adoption assessment with ConfigKit source tiers --- docs/roadmap/configkit-assessment.md | 260 ++++++++++++++++----------- 1 file changed, 151 insertions(+), 109 deletions(-) diff --git a/docs/roadmap/configkit-assessment.md b/docs/roadmap/configkit-assessment.md index 34a2479..2f316d7 100644 --- a/docs/roadmap/configkit-assessment.md +++ b/docs/roadmap/configkit-assessment.md @@ -13,8 +13,8 @@ sequences, file syntax, or Go signatures. Document intentional breaking changes and their migration paths. Recommend adoption through an eager, immutable definition-loading boundary. -ConfigKit should own YAML interpretation, strict typed decoding, presence, and -source diagnostics. PromptKit should own definition identity, named profile +ConfigKit should own YAML interpretation, source-tier combination, strict typed +decoding, presence, and source diagnostics. PromptKit should own definition identity, named profile relationships, LLM-specific rules, and execution. Do not retain the lazy loading architecture merely to minimize migration, or introduce a second generic configuration framework inside PromptKit. @@ -40,20 +40,18 @@ technically impossible using today's reader-based API. bundling default configuration. Passing an opened file to today's `Reader` supplies bytes and a label, but does not make filesystem-relative loading native to ConfigKit; that workaround is not the target architecture. -2. **Public in-memory layered loading with origins.** Support explicitly supplied - layers without temporary files or YAML serialization round trips, retaining - presence and meaningful source provenance. The upstream design must settle - which inputs it accepts: filesystem/byte sources, typed values, or both; how - typed values express omission; and how strict composition differs from - intentional overriding. Recommend covering embedded defaults, caller sources, - and programmatic values, with explicit ordering and useful conflict diagnostics. +2. **Source-backed underlays with origins and programmatic input.** Adopt + ConfigKit's working design: strictly compose contributions within each tier, + then recursively overlay explicitly ordered tiers. Support embedded, host-file, + byte/reader, and labeled programmatic sources without temporary files or YAML + serialization round trips. Prefer the proposed string-keyed programmatic + mappings: omitted keys convey absence; present values, including nil/null, + convey explicit input. Do not require typed-value ingestion as a prerequisite. -Keep these APIs application-neutral. A list of sources alone does not implement -PromptKit's whole-profile shadowing, named inheritance, or atomic provider maps. -Define which layer operations ConfigKit owns and which identity/domain decisions -remain in PromptKit; do not silently substitute recursive merging for an agreed -replacement rule. Revisit ConfigKit's current single-overlay/deferred boundaries -deliberately rather than working around them inside PromptKit. +Adopt ConfigKit's recursive combination model rather than preserving PromptKit's +whole-profile shadowing or atomic provider-map replacement across source tiers. +No hidden replacement pass should run after ConfigKit loading. Named inheritance +is a separate optional upstream request, not part of these prerequisites. The upstream design should show how the two capabilities work together, including source origins and efficient ownership/copying. Resolve these API questions before @@ -67,6 +65,15 @@ Reviewed local baselines on 2026-09-29: - PromptKit: `12d6d80024e80c552d122a076c22429bf6f2f6c2` (v0.11.2). - ConfigKit: `67c03690181c5af8dde6be325abac241dae6732f` (v0.4.2). +Design update: reviewed ConfigKit's working +`docs/roadmap/sources-and-layering.md` with checkout HEAD +`36172ed20e832b371506d6aceb02dc6c8f50f71b`. The reviewed roadmap included +uncommitted changes, so that commit alone does not identify the reviewed text. +It remains a proposed target, not implemented behavior. The recommendations here +now follow its underlay semantics and support its two open-decision recommendations: +labeled programmatic mappings and direct source-context queries. Verify the +final upstream contracts before preparing dependent implementation details. + PromptKit evidence: [consumer workflows](../consumers/pkg-promptkit.md), [file formats](../formats.md), [source implementation](../internal/sources.md), [runner](../internal/runner.md), and the implementation owners identified below. @@ -118,11 +125,11 @@ dropping execution features to simplify the configuration layer. | File/directory/embedded source selection and definition ID indexing | Retain a small PromptKit collection loader; ConfigKit loads documents, not a prompt catalog | | Configuration filesystem sources, relative imports, and generic layer origins | Delegate to the prerequisite ConfigKit APIs; do not maintain parallel reader-based loading machinery | | Definition discovery and relative message-content references | Retain PromptKit's domain resource adapters, using ConfigKit source context and standard `io/fs` facilities where sufficient | -| Profile-source priority and duplicate definition policy | Retain explicit PromptKit collection assembly; eliminate error-driven repository fallback | -| Named profile inheritance and backend selection | Retain a typed PromptKit resolver over the loaded collection, with explicit presence; consider generic inheritance upstream only with broader evidence | +| Profile-source priority and combination | PromptKit selects tiers; ConfigKit owns strict within-tier composition and recursive cross-tier overlay; eliminate repository fallback and whole-definition shadowing | +| Named profile inheritance and backend selection | PromptKit owns domain relationships unless the separate inheritance proposal is adopted; integrate resolution with ConfigKit's recorded lifecycle | | LLM setting bounds, role/session normalization, output rules, reserved provider fields | Retain shared PromptKit domain owners; ConfigKit validation callbacks may invoke them | | Definition defaults | Use ConfigKit defaults where they do not erase inheritance presence; apply effective execution defaults after inheritance | -| Programmatic configuration | Retain typed construction; normalize to the same presence-aware internal definitions without YAML round trips | +| Programmatic configuration | Prefer labeled mapping input with explicit presence; evaluate whether typed PromptKit convenience constructors justify conversion code, without YAML round trips or zero-value inference | | Runtime clients, observers, artifact readers, request data | Retain normal Go injection and request APIs; do not load them through ConfigKit | | Credentials | Retain optional environment-name and direct request-key semantics; do not replace these with ConfigKit's required typed secret resolution | | Catalog manifest, ownership, reserved names, secret-safety checks | Retain the catalog adapter; reuse the common ConfigKit profile decoder | @@ -135,19 +142,24 @@ dropping execution features to simplify the configuration layer. ### Definition Snapshot Create one immutable definition snapshot during engine construction. Discover -the explicitly supplied sources, decode each definition through ConfigKit, -normalize domain values, assemble identity indexes, and resolve profile chains +the explicitly supplied sources, assemble their contributions through ConfigKit, +decode the effective configuration, normalize domain values, and resolve relationships before publishing the engine. No provider call or credential availability check belongs in this phase. Publish no partially initialized engine on failure. -Use one typed document per prompt or profile initially. Existing files need not -be converted into a monolithic application configuration merely to adopt -ConfigKit. Support OS directories, individual files, embedded/custom `fs.FS` -sources, and programmatic profiles. Use the prerequisite native ConfigKit -filesystem and layered-loading APIs for configuration acquisition and generic -composition. PromptKit retains definition discovery and message-resource handling, -using the source context supplied by ConfigKit. Exact integration is contingent -on the upstream contracts; do not build a competing layer engine in PromptKit. +Prefer keyed collections, such as `profiles.`, assembled before one typed +loading lifecycle instead of independently decoding each profile and recombining +the results. Multiple files can contribute to those collections; this does not +require a monolithic file. Reconsider the previous one-typed-document-per-definition +recommendation. Settle prompt ID/version representation and catalog format migration +in the feature roadmap. Support OS directories, individual files, embedded/custom +`fs.FS` sources, and programmatic input through the new source model; source selection +and definition discovery remain PromptKit concerns, not a second composition engine. + +Perform domain-specific resolution through ConfigKit's recorded lifecycle before +semantic validation and result publication. Reuse its presence and provenance; +do not add a generic merge/defaulting pass or reconstruct origins after loading. +Runtime client assembly and resource interpretation remain outside typed decoding. Load and freeze referenced message text with definitions. Validate static prompt structure and template syntax at loading, using the same template function setup @@ -155,6 +167,15 @@ as execution; render only when request variables and artifacts exist. Preserve source-relative content resolution and containment without expanding the threat model. A snapshot is a set of captured reads, not an atomic filesystem transaction. +Resolve each effective content reference using its contributing source's context, +not the highest tier's directory. Support ConfigKit's proposed direct context +query rather than a consumer-maintained filesystem registry. Composite origins +must not be assigned an arbitrary directory; ambiguous or absent context needs +an actionable resource error or explicit consumer-supplied context. ConfigKit +does not snapshot external text: PromptKit opens it explicitly and freezes it +before engine publication. Caller-owned filesystem lifetimes remain caller-owned; +close resource handles acquired by PromptKit without retaining ConfigKit load handles. + Keep request artifacts and schema compilation at preparation. A request can replace the output contract, so an unused default schema should not need eager compilation. Prepared execution continues to freeze its effective resources. @@ -165,17 +186,22 @@ files, every schema file, or execution state. Use explicit source categories in the existing useful priority order: programmatic profiles, operator source, application fallback, PromptKit-owned -profiles, then external catalogs. Within a category, duplicate profile IDs fail; -across categories, a higher-priority definition replaces the whole lower one. -Prompt ID/version pairs must be unique; different versions remain available. -An omitted version remains ambiguous when more than one version is present. +profiles, then external catalogs. Express these as explicit ordered tiers, not +source discovery order. Within a tier, mappings compose recursively and overlapping +non-mapping contributions conflict, even when equal. Disjoint contributions to +the same profile may therefore combine; do not preserve a blanket duplicate-ID +rejection. Across tiers, mappings overlay recursively, retaining omitted lower +children. Effective prompt identities/versions must remain addressable, with an +omitted version ambiguous when multiple versions exist; finalize their keyed +representation without reintroducing whole-definition shadowing. -Strictly decode and locally validate every supplied definition, including -shadowed definitions. Thus an unused malformed file is a construction failure, -not a reason to perform a partial metadata scan. Resolve relationships for the -effective selected profile collection; shadowed definitions do not contribute -inheritance edges to that collection. Keep the catalog adapter's independent -validation of maintained data before consumer overrides. +All sources undergo syntax, acquisition, and composition checks; higher tiers +cannot hide those failures. Decode and validate effective values after assembly, +not every intermediate tier against the final typed schema. A replaced lower +value may consequently have a type that would not fit the final schema. An unused +invalid effective definition still fails construction. Catalog ownership, +reserved-name, and secret-safety checks remain distinct source-policy requirements, +not a reason to impose intermediate typed validation on ordinary inputs. Preserve profile templates that become usable through descendants or runtime overrides: eager loading must not require every consumer profile to be an @@ -185,43 +211,51 @@ requirements after runtime selection/overrides. Prompt default-profile reference can be checked against the assembled collection; a prompt with no default remains valid for callers that supply a profile explicitly. -Named-definition precedence is distinct from recursive application overlays. -Use the upstream layered API where its operations fit; retain the small identity -selection step in PromptKit where needed. The upstream design must make that -boundary explicit. Do not generate temporary overlay files or serialize maps to -YAML to imitate a missing operation. Imports are not definition discovery and -need not be enabled merely because native filesystem sources support them. +ConfigKit's ordering is typed defaults, source-underlay tiers, strictly composed +main input, selected overlay, then CLI overrides. PromptKit's API must map its +source priorities explicitly onto supported roles; do not mislabel programmatic +input as CLI flags. Underlay-only loading avoids inventing a main document. +Imports remain opt-in, and the named-overlay convenience need not be exposed by +PromptKit merely because ConfigKit supports it. -### Inheritance And Presence +A prompt's `version` is a domain identifier, not automatically a ConfigKit +application-schema version. Do not configure `VersionRule` for prompt versions. +If a separate configuration schema version is later needed, respect ConfigKit's +designated main-input ownership and pre-underlay check timing; do not assume +last-wins version selection across catalogs. -Recommend omission-based inheritance: absent fields inherit; explicit values -replace inherited settings, including zero, false, and empty strings where the -domain permits them. Preserve presence through inheritance, backend/framework +### Source Presence And Separate Named Inheritance + +Adopt ConfigKit source-layer semantics: absence retains lower input; explicit +zero, false, and empty strings replace it; mappings combine recursively; empty +mappings retain lower children; sequences replace as units; null explicitly +replaces, subject to final schema legality. No reset/delete operator is assumed. +Preserve presence through domain resolution, backend/framework default application, request overrides, and outbound request construction. Otherwise accepting `temperature: 0` during decoding would accomplish nothing. -Use ConfigKit metadata for file presence and presence-aware typed fields for -programmatic definitions. Both paths feed one typed resolver. Decode raw profiles -without effective execution defaults so defaults do not become child overrides. -Avoid a parallel reflection-based merge engine in PromptKit. +Use ConfigKit metadata for both file and programmatic input. A final Go value +alone does not distinguish absent from explicit zero. Avoid eagerly defaulting +profile fields in ways that erase that distinction before relationship resolution. -Keep provider `extra_params` replacement atomic: an explicitly supplied map -replaces the inherited map, including an empty map. This matches the useful -ability to choose a complete provider parameter set and avoids accidental mixing -of incompatible options. Do not generalize this domain rule into arbitrary -ConfigKit merge strategies. +Provider `extra_params` maps combine recursively across source tiers too: `{}` +does not clear lower keys. Retire the proposed atomic source-map replacement and +null-as-unspecified conventions; do not emulate them after loading. An operator +overriding only `model` also retains a lower `base_profile` declaration and other +omitted settings. Diagnose and document these consequences explicitly. -Recommend optional scalar pointers where null means no value/inherit, while -explicit zero/false/empty is a value. For optional maps, null likewise means no -override and `{}` explicitly clears. Required identity fields remain required; -an explicitly empty model or endpoint cannot produce an executable target. -Publish these conventions deliberately rather than pretending ConfigKit's -presence metadata automatically determines domain inheritance semantics. +Named inheritance still selects relationships between effective entries; source +layering does not implement it. The optional upstream request remains pending. +Finalize named-inheritance null/map semantics separately, favoring consistency +with source layering rather than automatically retaining atomic maps or null-as- +inherit. Any domain exception needs explicit justification. This also does not +silently change the existing request-level override contract, which is a separate +runtime boundary to review in the feature roadmap. -Resolve and memoize each effective chain once per snapshot. Retain source and -field-origin information sufficient to diagnose a bad base or inherited value; -do not claim that per-document ConfigKit metadata already represents the complete -cross-document inheritance history. +Resolve shared ancestors once per snapshot. Use recorded domain resolution to +keep resulting values and origins consistent. Confirm the upstream lifecycle can +express required relationship provenance; do not claim generic layer history +already explains named inheritance or invent a parallel audit framework. ### Reloading, Errors, And Application Integration @@ -259,18 +293,20 @@ without an identified consumer requirement. | Current implementation | Expected disposition | | --- | --- | -| `internal/profile/filesystem_repository.go`: metadata scans, selected strict decode, stream-end checks, malformed-file selection arbitration | Replace with eager document loads and indexed values; remove the metadata prepass and per-lookup filesystem path | +| `internal/profile/filesystem_repository.go`: metadata scans, selected strict decode, stream-end checks, malformed-file selection arbitration | Replace with source acquisition, ConfigKit assembly/decoding, and indexed effective values; remove the metadata prepass and per-lookup filesystem path | | `internal/promptdef/filesystem_repository.go`: metadata selection and repeated decoding/content opening | Replace with eager prompt loads and an ID/version index; keep domain normalization and contained content reading | | Profile filesystem, `fs.FS`, eager, memory, and overlay repository variants | Collapse loading into source adapters plus immutable collections; preserve only interfaces that still serve a real boundary | | `internal/profile/resolving_repository.go` | Replace per-operation traversal and zero-sentinel merging with once-per-snapshot presence-aware typed resolution; graph logic remains | -| `internal/engine` raw repository-stack assembly | Replace with explicit collection construction and precedence; engine capacity/client assembly remains | +| `internal/engine` raw repository-stack assembly | Replace with ConfigKit source tiers and one effective typed collection; engine capacity/client assembly remains | | `internal/catalog` profile metadata extraction | Obtain explicit field information from ConfigKit; retain manifests, ownership, namespace, connection-metadata and secret checks | | `internal/filecatalog` | Retain/reduce discovery and containment helpers actually needed by the eager loader; do not claim the entire package disappears | | `internal/domain` source YAML tags and runtime/source coupling | Separate configuration DTOs from execution values where presence requires it; reuse domain validation rather than duplicating it | | Direct `gopkg.in/yaml.v3` use for definition loading | Remove once all such consumers migrate; ConfigKit brings its own private YAML dependency | -New integration work is real: typed source DTOs, ConfigKit calls and error wrapping, -source-context records, collection indexing, and conversion to runtime definitions. +New integration work is real: collection DTOs, ConfigKit calls and error wrapping, +source-context access, collection indexing, programmatic mapping ergonomics, and +conversion to runtime definitions. Catalog documents may need new keyed shapes; +do not hide that migration behind a permanent second loader. Some defensive copies remain necessary at public/injected-client boundaries; ConfigKit's detached load result is not permission to expose shared engine maps. `internal/jsonvalue` still serves programmatic values, request parameters, and @@ -296,8 +332,9 @@ Efficiency is an explicit design criterion for both the upstream work and this shared-library refactor. Prefer optimal or near-optimal algorithms relative to input and produced-data size, not a blanket O(n) claim with an undefined n. -- Acquire and decode each definition once per snapshot; reuse captured content - rather than reopening or reparsing it for metadata, selection, and validation. +- Acquire/parse each accepted source occurrence once, assemble input, and run + one typed loading lifecycle rather than decoding intermediate tiers. Reuse + captured content instead of reparsing it for metadata, selection, and validation. - Build identity indexes in one pass, with expected constant-time map lookup. Do not scan whole collections for every lookup or precedence decision. - Process inheritance graph structure in O(V + E), where V is the number of @@ -328,7 +365,7 @@ a concrete dependency: | Candidate | Recommendation and tradeoff | | --- | --- | -| Generic named inheritance/presence-aware typed overlay | Consider an upstream proposal if a coherent application-neutral operation emerges; another concrete use case would strengthen the case but is not a prerequisite. PromptKit's named graph, atomic provider map, and routing rules alone do not justify a configurable general merge framework. Current fixed overlay semantics are not this API. | +| Generic named configuration inheritance | Requested separately in ConfigKit's `docs/roadmap/feature-request-promptkit-inheritance.md`, primarily for shared semantics across consumers. ConfigKit decides whether to adopt it; PromptKit can retain a small domain resolver if declined. Presence-aware source layering belongs to the prerequisite work, not this optional proposal. | | Context-aware loading | Optional for reusable cancellable construction if large-source consumers need it. Promise cooperative cancellation only; no detached reads or instant interruption of arbitrary readers. | Do not propose a public raw AST, permissive identity-only decoder, or dynamic @@ -345,36 +382,36 @@ or licensing change. ## 6. Decisions And Migration -The goal, permission to break compatibility, and these four recommendations are -the agreed working direction. Alternatives remain recorded for rationale, not as -unresolved votes. The prerequisite upstream API questions remain open and must be -settled before finalizing dependent implementation details. +The agreed direction now incorporates ConfigKit's working roadmap. This supersedes +the earlier recommendations for independently typed documents, whole-definition +shadowing, intermediate typed validation, and atomic maps/null-as-unspecified at +the source boundary. -1. **Loading lifetime and failure scope.** Recommend eager construction, frozen - definitions/content, and failure for malformed files anywhere in supplied - collections. This deletes lazy arbitration and repeated scans. Alternative: - explicit caller-loaded reusable definition snapshots separate from engine - construction; useful when multiple engines share data, but adds a public - lifecycle. Retaining automatic per-run reload is possible but sacrifices much - of the simplification and is not recommended. -2. **Source organization.** Recommend individual typed definition documents with - explicit collection precedence. This preserves embedded catalogs and distributed - authoring without copying application configuration machinery. Alternative: - one keyed PromptKit bundle composed using ConfigKit imports and overlays; - offers richer unified provenance but needs decisions about embedded sources, - provider-map replacement, profile identity, and catalog conversion. Do not - assume existing ConfigKit APIs already solve those issues. -3. **Presence and inheritance.** Recommend explicit-value replacement, null as - unspecified for optional settings, and atomic provider maps as described above. - Alternative: use ConfigKit-style recursive mapping overlays, where null clears; - this is more uniform with application overlays but changes provider-parameter - behavior and requires a supported cross-document mechanism. Historical - zero-as-absence is another viable option but loses useful standardization. -4. **Diagnostic boundary.** Recommend wrapping ConfigKit's public load errors - rather than mirroring them. Alternative: translate into a small PromptKit-owned - diagnostic value; that limits public dependency coupling but introduces mapping - code and risks inconsistent diagnostics. Either option must cover domain errors - and preserve actionable source information without inventing an audit subsystem. +1. **Loading lifetime and failure scope.** Keep eager construction and frozen + definitions/content. Syntax/acquisition/composition failures remain unmaskable; + typed and domain validation apply to effective values. A separate reusable + public snapshot lifecycle remains possible but is not currently needed. +2. **Source organization.** Prefer keyed collections composed across files and + tiers before typed loading. Settle exact profile and prompt/version shapes, + source-role mapping, and catalog migration in the feature roadmap. Independent + typed loads followed by custom combination are not the selected direction. +3. **Presence and relationships.** Accept recursive source-tier overlay, retained + children for empty mappings, explicit null replacement, and effective metadata. + Named inheritance remains separate pending the upstream decision; settle its + domain semantics before implementation rather than assuming either automatic + equivalence or an exception to ConfigKit's rules. +4. **Diagnostic boundary.** Keep wrapping ConfigKit's public loading errors rather + than mirroring them. Retain actionable domain relationship and resource errors + without duplicating ConfigKit's generic diagnostic/provenance system. + +Support the upstream recommendations for labeled mapping-based programmatic input +and direct filesystem resource-context queries. If real consumer examples show +mapping construction to be repetitive, consider small typed PromptKit conveniences +with explicit presence rather than requiring an upstream typed-ingestion API. +Final upstream signatures and ownership details remain open dependencies of the +feature roadmap, not implemented guarantees. Resolve the named-inheritance +direction there as well; a PromptKit-owned resolver remains viable if ConfigKit +declines or defers that optional request. Expected consumer migration: @@ -382,14 +419,19 @@ Expected consumer migration: anchors/aliases/tags, input limits, and null/type checks. Quote prompt version strings rather than introducing a special numeric-version conversion merely for historical YAML coercion. -- Remove unintended explicit zero/false/empty values that formerly meant - inheritance; express deliberate overrides using the new presence-aware Go - fields and file semantics. Review provider maps for explicit replacement. -- Handle definition errors at construction, clean up unused invalid files, and - deliberately rebuild/swap engines after definition changes. +- Move standalone definitions into the agreed keyed contribution format. Review + same-name contributions: disjoint fields within a tier can combine, and higher + tiers retain omitted lower fields, including `base_profile` references. +- Review zero/false/empty/null values explicitly. Provider maps merge across + tiers, `{}` does not clear them, and null replaces rather than means omission. + Use labeled programmatic mappings or approved presence-aware conveniences. + Document the separately selected inheritance semantics when settled. +- Handle effective-definition errors at construction, fix syntax/composition + failures even in overridden inputs, and deliberately rebuild/swap engines + after definition changes. - Keep runtime credential lookup optional, avoid baking resolved secrets into definition snapshots, and preserve request-scoped key overrides. -- Update maintained catalog data if strict parsing reveals incompatible syntax; +- Update maintained catalog data for collection shapes and any incompatible syntax; coordinate catalog releases before the PromptKit dependency update. Preserve maintained profile names and catalog ownership rules, not every old scalar spelling. Do not validate live provider/model availability.