eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Align the assessment with ConfigKit v0.5.0

Browse Source
This commit is contained in:
Eric Rakestraw
2026-09-30 00:25:10 +00:00
parent 34f5980973
commit 08409dd472
1 changed files with 115 additions and 103 deletions
Download Patch File Download Diff File Expand all files Collapse all files

218
docs/roadmap/configkit-assessment.md
View File

@@ -1,8 +1,9 @@
# ConfigKit Adoption Assessment
Status: assessment with agreed design direction; upstream API details remain
to be settled before the feature roadmap and implementation plan. No runtime
changes are part of this assessment.
Status: assessment aligned to released ConfigKit v0.5.0. Upstream capability
prerequisites are available; PromptKit integration and migration decisions remain
for the feature roadmap and implementation plan. No runtime changes are part of
this assessment.
## Goal And Recommendation
@@ -14,8 +15,8 @@ and their migration paths.
Recommend adoption through an eager, immutable definition-loading boundary.
ConfigKit should own YAML interpretation, source-tier combination, strict typed
decoding, presence, and source diagnostics. PromptKit should own definition identity, named profile
relationships, LLM-specific rules, and execution. Do not retain the lazy loading
decoding, sibling inheritance, presence, and source diagnostics. PromptKit should
own definition selection, LLM-specific rules, and execution. Do not retain the lazy loading
architecture merely to minimize migration, or introduce a second generic
configuration framework inside PromptKit.
@@ -25,69 +26,53 @@ lazy error arbitration, and repeated profile-chain resolution. Some of those
savings come from the loading redesign rather than ConfigKit itself. ConfigKit
adds ongoing value by owning the generic decoding behavior and its tests.
## Prerequisite: Proposed Upstream ConfigKit Work
## Released ConfigKit Capabilities
Design, implement, and release these two ConfigKit capabilities before the
dependent PromptKit refactor. The remaining recommendations assume this upstream
work; finalize their integration details against its agreed public contracts.
These are selected architectural prerequisites, not a claim that adoption is
technically impossible using today's reader-based API.
ConfigKit v0.5.0 satisfies the two requested prerequisites and also implements
sibling inheritance. Adopt the released public contracts rather than waiting for
the earlier proposals to be implemented:
1. **Native `fs.FS` sources and source-relative context.** Support embedded and
caller-supplied filesystems as first-class configuration sources. Define source
identity, relative path resolution, ownership, and how enabled imports and
overlays operate within that filesystem. This is broadly useful for libraries
bundling default configuration. Passing an opened file to today's `Reader`
supplies bytes and a label, but does not make filesystem-relative loading
native to ConfigKit; that workaround is not the target architecture.
2. **Source-backed underlays with origins and programmatic input.** Adopt
ConfigKit's working design: strictly compose contributions within each tier,
then recursively overlay explicitly ordered tiers. Support embedded, host-file,
byte/reader, and labeled programmatic sources without temporary files or YAML
serialization round trips. Prefer the proposed string-keyed programmatic
mappings: omitted keys convey absence; present values, including nil/null,
convey explicit input. Do not require typed-value ingestion as a prerequisite.
1. **Filesystem sources:** `NewFilesystem(label, fsys).File(name)` borrows an
`fs.FS` with reusable namespace identity. Imports and resource contexts retain
that namespace. `Result.SourceContext` and `Resolution.SourceContext` expose
passive `ResourceContext` values with `FS` and `BaseDir`.
2. **Source-backed underlays:** `LoadLayered` accepts a main `Input`
and ordered `Underlay` groups. Each group composes strictly; groups overlay
recursively from lowest to highest priority. `Input{Source: NoSource()}` omits
the main input. `Mapping(label, map[string]any)` snapshots supported built-in
scalar/list/map values with explicit presence; it does not ingest arbitrary
typed structs. Generic programmatic input is not represented as CLI overrides.
3. **Sibling inheritance:** the reserved `extends` key names one immediate sibling
mapping. Relationships resolve after precedence assembly and before typed
decoding, with inherited origins, history, and resource context preserved.
This is built-in behavior, not the opt-in/registered-collection API originally
suggested by PromptKit.
Adopt ConfigKit's recursive combination model rather than preserving PromptKit's
whole-profile shadowing or atomic provider-map replacement across source tiers.
No hidden replacement pass should run after ConfigKit loading. Named inheritance
is a separate optional upstream request, not part of these prerequisites.
The upstream design should show how the two capabilities work together, including
source origins and efficient ownership/copying. Resolve these API questions before
turning the assessment into a decision-complete PromptKit implementation plan.
This assessment does not authorize code changes in either repository.
No hidden replacement or inheritance pass should run after ConfigKit loading.
Use released sibling inheritance for profiles, leaving domain validation in
PromptKit. This assessment does not authorize code changes in either repository.
## Evidence And Scope
Reviewed local baselines on 2026-09-29:
PromptKit's runtime baseline remains v0.11.2; this assessment has since received
documentation-only revisions. Updated upstream review on 2026-09-30:
- PromptKit: `12d6d80024e80c552d122a076c22429bf6f2f6c2` (v0.11.2).
- ConfigKit: `67c03690181c5af8dde6be325abac241dae6732f` (v0.4.2).
Design update: reviewed ConfigKit's working
`docs/roadmap/sources-and-layering.md` with checkout HEAD
`36172ed20e832b371506d6aceb02dc6c8f50f71b`. The reviewed roadmap included
uncommitted changes, so that commit alone does not identify the reviewed text.
It remains a proposed target, not implemented behavior. The recommendations here
now follow its underlay semantics and support its two open-decision recommendations:
labeled programmatic mappings and direct source-context queries. Verify the
final upstream contracts before preparing dependent implementation details.
- ConfigKit: `dc09959cca4e105f49ec316c6d4ba585bd2340bd` (v0.5.0), verified
against the published tag. Its clean checkout passed `go test ./...` during
the release review. This replaces the earlier v0.4.2 and uncommitted-roadmap
comparison as the upstream basis for adoption.
PromptKit evidence: [consumer workflows](../consumers/pkg-promptkit.md),
[file formats](../formats.md), [source implementation](../internal/sources.md),
[runner](../internal/runner.md), and the implementation owners identified below.
ConfigKit evidence: its reviewed `docs/config.md`, `docs/roadmap/deferred.md`,
`lifecycle.go`, `source.go`, and ADR 0001 in the sibling checkout. That checkout
is review context, not a proposed build dependency or required directory layout.
ConfigKit currently exposes `Load[T]` over file, byte, reader, and defaults-only
sources. It supports strict decoding, presence/origin records, defaults,
recorded resolution, semantic validation, explicit imports, one selected
filesystem overlay, and secret handling. Its parser and generic conversion
machinery are private. It has no public arbitrary layer-composition, generic
inheritance, or `fs.FS` source API. Its deferred scope explicitly excludes several
of these generalizations; an upstream enhancement requires a deliberate decision.
ConfigKit evidence at that release: `docs/config.md`, `docs/releases/v0.5.0.md`,
`layered.go`, `source.go`, `resource_context.go`, and ADR 0009 in the sibling
checkout. That checkout is review context, not a build dependency or required
directory layout. The released configuration contract owns exact upstream behavior;
the summaries here identify its consequences for the proposed PromptKit design.
This is a source-and-contract assessment, not a performance benchmark or full
correctness audit. It does not claim measured allocation, latency, or line-count
@@ -132,7 +117,8 @@ provide configuration provenance.
| Configuration filesystem sources, relative imports, and generic layer origins | Delegate to the prerequisite ConfigKit APIs; do not maintain parallel reader-based loading machinery |
| Definition discovery and relative message-content references | Retain PromptKit's domain resource adapters, using ConfigKit source context and standard `io/fs` facilities where sufficient |
| Profile-source priority and combination | PromptKit selects tiers; ConfigKit owns strict within-tier composition and recursive cross-tier overlay; eliminate repository fallback and whole-definition shadowing |
| Named profile inheritance and backend selection | PromptKit owns domain relationships unless the separate inheritance proposal is adopted; integrate resolution with ConfigKit's recorded lifecycle |
| Named profile inheritance | Delegate sibling graph resolution, combination, and derivation to ConfigKit's `extends`; remove the custom profile resolver |
| Backend selection and domain normalization | Retain PromptKit domain rules; use recorded application resolution for configuration transformations not already handled by structural inheritance |
| LLM setting bounds, role/session normalization, output rules, reserved provider fields | Retain shared PromptKit domain owners; ConfigKit validation callbacks may invoke them |
| Definition defaults | Use ConfigKit defaults where they do not erase inheritance presence; apply effective execution defaults after inheritance |
| Programmatic configuration | Prefer labeled mapping input with explicit presence; evaluate whether typed PromptKit convenience constructors justify conversion code, without YAML round trips or zero-value inference |
@@ -162,7 +148,8 @@ in the feature roadmap. Support OS directories, individual files, embedded/custo
`fs.FS` sources, and programmatic input through the new source model; source selection
and definition discovery remain PromptKit concerns, not a second composition engine.
Perform domain-specific resolution through ConfigKit's recorded lifecycle before
ConfigKit expands structural inheritance before typed defaults/decoding. Perform
any remaining domain-specific resolution through its recorded lifecycle before
semantic validation and result publication. Reuse its presence and provenance;
do not add a generic merge/defaulting pass or reconstruct origins after loading.
Runtime client assembly and resource interpretation remain outside typed decoding.
@@ -174,7 +161,7 @@ source-relative content resolution and containment without expanding the threat
model. A snapshot is a set of captured reads, not an atomic filesystem transaction.
Resolve each effective content reference using its contributing source's context,
not the highest tier's directory. Support ConfigKit's proposed direct context
not the highest tier's directory. Use ConfigKit's released direct context
query rather than a consumer-maintained filesystem registry. Composite origins
must not be assigned an arbitrary directory; ambiguous or absent context needs
an actionable resource error or explicit consumer-supplied context. ConfigKit
@@ -212,7 +199,7 @@ not a reason to impose intermediate typed validation on ordinary inputs.
Preserve profile templates that become usable through descendants or runtime
overrides: eager loading must not require every consumer profile to be an
independently runnable endpoint/model pair. Validate supplied fields and graph
integrity eagerly (including missing bases and cycles), and check final execution
integrity eagerly (ConfigKit handles missing bases and cycles), and check final execution
requirements after runtime selection/overrides. Prompt default-profile references
can be checked against the assembled collection; a prompt with no default remains
valid for callers that supply a profile explicitly.
@@ -230,7 +217,7 @@ If a separate configuration schema version is later needed, respect ConfigKit's
designated main-input ownership and pre-underlay check timing; do not assume
last-wins version selection across catalogs.
### Source Presence And Separate Named Inheritance
### Source Presence And Released Sibling Inheritance
Adopt ConfigKit source-layer semantics: absence retains lower input; explicit
zero, false, and empty strings replace it; mappings combine recursively; empty
@@ -247,21 +234,40 @@ profile fields in ways that erase that distinction before relationship resolutio
Provider `extra_params` maps combine recursively across source tiers too: `{}`
does not clear lower keys. Retire the proposed atomic source-map replacement and
null-as-unspecified conventions; do not emulate them after loading. An operator
overriding only `model` also retains a lower `base_profile` declaration and other
overriding only `model` also retains a lower `extends` declaration and other
omitted settings. Diagnose and document these consequences explicitly.
Named inheritance still selects relationships between effective entries; source
layering does not implement it. The optional upstream request remains pending.
Finalize named-inheritance null/map semantics separately, favoring consistency
with source layering rather than automatically retaining atomic maps or null-as-
inherit. Any domain exception needs explicit justification. This also does not
silently change the existing request-level override contract, which is a separate
runtime boundary to review in the feature roadmap.
Replace `base_profile` with `extends` on sibling profile mappings. A string names
one exact immediate sibling; null cancels the relationship while preserving the
child's own settings. Omission retains a lower-tier directive. There is no ancestor
scope search or cross-collection reference. Superseded references are history,
not active relationships to validate; every effective relationship resolves,
including profiles not selected for execution.
Resolve shared ancestors once per snapshot. Use recorded domain resolution to
keep resulting values and origins consistent. Confirm the upstream lifecycle can
express required relationship provenance; do not claim generic layer history
already explains named inheritance or invent a parallel audit framework.
Children recursively overlay their resolved parents with the same value rules:
zero/false/empty strings and sequences/null replace, mappings combine, and `{}`
does not clear inherited keys. Inherited values count as explicit input for
presence queries and absence-based defaults, even when false, zero, or null.
Do not repeat inheritance in PromptKit's application-resolution callback. Use
`Origin.Inheritance` and retained history for explicit operator inspection.
Inherited resource references retain their declaring source's context.
Enclosing inheritance resolves before nested relationships; nested inherited
directives resolve at each destination against its siblings. Defaults and
application-derived values do not trigger another inheritance pass. Model this
deliberately in PromptKit configuration rather than promising arbitrary graph
resolution over runtime values. Partial bases must fit the typed schema and
PromptKit's validation; ConfigKit grants them no special validation exemption.
`extends` is reserved in every input mapping, including provider-parameter maps,
across all source kinds. There is no opt-in switch or ordinary-data escape in the
released contract. Audit catalog and consumer payload needs before finalizing
their format; a literal provider field with that name cannot silently pass through
the configuration loader. Do not invent an encoding workaround in this assessment.
The existing request-level override API remains a separate runtime boundary.
The feature roadmap must review its interaction with configured values without
silently subjecting runtime request data to configuration inheritance directives.
### Reloading, Errors, And Application Integration
@@ -335,7 +341,7 @@ cost, but explanation generation and provenance analysis stay off the run path.
| `internal/profile/filesystem_repository.go`: metadata scans, selected strict decode, stream-end checks, malformed-file selection arbitration | Replace with source acquisition, ConfigKit assembly/decoding, and indexed effective values; remove the metadata prepass and per-lookup filesystem path |
| `internal/promptdef/filesystem_repository.go`: metadata selection and repeated decoding/content opening | Replace with eager prompt loads and an ID/version index; keep domain normalization and contained content reading |
| Profile filesystem, `fs.FS`, eager, memory, and overlay repository variants | Collapse loading into source adapters plus immutable collections; preserve only interfaces that still serve a real boundary |
| `internal/profile/resolving_repository.go` | Replace per-operation traversal and zero-sentinel merging with once-per-snapshot presence-aware typed resolution; graph logic remains |
| `internal/profile/resolving_repository.go` | Remove custom graph traversal, cycle detection, and zero-sentinel merging in favor of ConfigKit sibling inheritance; retain only PromptKit domain validation |
| `internal/engine` raw repository-stack assembly | Replace with ConfigKit source tiers and one effective typed collection; engine capacity/client assembly remains |
| `internal/catalog` profile metadata extraction | Obtain explicit field information from ConfigKit; retain manifests, ownership, namespace, connection-metadata and secret checks |
| `internal/filecatalog` | Retain/reduce discovery and containment helpers actually needed by the eager loader; do not claim the entire package disappears |
@@ -377,10 +383,12 @@ input and produced-data size, not a blanket O(n) claim with an undefined n.
captured content instead of reparsing it for metadata, selection, and validation.
- Build identity indexes in one pass, with expected constant-time map lookup.
Do not scan whole collections for every lookup or precedence decision.
- Process inheritance graph structure in O(V + E), where V is the number of
profiles and E the inheritance edges. Resolve shared ancestors once. Account
separately for the size of materialized settings and retained origins; copying
large inherited maps to many independent results necessarily costs more than V.
- Delegate inheritance expansion to ConfigKit once during loading rather than
rebuilding its graph in PromptKit. Assess scaling against input, materialized
inherited values, nested destination resolution, and retained history; do not
claim unconditional O(V + E) total work from the count of profiles and edges.
ConfigKit bounds expansion and retained evidence through its existing budgets;
representative catalog sizes must fit them without new per-run processing.
- Avoid whole-collection copying per layer, repeated serialization, and repeated
prefix/history traversal. Analyze layer processing against total input nodes
and produced values/provenance, including intentionally retained history.
@@ -397,17 +405,13 @@ The implementation plan should include focused benchmarks for consequential
repeated-work risks, such as shared-ancestor profiles and many layers, rather than
machine-dependent timing gates or benchmarks for every helper.
## 5. Additional Upstream Opportunities
## 5. Upstream Readiness And Remaining Opportunities
Native filesystem sources and public layered loading with origins are the
prerequisites described near the beginning, not optional follow-ups. The following
remain optional and do not block adoption unless the upstream design establishes
a concrete dependency:
| Candidate | Recommendation and tradeoff |
| --- | --- |
| Generic named configuration inheritance | Requested separately in ConfigKit's `docs/roadmap/feature-request-promptkit-inheritance.md`, primarily for shared semantics across consumers. ConfigKit decides whether to adopt it; PromptKit can retain a small domain resolver if declined. Presence-aware source layering belongs to the prerequisite work, not this optional proposal. |
| Context-aware loading | Optional for reusable cancellable construction if large-source consumers need it. Promise cooperative cancellation only; no detached reads or instant interruption of arbitrary readers. |
Filesystem sources, underlays, programmatic mappings, source contexts, and sibling
inheritance are released in v0.5.0. No additional upstream feature is currently a
demonstrated prerequisite. Context-aware loading remains a possible future request
if concrete consumers need it, not a reason to delay adoption. Existing loading
is synchronous; do not simulate cancellation with detached background work.
Do not propose a public raw AST, permissive identity-only decoder, or dynamic
subtree conversion solely to preserve PromptKit's current scanning design.
@@ -418,12 +422,12 @@ ConfigKit and PromptKit both declare LGPL-3.0-or-later. Adoption must still upda
dependency notices under the OSS policy and inspect the exact selected release
and its transitive notices. Their current Go minimums do not require a PromptKit
toolchain increase. Use a released module dependency, not a sibling-directory
replacement. This assessment proposes upstream work, but makes no upstream edits
or licensing change.
replacement. Select v0.5.0 as the assessed adoption baseline; any newer selection
needs its own review. This assessment changes neither dependencies nor licenses.
## 6. Decisions And Migration
The agreed direction now incorporates ConfigKit's working roadmap. This supersedes
The agreed direction now incorporates ConfigKit's v0.5.0 contracts. This supersedes
the earlier recommendations for independently typed documents, whole-definition
shadowing, intermediate typed validation, and atomic maps/null-as-unspecified at
the source boundary.
@@ -438,9 +442,9 @@ the source boundary.
typed loads followed by custom combination are not the selected direction.
3. **Presence and relationships.** Accept recursive source-tier overlay, retained
children for empty mappings, explicit null replacement, and effective metadata.
Named inheritance remains separate pending the upstream decision; settle its
domain semantics before implementation rather than assuming either automatic
equivalence or an exception to ConfigKit's rules.
Adopt built-in sibling inheritance, including reserved `extends`, recursive
mapping combination, null cancellation, and inherited explicit presence. Do
not preserve a parallel PromptKit structural resolver.
4. **Diagnostic boundary.** Keep wrapping ConfigKit's public loading errors rather
than mirroring them. Retain actionable domain relationship and resource errors
without duplicating ConfigKit's generic diagnostic/provenance system.
@@ -450,14 +454,17 @@ the source boundary.
ordinary prompt execution. Finalize targeted query shapes and ownership in the
feature roadmap, with presentation remaining application-owned.
Support the upstream recommendations for labeled mapping-based programmatic input
and direct filesystem resource-context queries. If real consumer examples show
Use released mapping-based programmatic input and direct filesystem resource-context
queries. If real consumer examples show
mapping construction to be repetitive, consider small typed PromptKit conveniences
with explicit presence rather than requiring an upstream typed-ingestion API.
Final upstream signatures and ownership details remain open dependencies of the
feature roadmap, not implemented guarantees. Resolve the named-inheritance
direction there as well; a PromptKit-owned resolver remains viable if ConfigKit
declines or defers that optional request.
Remaining PromptKit decisions are the exact collection/version format, catalog
migration, programmatic ergonomics, operator-inspection surface, and validation
of partial base profiles. Also settle how operators discard unwanted provider
settings or old relationships using the released null semantics without a hidden
replacement pass. Check the reserved-key impact on arbitrary provider maps. These
are integration decisions, not unresolved ConfigKit API proposals.
Expected consumer migration:
@@ -467,11 +474,15 @@ Expected consumer migration:
for historical YAML coercion.
- Move standalone definitions into the agreed keyed contribution format. Review
same-name contributions: disjoint fields within a tier can combine, and higher
tiers retain omitted lower fields, including `base_profile` references.
tiers retain omitted lower fields, including `extends` references. Convert
`base_profile` to sibling `extends`; use `extends: null` to cancel a lower-tier
relationship, not to erase all settings already supplied to the child.
- Review zero/false/empty/null values explicitly. Provider maps merge across
tiers, `{}` does not clear them, and null replaces rather than means omission.
Use labeled programmatic mappings or approved presence-aware conveniences.
Document the separately selected inheritance semantics when settled.
Apply those same combination rules to inherited settings. Rename ordinary
configuration keys named `extends`; examine provider maps for cases where
renaming would change their substantive meaning and resolve those explicitly.
- Handle effective-definition errors at construction, fix syntax/composition
failures even in overridden inputs, and deliberately rebuild/swap engines
after definition changes.
@@ -486,8 +497,9 @@ Expected consumer migration:
custom backend, fallback, and embedded workflows in the migration guidance.
The subsequent feature roadmap should carry forward these choices, resolve the
upstream API details, and specify the target end state. The staged implementation
plan must sequence the prerequisite ConfigKit releases before dependent PromptKit
work, then cover deletions/replacements, contract tests, canonical documentation,
remaining PromptKit integration decisions, and specify the target end state.
The staged implementation plan can now adopt released ConfigKit support without
waiting for upstream feature work, coordinating any catalog format releases before
their consumption. Cover deletions/replacements, contract tests, canonical documentation,
examples, attribution, and minor-release migration notes. Neither document should
become a reason to preserve obsolete compatibility machinery.