eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Verify scoped collection behavior and complete roadmap

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-17 06:39:24 +00:00
parent dfd8da843e
commit 0221cd4610
3 changed files with 138 additions and 36 deletions
Download Patch File Download Diff File Expand all files Collapse all files

74
docs/roadmap/configuration-collections.md
View File

@@ -1,10 +1,10 @@
# Configuration Collections, Namespaces, And Scoped Lookup
Status: accepted feature roadmap; not implemented in PromptKit. This document
consolidates the namespace roadmap and selected collection proposal 3. The
configuration and namespace design decisions are settled. The companion
[implementation plan](implementation.md) defines concrete API delivery choices
and the staged work needed to reach this target state.
Status: implementation complete; roadmap retirement remains for normal release
preparation. This document consolidates the namespace roadmap and selected
collection proposal 3. The configuration and namespace design decisions are
settled. The companion [implementation plan](implementation.md) records concrete
API delivery choices and the completed work.
## Purpose
@@ -27,14 +27,14 @@ and provenance guarantees.
## Prerequisite: Adopt ConfigKit v0.8.0
The target selects released ConfigKit **v0.8.0** for the collection schema and
loading APIs. PromptKit currently selects v0.7.1. Dependency adoption and its
verification belong to the [implementation plan](implementation.md); local
filesystem replacements are not part of the distributed module.
loading APIs. At planning time PromptKit selected v0.7.1; the
[implementation plan](implementation.md) records adoption and verification of
v0.8.0. Local filesystem replacements are not part of the distributed module.
The v0.8.0 tag was reviewed at `137769f` on 2026-10-17. It supplies input
placement, relative import destinations, and shared source-version checks needed
by this design. No additional upstream extension is required. Recording this
prerequisite does not itself upgrade PromptKit's dependency.
by this design. No additional upstream extension is required. The implementation
plan records adoption of this prerequisite.
ConfigKit's public `docs/config.md`, ADRs 0011–0014, and v0.8.0 release notes own
the upstream contracts. The capabilities used here are:
@@ -175,15 +175,16 @@ weather, err := engine.WithNamespaces(Namespaces{
// Handle err, then use local prompt and profile names through weather.
```
These declarations do not exist yet. Empty scope fields mean no scope for that
resource type. Keep name qualification shared across the view and engine; do not
duplicate selection logic in each operation. Initially omit a one-string shorthand
that implicitly scopes all resource types.
The concrete declarations are recorded in the
[implementation plan](implementation.md). Empty scope fields mean no scope for
that resource type. Keep name qualification shared across the view and engine;
do not duplicate selection logic in each operation. Omit a one-string
shorthand that implicitly scopes all resource types.
The view would share the engine's captured configuration, clients, and execution
machinery. Selecting a namespace would not rebuild the engine or duplicate its
backend capacity accounting. Consumers could use different views concurrently
without changing a shared current-namespace setting.
The implemented views share the engine's captured configuration, clients, and
execution machinery. Selecting a namespace does not rebuild the engine or
duplicate its backend capacity accounting. Consumers can use different views
concurrently without changing a shared current-namespace setting.
The accepted resolution rules, independent of the final scope API, are:
@@ -680,10 +681,10 @@ automatically relocate schema or artifact stores or rewrite references when a
collection is renamed.
Retain the accepted removal of mandatory catalog modules and built-in profiles.
This is a breaking configuration/API change under the
[release policy](../policy/release.md); implementation must include consumer
migration guidance and update current contracts when it lands. The proposal
does not change today's supported configuration or API.
This was a breaking configuration/API change under the
[release policy](../policy/release.md); the implementation includes consumer
migration guidance and updates current contracts. The completed work changes
the supported configuration and API as described here.
Existing unscoped exact lookup remains available for IDs valid under the new
grammar. Compatibility with arbitrary slash-containing resource IDs is
@@ -703,8 +704,8 @@ intentionally broken. The migration must:
catalog adoption under the [release policy](../policy/release.md). Its catalog
profile-data exception does not cover this general format/API behavior change;
deliberately retire its built-in-profile preservation promise when the
accepted removal lands. Current policy and format documents remain unchanged
until implementation.
accepted removal landed. Current policy and format documents now reflect the
implemented behavior.
Renaming a backend changes its identity; duplicating it under another ID can
separate capacity accounting. Do not use blanket aliases or duplicate catalog
@@ -728,11 +729,11 @@ can be reconsidered for concrete needs; none blocks implementation.
## Target Acceptance
The target includes the collection schema and final configuration adapter,
catalog and built-in removal, ID-only prompt selection, immutable scoped views,
integrated filtering, and collection inventory. Runtime indexes use full
identities after ConfigKit handoff. Current contracts and consumer migration
guidance describe the implemented behavior when it lands. The
The completed target includes the collection schema and final configuration
adapter, catalog and built-in removal, ID-only prompt selection, immutable
scoped views, integrated filtering, and collection inventory. Runtime indexes
use full identities after ConfigKit handoff. Current contracts and consumer
migration guidance describe the implemented behavior. The
[implementation plan](implementation.md) owns sequencing, concrete public shapes,
documentation tasks, and focused offline validation under the
[testing policy](../policy/testing.md).
@@ -798,15 +799,17 @@ review of release v0.8.0 at `137769f` passed maintained consumer examples and
focused public API probes for sequence assembly, duplicate rejection, whole-list
replacement, inheritance limits, header consumption, and final identity validation
boundaries. The latter review informed the decision to retain mappings. These
checks establish upstream behavior, not an implemented PromptKit feature.
checks establish the upstream behavior relied on by the implemented PromptKit
feature.
## Delivery Reference
No substantive design question remains. ConfigKit v0.8.0 is the selected
dependency. The [implementation plan](implementation.md) specifies helper
dependency. The [implementation plan](implementation.md) records helper
signatures, inventory and optional-label representations, identifier migration
mappings, and consumer migration work. Its delivery choices implement these
accepted contracts without adding speculative options or reopening workflows.
mappings, and completed consumer migration work. Its delivery choices implement
these accepted contracts without adding speculative options or reopening
workflows.
## Consolidation And Retired Alternatives
@@ -838,5 +841,6 @@ part of that request is a prerequisite here.
The accepted configuration choices remain: optional main declarations and
if-present checks; source-root `version` reserved without banning that resource
name; removal of request-version machinery; current ConfigKit inheritance; and
flexible source assembly with a small convenience API. This consolidation does
not change those decisions or implement the feature.
flexible source assembly with a small convenience API. This consolidation
records those decisions and their implementation scope; current behavior is
maintained in the canonical format and consumer documents.

17
docs/roadmap/implementation.md
View File

@@ -1,6 +1,7 @@
# Configuration Collections Implementation Plan
Status: ready for implementation; no feature changes are implemented by this plan.
Status: implementation complete; roadmap retirement remains for normal release
preparation.
Implement the accepted [feature roadmap](configuration-collections.md). That
document owns intent, scope, and selected policy; this document owns API delivery
@@ -828,6 +829,20 @@ Scope: focused integration gaps and final regression validation, not new feature
release preparation. Do not publish, wait for CI, probe providers, or introduce
a catalog-release dependency to finish implementation.
Completion: acceptance coverage was traced across public collection assembly,
scoped execution and inspection, full-ID listings and inventories, exact stored
defaults, optional prompt labels, prepared selection capture, source headers and
origins, final-value validation, namespaced redaction, and retained effective
configuration shape. Added a controlled concurrent regression test proving an
ordinary call and a differently scoped call share one collection backend's
capacity. Audited current source, tests, and docs for obsolete selectors,
automatic catalogs, built-in profile promises, and local module replacements;
remaining version references are descriptive labels or migration guidance.
Marked the feature roadmap complete; normal release preparation owns retirement.
Validation passed: `GOWORK=off go test -race ./...`, `GOWORK=off go vet ./...`,
`GOWORK=off python3 scripts/check-examples.py`, `gofmt`, local Markdown link
checks, and `git diff --check`.
## Open Questions
None required to begin implementation. The feature roadmap's policy decisions

83
smoke_integration_contract_test.go
View File

@@ -59,6 +59,89 @@ func TestSmokeTestSharesBackendCapacityWithRunAndReleasesAfterCancellation(t *te
}
}
func TestScopedCollectionRunsShareBackendCapacity(t *testing.T) {
release := make(chan struct{})
client := newCapacityGateClient(release, 2)
var inputs []configkit.Input
for _, namespace := range []string{"north", "south"} {
input, err := promptkit.CollectionInput(namespace, configkit.Mapping(namespace+" resources", map[string]any{
"profiles": map[string]any{
"shared": map[string]any{"backend": "capacity/backend", "model": namespace + "-model"},
},
"prompts": map[string]any{
"shared": map[string]any{
"default_profile": namespace + "/shared",
"messages": []any{map[string]any{"role": "user", "content": "Hello"}},
"output": map[string]any{"format": "text", "validation_mode": "none"},
},
},
}))
if err != nil {
t.Fatal(err)
}
input.Imports = false
inputs = append(inputs, input)
}
capacityInput, err := promptkit.CollectionInput("capacity", configkit.Mapping("shared capacity", map[string]any{
"backends": map[string]any{
"backend": map[string]any{
"endpoint": "https://example.invalid/v1",
"concurrency_limit": 1,
"queue_capacity": 0,
},
},
}))
if err != nil {
t.Fatal(err)
}
capacityInput.Imports = false
inputs = append(inputs, capacityInput)
engine, err := promptkit.NewEngine(promptkit.Config{Sources: promptkit.ConfigurationSources{
Underlays: []configkit.Underlay{{Inputs: inputs}},
}}, promptkit.WithLLMClient(client))
if err != nil {
t.Fatalf("construct collection engine: %v", err)
}
south, err := engine.WithNamespaces(promptkit.Namespaces{Prompt: "south", Profile: "south"})
if err != nil {
t.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
results := make(chan capacityRunResult, 1)
go runCapacityRequest(engine, ctx, promptkit.RunRequest{
PromptID: "north/shared",
}, results)
select {
case <-client.started:
case <-time.After(5 * time.Second):
cancel()
t.Fatal("ordinary Run did not reach the shared backend")
}
scopedContext, cancelScoped := context.WithCancel(context.Background())
defer cancelScoped()
if _, err := south.Run(scopedContext, promptkit.RunRequest{
PromptID: "shared", ProfileID: "shared",
}); !errors.Is(err, promptkit.ErrCapacityExceeded) {
cancel()
cancelScoped()
t.Fatalf("concurrent scoped Run error = %v, want ErrCapacityExceeded", err)
}
cancel()
outcome := awaitCapacityRun(t, results)
if outcome.result != nil || !errors.Is(outcome.err, context.Canceled) {
t.Fatalf("ordinary Run outcome=(%+v, %v), want context cancellation", outcome.result, outcome.err)
}
_, peak, calls := client.snapshot()
if calls != 1 || peak != 1 {
t.Fatalf("shared backend calls=%d peak=%d, want one admitted call", calls, peak)
}
}
type failOnceSmokeClient struct {
mu sync.Mutex
calls int