eric/promptkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Retire completed implementation and review roadmaps

Browse Source
This commit is contained in:
Eric Rakestraw
2026-10-17 14:31:29 +00:00
parent 8627c751a4
commit 7091b51c87
4 changed files with 0 additions and 2480 deletions
Download Patch File Download Diff File Expand all files Collapse all files

846
docs/roadmap/configuration-collections.md
View File

@@ -1,846 +0,0 @@
# Configuration Collections, Namespaces, And Scoped Lookup
Status: implementation complete; roadmap retirement remains for normal release
preparation. This document consolidates the namespace roadmap and selected
collection proposal 3. The configuration and namespace design decisions are
settled. The companion [implementation plan](implementation.md) records concrete
API delivery choices and the completed work.
## Purpose
Simplify PromptKit configuration by loading independently maintained prompt,
profile, and backend libraries as YAML collections instead of mandatory Go
modules. Remove the unused prompt-version selection scheme. Specification
headers are optional: maintainers can declare compatibility in a collection
entry without adding a declaration to every file.
Consumers can select a namespace once and use local resource names while an
engine contains several collections with overlapping names. For example,
WeatherReporter and Homewatch can each define `summarize.alerts`, while provider
namespaces distinguish profiles targeting the same upstream model.
Conventions and small convenience APIs should reduce authoring and maintenance
work. Preserve consumer choice over sources, layouts, namespaces, and precedence
without adding speculative options, mandatory workflows, or new compatibility
and provenance guarantees.
## Prerequisite: Adopt ConfigKit v0.8.0
The target selects released ConfigKit **v0.8.0** for the collection schema and
loading APIs. At planning time PromptKit selected v0.7.1; the
[implementation plan](implementation.md) records adoption and verification of
v0.8.0. Local filesystem replacements are not part of the distributed module.
The v0.8.0 tag was reviewed at `137769f` on 2026-10-17. It supplies input
placement, relative import destinations, and shared source-version checks needed
by this design. No additional upstream extension is required. The implementation
plan records adoption of this prerequisite.
ConfigKit's public `docs/config.md`, ADRs 0011–0014, and v0.8.0 release notes own
the upstream contracts. The capabilities used here are:
- `Input.Destination` for root-relative placement and import `destination` for
placement relative to the immediate importing document.
- `Options.SourceVersionMode` and `LayeredOptions.SourceVersionMode`, applying
a shared `Definition.Version` policy in each document's source-local coordinates.
- Consumption of accepted non-main headers before composition, with the main
header retained and protected at its effective application path.
- Detached `SourceInfo.Destination` and `SourceInfo.VersionCheck` evidence,
together with existing source locations and resource contexts.
### Keep Mapping-Based Collections
ConfigKit v0.8.0 also offers sequence-item placement and registered composition
of distinct list records. These were reviewed and are not selected for this
round. A higher-priority list replaces the entire lower-priority list, duplicate
identities conflict within a strict group, and records cannot inherit from
sibling list records. Switching formats would sacrifice ordinary per-resource
partial overrides and local inheritance for flatter authoring.
Keep collection resources as mappings keyed by local ID. Do not add an
alternative list schema or a PromptKit identity-based merge layer. The existing
key supplies identity without an additional embedded `id`. This is a scope
choice for this round, not a prohibition on future upstream capabilities.
## Existing Foundation And Ownership
The [format reference](../formats.md) and
[consumer guide](../consumers/pkg-promptkit.md) remain authoritative for current
behavior until implementation lands. Directory and embedded configuration
loading already exist through `DirectoryInputs` and `FilesystemInputs`.
The current engine also supplies provider catalog modules and built-in profiles;
the accepted removal is described below.
[ADR 0001](../adr/0001-configuration-boundary.md) assigns source loading,
composition, and inheritance to ConfigKit. PromptKit interprets and validates
only final effective values, including their resource contexts. Source history
is available for explicit inspection, not additional validation of contributions.
Do not add a manifest pre-parser, source-level namespace ownership checks, or a
second composition pass. Two sources targeting the same namespace follow
ordinary ConfigKit composition; a repeated namespace is not a separate error.
The [architecture policy](../policy/architecture.md) continues to assign CLI
syntax, discovery policy, and application lifecycle to consuming applications.
## Collection Ownership And Built-In Removal
**Accepted: load collections as application/operator-supplied configuration,
and remove PromptKit's built-in profiles as a breaking change.** Remove the
mandatory OpenRouter and RakestrawHome catalog Go-module dependencies and their
automatic contribution to engine configuration. PromptKit owns the definition
format and execution machinery; collection publishers own their configuration
repositories, and applications/operators select and provision the collections
they use. One or more explicitly selected directories can supply backends,
profiles, and prompts through the existing ConfigKit integration.
Remove `promptkit-openrouter-standard` and `promptkit-openrouter-auto`, including
their implicit catalog dependencies. Do not replace them with another mandatory
starter bundle. Applications may embed their own defaults using the same
configuration format and loader, then layer operator configuration as desired.
Ordinary application overrides remain valid configuration and need not declare
a collection release of their own.
This decouples collection updates from PromptKit rebuilds and releases and
avoids imposing provider inventories on every consumer. The tradeoff is that
applications and deployments must supply their required configuration and
manage compatible collection revisions. PromptKit does not gain a collection
installer, automatic downloader, or implicit directory search policy.
Retain the captured-engine lifecycle: configuration changes take effect when
the application constructs and validates a replacement engine. A change to an
installed directory does not mutate an existing engine's captured definitions.
Distribution as configuration does not make new provider protocols or diagnostic
adapters data-only features; those can still require implementation work.
Prepare explicit identifier mappings, replacement configuration examples, and
consumer migration instructions. Update the built-in-profile promise in the
release policy and its canonical format documentation when removal lands.
Use a release permitted to break configuration and API contracts under the
current release policy: a minor release before v1, or a major release at/after
v1. Actual catalog mappings, packaging arrangements, and release versions remain
delivery work; independent collection releases need not track future PromptKit
software releases.
## Accepted Identity Convention
Reserve `/` exclusively as the namespace separator in PromptKit prompt, profile,
and backend identifiers and references. A qualified identifier has exactly two
nonempty components: `namespace/local-name`. Neither component can contain `/`.
Dots remain available within local names. An unqualified identifier has no slash
and remains valid under the existing nonblank-ID rule.
Reject resource identifiers or supplied references with multiple slashes or an
empty qualified component, such as `a/b/c`, `/name`, and `namespace/`. These
rules do not apply to provider model values, endpoints, resource paths, or prompt
versions. Preserve exact spelling; do not silently trim or rewrite IDs.
| Resource | Namespace | Local name | Qualified ID |
| --- | --- | --- | --- |
| Prompt | `weatherreporter` | `summarize.alerts` | `weatherreporter/summarize.alerts` |
| Prompt | `homewatch` | `summarize.alerts` | `homewatch/summarize.alerts` |
| Profile | `openrouter` | `google.gemma-4-31b-it` | `openrouter/google.gemma-4-31b-it` |
| Profile | `rakestrawhome` | `google.gemma-4-31b-it` | `rakestrawhome/google.gemma-4-31b-it` |
Prompt, profile, and backend identities remain separate resource types. The
same namespace can contain all three, and resources in it may come from
multiple configuration sources. A namespace does not imply a directory,
package, provider model identifier, or security boundary.
A profile ID identifies execution settings. Its configured model value is
still the identifier passed to the provider; qualifying the profile ID does
not modify that value. Both example profiles may retain the configured model
`google/gemma-4-31b-it`. Different profiles may select the same model with
different settings. Prompt version labels remain separate from namespaces. The
selected collection format installs one definition per prompt ID and removes
request-side version selection and assertions. Optional labels are descriptive.
Retain unqualified and dotted IDs. Existing slash-containing IDs need review:
those with one slash and nonempty components become qualified identities, while
those outside the new grammar require migration. This intentional format break
is accepted in favor of unambiguous resolution; the migration work is described
below.
## Accepted Consumer Scope
Provide a lightweight, immutable view over an existing engine with independent
prompt, profile, and backend scopes. The following sketch illustrates the
accepted shape; concrete declarations are specified in the
[implementation plan](implementation.md):
```go
weather, err := engine.WithNamespaces(Namespaces{
Prompt: "weatherreporter",
Profile: "openrouter",
})
// Handle err, then use local prompt and profile names through weather.
```
The concrete declarations are recorded in the
[implementation plan](implementation.md). Empty scope fields mean no scope for
that resource type. Keep name qualification shared across the view and engine;
do not duplicate selection logic in each operation. Omit a one-string
shorthand that implicitly scopes all resource types.
The implemented views share the engine's captured configuration, clients, and
execution machinery. Selecting a namespace does not rebuild the engine or
duplicate its backend capacity accounting. Consumers can use different views
concurrently without changing a shared current-namespace setting.
The accepted resolution rules, independent of the final scope API, are:
1. A supplied name without a slash is qualified using the selected scope for its
resource type, then looked up exactly. Without that scope, it is looked up as
an exact unqualified ID.
2. A missing scoped resource does not trigger a search in other namespaces or
among unqualified IDs. Loading another collection cannot redirect lookup.
3. A valid supplied name containing one slash is fully qualified and is looked
up exactly, regardless of scope. Malformed resource references are rejected.
4. Scope applies to caller-supplied resource names. Stored references, including
prompt default profiles, profile backends, inheritance references, and backend
smoke-test default profiles, retain their existing exact meaning. They are
not rebound to the invoking consumer's namespace.
5. Omitted optional IDs retain their existing selection/default behavior; an
omission is not converted into a namespace prefix or a lookup for an empty
local name.
6. Where outputs, inspection, errors, or diagnostic reports identify a selected
resource, preserve its resolved full identity. Any exposed local name should
be distinguishable from that identity.
Keep qualification consistent across execution, preparation, request validation,
prompt/profile inspection, and applicable smoke-test selections. Namespace
selection must not change request override precedence or provider model strings.
Prompts resolve by ID alone.
### Local And Qualified References In One Operation
Syntax alone distinguishes local and qualified references. Do not decide whether
to qualify a name by checking which candidates exist; that would make resolution
depend on loaded content.
Within a `weatherreporter` prompt scope, `summarize.alerts` resolves only to
`weatherreporter/summarize.alerts`, while `homewatch/summarize.alerts` explicitly
selects that other library's prompt. A qualified profile override similarly
bypasses the profile scope. Scoped operations accept both forms without a
separate reference type or requiring a return to the original engine for
cross-namespace access.
The original engine remains a way to select an exact unqualified ID without
applying a scope. No escape syntax for unqualified IDs is introduced here.
### Separate Resource Scopes
A consumer can select prompt shorthand under `weatherreporter` and profile
shorthand under `openrouter` in the same request. Each resource type has an
independent scope.
A prompt's configured default profile needs no caller profile scope because
the stored reference is already exact. Scope syntax and existence handling are
defined in [scope validation](#scope-validation).
### Scope Validation
**Accepted: validate syntax when creating the view; resolve existence when
an operation selects a resource.** In the scope value, the empty
string means no scope. Reject a nonempty namespace containing `/` or consisting
only of whitespace. Otherwise preserve its exact spelling, with no trimming,
case folding, or additional character whitelist. This distinguishes omission
from an unusable namespace without inventing normalization rules. Report invalid
scope construction through PromptKit's existing public error conventions.
Allow a well-formed namespace even when it has no current members. Its scoped
listing would be empty, and selection of a missing local resource would produce
the ordinary lookup error. Qualified selections would remain usable regardless
of the view's default scope. An empty optional request ID continues to mean
omission under that operation's existing contract; it is never qualified.
This keeps scope as a naming context and supports uniform application setup
across installations with different library contents. It also avoids treating
successful scope construction as validation of a prompt or execution profile.
The tradeoff is that a misspelled but syntactically valid namespace is detected
only when listed or used, rather than during view construction.
## Discovery And Namespace Filtering
**Accepted: integrated namespace filtering and full-identity results.**
Enumerate the distinct named namespaces present in final effective
prompts, profiles, and backends. Allow filtering that enumeration by resource
type so a prompt picker need not show profile-only namespaces. Provide scoped
listings for the supported resource types, adding a backend inventory surface
as part of this work if needed. Namespace filtering is an explicit option in
the library listing API, not a consumer-side filtering helper. Preserve unfiltered
engine listing behavior, with the accepted removal of prompt-version identity
and selection. The companion implementation plan defines the Go filter shape.
| Selection | Listing behavior |
| --- | --- |
| No explicit filter on the engine | All namespaces, including unqualified resources |
| No explicit filter on a scoped view | The selected scope for that resource type; all resources of that type if no scope is selected |
| Explicit named namespace | Resources in exactly that namespace |
| Explicit unqualified-only filter | Resources whose IDs have no namespace |
| Explicit all-namespaces filter | All resources of that type, including unqualified resources |
An explicit filter replaces the view's default for the current listing; it
does not intersect with that default or mutate the view. A named namespace
filter uses the namespace syntax rules above. A well-formed namespace
with no matching resources returns an empty listing. Represent default, named,
unqualified-only, and all-namespaces selections distinctly in the API; do not
overload an empty string to mean several of them.
Return full IDs with optional prompt version labels separately; the full ID
identifies the installed prompt. Do not invent a named namespace such as `default`
for unqualified resources. Namespace enumeration excludes that unnamed group.
Use deterministic ordering consistent with existing listing contracts.
Qualified result IDs can be passed unchanged to either scoped or unscoped
operations. Unqualified IDs remain literal unqualified identities: when an
explicit all-namespaces or unqualified-only filter exposes one through a scoped
view, use the original engine or a view without that resource scope to select
it exactly. A listing filter does not change subsequent execution scope. This
preserves the accepted lookup rules without introducing an escape syntax.
Integrated filtering gives consumers a direct way to discover the desired
resources, while full identities keep logs and subsequent selections
unambiguous. Namespace/local-name display decomposition is separate from
filtering; a public splitting helper is not required by this decision.
Namespace membership is exact: `weather` does not include `weatherreporter`.
Collection inventory describes final collection declarations and optional release
labels; it is distinct from enumeration of namespaces that contain resources.
See [collection inventory](#inspection-and-effective-collection-inventory).
## Authoring And Assembly
A file such as `weatherreporter/prompts/alerts.yml` would contain:
```yaml
summarize.alerts:
version: "1"
default_profile: provider/brief
messages:
- role: user
content: Summarize the weather alerts.
output:
format: text
validation_mode: none
```
The version may simply be omitted. Another collection can contribute the same
local prompt name, for example `homewatch/prompts/alerts.yml`:
```yaml
summarize.alerts:
default_profile: provider/brief
messages:
- role: user
content: Summarize the household alerts.
output:
format: text
validation_mode: none
```
A provider collection's `profiles/brief.yml` uses the same local-key convention:
```yaml
brief:
backend: provider/default
model: vendor/example-model
temperature: 0.2
```
Its `backends/default.yml` could contain this synthetic, non-live endpoint:
```yaml
default:
endpoint: https://inference.example.invalid/v1
```
A conventional `weatherreporter/collection.yml` entry can declare the shared
specification, optional collection release, and its layout:
```yaml
version: 1
release_version: "2.4.0"
configkit:
imports:
- glob: prompts/*.yml
destination: [prompts]
```
A provider entry can import `profiles/*.yml` at `[profiles]` and
`backends/*.yml` at `[backends]` in the same way. Collection publishers choose
which categories to include. A required glob must match; unused categories can
be omitted or use ConfigKit's existing optional-import behavior. Ordinary
fragments may omit specification headers under the selected `IfPresent` mode.
An entry header is the recommended publishing convention, not a load-time
requirement selectively enforced on entries by this mode.
The application selects each entry, enables its imports, and supplies its
namespace through the public input destination. For example, the following
partial Go snippet uses released ConfigKit v0.8.0 APIs:
```go
entry := configkit.Input{
Source: configkit.File("/usr/local/share/collections/weatherreporter/collection.yml"),
Imports: true,
Destination: configkit.Keys("collections", "weatherreporter"),
}
```
For independent peer collections, one `Underlay` group's `Inputs` is a useful
starting point, not a required grouping. Consumers choose precedence and groups.
Configure `LayeredOptions.SourceVersionMode` as `SourceVersionModeIfPresent` and
`Definition.Version.Path` as `configkit.Keys("version")`, with PromptKit's
supported-specification checker. The application main input remains at the
root and declares `version: 1` in the examples below; both the main input and
its declaration are optional. Namespace and source selection require no manifest parsing in PromptKit.
| Selected source | Effective destination |
| --- | --- |
| Application main document | Configuration root |
| `weatherreporter/collection.yml` | `[collections, weatherreporter]` |
| Its `prompts/*.yml` imports | `[collections, weatherreporter, prompts]` |
| Homewatch entry and its prompt imports | `[collections, homewatch]` and `[collections, homewatch, prompts]` |
| Provider entry and its profile/backend imports | `[collections, provider]`, then `[profiles]` or `[backends]` appended |
Directly selected fragments with `Input.Destination` remain possible. Entry
imports are the recommended packaging convention because publishers can describe
their layout. Filenames and directory basenames do not define resource identity;
no implicit directory discovery, namespace inference, or collection installer is
introduced. A small helper should construct inputs from an explicitly selected
source and namespace, without inspecting manifests to compute destinations. `collection.yml`
is a convention, not a required filename. Direct fragments and programmatic
sources remain supported through the source API; no parallel collection loader
or mandatory filesystem layout is introduced.
Use later underlay groups or main for intentional application/operator overrides.
Placement does not establish precedence. Do not also recursively scan files
already selected by entry imports: ConfigKit's repeated-source rules still apply.
The entry's source-root `version` is checked and consumed before its payload
composes. Its `release_version` is retained as ordinary collection metadata.
A prompt's nested optional `version` is also ordinary payload. These are three
distinct concepts; source-root release metadata must no longer use `version`.
The assembled application shape would be equivalent to:
```yaml
version: 1
collections:
weatherreporter:
release_version: "2.4.0"
prompts:
summarize.alerts:
version: "1"
default_profile: provider/brief
messages:
- role: user
content: Summarize the weather alerts.
output:
format: text
validation_mode: none
homewatch:
prompts:
summarize.alerts:
default_profile: provider/brief
messages:
- role: user
content: Summarize the household alerts.
output:
format: text
validation_mode: none
provider:
backends:
default:
endpoint: https://inference.example.invalid/v1
profiles:
brief:
backend: provider/default
model: vendor/example-model
temperature: 0.2
```
This yields `weatherreporter/summarize.alerts` and
`homewatch/summarize.alerts`, plus profile `provider/brief` and backend
`provider/default`. The provider's model string remains `vendor/example-model`;
it is not a PromptKit resource identifier.
One fragment may contain several local definitions. A fragment starting directly
with `messages:` would need a destination that also supplies the local prompt
key. ConfigKit mounting permits such placement, but with source checks enabled
its root `version` would be the specification header, not prompt metadata. Recommend
the local-key mapping above as the standard authoring form: identity travels
with the definition and files may be reorganized without changing it. Filename-derived IDs are not
needed.
## Final Schema And Identity
Use `collections` as a mapping of namespace to collection body, containing
optional `release_version` and optional `backends`, `profiles`, and `prompts`
mappings. Collection release labels are descriptive, with no SemVer ordering
or dependency-solving semantics. Metadata is optional: an installed collection
can contain resources without claiming a published release.
Each resource key inside a collection is local. The parent namespace and local
key follow the accepted slash-free component grammar; their public spelling is
exactly `namespace/local-name`. There are no duplicate embedded `id` or
`namespace` fields to reconcile. Dots remain ordinary characters.
Retain unqualified application resources through optional root-level
`backends`, `profiles`, and `prompts` mappings, also keyed by local name and
using the same definition shapes. They produce unqualified public IDs. Under
this proposal, namespaced definitions belong under `collections`, not a second
qualified-key spelling in those root maps. This avoids two configuration paths
producing the same public identity. This representation belongs to the selected
future direction; it does not change today's implemented format.
Stored references such as `default_profile`, profile `backend`, and smoke-test
profile defaults remain exact public references. Nesting does not automatically
qualify a slashless reference or rebind it to caller scope. For example, a
profile in `provider` still declares `backend: provider/default`. Caller-scoped
lookup follows [the scope rules above](#accepted-consumer-scope) after final
indexes are built.
### One Installed Prompt And Optional Version
The resource's qualified ID is its sole configuration identity. There is no
version-key mapping, implicit `latest`, version ordering, or parallel installed
history. Updating the optional version changes metadata on the same prompt.
An optional prompt version is an opaque string with no SemVer, nonblank, or
normalization requirement. Omission or explicit null means no declared label;
null can clear a lower-layer or inherited label. Quote numeric-looking labels
such as `"1"`. Preserve supplied strings, including empty strings, and distinguish
an absent label in inspection. Do not synthesize a version.
Remove request-side prompt-version fields, version selection, ordering, and
version defaults as part of the breaking change. Do not reinterpret those fields
as exact-match assertions. Execution and inspection resolve the installed prompt
by ID alone. Listings expose one prompt per ID with its optional descriptive
label separately. A label neither controls execution nor proves unchanged content.
## ConfigKit Composition Boundary
Use the implemented upstream contracts without duplicating them in PromptKit.
Input destinations use literal mapping-key segments, independent of filesystem
roots and precedence. Relative import destinations are sequences of strings;
omission and `[]` retain the importing destination. Null, scalar, and non-string
destination members are invalid. No logical path segment means parent traversal
or a reset to the application root.
Source-root import directives remain structural. Imported content contributes at
its resulting destination, while physical import paths and `content_file` bases
remain relative to their declaring sources. `SourceInfo.Destination` records
placement; normal origins and diagnostics retain real source locations. Selected
overlays and explicit CLI overrides remain rooted at the application root.
Mounting preserves source identity, cycle/repetition rules, and load limits.
An absent `NoSource` contributes no payload; an actual empty mapping can
materialize its destination containers without clearing lower contents.
ConfigKit's reserved structural keys continue to apply. PromptKit must not
strip headers, re-key source trees, merge fragments itself, or turn source
inspection into an additional enforcement pass.
### Composition And Overrides
Different collection keys prevent accidental collisions between equal local
resource names. Two sources targeting the same namespace are not automatically
invalid: disjoint mapping contributions combine. Overlapping scalar fields,
message sequences, nulls, and equal release labels conflict within a strict
group. Across deliberate layers they follow ordinary precedence instead.
A higher-layer prompt mapping is a partial override. Changing only its version
does not replace its messages or other settings. Omitting a field retains its
lower value, and an empty mapping does not clear a definition. Publishers and
operators must treat such changes as configuration customization, not as an
atomic installation of another prompt release. Do not introduce source audits
or a version-triggered replacement rule to conceal this distinction.
### Inheritance
Existing sibling inheritance remains useful within each collection's resource
mapping: `extends: brief` can name the local profile `brief`. Prompt versions,
now ordinary optional fields, would inherit and overlay like other fields;
`version: null` explicitly clears an inherited label.
Cross-collection profiles are no longer immediate siblings. Consequently,
`extends: provider/brief` cannot reach that collection under current ConfigKit
inheritance. Use existing local sibling inheritance for this development round;
applications can still select the qualified `provider/brief` profile directly.
Authors needing a customized derivative may publish an expanded definition.
Expansion is a migration option, not a mandatory authoring workflow.
Generic cross-path inheritance remains possible future ConfigKit work if a
concrete consumer need justifies it. It is not a prerequisite of this round.
PromptKit must not add its own inheritance resolver after ConfigKit handoff.
## Specification Versions And Collection Inventory
### Shared Specification Policy
Use one `Definition.Version` policy, with source-local path `version`, for the
candidate PromptKit specification. Initially accept specification text `1` from
string or strict decimal-integer declarations; reject explicit null and other
unsupported declarations through the ConfigKit checker. Accept an absent main
declaration, including an absent main input. Accept additional versions only
when PromptKit actually supports their format semantics: accepting
a header does not migrate its payload or independently validate a fragment's
complete schema. The retired proposals were alternatives, not released formats
sharing specification version 1.
Use `SourceVersionModeIfPresent` without adding a PromptKit mode-selection
option in this round. A stricter option can be considered for a demonstrated
consumer need. The upstream modes remain distinct:
| Mode | Implemented behavior | Role in this proposal |
| --- | --- | --- |
| `Default` | Existing main-only effective-path check and protected-path behavior | Not the proposed shared-source policy |
| `IfPresent` | Checks present declarations; skips absent non-main declarations | Initial choice for compact fragments |
| `Required` | Requires declarations on every participating document | Upstream capability; exposing a PromptKit option is deferred |
Participating documents include actual main and underlay inputs, transitive and
glob imports, and the selected overlay. Defaults, synthetic CLI overrides,
resolution output, absent optional imports, and empty groups do not need headers.
Explicit null is present and reaches the checker; malformed selected values
fail ConfigKit's version-type validation. A rejected declaration fails loading
even if a higher layer would supersede that source's payload.
In enabled modes, accepted non-main headers are consumed before composition.
The main declaration remains and is protected at its destination-prefixed path.
PromptKit's recommended main is unmounted, so that protected path is `/version`.
Do not confuse header consumption with permission to override the remaining
protected application field.
`IfPresent` does not require a collection entry header and does not infer a
fragment's compatibility from its entry. A publisher can declare compatibility
in the entry while leaving imported files headerless; the assembled configuration
still undergoes normal validation. A declaration is not a guarantee of future
compatibility or independent schema validation of every imported fragment.
ConfigKit checks the declarations actually present. Requiring headers only on
entry files is not supported by the load-wide mode. Do not emulate it using a
preliminary manifest load or validation of
source history. If that stronger guarantee becomes essential, seek an explicit
upstream contract instead.
### Naming And Optional Main Inputs
Reserve source-root `version` for specification declarations throughout the new
format. Use `release_version` for optional collection release labels and retain
nested prompt `version` for the optional installed-prompt label. An old release
file containing only `version: "2.4.0"` can fail the specification check; if its
text happens to be accepted, that field is consumed and lost as release metadata.
Migration must rename it rather than simply enable the new mode.
This reservation affects resource fragments whose local key is literally
`version`: a root mapping with that name is interpreted as a malformed header.
Allow that resource name through a nested resource mapping, for example
`prompts: {version: ...}` in a collection entry. Reserve the header position,
not the public resource name. Existing `configkit` and `extends` structural
reservations also remain applicable; destinations do not bypass them.
An absent header in an actual main document still reaches the checker under
`IfPresent`; `NoSource` invokes the absent-main checker as well. PromptKit's
checker accepts omission in both cases. Consumers may assemble only underlays,
fragments, or programmatic inputs without manufacturing a main document or
header. Do not inject a declaration or report undeclared sources as checked.
Present unsupported declarations still fail; final schema validation still runs.
### Inspection And Effective Collection Inventory
`SourceInfo.VersionCheck` is source evidence, not a generic success flag. A
non-nil record with `Present == false` can describe an omission for which the
checker did not run. Header paths and locations are source-local. Evidence is
detached and does not reread inputs; text is conservatively redacted during
callbacks and on failed loads, and successful final records respect completed
sensitivity classification. These records belong in explicit configuration
inspection, not in ordinary domain validation or collection-release identity.
Derive collection inventory from the final `collections` mapping: namespace
plus optional effective `release_version`, sorted and detached from the captured
engine. Metadata-only and resource-only entries can both appear. Root unqualified
resources do not invent a named collection; resource-based namespace enumeration
remains separate. The companion implementation plan defines exported inventory
types.
An inventory describes effective configuration, not immutable package identity,
historical source ownership, or proof of unchanged published contents. Operator
overrides may change resources while retaining a release label. Engine
construction captures definitions; edits require a replacement engine.
## Migration
Migration must choose one installed definition for each former multi-version
prompt, move any retained version into its optional field, and relocate
qualified resource keys into collection/local-key mappings. Update destination
declarations, stored references, inventories, and configuration-inspection paths.
Rename collection release metadata to
`release_version`, configure the shared checker and mode, and account for the
source-root header reservation and optional main declarations. Remove obsolete
request-version fields and version-selection logic from consumer examples and
public APIs rather than retaining them as assertions. Preserve provider model
strings and source resource bases. Namespace mounting does not
automatically relocate schema or artifact stores or rewrite references when a
collection is renamed.
Retain the accepted removal of mandatory catalog modules and built-in profiles.
This was a breaking configuration/API change under the
[release policy](../policy/release.md); the implementation includes consumer
migration guidance and updates current contracts. The completed work changes
the supported configuration and API as described here.
Existing unscoped exact lookup remains available for IDs valid under the new
grammar. Compatibility with arbitrary slash-containing resource IDs is
intentionally broken. The migration must:
- Audit maintained catalogs and representative consumer definitions for invalid
IDs and for valid one-slash IDs whose namespace meaning needs review.
- Choose explicit old-to-new mappings and check for collisions. Do not
automatically replace slashes with dots: distinct existing IDs could collapse
to the same new identity.
- Update renamed keys and their references together, including inheritance,
prompt default profiles, profile backend references, backend smoke-test
defaults, and affected examples and consumer request IDs.
- Keep provider model strings unchanged. Renaming a profile does not rename
the upstream model.
- Document consumer action and coordinate the PromptKit format change with
catalog adoption under the [release policy](../policy/release.md). Its catalog
profile-data exception does not cover this general format/API behavior change;
deliberately retire its built-in-profile preservation promise when the
accepted removal landed. Current policy and format documents now reflect the
implemented behavior.
Renaming a backend changes its identity; duplicating it under another ID can
separate capacity accounting. Do not use blanket aliases or duplicate catalog
entries as an automatic compatibility mechanism. Exact release versions and
catalog mappings remain implementation-planning work.
## Work Deferred Or Outside This Round
- Collection installation, automatic downloading, implicit directory discovery,
package registries, dependency solving, and live reload.
- Automatic rewriting of stored references when mounting or renaming collections.
- Implicit namespace search paths, fallback lookup, namespace-based access control,
and filesystem isolation.
- Changes to provider model names, inference behavior, or ConfigKit precedence.
- Sequence-item resource catalogs, identity-based overlays, cross-path inheritance,
and a PromptKit option requiring specification headers in every document.
Explicit source destinations, direct fragments, programmatic sources, and
consumer-selected precedence remain supported in the design. Deferred features
can be reconsidered for concrete needs; none blocks implementation.
## Target Acceptance
The completed target includes the collection schema and final configuration
adapter, catalog and built-in removal, ID-only prompt selection, immutable
scoped views, integrated filtering, and collection inventory. Runtime indexes
use full identities after ConfigKit handoff. Current contracts and consumer
migration guidance describe the implemented behavior. The
[implementation plan](implementation.md) owns sequencing, concrete public shapes,
documentation tasks, and focused offline validation under the
[testing policy](../policy/testing.md).
### Namespace And Consumer Acceptance
- Two prompt scopes resolve the same local name to their respective prompts;
a missing name never resolves from the other scope or an unqualified entry.
- Two provider profile scopes resolve the same local name, such as
`google.gemma-4-31b-it`, to different profiles without changing the configured
provider model value `google/gemma-4-31b-it`.
- A qualified reference bypasses the selected scope without double qualification;
adding another collection does not change local or qualified resolution.
- Multiple slashes and empty qualified components are rejected in resource IDs
and references, while provider model values retain their existing syntax.
- Prompt shorthand and a profile override use their intended scopes, while
stored default references and omitted selections retain their meaning.
- Scoped views coexist concurrently and share backend capacity with ordinary
calls on the underlying engine.
- Scope creation rejects invalid syntax, accepts well-formed absent namespaces,
and treats empty scope fields as no scope. Missing resources fail at lookup;
qualified references remain usable even when the default namespace is absent.
- Listing without an explicit filter uses the resource's view scope, or lists
all resources when no scope is selected. Explicit named, unqualified-only,
and all-namespaces filters override that default for the current listing only.
- A valid namespace with no matches produces an empty listing. Namespace
filtering respects resource type and exact namespace boundaries.
- Exact-ID calls using valid IDs and intentional overlays retain their behavior.
Migrated catalog references resolve without collisions; migration installs one
prompt per ID with an optional descriptive label.
- Listings preserve full IDs and separate optional prompt version labels.
Qualified results round-trip through either engine or view; unqualified results
round-trip through the engine without a corresponding scope.
- Representative validation, inspection, execution, and smoke-test paths use
the same qualification rules without duplicating every test at every layer.
- Engines use caller-selected definitions without automatically contributing
provider catalogs or PromptKit-owned profiles. Ordinary directory and embedded
configuration sources retain ConfigKit composition semantics.
- PromptKit does not add source-history-based duplicate-namespace rejection or
independently pre-validate collection definitions before the final handoff.
### Configuration Acceptance
Exercise typed assembly of multiple collections, relative/nested imports,
source-local header checking and consumption, ordinary release and prompt
metadata, unqualified resources, optional prompt labels, inventory without
release metadata, source-relative content, and intentional partial overrides.
Cover headerless entries/fragments, omitted main inputs and headers, and
resources named `version` in nested mappings. Verify ID-only lookup without
request-version selection or assertions. No automatic provider catalogs or
built-in profiles should be contributed.
Do not duplicate ConfigKit's generic source validation or enforce domain rules
through historical source inspection. Preserve local inheritance and ordinary
mapping overrides while migrating catalog definitions.
### Review Evidence
The 2026-10-16 review against ConfigKit development commit `e66c371` verified
mounted assembly, relative imports and resource contexts, shared header checks,
and this document's YAML assembly through public consumer tests. The 2026-10-17
review of release v0.8.0 at `137769f` passed maintained consumer examples and
focused public API probes for sequence assembly, duplicate rejection, whole-list
replacement, inheritance limits, header consumption, and final identity validation
boundaries. The latter review informed the decision to retain mappings. These
checks establish the upstream behavior relied on by the implemented PromptKit
feature.
## Delivery Reference
No substantive design question remains. ConfigKit v0.8.0 is the selected
dependency. The [implementation plan](implementation.md) records helper
signatures, inventory and optional-label representations, identifier migration
mappings, and completed consumer migration work. Its delivery choices implement
these accepted contracts without adding speculative options or reopening
workflows.
## Consolidation And Retired Alternatives
This document replaces the separate namespace roadmap and collection proposal 3.
Their pre-consolidation versions, including considered namespace API alternatives,
remain available at PromptKit commit `ddda2f8`:
```sh
git show ddda2f8:docs/roadmap/namespaces.md
git show ddda2f8:docs/roadmap/configuration-collections-mounted-fragments.md
```
The unselected collection proposals remain available at `b5866dd`:
```sh
git show b5866dd:docs/roadmap/configuration-collections-current-configkit.md
git show b5866dd:docs/roadmap/configuration-collections-identified-records.md
```
Proposal 1 used fully qualified mapping keys. Proposal 2 requested identified
record sequences, identity-based overlays, and record inheritance. ConfigKit
v0.8.0 implements a narrower sequence-composition capability; it does not supply
those identity-based overlays or inheritance. Its sequence feature was evaluated
and not selected for this roadmap. The original upstream feature-request lineage
is recorded in ConfigKit history under
`docs/roadmap/promptkit-identified-records-feature-request.md`; no outstanding
part of that request is a prerequisite here.
The accepted configuration choices remain: optional main declarations and
if-present checks; source-root `version` reserved without banning that resource
name; removal of request-version machinery; current ConfigKit inheritance; and
flexible source assembly with a small convenience API. This consolidation
records those decisions and their implementation scope; current behavior is
maintained in the canonical format and consumer documents.

852
docs/roadmap/implementation.md
View File

@@ -1,852 +0,0 @@
# Configuration Collections Implementation Plan
Status: implementation complete; roadmap retirement remains for normal release
preparation.
Implement the accepted [feature roadmap](configuration-collections.md). That
document owns intent, scope, and selected policy; this document owns API delivery
choices and the ordered implementation steps. Each numbered stage is sized as
one prompt for a Luna coding agent. Complete stages in order. Foundation and
migration stages are parts of one refactor, not separately publishable releases.
## Baseline And Working Rules
Reviewed PromptKit baseline: `683689a4f89d0fea6a8e0c6faf5830414adf9c7b`.
It selects ConfigKit `v0.7.1`, both `/v3` provider catalogs at
`v3.20261013.0`, and the completed smoke-test implementation and remediations.
The existing implementation plan described that completed work; Git retains it.
This plan replaces it without reopening the smoke-test design.
ConfigKit `v0.8.0` resolves to commit
`137769f3f18a370c68444242fb5d88e620e307e5` (the annotated tag object is distinct).
Its released public contracts in `docs/config.md`, release notes, and ADRs
0011–0014 supply the required placement and shared version-check behavior.
The adjacent checkout is a reference, not a committed build dependency.
Before each stage, inspect status, preserve unrelated work, read the owning
implementation/tests, and follow [development orientation](../development.md).
Apply the [agent](../policy/agents.md), [architecture](../policy/architecture.md),
[testing](../policy/testing.md), [documentation](../policy/documentation.md),
[OSS](../policy/oss.md), and relevant [release](../policy/release.md) policies.
Use the repository's code-discovery instructions. Reuse unchanged readings.
Update affected GoDoc and canonical contracts as each behavior lands; Stage 17
is a consistency pass, not permission to leave current documentation stale.
Record completion and validation beneath the stage, then commit each completed,
nonempty unit with a short plain-English message. These stages change PromptKit;
external collection publishing is separate work. Do not push, tag, release, or
introduce committed workspace/replacement files. The sprint requires neither
publication of a new catalog nor a live provider.
Dependency acquisition is setup, separate from offline behavioral validation.
Maintain [ADR 0001](../adr/0001-configuration-boundary.md): ConfigKit owns source
processing, composition, inheritance, and header checking through its configured
version callback. PromptKit interprets final effective values and their resource
contexts. Do not inspect contributions or source history to enforce domain
validity, including in tests. Explicit operator inspection remains available.
Preserve execution, retry/repair, credentials, appended messages, capacity,
observations, cancellation, and prepared-handle semantics except for the accepted
identity/version changes. Introduce no installer, implicit discovery, new
provider protocol, cross-collection inheritance resolver, compatibility alias
bundle, live reload, or additional release gate.
## Delivery Decisions
These names and signatures settle delivery details left by the feature roadmap.
Private helper names below are implementation guidance, not public invariants.
### Final Configuration And Runtime Identity
Use a typed root document with optional `version`, root `backends`, `profiles`,
and `prompts`, and `collections`. Each collection body has optional
`release_version` and the same three resource mappings. Root and collection
resource maps are keyed by local ID; `prompts` maps directly to definitions,
not another version map. Keep ordinary strict ConfigKit decoding and existing
definition rules. Collection bodies are mappings; an empty body is useful.
Represent source-root `version`, collection `release_version`, and prompt
`version` with `*string` in the typed document. Configure
`Definition.Version.Path = configkit.Keys("version")` and
`SourceVersionModeIfPresent`. The checker accepts absence, accepts exact text
`1` from a string or strict decimal integer, and rejects explicit null or other
texts with a safe ConfigKit diagnostic. ConfigKit owns type checking and
non-main header consumption. Do not synthesize a main input or declaration.
Collection release and prompt labels are opaque strings: nil means absent/null,
and a non-nil empty string remains explicitly declared. Do not trim, order,
interpret as SemVer, or require these labels.
Keep the retained ConfigKit result in its authored effective shape. Refactor
configuration helpers to receive the effective resource base path rather than
reconstructing it from a runtime ID. Visit root and collection mappings with
their actual paths; after handoff, index each resource under its full public ID.
Do not flatten/re-key the ConfigKit result or make a second load. Presence,
normalization records, content-file origins, secret classification, explanations,
and diffs must keep the effective paths, including `collections/<namespace>`.
Create shared private identity helpers under `internal/domain`. Follow the
roadmap's exact grammar: a nonblank unqualified ID has no slash; a qualified
reference has exactly one slash and two nonempty components. A configured local
key is nonblank and slash-free. A collection namespace is nonblank and
slash-free, using the same syntax as a nonempty scope. Preserve spelling,
including otherwise permitted whitespace, dots, case, and punctuation. Add no
other character whitelist. Never apply this grammar to models, paths, URLs,
artifact/input names, session IDs, or descriptive labels.
Validate final configured resource keys and effective PromptKit references.
Retain existing existence/readiness timing: for example, profile backend
existence is checked as today, but an unselected backend diagnostic default is
not made eagerly resolvable. ConfigKit interprets `extends` as a sibling key,
not a PromptKit qualified reference. PromptKit does not validate or reinterpret
consumed inheritance directives.
### Prompt Selection And Optional Labels
Remove `PromptVersion` from public/internal `RunRequest`,
`RequirementInspectionRequest`, and `SmokeTestRequest`. `ValidateRequest`,
`Prepare`, and `PrepareExecution` continue to use `RunRequest`.
Change `InspectPrompt` to:
```go
func (e *Engine) InspectPrompt(ctx context.Context, promptID string) (*PromptInspection, error)
```
All prompt repository, compiled-template, and requirement-snapshot lookups use
ID alone. Remove ambiguity tracking, version selectors, version ordering, and
request assertions. Keep `PromptIdentity.ID`; change `PromptIdentity.Version`
to `*string`. Keep output field names `PromptVersion`, changing their types to
`*string` wherever they describe the selected label: prepared details, run
results, prompt/requirement inspection, execution events, and smoke-test targets.
The internal definition uses `Version *string` with equivalent semantics.
Each returned mutable pointer is owned by its result, including events and
prepared snapshots. JSON `omitempty` omits nil while preserving a declared
empty string. Preserve other existing serialization choices.
The optional label remains part of captured definition metadata and its existing
definition-hash projection. Copy the pointer; do not dereference it as a cache
or selection key. No cross-release hash-stability promise is introduced.
Derive automatic structured-output schema names from the selected prompt ID
using the existing sanitization, not from a version-selection token. Keep explicit
schema selection and structural validation behavior unchanged.
### Immutable Scope On The Existing Engine
```go
type Namespaces struct {
Prompt string
Profile string
Backend string
}
func (e *Engine) WithNamespaces(namespaces Namespaces) (*Engine, error)
```
Return another engine facade over the same assembled engine components with
immutable scope values. An internal engine view may be a shallow copy containing
the same runner, smoke tester, inspection, and inventory collaborators. It does
not invoke `NewEngine` or build another capacity manager. `WithNamespaces`
replaces all three scopes on the new view; empty fields clear those scopes.
The receiver is unchanged. Validate syntax, not namespace existence. Use the
existing invalid-configuration convention for an invalid engine or scope.
At the internal engine entry boundary, use shared qualification helpers for
caller-supplied IDs. Validate supplied syntax; leave omitted optional IDs empty;
qualify slashless nonempty IDs for the applicable scope; pass qualified IDs
unchanged. The runner continues exact lookup. Do not requalify configured default
profiles, backend references, sibling inheritance, or diagnostic defaults.
Apply the same policy to `Run`, `Prepare`, `PrepareExecution`, `ValidateRequest`,
`InspectPrompt`, `InspectProfile`, `InspectRequirements`, and all three selection
fields of `SmokeTest`. `ValidateRequest` already delegates to preparation; use
that route rather than adding another validator. `RunPrepared` consumes its
captured selection without qualification. Preserve its existing runner-ownership
check; scope does not introduce another handle owner. `InspectConfiguration`
continues to expose the complete retained configuration, not a filtered/re-keyed
view. Results and observer metadata carry full selected identities.
### Listing Filters And Inventory
Use one concrete optional filter argument for profile, prompt, and new backend
listings:
```go
type NamespaceFilterMode uint8
const (
NamespaceDefault NamespaceFilterMode = iota
NamespaceNamed
NamespaceUnqualified
NamespaceAll
)
type NamespaceFilter struct {
Mode NamespaceFilterMode
Name string // Used only by NamespaceNamed.
}
func (e *Engine) ListProfiles(ctx context.Context, filter ...NamespaceFilter) ([]string, error)
func (e *Engine) ListPrompts(ctx context.Context, filter ...NamespaceFilter) ([]PromptIdentity, error)
func (e *Engine) ListBackends(ctx context.Context, filter ...NamespaceFilter) ([]string, error)
```
Zero or one filter is accepted; multiple filters or an unknown mode return the
ordinary invalid-request error rather than guessing precedence. Omission and a
zero filter mean default: use that resource's scope, or all resources when it is
unset. Named requires a nonempty valid namespace. Unqualified selects IDs without
a slash; All includes every ID. A non-Named mode ignores `Name`. An explicit
filter replaces, not intersects, the view's default for that call. Keep IDs
full, sort lexically by full ID, and retain existing context, ownership, and
empty-list behavior. Copy optional labels on returned prompt entries.
Qualified results can be reused through any view. To select an unqualified ID
returned by an explicit All/Unqualified filter, use the original engine or a view
without that resource scope; a listing filter never changes execution scope.
Add the following small discovery surfaces:
```go
type ResourceKind string
const (
ResourcePrompt ResourceKind = "prompt"
ResourceProfile ResourceKind = "profile"
ResourceBackend ResourceKind = "backend"
)
func (e *Engine) ListNamespaces(ctx context.Context, kinds ...ResourceKind) ([]string, error)
type CollectionInfo struct {
Namespace string
ReleaseVersion *string
}
func (e *Engine) ListCollections(ctx context.Context) ([]CollectionInfo, error)
```
No kinds means the union of all three resource types. Supplied kinds select their
union; repeats are harmless and unknown kinds are invalid requests. Namespace
enumeration describes resource membership across the captured engine, independent
of view scopes. Return distinct named namespaces sorted lexically; exclude the
unqualified group. Collection inventory likewise covers the entire captured
`collections` mapping, sorted by namespace, including metadata-only and
resource-only entries. It is not filtered by a view and does not include an
invented root collection. Both surfaces use the existing listing-style nil-engine
and context error conventions and return detached values. They do not inspect
historical sources, reopen files, or assert package authenticity.
Capture sorted inventories once at construction. Namespace buckets or sorted
ranges may avoid repeated full scans for filtered listings. Prefer a single
resource traversal plus sorting and ordinary indexed lookups; no repeated
ConfigKit queries, definition deep copies for identity-only listings, global
caches, or performance/complexity promises in the public API are needed.
### Collection Input Convenience
```go
func CollectionInput(namespace string, source configkit.Source) (configkit.Input, error)
```
Validate the required namespace syntax and return an input with that source,
`Destination: configkit.Keys("collections", namespace)`, and `Imports: true`.
Invalid namespace syntax returns an error matching `ErrInvalidConfig`.
Do not open/read the source or infer a namespace/filename. ConfigKit still owns
source validity and loading errors. The caller may change the returned input's
imports, filesystem context, or destination, or construct an input directly.
This supports host, embedded, and mapping sources with one helper, without
introducing another loader or options framework. Directory/Filesystem helpers
retain their existing explicit discovery and imports-disabled semantics.
`ProfileSource` and `BackendSource` retain their root-document shapes and can be
mounted with this helper. Direct local-key fragments use explicit ConfigKit
destinations that include the resource category.
### Migration Choices
Do not mass-copy the provider inventories into PromptKit or create a mandatory
replacement bundle. Provide synthetic, complete collection examples and guidance
for adapting selected existing catalog configuration. Adjacent repositories and
their publication are outside this sprint. The reviewed pinned catalog assets
have slash-free profile keys, so these explicit mappings are collision-free
within those assets:
| Previous identity | Documented migration identity |
| --- | --- |
| Backend `openrouter` | `openrouter/default` |
| Backend `rakestrawhome` | `rakestrawhome/default` |
| Each OpenRouter catalog profile `p` | `openrouter/p`, retaining the exact local key `p` |
| Profile `rakestrawhome-gemma-4-31b` | `rakestrawhome/rakestrawhome-gemma-4-31b` |
| Built-in `promptkit-openrouter-standard` | No automatic replacement; optional caller profile `openrouter/standard` |
| Built-in `promptkit-openrouter-auto` | No automatic replacement; optional caller profile `openrouter/auto` |
These are documented conversions, not installed aliases. Simply mounting old
`/v3` assets does not update their stored backend/profile references; those
references and diagnostic defaults need the explicit edits shown here.
For an optional caller standard profile, `extends: gemini-flash-latest` is valid
only within the same `openrouter` profiles mapping containing that base. The
caller auto profile may explicitly set backend `openrouter/default` and model
`openrouter/auto`. Neither name becomes a PromptKit promise. Move backend
diagnostic defaults to exact new profile IDs. Preserve endpoint/settings and
provider model strings, including the pinned Gemini model's leading `~`.
Application-specific IDs require a deliberate collision review, not automatic
slash-to-dot replacement. For a formerly multi-version prompt, its maintainer
chooses the installed definition; examples use the single definition already
present. Put its old version string into optional metadata where useful. Do not
select a winner from a consumer's multi-version history by guessed ordering.
Document exact references, paths, backend capacity identity changes, source-root
header reservations, label null/empty semantics, and the loss of cross-mapping
inheritance. Migration documentation is not a runtime migration tool.
## Stage 1: Adopt The Released ConfigKit Prerequisite
Scope: dependency selection and existing integration compatibility only.
1. Select ConfigKit `v0.8.0` with normal Go module tooling; update `go.mod` and
`go.sum` without filesystem replacements. Verify the selected release commit
and notices; update its attribution entry with the full commit above.
2. Check affected keyed literals and source-inspection handling for upstream
additions. `SourceInfo` is no longer comparable; compare relevant fields if
PromptKit currently relies on comparability. Do not enable new schema behavior.
3. Leave Go's consumer minimum at `1.26.0` unless the selected dependency actually
requires a higher minimum. Leave unrelated dependencies/tooling unchanged.
Validation: `GOWORK=off go test ./internal/configuration .` and the three offline
examples, including configuration inspection/diff coverage. This stage establishes
the unchanged PromptKit baseline on the new dependency.
Completion: ConfigKit `v0.8.0` is selected at commit
`137769f3f18a370c68444242fb5d88e620e307e5`; its attribution links and module
checksums are updated. The Go consumer minimum remains `1.26.0`. No
`SourceInfo` comparability-dependent code required adjustment. Validation passed:
`GOWORK=off go test ./internal/configuration .` and
`GOWORK=off python3 scripts/check-examples.py`.
## Stage 2: Add Shared Resource Naming Primitives
Scope: small private domain helpers and focused behavior tests.
1. Implement exact ID parsing/validation, namespace syntax validation, local-key
validation, qualification, and namespace extraction as needed by later stages.
Keep omission policy in operation callers, not a universal required-ID rule.
2. Cover unqualified/dotted names, qualified bypass, missing/empty scope,
malformed slash forms, and preserved spelling. Check that namespace matching
is exact rather than a prefix match.
3. Do not wire names into all public operations yet or export a splitting API.
Validation: `GOWORK=off go test ./internal/domain`. Test selected naming behavior,
not a helper call sequence or invented character restrictions.
Completion: private resource-ID, namespace, local-key, qualification, and
namespace-extraction helpers are implemented with focused grammar and spelling
tests. Validation passed: `GOWORK=off go test ./internal/domain` and
`git diff --check`.
## Stage 3: Make Configuration Processing Path-Aware
Scope: preparation refactor inside `internal/configuration`; current format stays.
1. Refactor `captureFields`, `resolveBackends`, `resolveProfiles`,
`validateEffective`, prompt metadata capture, and prompt materialization to
accept/use explicit effective base paths. Preserve current values and errors.
2. Carry a resource's public ID separately from its ConfigKit path. Stop deriving
an effective path by splitting a runtime ID. Use ConfigKit path segments for
literal keys and preserve inherited `content_file` origins.
3. Factor resource-map traversal only as needed to reuse the same processing
for root and collection mappings later. Do not add a generic visitor framework.
4. Prepare the existing provider-secret predicate to recognize actual resource
paths without mistaking arbitrary nested `extra_params` data for resources.
Keep its existing protected-key policy.
Validation: configuration load, target, prompt-origin, inheritance, and inspection
tests; `GOWORK=off go test ./internal/configuration .`. A behavior-preserving
refactor should not rewrite unrelated expectations.
Completion: configuration capture, normalization, effective validation, and
prompt materialization now receive explicit ConfigKit base paths. Existing
resource IDs remain independent from those paths; source-origin reads and the
protected provider-key policy are preserved. Validation passed:
`GOWORK=off go test ./internal/configuration .`.
## Stage 4: Install One Prompt Definition Per ID
Scope: prompt storage format, materialization, and fixture-format migration.
1. Change root `prompts` to `map[string]promptConfig`; add nullable
`Version *string` to the raw prompt definition. Materialize one entry per ID,
preserving the label text without normalization or a generated label.
2. Remove nested version indexes, ambiguity tracking, and selection-by-version
from the configuration adapter. Use the actual prompt path without the old
version segment for metadata and message resources.
3. To keep this stage bounded, existing repository method parameters and public
selector fields may remain only until Stage 5. At this adapter boundary their
temporary implementation delegates to ID-only lookup; do not compare them
with labels or create an assertion mode. Existing string-valued output labels
are converted to presence-aware values in Stage 6. These are migration seams,
not new compatibility behavior to retain or publish.
4. Mechanically migrate YAML fixtures, mapping builders, and example configuration
files to the direct definition shape. Move retained version keys to quoted
metadata. Replace tests of parallel versions/ambiguity with one-entry lookup
and ordinary mapping-overlay cases. Keep unrelated validation coverage.
Validation: `GOWORK=off go test ./...`, including consumers of migrated fixtures.
Test ID-only selection, source-relative content, and inherited/overlaid prompt
settings. Do not add tests preserving temporary obsolete selector parameters.
Completion: prompts now map directly from each ID to one definition, with an
optional descriptive version label. Configuration lookup and message/input
metadata paths are ID-only; ambiguity handling and version-keyed indexes are
removed. Fixtures, examples, and the canonical format reference use the direct
shape, including layered overlays. Validation passed: `GOWORK=off go test ./...`,
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
## Stage 5: Remove Request-Side Version Selection
Scope: compile-directed removal of obsolete selector plumbing; no new lookup rules.
1. Remove request `PromptVersion` fields and the version parameter from public
`InspectPrompt` and internal prompt repository/snapshot/template interfaces.
Update runner preparation, inspection, requirements, and smoke selection.
2. Update public-to-internal conversion, redacted formatting, fakes, tests,
examples, and call sites. Remove version-without-prompt checks and the
transitional signatures from Stage 4. Preserve ordinary not-found identities.
3. Use the selected prompt ID for generated structured-schema names; retain the
existing sanitization and explicit schema behavior. Do not remove output
metadata fields merely because selectors are gone.
4. Update affected GoDoc now so obsolete selectors are no longer advertised.
Validation: `GOWORK=off go test ./...` and focused vet of modified packages.
Search declarations/call sites for remaining selection/ambiguity logic; descriptive
label references are valid and should not be blanket-deleted.
Completion: request-side version selectors and version parameters have been
removed from prompt lookup, inspection, requirement analysis, and smoke-test
selection. Generated structured-schema names now derive from the prompt ID;
result-side descriptive label fields remain. Updated the consumer guide and
GoDoc to reflect ID-only selection. Validation passed: `GOWORK=off go test
./...`, focused `GOWORK=off go vet` for modified packages,
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
## Stage 6: Carry Optional Prompt Labels Through Results
Scope: presence-aware metadata across internal values and the facade.
1. Change internal definition `Version` and all selected-label output fields
identified above to `*string`. Read the raw optional field captured by Stage 4.
Update conversion and ownership copies, hashing projection, requirement
snapshots, observation selection/events, prepared details/results, and smoke
target projection. Optional labels never become lookup keys.
2. Distinguish nil from a declared empty string in inspection/listings and JSON.
Both omitted and null mean no label; a higher-layer or inherited null clears
the label through ordinary ConfigKit composition.
3. Update the example summary types and test expectations. Add focused boundary
coverage for absence, null clearing, empty/exact text, and pointer ownership;
reuse existing report/prepared/event detachment tests where practical.
Validation: domain, promptdef, configuration, observation, usecase, and root tests;
`GOWORK=off go test ./...` and the offline examples. Verify the existing selected
definition hash still agrees between inspection and preparation.
Completion: prompt labels are presence-aware from configuration capture through
inspection, listing, requirements, prepared and completed runs, observation
events, and smoke targets. Nil and declared-empty labels remain distinct in JSON;
null overlays clear inherited labels. Returned label pointers are independently
owned, and inspection/preparation retain matching definition hashes. Validation
passed: `GOWORK=off go test ./...`, `GOWORK=off go vet ./...`,
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
## Stage 7: Enable Shared Specification Checking
Scope: root declaration policy through ConfigKit's released version API.
1. Add optional root `version` to the typed document. Configure the version rule
and `SourceVersionModeIfPresent` with the exact checker specified above.
Keep no-main and headerless sources valid.
2. Use ConfigKit's header consumption and source evidence without a source
pre-parser, validation loop, or PromptKit mode option. Nested prompt labels
remain ordinary metadata.
3. Cover accepted string/integer `1`, omitted main/header, rejected null or
unsupported declarations, consumed non-main headers, and an explicit main
declaration retained in inspection. A headerless imported prompt fragment
still undergoes ordinary final-value validation.
Validation: focused real-ConfigKit tests in `internal/configuration` and public
construction/inspection tests. Do not replicate upstream's full version-type or
import matrix. Run `GOWORK=off go test ./internal/configuration .`.
Completion: the optional root declaration is checked through ConfigKit's shared
source-version API, accepting only exact string or integer `1` while allowing
absence and preserving the main declaration in inspection. Accepted underlay
declarations are consumed; null and unsupported values produce safe diagnostics,
and headerless payloads still receive final-value validation. Validation passed:
`GOWORK=off go test ./internal/configuration .`.
## Stage 8: Assemble And Index Final Collection Resources
Scope: namespaced schema, final resource interpretation, and runtime indexes.
1. Add `collections` and collection `release_version`/resource mappings to the
typed document. Visit root and collection maps using Stage 3's effective
paths. Validate namespaces/local keys; build full IDs only in runtime indexes.
2. Include backend/profile snapshots and prompt metadata/templates in that
traversal. Keep the retained ConfigKit result structurally unchanged.
Preserve the distinction between root unqualified and collection-local keys.
Migrate maintained qualified-key fixtures into collection/local-key mappings,
updating exact references and inspection paths together. Use ordinary explicit
destinations until the convenience helper lands; do not keep a second root
spelling for qualified identities.
3. Validate effective references with the naming helper and existing existence
policy. Local `extends` stays with ConfigKit. Do not reject repeated namespace
contributions, require collection releases, or qualify stored references.
4. Extend secret-path matching to backend/profile extras beneath collections,
preserving the existing protected-key list. Use exact effective paths for
field presence, normalization, `content_file`, explanations, and diagnostics.
5. Preserve backend-ID capacity policy wiring. The same final backend is one
capacity identity regardless of the sources that contributed its settings.
Validation: focused real-ConfigKit tests for two collections with duplicate local
names, root resources, partial overrides, inherited relative content, explicit
zero/null settings, and redacted namespaced provider data. Include retained
release metadata, sibling inheritance, headerless collection fragments, resources
named `version` in nested mappings, and header/declaration distinctions.
Run `GOWORK=off go test ./internal/configuration .`.
Completion: final root and collection resources are interpreted from their
effective ConfigKit paths and indexed under full runtime IDs without re-keying
the retained result. Collection namespaces and local keys are validated;
collection release labels, sibling inheritance, inherited relative content,
presence/null handling, and namespaced secret redaction are covered. Capacity
and profile snapshots use full backend identities. Validation passed:
`GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, and `git diff --check`.
## Stage 9: Make Application Tests Independent Of Bundled Catalogs
Scope: owned fixtures and examples that currently assume automatic provider data.
1. Locate tests/examples using bundled backend/profile IDs, inherited catalog
settings, default smoke profiles, or baseline listing counts. Supply the
explicit synthetic definitions each behavior needs through existing sources.
2. Keep generic test helpers small and local; no full catalog snapshot or shared
test fixture framework. HTTP adapter tests may retain provider adapter names
but use controlled transports and synthetic configuration.
3. Keep catalog-specific assembly tests until Stage 10 removes the integration.
Do not preserve provider inventory assertions as application contracts.
Validation: the affected root, engine, and usecase tests plus examples. Ordinary
coverage should now express its own backend, profile, and prompt requirements.
Completion: ordinary application tests now provide the backend, profile, and
smoke metadata they exercise instead of relying on bundled provider entries.
The listing and inheritance contracts use synthetic resources; catalog-specific
assembly tests remain isolated for the later integration removal. Maintained
examples already supplied their own definitions and needed no changes.
Validation passed: `GOWORK=off go test ./...`,
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
## Stage 10: Remove Automatic Catalogs And Built-In Profiles
Scope: engine composition, dependency/notices cleanup, and the retired promise.
1. Remove `catalog.Underlays()` from `internal/engine.New`; load only caller
underlays and optional main. Preserve their order and strict-group semantics.
Empty configuration constructs an engine with empty resource inventories.
2. Delete `internal/catalog`, its owned profiles, and integration-specific tests
after checking references. Do not delete generic file-discovery helpers.
3. Remove both catalog module requirements and unused checksums through module
tooling. Do not update/publish those repositories or copy their inventories
into PromptKit as a replacement.
4. Update attribution for actual retained material; remove catalog dependency
entries when nothing incorporated remains. Keep a notice if other copied
material still requires it. Do not strip a shared license text still in use.
5. Retire the release policy's catalog-update exception and literal built-in-name
promise. Apply ordinary compatibility rules to PromptKit format/API changes;
independently selected configuration is no longer a bundled release inventory.
Update architecture/internal inventory and current format documentation for
the removed component. Preserve historical release notes.
Validation: `GOWORK=off go test ./...`; verify empty inventories, explicit former
backend names still work when configured, and inference smoke tests retain their
selection/capability behavior with caller-supplied metadata. Review `go.mod`,
`go.sum`, imports, and notices for residual automatic catalog coupling.
Completion: engine construction now composes only caller-selected underlays and
the optional main input; an empty configuration has empty profile and prompt
inventories. Removed the catalog adapter and its tests, provider catalog module
requirements and checksums, catalog attribution entries, and the built-in
profile compatibility exception. Current architecture, source, format, and
consumer documentation now describe caller-supplied definitions; historical
release notes remain unchanged. Explicit OpenRouter and Rakestrawhome backend
IDs continue to work when configured. Validation passed: `GOWORK=off go test
./...`, `GOWORK=off go vet ./...`, `GOWORK=off python3
scripts/check-examples.py`, and `git diff --check`.
## Stage 11: Add The Explicit Collection Input Helper
Scope: the single public convenience function and its GoDoc.
1. Implement `CollectionInput` exactly as specified above, reusing namespace
validation and existing facade error conventions. It creates an input, not
an engine and not a configuration pre-parser.
2. Keep raw ConfigKit inputs and existing directory/filesystem/programmatic
helpers usable. Do not reinterpret their IDs or introduce root discovery.
3. Add one native-file and one embedded/mapping assembled case with entry
imports and a later deliberate override. Verify resource contexts survive.
Validation: source-helper contract tests and focused configuration/root tests.
Verify helper construction itself performs no source read using an existing
controlled source seam where needed, rather than adding a production interface.
Completion: added `CollectionInput` with shared namespace validation, a
`collections/<namespace>` destination, and imports enabled without reading the
source. Contract tests cover invalid namespaces and deferred source loading;
native-file and embedded-filesystem imports assemble successfully, preserve
their source context, and accept a later override. Validation passed:
`GOWORK=off go test ./...`, `GOWORK=off go vet .`, and `git diff --check`.
## Stage 12: Add Immutable Scopes To Ordinary Execution
Scope: public `Namespaces`/`WithNamespaces` and shared engine-boundary routing.
1. Add the public facade and internal immutable view. Replace all scopes on each
new view without mutating its receiver. Reuse assembled collaborators.
2. Route caller prompt/profile selections in `Run`, `Prepare`, and
`PrepareExecution` through shared qualification. `ValidateRequest` follows
its current preparation route. Stored defaults and omitted IDs stay untouched.
3. Preserve resolved full IDs through prepared details, results, events, errors,
and selected backend identity. Leave `RunPrepared` captured and single-claim;
do not add a scope mismatch check or rebuild shared state.
4. Add public behavior tests for two independent prompt/profile scopes,
qualified bypass, no fallback, absent well-formed namespaces, and invalid
scope syntax. Assert the receiver retains its original behavior.
Validation: root/prepared/usecase tests and focused race-enabled scope tests.
Use injected generation or a controlled HTTP boundary; never a live endpoint.
Completion: added immutable `WithNamespaces` views that share the assembled
runner and captured configuration. Ordinary `Run`, `Prepare`,
`PrepareExecution`, and the preparation-backed `ValidateRequest` now qualify
slashless caller prompt/profile IDs while preserving qualified IDs, captured
defaults, full result/event identities, and prepared-handle ownership behavior.
Tests cover independent views, replacement and clearing, no fallback, absent
well-formed namespaces, invalid syntax, and concurrent preparation. Validation
passed: `GOWORK=off go test ./...`, focused `GOWORK=off go test -race . -run
'TestWithNamespaces' -count=1`, `GOWORK=off go vet ./...`, and `git diff --check`.
## Stage 13: Apply Scopes To Inspection And Diagnostics
Scope: complete the existing operation inventory without a second resolver.
1. Route `InspectPrompt`, `InspectProfile`, and `InspectRequirements` through the
same helpers; resolve each request's explicitly supplied prompt/profile fields
independently. Keep configured references exact.
2. Route `SmokeTest`'s supplied PromptID, ProfileID, and BackendID by their own
scopes before ordinary diagnostic selection. Keep its precedence, optional
omissions, backend defaults, factual reports, and errors unchanged.
3. Keep `InspectConfiguration`/Diff global with effective ConfigKit paths.
Do not qualify schema paths, provider models, input identifiers, or overrides.
4. Review all public resource-selection entry points for omissions. Validate
malformed supplied reference syntax through the existing invalid-argument
boundary; retain ordinary not-found behavior for well-formed missing IDs.
Validation: representative public inspection/requirements/validation/smoke cases
prove shared behavior and full identities. Reuse naming-helper tests instead of
duplicating the full syntax matrix in every API test. Run relevant root/engine
tests and focused race checks.
Completion: prompt, profile, and requirement inspections now qualify each
explicit caller selection independently, and smoke diagnostics qualify supplied
prompt, profile, and backend IDs before applying their unchanged selection
precedence. Captured references and global configuration inspection remain
unchanged. Contract tests verify full returned identities, exact configured
prompt defaults and backend references, malformed caller IDs, and ordinary
not-found behavior.
Validation passed: `GOWORK=off go test ./...`, focused
`GOWORK=off go test -race . -run 'TestWithNamespaces' -count=1`,
`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`,
and `git diff --check`.
## Stage 14: Add Backend Listing And Namespace Filters
Scope: the existing identity-inventory path, not the execution runner.
1. Add the filter declarations and optional arguments to profile/prompt listing;
add backend identity listing through configuration, internal engine, and facade.
Existing no-filter engine calls still enumerate everything.
2. Resolve default/explicit filters once per listing using the relevant view
scope. Validate named filters and modes; filter exact namespace membership.
Use captured sorted inventories, optionally indexed by namespace.
3. Return full IDs and detached optional prompt labels, with existing cancellation
and empty-result behavior. Do not materialize definitions for identity listing.
Validation: extend listing tests with scoped defaults, explicit named/All/
Unqualified overrides, no-scope defaults, absent namespaces, same-prefix namespace
names, full-ID round trips (and unqualified selection through the original
engine), invalid filters, ownership, and canceled contexts. Use a compact matrix
across resource types, not three copies of all tests. Run configuration/root tests.
Completion: prompt and profile listings now accept the documented optional
namespace filters, and backend identities are captured and listed through the
same path. Default filters use the view's per-resource scope, explicit filters
replace it, results retain full IDs and detached prompt labels, and ordinary
no-filter calls still return the full inventory. Contract tests cover three
resource types, exact namespace boundaries, round trips, invalid filters,
ownership, and cancellation. Validation passed: `GOWORK=off go test ./...`,
focused `GOWORK=off go test -race . -run
'TestScopedIdentityListingsApplyNamespaceFilters|TestListIdentitiesAreOwnedAndSafeForConcurrentUse|TestListIdentitiesPreserveContextErrorsWithoutPartialResults' -count=1`,
`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`,
and `git diff --check`.
## Stage 15: Expose Namespace And Effective Collection Inventories
Scope: the two discovery APIs and construction-time inventory capture.
1. Capture namespace sets by resource kind from final indexed IDs; implement
`ListNamespaces` with the chosen union, kind validation, and global semantics.
2. Capture `CollectionInfo` values from final `collections` declarations. Preserve
absent/null versus empty/exact release labels, including metadata-only and
resource-only collections. Copy label pointers on output.
3. Delegate through the internal engine's narrow inventory boundary and root
facade. Do not derive collection releases from source header evidence or
confuse resource namespace membership with collection declarations.
Validation: one representative inventory fixture covers cross-kind membership,
deduplication/sorting, unqualified exclusion, metadata-only collections,
scope-independent enumeration, effective overrides, cancellation, and ownership.
Run configuration/root tests; no file access or generation should be needed.
Completion: added global resource-namespace and effective collection inventory
APIs. Namespace results are distinct, sorted, kind-selectable, and exclude
unqualified resources; collection results preserve absent/null versus empty
release labels and include metadata-only declarations. Inventories remain
independent of namespace views and return caller-owned values. Validation passed:
`GOWORK=off go test ./...`, focused
`GOWORK=off go test -race . -run 'TestResourceAndCollectionInventories' -count=1`,
`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`,
and `git diff --check`.
## Stage 16: Maintain Complete Consumer And Migration Examples
Scope: runnable/synthetic examples and precise migration guidance.
1. Keep the existing prepare/run/observe examples valid with ID-only selection
and optional labels. Extend the preparation example, or its existing input
assets, to demonstrate two collections, explicit input selection, a scoped
view, filtering, and operator inventory without a provider call. Keep the
run/observe examples focused on their distinct existing workflows.
2. Add small complete collection entry/fragments under `examples/`, with optional
headers/releases, source-relative message content, exact stored references,
sibling inheritance, and an intentional later partial override. Use synthetic
endpoints/models rather than another maintained provider catalog.
3. Add `docs/consumers/configuration-migration.md` explaining the breaking format
and signatures, the explicit mapping table above, optional caller standard/auto
choices, and deployment selection responsibility. Do not promise arbitrary
consumer migrations are automatic or validate upstream provider availability.
4. Update semantic example checks only for the changed example outputs. If
script logic changes, run its focused checks; do not rewrite release tooling.
Validation: `GOWORK=off python3 scripts/check-examples.py`. Assemble the maintained
collection example through real ConfigKit and PromptKit. Keep copied material's
actual attribution obligations aligned if any is introduced.
Completion: the offline preparation example now explicitly composes `support`
and `review` collections, including an imported source-relative message file,
sibling profile inheritance, and a later partial profile override. It prepares
an explicitly selected prompt/profile through a scoped view and reports global
namespace/collection inventories plus a namespace-filtered prompt listing. The
synthetic entries demonstrate present and absent source/release metadata; the
run and observe examples remain focused on their existing workflows. Added a
consumer migration guide with the format/API changes, identity mappings,
caller-owned optional profiles, and deployment-selection responsibility.
Validation passed: `GOWORK=off go test ./...`, `GOWORK=off go vet ./...`,
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
## Stage 17: Finalize Canonical Contracts And Migration Documentation
Scope: documentation consistency after the APIs and examples exist.
1. Update GoDoc for all changed/new public declarations, errors, optional labels,
ownership, scopes, and listing behavior. Do not leave obsolete selectors or
an implication that a descriptive label controls execution.
2. Update `docs/formats.md` for the final schema, specification header policy,
source-root reservations, one installed prompt, exact references, local
inheritance, and explicit provider configuration. Keep current contracts here,
not in the migration guide or roadmap.
3. Update `docs/consumers/pkg-promptkit.md`, README quickstart/navigation,
`docs/internal/sources.md`, internal inventory, and affected runner/rendering/
observation documents. Explain unchanged resource contexts and operator-only
provenance use. Remove live claims about automatically bundled inventories.
4. Review architecture, release policy/procedure, development reading guidance,
and attribution for obsolete catalog assumptions. Revise only affected local
statements; do not rewrite upstream policy language unnecessarily or weaken
ordinary compatibility promises. Do not revise historical release notes or
immutable ADR decisions.
Validation: review changed Markdown links and example references, check GoDoc
against declarations, and run `git diff --check`. Release notes and version/tag
selection belong to the later requested release preparation, not this stage.
Completion: canonical format, consumer, README, internal, and public GoDoc
contracts now describe collection identity, source headers and origins,
namespace-scoped selection, global inventories, descriptive labels, and
caller-supplied provider configuration. Reviewed architecture, release policy
and procedure, development guidance, and attribution; no additional local
changes were needed, and historical release notes and immutable ADRs remain
unchanged. Validation passed: local Markdown link targets and example references
were reviewed, affected GoDoc was checked against declarations, and
`git diff --check`.
## Stage 18: Verify Cross-Boundary Behavior And Close The Sprint
Scope: focused integration gaps and final regression validation, not new features.
1. Trace the feature roadmap's acceptance criteria to focused existing/new tests.
Add only missing assembled-boundary coverage. One synthetic fixture can prove
independent collection names, scoped execution/inspection, full-ID filtering,
exact defaults, optional labels, and inventory without duplicating every
assertion at each internal layer.
2. Use a controlled concurrent model boundary to verify ordinary and differently
scoped calls share the same configured backend's capacity. Reuse existing
admission/cancellation tests for queue mechanics. Verify prepared selections
remain captured when another view is used; no namespace is mutable global state.
3. Confirm final-value-only validation, source-local headers versus ordinary
metadata, retained inherited/overlaid content origins, and redaction under
namespaced paths. Final inspection still describes ConfigKit's original
effective shape rather than a rewritten runtime index.
4. Remove all temporary selector signatures/adapters from the migration stages.
Review source/tests/docs for old version-key maps, version selection/assertions,
automatic catalog contributions, mandatory built-in names, and local module
replacements. Historical notes and optional metadata references may remain.
5. Run the final offline checks once for the resulting content:
```sh
GOWORK=off go test -race ./...
GOWORK=off go vet ./...
GOWORK=off python3 scripts/check-examples.py
git diff --check
```
Format changed Go files and validate affected Markdown links. Review the full
diff/status for unintended files and private data; commit the completed unit.
Mark this plan and the feature roadmap complete, leaving retirement to normal
release preparation. Do not publish, wait for CI, probe providers, or introduce
a catalog-release dependency to finish implementation.
Completion: acceptance coverage was traced across public collection assembly,
scoped execution and inspection, full-ID listings and inventories, exact stored
defaults, optional prompt labels, prepared selection capture, source headers and
origins, final-value validation, namespaced redaction, and retained effective
configuration shape. Added a controlled concurrent regression test proving an
ordinary call and a differently scoped call share one collection backend's
capacity. Audited current source, tests, and docs for obsolete selectors,
automatic catalogs, built-in profile promises, and local module replacements;
remaining version references are descriptive labels or migration guidance.
Marked the feature roadmap complete; normal release preparation owns retirement.
Validation passed: `GOWORK=off go test -race ./...`, `GOWORK=off go vet ./...`,
`GOWORK=off python3 scripts/check-examples.py`, `gofmt`, local Markdown link
checks, and `git diff --check`.
## Open Questions
None required to begin implementation. The feature roadmap's policy decisions
are settled; the delivery decisions above specify the public shapes and migration
path. A newly discovered upstream incompatibility or genuine change of scope
should be reported with a recommendation and viable alternative rather than
silently changing the accepted configuration boundary or adding restrictions.

190
docs/roadmap/remediation.md
View File

@@ -1,190 +0,0 @@
# Configuration Collections Implementation Review Findings
Status: all three findings completed 2026-10-17.
Reviewed implementation after `89af2a1` through `0221cd4` against the accepted
[feature roadmap](configuration-collections.md), [implementation plan](implementation.md),
[configuration-boundary ADR](../adr/0001-configuration-boundary.md), current
GoDoc, and applicable repository policies. This replaces the previous completed
smoke-test remediation log; its findings and completion records remain in Git.
## Architecture Assessment
The targeted architecture is substantially in place:
- ConfigKit v0.8.0 composes caller-selected inputs and applies the shared optional
specification-header policy. PromptKit interprets final effective values and
resource contexts without flattening the retained configuration or replaying
source history.
- Root resources retain unqualified identities; collection resources use full
namespace/local-key identities in captured runtime indexes. Stored references
remain exact, and inheritance stays with ConfigKit.
- Scoped engine views share the assembled runner, model clients, inspection,
inventory, and capacity machinery. Qualification is centralized at engine
entry points; prepared execution retains its captured selections.
- Prompt selection is ID-only, with optional descriptive version labels.
Namespace-filtered prompt/profile/backend listing, resource namespace discovery,
and effective collection inventory are available.
- Automatic catalog dependencies and built-in profiles are removed. Consumer
migration guidance and offline collection examples are present.
The findings below are focused ownership, construction-efficiency, and contract
documentation corrections, not reasons to redesign the feature. No additional
unrequested runtime restrictions were identified in the reviewed paths. Finding 3
addresses a retained public guarantee that no longer describes scoped views.
## Finding 1: Share Origin Metadata Across Collection Materialization
Priority: P2. Status: completed 2026-10-17.
Locations: [load.go](../../internal/configuration/load.go), lines 257–262, and
[materializePrompts](../../internal/configuration/prompts.go), lines 84–95.
The loader now calls `materializePrompts` separately for every collection. Each
call creates its own lazy metadata cache and calls `state.result.Metadata()`
when it encounters a file-backed message. In the selected ConfigKit v0.8.0,
`Result.Metadata` defensively clones the whole metadata snapshot, including
field/history maps and paths. The cache is consequently reused within one
collection but discarded before the next.
With C collections requiring origins and M total metadata entries, this adds
O(C × M) copying during construction; when resource count grows with collection
count, this portion becomes quadratic. Before the collection refactor, the same
cache covered the single prompt-materialization pass.
An offline diagnostic benchmark used 4, 16, and 64 collections, each containing
one prompt with one `content_file`, all reading a tiny in-memory file. A single
iteration per case reported:
| Collections | Allocated bytes per construction |
| --- | ---: |
| 4 | 1,662,192 |
| 16 | 10,267,520 |
| 64 | 104,352,336 |
An allocation-space profile across those diagnostic loads attributed about 79%
of allocated space to ConfigKit metadata cloning reached through
`materializePrompts`' origin callback. These are illustrative local measurements,
not stable thresholds or a claim about all constructor workloads.
Required outcome: reuse one construction-wide lazy metadata snapshot or an
equivalent narrow lookup across root and collection prompt materialization.
Preserve exact effective paths, inherited origins, resource contexts, and
handle closure. Keep the change local and construction-scoped; no new source
validation, global cache, or upstream API extension is needed.
Validation: retain origin/inheritance/content-capture coverage with more than
one file-backed collection, and use a small scaling benchmark to verify the
repeated full-snapshot cost is removed. Do not make exact allocations, elapsed
times, or private helper call order a new contract.
Completion: prompt materialization now traverses root and collection resources
within one construction-scoped pass sharing the lazy origin metadata snapshot.
Extended the imported/inherited-content test across distinct collection resource
contexts and added a scaling benchmark without allocation or timing assertions.
At 64 collections the same benchmark dropped from 104,574,200 to 21,239,712
allocated bytes per construction in single-iteration local measurements.
Validation passed: focused origin tests and benchmark, race-enabled configuration
and public API tests, configuration vet, and `git diff --check`.
## Finding 2: Detach The Optional Label In Internal Prepared Snapshots
Priority: P3. Status: completed 2026-10-17.
Location: [clonePreparedRun](../../internal/usecase/prepared_execution.go),
lines 226–244; also its use by `PrepareExecution` and `PreparedExecution.Details`.
`domain.PreparedRun.PromptVersion` changed from a string to `*string`, but
`clonePreparedRun` still performs only the initial struct assignment for that
field. Its returned snapshot shares the label pointer with its source.
`PrepareExecution` uses this copier to produce retained details from the
execution snapshot, and `Details` uses it again for each returned copy.
A temporary package-level probe constructed a prepared handle with label
`original`, obtained its internal `Details`, changed that result's label to
`mutated`, then obtained `Details` again. The second result incorrectly contained
`mutated`. This contradicts the internal method's fresh-copy contract and the
planned ownership of optional metadata. The same copier leaves retained details
and execution metadata sharing this pointer.
The public facade's `fromDomainPreparedRun` currently copies the pointer again,
so public callers remain protected by that additional boundary. This is an
internal ownership gap, not a reproduced public mutation or credential leak.
Required outcome: copy the optional label with the existing nil-aware string
copy helper when cloning prepared runs. Retain absent versus declared-empty
semantics and the existing public ownership protection; a cloning-framework
refactor is unnecessary.
Validation: extend the existing owning use-case prepared-details test with a
non-nil version label. Mutation of one returned internal snapshot must not change
retained details, the execution snapshot, or a later result. Reuse existing
nil/empty public metadata coverage rather than reproducing it at every layer.
Completion: the prepared-run copier now detaches the optional version label with
the existing nil-aware helper. Extended the owning prepared-execution snapshot
test to mutate source and returned labels and verify later details, execution
metadata, and post-execution details remain unchanged. The regression failed
before the fix and passes afterward. Validation passed with `GOWORK=off`:
the focused regression, `go test -race ./...`, `go vet ./...`, all three offline
examples, and `git diff --check`.
## Finding 3: Document Engine Ownership In Terms Of The Shared Assembly
Priority: P3. Status: completed 2026-10-17.
Location: [Engine GoDoc](../../engine.go), lines 85–101; related
[prepared-handle GoDoc](../../prepared_execution.go) and
[WithNamespaces GoDoc](../../namespaces.go).
The Engine declaration still says that each Engine owns independent
backend-capacity pools. `WithNamespaces` now returns another `*Engine` sharing
the same runner and capacity manager, as expressly required by the feature.
The new `TestScopedCollectionRunsShareBackendCapacity` demonstrates this sharing;
the runtime is correct, but the public declaration promises the opposite for
these returned Engine values.
Prepared handles are likewise described as bound to the Engine that created
them, without clarifying that namespace views share that underlying owner.
The current public tests intentionally run a scoped handle through the original
engine. Documentation should make that distinction clear rather than imply
per-facade capacity or handle ownership.
Required outcome: distinguish independently constructed engines from namespace
views of one assembled engine. State that views share clients/capacity and the
prepared-execution owner, with selections already frozen in prepared handles.
Preserve independent ownership between separate `NewEngine` constructions.
Do not change the implementation to satisfy the stale wording or introduce
scope-mismatch rejection.
Validation: review the affected GoDoc together. Reuse the passing shared-capacity
and cross-view prepared-handle tests; no new behavior or test matrix is needed.
Completion: Engine, WithNamespaces, PrepareExecution, RunPrepared, and the
prepared-handle GoDoc now distinguish independent constructions from shared
namespace views. They describe shared clients/capacity and handle ownership,
with captured selections unchanged by the invoking view. No runtime behavior
was changed. Reviewed rendered `go doc` output; the existing shared-capacity and
cross-view prepared-handle tests and `git diff --check` passed. The full
race-enabled suite, vet, and offline examples from Finding 2 cover the unchanged
implementation.
## Review Validation
Passed on the reviewed committed tree with `GOWORK=off`:
- `go test -race ./...`;
- `go vet ./...`;
- `python3 scripts/check-examples.py`.
Source review covered the configuration handoff, header policy, paths/origins
and sensitivity, root/collection indexing, optional metadata copying, all scoped
selection paths, listing/inventory behavior, shared capacity, dependency removal,
and migration contracts. Representative existing tests were inspected as well as
executed; the green suite does not cover the internal label-aliasing probe.
The package-level ownership probe failed as described in Finding 2. The
construction benchmark and allocation profile support Finding 1. Both probes
used temporary Go test overlays outside the repository; neither changes the
implementation or adds permanent tests. No live provider or real credential
was used. Only this remediation document is changed by the review.

592
docs/roadmap/smoke-tests.md
View File

@@ -1,592 +0,0 @@
# Explicit Smoke Test Diagnostics
Status: accepted target state. Implementation has not started.
This roadmap replaces the earlier capability proposal and incorporates the
subsequent discussion. It defines the target feature, not a staged implementation
plan or current public API. The companion [implementation plan](implementation.md)
owns the ordered work and delivery mechanics.
## Purpose And Ownership
Provide reusable, explicitly invoked diagnostics for endpoint reachability,
authentication, plain-text inference, and structured-output inference. Consumers
can build setup and diagnostic workflows without maintaining their own synthetic
prompts, schemas, and provider probes.
PromptKit is opinionated about what each test checks and how it operates. It owns
fixed content, provider adapters, evaluation rules, and neutral factual reports.
Consumers select tests, targets, and supported execution settings; they interpret
the results and decide what action to take. There is no overall success,
application-readiness verdict, or configurable criteria framework.
For structured output, report whether structured output was requested, a response
was received, generated content was valid JSON, and that JSON matched the fixed
test schema. Four positive observations do not prove provider enforcement; no
field attempts to infer it.
For authentication, distinguish credentials supplied with the dispatched request,
a reachable authentication endpoint, and credentials accepted by that operation.
A reachable endpoint can reject credentials, and a transport failure can leave
acceptance undetermined. Authentication is inapplicable on the maintainer's
credential-free RakestrawHome deployment; that is neither an error nor a failure.
## Single-Sprint Delivery Objective
The staged implementation must be completable from start to finish in one sprint,
including built-in defaults, tests, and documentation. It must not require
intermediate human-review approvals, upstream releases, external publication, or
live-provider access to progress or establish completion. Automated checks and
focused local commits remain part of ordinary implementation.
Built-in diagnostic metadata belongs in the local OpenRouter and RakestrawHome
catalogs, with their validated revisions identified by local candidate tags.
There is no intermediate PromptKit-owned metadata underlay or requirement for
separate loader/catalog publication before integration.
A temporary, uncommitted Go workspace may support coordinated development.
The completed integration uses normal module requirements and Go-generated
checksums and is validated with `GOWORK=off`, without filesystem replacements.
Exact local tag contents can be supplied through a temporary filesystem module
proxy, without public HTTPS availability or upstream publication. Local tags
alone do not make Go discover adjacent checkouts.
Keep workspace/proxy artifacts outside committed repository contents, following
the [repository-independence policy](../policy/architecture.md#repository-independence).
PromptKit's completed sprint commits may reference those not-yet-published
catalog versions in `go.mod` and their corresponding Go-generated checksums in
`go.sum`. Implementation completion and local validation do not claim that other
machines can fetch them. Public distribution is separate: consumers eventually
need those exact contents available through their configured module sources.
Normal release preparation and publication follow implementation separately;
this objective does not bypass the repository's release process or authorize
publication during the sprint.
## Current Baseline
At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
checks a concrete application request without model generation. `Run` executes
that request and may structurally validate and repair its output. These operations
remain unchanged; smoke tests need not validate an intended application request.
The [runner](../internal/runner.md) shares captured prompt/profile selection and
execution-target resolution. The [generation coordinator](../internal/generation.md)
handles attempts, retries, usage, and active backend permits; the
[model client](../internal/llm.md) handles transport and decoding. Ordinary `Run`
requires an application prompt and returns no partial result on operational
failure, so calling it directly is not sufficient for the proposed diagnostic API.
Backend configuration has no diagnostic endpoints, capabilities, or default test
profile. New fields must follow [ADR 0001](../adr/0001-configuration-boundary.md):
ConfigKit owns loading and composition; PromptKit interprets and validates final
effective values. Provenance stays available for explicit inspection and is not
an execution-time resolution or validation mechanism.
## Public Operation And Request
One synchronous call runs one independently selected operation:
```go
func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error)
type SmokeTestKind string
const (
SmokeTestReachability SmokeTestKind = "reachability"
SmokeTestAuthentication SmokeTestKind = "authentication"
SmokeTestTextInference SmokeTestKind = "text_inference"
SmokeTestStructuredOutput SmokeTestKind = "structured_output"
)
type SmokeTestRequest struct {
Kind SmokeTestKind
BackendID string
PromptID string
PromptVersion string
ProfileID string
Execution *ExecutionTargetOverride
APIKey string `json:"-"`
TransientRetries *int
}
```
The request supplies no variables, inputs, appended messages, schema, or output
contract. An application prompt is a profile-selection aid only: its messages,
session template, input requirements, and output contract are not executed.
Consumers can compose several calls themselves; custom application tests remain
available through ordinary `Run`.
Reuse existing `ExecutionTargetOverride` semantics. Inference uses its endpoint,
model, numeric, reasoning, timeout, credential-source, and provider-extra controls.
Backend-only probes use endpoint, credential-source, and timeout settings as
applicable; they do not require model completeness or execute generation merely
because a model setting was supplied. Profile settings remain the inference
baseline unless overridden; no silent cheaper-model substitution is proposed.
Direct credentials take precedence over environment credentials and follow
existing request-scoped ownership and redacted-formatting conventions. Optional
missing/blank credentials are omitted from the upstream request. Authentication
can test the resulting unauthenticated request rather than reject missing keys
locally. If an explicitly required credential prevents inference dispatch, report
that local diagnostic finding without pretending the provider rejected it.
Ordinary execution's required-credential behavior remains unchanged.
## Selection And Routing
Preserve the accepted inference selection order:
| Caller selection | Execution profile |
| --- | --- |
| Prompt ID only | Selected prompt version's default profile |
| Profile ID only | Explicit profile |
| Both | Explicit profile overrides the prompt default |
| Neither | Explicit backend's configured default smoke-test profile |
Reuse exact identity and omitted-version semantics. An invalid explicit selection
does not fall back to a backend default. Missing backend/default selection is an
error when inference needs it, not a reason to substitute an arbitrary profile.
Backend-only probes require neither a default profile nor unrelated prompt data.
Whenever inference selects a profile, that profile supplies the resolved endpoint,
credential settings, and backend/capacity identity, subject to ordinary execution
overrides. This applies equally to explicit profiles, prompt defaults, and
backend-default profiles. A supplied `BackendID` is a selection fallback, not a
constraint or a request to rebind the selected profile's backend identity.
For example, if backend A names a default profile that resolves to backend B,
inference uses B's resolved target and capacity identity. If that default profile
is endpoint-only, inference retains an empty backend identity. Report A as the
default-selection source and B (or the empty identity) as the effective backend.
No recursive backend-default lookup or mismatch rejection is introduced.
For backend-only probes, an explicit or prompt-selected profile likewise supplies
the target and its backend's diagnostic metadata. Otherwise `BackendID` selects
the backend directly, without consulting its default inference profile. The
profile wins when both selectors are supplied, even when they name different
backends.
An endpoint-only profile remains usable for inference and reachability without
an inferred backend identity or provider-specific authentication adapter.
Endpoint overrides retain the effective backend's ordinary capacity identity.
Backend-only probes can resolve profile endpoint/credential metadata without
requiring its model. Prompt selection uses captured identity/default-profile
metadata without rendering or loading the application's artifacts or schemas.
## Report Shapes
Use factual Boolean observations with presence, rather than a caller-configurable
list of criteria. These Go-facing values need no stable JSON contract initially.
```go
type SmokeTestReport struct {
Kind SmokeTestKind
Supported bool
Applicable *bool
Completed bool
Reason SmokeTestReason
Target SmokeTestTarget
Facts SmokeTestFacts
Timing SmokeTestTiming
Attempts []SmokeTestAttempt
Usage ObservedTokenUsage
Failure ExecutionFailureKind
}
type SmokeTestTarget struct {
BackendID, ProfileID, PromptID, PromptVersion string
DefaultSourceBackendID string // backend whose default selected the profile, if any
SelectionSource string // "backend_default", "profile", "prompt", or "backend"
Endpoint string // initially resolved diagnostic destination
EffectiveSettings ExecutionTarget
SettingsPresence ExecutionTargetPresence
}
type SmokeTestFacts struct {
CredentialsSupplied *bool
EndpointReachable *bool
CredentialsAccepted *bool
ResponseReceived *bool
GenerationCompleted *bool
TextReturned *bool
ExpectedTextMatched *bool
StructuredOutputRequested *bool
ValidJSON *bool
SchemaMatched *bool
}
type SmokeTestTiming struct {
StartedAt time.Time
Duration time.Duration
CapacityWait *time.Duration
RequestDuration *time.Duration
RetryDelay *time.Duration
EvaluationDuration *time.Duration
}
type SmokeTestAttempt struct {
Dispatched bool
InitialEndpoint string // resolved destination before redirects
Endpoint string // last observed request destination; facts refer to this destination
HTTPStatus *int
Facts SmokeTestFacts
Timing SmokeTestTiming
Failure ExecutionFailureKind
Usage ObservedTokenUsage
}
type SmokeTestReason string
const (
SmokeTestReasonNone SmokeTestReason = ""
SmokeTestReasonAuthenticationNotApplicable SmokeTestReason = "authentication_not_applicable"
SmokeTestReasonProbeNotConfigured SmokeTestReason = "probe_not_configured"
SmokeTestReasonHTTPDiagnosticsUnavailable SmokeTestReason = "http_diagnostics_unavailable"
SmokeTestReasonRequiredCredentialMissing SmokeTestReason = "required_credential_missing"
SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized"
)
```
`Supported` records whether an implementation is available for the target and
selected client. `Applicable=false` records known inapplicability; nil leaves
applicability undetermined, for example when no authentication probe is configured.
Missing probe metadata does not imply that a provider accepts anonymous access.
`Reason` is a named factual code, not application severity or free-form diagnostic
text. The constants distinguish, respectively, no additional reason, explicitly
inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics
for the selected client, a required credential missing before inference dispatch,
and an authentication response whose documented meaning cannot be established.
HTTP status and `Failure` classify ordinary rejection and transport outcomes;
they do not need duplicate reason codes.
`Completed` means the diagnostic concluded, not that checks passed. A rejected,
failed, unsupported, or inapplicable diagnostic can conclude with nil Go error.
Caller cancellation before completion leaves it false and returns available facts.
**A nil Go error means the diagnostic concluded, not that the tested operation
succeeded.** Make this distinction prominent in GoDoc and consumer examples.
A fact is true or false only when established; nil means not evaluated,
undetermined, or inapplicable. Operation metadata and failure classification
provide context. In particular:
- `CredentialsSupplied`: a nonempty credential was attached to the observed
dispatched request for the attempt's `Endpoint`, not merely named in configuration
or attached to an earlier redirect hop. No observed request leaves this fact
absent. For injected clients, do not claim upstream transmission PromptKit cannot
observe. Observation at the transport boundary is not proof of wire transmission.
- `EndpointReachable`: an HTTP response was received from the tested destination,
regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation
ending without a response yields false. An injected generation result alone
does not establish HTTP reachability.
- `CredentialsAccepted`: the designated authentication adapter recognized acceptance
or rejection. A rejection yields false; transport failure, server error, or an
uninterpretable response can leave it nil. Acceptance does not certify balance,
model permissions, or future access. Absence of credentials is not an invalid-key
assertion: a documented 401 with `CredentialsSupplied=false` records rejection
of an unauthenticated request, not rejection of a supplied key.
- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a
response was returned by the injected generation boundary. A non-success or
malformed HTTP response still counts as received.
- `GenerationCompleted`: generation decoded/completed successfully, separately from
generated-content checks. Empty decoded content can complete generation.
- `TextReturned`: generated content is non-whitespace. `ExpectedTextMatched` compares
against the private expected text using trimming only, without case folding or
semantic matching.
- `StructuredOutputRequested`: the fixed native schema constraint was supplied to
the built-in provider request or injected generation boundary. This is not proof
of upstream transmission by an injected client or provider enforcement.
- `ValidJSON`: the generated content, not the HTTP envelope, parsed as JSON.
`SchemaMatched` evaluates parsed JSON against the fixed test schema and remains
absent when parsing or response acquisition prevents evaluation.
Known inapplicable authentication performs no dispatch and leaves request and
acceptance facts absent. Unknown measurements remain unknown, including phase
measurements for unperformed work. Report-level capacity wait, request duration,
evaluation duration, and retry delay sum the corresponding measured work across
the diagnostic. Request duration excludes capacity waiting, evaluation, and retry
backoff. Retry delay includes completed and interrupted backoff. Per-attempt
timings retain their own measurements; backoff is reported at the operation level.
Total duration is wall time, including selection and other orchestration overhead,
so it need not equal the phase sum. Timings impose no application latency threshold
or universal deadline.
If retries are enabled, top-level facts describe the most recent dispatched
attempt. Cancellation during backoff or a later capacity wait does not replace
those facts with an undispatched attempt's empty facts. The operation's `Failure`
and `Completed` still describe the interruption; earlier attempt records retain
their own outcomes. A canceled capacity acquisition can add a `Dispatched=false`
record, but no synthetic attempt is added for cancellation during backoff. With
no dispatched attempt, request facts remain absent. Usage reuses presence-aware
accounting without claiming complete provider billing. No facts are inferred from
arbitrary error text.
`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt
observations; it is not proof that packets were sent or billable work occurred.
Reports omit private prompts, expected text, schemas, generated content, credentials,
raw bodies, and headers. Use content-free failure classifications and HTTP status,
not arbitrary provider messages, to preserve this exposure boundary. Effective
settings, provider-extra values, endpoints, and identities may still be sensitive
operator data; applications own their disclosure. Apply existing credential
exclusion and formatting rules rather than promise that all configuration is safe
for unrestricted logging.
## Diagnostic Procedures And Capability Configuration
### Reachability
Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or
the effective base endpoint when no separate endpoint is supplied. Any HTTP
response establishes reachability, including 401, 404, or 405. It does not establish
that the completion route, credentials, or a model work. Authentication is tested
separately. Classify transport failure where available without promising exhaustive
DNS/TLS diagnosis.
### Authentication
Metadata names a supported provider adapter or explicitly marks authentication
inapplicable. The first adapter is `openrouter_key`, using GET on the
selected base endpoint's `key` path with ordinary bearer credential resolution.
OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key)
documents this authenticated operation and its 200 and 401 responses. Recognize
the documented successful shape and unauthorized response; leave acceptance
unknown on other outcomes instead of guessing. Returned account/key details
are not exposed.
Configure RakestrawHome authentication as `none`, pursuant to the maintainer's
stated deployment behavior. Other backends can declare supported adapters.
An arbitrary models-list or health endpoint is not credential-acceptance evidence.
### Inference
Plain-text inference uses a small fixed private prompt and expected answer.
Structured-output inference uses a fixed private prompt and self-contained schema,
requests native JSON Schema output through the existing client path, and evaluates
JSON and schema conformance locally. It imports no application output contract.
Exact content is an implementation detail and does not enter public prompt listings,
inspection, configuration inspection, prepared snapshots, or reports. Injected
clients and providers necessarily receive the content they execute: privacy here
is an API-exposure boundary, not secrecy from executing collaborators. There is
no persistent diagnostic prompt or session state.
### Backend Metadata
Extend optional backend configuration through the final-value loader and the
`BackendConfig`/`BackendSource` convenience. Illustrative shape:
```yaml
backends:
openrouter:
smoke_tests:
default_profile: promptkit-openrouter-standard
authentication:
kind: openrouter_key
rakestrawhome:
smoke_tests:
default_profile: rakestrawhome-gemma-4-31b
authentication:
kind: none
```
Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted
reachability metadata uses the effective base for the generic HTTP probe;
omitted authentication metadata makes that operation unsupported, with applicability
undetermined. Missing default-profile selection matters when inference selects it,
not for backend-only checks or an explicit profile. Retain normal effective
presence/null and layering semantics. Validate syntax and recognized configuration
at the final handoff, without contacting providers or validating intermediate layers.
Provider paths follow the selected endpoint override, not silently the catalog's
original endpoint. Explicit absolute diagnostic endpoints are supported for separate
health/auth routes and take precedence for their operation. Configuration stays
limited to named adapters and endpoint settings, not arbitrary methods or response
predicates. The companion implementation plan specifies convenience types and
presence/null interpretation for these fields.
Reuse the configured HTTP client and its redirect handling, without adding custom
credential forwarding. `Target.Endpoint` records the resolved starting destination.
Each HTTP attempt records `InitialEndpoint` and the last observed dispatched
request destination as `Endpoint`; its credential, reachability, response, and
acceptance facts describe that destination. If a redirect leads to a failed request,
do not attribute an earlier hop's response or credentials to the later destination.
When a client prevents observing the relevant request, leave the affected evidence
unknown. Injected generation alone supplies no HTTP destination evidence.
Authentication acceptance is interpreted from the response for the observed
destination under the selected adapter's contract. It does not assert acceptance
at the initial destination or at every redirect hop. These endpoint fields provide
attribution without introducing a public redirect trace.
Deliver built-in metadata in the local catalog repositories at
`../promptkit-backend-openrouter` and `../promptkit-backend-rakestrawhome`.
Catalog changes and PromptKit adoption are included in the sprint; publication
and remote module availability are not intermediate dependencies. Caller
overlays remain available through ordinary composition. Interpret and validate
only the assembled final configuration. Built-in reachability follows the
effective endpoint rather than pinning a separate catalog endpoint.
The catalogs expose diagnostic metadata through an additive, opt-in embedded
asset root. Existing `/v3` catalog roots remain usable by older strict loaders;
new PromptKit explicitly selects the additional assets. Provider metadata stays
owned by its catalog rather than being duplicated in PromptKit.
Local candidate tags identify validated contents; they are not published releases.
The implementation plan owns candidate selection and local proxy mechanics.
The catalogs' publication-date versioning convention still applies to later
publication. If that requires new final version names, update PromptKit's pins
and checksums accordingly; retain the validated catalog contents and never move
a published tag. Selecting the new metadata requires a compatible PromptKit
loader. No intervening PromptKit loader release or external module-download
verification is required to finish implementation.
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
setting. Verify each shipped default resolves in the supported assembled effective
configuration, including any PromptKit-owned profiles. This is final-value
integration validation, not a second validation pass over intermediate sources.
A catalog default is replaceable metadata,
not a promise of upstream availability, cheap inference, or a stable model choice.
## Execution And Error Boundary
Reuse target resolution, generation coordination, HTTP decoding, structural
validation, and usage accounting internally. A separate diagnostic orchestration
entry point may retain evidence that ordinary `Run` intentionally discards.
Do not duplicate inference serialization or build a second inference stack.
Add a narrow internal evidence path at the transport/coordinator boundary. Existing
execution events do not carry redirect destinations or dispatched credential
presence, and the coordinator can discard a completed response when cancellation
is observed before return. Capture available facts before that discard and retain
them for diagnostics. Keep ordinary execution's error/result contracts unchanged;
do not expand public execution events just to reconstruct smoke-test evidence.
Inference participates in existing engine-local admission and active backend
limits. HTTP-only probes need no generation permit or unrelated model selection.
Caller context and configured timeouts govern operations. Inference defaults to
one attempt: nil `TransientRetries` means zero for this operation rather than
inheritance from the engine retry budget. Positive values reuse existing built-in
inference status retries and retain attempt records; injected generation remains
single-dispatch. Reachability and authentication remain single-attempt regardless
of a valid inference retry setting. Structural repair is excluded from the initial
procedure. Otherwise use the selected profile plus explicit overrides. Ordinary
engine defaults and repair behavior do not change. Small test content does not
establish small output, a price estimate, or a cost ceiling.
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
new required methods. HTTP diagnostics must describe an explicitly selected
transport, not secretly probe a different transport in an injected-client engine.
For injected-client engines, the initial delivery reports HTTP diagnostics
unavailable and adds no separate diagnostic HTTP-client option. Inference
diagnostics remain available through the injected client. Known inapplicability,
such as authentication configured as `none`, can be reported without any transport.
Do not silently fall back to a default HTTP client to diagnose an injected
generation client. A future explicit diagnostic transport option remains possible
but is not needed to complete this feature.
Return conventions:
- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections
return ordinary public errors before probe work. Unsupported capability is not
the same as invalid selection.
- Concluded diagnostics return reports with nil Go error, including credential
rejection, provider rejection, transport or configured request-timeout failure,
invalid JSON, schema mismatch, unsupported capability, or inapplicability.
Local capacity exhaustion is a diagnostic execution finding, not provider failure;
request facts remain absent if nothing was dispatched.
- Caller cancellation or caller-context deadline expiration returns accumulated
observations with an error recognizable through `errors.Is` against
`context.Canceled` or `context.DeadlineExceeded`. An already-done context can yield
a report with no observations. Do not dispatch again merely to fill missing facts.
As elsewhere, arbitrary synchronous injected code must return before cancellation
can be observed.
The cancellation contract applies to this new operation, not to existing
preparation APIs. Internal faults preventing meaningful diagnostic execution may
still return an ordinary error; the implementation plan should identify those
boundaries without promising that every conceivable failure becomes a report.
## Representative Requests And Results
Request examples use ordinary exact selections, without private test language:
```go
// Select the application's default profile, but do not run its prompt.
req := promptkit.SmokeTestRequest{
Kind: promptkit.SmokeTestTextInference,
PromptID: "meeting.summary",
}
// An explicit profile overrides that default.
req.ProfileID = "promptkit-openrouter-standard"
// Select a backend's configured default for structured inference.
structured := promptkit.SmokeTestRequest{
Kind: promptkit.SmokeTestStructuredOutput,
BackendID: "openrouter",
}
```
Expected report examples:
| Operation/outcome | Observations | Go error |
| --- | --- | --- |
| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil |
| RakestrawHome authentication | Supported true, applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil |
| HTTP diagnostics unavailable for injected client | Supported false, applicability unknown, completed, `http_diagnostics_unavailable`; no dispatch | nil |
| Authentication without credentials receives documented 401 | Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request | nil |
| Required credential absent before inference | Supported/applicable/completed; `required_credential_missing`, failure `invalid_request`; no dispatch and no request facts | nil |
| Authentication receives uninterpretable HTTP 200 | Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; `authentication_response_unrecognized`, failure `malformed_response` | nil |
| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil |
| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil |
| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil |
| Configured request timeout while caller context remains active | Supported/applicable/completed; failure `deadline`; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts | nil |
| Local capacity exhausted | Supported/applicable/completed; failure `capacity`; no dispatch or request facts | nil |
| Caller cancels during initial capacity wait | Supported/applicable, completed false; failure `canceled`, measured capacity wait, undispatched attempt; request facts absent | `context.Canceled` identity |
| Caller deadline expires during a dispatched request | Supported/applicable, completed false; failure `deadline`; retain available attempt evidence | `context.DeadlineExceeded` identity |
| Caller cancels during retry backoff after HTTP 503 | Completed false, failure `canceled`; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt | `context.Canceled` identity |
| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error |
Consumers decide whether any of these findings warrant success, a warning, or a
failure in their own application. They own CLI diagnostics, exit statuses,
scheduling, and any combination with `ValidateRequest` or normal execution.
## Work Needed And Target Completion
- Add the public request/report operation and GoDoc, keeping business logic internal.
- Extend final backend metadata interpretation and captured lookup for capabilities
and defaults. Preserve explicit configuration inspection without exposing test
content or using source history for validation.
- Connect private inference content to shared selection, execution resolution,
capacity, generation, decoding, and structural evaluation without preparing
application inputs or messages.
- Add narrow HTTP probes, credential handling, evidence retention, factual results,
and partial cancellation reporting through the internal evidence path while
preserving existing client contracts. Specify destination/credential attribution
without exposing headers or a redirect trace.
- Deliver the agreed metadata through both local catalogs with ordinary
catalog/caller precedence and no provider-availability guarantee. Shipped
defaults resolve in the assembled final configuration, and selected versions
work without workspace/replacement substitution. The single-sprint objective
applies to the complete integration.
- Add focused offline tests with controlled transports and injected clients for
selection, rejected/absent credentials, applicability/support, transport/malformed
responses, content/JSON/schema facts, cancellation, shared inference capacity,
and credential/private-content exposure. Assign distinct test owners; do not
repeat existing transport, capacity, or validation matrices or snapshot private
test language.
- Include the report-state matrix, cross-backend and endpoint-only default profiles,
redirect attribution, cancellation after a response, and interrupted retry waits
in focused contract coverage. Reuse existing lower-level checks where sufficient.
- Update canonical consumer, backend-format, integration, and internal documentation
when implementation lands, with release notes explaining the optional addition.
Live diagnostics are explicit consumer operations, not default tests or new
release gates.
Completion means consumers can select the four kinds, obtain supported factual
observations, distinguish false from unknown/inapplicable, and retain evidence
on cancellation. Existing preparation and execution continue unchanged. Initial
scope excludes application CLI/service work, readiness policy, continuous monitoring,
model-status tests, batch orchestration, customizable test language, cost estimates,
and guarantees about later generation.