Retire completed implementation and review roadmaps
This commit is contained in:
@@ -1,846 +0,0 @@
|
||||
# Configuration Collections, Namespaces, And Scoped Lookup
|
||||
|
||||
Status: implementation complete; roadmap retirement remains for normal release
|
||||
preparation. This document consolidates the namespace roadmap and selected
|
||||
collection proposal 3. The configuration and namespace design decisions are
|
||||
settled. The companion [implementation plan](implementation.md) records concrete
|
||||
API delivery choices and the completed work.
|
||||
|
||||
## Purpose
|
||||
|
||||
Simplify PromptKit configuration by loading independently maintained prompt,
|
||||
profile, and backend libraries as YAML collections instead of mandatory Go
|
||||
modules. Remove the unused prompt-version selection scheme. Specification
|
||||
headers are optional: maintainers can declare compatibility in a collection
|
||||
entry without adding a declaration to every file.
|
||||
|
||||
Consumers can select a namespace once and use local resource names while an
|
||||
engine contains several collections with overlapping names. For example,
|
||||
WeatherReporter and Homewatch can each define `summarize.alerts`, while provider
|
||||
namespaces distinguish profiles targeting the same upstream model.
|
||||
|
||||
Conventions and small convenience APIs should reduce authoring and maintenance
|
||||
work. Preserve consumer choice over sources, layouts, namespaces, and precedence
|
||||
without adding speculative options, mandatory workflows, or new compatibility
|
||||
and provenance guarantees.
|
||||
|
||||
## Prerequisite: Adopt ConfigKit v0.8.0
|
||||
|
||||
The target selects released ConfigKit **v0.8.0** for the collection schema and
|
||||
loading APIs. At planning time PromptKit selected v0.7.1; the
|
||||
[implementation plan](implementation.md) records adoption and verification of
|
||||
v0.8.0. Local filesystem replacements are not part of the distributed module.
|
||||
|
||||
The v0.8.0 tag was reviewed at `137769f` on 2026-10-17. It supplies input
|
||||
placement, relative import destinations, and shared source-version checks needed
|
||||
by this design. No additional upstream extension is required. The implementation
|
||||
plan records adoption of this prerequisite.
|
||||
|
||||
ConfigKit's public `docs/config.md`, ADRs 0011–0014, and v0.8.0 release notes own
|
||||
the upstream contracts. The capabilities used here are:
|
||||
|
||||
- `Input.Destination` for root-relative placement and import `destination` for
|
||||
placement relative to the immediate importing document.
|
||||
- `Options.SourceVersionMode` and `LayeredOptions.SourceVersionMode`, applying
|
||||
a shared `Definition.Version` policy in each document's source-local coordinates.
|
||||
- Consumption of accepted non-main headers before composition, with the main
|
||||
header retained and protected at its effective application path.
|
||||
- Detached `SourceInfo.Destination` and `SourceInfo.VersionCheck` evidence,
|
||||
together with existing source locations and resource contexts.
|
||||
|
||||
### Keep Mapping-Based Collections
|
||||
|
||||
ConfigKit v0.8.0 also offers sequence-item placement and registered composition
|
||||
of distinct list records. These were reviewed and are not selected for this
|
||||
round. A higher-priority list replaces the entire lower-priority list, duplicate
|
||||
identities conflict within a strict group, and records cannot inherit from
|
||||
sibling list records. Switching formats would sacrifice ordinary per-resource
|
||||
partial overrides and local inheritance for flatter authoring.
|
||||
|
||||
Keep collection resources as mappings keyed by local ID. Do not add an
|
||||
alternative list schema or a PromptKit identity-based merge layer. The existing
|
||||
key supplies identity without an additional embedded `id`. This is a scope
|
||||
choice for this round, not a prohibition on future upstream capabilities.
|
||||
|
||||
## Existing Foundation And Ownership
|
||||
|
||||
The [format reference](../formats.md) and
|
||||
[consumer guide](../consumers/pkg-promptkit.md) remain authoritative for current
|
||||
behavior until implementation lands. Directory and embedded configuration
|
||||
loading already exist through `DirectoryInputs` and `FilesystemInputs`.
|
||||
The current engine also supplies provider catalog modules and built-in profiles;
|
||||
the accepted removal is described below.
|
||||
|
||||
[ADR 0001](../adr/0001-configuration-boundary.md) assigns source loading,
|
||||
composition, and inheritance to ConfigKit. PromptKit interprets and validates
|
||||
only final effective values, including their resource contexts. Source history
|
||||
is available for explicit inspection, not additional validation of contributions.
|
||||
Do not add a manifest pre-parser, source-level namespace ownership checks, or a
|
||||
second composition pass. Two sources targeting the same namespace follow
|
||||
ordinary ConfigKit composition; a repeated namespace is not a separate error.
|
||||
|
||||
The [architecture policy](../policy/architecture.md) continues to assign CLI
|
||||
syntax, discovery policy, and application lifecycle to consuming applications.
|
||||
|
||||
## Collection Ownership And Built-In Removal
|
||||
|
||||
**Accepted: load collections as application/operator-supplied configuration,
|
||||
and remove PromptKit's built-in profiles as a breaking change.** Remove the
|
||||
mandatory OpenRouter and RakestrawHome catalog Go-module dependencies and their
|
||||
automatic contribution to engine configuration. PromptKit owns the definition
|
||||
format and execution machinery; collection publishers own their configuration
|
||||
repositories, and applications/operators select and provision the collections
|
||||
they use. One or more explicitly selected directories can supply backends,
|
||||
profiles, and prompts through the existing ConfigKit integration.
|
||||
|
||||
Remove `promptkit-openrouter-standard` and `promptkit-openrouter-auto`, including
|
||||
their implicit catalog dependencies. Do not replace them with another mandatory
|
||||
starter bundle. Applications may embed their own defaults using the same
|
||||
configuration format and loader, then layer operator configuration as desired.
|
||||
Ordinary application overrides remain valid configuration and need not declare
|
||||
a collection release of their own.
|
||||
|
||||
This decouples collection updates from PromptKit rebuilds and releases and
|
||||
avoids imposing provider inventories on every consumer. The tradeoff is that
|
||||
applications and deployments must supply their required configuration and
|
||||
manage compatible collection revisions. PromptKit does not gain a collection
|
||||
installer, automatic downloader, or implicit directory search policy.
|
||||
|
||||
Retain the captured-engine lifecycle: configuration changes take effect when
|
||||
the application constructs and validates a replacement engine. A change to an
|
||||
installed directory does not mutate an existing engine's captured definitions.
|
||||
Distribution as configuration does not make new provider protocols or diagnostic
|
||||
adapters data-only features; those can still require implementation work.
|
||||
|
||||
Prepare explicit identifier mappings, replacement configuration examples, and
|
||||
consumer migration instructions. Update the built-in-profile promise in the
|
||||
release policy and its canonical format documentation when removal lands.
|
||||
Use a release permitted to break configuration and API contracts under the
|
||||
current release policy: a minor release before v1, or a major release at/after
|
||||
v1. Actual catalog mappings, packaging arrangements, and release versions remain
|
||||
delivery work; independent collection releases need not track future PromptKit
|
||||
software releases.
|
||||
|
||||
## Accepted Identity Convention
|
||||
|
||||
Reserve `/` exclusively as the namespace separator in PromptKit prompt, profile,
|
||||
and backend identifiers and references. A qualified identifier has exactly two
|
||||
nonempty components: `namespace/local-name`. Neither component can contain `/`.
|
||||
Dots remain available within local names. An unqualified identifier has no slash
|
||||
and remains valid under the existing nonblank-ID rule.
|
||||
|
||||
Reject resource identifiers or supplied references with multiple slashes or an
|
||||
empty qualified component, such as `a/b/c`, `/name`, and `namespace/`. These
|
||||
rules do not apply to provider model values, endpoints, resource paths, or prompt
|
||||
versions. Preserve exact spelling; do not silently trim or rewrite IDs.
|
||||
|
||||
| Resource | Namespace | Local name | Qualified ID |
|
||||
| --- | --- | --- | --- |
|
||||
| Prompt | `weatherreporter` | `summarize.alerts` | `weatherreporter/summarize.alerts` |
|
||||
| Prompt | `homewatch` | `summarize.alerts` | `homewatch/summarize.alerts` |
|
||||
| Profile | `openrouter` | `google.gemma-4-31b-it` | `openrouter/google.gemma-4-31b-it` |
|
||||
| Profile | `rakestrawhome` | `google.gemma-4-31b-it` | `rakestrawhome/google.gemma-4-31b-it` |
|
||||
|
||||
Prompt, profile, and backend identities remain separate resource types. The
|
||||
same namespace can contain all three, and resources in it may come from
|
||||
multiple configuration sources. A namespace does not imply a directory,
|
||||
package, provider model identifier, or security boundary.
|
||||
|
||||
A profile ID identifies execution settings. Its configured model value is
|
||||
still the identifier passed to the provider; qualifying the profile ID does
|
||||
not modify that value. Both example profiles may retain the configured model
|
||||
`google/gemma-4-31b-it`. Different profiles may select the same model with
|
||||
different settings. Prompt version labels remain separate from namespaces. The
|
||||
selected collection format installs one definition per prompt ID and removes
|
||||
request-side version selection and assertions. Optional labels are descriptive.
|
||||
|
||||
Retain unqualified and dotted IDs. Existing slash-containing IDs need review:
|
||||
those with one slash and nonempty components become qualified identities, while
|
||||
those outside the new grammar require migration. This intentional format break
|
||||
is accepted in favor of unambiguous resolution; the migration work is described
|
||||
below.
|
||||
|
||||
## Accepted Consumer Scope
|
||||
|
||||
Provide a lightweight, immutable view over an existing engine with independent
|
||||
prompt, profile, and backend scopes. The following sketch illustrates the
|
||||
accepted shape; concrete declarations are specified in the
|
||||
[implementation plan](implementation.md):
|
||||
|
||||
```go
|
||||
weather, err := engine.WithNamespaces(Namespaces{
|
||||
Prompt: "weatherreporter",
|
||||
Profile: "openrouter",
|
||||
})
|
||||
// Handle err, then use local prompt and profile names through weather.
|
||||
```
|
||||
|
||||
The concrete declarations are recorded in the
|
||||
[implementation plan](implementation.md). Empty scope fields mean no scope for
|
||||
that resource type. Keep name qualification shared across the view and engine;
|
||||
do not duplicate selection logic in each operation. Omit a one-string
|
||||
shorthand that implicitly scopes all resource types.
|
||||
|
||||
The implemented views share the engine's captured configuration, clients, and
|
||||
execution machinery. Selecting a namespace does not rebuild the engine or
|
||||
duplicate its backend capacity accounting. Consumers can use different views
|
||||
concurrently without changing a shared current-namespace setting.
|
||||
|
||||
The accepted resolution rules, independent of the final scope API, are:
|
||||
|
||||
1. A supplied name without a slash is qualified using the selected scope for its
|
||||
resource type, then looked up exactly. Without that scope, it is looked up as
|
||||
an exact unqualified ID.
|
||||
2. A missing scoped resource does not trigger a search in other namespaces or
|
||||
among unqualified IDs. Loading another collection cannot redirect lookup.
|
||||
3. A valid supplied name containing one slash is fully qualified and is looked
|
||||
up exactly, regardless of scope. Malformed resource references are rejected.
|
||||
4. Scope applies to caller-supplied resource names. Stored references, including
|
||||
prompt default profiles, profile backends, inheritance references, and backend
|
||||
smoke-test default profiles, retain their existing exact meaning. They are
|
||||
not rebound to the invoking consumer's namespace.
|
||||
5. Omitted optional IDs retain their existing selection/default behavior; an
|
||||
omission is not converted into a namespace prefix or a lookup for an empty
|
||||
local name.
|
||||
6. Where outputs, inspection, errors, or diagnostic reports identify a selected
|
||||
resource, preserve its resolved full identity. Any exposed local name should
|
||||
be distinguishable from that identity.
|
||||
|
||||
Keep qualification consistent across execution, preparation, request validation,
|
||||
prompt/profile inspection, and applicable smoke-test selections. Namespace
|
||||
selection must not change request override precedence or provider model strings.
|
||||
Prompts resolve by ID alone.
|
||||
|
||||
### Local And Qualified References In One Operation
|
||||
|
||||
Syntax alone distinguishes local and qualified references. Do not decide whether
|
||||
to qualify a name by checking which candidates exist; that would make resolution
|
||||
depend on loaded content.
|
||||
|
||||
Within a `weatherreporter` prompt scope, `summarize.alerts` resolves only to
|
||||
`weatherreporter/summarize.alerts`, while `homewatch/summarize.alerts` explicitly
|
||||
selects that other library's prompt. A qualified profile override similarly
|
||||
bypasses the profile scope. Scoped operations accept both forms without a
|
||||
separate reference type or requiring a return to the original engine for
|
||||
cross-namespace access.
|
||||
|
||||
The original engine remains a way to select an exact unqualified ID without
|
||||
applying a scope. No escape syntax for unqualified IDs is introduced here.
|
||||
|
||||
### Separate Resource Scopes
|
||||
|
||||
A consumer can select prompt shorthand under `weatherreporter` and profile
|
||||
shorthand under `openrouter` in the same request. Each resource type has an
|
||||
independent scope.
|
||||
A prompt's configured default profile needs no caller profile scope because
|
||||
the stored reference is already exact. Scope syntax and existence handling are
|
||||
defined in [scope validation](#scope-validation).
|
||||
|
||||
### Scope Validation
|
||||
|
||||
**Accepted: validate syntax when creating the view; resolve existence when
|
||||
an operation selects a resource.** In the scope value, the empty
|
||||
string means no scope. Reject a nonempty namespace containing `/` or consisting
|
||||
only of whitespace. Otherwise preserve its exact spelling, with no trimming,
|
||||
case folding, or additional character whitelist. This distinguishes omission
|
||||
from an unusable namespace without inventing normalization rules. Report invalid
|
||||
scope construction through PromptKit's existing public error conventions.
|
||||
|
||||
Allow a well-formed namespace even when it has no current members. Its scoped
|
||||
listing would be empty, and selection of a missing local resource would produce
|
||||
the ordinary lookup error. Qualified selections would remain usable regardless
|
||||
of the view's default scope. An empty optional request ID continues to mean
|
||||
omission under that operation's existing contract; it is never qualified.
|
||||
|
||||
This keeps scope as a naming context and supports uniform application setup
|
||||
across installations with different library contents. It also avoids treating
|
||||
successful scope construction as validation of a prompt or execution profile.
|
||||
The tradeoff is that a misspelled but syntactically valid namespace is detected
|
||||
only when listed or used, rather than during view construction.
|
||||
|
||||
## Discovery And Namespace Filtering
|
||||
|
||||
**Accepted: integrated namespace filtering and full-identity results.**
|
||||
Enumerate the distinct named namespaces present in final effective
|
||||
prompts, profiles, and backends. Allow filtering that enumeration by resource
|
||||
type so a prompt picker need not show profile-only namespaces. Provide scoped
|
||||
listings for the supported resource types, adding a backend inventory surface
|
||||
as part of this work if needed. Namespace filtering is an explicit option in
|
||||
the library listing API, not a consumer-side filtering helper. Preserve unfiltered
|
||||
engine listing behavior, with the accepted removal of prompt-version identity
|
||||
and selection. The companion implementation plan defines the Go filter shape.
|
||||
|
||||
| Selection | Listing behavior |
|
||||
| --- | --- |
|
||||
| No explicit filter on the engine | All namespaces, including unqualified resources |
|
||||
| No explicit filter on a scoped view | The selected scope for that resource type; all resources of that type if no scope is selected |
|
||||
| Explicit named namespace | Resources in exactly that namespace |
|
||||
| Explicit unqualified-only filter | Resources whose IDs have no namespace |
|
||||
| Explicit all-namespaces filter | All resources of that type, including unqualified resources |
|
||||
|
||||
An explicit filter replaces the view's default for the current listing; it
|
||||
does not intersect with that default or mutate the view. A named namespace
|
||||
filter uses the namespace syntax rules above. A well-formed namespace
|
||||
with no matching resources returns an empty listing. Represent default, named,
|
||||
unqualified-only, and all-namespaces selections distinctly in the API; do not
|
||||
overload an empty string to mean several of them.
|
||||
|
||||
Return full IDs with optional prompt version labels separately; the full ID
|
||||
identifies the installed prompt. Do not invent a named namespace such as `default`
|
||||
for unqualified resources. Namespace enumeration excludes that unnamed group.
|
||||
Use deterministic ordering consistent with existing listing contracts.
|
||||
|
||||
Qualified result IDs can be passed unchanged to either scoped or unscoped
|
||||
operations. Unqualified IDs remain literal unqualified identities: when an
|
||||
explicit all-namespaces or unqualified-only filter exposes one through a scoped
|
||||
view, use the original engine or a view without that resource scope to select
|
||||
it exactly. A listing filter does not change subsequent execution scope. This
|
||||
preserves the accepted lookup rules without introducing an escape syntax.
|
||||
|
||||
Integrated filtering gives consumers a direct way to discover the desired
|
||||
resources, while full identities keep logs and subsequent selections
|
||||
unambiguous. Namespace/local-name display decomposition is separate from
|
||||
filtering; a public splitting helper is not required by this decision.
|
||||
|
||||
Namespace membership is exact: `weather` does not include `weatherreporter`.
|
||||
Collection inventory describes final collection declarations and optional release
|
||||
labels; it is distinct from enumeration of namespaces that contain resources.
|
||||
See [collection inventory](#inspection-and-effective-collection-inventory).
|
||||
|
||||
## Authoring And Assembly
|
||||
|
||||
A file such as `weatherreporter/prompts/alerts.yml` would contain:
|
||||
|
||||
```yaml
|
||||
summarize.alerts:
|
||||
version: "1"
|
||||
default_profile: provider/brief
|
||||
messages:
|
||||
- role: user
|
||||
content: Summarize the weather alerts.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
```
|
||||
|
||||
The version may simply be omitted. Another collection can contribute the same
|
||||
local prompt name, for example `homewatch/prompts/alerts.yml`:
|
||||
|
||||
```yaml
|
||||
summarize.alerts:
|
||||
default_profile: provider/brief
|
||||
messages:
|
||||
- role: user
|
||||
content: Summarize the household alerts.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
```
|
||||
|
||||
A provider collection's `profiles/brief.yml` uses the same local-key convention:
|
||||
|
||||
```yaml
|
||||
brief:
|
||||
backend: provider/default
|
||||
model: vendor/example-model
|
||||
temperature: 0.2
|
||||
```
|
||||
|
||||
Its `backends/default.yml` could contain this synthetic, non-live endpoint:
|
||||
|
||||
```yaml
|
||||
default:
|
||||
endpoint: https://inference.example.invalid/v1
|
||||
```
|
||||
|
||||
A conventional `weatherreporter/collection.yml` entry can declare the shared
|
||||
specification, optional collection release, and its layout:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
release_version: "2.4.0"
|
||||
configkit:
|
||||
imports:
|
||||
- glob: prompts/*.yml
|
||||
destination: [prompts]
|
||||
```
|
||||
|
||||
A provider entry can import `profiles/*.yml` at `[profiles]` and
|
||||
`backends/*.yml` at `[backends]` in the same way. Collection publishers choose
|
||||
which categories to include. A required glob must match; unused categories can
|
||||
be omitted or use ConfigKit's existing optional-import behavior. Ordinary
|
||||
fragments may omit specification headers under the selected `IfPresent` mode.
|
||||
An entry header is the recommended publishing convention, not a load-time
|
||||
requirement selectively enforced on entries by this mode.
|
||||
|
||||
The application selects each entry, enables its imports, and supplies its
|
||||
namespace through the public input destination. For example, the following
|
||||
partial Go snippet uses released ConfigKit v0.8.0 APIs:
|
||||
|
||||
```go
|
||||
entry := configkit.Input{
|
||||
Source: configkit.File("/usr/local/share/collections/weatherreporter/collection.yml"),
|
||||
Imports: true,
|
||||
Destination: configkit.Keys("collections", "weatherreporter"),
|
||||
}
|
||||
```
|
||||
|
||||
For independent peer collections, one `Underlay` group's `Inputs` is a useful
|
||||
starting point, not a required grouping. Consumers choose precedence and groups.
|
||||
Configure `LayeredOptions.SourceVersionMode` as `SourceVersionModeIfPresent` and
|
||||
`Definition.Version.Path` as `configkit.Keys("version")`, with PromptKit's
|
||||
supported-specification checker. The application main input remains at the
|
||||
root and declares `version: 1` in the examples below; both the main input and
|
||||
its declaration are optional. Namespace and source selection require no manifest parsing in PromptKit.
|
||||
|
||||
| Selected source | Effective destination |
|
||||
| --- | --- |
|
||||
| Application main document | Configuration root |
|
||||
| `weatherreporter/collection.yml` | `[collections, weatherreporter]` |
|
||||
| Its `prompts/*.yml` imports | `[collections, weatherreporter, prompts]` |
|
||||
| Homewatch entry and its prompt imports | `[collections, homewatch]` and `[collections, homewatch, prompts]` |
|
||||
| Provider entry and its profile/backend imports | `[collections, provider]`, then `[profiles]` or `[backends]` appended |
|
||||
|
||||
Directly selected fragments with `Input.Destination` remain possible. Entry
|
||||
imports are the recommended packaging convention because publishers can describe
|
||||
their layout. Filenames and directory basenames do not define resource identity;
|
||||
no implicit directory discovery, namespace inference, or collection installer is
|
||||
introduced. A small helper should construct inputs from an explicitly selected
|
||||
source and namespace, without inspecting manifests to compute destinations. `collection.yml`
|
||||
is a convention, not a required filename. Direct fragments and programmatic
|
||||
sources remain supported through the source API; no parallel collection loader
|
||||
or mandatory filesystem layout is introduced.
|
||||
|
||||
Use later underlay groups or main for intentional application/operator overrides.
|
||||
Placement does not establish precedence. Do not also recursively scan files
|
||||
already selected by entry imports: ConfigKit's repeated-source rules still apply.
|
||||
|
||||
The entry's source-root `version` is checked and consumed before its payload
|
||||
composes. Its `release_version` is retained as ordinary collection metadata.
|
||||
A prompt's nested optional `version` is also ordinary payload. These are three
|
||||
distinct concepts; source-root release metadata must no longer use `version`.
|
||||
|
||||
The assembled application shape would be equivalent to:
|
||||
|
||||
```yaml
|
||||
version: 1
|
||||
collections:
|
||||
weatherreporter:
|
||||
release_version: "2.4.0"
|
||||
prompts:
|
||||
summarize.alerts:
|
||||
version: "1"
|
||||
default_profile: provider/brief
|
||||
messages:
|
||||
- role: user
|
||||
content: Summarize the weather alerts.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
homewatch:
|
||||
prompts:
|
||||
summarize.alerts:
|
||||
default_profile: provider/brief
|
||||
messages:
|
||||
- role: user
|
||||
content: Summarize the household alerts.
|
||||
output:
|
||||
format: text
|
||||
validation_mode: none
|
||||
provider:
|
||||
backends:
|
||||
default:
|
||||
endpoint: https://inference.example.invalid/v1
|
||||
profiles:
|
||||
brief:
|
||||
backend: provider/default
|
||||
model: vendor/example-model
|
||||
temperature: 0.2
|
||||
```
|
||||
|
||||
This yields `weatherreporter/summarize.alerts` and
|
||||
`homewatch/summarize.alerts`, plus profile `provider/brief` and backend
|
||||
`provider/default`. The provider's model string remains `vendor/example-model`;
|
||||
it is not a PromptKit resource identifier.
|
||||
|
||||
One fragment may contain several local definitions. A fragment starting directly
|
||||
with `messages:` would need a destination that also supplies the local prompt
|
||||
key. ConfigKit mounting permits such placement, but with source checks enabled
|
||||
its root `version` would be the specification header, not prompt metadata. Recommend
|
||||
the local-key mapping above as the standard authoring form: identity travels
|
||||
with the definition and files may be reorganized without changing it. Filename-derived IDs are not
|
||||
needed.
|
||||
|
||||
## Final Schema And Identity
|
||||
|
||||
Use `collections` as a mapping of namespace to collection body, containing
|
||||
optional `release_version` and optional `backends`, `profiles`, and `prompts`
|
||||
mappings. Collection release labels are descriptive, with no SemVer ordering
|
||||
or dependency-solving semantics. Metadata is optional: an installed collection
|
||||
can contain resources without claiming a published release.
|
||||
|
||||
Each resource key inside a collection is local. The parent namespace and local
|
||||
key follow the accepted slash-free component grammar; their public spelling is
|
||||
exactly `namespace/local-name`. There are no duplicate embedded `id` or
|
||||
`namespace` fields to reconcile. Dots remain ordinary characters.
|
||||
|
||||
Retain unqualified application resources through optional root-level
|
||||
`backends`, `profiles`, and `prompts` mappings, also keyed by local name and
|
||||
using the same definition shapes. They produce unqualified public IDs. Under
|
||||
this proposal, namespaced definitions belong under `collections`, not a second
|
||||
qualified-key spelling in those root maps. This avoids two configuration paths
|
||||
producing the same public identity. This representation belongs to the selected
|
||||
future direction; it does not change today's implemented format.
|
||||
|
||||
Stored references such as `default_profile`, profile `backend`, and smoke-test
|
||||
profile defaults remain exact public references. Nesting does not automatically
|
||||
qualify a slashless reference or rebind it to caller scope. For example, a
|
||||
profile in `provider` still declares `backend: provider/default`. Caller-scoped
|
||||
lookup follows [the scope rules above](#accepted-consumer-scope) after final
|
||||
indexes are built.
|
||||
|
||||
### One Installed Prompt And Optional Version
|
||||
|
||||
The resource's qualified ID is its sole configuration identity. There is no
|
||||
version-key mapping, implicit `latest`, version ordering, or parallel installed
|
||||
history. Updating the optional version changes metadata on the same prompt.
|
||||
|
||||
An optional prompt version is an opaque string with no SemVer, nonblank, or
|
||||
normalization requirement. Omission or explicit null means no declared label;
|
||||
null can clear a lower-layer or inherited label. Quote numeric-looking labels
|
||||
such as `"1"`. Preserve supplied strings, including empty strings, and distinguish
|
||||
an absent label in inspection. Do not synthesize a version.
|
||||
|
||||
Remove request-side prompt-version fields, version selection, ordering, and
|
||||
version defaults as part of the breaking change. Do not reinterpret those fields
|
||||
as exact-match assertions. Execution and inspection resolve the installed prompt
|
||||
by ID alone. Listings expose one prompt per ID with its optional descriptive
|
||||
label separately. A label neither controls execution nor proves unchanged content.
|
||||
|
||||
## ConfigKit Composition Boundary
|
||||
|
||||
Use the implemented upstream contracts without duplicating them in PromptKit.
|
||||
Input destinations use literal mapping-key segments, independent of filesystem
|
||||
roots and precedence. Relative import destinations are sequences of strings;
|
||||
omission and `[]` retain the importing destination. Null, scalar, and non-string
|
||||
destination members are invalid. No logical path segment means parent traversal
|
||||
or a reset to the application root.
|
||||
|
||||
Source-root import directives remain structural. Imported content contributes at
|
||||
its resulting destination, while physical import paths and `content_file` bases
|
||||
remain relative to their declaring sources. `SourceInfo.Destination` records
|
||||
placement; normal origins and diagnostics retain real source locations. Selected
|
||||
overlays and explicit CLI overrides remain rooted at the application root.
|
||||
|
||||
Mounting preserves source identity, cycle/repetition rules, and load limits.
|
||||
An absent `NoSource` contributes no payload; an actual empty mapping can
|
||||
materialize its destination containers without clearing lower contents.
|
||||
ConfigKit's reserved structural keys continue to apply. PromptKit must not
|
||||
strip headers, re-key source trees, merge fragments itself, or turn source
|
||||
inspection into an additional enforcement pass.
|
||||
|
||||
### Composition And Overrides
|
||||
|
||||
Different collection keys prevent accidental collisions between equal local
|
||||
resource names. Two sources targeting the same namespace are not automatically
|
||||
invalid: disjoint mapping contributions combine. Overlapping scalar fields,
|
||||
message sequences, nulls, and equal release labels conflict within a strict
|
||||
group. Across deliberate layers they follow ordinary precedence instead.
|
||||
|
||||
A higher-layer prompt mapping is a partial override. Changing only its version
|
||||
does not replace its messages or other settings. Omitting a field retains its
|
||||
lower value, and an empty mapping does not clear a definition. Publishers and
|
||||
operators must treat such changes as configuration customization, not as an
|
||||
atomic installation of another prompt release. Do not introduce source audits
|
||||
or a version-triggered replacement rule to conceal this distinction.
|
||||
|
||||
### Inheritance
|
||||
|
||||
Existing sibling inheritance remains useful within each collection's resource
|
||||
mapping: `extends: brief` can name the local profile `brief`. Prompt versions,
|
||||
now ordinary optional fields, would inherit and overlay like other fields;
|
||||
`version: null` explicitly clears an inherited label.
|
||||
|
||||
Cross-collection profiles are no longer immediate siblings. Consequently,
|
||||
`extends: provider/brief` cannot reach that collection under current ConfigKit
|
||||
inheritance. Use existing local sibling inheritance for this development round;
|
||||
applications can still select the qualified `provider/brief` profile directly.
|
||||
Authors needing a customized derivative may publish an expanded definition.
|
||||
Expansion is a migration option, not a mandatory authoring workflow.
|
||||
|
||||
Generic cross-path inheritance remains possible future ConfigKit work if a
|
||||
concrete consumer need justifies it. It is not a prerequisite of this round.
|
||||
PromptKit must not add its own inheritance resolver after ConfigKit handoff.
|
||||
|
||||
## Specification Versions And Collection Inventory
|
||||
|
||||
### Shared Specification Policy
|
||||
|
||||
Use one `Definition.Version` policy, with source-local path `version`, for the
|
||||
candidate PromptKit specification. Initially accept specification text `1` from
|
||||
string or strict decimal-integer declarations; reject explicit null and other
|
||||
unsupported declarations through the ConfigKit checker. Accept an absent main
|
||||
declaration, including an absent main input. Accept additional versions only
|
||||
when PromptKit actually supports their format semantics: accepting
|
||||
a header does not migrate its payload or independently validate a fragment's
|
||||
complete schema. The retired proposals were alternatives, not released formats
|
||||
sharing specification version 1.
|
||||
|
||||
Use `SourceVersionModeIfPresent` without adding a PromptKit mode-selection
|
||||
option in this round. A stricter option can be considered for a demonstrated
|
||||
consumer need. The upstream modes remain distinct:
|
||||
|
||||
| Mode | Implemented behavior | Role in this proposal |
|
||||
| --- | --- | --- |
|
||||
| `Default` | Existing main-only effective-path check and protected-path behavior | Not the proposed shared-source policy |
|
||||
| `IfPresent` | Checks present declarations; skips absent non-main declarations | Initial choice for compact fragments |
|
||||
| `Required` | Requires declarations on every participating document | Upstream capability; exposing a PromptKit option is deferred |
|
||||
|
||||
Participating documents include actual main and underlay inputs, transitive and
|
||||
glob imports, and the selected overlay. Defaults, synthetic CLI overrides,
|
||||
resolution output, absent optional imports, and empty groups do not need headers.
|
||||
Explicit null is present and reaches the checker; malformed selected values
|
||||
fail ConfigKit's version-type validation. A rejected declaration fails loading
|
||||
even if a higher layer would supersede that source's payload.
|
||||
|
||||
In enabled modes, accepted non-main headers are consumed before composition.
|
||||
The main declaration remains and is protected at its destination-prefixed path.
|
||||
PromptKit's recommended main is unmounted, so that protected path is `/version`.
|
||||
Do not confuse header consumption with permission to override the remaining
|
||||
protected application field.
|
||||
|
||||
`IfPresent` does not require a collection entry header and does not infer a
|
||||
fragment's compatibility from its entry. A publisher can declare compatibility
|
||||
in the entry while leaving imported files headerless; the assembled configuration
|
||||
still undergoes normal validation. A declaration is not a guarantee of future
|
||||
compatibility or independent schema validation of every imported fragment.
|
||||
ConfigKit checks the declarations actually present. Requiring headers only on
|
||||
entry files is not supported by the load-wide mode. Do not emulate it using a
|
||||
preliminary manifest load or validation of
|
||||
source history. If that stronger guarantee becomes essential, seek an explicit
|
||||
upstream contract instead.
|
||||
|
||||
### Naming And Optional Main Inputs
|
||||
|
||||
Reserve source-root `version` for specification declarations throughout the new
|
||||
format. Use `release_version` for optional collection release labels and retain
|
||||
nested prompt `version` for the optional installed-prompt label. An old release
|
||||
file containing only `version: "2.4.0"` can fail the specification check; if its
|
||||
text happens to be accepted, that field is consumed and lost as release metadata.
|
||||
Migration must rename it rather than simply enable the new mode.
|
||||
|
||||
This reservation affects resource fragments whose local key is literally
|
||||
`version`: a root mapping with that name is interpreted as a malformed header.
|
||||
Allow that resource name through a nested resource mapping, for example
|
||||
`prompts: {version: ...}` in a collection entry. Reserve the header position,
|
||||
not the public resource name. Existing `configkit` and `extends` structural
|
||||
reservations also remain applicable; destinations do not bypass them.
|
||||
|
||||
An absent header in an actual main document still reaches the checker under
|
||||
`IfPresent`; `NoSource` invokes the absent-main checker as well. PromptKit's
|
||||
checker accepts omission in both cases. Consumers may assemble only underlays,
|
||||
fragments, or programmatic inputs without manufacturing a main document or
|
||||
header. Do not inject a declaration or report undeclared sources as checked.
|
||||
Present unsupported declarations still fail; final schema validation still runs.
|
||||
|
||||
### Inspection And Effective Collection Inventory
|
||||
|
||||
`SourceInfo.VersionCheck` is source evidence, not a generic success flag. A
|
||||
non-nil record with `Present == false` can describe an omission for which the
|
||||
checker did not run. Header paths and locations are source-local. Evidence is
|
||||
detached and does not reread inputs; text is conservatively redacted during
|
||||
callbacks and on failed loads, and successful final records respect completed
|
||||
sensitivity classification. These records belong in explicit configuration
|
||||
inspection, not in ordinary domain validation or collection-release identity.
|
||||
|
||||
Derive collection inventory from the final `collections` mapping: namespace
|
||||
plus optional effective `release_version`, sorted and detached from the captured
|
||||
engine. Metadata-only and resource-only entries can both appear. Root unqualified
|
||||
resources do not invent a named collection; resource-based namespace enumeration
|
||||
remains separate. The companion implementation plan defines exported inventory
|
||||
types.
|
||||
|
||||
An inventory describes effective configuration, not immutable package identity,
|
||||
historical source ownership, or proof of unchanged published contents. Operator
|
||||
overrides may change resources while retaining a release label. Engine
|
||||
construction captures definitions; edits require a replacement engine.
|
||||
|
||||
## Migration
|
||||
|
||||
Migration must choose one installed definition for each former multi-version
|
||||
prompt, move any retained version into its optional field, and relocate
|
||||
qualified resource keys into collection/local-key mappings. Update destination
|
||||
declarations, stored references, inventories, and configuration-inspection paths.
|
||||
Rename collection release metadata to
|
||||
`release_version`, configure the shared checker and mode, and account for the
|
||||
source-root header reservation and optional main declarations. Remove obsolete
|
||||
request-version fields and version-selection logic from consumer examples and
|
||||
public APIs rather than retaining them as assertions. Preserve provider model
|
||||
strings and source resource bases. Namespace mounting does not
|
||||
automatically relocate schema or artifact stores or rewrite references when a
|
||||
collection is renamed.
|
||||
|
||||
Retain the accepted removal of mandatory catalog modules and built-in profiles.
|
||||
This was a breaking configuration/API change under the
|
||||
[release policy](../policy/release.md); the implementation includes consumer
|
||||
migration guidance and updates current contracts. The completed work changes
|
||||
the supported configuration and API as described here.
|
||||
|
||||
Existing unscoped exact lookup remains available for IDs valid under the new
|
||||
grammar. Compatibility with arbitrary slash-containing resource IDs is
|
||||
intentionally broken. The migration must:
|
||||
|
||||
- Audit maintained catalogs and representative consumer definitions for invalid
|
||||
IDs and for valid one-slash IDs whose namespace meaning needs review.
|
||||
- Choose explicit old-to-new mappings and check for collisions. Do not
|
||||
automatically replace slashes with dots: distinct existing IDs could collapse
|
||||
to the same new identity.
|
||||
- Update renamed keys and their references together, including inheritance,
|
||||
prompt default profiles, profile backend references, backend smoke-test
|
||||
defaults, and affected examples and consumer request IDs.
|
||||
- Keep provider model strings unchanged. Renaming a profile does not rename
|
||||
the upstream model.
|
||||
- Document consumer action and coordinate the PromptKit format change with
|
||||
catalog adoption under the [release policy](../policy/release.md). Its catalog
|
||||
profile-data exception does not cover this general format/API behavior change;
|
||||
deliberately retire its built-in-profile preservation promise when the
|
||||
accepted removal landed. Current policy and format documents now reflect the
|
||||
implemented behavior.
|
||||
|
||||
Renaming a backend changes its identity; duplicating it under another ID can
|
||||
separate capacity accounting. Do not use blanket aliases or duplicate catalog
|
||||
entries as an automatic compatibility mechanism. Exact release versions and
|
||||
catalog mappings remain implementation-planning work.
|
||||
|
||||
## Work Deferred Or Outside This Round
|
||||
|
||||
- Collection installation, automatic downloading, implicit directory discovery,
|
||||
package registries, dependency solving, and live reload.
|
||||
- Automatic rewriting of stored references when mounting or renaming collections.
|
||||
- Implicit namespace search paths, fallback lookup, namespace-based access control,
|
||||
and filesystem isolation.
|
||||
- Changes to provider model names, inference behavior, or ConfigKit precedence.
|
||||
- Sequence-item resource catalogs, identity-based overlays, cross-path inheritance,
|
||||
and a PromptKit option requiring specification headers in every document.
|
||||
|
||||
Explicit source destinations, direct fragments, programmatic sources, and
|
||||
consumer-selected precedence remain supported in the design. Deferred features
|
||||
can be reconsidered for concrete needs; none blocks implementation.
|
||||
|
||||
## Target Acceptance
|
||||
|
||||
The completed target includes the collection schema and final configuration
|
||||
adapter, catalog and built-in removal, ID-only prompt selection, immutable
|
||||
scoped views, integrated filtering, and collection inventory. Runtime indexes
|
||||
use full identities after ConfigKit handoff. Current contracts and consumer
|
||||
migration guidance describe the implemented behavior. The
|
||||
[implementation plan](implementation.md) owns sequencing, concrete public shapes,
|
||||
documentation tasks, and focused offline validation under the
|
||||
[testing policy](../policy/testing.md).
|
||||
|
||||
### Namespace And Consumer Acceptance
|
||||
|
||||
- Two prompt scopes resolve the same local name to their respective prompts;
|
||||
a missing name never resolves from the other scope or an unqualified entry.
|
||||
- Two provider profile scopes resolve the same local name, such as
|
||||
`google.gemma-4-31b-it`, to different profiles without changing the configured
|
||||
provider model value `google/gemma-4-31b-it`.
|
||||
- A qualified reference bypasses the selected scope without double qualification;
|
||||
adding another collection does not change local or qualified resolution.
|
||||
- Multiple slashes and empty qualified components are rejected in resource IDs
|
||||
and references, while provider model values retain their existing syntax.
|
||||
- Prompt shorthand and a profile override use their intended scopes, while
|
||||
stored default references and omitted selections retain their meaning.
|
||||
- Scoped views coexist concurrently and share backend capacity with ordinary
|
||||
calls on the underlying engine.
|
||||
- Scope creation rejects invalid syntax, accepts well-formed absent namespaces,
|
||||
and treats empty scope fields as no scope. Missing resources fail at lookup;
|
||||
qualified references remain usable even when the default namespace is absent.
|
||||
- Listing without an explicit filter uses the resource's view scope, or lists
|
||||
all resources when no scope is selected. Explicit named, unqualified-only,
|
||||
and all-namespaces filters override that default for the current listing only.
|
||||
- A valid namespace with no matches produces an empty listing. Namespace
|
||||
filtering respects resource type and exact namespace boundaries.
|
||||
- Exact-ID calls using valid IDs and intentional overlays retain their behavior.
|
||||
Migrated catalog references resolve without collisions; migration installs one
|
||||
prompt per ID with an optional descriptive label.
|
||||
- Listings preserve full IDs and separate optional prompt version labels.
|
||||
Qualified results round-trip through either engine or view; unqualified results
|
||||
round-trip through the engine without a corresponding scope.
|
||||
- Representative validation, inspection, execution, and smoke-test paths use
|
||||
the same qualification rules without duplicating every test at every layer.
|
||||
- Engines use caller-selected definitions without automatically contributing
|
||||
provider catalogs or PromptKit-owned profiles. Ordinary directory and embedded
|
||||
configuration sources retain ConfigKit composition semantics.
|
||||
- PromptKit does not add source-history-based duplicate-namespace rejection or
|
||||
independently pre-validate collection definitions before the final handoff.
|
||||
|
||||
### Configuration Acceptance
|
||||
|
||||
Exercise typed assembly of multiple collections, relative/nested imports,
|
||||
source-local header checking and consumption, ordinary release and prompt
|
||||
metadata, unqualified resources, optional prompt labels, inventory without
|
||||
release metadata, source-relative content, and intentional partial overrides.
|
||||
Cover headerless entries/fragments, omitted main inputs and headers, and
|
||||
resources named `version` in nested mappings. Verify ID-only lookup without
|
||||
request-version selection or assertions. No automatic provider catalogs or
|
||||
built-in profiles should be contributed.
|
||||
|
||||
Do not duplicate ConfigKit's generic source validation or enforce domain rules
|
||||
through historical source inspection. Preserve local inheritance and ordinary
|
||||
mapping overrides while migrating catalog definitions.
|
||||
|
||||
### Review Evidence
|
||||
|
||||
The 2026-10-16 review against ConfigKit development commit `e66c371` verified
|
||||
mounted assembly, relative imports and resource contexts, shared header checks,
|
||||
and this document's YAML assembly through public consumer tests. The 2026-10-17
|
||||
review of release v0.8.0 at `137769f` passed maintained consumer examples and
|
||||
focused public API probes for sequence assembly, duplicate rejection, whole-list
|
||||
replacement, inheritance limits, header consumption, and final identity validation
|
||||
boundaries. The latter review informed the decision to retain mappings. These
|
||||
checks establish the upstream behavior relied on by the implemented PromptKit
|
||||
feature.
|
||||
|
||||
## Delivery Reference
|
||||
|
||||
No substantive design question remains. ConfigKit v0.8.0 is the selected
|
||||
dependency. The [implementation plan](implementation.md) records helper
|
||||
signatures, inventory and optional-label representations, identifier migration
|
||||
mappings, and completed consumer migration work. Its delivery choices implement
|
||||
these accepted contracts without adding speculative options or reopening
|
||||
workflows.
|
||||
|
||||
## Consolidation And Retired Alternatives
|
||||
|
||||
This document replaces the separate namespace roadmap and collection proposal 3.
|
||||
Their pre-consolidation versions, including considered namespace API alternatives,
|
||||
remain available at PromptKit commit `ddda2f8`:
|
||||
|
||||
```sh
|
||||
git show ddda2f8:docs/roadmap/namespaces.md
|
||||
git show ddda2f8:docs/roadmap/configuration-collections-mounted-fragments.md
|
||||
```
|
||||
|
||||
The unselected collection proposals remain available at `b5866dd`:
|
||||
|
||||
```sh
|
||||
git show b5866dd:docs/roadmap/configuration-collections-current-configkit.md
|
||||
git show b5866dd:docs/roadmap/configuration-collections-identified-records.md
|
||||
```
|
||||
|
||||
Proposal 1 used fully qualified mapping keys. Proposal 2 requested identified
|
||||
record sequences, identity-based overlays, and record inheritance. ConfigKit
|
||||
v0.8.0 implements a narrower sequence-composition capability; it does not supply
|
||||
those identity-based overlays or inheritance. Its sequence feature was evaluated
|
||||
and not selected for this roadmap. The original upstream feature-request lineage
|
||||
is recorded in ConfigKit history under
|
||||
`docs/roadmap/promptkit-identified-records-feature-request.md`; no outstanding
|
||||
part of that request is a prerequisite here.
|
||||
|
||||
The accepted configuration choices remain: optional main declarations and
|
||||
if-present checks; source-root `version` reserved without banning that resource
|
||||
name; removal of request-version machinery; current ConfigKit inheritance; and
|
||||
flexible source assembly with a small convenience API. This consolidation
|
||||
records those decisions and their implementation scope; current behavior is
|
||||
maintained in the canonical format and consumer documents.
|
||||
@@ -1,852 +0,0 @@
|
||||
# Configuration Collections Implementation Plan
|
||||
|
||||
Status: implementation complete; roadmap retirement remains for normal release
|
||||
preparation.
|
||||
|
||||
Implement the accepted [feature roadmap](configuration-collections.md). That
|
||||
document owns intent, scope, and selected policy; this document owns API delivery
|
||||
choices and the ordered implementation steps. Each numbered stage is sized as
|
||||
one prompt for a Luna coding agent. Complete stages in order. Foundation and
|
||||
migration stages are parts of one refactor, not separately publishable releases.
|
||||
|
||||
## Baseline And Working Rules
|
||||
|
||||
Reviewed PromptKit baseline: `683689a4f89d0fea6a8e0c6faf5830414adf9c7b`.
|
||||
It selects ConfigKit `v0.7.1`, both `/v3` provider catalogs at
|
||||
`v3.20261013.0`, and the completed smoke-test implementation and remediations.
|
||||
The existing implementation plan described that completed work; Git retains it.
|
||||
This plan replaces it without reopening the smoke-test design.
|
||||
|
||||
ConfigKit `v0.8.0` resolves to commit
|
||||
`137769f3f18a370c68444242fb5d88e620e307e5` (the annotated tag object is distinct).
|
||||
Its released public contracts in `docs/config.md`, release notes, and ADRs
|
||||
0011–0014 supply the required placement and shared version-check behavior.
|
||||
The adjacent checkout is a reference, not a committed build dependency.
|
||||
|
||||
Before each stage, inspect status, preserve unrelated work, read the owning
|
||||
implementation/tests, and follow [development orientation](../development.md).
|
||||
Apply the [agent](../policy/agents.md), [architecture](../policy/architecture.md),
|
||||
[testing](../policy/testing.md), [documentation](../policy/documentation.md),
|
||||
[OSS](../policy/oss.md), and relevant [release](../policy/release.md) policies.
|
||||
Use the repository's code-discovery instructions. Reuse unchanged readings.
|
||||
Update affected GoDoc and canonical contracts as each behavior lands; Stage 17
|
||||
is a consistency pass, not permission to leave current documentation stale.
|
||||
|
||||
Record completion and validation beneath the stage, then commit each completed,
|
||||
nonempty unit with a short plain-English message. These stages change PromptKit;
|
||||
external collection publishing is separate work. Do not push, tag, release, or
|
||||
introduce committed workspace/replacement files. The sprint requires neither
|
||||
publication of a new catalog nor a live provider.
|
||||
Dependency acquisition is setup, separate from offline behavioral validation.
|
||||
|
||||
Maintain [ADR 0001](../adr/0001-configuration-boundary.md): ConfigKit owns source
|
||||
processing, composition, inheritance, and header checking through its configured
|
||||
version callback. PromptKit interprets final effective values and their resource
|
||||
contexts. Do not inspect contributions or source history to enforce domain
|
||||
validity, including in tests. Explicit operator inspection remains available.
|
||||
|
||||
Preserve execution, retry/repair, credentials, appended messages, capacity,
|
||||
observations, cancellation, and prepared-handle semantics except for the accepted
|
||||
identity/version changes. Introduce no installer, implicit discovery, new
|
||||
provider protocol, cross-collection inheritance resolver, compatibility alias
|
||||
bundle, live reload, or additional release gate.
|
||||
|
||||
## Delivery Decisions
|
||||
|
||||
These names and signatures settle delivery details left by the feature roadmap.
|
||||
Private helper names below are implementation guidance, not public invariants.
|
||||
|
||||
### Final Configuration And Runtime Identity
|
||||
|
||||
Use a typed root document with optional `version`, root `backends`, `profiles`,
|
||||
and `prompts`, and `collections`. Each collection body has optional
|
||||
`release_version` and the same three resource mappings. Root and collection
|
||||
resource maps are keyed by local ID; `prompts` maps directly to definitions,
|
||||
not another version map. Keep ordinary strict ConfigKit decoding and existing
|
||||
definition rules. Collection bodies are mappings; an empty body is useful.
|
||||
|
||||
Represent source-root `version`, collection `release_version`, and prompt
|
||||
`version` with `*string` in the typed document. Configure
|
||||
`Definition.Version.Path = configkit.Keys("version")` and
|
||||
`SourceVersionModeIfPresent`. The checker accepts absence, accepts exact text
|
||||
`1` from a string or strict decimal integer, and rejects explicit null or other
|
||||
texts with a safe ConfigKit diagnostic. ConfigKit owns type checking and
|
||||
non-main header consumption. Do not synthesize a main input or declaration.
|
||||
Collection release and prompt labels are opaque strings: nil means absent/null,
|
||||
and a non-nil empty string remains explicitly declared. Do not trim, order,
|
||||
interpret as SemVer, or require these labels.
|
||||
|
||||
Keep the retained ConfigKit result in its authored effective shape. Refactor
|
||||
configuration helpers to receive the effective resource base path rather than
|
||||
reconstructing it from a runtime ID. Visit root and collection mappings with
|
||||
their actual paths; after handoff, index each resource under its full public ID.
|
||||
Do not flatten/re-key the ConfigKit result or make a second load. Presence,
|
||||
normalization records, content-file origins, secret classification, explanations,
|
||||
and diffs must keep the effective paths, including `collections/<namespace>`.
|
||||
|
||||
Create shared private identity helpers under `internal/domain`. Follow the
|
||||
roadmap's exact grammar: a nonblank unqualified ID has no slash; a qualified
|
||||
reference has exactly one slash and two nonempty components. A configured local
|
||||
key is nonblank and slash-free. A collection namespace is nonblank and
|
||||
slash-free, using the same syntax as a nonempty scope. Preserve spelling,
|
||||
including otherwise permitted whitespace, dots, case, and punctuation. Add no
|
||||
other character whitelist. Never apply this grammar to models, paths, URLs,
|
||||
artifact/input names, session IDs, or descriptive labels.
|
||||
|
||||
Validate final configured resource keys and effective PromptKit references.
|
||||
Retain existing existence/readiness timing: for example, profile backend
|
||||
existence is checked as today, but an unselected backend diagnostic default is
|
||||
not made eagerly resolvable. ConfigKit interprets `extends` as a sibling key,
|
||||
not a PromptKit qualified reference. PromptKit does not validate or reinterpret
|
||||
consumed inheritance directives.
|
||||
|
||||
### Prompt Selection And Optional Labels
|
||||
|
||||
Remove `PromptVersion` from public/internal `RunRequest`,
|
||||
`RequirementInspectionRequest`, and `SmokeTestRequest`. `ValidateRequest`,
|
||||
`Prepare`, and `PrepareExecution` continue to use `RunRequest`.
|
||||
Change `InspectPrompt` to:
|
||||
|
||||
```go
|
||||
func (e *Engine) InspectPrompt(ctx context.Context, promptID string) (*PromptInspection, error)
|
||||
```
|
||||
|
||||
All prompt repository, compiled-template, and requirement-snapshot lookups use
|
||||
ID alone. Remove ambiguity tracking, version selectors, version ordering, and
|
||||
request assertions. Keep `PromptIdentity.ID`; change `PromptIdentity.Version`
|
||||
to `*string`. Keep output field names `PromptVersion`, changing their types to
|
||||
`*string` wherever they describe the selected label: prepared details, run
|
||||
results, prompt/requirement inspection, execution events, and smoke-test targets.
|
||||
The internal definition uses `Version *string` with equivalent semantics.
|
||||
Each returned mutable pointer is owned by its result, including events and
|
||||
prepared snapshots. JSON `omitempty` omits nil while preserving a declared
|
||||
empty string. Preserve other existing serialization choices.
|
||||
|
||||
The optional label remains part of captured definition metadata and its existing
|
||||
definition-hash projection. Copy the pointer; do not dereference it as a cache
|
||||
or selection key. No cross-release hash-stability promise is introduced.
|
||||
Derive automatic structured-output schema names from the selected prompt ID
|
||||
using the existing sanitization, not from a version-selection token. Keep explicit
|
||||
schema selection and structural validation behavior unchanged.
|
||||
|
||||
### Immutable Scope On The Existing Engine
|
||||
|
||||
```go
|
||||
type Namespaces struct {
|
||||
Prompt string
|
||||
Profile string
|
||||
Backend string
|
||||
}
|
||||
|
||||
func (e *Engine) WithNamespaces(namespaces Namespaces) (*Engine, error)
|
||||
```
|
||||
|
||||
Return another engine facade over the same assembled engine components with
|
||||
immutable scope values. An internal engine view may be a shallow copy containing
|
||||
the same runner, smoke tester, inspection, and inventory collaborators. It does
|
||||
not invoke `NewEngine` or build another capacity manager. `WithNamespaces`
|
||||
replaces all three scopes on the new view; empty fields clear those scopes.
|
||||
The receiver is unchanged. Validate syntax, not namespace existence. Use the
|
||||
existing invalid-configuration convention for an invalid engine or scope.
|
||||
|
||||
At the internal engine entry boundary, use shared qualification helpers for
|
||||
caller-supplied IDs. Validate supplied syntax; leave omitted optional IDs empty;
|
||||
qualify slashless nonempty IDs for the applicable scope; pass qualified IDs
|
||||
unchanged. The runner continues exact lookup. Do not requalify configured default
|
||||
profiles, backend references, sibling inheritance, or diagnostic defaults.
|
||||
|
||||
Apply the same policy to `Run`, `Prepare`, `PrepareExecution`, `ValidateRequest`,
|
||||
`InspectPrompt`, `InspectProfile`, `InspectRequirements`, and all three selection
|
||||
fields of `SmokeTest`. `ValidateRequest` already delegates to preparation; use
|
||||
that route rather than adding another validator. `RunPrepared` consumes its
|
||||
captured selection without qualification. Preserve its existing runner-ownership
|
||||
check; scope does not introduce another handle owner. `InspectConfiguration`
|
||||
continues to expose the complete retained configuration, not a filtered/re-keyed
|
||||
view. Results and observer metadata carry full selected identities.
|
||||
|
||||
### Listing Filters And Inventory
|
||||
|
||||
Use one concrete optional filter argument for profile, prompt, and new backend
|
||||
listings:
|
||||
|
||||
```go
|
||||
type NamespaceFilterMode uint8
|
||||
|
||||
const (
|
||||
NamespaceDefault NamespaceFilterMode = iota
|
||||
NamespaceNamed
|
||||
NamespaceUnqualified
|
||||
NamespaceAll
|
||||
)
|
||||
|
||||
type NamespaceFilter struct {
|
||||
Mode NamespaceFilterMode
|
||||
Name string // Used only by NamespaceNamed.
|
||||
}
|
||||
|
||||
func (e *Engine) ListProfiles(ctx context.Context, filter ...NamespaceFilter) ([]string, error)
|
||||
func (e *Engine) ListPrompts(ctx context.Context, filter ...NamespaceFilter) ([]PromptIdentity, error)
|
||||
func (e *Engine) ListBackends(ctx context.Context, filter ...NamespaceFilter) ([]string, error)
|
||||
```
|
||||
|
||||
Zero or one filter is accepted; multiple filters or an unknown mode return the
|
||||
ordinary invalid-request error rather than guessing precedence. Omission and a
|
||||
zero filter mean default: use that resource's scope, or all resources when it is
|
||||
unset. Named requires a nonempty valid namespace. Unqualified selects IDs without
|
||||
a slash; All includes every ID. A non-Named mode ignores `Name`. An explicit
|
||||
filter replaces, not intersects, the view's default for that call. Keep IDs
|
||||
full, sort lexically by full ID, and retain existing context, ownership, and
|
||||
empty-list behavior. Copy optional labels on returned prompt entries.
|
||||
Qualified results can be reused through any view. To select an unqualified ID
|
||||
returned by an explicit All/Unqualified filter, use the original engine or a view
|
||||
without that resource scope; a listing filter never changes execution scope.
|
||||
|
||||
Add the following small discovery surfaces:
|
||||
|
||||
```go
|
||||
type ResourceKind string
|
||||
|
||||
const (
|
||||
ResourcePrompt ResourceKind = "prompt"
|
||||
ResourceProfile ResourceKind = "profile"
|
||||
ResourceBackend ResourceKind = "backend"
|
||||
)
|
||||
|
||||
func (e *Engine) ListNamespaces(ctx context.Context, kinds ...ResourceKind) ([]string, error)
|
||||
|
||||
type CollectionInfo struct {
|
||||
Namespace string
|
||||
ReleaseVersion *string
|
||||
}
|
||||
|
||||
func (e *Engine) ListCollections(ctx context.Context) ([]CollectionInfo, error)
|
||||
```
|
||||
|
||||
No kinds means the union of all three resource types. Supplied kinds select their
|
||||
union; repeats are harmless and unknown kinds are invalid requests. Namespace
|
||||
enumeration describes resource membership across the captured engine, independent
|
||||
of view scopes. Return distinct named namespaces sorted lexically; exclude the
|
||||
unqualified group. Collection inventory likewise covers the entire captured
|
||||
`collections` mapping, sorted by namespace, including metadata-only and
|
||||
resource-only entries. It is not filtered by a view and does not include an
|
||||
invented root collection. Both surfaces use the existing listing-style nil-engine
|
||||
and context error conventions and return detached values. They do not inspect
|
||||
historical sources, reopen files, or assert package authenticity.
|
||||
|
||||
Capture sorted inventories once at construction. Namespace buckets or sorted
|
||||
ranges may avoid repeated full scans for filtered listings. Prefer a single
|
||||
resource traversal plus sorting and ordinary indexed lookups; no repeated
|
||||
ConfigKit queries, definition deep copies for identity-only listings, global
|
||||
caches, or performance/complexity promises in the public API are needed.
|
||||
|
||||
### Collection Input Convenience
|
||||
|
||||
```go
|
||||
func CollectionInput(namespace string, source configkit.Source) (configkit.Input, error)
|
||||
```
|
||||
|
||||
Validate the required namespace syntax and return an input with that source,
|
||||
`Destination: configkit.Keys("collections", namespace)`, and `Imports: true`.
|
||||
Invalid namespace syntax returns an error matching `ErrInvalidConfig`.
|
||||
Do not open/read the source or infer a namespace/filename. ConfigKit still owns
|
||||
source validity and loading errors. The caller may change the returned input's
|
||||
imports, filesystem context, or destination, or construct an input directly.
|
||||
This supports host, embedded, and mapping sources with one helper, without
|
||||
introducing another loader or options framework. Directory/Filesystem helpers
|
||||
retain their existing explicit discovery and imports-disabled semantics.
|
||||
`ProfileSource` and `BackendSource` retain their root-document shapes and can be
|
||||
mounted with this helper. Direct local-key fragments use explicit ConfigKit
|
||||
destinations that include the resource category.
|
||||
|
||||
### Migration Choices
|
||||
|
||||
Do not mass-copy the provider inventories into PromptKit or create a mandatory
|
||||
replacement bundle. Provide synthetic, complete collection examples and guidance
|
||||
for adapting selected existing catalog configuration. Adjacent repositories and
|
||||
their publication are outside this sprint. The reviewed pinned catalog assets
|
||||
have slash-free profile keys, so these explicit mappings are collision-free
|
||||
within those assets:
|
||||
|
||||
| Previous identity | Documented migration identity |
|
||||
| --- | --- |
|
||||
| Backend `openrouter` | `openrouter/default` |
|
||||
| Backend `rakestrawhome` | `rakestrawhome/default` |
|
||||
| Each OpenRouter catalog profile `p` | `openrouter/p`, retaining the exact local key `p` |
|
||||
| Profile `rakestrawhome-gemma-4-31b` | `rakestrawhome/rakestrawhome-gemma-4-31b` |
|
||||
| Built-in `promptkit-openrouter-standard` | No automatic replacement; optional caller profile `openrouter/standard` |
|
||||
| Built-in `promptkit-openrouter-auto` | No automatic replacement; optional caller profile `openrouter/auto` |
|
||||
|
||||
These are documented conversions, not installed aliases. Simply mounting old
|
||||
`/v3` assets does not update their stored backend/profile references; those
|
||||
references and diagnostic defaults need the explicit edits shown here.
|
||||
|
||||
For an optional caller standard profile, `extends: gemini-flash-latest` is valid
|
||||
only within the same `openrouter` profiles mapping containing that base. The
|
||||
caller auto profile may explicitly set backend `openrouter/default` and model
|
||||
`openrouter/auto`. Neither name becomes a PromptKit promise. Move backend
|
||||
diagnostic defaults to exact new profile IDs. Preserve endpoint/settings and
|
||||
provider model strings, including the pinned Gemini model's leading `~`.
|
||||
|
||||
Application-specific IDs require a deliberate collision review, not automatic
|
||||
slash-to-dot replacement. For a formerly multi-version prompt, its maintainer
|
||||
chooses the installed definition; examples use the single definition already
|
||||
present. Put its old version string into optional metadata where useful. Do not
|
||||
select a winner from a consumer's multi-version history by guessed ordering.
|
||||
Document exact references, paths, backend capacity identity changes, source-root
|
||||
header reservations, label null/empty semantics, and the loss of cross-mapping
|
||||
inheritance. Migration documentation is not a runtime migration tool.
|
||||
|
||||
## Stage 1: Adopt The Released ConfigKit Prerequisite
|
||||
|
||||
Scope: dependency selection and existing integration compatibility only.
|
||||
|
||||
1. Select ConfigKit `v0.8.0` with normal Go module tooling; update `go.mod` and
|
||||
`go.sum` without filesystem replacements. Verify the selected release commit
|
||||
and notices; update its attribution entry with the full commit above.
|
||||
2. Check affected keyed literals and source-inspection handling for upstream
|
||||
additions. `SourceInfo` is no longer comparable; compare relevant fields if
|
||||
PromptKit currently relies on comparability. Do not enable new schema behavior.
|
||||
3. Leave Go's consumer minimum at `1.26.0` unless the selected dependency actually
|
||||
requires a higher minimum. Leave unrelated dependencies/tooling unchanged.
|
||||
|
||||
Validation: `GOWORK=off go test ./internal/configuration .` and the three offline
|
||||
examples, including configuration inspection/diff coverage. This stage establishes
|
||||
the unchanged PromptKit baseline on the new dependency.
|
||||
|
||||
Completion: ConfigKit `v0.8.0` is selected at commit
|
||||
`137769f3f18a370c68444242fb5d88e620e307e5`; its attribution links and module
|
||||
checksums are updated. The Go consumer minimum remains `1.26.0`. No
|
||||
`SourceInfo` comparability-dependent code required adjustment. Validation passed:
|
||||
`GOWORK=off go test ./internal/configuration .` and
|
||||
`GOWORK=off python3 scripts/check-examples.py`.
|
||||
|
||||
## Stage 2: Add Shared Resource Naming Primitives
|
||||
|
||||
Scope: small private domain helpers and focused behavior tests.
|
||||
|
||||
1. Implement exact ID parsing/validation, namespace syntax validation, local-key
|
||||
validation, qualification, and namespace extraction as needed by later stages.
|
||||
Keep omission policy in operation callers, not a universal required-ID rule.
|
||||
2. Cover unqualified/dotted names, qualified bypass, missing/empty scope,
|
||||
malformed slash forms, and preserved spelling. Check that namespace matching
|
||||
is exact rather than a prefix match.
|
||||
3. Do not wire names into all public operations yet or export a splitting API.
|
||||
|
||||
Validation: `GOWORK=off go test ./internal/domain`. Test selected naming behavior,
|
||||
not a helper call sequence or invented character restrictions.
|
||||
|
||||
Completion: private resource-ID, namespace, local-key, qualification, and
|
||||
namespace-extraction helpers are implemented with focused grammar and spelling
|
||||
tests. Validation passed: `GOWORK=off go test ./internal/domain` and
|
||||
`git diff --check`.
|
||||
|
||||
## Stage 3: Make Configuration Processing Path-Aware
|
||||
|
||||
Scope: preparation refactor inside `internal/configuration`; current format stays.
|
||||
|
||||
1. Refactor `captureFields`, `resolveBackends`, `resolveProfiles`,
|
||||
`validateEffective`, prompt metadata capture, and prompt materialization to
|
||||
accept/use explicit effective base paths. Preserve current values and errors.
|
||||
2. Carry a resource's public ID separately from its ConfigKit path. Stop deriving
|
||||
an effective path by splitting a runtime ID. Use ConfigKit path segments for
|
||||
literal keys and preserve inherited `content_file` origins.
|
||||
3. Factor resource-map traversal only as needed to reuse the same processing
|
||||
for root and collection mappings later. Do not add a generic visitor framework.
|
||||
4. Prepare the existing provider-secret predicate to recognize actual resource
|
||||
paths without mistaking arbitrary nested `extra_params` data for resources.
|
||||
Keep its existing protected-key policy.
|
||||
|
||||
Validation: configuration load, target, prompt-origin, inheritance, and inspection
|
||||
tests; `GOWORK=off go test ./internal/configuration .`. A behavior-preserving
|
||||
refactor should not rewrite unrelated expectations.
|
||||
|
||||
Completion: configuration capture, normalization, effective validation, and
|
||||
prompt materialization now receive explicit ConfigKit base paths. Existing
|
||||
resource IDs remain independent from those paths; source-origin reads and the
|
||||
protected provider-key policy are preserved. Validation passed:
|
||||
`GOWORK=off go test ./internal/configuration .`.
|
||||
|
||||
## Stage 4: Install One Prompt Definition Per ID
|
||||
|
||||
Scope: prompt storage format, materialization, and fixture-format migration.
|
||||
|
||||
1. Change root `prompts` to `map[string]promptConfig`; add nullable
|
||||
`Version *string` to the raw prompt definition. Materialize one entry per ID,
|
||||
preserving the label text without normalization or a generated label.
|
||||
2. Remove nested version indexes, ambiguity tracking, and selection-by-version
|
||||
from the configuration adapter. Use the actual prompt path without the old
|
||||
version segment for metadata and message resources.
|
||||
3. To keep this stage bounded, existing repository method parameters and public
|
||||
selector fields may remain only until Stage 5. At this adapter boundary their
|
||||
temporary implementation delegates to ID-only lookup; do not compare them
|
||||
with labels or create an assertion mode. Existing string-valued output labels
|
||||
are converted to presence-aware values in Stage 6. These are migration seams,
|
||||
not new compatibility behavior to retain or publish.
|
||||
4. Mechanically migrate YAML fixtures, mapping builders, and example configuration
|
||||
files to the direct definition shape. Move retained version keys to quoted
|
||||
metadata. Replace tests of parallel versions/ambiguity with one-entry lookup
|
||||
and ordinary mapping-overlay cases. Keep unrelated validation coverage.
|
||||
|
||||
Validation: `GOWORK=off go test ./...`, including consumers of migrated fixtures.
|
||||
Test ID-only selection, source-relative content, and inherited/overlaid prompt
|
||||
settings. Do not add tests preserving temporary obsolete selector parameters.
|
||||
|
||||
Completion: prompts now map directly from each ID to one definition, with an
|
||||
optional descriptive version label. Configuration lookup and message/input
|
||||
metadata paths are ID-only; ambiguity handling and version-keyed indexes are
|
||||
removed. Fixtures, examples, and the canonical format reference use the direct
|
||||
shape, including layered overlays. Validation passed: `GOWORK=off go test ./...`,
|
||||
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
|
||||
|
||||
## Stage 5: Remove Request-Side Version Selection
|
||||
|
||||
Scope: compile-directed removal of obsolete selector plumbing; no new lookup rules.
|
||||
|
||||
1. Remove request `PromptVersion` fields and the version parameter from public
|
||||
`InspectPrompt` and internal prompt repository/snapshot/template interfaces.
|
||||
Update runner preparation, inspection, requirements, and smoke selection.
|
||||
2. Update public-to-internal conversion, redacted formatting, fakes, tests,
|
||||
examples, and call sites. Remove version-without-prompt checks and the
|
||||
transitional signatures from Stage 4. Preserve ordinary not-found identities.
|
||||
3. Use the selected prompt ID for generated structured-schema names; retain the
|
||||
existing sanitization and explicit schema behavior. Do not remove output
|
||||
metadata fields merely because selectors are gone.
|
||||
4. Update affected GoDoc now so obsolete selectors are no longer advertised.
|
||||
|
||||
Validation: `GOWORK=off go test ./...` and focused vet of modified packages.
|
||||
Search declarations/call sites for remaining selection/ambiguity logic; descriptive
|
||||
label references are valid and should not be blanket-deleted.
|
||||
|
||||
Completion: request-side version selectors and version parameters have been
|
||||
removed from prompt lookup, inspection, requirement analysis, and smoke-test
|
||||
selection. Generated structured-schema names now derive from the prompt ID;
|
||||
result-side descriptive label fields remain. Updated the consumer guide and
|
||||
GoDoc to reflect ID-only selection. Validation passed: `GOWORK=off go test
|
||||
./...`, focused `GOWORK=off go vet` for modified packages,
|
||||
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
|
||||
|
||||
## Stage 6: Carry Optional Prompt Labels Through Results
|
||||
|
||||
Scope: presence-aware metadata across internal values and the facade.
|
||||
|
||||
1. Change internal definition `Version` and all selected-label output fields
|
||||
identified above to `*string`. Read the raw optional field captured by Stage 4.
|
||||
Update conversion and ownership copies, hashing projection, requirement
|
||||
snapshots, observation selection/events, prepared details/results, and smoke
|
||||
target projection. Optional labels never become lookup keys.
|
||||
2. Distinguish nil from a declared empty string in inspection/listings and JSON.
|
||||
Both omitted and null mean no label; a higher-layer or inherited null clears
|
||||
the label through ordinary ConfigKit composition.
|
||||
3. Update the example summary types and test expectations. Add focused boundary
|
||||
coverage for absence, null clearing, empty/exact text, and pointer ownership;
|
||||
reuse existing report/prepared/event detachment tests where practical.
|
||||
|
||||
Validation: domain, promptdef, configuration, observation, usecase, and root tests;
|
||||
`GOWORK=off go test ./...` and the offline examples. Verify the existing selected
|
||||
definition hash still agrees between inspection and preparation.
|
||||
|
||||
Completion: prompt labels are presence-aware from configuration capture through
|
||||
inspection, listing, requirements, prepared and completed runs, observation
|
||||
events, and smoke targets. Nil and declared-empty labels remain distinct in JSON;
|
||||
null overlays clear inherited labels. Returned label pointers are independently
|
||||
owned, and inspection/preparation retain matching definition hashes. Validation
|
||||
passed: `GOWORK=off go test ./...`, `GOWORK=off go vet ./...`,
|
||||
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
|
||||
|
||||
## Stage 7: Enable Shared Specification Checking
|
||||
|
||||
Scope: root declaration policy through ConfigKit's released version API.
|
||||
|
||||
1. Add optional root `version` to the typed document. Configure the version rule
|
||||
and `SourceVersionModeIfPresent` with the exact checker specified above.
|
||||
Keep no-main and headerless sources valid.
|
||||
2. Use ConfigKit's header consumption and source evidence without a source
|
||||
pre-parser, validation loop, or PromptKit mode option. Nested prompt labels
|
||||
remain ordinary metadata.
|
||||
3. Cover accepted string/integer `1`, omitted main/header, rejected null or
|
||||
unsupported declarations, consumed non-main headers, and an explicit main
|
||||
declaration retained in inspection. A headerless imported prompt fragment
|
||||
still undergoes ordinary final-value validation.
|
||||
|
||||
Validation: focused real-ConfigKit tests in `internal/configuration` and public
|
||||
construction/inspection tests. Do not replicate upstream's full version-type or
|
||||
import matrix. Run `GOWORK=off go test ./internal/configuration .`.
|
||||
|
||||
Completion: the optional root declaration is checked through ConfigKit's shared
|
||||
source-version API, accepting only exact string or integer `1` while allowing
|
||||
absence and preserving the main declaration in inspection. Accepted underlay
|
||||
declarations are consumed; null and unsupported values produce safe diagnostics,
|
||||
and headerless payloads still receive final-value validation. Validation passed:
|
||||
`GOWORK=off go test ./internal/configuration .`.
|
||||
|
||||
## Stage 8: Assemble And Index Final Collection Resources
|
||||
|
||||
Scope: namespaced schema, final resource interpretation, and runtime indexes.
|
||||
|
||||
1. Add `collections` and collection `release_version`/resource mappings to the
|
||||
typed document. Visit root and collection maps using Stage 3's effective
|
||||
paths. Validate namespaces/local keys; build full IDs only in runtime indexes.
|
||||
2. Include backend/profile snapshots and prompt metadata/templates in that
|
||||
traversal. Keep the retained ConfigKit result structurally unchanged.
|
||||
Preserve the distinction between root unqualified and collection-local keys.
|
||||
Migrate maintained qualified-key fixtures into collection/local-key mappings,
|
||||
updating exact references and inspection paths together. Use ordinary explicit
|
||||
destinations until the convenience helper lands; do not keep a second root
|
||||
spelling for qualified identities.
|
||||
3. Validate effective references with the naming helper and existing existence
|
||||
policy. Local `extends` stays with ConfigKit. Do not reject repeated namespace
|
||||
contributions, require collection releases, or qualify stored references.
|
||||
4. Extend secret-path matching to backend/profile extras beneath collections,
|
||||
preserving the existing protected-key list. Use exact effective paths for
|
||||
field presence, normalization, `content_file`, explanations, and diagnostics.
|
||||
5. Preserve backend-ID capacity policy wiring. The same final backend is one
|
||||
capacity identity regardless of the sources that contributed its settings.
|
||||
|
||||
Validation: focused real-ConfigKit tests for two collections with duplicate local
|
||||
names, root resources, partial overrides, inherited relative content, explicit
|
||||
zero/null settings, and redacted namespaced provider data. Include retained
|
||||
release metadata, sibling inheritance, headerless collection fragments, resources
|
||||
named `version` in nested mappings, and header/declaration distinctions.
|
||||
Run `GOWORK=off go test ./internal/configuration .`.
|
||||
|
||||
Completion: final root and collection resources are interpreted from their
|
||||
effective ConfigKit paths and indexed under full runtime IDs without re-keying
|
||||
the retained result. Collection namespaces and local keys are validated;
|
||||
collection release labels, sibling inheritance, inherited relative content,
|
||||
presence/null handling, and namespaced secret redaction are covered. Capacity
|
||||
and profile snapshots use full backend identities. Validation passed:
|
||||
`GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, and `git diff --check`.
|
||||
|
||||
## Stage 9: Make Application Tests Independent Of Bundled Catalogs
|
||||
|
||||
Scope: owned fixtures and examples that currently assume automatic provider data.
|
||||
|
||||
1. Locate tests/examples using bundled backend/profile IDs, inherited catalog
|
||||
settings, default smoke profiles, or baseline listing counts. Supply the
|
||||
explicit synthetic definitions each behavior needs through existing sources.
|
||||
2. Keep generic test helpers small and local; no full catalog snapshot or shared
|
||||
test fixture framework. HTTP adapter tests may retain provider adapter names
|
||||
but use controlled transports and synthetic configuration.
|
||||
3. Keep catalog-specific assembly tests until Stage 10 removes the integration.
|
||||
Do not preserve provider inventory assertions as application contracts.
|
||||
|
||||
Validation: the affected root, engine, and usecase tests plus examples. Ordinary
|
||||
coverage should now express its own backend, profile, and prompt requirements.
|
||||
|
||||
Completion: ordinary application tests now provide the backend, profile, and
|
||||
smoke metadata they exercise instead of relying on bundled provider entries.
|
||||
The listing and inheritance contracts use synthetic resources; catalog-specific
|
||||
assembly tests remain isolated for the later integration removal. Maintained
|
||||
examples already supplied their own definitions and needed no changes.
|
||||
Validation passed: `GOWORK=off go test ./...`,
|
||||
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
|
||||
|
||||
## Stage 10: Remove Automatic Catalogs And Built-In Profiles
|
||||
|
||||
Scope: engine composition, dependency/notices cleanup, and the retired promise.
|
||||
|
||||
1. Remove `catalog.Underlays()` from `internal/engine.New`; load only caller
|
||||
underlays and optional main. Preserve their order and strict-group semantics.
|
||||
Empty configuration constructs an engine with empty resource inventories.
|
||||
2. Delete `internal/catalog`, its owned profiles, and integration-specific tests
|
||||
after checking references. Do not delete generic file-discovery helpers.
|
||||
3. Remove both catalog module requirements and unused checksums through module
|
||||
tooling. Do not update/publish those repositories or copy their inventories
|
||||
into PromptKit as a replacement.
|
||||
4. Update attribution for actual retained material; remove catalog dependency
|
||||
entries when nothing incorporated remains. Keep a notice if other copied
|
||||
material still requires it. Do not strip a shared license text still in use.
|
||||
5. Retire the release policy's catalog-update exception and literal built-in-name
|
||||
promise. Apply ordinary compatibility rules to PromptKit format/API changes;
|
||||
independently selected configuration is no longer a bundled release inventory.
|
||||
Update architecture/internal inventory and current format documentation for
|
||||
the removed component. Preserve historical release notes.
|
||||
|
||||
Validation: `GOWORK=off go test ./...`; verify empty inventories, explicit former
|
||||
backend names still work when configured, and inference smoke tests retain their
|
||||
selection/capability behavior with caller-supplied metadata. Review `go.mod`,
|
||||
`go.sum`, imports, and notices for residual automatic catalog coupling.
|
||||
|
||||
Completion: engine construction now composes only caller-selected underlays and
|
||||
the optional main input; an empty configuration has empty profile and prompt
|
||||
inventories. Removed the catalog adapter and its tests, provider catalog module
|
||||
requirements and checksums, catalog attribution entries, and the built-in
|
||||
profile compatibility exception. Current architecture, source, format, and
|
||||
consumer documentation now describe caller-supplied definitions; historical
|
||||
release notes remain unchanged. Explicit OpenRouter and Rakestrawhome backend
|
||||
IDs continue to work when configured. Validation passed: `GOWORK=off go test
|
||||
./...`, `GOWORK=off go vet ./...`, `GOWORK=off python3
|
||||
scripts/check-examples.py`, and `git diff --check`.
|
||||
|
||||
## Stage 11: Add The Explicit Collection Input Helper
|
||||
|
||||
Scope: the single public convenience function and its GoDoc.
|
||||
|
||||
1. Implement `CollectionInput` exactly as specified above, reusing namespace
|
||||
validation and existing facade error conventions. It creates an input, not
|
||||
an engine and not a configuration pre-parser.
|
||||
2. Keep raw ConfigKit inputs and existing directory/filesystem/programmatic
|
||||
helpers usable. Do not reinterpret their IDs or introduce root discovery.
|
||||
3. Add one native-file and one embedded/mapping assembled case with entry
|
||||
imports and a later deliberate override. Verify resource contexts survive.
|
||||
|
||||
Validation: source-helper contract tests and focused configuration/root tests.
|
||||
Verify helper construction itself performs no source read using an existing
|
||||
controlled source seam where needed, rather than adding a production interface.
|
||||
|
||||
Completion: added `CollectionInput` with shared namespace validation, a
|
||||
`collections/<namespace>` destination, and imports enabled without reading the
|
||||
source. Contract tests cover invalid namespaces and deferred source loading;
|
||||
native-file and embedded-filesystem imports assemble successfully, preserve
|
||||
their source context, and accept a later override. Validation passed:
|
||||
`GOWORK=off go test ./...`, `GOWORK=off go vet .`, and `git diff --check`.
|
||||
|
||||
## Stage 12: Add Immutable Scopes To Ordinary Execution
|
||||
|
||||
Scope: public `Namespaces`/`WithNamespaces` and shared engine-boundary routing.
|
||||
|
||||
1. Add the public facade and internal immutable view. Replace all scopes on each
|
||||
new view without mutating its receiver. Reuse assembled collaborators.
|
||||
2. Route caller prompt/profile selections in `Run`, `Prepare`, and
|
||||
`PrepareExecution` through shared qualification. `ValidateRequest` follows
|
||||
its current preparation route. Stored defaults and omitted IDs stay untouched.
|
||||
3. Preserve resolved full IDs through prepared details, results, events, errors,
|
||||
and selected backend identity. Leave `RunPrepared` captured and single-claim;
|
||||
do not add a scope mismatch check or rebuild shared state.
|
||||
4. Add public behavior tests for two independent prompt/profile scopes,
|
||||
qualified bypass, no fallback, absent well-formed namespaces, and invalid
|
||||
scope syntax. Assert the receiver retains its original behavior.
|
||||
|
||||
Validation: root/prepared/usecase tests and focused race-enabled scope tests.
|
||||
Use injected generation or a controlled HTTP boundary; never a live endpoint.
|
||||
|
||||
Completion: added immutable `WithNamespaces` views that share the assembled
|
||||
runner and captured configuration. Ordinary `Run`, `Prepare`,
|
||||
`PrepareExecution`, and the preparation-backed `ValidateRequest` now qualify
|
||||
slashless caller prompt/profile IDs while preserving qualified IDs, captured
|
||||
defaults, full result/event identities, and prepared-handle ownership behavior.
|
||||
Tests cover independent views, replacement and clearing, no fallback, absent
|
||||
well-formed namespaces, invalid syntax, and concurrent preparation. Validation
|
||||
passed: `GOWORK=off go test ./...`, focused `GOWORK=off go test -race . -run
|
||||
'TestWithNamespaces' -count=1`, `GOWORK=off go vet ./...`, and `git diff --check`.
|
||||
|
||||
## Stage 13: Apply Scopes To Inspection And Diagnostics
|
||||
|
||||
Scope: complete the existing operation inventory without a second resolver.
|
||||
|
||||
1. Route `InspectPrompt`, `InspectProfile`, and `InspectRequirements` through the
|
||||
same helpers; resolve each request's explicitly supplied prompt/profile fields
|
||||
independently. Keep configured references exact.
|
||||
2. Route `SmokeTest`'s supplied PromptID, ProfileID, and BackendID by their own
|
||||
scopes before ordinary diagnostic selection. Keep its precedence, optional
|
||||
omissions, backend defaults, factual reports, and errors unchanged.
|
||||
3. Keep `InspectConfiguration`/Diff global with effective ConfigKit paths.
|
||||
Do not qualify schema paths, provider models, input identifiers, or overrides.
|
||||
4. Review all public resource-selection entry points for omissions. Validate
|
||||
malformed supplied reference syntax through the existing invalid-argument
|
||||
boundary; retain ordinary not-found behavior for well-formed missing IDs.
|
||||
|
||||
Validation: representative public inspection/requirements/validation/smoke cases
|
||||
prove shared behavior and full identities. Reuse naming-helper tests instead of
|
||||
duplicating the full syntax matrix in every API test. Run relevant root/engine
|
||||
tests and focused race checks.
|
||||
|
||||
Completion: prompt, profile, and requirement inspections now qualify each
|
||||
explicit caller selection independently, and smoke diagnostics qualify supplied
|
||||
prompt, profile, and backend IDs before applying their unchanged selection
|
||||
precedence. Captured references and global configuration inspection remain
|
||||
unchanged. Contract tests verify full returned identities, exact configured
|
||||
prompt defaults and backend references, malformed caller IDs, and ordinary
|
||||
not-found behavior.
|
||||
Validation passed: `GOWORK=off go test ./...`, focused
|
||||
`GOWORK=off go test -race . -run 'TestWithNamespaces' -count=1`,
|
||||
`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`,
|
||||
and `git diff --check`.
|
||||
|
||||
## Stage 14: Add Backend Listing And Namespace Filters
|
||||
|
||||
Scope: the existing identity-inventory path, not the execution runner.
|
||||
|
||||
1. Add the filter declarations and optional arguments to profile/prompt listing;
|
||||
add backend identity listing through configuration, internal engine, and facade.
|
||||
Existing no-filter engine calls still enumerate everything.
|
||||
2. Resolve default/explicit filters once per listing using the relevant view
|
||||
scope. Validate named filters and modes; filter exact namespace membership.
|
||||
Use captured sorted inventories, optionally indexed by namespace.
|
||||
3. Return full IDs and detached optional prompt labels, with existing cancellation
|
||||
and empty-result behavior. Do not materialize definitions for identity listing.
|
||||
|
||||
Validation: extend listing tests with scoped defaults, explicit named/All/
|
||||
Unqualified overrides, no-scope defaults, absent namespaces, same-prefix namespace
|
||||
names, full-ID round trips (and unqualified selection through the original
|
||||
engine), invalid filters, ownership, and canceled contexts. Use a compact matrix
|
||||
across resource types, not three copies of all tests. Run configuration/root tests.
|
||||
|
||||
Completion: prompt and profile listings now accept the documented optional
|
||||
namespace filters, and backend identities are captured and listed through the
|
||||
same path. Default filters use the view's per-resource scope, explicit filters
|
||||
replace it, results retain full IDs and detached prompt labels, and ordinary
|
||||
no-filter calls still return the full inventory. Contract tests cover three
|
||||
resource types, exact namespace boundaries, round trips, invalid filters,
|
||||
ownership, and cancellation. Validation passed: `GOWORK=off go test ./...`,
|
||||
focused `GOWORK=off go test -race . -run
|
||||
'TestScopedIdentityListingsApplyNamespaceFilters|TestListIdentitiesAreOwnedAndSafeForConcurrentUse|TestListIdentitiesPreserveContextErrorsWithoutPartialResults' -count=1`,
|
||||
`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`,
|
||||
and `git diff --check`.
|
||||
|
||||
## Stage 15: Expose Namespace And Effective Collection Inventories
|
||||
|
||||
Scope: the two discovery APIs and construction-time inventory capture.
|
||||
|
||||
1. Capture namespace sets by resource kind from final indexed IDs; implement
|
||||
`ListNamespaces` with the chosen union, kind validation, and global semantics.
|
||||
2. Capture `CollectionInfo` values from final `collections` declarations. Preserve
|
||||
absent/null versus empty/exact release labels, including metadata-only and
|
||||
resource-only collections. Copy label pointers on output.
|
||||
3. Delegate through the internal engine's narrow inventory boundary and root
|
||||
facade. Do not derive collection releases from source header evidence or
|
||||
confuse resource namespace membership with collection declarations.
|
||||
|
||||
Validation: one representative inventory fixture covers cross-kind membership,
|
||||
deduplication/sorting, unqualified exclusion, metadata-only collections,
|
||||
scope-independent enumeration, effective overrides, cancellation, and ownership.
|
||||
Run configuration/root tests; no file access or generation should be needed.
|
||||
|
||||
Completion: added global resource-namespace and effective collection inventory
|
||||
APIs. Namespace results are distinct, sorted, kind-selectable, and exclude
|
||||
unqualified resources; collection results preserve absent/null versus empty
|
||||
release labels and include metadata-only declarations. Inventories remain
|
||||
independent of namespace views and return caller-owned values. Validation passed:
|
||||
`GOWORK=off go test ./...`, focused
|
||||
`GOWORK=off go test -race . -run 'TestResourceAndCollectionInventories' -count=1`,
|
||||
`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`,
|
||||
and `git diff --check`.
|
||||
|
||||
## Stage 16: Maintain Complete Consumer And Migration Examples
|
||||
|
||||
Scope: runnable/synthetic examples and precise migration guidance.
|
||||
|
||||
1. Keep the existing prepare/run/observe examples valid with ID-only selection
|
||||
and optional labels. Extend the preparation example, or its existing input
|
||||
assets, to demonstrate two collections, explicit input selection, a scoped
|
||||
view, filtering, and operator inventory without a provider call. Keep the
|
||||
run/observe examples focused on their distinct existing workflows.
|
||||
2. Add small complete collection entry/fragments under `examples/`, with optional
|
||||
headers/releases, source-relative message content, exact stored references,
|
||||
sibling inheritance, and an intentional later partial override. Use synthetic
|
||||
endpoints/models rather than another maintained provider catalog.
|
||||
3. Add `docs/consumers/configuration-migration.md` explaining the breaking format
|
||||
and signatures, the explicit mapping table above, optional caller standard/auto
|
||||
choices, and deployment selection responsibility. Do not promise arbitrary
|
||||
consumer migrations are automatic or validate upstream provider availability.
|
||||
4. Update semantic example checks only for the changed example outputs. If
|
||||
script logic changes, run its focused checks; do not rewrite release tooling.
|
||||
|
||||
Validation: `GOWORK=off python3 scripts/check-examples.py`. Assemble the maintained
|
||||
collection example through real ConfigKit and PromptKit. Keep copied material's
|
||||
actual attribution obligations aligned if any is introduced.
|
||||
|
||||
Completion: the offline preparation example now explicitly composes `support`
|
||||
and `review` collections, including an imported source-relative message file,
|
||||
sibling profile inheritance, and a later partial profile override. It prepares
|
||||
an explicitly selected prompt/profile through a scoped view and reports global
|
||||
namespace/collection inventories plus a namespace-filtered prompt listing. The
|
||||
synthetic entries demonstrate present and absent source/release metadata; the
|
||||
run and observe examples remain focused on their existing workflows. Added a
|
||||
consumer migration guide with the format/API changes, identity mappings,
|
||||
caller-owned optional profiles, and deployment-selection responsibility.
|
||||
Validation passed: `GOWORK=off go test ./...`, `GOWORK=off go vet ./...`,
|
||||
`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`.
|
||||
|
||||
## Stage 17: Finalize Canonical Contracts And Migration Documentation
|
||||
|
||||
Scope: documentation consistency after the APIs and examples exist.
|
||||
|
||||
1. Update GoDoc for all changed/new public declarations, errors, optional labels,
|
||||
ownership, scopes, and listing behavior. Do not leave obsolete selectors or
|
||||
an implication that a descriptive label controls execution.
|
||||
2. Update `docs/formats.md` for the final schema, specification header policy,
|
||||
source-root reservations, one installed prompt, exact references, local
|
||||
inheritance, and explicit provider configuration. Keep current contracts here,
|
||||
not in the migration guide or roadmap.
|
||||
3. Update `docs/consumers/pkg-promptkit.md`, README quickstart/navigation,
|
||||
`docs/internal/sources.md`, internal inventory, and affected runner/rendering/
|
||||
observation documents. Explain unchanged resource contexts and operator-only
|
||||
provenance use. Remove live claims about automatically bundled inventories.
|
||||
4. Review architecture, release policy/procedure, development reading guidance,
|
||||
and attribution for obsolete catalog assumptions. Revise only affected local
|
||||
statements; do not rewrite upstream policy language unnecessarily or weaken
|
||||
ordinary compatibility promises. Do not revise historical release notes or
|
||||
immutable ADR decisions.
|
||||
|
||||
Validation: review changed Markdown links and example references, check GoDoc
|
||||
against declarations, and run `git diff --check`. Release notes and version/tag
|
||||
selection belong to the later requested release preparation, not this stage.
|
||||
|
||||
Completion: canonical format, consumer, README, internal, and public GoDoc
|
||||
contracts now describe collection identity, source headers and origins,
|
||||
namespace-scoped selection, global inventories, descriptive labels, and
|
||||
caller-supplied provider configuration. Reviewed architecture, release policy
|
||||
and procedure, development guidance, and attribution; no additional local
|
||||
changes were needed, and historical release notes and immutable ADRs remain
|
||||
unchanged. Validation passed: local Markdown link targets and example references
|
||||
were reviewed, affected GoDoc was checked against declarations, and
|
||||
`git diff --check`.
|
||||
|
||||
## Stage 18: Verify Cross-Boundary Behavior And Close The Sprint
|
||||
|
||||
Scope: focused integration gaps and final regression validation, not new features.
|
||||
|
||||
1. Trace the feature roadmap's acceptance criteria to focused existing/new tests.
|
||||
Add only missing assembled-boundary coverage. One synthetic fixture can prove
|
||||
independent collection names, scoped execution/inspection, full-ID filtering,
|
||||
exact defaults, optional labels, and inventory without duplicating every
|
||||
assertion at each internal layer.
|
||||
2. Use a controlled concurrent model boundary to verify ordinary and differently
|
||||
scoped calls share the same configured backend's capacity. Reuse existing
|
||||
admission/cancellation tests for queue mechanics. Verify prepared selections
|
||||
remain captured when another view is used; no namespace is mutable global state.
|
||||
3. Confirm final-value-only validation, source-local headers versus ordinary
|
||||
metadata, retained inherited/overlaid content origins, and redaction under
|
||||
namespaced paths. Final inspection still describes ConfigKit's original
|
||||
effective shape rather than a rewritten runtime index.
|
||||
4. Remove all temporary selector signatures/adapters from the migration stages.
|
||||
Review source/tests/docs for old version-key maps, version selection/assertions,
|
||||
automatic catalog contributions, mandatory built-in names, and local module
|
||||
replacements. Historical notes and optional metadata references may remain.
|
||||
5. Run the final offline checks once for the resulting content:
|
||||
|
||||
```sh
|
||||
GOWORK=off go test -race ./...
|
||||
GOWORK=off go vet ./...
|
||||
GOWORK=off python3 scripts/check-examples.py
|
||||
git diff --check
|
||||
```
|
||||
|
||||
Format changed Go files and validate affected Markdown links. Review the full
|
||||
diff/status for unintended files and private data; commit the completed unit.
|
||||
Mark this plan and the feature roadmap complete, leaving retirement to normal
|
||||
release preparation. Do not publish, wait for CI, probe providers, or introduce
|
||||
a catalog-release dependency to finish implementation.
|
||||
|
||||
Completion: acceptance coverage was traced across public collection assembly,
|
||||
scoped execution and inspection, full-ID listings and inventories, exact stored
|
||||
defaults, optional prompt labels, prepared selection capture, source headers and
|
||||
origins, final-value validation, namespaced redaction, and retained effective
|
||||
configuration shape. Added a controlled concurrent regression test proving an
|
||||
ordinary call and a differently scoped call share one collection backend's
|
||||
capacity. Audited current source, tests, and docs for obsolete selectors,
|
||||
automatic catalogs, built-in profile promises, and local module replacements;
|
||||
remaining version references are descriptive labels or migration guidance.
|
||||
Marked the feature roadmap complete; normal release preparation owns retirement.
|
||||
Validation passed: `GOWORK=off go test -race ./...`, `GOWORK=off go vet ./...`,
|
||||
`GOWORK=off python3 scripts/check-examples.py`, `gofmt`, local Markdown link
|
||||
checks, and `git diff --check`.
|
||||
|
||||
## Open Questions
|
||||
|
||||
None required to begin implementation. The feature roadmap's policy decisions
|
||||
are settled; the delivery decisions above specify the public shapes and migration
|
||||
path. A newly discovered upstream incompatibility or genuine change of scope
|
||||
should be reported with a recommendation and viable alternative rather than
|
||||
silently changing the accepted configuration boundary or adding restrictions.
|
||||
@@ -1,190 +0,0 @@
|
||||
# Configuration Collections Implementation Review Findings
|
||||
|
||||
Status: all three findings completed 2026-10-17.
|
||||
|
||||
Reviewed implementation after `89af2a1` through `0221cd4` against the accepted
|
||||
[feature roadmap](configuration-collections.md), [implementation plan](implementation.md),
|
||||
[configuration-boundary ADR](../adr/0001-configuration-boundary.md), current
|
||||
GoDoc, and applicable repository policies. This replaces the previous completed
|
||||
smoke-test remediation log; its findings and completion records remain in Git.
|
||||
|
||||
## Architecture Assessment
|
||||
|
||||
The targeted architecture is substantially in place:
|
||||
|
||||
- ConfigKit v0.8.0 composes caller-selected inputs and applies the shared optional
|
||||
specification-header policy. PromptKit interprets final effective values and
|
||||
resource contexts without flattening the retained configuration or replaying
|
||||
source history.
|
||||
- Root resources retain unqualified identities; collection resources use full
|
||||
namespace/local-key identities in captured runtime indexes. Stored references
|
||||
remain exact, and inheritance stays with ConfigKit.
|
||||
- Scoped engine views share the assembled runner, model clients, inspection,
|
||||
inventory, and capacity machinery. Qualification is centralized at engine
|
||||
entry points; prepared execution retains its captured selections.
|
||||
- Prompt selection is ID-only, with optional descriptive version labels.
|
||||
Namespace-filtered prompt/profile/backend listing, resource namespace discovery,
|
||||
and effective collection inventory are available.
|
||||
- Automatic catalog dependencies and built-in profiles are removed. Consumer
|
||||
migration guidance and offline collection examples are present.
|
||||
|
||||
The findings below are focused ownership, construction-efficiency, and contract
|
||||
documentation corrections, not reasons to redesign the feature. No additional
|
||||
unrequested runtime restrictions were identified in the reviewed paths. Finding 3
|
||||
addresses a retained public guarantee that no longer describes scoped views.
|
||||
|
||||
## Finding 1: Share Origin Metadata Across Collection Materialization
|
||||
|
||||
Priority: P2. Status: completed 2026-10-17.
|
||||
|
||||
Locations: [load.go](../../internal/configuration/load.go), lines 257–262, and
|
||||
[materializePrompts](../../internal/configuration/prompts.go), lines 84–95.
|
||||
|
||||
The loader now calls `materializePrompts` separately for every collection. Each
|
||||
call creates its own lazy metadata cache and calls `state.result.Metadata()`
|
||||
when it encounters a file-backed message. In the selected ConfigKit v0.8.0,
|
||||
`Result.Metadata` defensively clones the whole metadata snapshot, including
|
||||
field/history maps and paths. The cache is consequently reused within one
|
||||
collection but discarded before the next.
|
||||
|
||||
With C collections requiring origins and M total metadata entries, this adds
|
||||
O(C × M) copying during construction; when resource count grows with collection
|
||||
count, this portion becomes quadratic. Before the collection refactor, the same
|
||||
cache covered the single prompt-materialization pass.
|
||||
|
||||
An offline diagnostic benchmark used 4, 16, and 64 collections, each containing
|
||||
one prompt with one `content_file`, all reading a tiny in-memory file. A single
|
||||
iteration per case reported:
|
||||
|
||||
| Collections | Allocated bytes per construction |
|
||||
| --- | ---: |
|
||||
| 4 | 1,662,192 |
|
||||
| 16 | 10,267,520 |
|
||||
| 64 | 104,352,336 |
|
||||
|
||||
An allocation-space profile across those diagnostic loads attributed about 79%
|
||||
of allocated space to ConfigKit metadata cloning reached through
|
||||
`materializePrompts`' origin callback. These are illustrative local measurements,
|
||||
not stable thresholds or a claim about all constructor workloads.
|
||||
|
||||
Required outcome: reuse one construction-wide lazy metadata snapshot or an
|
||||
equivalent narrow lookup across root and collection prompt materialization.
|
||||
Preserve exact effective paths, inherited origins, resource contexts, and
|
||||
handle closure. Keep the change local and construction-scoped; no new source
|
||||
validation, global cache, or upstream API extension is needed.
|
||||
|
||||
Validation: retain origin/inheritance/content-capture coverage with more than
|
||||
one file-backed collection, and use a small scaling benchmark to verify the
|
||||
repeated full-snapshot cost is removed. Do not make exact allocations, elapsed
|
||||
times, or private helper call order a new contract.
|
||||
|
||||
Completion: prompt materialization now traverses root and collection resources
|
||||
within one construction-scoped pass sharing the lazy origin metadata snapshot.
|
||||
Extended the imported/inherited-content test across distinct collection resource
|
||||
contexts and added a scaling benchmark without allocation or timing assertions.
|
||||
At 64 collections the same benchmark dropped from 104,574,200 to 21,239,712
|
||||
allocated bytes per construction in single-iteration local measurements.
|
||||
Validation passed: focused origin tests and benchmark, race-enabled configuration
|
||||
and public API tests, configuration vet, and `git diff --check`.
|
||||
|
||||
## Finding 2: Detach The Optional Label In Internal Prepared Snapshots
|
||||
|
||||
Priority: P3. Status: completed 2026-10-17.
|
||||
|
||||
Location: [clonePreparedRun](../../internal/usecase/prepared_execution.go),
|
||||
lines 226–244; also its use by `PrepareExecution` and `PreparedExecution.Details`.
|
||||
|
||||
`domain.PreparedRun.PromptVersion` changed from a string to `*string`, but
|
||||
`clonePreparedRun` still performs only the initial struct assignment for that
|
||||
field. Its returned snapshot shares the label pointer with its source.
|
||||
`PrepareExecution` uses this copier to produce retained details from the
|
||||
execution snapshot, and `Details` uses it again for each returned copy.
|
||||
|
||||
A temporary package-level probe constructed a prepared handle with label
|
||||
`original`, obtained its internal `Details`, changed that result's label to
|
||||
`mutated`, then obtained `Details` again. The second result incorrectly contained
|
||||
`mutated`. This contradicts the internal method's fresh-copy contract and the
|
||||
planned ownership of optional metadata. The same copier leaves retained details
|
||||
and execution metadata sharing this pointer.
|
||||
|
||||
The public facade's `fromDomainPreparedRun` currently copies the pointer again,
|
||||
so public callers remain protected by that additional boundary. This is an
|
||||
internal ownership gap, not a reproduced public mutation or credential leak.
|
||||
|
||||
Required outcome: copy the optional label with the existing nil-aware string
|
||||
copy helper when cloning prepared runs. Retain absent versus declared-empty
|
||||
semantics and the existing public ownership protection; a cloning-framework
|
||||
refactor is unnecessary.
|
||||
|
||||
Validation: extend the existing owning use-case prepared-details test with a
|
||||
non-nil version label. Mutation of one returned internal snapshot must not change
|
||||
retained details, the execution snapshot, or a later result. Reuse existing
|
||||
nil/empty public metadata coverage rather than reproducing it at every layer.
|
||||
|
||||
Completion: the prepared-run copier now detaches the optional version label with
|
||||
the existing nil-aware helper. Extended the owning prepared-execution snapshot
|
||||
test to mutate source and returned labels and verify later details, execution
|
||||
metadata, and post-execution details remain unchanged. The regression failed
|
||||
before the fix and passes afterward. Validation passed with `GOWORK=off`:
|
||||
the focused regression, `go test -race ./...`, `go vet ./...`, all three offline
|
||||
examples, and `git diff --check`.
|
||||
|
||||
## Finding 3: Document Engine Ownership In Terms Of The Shared Assembly
|
||||
|
||||
Priority: P3. Status: completed 2026-10-17.
|
||||
|
||||
Location: [Engine GoDoc](../../engine.go), lines 85–101; related
|
||||
[prepared-handle GoDoc](../../prepared_execution.go) and
|
||||
[WithNamespaces GoDoc](../../namespaces.go).
|
||||
|
||||
The Engine declaration still says that each Engine owns independent
|
||||
backend-capacity pools. `WithNamespaces` now returns another `*Engine` sharing
|
||||
the same runner and capacity manager, as expressly required by the feature.
|
||||
The new `TestScopedCollectionRunsShareBackendCapacity` demonstrates this sharing;
|
||||
the runtime is correct, but the public declaration promises the opposite for
|
||||
these returned Engine values.
|
||||
|
||||
Prepared handles are likewise described as bound to the Engine that created
|
||||
them, without clarifying that namespace views share that underlying owner.
|
||||
The current public tests intentionally run a scoped handle through the original
|
||||
engine. Documentation should make that distinction clear rather than imply
|
||||
per-facade capacity or handle ownership.
|
||||
|
||||
Required outcome: distinguish independently constructed engines from namespace
|
||||
views of one assembled engine. State that views share clients/capacity and the
|
||||
prepared-execution owner, with selections already frozen in prepared handles.
|
||||
Preserve independent ownership between separate `NewEngine` constructions.
|
||||
Do not change the implementation to satisfy the stale wording or introduce
|
||||
scope-mismatch rejection.
|
||||
|
||||
Validation: review the affected GoDoc together. Reuse the passing shared-capacity
|
||||
and cross-view prepared-handle tests; no new behavior or test matrix is needed.
|
||||
|
||||
Completion: Engine, WithNamespaces, PrepareExecution, RunPrepared, and the
|
||||
prepared-handle GoDoc now distinguish independent constructions from shared
|
||||
namespace views. They describe shared clients/capacity and handle ownership,
|
||||
with captured selections unchanged by the invoking view. No runtime behavior
|
||||
was changed. Reviewed rendered `go doc` output; the existing shared-capacity and
|
||||
cross-view prepared-handle tests and `git diff --check` passed. The full
|
||||
race-enabled suite, vet, and offline examples from Finding 2 cover the unchanged
|
||||
implementation.
|
||||
|
||||
## Review Validation
|
||||
|
||||
Passed on the reviewed committed tree with `GOWORK=off`:
|
||||
|
||||
- `go test -race ./...`;
|
||||
- `go vet ./...`;
|
||||
- `python3 scripts/check-examples.py`.
|
||||
|
||||
Source review covered the configuration handoff, header policy, paths/origins
|
||||
and sensitivity, root/collection indexing, optional metadata copying, all scoped
|
||||
selection paths, listing/inventory behavior, shared capacity, dependency removal,
|
||||
and migration contracts. Representative existing tests were inspected as well as
|
||||
executed; the green suite does not cover the internal label-aliasing probe.
|
||||
|
||||
The package-level ownership probe failed as described in Finding 2. The
|
||||
construction benchmark and allocation profile support Finding 1. Both probes
|
||||
used temporary Go test overlays outside the repository; neither changes the
|
||||
implementation or adds permanent tests. No live provider or real credential
|
||||
was used. Only this remediation document is changed by the review.
|
||||
@@ -1,592 +0,0 @@
|
||||
# Explicit Smoke Test Diagnostics
|
||||
|
||||
Status: accepted target state. Implementation has not started.
|
||||
|
||||
This roadmap replaces the earlier capability proposal and incorporates the
|
||||
subsequent discussion. It defines the target feature, not a staged implementation
|
||||
plan or current public API. The companion [implementation plan](implementation.md)
|
||||
owns the ordered work and delivery mechanics.
|
||||
|
||||
## Purpose And Ownership
|
||||
|
||||
Provide reusable, explicitly invoked diagnostics for endpoint reachability,
|
||||
authentication, plain-text inference, and structured-output inference. Consumers
|
||||
can build setup and diagnostic workflows without maintaining their own synthetic
|
||||
prompts, schemas, and provider probes.
|
||||
|
||||
PromptKit is opinionated about what each test checks and how it operates. It owns
|
||||
fixed content, provider adapters, evaluation rules, and neutral factual reports.
|
||||
Consumers select tests, targets, and supported execution settings; they interpret
|
||||
the results and decide what action to take. There is no overall success,
|
||||
application-readiness verdict, or configurable criteria framework.
|
||||
|
||||
For structured output, report whether structured output was requested, a response
|
||||
was received, generated content was valid JSON, and that JSON matched the fixed
|
||||
test schema. Four positive observations do not prove provider enforcement; no
|
||||
field attempts to infer it.
|
||||
|
||||
For authentication, distinguish credentials supplied with the dispatched request,
|
||||
a reachable authentication endpoint, and credentials accepted by that operation.
|
||||
A reachable endpoint can reject credentials, and a transport failure can leave
|
||||
acceptance undetermined. Authentication is inapplicable on the maintainer's
|
||||
credential-free RakestrawHome deployment; that is neither an error nor a failure.
|
||||
|
||||
## Single-Sprint Delivery Objective
|
||||
|
||||
The staged implementation must be completable from start to finish in one sprint,
|
||||
including built-in defaults, tests, and documentation. It must not require
|
||||
intermediate human-review approvals, upstream releases, external publication, or
|
||||
live-provider access to progress or establish completion. Automated checks and
|
||||
focused local commits remain part of ordinary implementation.
|
||||
|
||||
Built-in diagnostic metadata belongs in the local OpenRouter and RakestrawHome
|
||||
catalogs, with their validated revisions identified by local candidate tags.
|
||||
There is no intermediate PromptKit-owned metadata underlay or requirement for
|
||||
separate loader/catalog publication before integration.
|
||||
|
||||
A temporary, uncommitted Go workspace may support coordinated development.
|
||||
The completed integration uses normal module requirements and Go-generated
|
||||
checksums and is validated with `GOWORK=off`, without filesystem replacements.
|
||||
Exact local tag contents can be supplied through a temporary filesystem module
|
||||
proxy, without public HTTPS availability or upstream publication. Local tags
|
||||
alone do not make Go discover adjacent checkouts.
|
||||
Keep workspace/proxy artifacts outside committed repository contents, following
|
||||
the [repository-independence policy](../policy/architecture.md#repository-independence).
|
||||
|
||||
PromptKit's completed sprint commits may reference those not-yet-published
|
||||
catalog versions in `go.mod` and their corresponding Go-generated checksums in
|
||||
`go.sum`. Implementation completion and local validation do not claim that other
|
||||
machines can fetch them. Public distribution is separate: consumers eventually
|
||||
need those exact contents available through their configured module sources.
|
||||
|
||||
Normal release preparation and publication follow implementation separately;
|
||||
this objective does not bypass the repository's release process or authorize
|
||||
publication during the sprint.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation.
|
||||
[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request)
|
||||
checks a concrete application request without model generation. `Run` executes
|
||||
that request and may structurally validate and repair its output. These operations
|
||||
remain unchanged; smoke tests need not validate an intended application request.
|
||||
|
||||
The [runner](../internal/runner.md) shares captured prompt/profile selection and
|
||||
execution-target resolution. The [generation coordinator](../internal/generation.md)
|
||||
handles attempts, retries, usage, and active backend permits; the
|
||||
[model client](../internal/llm.md) handles transport and decoding. Ordinary `Run`
|
||||
requires an application prompt and returns no partial result on operational
|
||||
failure, so calling it directly is not sufficient for the proposed diagnostic API.
|
||||
|
||||
Backend configuration has no diagnostic endpoints, capabilities, or default test
|
||||
profile. New fields must follow [ADR 0001](../adr/0001-configuration-boundary.md):
|
||||
ConfigKit owns loading and composition; PromptKit interprets and validates final
|
||||
effective values. Provenance stays available for explicit inspection and is not
|
||||
an execution-time resolution or validation mechanism.
|
||||
|
||||
## Public Operation And Request
|
||||
|
||||
One synchronous call runs one independently selected operation:
|
||||
|
||||
```go
|
||||
func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error)
|
||||
|
||||
type SmokeTestKind string
|
||||
|
||||
const (
|
||||
SmokeTestReachability SmokeTestKind = "reachability"
|
||||
SmokeTestAuthentication SmokeTestKind = "authentication"
|
||||
SmokeTestTextInference SmokeTestKind = "text_inference"
|
||||
SmokeTestStructuredOutput SmokeTestKind = "structured_output"
|
||||
)
|
||||
|
||||
type SmokeTestRequest struct {
|
||||
Kind SmokeTestKind
|
||||
BackendID string
|
||||
PromptID string
|
||||
PromptVersion string
|
||||
ProfileID string
|
||||
Execution *ExecutionTargetOverride
|
||||
APIKey string `json:"-"`
|
||||
TransientRetries *int
|
||||
}
|
||||
```
|
||||
|
||||
The request supplies no variables, inputs, appended messages, schema, or output
|
||||
contract. An application prompt is a profile-selection aid only: its messages,
|
||||
session template, input requirements, and output contract are not executed.
|
||||
Consumers can compose several calls themselves; custom application tests remain
|
||||
available through ordinary `Run`.
|
||||
|
||||
Reuse existing `ExecutionTargetOverride` semantics. Inference uses its endpoint,
|
||||
model, numeric, reasoning, timeout, credential-source, and provider-extra controls.
|
||||
Backend-only probes use endpoint, credential-source, and timeout settings as
|
||||
applicable; they do not require model completeness or execute generation merely
|
||||
because a model setting was supplied. Profile settings remain the inference
|
||||
baseline unless overridden; no silent cheaper-model substitution is proposed.
|
||||
|
||||
Direct credentials take precedence over environment credentials and follow
|
||||
existing request-scoped ownership and redacted-formatting conventions. Optional
|
||||
missing/blank credentials are omitted from the upstream request. Authentication
|
||||
can test the resulting unauthenticated request rather than reject missing keys
|
||||
locally. If an explicitly required credential prevents inference dispatch, report
|
||||
that local diagnostic finding without pretending the provider rejected it.
|
||||
Ordinary execution's required-credential behavior remains unchanged.
|
||||
|
||||
## Selection And Routing
|
||||
|
||||
Preserve the accepted inference selection order:
|
||||
|
||||
| Caller selection | Execution profile |
|
||||
| --- | --- |
|
||||
| Prompt ID only | Selected prompt version's default profile |
|
||||
| Profile ID only | Explicit profile |
|
||||
| Both | Explicit profile overrides the prompt default |
|
||||
| Neither | Explicit backend's configured default smoke-test profile |
|
||||
|
||||
Reuse exact identity and omitted-version semantics. An invalid explicit selection
|
||||
does not fall back to a backend default. Missing backend/default selection is an
|
||||
error when inference needs it, not a reason to substitute an arbitrary profile.
|
||||
Backend-only probes require neither a default profile nor unrelated prompt data.
|
||||
|
||||
Whenever inference selects a profile, that profile supplies the resolved endpoint,
|
||||
credential settings, and backend/capacity identity, subject to ordinary execution
|
||||
overrides. This applies equally to explicit profiles, prompt defaults, and
|
||||
backend-default profiles. A supplied `BackendID` is a selection fallback, not a
|
||||
constraint or a request to rebind the selected profile's backend identity.
|
||||
|
||||
For example, if backend A names a default profile that resolves to backend B,
|
||||
inference uses B's resolved target and capacity identity. If that default profile
|
||||
is endpoint-only, inference retains an empty backend identity. Report A as the
|
||||
default-selection source and B (or the empty identity) as the effective backend.
|
||||
No recursive backend-default lookup or mismatch rejection is introduced.
|
||||
|
||||
For backend-only probes, an explicit or prompt-selected profile likewise supplies
|
||||
the target and its backend's diagnostic metadata. Otherwise `BackendID` selects
|
||||
the backend directly, without consulting its default inference profile. The
|
||||
profile wins when both selectors are supplied, even when they name different
|
||||
backends.
|
||||
|
||||
An endpoint-only profile remains usable for inference and reachability without
|
||||
an inferred backend identity or provider-specific authentication adapter.
|
||||
Endpoint overrides retain the effective backend's ordinary capacity identity.
|
||||
Backend-only probes can resolve profile endpoint/credential metadata without
|
||||
requiring its model. Prompt selection uses captured identity/default-profile
|
||||
metadata without rendering or loading the application's artifacts or schemas.
|
||||
|
||||
## Report Shapes
|
||||
|
||||
Use factual Boolean observations with presence, rather than a caller-configurable
|
||||
list of criteria. These Go-facing values need no stable JSON contract initially.
|
||||
|
||||
```go
|
||||
type SmokeTestReport struct {
|
||||
Kind SmokeTestKind
|
||||
Supported bool
|
||||
Applicable *bool
|
||||
Completed bool
|
||||
Reason SmokeTestReason
|
||||
Target SmokeTestTarget
|
||||
Facts SmokeTestFacts
|
||||
Timing SmokeTestTiming
|
||||
Attempts []SmokeTestAttempt
|
||||
Usage ObservedTokenUsage
|
||||
Failure ExecutionFailureKind
|
||||
}
|
||||
|
||||
type SmokeTestTarget struct {
|
||||
BackendID, ProfileID, PromptID, PromptVersion string
|
||||
DefaultSourceBackendID string // backend whose default selected the profile, if any
|
||||
SelectionSource string // "backend_default", "profile", "prompt", or "backend"
|
||||
Endpoint string // initially resolved diagnostic destination
|
||||
EffectiveSettings ExecutionTarget
|
||||
SettingsPresence ExecutionTargetPresence
|
||||
}
|
||||
|
||||
type SmokeTestFacts struct {
|
||||
CredentialsSupplied *bool
|
||||
EndpointReachable *bool
|
||||
CredentialsAccepted *bool
|
||||
ResponseReceived *bool
|
||||
GenerationCompleted *bool
|
||||
TextReturned *bool
|
||||
ExpectedTextMatched *bool
|
||||
StructuredOutputRequested *bool
|
||||
ValidJSON *bool
|
||||
SchemaMatched *bool
|
||||
}
|
||||
|
||||
type SmokeTestTiming struct {
|
||||
StartedAt time.Time
|
||||
Duration time.Duration
|
||||
CapacityWait *time.Duration
|
||||
RequestDuration *time.Duration
|
||||
RetryDelay *time.Duration
|
||||
EvaluationDuration *time.Duration
|
||||
}
|
||||
|
||||
type SmokeTestAttempt struct {
|
||||
Dispatched bool
|
||||
InitialEndpoint string // resolved destination before redirects
|
||||
Endpoint string // last observed request destination; facts refer to this destination
|
||||
HTTPStatus *int
|
||||
Facts SmokeTestFacts
|
||||
Timing SmokeTestTiming
|
||||
Failure ExecutionFailureKind
|
||||
Usage ObservedTokenUsage
|
||||
}
|
||||
|
||||
type SmokeTestReason string
|
||||
|
||||
const (
|
||||
SmokeTestReasonNone SmokeTestReason = ""
|
||||
SmokeTestReasonAuthenticationNotApplicable SmokeTestReason = "authentication_not_applicable"
|
||||
SmokeTestReasonProbeNotConfigured SmokeTestReason = "probe_not_configured"
|
||||
SmokeTestReasonHTTPDiagnosticsUnavailable SmokeTestReason = "http_diagnostics_unavailable"
|
||||
SmokeTestReasonRequiredCredentialMissing SmokeTestReason = "required_credential_missing"
|
||||
SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized"
|
||||
)
|
||||
```
|
||||
|
||||
`Supported` records whether an implementation is available for the target and
|
||||
selected client. `Applicable=false` records known inapplicability; nil leaves
|
||||
applicability undetermined, for example when no authentication probe is configured.
|
||||
Missing probe metadata does not imply that a provider accepts anonymous access.
|
||||
`Reason` is a named factual code, not application severity or free-form diagnostic
|
||||
text. The constants distinguish, respectively, no additional reason, explicitly
|
||||
inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics
|
||||
for the selected client, a required credential missing before inference dispatch,
|
||||
and an authentication response whose documented meaning cannot be established.
|
||||
HTTP status and `Failure` classify ordinary rejection and transport outcomes;
|
||||
they do not need duplicate reason codes.
|
||||
|
||||
`Completed` means the diagnostic concluded, not that checks passed. A rejected,
|
||||
failed, unsupported, or inapplicable diagnostic can conclude with nil Go error.
|
||||
Caller cancellation before completion leaves it false and returns available facts.
|
||||
**A nil Go error means the diagnostic concluded, not that the tested operation
|
||||
succeeded.** Make this distinction prominent in GoDoc and consumer examples.
|
||||
|
||||
A fact is true or false only when established; nil means not evaluated,
|
||||
undetermined, or inapplicable. Operation metadata and failure classification
|
||||
provide context. In particular:
|
||||
|
||||
- `CredentialsSupplied`: a nonempty credential was attached to the observed
|
||||
dispatched request for the attempt's `Endpoint`, not merely named in configuration
|
||||
or attached to an earlier redirect hop. No observed request leaves this fact
|
||||
absent. For injected clients, do not claim upstream transmission PromptKit cannot
|
||||
observe. Observation at the transport boundary is not proof of wire transmission.
|
||||
- `EndpointReachable`: an HTTP response was received from the tested destination,
|
||||
regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation
|
||||
ending without a response yields false. An injected generation result alone
|
||||
does not establish HTTP reachability.
|
||||
- `CredentialsAccepted`: the designated authentication adapter recognized acceptance
|
||||
or rejection. A rejection yields false; transport failure, server error, or an
|
||||
uninterpretable response can leave it nil. Acceptance does not certify balance,
|
||||
model permissions, or future access. Absence of credentials is not an invalid-key
|
||||
assertion: a documented 401 with `CredentialsSupplied=false` records rejection
|
||||
of an unauthenticated request, not rejection of a supplied key.
|
||||
- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a
|
||||
response was returned by the injected generation boundary. A non-success or
|
||||
malformed HTTP response still counts as received.
|
||||
- `GenerationCompleted`: generation decoded/completed successfully, separately from
|
||||
generated-content checks. Empty decoded content can complete generation.
|
||||
- `TextReturned`: generated content is non-whitespace. `ExpectedTextMatched` compares
|
||||
against the private expected text using trimming only, without case folding or
|
||||
semantic matching.
|
||||
- `StructuredOutputRequested`: the fixed native schema constraint was supplied to
|
||||
the built-in provider request or injected generation boundary. This is not proof
|
||||
of upstream transmission by an injected client or provider enforcement.
|
||||
- `ValidJSON`: the generated content, not the HTTP envelope, parsed as JSON.
|
||||
`SchemaMatched` evaluates parsed JSON against the fixed test schema and remains
|
||||
absent when parsing or response acquisition prevents evaluation.
|
||||
|
||||
Known inapplicable authentication performs no dispatch and leaves request and
|
||||
acceptance facts absent. Unknown measurements remain unknown, including phase
|
||||
measurements for unperformed work. Report-level capacity wait, request duration,
|
||||
evaluation duration, and retry delay sum the corresponding measured work across
|
||||
the diagnostic. Request duration excludes capacity waiting, evaluation, and retry
|
||||
backoff. Retry delay includes completed and interrupted backoff. Per-attempt
|
||||
timings retain their own measurements; backoff is reported at the operation level.
|
||||
Total duration is wall time, including selection and other orchestration overhead,
|
||||
so it need not equal the phase sum. Timings impose no application latency threshold
|
||||
or universal deadline.
|
||||
|
||||
If retries are enabled, top-level facts describe the most recent dispatched
|
||||
attempt. Cancellation during backoff or a later capacity wait does not replace
|
||||
those facts with an undispatched attempt's empty facts. The operation's `Failure`
|
||||
and `Completed` still describe the interruption; earlier attempt records retain
|
||||
their own outcomes. A canceled capacity acquisition can add a `Dispatched=false`
|
||||
record, but no synthetic attempt is added for cancellation during backoff. With
|
||||
no dispatched attempt, request facts remain absent. Usage reuses presence-aware
|
||||
accounting without claiming complete provider billing. No facts are inferred from
|
||||
arbitrary error text.
|
||||
`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt
|
||||
observations; it is not proof that packets were sent or billable work occurred.
|
||||
|
||||
Reports omit private prompts, expected text, schemas, generated content, credentials,
|
||||
raw bodies, and headers. Use content-free failure classifications and HTTP status,
|
||||
not arbitrary provider messages, to preserve this exposure boundary. Effective
|
||||
settings, provider-extra values, endpoints, and identities may still be sensitive
|
||||
operator data; applications own their disclosure. Apply existing credential
|
||||
exclusion and formatting rules rather than promise that all configuration is safe
|
||||
for unrestricted logging.
|
||||
|
||||
## Diagnostic Procedures And Capability Configuration
|
||||
|
||||
### Reachability
|
||||
|
||||
Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or
|
||||
the effective base endpoint when no separate endpoint is supplied. Any HTTP
|
||||
response establishes reachability, including 401, 404, or 405. It does not establish
|
||||
that the completion route, credentials, or a model work. Authentication is tested
|
||||
separately. Classify transport failure where available without promising exhaustive
|
||||
DNS/TLS diagnosis.
|
||||
|
||||
### Authentication
|
||||
|
||||
Metadata names a supported provider adapter or explicitly marks authentication
|
||||
inapplicable. The first adapter is `openrouter_key`, using GET on the
|
||||
selected base endpoint's `key` path with ordinary bearer credential resolution.
|
||||
OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key)
|
||||
documents this authenticated operation and its 200 and 401 responses. Recognize
|
||||
the documented successful shape and unauthorized response; leave acceptance
|
||||
unknown on other outcomes instead of guessing. Returned account/key details
|
||||
are not exposed.
|
||||
|
||||
Configure RakestrawHome authentication as `none`, pursuant to the maintainer's
|
||||
stated deployment behavior. Other backends can declare supported adapters.
|
||||
An arbitrary models-list or health endpoint is not credential-acceptance evidence.
|
||||
|
||||
### Inference
|
||||
|
||||
Plain-text inference uses a small fixed private prompt and expected answer.
|
||||
Structured-output inference uses a fixed private prompt and self-contained schema,
|
||||
requests native JSON Schema output through the existing client path, and evaluates
|
||||
JSON and schema conformance locally. It imports no application output contract.
|
||||
|
||||
Exact content is an implementation detail and does not enter public prompt listings,
|
||||
inspection, configuration inspection, prepared snapshots, or reports. Injected
|
||||
clients and providers necessarily receive the content they execute: privacy here
|
||||
is an API-exposure boundary, not secrecy from executing collaborators. There is
|
||||
no persistent diagnostic prompt or session state.
|
||||
|
||||
### Backend Metadata
|
||||
|
||||
Extend optional backend configuration through the final-value loader and the
|
||||
`BackendConfig`/`BackendSource` convenience. Illustrative shape:
|
||||
|
||||
```yaml
|
||||
backends:
|
||||
openrouter:
|
||||
smoke_tests:
|
||||
default_profile: promptkit-openrouter-standard
|
||||
authentication:
|
||||
kind: openrouter_key
|
||||
rakestrawhome:
|
||||
smoke_tests:
|
||||
default_profile: rakestrawhome-gemma-4-31b
|
||||
authentication:
|
||||
kind: none
|
||||
```
|
||||
|
||||
Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted
|
||||
reachability metadata uses the effective base for the generic HTTP probe;
|
||||
omitted authentication metadata makes that operation unsupported, with applicability
|
||||
undetermined. Missing default-profile selection matters when inference selects it,
|
||||
not for backend-only checks or an explicit profile. Retain normal effective
|
||||
presence/null and layering semantics. Validate syntax and recognized configuration
|
||||
at the final handoff, without contacting providers or validating intermediate layers.
|
||||
|
||||
Provider paths follow the selected endpoint override, not silently the catalog's
|
||||
original endpoint. Explicit absolute diagnostic endpoints are supported for separate
|
||||
health/auth routes and take precedence for their operation. Configuration stays
|
||||
limited to named adapters and endpoint settings, not arbitrary methods or response
|
||||
predicates. The companion implementation plan specifies convenience types and
|
||||
presence/null interpretation for these fields.
|
||||
|
||||
Reuse the configured HTTP client and its redirect handling, without adding custom
|
||||
credential forwarding. `Target.Endpoint` records the resolved starting destination.
|
||||
Each HTTP attempt records `InitialEndpoint` and the last observed dispatched
|
||||
request destination as `Endpoint`; its credential, reachability, response, and
|
||||
acceptance facts describe that destination. If a redirect leads to a failed request,
|
||||
do not attribute an earlier hop's response or credentials to the later destination.
|
||||
When a client prevents observing the relevant request, leave the affected evidence
|
||||
unknown. Injected generation alone supplies no HTTP destination evidence.
|
||||
|
||||
Authentication acceptance is interpreted from the response for the observed
|
||||
destination under the selected adapter's contract. It does not assert acceptance
|
||||
at the initial destination or at every redirect hop. These endpoint fields provide
|
||||
attribution without introducing a public redirect trace.
|
||||
|
||||
Deliver built-in metadata in the local catalog repositories at
|
||||
`../promptkit-backend-openrouter` and `../promptkit-backend-rakestrawhome`.
|
||||
Catalog changes and PromptKit adoption are included in the sprint; publication
|
||||
and remote module availability are not intermediate dependencies. Caller
|
||||
overlays remain available through ordinary composition. Interpret and validate
|
||||
only the assembled final configuration. Built-in reachability follows the
|
||||
effective endpoint rather than pinning a separate catalog endpoint.
|
||||
|
||||
The catalogs expose diagnostic metadata through an additive, opt-in embedded
|
||||
asset root. Existing `/v3` catalog roots remain usable by older strict loaders;
|
||||
new PromptKit explicitly selects the additional assets. Provider metadata stays
|
||||
owned by its catalog rather than being duplicated in PromptKit.
|
||||
|
||||
Local candidate tags identify validated contents; they are not published releases.
|
||||
The implementation plan owns candidate selection and local proxy mechanics.
|
||||
The catalogs' publication-date versioning convention still applies to later
|
||||
publication. If that requires new final version names, update PromptKit's pins
|
||||
and checksums accordingly; retain the validated catalog contents and never move
|
||||
a published tag. Selecting the new metadata requires a compatible PromptKit
|
||||
loader. No intervening PromptKit loader release or external module-download
|
||||
verification is required to finish implementation.
|
||||
|
||||
Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key`
|
||||
adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication
|
||||
setting. Verify each shipped default resolves in the supported assembled effective
|
||||
configuration, including any PromptKit-owned profiles. This is final-value
|
||||
integration validation, not a second validation pass over intermediate sources.
|
||||
A catalog default is replaceable metadata,
|
||||
not a promise of upstream availability, cheap inference, or a stable model choice.
|
||||
|
||||
## Execution And Error Boundary
|
||||
|
||||
Reuse target resolution, generation coordination, HTTP decoding, structural
|
||||
validation, and usage accounting internally. A separate diagnostic orchestration
|
||||
entry point may retain evidence that ordinary `Run` intentionally discards.
|
||||
Do not duplicate inference serialization or build a second inference stack.
|
||||
|
||||
Add a narrow internal evidence path at the transport/coordinator boundary. Existing
|
||||
execution events do not carry redirect destinations or dispatched credential
|
||||
presence, and the coordinator can discard a completed response when cancellation
|
||||
is observed before return. Capture available facts before that discard and retain
|
||||
them for diagnostics. Keep ordinary execution's error/result contracts unchanged;
|
||||
do not expand public execution events just to reconstruct smoke-test evidence.
|
||||
|
||||
Inference participates in existing engine-local admission and active backend
|
||||
limits. HTTP-only probes need no generation permit or unrelated model selection.
|
||||
Caller context and configured timeouts govern operations. Inference defaults to
|
||||
one attempt: nil `TransientRetries` means zero for this operation rather than
|
||||
inheritance from the engine retry budget. Positive values reuse existing built-in
|
||||
inference status retries and retain attempt records; injected generation remains
|
||||
single-dispatch. Reachability and authentication remain single-attempt regardless
|
||||
of a valid inference retry setting. Structural repair is excluded from the initial
|
||||
procedure. Otherwise use the selected profile plus explicit overrides. Ordinary
|
||||
engine defaults and repair behavior do not change. Small test content does not
|
||||
establish small output, a price estimate, or a cost ceiling.
|
||||
|
||||
Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without
|
||||
new required methods. HTTP diagnostics must describe an explicitly selected
|
||||
transport, not secretly probe a different transport in an injected-client engine.
|
||||
For injected-client engines, the initial delivery reports HTTP diagnostics
|
||||
unavailable and adds no separate diagnostic HTTP-client option. Inference
|
||||
diagnostics remain available through the injected client. Known inapplicability,
|
||||
such as authentication configured as `none`, can be reported without any transport.
|
||||
Do not silently fall back to a default HTTP client to diagnose an injected
|
||||
generation client. A future explicit diagnostic transport option remains possible
|
||||
but is not needed to complete this feature.
|
||||
|
||||
Return conventions:
|
||||
|
||||
- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections
|
||||
return ordinary public errors before probe work. Unsupported capability is not
|
||||
the same as invalid selection.
|
||||
- Concluded diagnostics return reports with nil Go error, including credential
|
||||
rejection, provider rejection, transport or configured request-timeout failure,
|
||||
invalid JSON, schema mismatch, unsupported capability, or inapplicability.
|
||||
Local capacity exhaustion is a diagnostic execution finding, not provider failure;
|
||||
request facts remain absent if nothing was dispatched.
|
||||
- Caller cancellation or caller-context deadline expiration returns accumulated
|
||||
observations with an error recognizable through `errors.Is` against
|
||||
`context.Canceled` or `context.DeadlineExceeded`. An already-done context can yield
|
||||
a report with no observations. Do not dispatch again merely to fill missing facts.
|
||||
As elsewhere, arbitrary synchronous injected code must return before cancellation
|
||||
can be observed.
|
||||
|
||||
The cancellation contract applies to this new operation, not to existing
|
||||
preparation APIs. Internal faults preventing meaningful diagnostic execution may
|
||||
still return an ordinary error; the implementation plan should identify those
|
||||
boundaries without promising that every conceivable failure becomes a report.
|
||||
|
||||
## Representative Requests And Results
|
||||
|
||||
Request examples use ordinary exact selections, without private test language:
|
||||
|
||||
```go
|
||||
// Select the application's default profile, but do not run its prompt.
|
||||
req := promptkit.SmokeTestRequest{
|
||||
Kind: promptkit.SmokeTestTextInference,
|
||||
PromptID: "meeting.summary",
|
||||
}
|
||||
// An explicit profile overrides that default.
|
||||
req.ProfileID = "promptkit-openrouter-standard"
|
||||
|
||||
// Select a backend's configured default for structured inference.
|
||||
structured := promptkit.SmokeTestRequest{
|
||||
Kind: promptkit.SmokeTestStructuredOutput,
|
||||
BackendID: "openrouter",
|
||||
}
|
||||
```
|
||||
|
||||
Expected report examples:
|
||||
|
||||
| Operation/outcome | Observations | Go error |
|
||||
| --- | --- | --- |
|
||||
| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil |
|
||||
| RakestrawHome authentication | Supported true, applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil |
|
||||
| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil |
|
||||
| HTTP diagnostics unavailable for injected client | Supported false, applicability unknown, completed, `http_diagnostics_unavailable`; no dispatch | nil |
|
||||
| Authentication without credentials receives documented 401 | Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request | nil |
|
||||
| Required credential absent before inference | Supported/applicable/completed; `required_credential_missing`, failure `invalid_request`; no dispatch and no request facts | nil |
|
||||
| Authentication receives uninterpretable HTTP 200 | Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; `authentication_response_unrecognized`, failure `malformed_response` | nil |
|
||||
| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil |
|
||||
| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil |
|
||||
| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil |
|
||||
| Configured request timeout while caller context remains active | Supported/applicable/completed; failure `deadline`; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts | nil |
|
||||
| Local capacity exhausted | Supported/applicable/completed; failure `capacity`; no dispatch or request facts | nil |
|
||||
| Caller cancels during initial capacity wait | Supported/applicable, completed false; failure `canceled`, measured capacity wait, undispatched attempt; request facts absent | `context.Canceled` identity |
|
||||
| Caller deadline expires during a dispatched request | Supported/applicable, completed false; failure `deadline`; retain available attempt evidence | `context.DeadlineExceeded` identity |
|
||||
| Caller cancels during retry backoff after HTTP 503 | Completed false, failure `canceled`; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt | `context.Canceled` identity |
|
||||
| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error |
|
||||
|
||||
Consumers decide whether any of these findings warrant success, a warning, or a
|
||||
failure in their own application. They own CLI diagnostics, exit statuses,
|
||||
scheduling, and any combination with `ValidateRequest` or normal execution.
|
||||
|
||||
## Work Needed And Target Completion
|
||||
|
||||
- Add the public request/report operation and GoDoc, keeping business logic internal.
|
||||
- Extend final backend metadata interpretation and captured lookup for capabilities
|
||||
and defaults. Preserve explicit configuration inspection without exposing test
|
||||
content or using source history for validation.
|
||||
- Connect private inference content to shared selection, execution resolution,
|
||||
capacity, generation, decoding, and structural evaluation without preparing
|
||||
application inputs or messages.
|
||||
- Add narrow HTTP probes, credential handling, evidence retention, factual results,
|
||||
and partial cancellation reporting through the internal evidence path while
|
||||
preserving existing client contracts. Specify destination/credential attribution
|
||||
without exposing headers or a redirect trace.
|
||||
- Deliver the agreed metadata through both local catalogs with ordinary
|
||||
catalog/caller precedence and no provider-availability guarantee. Shipped
|
||||
defaults resolve in the assembled final configuration, and selected versions
|
||||
work without workspace/replacement substitution. The single-sprint objective
|
||||
applies to the complete integration.
|
||||
- Add focused offline tests with controlled transports and injected clients for
|
||||
selection, rejected/absent credentials, applicability/support, transport/malformed
|
||||
responses, content/JSON/schema facts, cancellation, shared inference capacity,
|
||||
and credential/private-content exposure. Assign distinct test owners; do not
|
||||
repeat existing transport, capacity, or validation matrices or snapshot private
|
||||
test language.
|
||||
- Include the report-state matrix, cross-backend and endpoint-only default profiles,
|
||||
redirect attribution, cancellation after a response, and interrupted retry waits
|
||||
in focused contract coverage. Reuse existing lower-level checks where sufficient.
|
||||
- Update canonical consumer, backend-format, integration, and internal documentation
|
||||
when implementation lands, with release notes explaining the optional addition.
|
||||
Live diagnostics are explicit consumer operations, not default tests or new
|
||||
release gates.
|
||||
|
||||
Completion means consumers can select the four kinds, obtain supported factual
|
||||
observations, distinguish false from unknown/inapplicable, and retain evidence
|
||||
on cancellation. Existing preparation and execution continue unchanged. Initial
|
||||
scope excludes application CLI/service work, readiness policy, continuous monitoring,
|
||||
model-status tests, batch orchestration, customizable test language, cost estimates,
|
||||
and guarantees about later generation.
|
||||
Reference in New Issue
Block a user