diff --git a/docs/roadmap/configuration-collections.md b/docs/roadmap/configuration-collections.md deleted file mode 100644 index 6d57229..0000000 --- a/docs/roadmap/configuration-collections.md +++ /dev/null @@ -1,846 +0,0 @@ -# Configuration Collections, Namespaces, And Scoped Lookup - -Status: implementation complete; roadmap retirement remains for normal release -preparation. This document consolidates the namespace roadmap and selected -collection proposal 3. The configuration and namespace design decisions are -settled. The companion [implementation plan](implementation.md) records concrete -API delivery choices and the completed work. - -## Purpose - -Simplify PromptKit configuration by loading independently maintained prompt, -profile, and backend libraries as YAML collections instead of mandatory Go -modules. Remove the unused prompt-version selection scheme. Specification -headers are optional: maintainers can declare compatibility in a collection -entry without adding a declaration to every file. - -Consumers can select a namespace once and use local resource names while an -engine contains several collections with overlapping names. For example, -WeatherReporter and Homewatch can each define `summarize.alerts`, while provider -namespaces distinguish profiles targeting the same upstream model. - -Conventions and small convenience APIs should reduce authoring and maintenance -work. Preserve consumer choice over sources, layouts, namespaces, and precedence -without adding speculative options, mandatory workflows, or new compatibility -and provenance guarantees. - -## Prerequisite: Adopt ConfigKit v0.8.0 - -The target selects released ConfigKit **v0.8.0** for the collection schema and -loading APIs. At planning time PromptKit selected v0.7.1; the -[implementation plan](implementation.md) records adoption and verification of -v0.8.0. Local filesystem replacements are not part of the distributed module. - -The v0.8.0 tag was reviewed at `137769f` on 2026-10-17. It supplies input -placement, relative import destinations, and shared source-version checks needed -by this design. No additional upstream extension is required. The implementation -plan records adoption of this prerequisite. - -ConfigKit's public `docs/config.md`, ADRs 0011–0014, and v0.8.0 release notes own -the upstream contracts. The capabilities used here are: - -- `Input.Destination` for root-relative placement and import `destination` for - placement relative to the immediate importing document. -- `Options.SourceVersionMode` and `LayeredOptions.SourceVersionMode`, applying - a shared `Definition.Version` policy in each document's source-local coordinates. -- Consumption of accepted non-main headers before composition, with the main - header retained and protected at its effective application path. -- Detached `SourceInfo.Destination` and `SourceInfo.VersionCheck` evidence, - together with existing source locations and resource contexts. - -### Keep Mapping-Based Collections - -ConfigKit v0.8.0 also offers sequence-item placement and registered composition -of distinct list records. These were reviewed and are not selected for this -round. A higher-priority list replaces the entire lower-priority list, duplicate -identities conflict within a strict group, and records cannot inherit from -sibling list records. Switching formats would sacrifice ordinary per-resource -partial overrides and local inheritance for flatter authoring. - -Keep collection resources as mappings keyed by local ID. Do not add an -alternative list schema or a PromptKit identity-based merge layer. The existing -key supplies identity without an additional embedded `id`. This is a scope -choice for this round, not a prohibition on future upstream capabilities. - -## Existing Foundation And Ownership - -The [format reference](../formats.md) and -[consumer guide](../consumers/pkg-promptkit.md) remain authoritative for current -behavior until implementation lands. Directory and embedded configuration -loading already exist through `DirectoryInputs` and `FilesystemInputs`. -The current engine also supplies provider catalog modules and built-in profiles; -the accepted removal is described below. - -[ADR 0001](../adr/0001-configuration-boundary.md) assigns source loading, -composition, and inheritance to ConfigKit. PromptKit interprets and validates -only final effective values, including their resource contexts. Source history -is available for explicit inspection, not additional validation of contributions. -Do not add a manifest pre-parser, source-level namespace ownership checks, or a -second composition pass. Two sources targeting the same namespace follow -ordinary ConfigKit composition; a repeated namespace is not a separate error. - -The [architecture policy](../policy/architecture.md) continues to assign CLI -syntax, discovery policy, and application lifecycle to consuming applications. - -## Collection Ownership And Built-In Removal - -**Accepted: load collections as application/operator-supplied configuration, -and remove PromptKit's built-in profiles as a breaking change.** Remove the -mandatory OpenRouter and RakestrawHome catalog Go-module dependencies and their -automatic contribution to engine configuration. PromptKit owns the definition -format and execution machinery; collection publishers own their configuration -repositories, and applications/operators select and provision the collections -they use. One or more explicitly selected directories can supply backends, -profiles, and prompts through the existing ConfigKit integration. - -Remove `promptkit-openrouter-standard` and `promptkit-openrouter-auto`, including -their implicit catalog dependencies. Do not replace them with another mandatory -starter bundle. Applications may embed their own defaults using the same -configuration format and loader, then layer operator configuration as desired. -Ordinary application overrides remain valid configuration and need not declare -a collection release of their own. - -This decouples collection updates from PromptKit rebuilds and releases and -avoids imposing provider inventories on every consumer. The tradeoff is that -applications and deployments must supply their required configuration and -manage compatible collection revisions. PromptKit does not gain a collection -installer, automatic downloader, or implicit directory search policy. - -Retain the captured-engine lifecycle: configuration changes take effect when -the application constructs and validates a replacement engine. A change to an -installed directory does not mutate an existing engine's captured definitions. -Distribution as configuration does not make new provider protocols or diagnostic -adapters data-only features; those can still require implementation work. - -Prepare explicit identifier mappings, replacement configuration examples, and -consumer migration instructions. Update the built-in-profile promise in the -release policy and its canonical format documentation when removal lands. -Use a release permitted to break configuration and API contracts under the -current release policy: a minor release before v1, or a major release at/after -v1. Actual catalog mappings, packaging arrangements, and release versions remain -delivery work; independent collection releases need not track future PromptKit -software releases. - -## Accepted Identity Convention - -Reserve `/` exclusively as the namespace separator in PromptKit prompt, profile, -and backend identifiers and references. A qualified identifier has exactly two -nonempty components: `namespace/local-name`. Neither component can contain `/`. -Dots remain available within local names. An unqualified identifier has no slash -and remains valid under the existing nonblank-ID rule. - -Reject resource identifiers or supplied references with multiple slashes or an -empty qualified component, such as `a/b/c`, `/name`, and `namespace/`. These -rules do not apply to provider model values, endpoints, resource paths, or prompt -versions. Preserve exact spelling; do not silently trim or rewrite IDs. - -| Resource | Namespace | Local name | Qualified ID | -| --- | --- | --- | --- | -| Prompt | `weatherreporter` | `summarize.alerts` | `weatherreporter/summarize.alerts` | -| Prompt | `homewatch` | `summarize.alerts` | `homewatch/summarize.alerts` | -| Profile | `openrouter` | `google.gemma-4-31b-it` | `openrouter/google.gemma-4-31b-it` | -| Profile | `rakestrawhome` | `google.gemma-4-31b-it` | `rakestrawhome/google.gemma-4-31b-it` | - -Prompt, profile, and backend identities remain separate resource types. The -same namespace can contain all three, and resources in it may come from -multiple configuration sources. A namespace does not imply a directory, -package, provider model identifier, or security boundary. - -A profile ID identifies execution settings. Its configured model value is -still the identifier passed to the provider; qualifying the profile ID does -not modify that value. Both example profiles may retain the configured model -`google/gemma-4-31b-it`. Different profiles may select the same model with -different settings. Prompt version labels remain separate from namespaces. The -selected collection format installs one definition per prompt ID and removes -request-side version selection and assertions. Optional labels are descriptive. - -Retain unqualified and dotted IDs. Existing slash-containing IDs need review: -those with one slash and nonempty components become qualified identities, while -those outside the new grammar require migration. This intentional format break -is accepted in favor of unambiguous resolution; the migration work is described -below. - -## Accepted Consumer Scope - -Provide a lightweight, immutable view over an existing engine with independent -prompt, profile, and backend scopes. The following sketch illustrates the -accepted shape; concrete declarations are specified in the -[implementation plan](implementation.md): - -```go -weather, err := engine.WithNamespaces(Namespaces{ - Prompt: "weatherreporter", - Profile: "openrouter", -}) -// Handle err, then use local prompt and profile names through weather. -``` - -The concrete declarations are recorded in the -[implementation plan](implementation.md). Empty scope fields mean no scope for -that resource type. Keep name qualification shared across the view and engine; -do not duplicate selection logic in each operation. Omit a one-string -shorthand that implicitly scopes all resource types. - -The implemented views share the engine's captured configuration, clients, and -execution machinery. Selecting a namespace does not rebuild the engine or -duplicate its backend capacity accounting. Consumers can use different views -concurrently without changing a shared current-namespace setting. - -The accepted resolution rules, independent of the final scope API, are: - -1. A supplied name without a slash is qualified using the selected scope for its - resource type, then looked up exactly. Without that scope, it is looked up as - an exact unqualified ID. -2. A missing scoped resource does not trigger a search in other namespaces or - among unqualified IDs. Loading another collection cannot redirect lookup. -3. A valid supplied name containing one slash is fully qualified and is looked - up exactly, regardless of scope. Malformed resource references are rejected. -4. Scope applies to caller-supplied resource names. Stored references, including - prompt default profiles, profile backends, inheritance references, and backend - smoke-test default profiles, retain their existing exact meaning. They are - not rebound to the invoking consumer's namespace. -5. Omitted optional IDs retain their existing selection/default behavior; an - omission is not converted into a namespace prefix or a lookup for an empty - local name. -6. Where outputs, inspection, errors, or diagnostic reports identify a selected - resource, preserve its resolved full identity. Any exposed local name should - be distinguishable from that identity. - -Keep qualification consistent across execution, preparation, request validation, -prompt/profile inspection, and applicable smoke-test selections. Namespace -selection must not change request override precedence or provider model strings. -Prompts resolve by ID alone. - -### Local And Qualified References In One Operation - -Syntax alone distinguishes local and qualified references. Do not decide whether -to qualify a name by checking which candidates exist; that would make resolution -depend on loaded content. - -Within a `weatherreporter` prompt scope, `summarize.alerts` resolves only to -`weatherreporter/summarize.alerts`, while `homewatch/summarize.alerts` explicitly -selects that other library's prompt. A qualified profile override similarly -bypasses the profile scope. Scoped operations accept both forms without a -separate reference type or requiring a return to the original engine for -cross-namespace access. - -The original engine remains a way to select an exact unqualified ID without -applying a scope. No escape syntax for unqualified IDs is introduced here. - -### Separate Resource Scopes - -A consumer can select prompt shorthand under `weatherreporter` and profile -shorthand under `openrouter` in the same request. Each resource type has an -independent scope. -A prompt's configured default profile needs no caller profile scope because -the stored reference is already exact. Scope syntax and existence handling are -defined in [scope validation](#scope-validation). - -### Scope Validation - -**Accepted: validate syntax when creating the view; resolve existence when -an operation selects a resource.** In the scope value, the empty -string means no scope. Reject a nonempty namespace containing `/` or consisting -only of whitespace. Otherwise preserve its exact spelling, with no trimming, -case folding, or additional character whitelist. This distinguishes omission -from an unusable namespace without inventing normalization rules. Report invalid -scope construction through PromptKit's existing public error conventions. - -Allow a well-formed namespace even when it has no current members. Its scoped -listing would be empty, and selection of a missing local resource would produce -the ordinary lookup error. Qualified selections would remain usable regardless -of the view's default scope. An empty optional request ID continues to mean -omission under that operation's existing contract; it is never qualified. - -This keeps scope as a naming context and supports uniform application setup -across installations with different library contents. It also avoids treating -successful scope construction as validation of a prompt or execution profile. -The tradeoff is that a misspelled but syntactically valid namespace is detected -only when listed or used, rather than during view construction. - -## Discovery And Namespace Filtering - -**Accepted: integrated namespace filtering and full-identity results.** -Enumerate the distinct named namespaces present in final effective -prompts, profiles, and backends. Allow filtering that enumeration by resource -type so a prompt picker need not show profile-only namespaces. Provide scoped -listings for the supported resource types, adding a backend inventory surface -as part of this work if needed. Namespace filtering is an explicit option in -the library listing API, not a consumer-side filtering helper. Preserve unfiltered -engine listing behavior, with the accepted removal of prompt-version identity -and selection. The companion implementation plan defines the Go filter shape. - -| Selection | Listing behavior | -| --- | --- | -| No explicit filter on the engine | All namespaces, including unqualified resources | -| No explicit filter on a scoped view | The selected scope for that resource type; all resources of that type if no scope is selected | -| Explicit named namespace | Resources in exactly that namespace | -| Explicit unqualified-only filter | Resources whose IDs have no namespace | -| Explicit all-namespaces filter | All resources of that type, including unqualified resources | - -An explicit filter replaces the view's default for the current listing; it -does not intersect with that default or mutate the view. A named namespace -filter uses the namespace syntax rules above. A well-formed namespace -with no matching resources returns an empty listing. Represent default, named, -unqualified-only, and all-namespaces selections distinctly in the API; do not -overload an empty string to mean several of them. - -Return full IDs with optional prompt version labels separately; the full ID -identifies the installed prompt. Do not invent a named namespace such as `default` -for unqualified resources. Namespace enumeration excludes that unnamed group. -Use deterministic ordering consistent with existing listing contracts. - -Qualified result IDs can be passed unchanged to either scoped or unscoped -operations. Unqualified IDs remain literal unqualified identities: when an -explicit all-namespaces or unqualified-only filter exposes one through a scoped -view, use the original engine or a view without that resource scope to select -it exactly. A listing filter does not change subsequent execution scope. This -preserves the accepted lookup rules without introducing an escape syntax. - -Integrated filtering gives consumers a direct way to discover the desired -resources, while full identities keep logs and subsequent selections -unambiguous. Namespace/local-name display decomposition is separate from -filtering; a public splitting helper is not required by this decision. - -Namespace membership is exact: `weather` does not include `weatherreporter`. -Collection inventory describes final collection declarations and optional release -labels; it is distinct from enumeration of namespaces that contain resources. -See [collection inventory](#inspection-and-effective-collection-inventory). - -## Authoring And Assembly - -A file such as `weatherreporter/prompts/alerts.yml` would contain: - -```yaml -summarize.alerts: - version: "1" - default_profile: provider/brief - messages: - - role: user - content: Summarize the weather alerts. - output: - format: text - validation_mode: none -``` - -The version may simply be omitted. Another collection can contribute the same -local prompt name, for example `homewatch/prompts/alerts.yml`: - -```yaml -summarize.alerts: - default_profile: provider/brief - messages: - - role: user - content: Summarize the household alerts. - output: - format: text - validation_mode: none -``` - -A provider collection's `profiles/brief.yml` uses the same local-key convention: - -```yaml -brief: - backend: provider/default - model: vendor/example-model - temperature: 0.2 -``` - -Its `backends/default.yml` could contain this synthetic, non-live endpoint: - -```yaml -default: - endpoint: https://inference.example.invalid/v1 -``` - -A conventional `weatherreporter/collection.yml` entry can declare the shared -specification, optional collection release, and its layout: - -```yaml -version: 1 -release_version: "2.4.0" -configkit: - imports: - - glob: prompts/*.yml - destination: [prompts] -``` - -A provider entry can import `profiles/*.yml` at `[profiles]` and -`backends/*.yml` at `[backends]` in the same way. Collection publishers choose -which categories to include. A required glob must match; unused categories can -be omitted or use ConfigKit's existing optional-import behavior. Ordinary -fragments may omit specification headers under the selected `IfPresent` mode. -An entry header is the recommended publishing convention, not a load-time -requirement selectively enforced on entries by this mode. - -The application selects each entry, enables its imports, and supplies its -namespace through the public input destination. For example, the following -partial Go snippet uses released ConfigKit v0.8.0 APIs: - -```go -entry := configkit.Input{ - Source: configkit.File("/usr/local/share/collections/weatherreporter/collection.yml"), - Imports: true, - Destination: configkit.Keys("collections", "weatherreporter"), -} -``` - -For independent peer collections, one `Underlay` group's `Inputs` is a useful -starting point, not a required grouping. Consumers choose precedence and groups. -Configure `LayeredOptions.SourceVersionMode` as `SourceVersionModeIfPresent` and -`Definition.Version.Path` as `configkit.Keys("version")`, with PromptKit's -supported-specification checker. The application main input remains at the -root and declares `version: 1` in the examples below; both the main input and -its declaration are optional. Namespace and source selection require no manifest parsing in PromptKit. - -| Selected source | Effective destination | -| --- | --- | -| Application main document | Configuration root | -| `weatherreporter/collection.yml` | `[collections, weatherreporter]` | -| Its `prompts/*.yml` imports | `[collections, weatherreporter, prompts]` | -| Homewatch entry and its prompt imports | `[collections, homewatch]` and `[collections, homewatch, prompts]` | -| Provider entry and its profile/backend imports | `[collections, provider]`, then `[profiles]` or `[backends]` appended | - -Directly selected fragments with `Input.Destination` remain possible. Entry -imports are the recommended packaging convention because publishers can describe -their layout. Filenames and directory basenames do not define resource identity; -no implicit directory discovery, namespace inference, or collection installer is -introduced. A small helper should construct inputs from an explicitly selected -source and namespace, without inspecting manifests to compute destinations. `collection.yml` -is a convention, not a required filename. Direct fragments and programmatic -sources remain supported through the source API; no parallel collection loader -or mandatory filesystem layout is introduced. - -Use later underlay groups or main for intentional application/operator overrides. -Placement does not establish precedence. Do not also recursively scan files -already selected by entry imports: ConfigKit's repeated-source rules still apply. - -The entry's source-root `version` is checked and consumed before its payload -composes. Its `release_version` is retained as ordinary collection metadata. -A prompt's nested optional `version` is also ordinary payload. These are three -distinct concepts; source-root release metadata must no longer use `version`. - -The assembled application shape would be equivalent to: - -```yaml -version: 1 -collections: - weatherreporter: - release_version: "2.4.0" - prompts: - summarize.alerts: - version: "1" - default_profile: provider/brief - messages: - - role: user - content: Summarize the weather alerts. - output: - format: text - validation_mode: none - homewatch: - prompts: - summarize.alerts: - default_profile: provider/brief - messages: - - role: user - content: Summarize the household alerts. - output: - format: text - validation_mode: none - provider: - backends: - default: - endpoint: https://inference.example.invalid/v1 - profiles: - brief: - backend: provider/default - model: vendor/example-model - temperature: 0.2 -``` - -This yields `weatherreporter/summarize.alerts` and -`homewatch/summarize.alerts`, plus profile `provider/brief` and backend -`provider/default`. The provider's model string remains `vendor/example-model`; -it is not a PromptKit resource identifier. - -One fragment may contain several local definitions. A fragment starting directly -with `messages:` would need a destination that also supplies the local prompt -key. ConfigKit mounting permits such placement, but with source checks enabled -its root `version` would be the specification header, not prompt metadata. Recommend -the local-key mapping above as the standard authoring form: identity travels -with the definition and files may be reorganized without changing it. Filename-derived IDs are not -needed. - -## Final Schema And Identity - -Use `collections` as a mapping of namespace to collection body, containing -optional `release_version` and optional `backends`, `profiles`, and `prompts` -mappings. Collection release labels are descriptive, with no SemVer ordering -or dependency-solving semantics. Metadata is optional: an installed collection -can contain resources without claiming a published release. - -Each resource key inside a collection is local. The parent namespace and local -key follow the accepted slash-free component grammar; their public spelling is -exactly `namespace/local-name`. There are no duplicate embedded `id` or -`namespace` fields to reconcile. Dots remain ordinary characters. - -Retain unqualified application resources through optional root-level -`backends`, `profiles`, and `prompts` mappings, also keyed by local name and -using the same definition shapes. They produce unqualified public IDs. Under -this proposal, namespaced definitions belong under `collections`, not a second -qualified-key spelling in those root maps. This avoids two configuration paths -producing the same public identity. This representation belongs to the selected -future direction; it does not change today's implemented format. - -Stored references such as `default_profile`, profile `backend`, and smoke-test -profile defaults remain exact public references. Nesting does not automatically -qualify a slashless reference or rebind it to caller scope. For example, a -profile in `provider` still declares `backend: provider/default`. Caller-scoped -lookup follows [the scope rules above](#accepted-consumer-scope) after final -indexes are built. - -### One Installed Prompt And Optional Version - -The resource's qualified ID is its sole configuration identity. There is no -version-key mapping, implicit `latest`, version ordering, or parallel installed -history. Updating the optional version changes metadata on the same prompt. - -An optional prompt version is an opaque string with no SemVer, nonblank, or -normalization requirement. Omission or explicit null means no declared label; -null can clear a lower-layer or inherited label. Quote numeric-looking labels -such as `"1"`. Preserve supplied strings, including empty strings, and distinguish -an absent label in inspection. Do not synthesize a version. - -Remove request-side prompt-version fields, version selection, ordering, and -version defaults as part of the breaking change. Do not reinterpret those fields -as exact-match assertions. Execution and inspection resolve the installed prompt -by ID alone. Listings expose one prompt per ID with its optional descriptive -label separately. A label neither controls execution nor proves unchanged content. - -## ConfigKit Composition Boundary - -Use the implemented upstream contracts without duplicating them in PromptKit. -Input destinations use literal mapping-key segments, independent of filesystem -roots and precedence. Relative import destinations are sequences of strings; -omission and `[]` retain the importing destination. Null, scalar, and non-string -destination members are invalid. No logical path segment means parent traversal -or a reset to the application root. - -Source-root import directives remain structural. Imported content contributes at -its resulting destination, while physical import paths and `content_file` bases -remain relative to their declaring sources. `SourceInfo.Destination` records -placement; normal origins and diagnostics retain real source locations. Selected -overlays and explicit CLI overrides remain rooted at the application root. - -Mounting preserves source identity, cycle/repetition rules, and load limits. -An absent `NoSource` contributes no payload; an actual empty mapping can -materialize its destination containers without clearing lower contents. -ConfigKit's reserved structural keys continue to apply. PromptKit must not -strip headers, re-key source trees, merge fragments itself, or turn source -inspection into an additional enforcement pass. - -### Composition And Overrides - -Different collection keys prevent accidental collisions between equal local -resource names. Two sources targeting the same namespace are not automatically -invalid: disjoint mapping contributions combine. Overlapping scalar fields, -message sequences, nulls, and equal release labels conflict within a strict -group. Across deliberate layers they follow ordinary precedence instead. - -A higher-layer prompt mapping is a partial override. Changing only its version -does not replace its messages or other settings. Omitting a field retains its -lower value, and an empty mapping does not clear a definition. Publishers and -operators must treat such changes as configuration customization, not as an -atomic installation of another prompt release. Do not introduce source audits -or a version-triggered replacement rule to conceal this distinction. - -### Inheritance - -Existing sibling inheritance remains useful within each collection's resource -mapping: `extends: brief` can name the local profile `brief`. Prompt versions, -now ordinary optional fields, would inherit and overlay like other fields; -`version: null` explicitly clears an inherited label. - -Cross-collection profiles are no longer immediate siblings. Consequently, -`extends: provider/brief` cannot reach that collection under current ConfigKit -inheritance. Use existing local sibling inheritance for this development round; -applications can still select the qualified `provider/brief` profile directly. -Authors needing a customized derivative may publish an expanded definition. -Expansion is a migration option, not a mandatory authoring workflow. - -Generic cross-path inheritance remains possible future ConfigKit work if a -concrete consumer need justifies it. It is not a prerequisite of this round. -PromptKit must not add its own inheritance resolver after ConfigKit handoff. - -## Specification Versions And Collection Inventory - -### Shared Specification Policy - -Use one `Definition.Version` policy, with source-local path `version`, for the -candidate PromptKit specification. Initially accept specification text `1` from -string or strict decimal-integer declarations; reject explicit null and other -unsupported declarations through the ConfigKit checker. Accept an absent main -declaration, including an absent main input. Accept additional versions only -when PromptKit actually supports their format semantics: accepting -a header does not migrate its payload or independently validate a fragment's -complete schema. The retired proposals were alternatives, not released formats -sharing specification version 1. - -Use `SourceVersionModeIfPresent` without adding a PromptKit mode-selection -option in this round. A stricter option can be considered for a demonstrated -consumer need. The upstream modes remain distinct: - -| Mode | Implemented behavior | Role in this proposal | -| --- | --- | --- | -| `Default` | Existing main-only effective-path check and protected-path behavior | Not the proposed shared-source policy | -| `IfPresent` | Checks present declarations; skips absent non-main declarations | Initial choice for compact fragments | -| `Required` | Requires declarations on every participating document | Upstream capability; exposing a PromptKit option is deferred | - -Participating documents include actual main and underlay inputs, transitive and -glob imports, and the selected overlay. Defaults, synthetic CLI overrides, -resolution output, absent optional imports, and empty groups do not need headers. -Explicit null is present and reaches the checker; malformed selected values -fail ConfigKit's version-type validation. A rejected declaration fails loading -even if a higher layer would supersede that source's payload. - -In enabled modes, accepted non-main headers are consumed before composition. -The main declaration remains and is protected at its destination-prefixed path. -PromptKit's recommended main is unmounted, so that protected path is `/version`. -Do not confuse header consumption with permission to override the remaining -protected application field. - -`IfPresent` does not require a collection entry header and does not infer a -fragment's compatibility from its entry. A publisher can declare compatibility -in the entry while leaving imported files headerless; the assembled configuration -still undergoes normal validation. A declaration is not a guarantee of future -compatibility or independent schema validation of every imported fragment. -ConfigKit checks the declarations actually present. Requiring headers only on -entry files is not supported by the load-wide mode. Do not emulate it using a -preliminary manifest load or validation of -source history. If that stronger guarantee becomes essential, seek an explicit -upstream contract instead. - -### Naming And Optional Main Inputs - -Reserve source-root `version` for specification declarations throughout the new -format. Use `release_version` for optional collection release labels and retain -nested prompt `version` for the optional installed-prompt label. An old release -file containing only `version: "2.4.0"` can fail the specification check; if its -text happens to be accepted, that field is consumed and lost as release metadata. -Migration must rename it rather than simply enable the new mode. - -This reservation affects resource fragments whose local key is literally -`version`: a root mapping with that name is interpreted as a malformed header. -Allow that resource name through a nested resource mapping, for example -`prompts: {version: ...}` in a collection entry. Reserve the header position, -not the public resource name. Existing `configkit` and `extends` structural -reservations also remain applicable; destinations do not bypass them. - -An absent header in an actual main document still reaches the checker under -`IfPresent`; `NoSource` invokes the absent-main checker as well. PromptKit's -checker accepts omission in both cases. Consumers may assemble only underlays, -fragments, or programmatic inputs without manufacturing a main document or -header. Do not inject a declaration or report undeclared sources as checked. -Present unsupported declarations still fail; final schema validation still runs. - -### Inspection And Effective Collection Inventory - -`SourceInfo.VersionCheck` is source evidence, not a generic success flag. A -non-nil record with `Present == false` can describe an omission for which the -checker did not run. Header paths and locations are source-local. Evidence is -detached and does not reread inputs; text is conservatively redacted during -callbacks and on failed loads, and successful final records respect completed -sensitivity classification. These records belong in explicit configuration -inspection, not in ordinary domain validation or collection-release identity. - -Derive collection inventory from the final `collections` mapping: namespace -plus optional effective `release_version`, sorted and detached from the captured -engine. Metadata-only and resource-only entries can both appear. Root unqualified -resources do not invent a named collection; resource-based namespace enumeration -remains separate. The companion implementation plan defines exported inventory -types. - -An inventory describes effective configuration, not immutable package identity, -historical source ownership, or proof of unchanged published contents. Operator -overrides may change resources while retaining a release label. Engine -construction captures definitions; edits require a replacement engine. - -## Migration - -Migration must choose one installed definition for each former multi-version -prompt, move any retained version into its optional field, and relocate -qualified resource keys into collection/local-key mappings. Update destination -declarations, stored references, inventories, and configuration-inspection paths. -Rename collection release metadata to -`release_version`, configure the shared checker and mode, and account for the -source-root header reservation and optional main declarations. Remove obsolete -request-version fields and version-selection logic from consumer examples and -public APIs rather than retaining them as assertions. Preserve provider model -strings and source resource bases. Namespace mounting does not -automatically relocate schema or artifact stores or rewrite references when a -collection is renamed. - -Retain the accepted removal of mandatory catalog modules and built-in profiles. -This was a breaking configuration/API change under the -[release policy](../policy/release.md); the implementation includes consumer -migration guidance and updates current contracts. The completed work changes -the supported configuration and API as described here. - -Existing unscoped exact lookup remains available for IDs valid under the new -grammar. Compatibility with arbitrary slash-containing resource IDs is -intentionally broken. The migration must: - -- Audit maintained catalogs and representative consumer definitions for invalid - IDs and for valid one-slash IDs whose namespace meaning needs review. -- Choose explicit old-to-new mappings and check for collisions. Do not - automatically replace slashes with dots: distinct existing IDs could collapse - to the same new identity. -- Update renamed keys and their references together, including inheritance, - prompt default profiles, profile backend references, backend smoke-test - defaults, and affected examples and consumer request IDs. -- Keep provider model strings unchanged. Renaming a profile does not rename - the upstream model. -- Document consumer action and coordinate the PromptKit format change with - catalog adoption under the [release policy](../policy/release.md). Its catalog - profile-data exception does not cover this general format/API behavior change; - deliberately retire its built-in-profile preservation promise when the - accepted removal landed. Current policy and format documents now reflect the - implemented behavior. - -Renaming a backend changes its identity; duplicating it under another ID can -separate capacity accounting. Do not use blanket aliases or duplicate catalog -entries as an automatic compatibility mechanism. Exact release versions and -catalog mappings remain implementation-planning work. - -## Work Deferred Or Outside This Round - -- Collection installation, automatic downloading, implicit directory discovery, - package registries, dependency solving, and live reload. -- Automatic rewriting of stored references when mounting or renaming collections. -- Implicit namespace search paths, fallback lookup, namespace-based access control, - and filesystem isolation. -- Changes to provider model names, inference behavior, or ConfigKit precedence. -- Sequence-item resource catalogs, identity-based overlays, cross-path inheritance, - and a PromptKit option requiring specification headers in every document. - -Explicit source destinations, direct fragments, programmatic sources, and -consumer-selected precedence remain supported in the design. Deferred features -can be reconsidered for concrete needs; none blocks implementation. - -## Target Acceptance - -The completed target includes the collection schema and final configuration -adapter, catalog and built-in removal, ID-only prompt selection, immutable -scoped views, integrated filtering, and collection inventory. Runtime indexes -use full identities after ConfigKit handoff. Current contracts and consumer -migration guidance describe the implemented behavior. The -[implementation plan](implementation.md) owns sequencing, concrete public shapes, -documentation tasks, and focused offline validation under the -[testing policy](../policy/testing.md). - -### Namespace And Consumer Acceptance - -- Two prompt scopes resolve the same local name to their respective prompts; - a missing name never resolves from the other scope or an unqualified entry. -- Two provider profile scopes resolve the same local name, such as - `google.gemma-4-31b-it`, to different profiles without changing the configured - provider model value `google/gemma-4-31b-it`. -- A qualified reference bypasses the selected scope without double qualification; - adding another collection does not change local or qualified resolution. -- Multiple slashes and empty qualified components are rejected in resource IDs - and references, while provider model values retain their existing syntax. -- Prompt shorthand and a profile override use their intended scopes, while - stored default references and omitted selections retain their meaning. -- Scoped views coexist concurrently and share backend capacity with ordinary - calls on the underlying engine. -- Scope creation rejects invalid syntax, accepts well-formed absent namespaces, - and treats empty scope fields as no scope. Missing resources fail at lookup; - qualified references remain usable even when the default namespace is absent. -- Listing without an explicit filter uses the resource's view scope, or lists - all resources when no scope is selected. Explicit named, unqualified-only, - and all-namespaces filters override that default for the current listing only. -- A valid namespace with no matches produces an empty listing. Namespace - filtering respects resource type and exact namespace boundaries. -- Exact-ID calls using valid IDs and intentional overlays retain their behavior. - Migrated catalog references resolve without collisions; migration installs one - prompt per ID with an optional descriptive label. -- Listings preserve full IDs and separate optional prompt version labels. - Qualified results round-trip through either engine or view; unqualified results - round-trip through the engine without a corresponding scope. -- Representative validation, inspection, execution, and smoke-test paths use - the same qualification rules without duplicating every test at every layer. -- Engines use caller-selected definitions without automatically contributing - provider catalogs or PromptKit-owned profiles. Ordinary directory and embedded - configuration sources retain ConfigKit composition semantics. -- PromptKit does not add source-history-based duplicate-namespace rejection or - independently pre-validate collection definitions before the final handoff. - -### Configuration Acceptance - -Exercise typed assembly of multiple collections, relative/nested imports, -source-local header checking and consumption, ordinary release and prompt -metadata, unqualified resources, optional prompt labels, inventory without -release metadata, source-relative content, and intentional partial overrides. -Cover headerless entries/fragments, omitted main inputs and headers, and -resources named `version` in nested mappings. Verify ID-only lookup without -request-version selection or assertions. No automatic provider catalogs or -built-in profiles should be contributed. - -Do not duplicate ConfigKit's generic source validation or enforce domain rules -through historical source inspection. Preserve local inheritance and ordinary -mapping overrides while migrating catalog definitions. - -### Review Evidence - -The 2026-10-16 review against ConfigKit development commit `e66c371` verified -mounted assembly, relative imports and resource contexts, shared header checks, -and this document's YAML assembly through public consumer tests. The 2026-10-17 -review of release v0.8.0 at `137769f` passed maintained consumer examples and -focused public API probes for sequence assembly, duplicate rejection, whole-list -replacement, inheritance limits, header consumption, and final identity validation -boundaries. The latter review informed the decision to retain mappings. These -checks establish the upstream behavior relied on by the implemented PromptKit -feature. - -## Delivery Reference - -No substantive design question remains. ConfigKit v0.8.0 is the selected -dependency. The [implementation plan](implementation.md) records helper -signatures, inventory and optional-label representations, identifier migration -mappings, and completed consumer migration work. Its delivery choices implement -these accepted contracts without adding speculative options or reopening -workflows. - -## Consolidation And Retired Alternatives - -This document replaces the separate namespace roadmap and collection proposal 3. -Their pre-consolidation versions, including considered namespace API alternatives, -remain available at PromptKit commit `ddda2f8`: - -```sh -git show ddda2f8:docs/roadmap/namespaces.md -git show ddda2f8:docs/roadmap/configuration-collections-mounted-fragments.md -``` - -The unselected collection proposals remain available at `b5866dd`: - -```sh -git show b5866dd:docs/roadmap/configuration-collections-current-configkit.md -git show b5866dd:docs/roadmap/configuration-collections-identified-records.md -``` - -Proposal 1 used fully qualified mapping keys. Proposal 2 requested identified -record sequences, identity-based overlays, and record inheritance. ConfigKit -v0.8.0 implements a narrower sequence-composition capability; it does not supply -those identity-based overlays or inheritance. Its sequence feature was evaluated -and not selected for this roadmap. The original upstream feature-request lineage -is recorded in ConfigKit history under -`docs/roadmap/promptkit-identified-records-feature-request.md`; no outstanding -part of that request is a prerequisite here. - -The accepted configuration choices remain: optional main declarations and -if-present checks; source-root `version` reserved without banning that resource -name; removal of request-version machinery; current ConfigKit inheritance; and -flexible source assembly with a small convenience API. This consolidation -records those decisions and their implementation scope; current behavior is -maintained in the canonical format and consumer documents. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index f9387d1..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,852 +0,0 @@ -# Configuration Collections Implementation Plan - -Status: implementation complete; roadmap retirement remains for normal release -preparation. - -Implement the accepted [feature roadmap](configuration-collections.md). That -document owns intent, scope, and selected policy; this document owns API delivery -choices and the ordered implementation steps. Each numbered stage is sized as -one prompt for a Luna coding agent. Complete stages in order. Foundation and -migration stages are parts of one refactor, not separately publishable releases. - -## Baseline And Working Rules - -Reviewed PromptKit baseline: `683689a4f89d0fea6a8e0c6faf5830414adf9c7b`. -It selects ConfigKit `v0.7.1`, both `/v3` provider catalogs at -`v3.20261013.0`, and the completed smoke-test implementation and remediations. -The existing implementation plan described that completed work; Git retains it. -This plan replaces it without reopening the smoke-test design. - -ConfigKit `v0.8.0` resolves to commit -`137769f3f18a370c68444242fb5d88e620e307e5` (the annotated tag object is distinct). -Its released public contracts in `docs/config.md`, release notes, and ADRs -0011–0014 supply the required placement and shared version-check behavior. -The adjacent checkout is a reference, not a committed build dependency. - -Before each stage, inspect status, preserve unrelated work, read the owning -implementation/tests, and follow [development orientation](../development.md). -Apply the [agent](../policy/agents.md), [architecture](../policy/architecture.md), -[testing](../policy/testing.md), [documentation](../policy/documentation.md), -[OSS](../policy/oss.md), and relevant [release](../policy/release.md) policies. -Use the repository's code-discovery instructions. Reuse unchanged readings. -Update affected GoDoc and canonical contracts as each behavior lands; Stage 17 -is a consistency pass, not permission to leave current documentation stale. - -Record completion and validation beneath the stage, then commit each completed, -nonempty unit with a short plain-English message. These stages change PromptKit; -external collection publishing is separate work. Do not push, tag, release, or -introduce committed workspace/replacement files. The sprint requires neither -publication of a new catalog nor a live provider. -Dependency acquisition is setup, separate from offline behavioral validation. - -Maintain [ADR 0001](../adr/0001-configuration-boundary.md): ConfigKit owns source -processing, composition, inheritance, and header checking through its configured -version callback. PromptKit interprets final effective values and their resource -contexts. Do not inspect contributions or source history to enforce domain -validity, including in tests. Explicit operator inspection remains available. - -Preserve execution, retry/repair, credentials, appended messages, capacity, -observations, cancellation, and prepared-handle semantics except for the accepted -identity/version changes. Introduce no installer, implicit discovery, new -provider protocol, cross-collection inheritance resolver, compatibility alias -bundle, live reload, or additional release gate. - -## Delivery Decisions - -These names and signatures settle delivery details left by the feature roadmap. -Private helper names below are implementation guidance, not public invariants. - -### Final Configuration And Runtime Identity - -Use a typed root document with optional `version`, root `backends`, `profiles`, -and `prompts`, and `collections`. Each collection body has optional -`release_version` and the same three resource mappings. Root and collection -resource maps are keyed by local ID; `prompts` maps directly to definitions, -not another version map. Keep ordinary strict ConfigKit decoding and existing -definition rules. Collection bodies are mappings; an empty body is useful. - -Represent source-root `version`, collection `release_version`, and prompt -`version` with `*string` in the typed document. Configure -`Definition.Version.Path = configkit.Keys("version")` and -`SourceVersionModeIfPresent`. The checker accepts absence, accepts exact text -`1` from a string or strict decimal integer, and rejects explicit null or other -texts with a safe ConfigKit diagnostic. ConfigKit owns type checking and -non-main header consumption. Do not synthesize a main input or declaration. -Collection release and prompt labels are opaque strings: nil means absent/null, -and a non-nil empty string remains explicitly declared. Do not trim, order, -interpret as SemVer, or require these labels. - -Keep the retained ConfigKit result in its authored effective shape. Refactor -configuration helpers to receive the effective resource base path rather than -reconstructing it from a runtime ID. Visit root and collection mappings with -their actual paths; after handoff, index each resource under its full public ID. -Do not flatten/re-key the ConfigKit result or make a second load. Presence, -normalization records, content-file origins, secret classification, explanations, -and diffs must keep the effective paths, including `collections/`. - -Create shared private identity helpers under `internal/domain`. Follow the -roadmap's exact grammar: a nonblank unqualified ID has no slash; a qualified -reference has exactly one slash and two nonempty components. A configured local -key is nonblank and slash-free. A collection namespace is nonblank and -slash-free, using the same syntax as a nonempty scope. Preserve spelling, -including otherwise permitted whitespace, dots, case, and punctuation. Add no -other character whitelist. Never apply this grammar to models, paths, URLs, -artifact/input names, session IDs, or descriptive labels. - -Validate final configured resource keys and effective PromptKit references. -Retain existing existence/readiness timing: for example, profile backend -existence is checked as today, but an unselected backend diagnostic default is -not made eagerly resolvable. ConfigKit interprets `extends` as a sibling key, -not a PromptKit qualified reference. PromptKit does not validate or reinterpret -consumed inheritance directives. - -### Prompt Selection And Optional Labels - -Remove `PromptVersion` from public/internal `RunRequest`, -`RequirementInspectionRequest`, and `SmokeTestRequest`. `ValidateRequest`, -`Prepare`, and `PrepareExecution` continue to use `RunRequest`. -Change `InspectPrompt` to: - -```go -func (e *Engine) InspectPrompt(ctx context.Context, promptID string) (*PromptInspection, error) -``` - -All prompt repository, compiled-template, and requirement-snapshot lookups use -ID alone. Remove ambiguity tracking, version selectors, version ordering, and -request assertions. Keep `PromptIdentity.ID`; change `PromptIdentity.Version` -to `*string`. Keep output field names `PromptVersion`, changing their types to -`*string` wherever they describe the selected label: prepared details, run -results, prompt/requirement inspection, execution events, and smoke-test targets. -The internal definition uses `Version *string` with equivalent semantics. -Each returned mutable pointer is owned by its result, including events and -prepared snapshots. JSON `omitempty` omits nil while preserving a declared -empty string. Preserve other existing serialization choices. - -The optional label remains part of captured definition metadata and its existing -definition-hash projection. Copy the pointer; do not dereference it as a cache -or selection key. No cross-release hash-stability promise is introduced. -Derive automatic structured-output schema names from the selected prompt ID -using the existing sanitization, not from a version-selection token. Keep explicit -schema selection and structural validation behavior unchanged. - -### Immutable Scope On The Existing Engine - -```go -type Namespaces struct { - Prompt string - Profile string - Backend string -} - -func (e *Engine) WithNamespaces(namespaces Namespaces) (*Engine, error) -``` - -Return another engine facade over the same assembled engine components with -immutable scope values. An internal engine view may be a shallow copy containing -the same runner, smoke tester, inspection, and inventory collaborators. It does -not invoke `NewEngine` or build another capacity manager. `WithNamespaces` -replaces all three scopes on the new view; empty fields clear those scopes. -The receiver is unchanged. Validate syntax, not namespace existence. Use the -existing invalid-configuration convention for an invalid engine or scope. - -At the internal engine entry boundary, use shared qualification helpers for -caller-supplied IDs. Validate supplied syntax; leave omitted optional IDs empty; -qualify slashless nonempty IDs for the applicable scope; pass qualified IDs -unchanged. The runner continues exact lookup. Do not requalify configured default -profiles, backend references, sibling inheritance, or diagnostic defaults. - -Apply the same policy to `Run`, `Prepare`, `PrepareExecution`, `ValidateRequest`, -`InspectPrompt`, `InspectProfile`, `InspectRequirements`, and all three selection -fields of `SmokeTest`. `ValidateRequest` already delegates to preparation; use -that route rather than adding another validator. `RunPrepared` consumes its -captured selection without qualification. Preserve its existing runner-ownership -check; scope does not introduce another handle owner. `InspectConfiguration` -continues to expose the complete retained configuration, not a filtered/re-keyed -view. Results and observer metadata carry full selected identities. - -### Listing Filters And Inventory - -Use one concrete optional filter argument for profile, prompt, and new backend -listings: - -```go -type NamespaceFilterMode uint8 - -const ( - NamespaceDefault NamespaceFilterMode = iota - NamespaceNamed - NamespaceUnqualified - NamespaceAll -) - -type NamespaceFilter struct { - Mode NamespaceFilterMode - Name string // Used only by NamespaceNamed. -} - -func (e *Engine) ListProfiles(ctx context.Context, filter ...NamespaceFilter) ([]string, error) -func (e *Engine) ListPrompts(ctx context.Context, filter ...NamespaceFilter) ([]PromptIdentity, error) -func (e *Engine) ListBackends(ctx context.Context, filter ...NamespaceFilter) ([]string, error) -``` - -Zero or one filter is accepted; multiple filters or an unknown mode return the -ordinary invalid-request error rather than guessing precedence. Omission and a -zero filter mean default: use that resource's scope, or all resources when it is -unset. Named requires a nonempty valid namespace. Unqualified selects IDs without -a slash; All includes every ID. A non-Named mode ignores `Name`. An explicit -filter replaces, not intersects, the view's default for that call. Keep IDs -full, sort lexically by full ID, and retain existing context, ownership, and -empty-list behavior. Copy optional labels on returned prompt entries. -Qualified results can be reused through any view. To select an unqualified ID -returned by an explicit All/Unqualified filter, use the original engine or a view -without that resource scope; a listing filter never changes execution scope. - -Add the following small discovery surfaces: - -```go -type ResourceKind string - -const ( - ResourcePrompt ResourceKind = "prompt" - ResourceProfile ResourceKind = "profile" - ResourceBackend ResourceKind = "backend" -) - -func (e *Engine) ListNamespaces(ctx context.Context, kinds ...ResourceKind) ([]string, error) - -type CollectionInfo struct { - Namespace string - ReleaseVersion *string -} - -func (e *Engine) ListCollections(ctx context.Context) ([]CollectionInfo, error) -``` - -No kinds means the union of all three resource types. Supplied kinds select their -union; repeats are harmless and unknown kinds are invalid requests. Namespace -enumeration describes resource membership across the captured engine, independent -of view scopes. Return distinct named namespaces sorted lexically; exclude the -unqualified group. Collection inventory likewise covers the entire captured -`collections` mapping, sorted by namespace, including metadata-only and -resource-only entries. It is not filtered by a view and does not include an -invented root collection. Both surfaces use the existing listing-style nil-engine -and context error conventions and return detached values. They do not inspect -historical sources, reopen files, or assert package authenticity. - -Capture sorted inventories once at construction. Namespace buckets or sorted -ranges may avoid repeated full scans for filtered listings. Prefer a single -resource traversal plus sorting and ordinary indexed lookups; no repeated -ConfigKit queries, definition deep copies for identity-only listings, global -caches, or performance/complexity promises in the public API are needed. - -### Collection Input Convenience - -```go -func CollectionInput(namespace string, source configkit.Source) (configkit.Input, error) -``` - -Validate the required namespace syntax and return an input with that source, -`Destination: configkit.Keys("collections", namespace)`, and `Imports: true`. -Invalid namespace syntax returns an error matching `ErrInvalidConfig`. -Do not open/read the source or infer a namespace/filename. ConfigKit still owns -source validity and loading errors. The caller may change the returned input's -imports, filesystem context, or destination, or construct an input directly. -This supports host, embedded, and mapping sources with one helper, without -introducing another loader or options framework. Directory/Filesystem helpers -retain their existing explicit discovery and imports-disabled semantics. -`ProfileSource` and `BackendSource` retain their root-document shapes and can be -mounted with this helper. Direct local-key fragments use explicit ConfigKit -destinations that include the resource category. - -### Migration Choices - -Do not mass-copy the provider inventories into PromptKit or create a mandatory -replacement bundle. Provide synthetic, complete collection examples and guidance -for adapting selected existing catalog configuration. Adjacent repositories and -their publication are outside this sprint. The reviewed pinned catalog assets -have slash-free profile keys, so these explicit mappings are collision-free -within those assets: - -| Previous identity | Documented migration identity | -| --- | --- | -| Backend `openrouter` | `openrouter/default` | -| Backend `rakestrawhome` | `rakestrawhome/default` | -| Each OpenRouter catalog profile `p` | `openrouter/p`, retaining the exact local key `p` | -| Profile `rakestrawhome-gemma-4-31b` | `rakestrawhome/rakestrawhome-gemma-4-31b` | -| Built-in `promptkit-openrouter-standard` | No automatic replacement; optional caller profile `openrouter/standard` | -| Built-in `promptkit-openrouter-auto` | No automatic replacement; optional caller profile `openrouter/auto` | - -These are documented conversions, not installed aliases. Simply mounting old -`/v3` assets does not update their stored backend/profile references; those -references and diagnostic defaults need the explicit edits shown here. - -For an optional caller standard profile, `extends: gemini-flash-latest` is valid -only within the same `openrouter` profiles mapping containing that base. The -caller auto profile may explicitly set backend `openrouter/default` and model -`openrouter/auto`. Neither name becomes a PromptKit promise. Move backend -diagnostic defaults to exact new profile IDs. Preserve endpoint/settings and -provider model strings, including the pinned Gemini model's leading `~`. - -Application-specific IDs require a deliberate collision review, not automatic -slash-to-dot replacement. For a formerly multi-version prompt, its maintainer -chooses the installed definition; examples use the single definition already -present. Put its old version string into optional metadata where useful. Do not -select a winner from a consumer's multi-version history by guessed ordering. -Document exact references, paths, backend capacity identity changes, source-root -header reservations, label null/empty semantics, and the loss of cross-mapping -inheritance. Migration documentation is not a runtime migration tool. - -## Stage 1: Adopt The Released ConfigKit Prerequisite - -Scope: dependency selection and existing integration compatibility only. - -1. Select ConfigKit `v0.8.0` with normal Go module tooling; update `go.mod` and - `go.sum` without filesystem replacements. Verify the selected release commit - and notices; update its attribution entry with the full commit above. -2. Check affected keyed literals and source-inspection handling for upstream - additions. `SourceInfo` is no longer comparable; compare relevant fields if - PromptKit currently relies on comparability. Do not enable new schema behavior. -3. Leave Go's consumer minimum at `1.26.0` unless the selected dependency actually - requires a higher minimum. Leave unrelated dependencies/tooling unchanged. - -Validation: `GOWORK=off go test ./internal/configuration .` and the three offline -examples, including configuration inspection/diff coverage. This stage establishes -the unchanged PromptKit baseline on the new dependency. - -Completion: ConfigKit `v0.8.0` is selected at commit -`137769f3f18a370c68444242fb5d88e620e307e5`; its attribution links and module -checksums are updated. The Go consumer minimum remains `1.26.0`. No -`SourceInfo` comparability-dependent code required adjustment. Validation passed: -`GOWORK=off go test ./internal/configuration .` and -`GOWORK=off python3 scripts/check-examples.py`. - -## Stage 2: Add Shared Resource Naming Primitives - -Scope: small private domain helpers and focused behavior tests. - -1. Implement exact ID parsing/validation, namespace syntax validation, local-key - validation, qualification, and namespace extraction as needed by later stages. - Keep omission policy in operation callers, not a universal required-ID rule. -2. Cover unqualified/dotted names, qualified bypass, missing/empty scope, - malformed slash forms, and preserved spelling. Check that namespace matching - is exact rather than a prefix match. -3. Do not wire names into all public operations yet or export a splitting API. - -Validation: `GOWORK=off go test ./internal/domain`. Test selected naming behavior, -not a helper call sequence or invented character restrictions. - -Completion: private resource-ID, namespace, local-key, qualification, and -namespace-extraction helpers are implemented with focused grammar and spelling -tests. Validation passed: `GOWORK=off go test ./internal/domain` and -`git diff --check`. - -## Stage 3: Make Configuration Processing Path-Aware - -Scope: preparation refactor inside `internal/configuration`; current format stays. - -1. Refactor `captureFields`, `resolveBackends`, `resolveProfiles`, - `validateEffective`, prompt metadata capture, and prompt materialization to - accept/use explicit effective base paths. Preserve current values and errors. -2. Carry a resource's public ID separately from its ConfigKit path. Stop deriving - an effective path by splitting a runtime ID. Use ConfigKit path segments for - literal keys and preserve inherited `content_file` origins. -3. Factor resource-map traversal only as needed to reuse the same processing - for root and collection mappings later. Do not add a generic visitor framework. -4. Prepare the existing provider-secret predicate to recognize actual resource - paths without mistaking arbitrary nested `extra_params` data for resources. - Keep its existing protected-key policy. - -Validation: configuration load, target, prompt-origin, inheritance, and inspection -tests; `GOWORK=off go test ./internal/configuration .`. A behavior-preserving -refactor should not rewrite unrelated expectations. - -Completion: configuration capture, normalization, effective validation, and -prompt materialization now receive explicit ConfigKit base paths. Existing -resource IDs remain independent from those paths; source-origin reads and the -protected provider-key policy are preserved. Validation passed: -`GOWORK=off go test ./internal/configuration .`. - -## Stage 4: Install One Prompt Definition Per ID - -Scope: prompt storage format, materialization, and fixture-format migration. - -1. Change root `prompts` to `map[string]promptConfig`; add nullable - `Version *string` to the raw prompt definition. Materialize one entry per ID, - preserving the label text without normalization or a generated label. -2. Remove nested version indexes, ambiguity tracking, and selection-by-version - from the configuration adapter. Use the actual prompt path without the old - version segment for metadata and message resources. -3. To keep this stage bounded, existing repository method parameters and public - selector fields may remain only until Stage 5. At this adapter boundary their - temporary implementation delegates to ID-only lookup; do not compare them - with labels or create an assertion mode. Existing string-valued output labels - are converted to presence-aware values in Stage 6. These are migration seams, - not new compatibility behavior to retain or publish. -4. Mechanically migrate YAML fixtures, mapping builders, and example configuration - files to the direct definition shape. Move retained version keys to quoted - metadata. Replace tests of parallel versions/ambiguity with one-entry lookup - and ordinary mapping-overlay cases. Keep unrelated validation coverage. - -Validation: `GOWORK=off go test ./...`, including consumers of migrated fixtures. -Test ID-only selection, source-relative content, and inherited/overlaid prompt -settings. Do not add tests preserving temporary obsolete selector parameters. - -Completion: prompts now map directly from each ID to one definition, with an -optional descriptive version label. Configuration lookup and message/input -metadata paths are ID-only; ambiguity handling and version-keyed indexes are -removed. Fixtures, examples, and the canonical format reference use the direct -shape, including layered overlays. Validation passed: `GOWORK=off go test ./...`, -`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`. - -## Stage 5: Remove Request-Side Version Selection - -Scope: compile-directed removal of obsolete selector plumbing; no new lookup rules. - -1. Remove request `PromptVersion` fields and the version parameter from public - `InspectPrompt` and internal prompt repository/snapshot/template interfaces. - Update runner preparation, inspection, requirements, and smoke selection. -2. Update public-to-internal conversion, redacted formatting, fakes, tests, - examples, and call sites. Remove version-without-prompt checks and the - transitional signatures from Stage 4. Preserve ordinary not-found identities. -3. Use the selected prompt ID for generated structured-schema names; retain the - existing sanitization and explicit schema behavior. Do not remove output - metadata fields merely because selectors are gone. -4. Update affected GoDoc now so obsolete selectors are no longer advertised. - -Validation: `GOWORK=off go test ./...` and focused vet of modified packages. -Search declarations/call sites for remaining selection/ambiguity logic; descriptive -label references are valid and should not be blanket-deleted. - -Completion: request-side version selectors and version parameters have been -removed from prompt lookup, inspection, requirement analysis, and smoke-test -selection. Generated structured-schema names now derive from the prompt ID; -result-side descriptive label fields remain. Updated the consumer guide and -GoDoc to reflect ID-only selection. Validation passed: `GOWORK=off go test -./...`, focused `GOWORK=off go vet` for modified packages, -`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`. - -## Stage 6: Carry Optional Prompt Labels Through Results - -Scope: presence-aware metadata across internal values and the facade. - -1. Change internal definition `Version` and all selected-label output fields - identified above to `*string`. Read the raw optional field captured by Stage 4. - Update conversion and ownership copies, hashing projection, requirement - snapshots, observation selection/events, prepared details/results, and smoke - target projection. Optional labels never become lookup keys. -2. Distinguish nil from a declared empty string in inspection/listings and JSON. - Both omitted and null mean no label; a higher-layer or inherited null clears - the label through ordinary ConfigKit composition. -3. Update the example summary types and test expectations. Add focused boundary - coverage for absence, null clearing, empty/exact text, and pointer ownership; - reuse existing report/prepared/event detachment tests where practical. - -Validation: domain, promptdef, configuration, observation, usecase, and root tests; -`GOWORK=off go test ./...` and the offline examples. Verify the existing selected -definition hash still agrees between inspection and preparation. - -Completion: prompt labels are presence-aware from configuration capture through -inspection, listing, requirements, prepared and completed runs, observation -events, and smoke targets. Nil and declared-empty labels remain distinct in JSON; -null overlays clear inherited labels. Returned label pointers are independently -owned, and inspection/preparation retain matching definition hashes. Validation -passed: `GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, -`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`. - -## Stage 7: Enable Shared Specification Checking - -Scope: root declaration policy through ConfigKit's released version API. - -1. Add optional root `version` to the typed document. Configure the version rule - and `SourceVersionModeIfPresent` with the exact checker specified above. - Keep no-main and headerless sources valid. -2. Use ConfigKit's header consumption and source evidence without a source - pre-parser, validation loop, or PromptKit mode option. Nested prompt labels - remain ordinary metadata. -3. Cover accepted string/integer `1`, omitted main/header, rejected null or - unsupported declarations, consumed non-main headers, and an explicit main - declaration retained in inspection. A headerless imported prompt fragment - still undergoes ordinary final-value validation. - -Validation: focused real-ConfigKit tests in `internal/configuration` and public -construction/inspection tests. Do not replicate upstream's full version-type or -import matrix. Run `GOWORK=off go test ./internal/configuration .`. - -Completion: the optional root declaration is checked through ConfigKit's shared -source-version API, accepting only exact string or integer `1` while allowing -absence and preserving the main declaration in inspection. Accepted underlay -declarations are consumed; null and unsupported values produce safe diagnostics, -and headerless payloads still receive final-value validation. Validation passed: -`GOWORK=off go test ./internal/configuration .`. - -## Stage 8: Assemble And Index Final Collection Resources - -Scope: namespaced schema, final resource interpretation, and runtime indexes. - -1. Add `collections` and collection `release_version`/resource mappings to the - typed document. Visit root and collection maps using Stage 3's effective - paths. Validate namespaces/local keys; build full IDs only in runtime indexes. -2. Include backend/profile snapshots and prompt metadata/templates in that - traversal. Keep the retained ConfigKit result structurally unchanged. - Preserve the distinction between root unqualified and collection-local keys. - Migrate maintained qualified-key fixtures into collection/local-key mappings, - updating exact references and inspection paths together. Use ordinary explicit - destinations until the convenience helper lands; do not keep a second root - spelling for qualified identities. -3. Validate effective references with the naming helper and existing existence - policy. Local `extends` stays with ConfigKit. Do not reject repeated namespace - contributions, require collection releases, or qualify stored references. -4. Extend secret-path matching to backend/profile extras beneath collections, - preserving the existing protected-key list. Use exact effective paths for - field presence, normalization, `content_file`, explanations, and diagnostics. -5. Preserve backend-ID capacity policy wiring. The same final backend is one - capacity identity regardless of the sources that contributed its settings. - -Validation: focused real-ConfigKit tests for two collections with duplicate local -names, root resources, partial overrides, inherited relative content, explicit -zero/null settings, and redacted namespaced provider data. Include retained -release metadata, sibling inheritance, headerless collection fragments, resources -named `version` in nested mappings, and header/declaration distinctions. -Run `GOWORK=off go test ./internal/configuration .`. - -Completion: final root and collection resources are interpreted from their -effective ConfigKit paths and indexed under full runtime IDs without re-keying -the retained result. Collection namespaces and local keys are validated; -collection release labels, sibling inheritance, inherited relative content, -presence/null handling, and namespaced secret redaction are covered. Capacity -and profile snapshots use full backend identities. Validation passed: -`GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, and `git diff --check`. - -## Stage 9: Make Application Tests Independent Of Bundled Catalogs - -Scope: owned fixtures and examples that currently assume automatic provider data. - -1. Locate tests/examples using bundled backend/profile IDs, inherited catalog - settings, default smoke profiles, or baseline listing counts. Supply the - explicit synthetic definitions each behavior needs through existing sources. -2. Keep generic test helpers small and local; no full catalog snapshot or shared - test fixture framework. HTTP adapter tests may retain provider adapter names - but use controlled transports and synthetic configuration. -3. Keep catalog-specific assembly tests until Stage 10 removes the integration. - Do not preserve provider inventory assertions as application contracts. - -Validation: the affected root, engine, and usecase tests plus examples. Ordinary -coverage should now express its own backend, profile, and prompt requirements. - -Completion: ordinary application tests now provide the backend, profile, and -smoke metadata they exercise instead of relying on bundled provider entries. -The listing and inheritance contracts use synthetic resources; catalog-specific -assembly tests remain isolated for the later integration removal. Maintained -examples already supplied their own definitions and needed no changes. -Validation passed: `GOWORK=off go test ./...`, -`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`. - -## Stage 10: Remove Automatic Catalogs And Built-In Profiles - -Scope: engine composition, dependency/notices cleanup, and the retired promise. - -1. Remove `catalog.Underlays()` from `internal/engine.New`; load only caller - underlays and optional main. Preserve their order and strict-group semantics. - Empty configuration constructs an engine with empty resource inventories. -2. Delete `internal/catalog`, its owned profiles, and integration-specific tests - after checking references. Do not delete generic file-discovery helpers. -3. Remove both catalog module requirements and unused checksums through module - tooling. Do not update/publish those repositories or copy their inventories - into PromptKit as a replacement. -4. Update attribution for actual retained material; remove catalog dependency - entries when nothing incorporated remains. Keep a notice if other copied - material still requires it. Do not strip a shared license text still in use. -5. Retire the release policy's catalog-update exception and literal built-in-name - promise. Apply ordinary compatibility rules to PromptKit format/API changes; - independently selected configuration is no longer a bundled release inventory. - Update architecture/internal inventory and current format documentation for - the removed component. Preserve historical release notes. - -Validation: `GOWORK=off go test ./...`; verify empty inventories, explicit former -backend names still work when configured, and inference smoke tests retain their -selection/capability behavior with caller-supplied metadata. Review `go.mod`, -`go.sum`, imports, and notices for residual automatic catalog coupling. - -Completion: engine construction now composes only caller-selected underlays and -the optional main input; an empty configuration has empty profile and prompt -inventories. Removed the catalog adapter and its tests, provider catalog module -requirements and checksums, catalog attribution entries, and the built-in -profile compatibility exception. Current architecture, source, format, and -consumer documentation now describe caller-supplied definitions; historical -release notes remain unchanged. Explicit OpenRouter and Rakestrawhome backend -IDs continue to work when configured. Validation passed: `GOWORK=off go test -./...`, `GOWORK=off go vet ./...`, `GOWORK=off python3 -scripts/check-examples.py`, and `git diff --check`. - -## Stage 11: Add The Explicit Collection Input Helper - -Scope: the single public convenience function and its GoDoc. - -1. Implement `CollectionInput` exactly as specified above, reusing namespace - validation and existing facade error conventions. It creates an input, not - an engine and not a configuration pre-parser. -2. Keep raw ConfigKit inputs and existing directory/filesystem/programmatic - helpers usable. Do not reinterpret their IDs or introduce root discovery. -3. Add one native-file and one embedded/mapping assembled case with entry - imports and a later deliberate override. Verify resource contexts survive. - -Validation: source-helper contract tests and focused configuration/root tests. -Verify helper construction itself performs no source read using an existing -controlled source seam where needed, rather than adding a production interface. - -Completion: added `CollectionInput` with shared namespace validation, a -`collections/` destination, and imports enabled without reading the -source. Contract tests cover invalid namespaces and deferred source loading; -native-file and embedded-filesystem imports assemble successfully, preserve -their source context, and accept a later override. Validation passed: -`GOWORK=off go test ./...`, `GOWORK=off go vet .`, and `git diff --check`. - -## Stage 12: Add Immutable Scopes To Ordinary Execution - -Scope: public `Namespaces`/`WithNamespaces` and shared engine-boundary routing. - -1. Add the public facade and internal immutable view. Replace all scopes on each - new view without mutating its receiver. Reuse assembled collaborators. -2. Route caller prompt/profile selections in `Run`, `Prepare`, and - `PrepareExecution` through shared qualification. `ValidateRequest` follows - its current preparation route. Stored defaults and omitted IDs stay untouched. -3. Preserve resolved full IDs through prepared details, results, events, errors, - and selected backend identity. Leave `RunPrepared` captured and single-claim; - do not add a scope mismatch check or rebuild shared state. -4. Add public behavior tests for two independent prompt/profile scopes, - qualified bypass, no fallback, absent well-formed namespaces, and invalid - scope syntax. Assert the receiver retains its original behavior. - -Validation: root/prepared/usecase tests and focused race-enabled scope tests. -Use injected generation or a controlled HTTP boundary; never a live endpoint. - -Completion: added immutable `WithNamespaces` views that share the assembled -runner and captured configuration. Ordinary `Run`, `Prepare`, -`PrepareExecution`, and the preparation-backed `ValidateRequest` now qualify -slashless caller prompt/profile IDs while preserving qualified IDs, captured -defaults, full result/event identities, and prepared-handle ownership behavior. -Tests cover independent views, replacement and clearing, no fallback, absent -well-formed namespaces, invalid syntax, and concurrent preparation. Validation -passed: `GOWORK=off go test ./...`, focused `GOWORK=off go test -race . -run -'TestWithNamespaces' -count=1`, `GOWORK=off go vet ./...`, and `git diff --check`. - -## Stage 13: Apply Scopes To Inspection And Diagnostics - -Scope: complete the existing operation inventory without a second resolver. - -1. Route `InspectPrompt`, `InspectProfile`, and `InspectRequirements` through the - same helpers; resolve each request's explicitly supplied prompt/profile fields - independently. Keep configured references exact. -2. Route `SmokeTest`'s supplied PromptID, ProfileID, and BackendID by their own - scopes before ordinary diagnostic selection. Keep its precedence, optional - omissions, backend defaults, factual reports, and errors unchanged. -3. Keep `InspectConfiguration`/Diff global with effective ConfigKit paths. - Do not qualify schema paths, provider models, input identifiers, or overrides. -4. Review all public resource-selection entry points for omissions. Validate - malformed supplied reference syntax through the existing invalid-argument - boundary; retain ordinary not-found behavior for well-formed missing IDs. - -Validation: representative public inspection/requirements/validation/smoke cases -prove shared behavior and full identities. Reuse naming-helper tests instead of -duplicating the full syntax matrix in every API test. Run relevant root/engine -tests and focused race checks. - -Completion: prompt, profile, and requirement inspections now qualify each -explicit caller selection independently, and smoke diagnostics qualify supplied -prompt, profile, and backend IDs before applying their unchanged selection -precedence. Captured references and global configuration inspection remain -unchanged. Contract tests verify full returned identities, exact configured -prompt defaults and backend references, malformed caller IDs, and ordinary -not-found behavior. -Validation passed: `GOWORK=off go test ./...`, focused -`GOWORK=off go test -race . -run 'TestWithNamespaces' -count=1`, -`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`, -and `git diff --check`. - -## Stage 14: Add Backend Listing And Namespace Filters - -Scope: the existing identity-inventory path, not the execution runner. - -1. Add the filter declarations and optional arguments to profile/prompt listing; - add backend identity listing through configuration, internal engine, and facade. - Existing no-filter engine calls still enumerate everything. -2. Resolve default/explicit filters once per listing using the relevant view - scope. Validate named filters and modes; filter exact namespace membership. - Use captured sorted inventories, optionally indexed by namespace. -3. Return full IDs and detached optional prompt labels, with existing cancellation - and empty-result behavior. Do not materialize definitions for identity listing. - -Validation: extend listing tests with scoped defaults, explicit named/All/ -Unqualified overrides, no-scope defaults, absent namespaces, same-prefix namespace -names, full-ID round trips (and unqualified selection through the original -engine), invalid filters, ownership, and canceled contexts. Use a compact matrix -across resource types, not three copies of all tests. Run configuration/root tests. - -Completion: prompt and profile listings now accept the documented optional -namespace filters, and backend identities are captured and listed through the -same path. Default filters use the view's per-resource scope, explicit filters -replace it, results retain full IDs and detached prompt labels, and ordinary -no-filter calls still return the full inventory. Contract tests cover three -resource types, exact namespace boundaries, round trips, invalid filters, -ownership, and cancellation. Validation passed: `GOWORK=off go test ./...`, -focused `GOWORK=off go test -race . -run -'TestScopedIdentityListingsApplyNamespaceFilters|TestListIdentitiesAreOwnedAndSafeForConcurrentUse|TestListIdentitiesPreserveContextErrorsWithoutPartialResults' -count=1`, -`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`, -and `git diff --check`. - -## Stage 15: Expose Namespace And Effective Collection Inventories - -Scope: the two discovery APIs and construction-time inventory capture. - -1. Capture namespace sets by resource kind from final indexed IDs; implement - `ListNamespaces` with the chosen union, kind validation, and global semantics. -2. Capture `CollectionInfo` values from final `collections` declarations. Preserve - absent/null versus empty/exact release labels, including metadata-only and - resource-only collections. Copy label pointers on output. -3. Delegate through the internal engine's narrow inventory boundary and root - facade. Do not derive collection releases from source header evidence or - confuse resource namespace membership with collection declarations. - -Validation: one representative inventory fixture covers cross-kind membership, -deduplication/sorting, unqualified exclusion, metadata-only collections, -scope-independent enumeration, effective overrides, cancellation, and ownership. -Run configuration/root tests; no file access or generation should be needed. - -Completion: added global resource-namespace and effective collection inventory -APIs. Namespace results are distinct, sorted, kind-selectable, and exclude -unqualified resources; collection results preserve absent/null versus empty -release labels and include metadata-only declarations. Inventories remain -independent of namespace views and return caller-owned values. Validation passed: -`GOWORK=off go test ./...`, focused -`GOWORK=off go test -race . -run 'TestResourceAndCollectionInventories' -count=1`, -`GOWORK=off go vet ./...`, `GOWORK=off python3 scripts/check-examples.py`, -and `git diff --check`. - -## Stage 16: Maintain Complete Consumer And Migration Examples - -Scope: runnable/synthetic examples and precise migration guidance. - -1. Keep the existing prepare/run/observe examples valid with ID-only selection - and optional labels. Extend the preparation example, or its existing input - assets, to demonstrate two collections, explicit input selection, a scoped - view, filtering, and operator inventory without a provider call. Keep the - run/observe examples focused on their distinct existing workflows. -2. Add small complete collection entry/fragments under `examples/`, with optional - headers/releases, source-relative message content, exact stored references, - sibling inheritance, and an intentional later partial override. Use synthetic - endpoints/models rather than another maintained provider catalog. -3. Add `docs/consumers/configuration-migration.md` explaining the breaking format - and signatures, the explicit mapping table above, optional caller standard/auto - choices, and deployment selection responsibility. Do not promise arbitrary - consumer migrations are automatic or validate upstream provider availability. -4. Update semantic example checks only for the changed example outputs. If - script logic changes, run its focused checks; do not rewrite release tooling. - -Validation: `GOWORK=off python3 scripts/check-examples.py`. Assemble the maintained -collection example through real ConfigKit and PromptKit. Keep copied material's -actual attribution obligations aligned if any is introduced. - -Completion: the offline preparation example now explicitly composes `support` -and `review` collections, including an imported source-relative message file, -sibling profile inheritance, and a later partial profile override. It prepares -an explicitly selected prompt/profile through a scoped view and reports global -namespace/collection inventories plus a namespace-filtered prompt listing. The -synthetic entries demonstrate present and absent source/release metadata; the -run and observe examples remain focused on their existing workflows. Added a -consumer migration guide with the format/API changes, identity mappings, -caller-owned optional profiles, and deployment-selection responsibility. -Validation passed: `GOWORK=off go test ./...`, `GOWORK=off go vet ./...`, -`GOWORK=off python3 scripts/check-examples.py`, and `git diff --check`. - -## Stage 17: Finalize Canonical Contracts And Migration Documentation - -Scope: documentation consistency after the APIs and examples exist. - -1. Update GoDoc for all changed/new public declarations, errors, optional labels, - ownership, scopes, and listing behavior. Do not leave obsolete selectors or - an implication that a descriptive label controls execution. -2. Update `docs/formats.md` for the final schema, specification header policy, - source-root reservations, one installed prompt, exact references, local - inheritance, and explicit provider configuration. Keep current contracts here, - not in the migration guide or roadmap. -3. Update `docs/consumers/pkg-promptkit.md`, README quickstart/navigation, - `docs/internal/sources.md`, internal inventory, and affected runner/rendering/ - observation documents. Explain unchanged resource contexts and operator-only - provenance use. Remove live claims about automatically bundled inventories. -4. Review architecture, release policy/procedure, development reading guidance, - and attribution for obsolete catalog assumptions. Revise only affected local - statements; do not rewrite upstream policy language unnecessarily or weaken - ordinary compatibility promises. Do not revise historical release notes or - immutable ADR decisions. - -Validation: review changed Markdown links and example references, check GoDoc -against declarations, and run `git diff --check`. Release notes and version/tag -selection belong to the later requested release preparation, not this stage. - -Completion: canonical format, consumer, README, internal, and public GoDoc -contracts now describe collection identity, source headers and origins, -namespace-scoped selection, global inventories, descriptive labels, and -caller-supplied provider configuration. Reviewed architecture, release policy -and procedure, development guidance, and attribution; no additional local -changes were needed, and historical release notes and immutable ADRs remain -unchanged. Validation passed: local Markdown link targets and example references -were reviewed, affected GoDoc was checked against declarations, and -`git diff --check`. - -## Stage 18: Verify Cross-Boundary Behavior And Close The Sprint - -Scope: focused integration gaps and final regression validation, not new features. - -1. Trace the feature roadmap's acceptance criteria to focused existing/new tests. - Add only missing assembled-boundary coverage. One synthetic fixture can prove - independent collection names, scoped execution/inspection, full-ID filtering, - exact defaults, optional labels, and inventory without duplicating every - assertion at each internal layer. -2. Use a controlled concurrent model boundary to verify ordinary and differently - scoped calls share the same configured backend's capacity. Reuse existing - admission/cancellation tests for queue mechanics. Verify prepared selections - remain captured when another view is used; no namespace is mutable global state. -3. Confirm final-value-only validation, source-local headers versus ordinary - metadata, retained inherited/overlaid content origins, and redaction under - namespaced paths. Final inspection still describes ConfigKit's original - effective shape rather than a rewritten runtime index. -4. Remove all temporary selector signatures/adapters from the migration stages. - Review source/tests/docs for old version-key maps, version selection/assertions, - automatic catalog contributions, mandatory built-in names, and local module - replacements. Historical notes and optional metadata references may remain. -5. Run the final offline checks once for the resulting content: - - ```sh - GOWORK=off go test -race ./... - GOWORK=off go vet ./... - GOWORK=off python3 scripts/check-examples.py - git diff --check - ``` - - Format changed Go files and validate affected Markdown links. Review the full - diff/status for unintended files and private data; commit the completed unit. - Mark this plan and the feature roadmap complete, leaving retirement to normal - release preparation. Do not publish, wait for CI, probe providers, or introduce - a catalog-release dependency to finish implementation. - -Completion: acceptance coverage was traced across public collection assembly, -scoped execution and inspection, full-ID listings and inventories, exact stored -defaults, optional prompt labels, prepared selection capture, source headers and -origins, final-value validation, namespaced redaction, and retained effective -configuration shape. Added a controlled concurrent regression test proving an -ordinary call and a differently scoped call share one collection backend's -capacity. Audited current source, tests, and docs for obsolete selectors, -automatic catalogs, built-in profile promises, and local module replacements; -remaining version references are descriptive labels or migration guidance. -Marked the feature roadmap complete; normal release preparation owns retirement. -Validation passed: `GOWORK=off go test -race ./...`, `GOWORK=off go vet ./...`, -`GOWORK=off python3 scripts/check-examples.py`, `gofmt`, local Markdown link -checks, and `git diff --check`. - -## Open Questions - -None required to begin implementation. The feature roadmap's policy decisions -are settled; the delivery decisions above specify the public shapes and migration -path. A newly discovered upstream incompatibility or genuine change of scope -should be reported with a recommendation and viable alternative rather than -silently changing the accepted configuration boundary or adding restrictions. diff --git a/docs/roadmap/remediation.md b/docs/roadmap/remediation.md deleted file mode 100644 index 496d41f..0000000 --- a/docs/roadmap/remediation.md +++ /dev/null @@ -1,190 +0,0 @@ -# Configuration Collections Implementation Review Findings - -Status: all three findings completed 2026-10-17. - -Reviewed implementation after `89af2a1` through `0221cd4` against the accepted -[feature roadmap](configuration-collections.md), [implementation plan](implementation.md), -[configuration-boundary ADR](../adr/0001-configuration-boundary.md), current -GoDoc, and applicable repository policies. This replaces the previous completed -smoke-test remediation log; its findings and completion records remain in Git. - -## Architecture Assessment - -The targeted architecture is substantially in place: - -- ConfigKit v0.8.0 composes caller-selected inputs and applies the shared optional - specification-header policy. PromptKit interprets final effective values and - resource contexts without flattening the retained configuration or replaying - source history. -- Root resources retain unqualified identities; collection resources use full - namespace/local-key identities in captured runtime indexes. Stored references - remain exact, and inheritance stays with ConfigKit. -- Scoped engine views share the assembled runner, model clients, inspection, - inventory, and capacity machinery. Qualification is centralized at engine - entry points; prepared execution retains its captured selections. -- Prompt selection is ID-only, with optional descriptive version labels. - Namespace-filtered prompt/profile/backend listing, resource namespace discovery, - and effective collection inventory are available. -- Automatic catalog dependencies and built-in profiles are removed. Consumer - migration guidance and offline collection examples are present. - -The findings below are focused ownership, construction-efficiency, and contract -documentation corrections, not reasons to redesign the feature. No additional -unrequested runtime restrictions were identified in the reviewed paths. Finding 3 -addresses a retained public guarantee that no longer describes scoped views. - -## Finding 1: Share Origin Metadata Across Collection Materialization - -Priority: P2. Status: completed 2026-10-17. - -Locations: [load.go](../../internal/configuration/load.go), lines 257–262, and -[materializePrompts](../../internal/configuration/prompts.go), lines 84–95. - -The loader now calls `materializePrompts` separately for every collection. Each -call creates its own lazy metadata cache and calls `state.result.Metadata()` -when it encounters a file-backed message. In the selected ConfigKit v0.8.0, -`Result.Metadata` defensively clones the whole metadata snapshot, including -field/history maps and paths. The cache is consequently reused within one -collection but discarded before the next. - -With C collections requiring origins and M total metadata entries, this adds -O(C × M) copying during construction; when resource count grows with collection -count, this portion becomes quadratic. Before the collection refactor, the same -cache covered the single prompt-materialization pass. - -An offline diagnostic benchmark used 4, 16, and 64 collections, each containing -one prompt with one `content_file`, all reading a tiny in-memory file. A single -iteration per case reported: - -| Collections | Allocated bytes per construction | -| --- | ---: | -| 4 | 1,662,192 | -| 16 | 10,267,520 | -| 64 | 104,352,336 | - -An allocation-space profile across those diagnostic loads attributed about 79% -of allocated space to ConfigKit metadata cloning reached through -`materializePrompts`' origin callback. These are illustrative local measurements, -not stable thresholds or a claim about all constructor workloads. - -Required outcome: reuse one construction-wide lazy metadata snapshot or an -equivalent narrow lookup across root and collection prompt materialization. -Preserve exact effective paths, inherited origins, resource contexts, and -handle closure. Keep the change local and construction-scoped; no new source -validation, global cache, or upstream API extension is needed. - -Validation: retain origin/inheritance/content-capture coverage with more than -one file-backed collection, and use a small scaling benchmark to verify the -repeated full-snapshot cost is removed. Do not make exact allocations, elapsed -times, or private helper call order a new contract. - -Completion: prompt materialization now traverses root and collection resources -within one construction-scoped pass sharing the lazy origin metadata snapshot. -Extended the imported/inherited-content test across distinct collection resource -contexts and added a scaling benchmark without allocation or timing assertions. -At 64 collections the same benchmark dropped from 104,574,200 to 21,239,712 -allocated bytes per construction in single-iteration local measurements. -Validation passed: focused origin tests and benchmark, race-enabled configuration -and public API tests, configuration vet, and `git diff --check`. - -## Finding 2: Detach The Optional Label In Internal Prepared Snapshots - -Priority: P3. Status: completed 2026-10-17. - -Location: [clonePreparedRun](../../internal/usecase/prepared_execution.go), -lines 226–244; also its use by `PrepareExecution` and `PreparedExecution.Details`. - -`domain.PreparedRun.PromptVersion` changed from a string to `*string`, but -`clonePreparedRun` still performs only the initial struct assignment for that -field. Its returned snapshot shares the label pointer with its source. -`PrepareExecution` uses this copier to produce retained details from the -execution snapshot, and `Details` uses it again for each returned copy. - -A temporary package-level probe constructed a prepared handle with label -`original`, obtained its internal `Details`, changed that result's label to -`mutated`, then obtained `Details` again. The second result incorrectly contained -`mutated`. This contradicts the internal method's fresh-copy contract and the -planned ownership of optional metadata. The same copier leaves retained details -and execution metadata sharing this pointer. - -The public facade's `fromDomainPreparedRun` currently copies the pointer again, -so public callers remain protected by that additional boundary. This is an -internal ownership gap, not a reproduced public mutation or credential leak. - -Required outcome: copy the optional label with the existing nil-aware string -copy helper when cloning prepared runs. Retain absent versus declared-empty -semantics and the existing public ownership protection; a cloning-framework -refactor is unnecessary. - -Validation: extend the existing owning use-case prepared-details test with a -non-nil version label. Mutation of one returned internal snapshot must not change -retained details, the execution snapshot, or a later result. Reuse existing -nil/empty public metadata coverage rather than reproducing it at every layer. - -Completion: the prepared-run copier now detaches the optional version label with -the existing nil-aware helper. Extended the owning prepared-execution snapshot -test to mutate source and returned labels and verify later details, execution -metadata, and post-execution details remain unchanged. The regression failed -before the fix and passes afterward. Validation passed with `GOWORK=off`: -the focused regression, `go test -race ./...`, `go vet ./...`, all three offline -examples, and `git diff --check`. - -## Finding 3: Document Engine Ownership In Terms Of The Shared Assembly - -Priority: P3. Status: completed 2026-10-17. - -Location: [Engine GoDoc](../../engine.go), lines 85–101; related -[prepared-handle GoDoc](../../prepared_execution.go) and -[WithNamespaces GoDoc](../../namespaces.go). - -The Engine declaration still says that each Engine owns independent -backend-capacity pools. `WithNamespaces` now returns another `*Engine` sharing -the same runner and capacity manager, as expressly required by the feature. -The new `TestScopedCollectionRunsShareBackendCapacity` demonstrates this sharing; -the runtime is correct, but the public declaration promises the opposite for -these returned Engine values. - -Prepared handles are likewise described as bound to the Engine that created -them, without clarifying that namespace views share that underlying owner. -The current public tests intentionally run a scoped handle through the original -engine. Documentation should make that distinction clear rather than imply -per-facade capacity or handle ownership. - -Required outcome: distinguish independently constructed engines from namespace -views of one assembled engine. State that views share clients/capacity and the -prepared-execution owner, with selections already frozen in prepared handles. -Preserve independent ownership between separate `NewEngine` constructions. -Do not change the implementation to satisfy the stale wording or introduce -scope-mismatch rejection. - -Validation: review the affected GoDoc together. Reuse the passing shared-capacity -and cross-view prepared-handle tests; no new behavior or test matrix is needed. - -Completion: Engine, WithNamespaces, PrepareExecution, RunPrepared, and the -prepared-handle GoDoc now distinguish independent constructions from shared -namespace views. They describe shared clients/capacity and handle ownership, -with captured selections unchanged by the invoking view. No runtime behavior -was changed. Reviewed rendered `go doc` output; the existing shared-capacity and -cross-view prepared-handle tests and `git diff --check` passed. The full -race-enabled suite, vet, and offline examples from Finding 2 cover the unchanged -implementation. - -## Review Validation - -Passed on the reviewed committed tree with `GOWORK=off`: - -- `go test -race ./...`; -- `go vet ./...`; -- `python3 scripts/check-examples.py`. - -Source review covered the configuration handoff, header policy, paths/origins -and sensitivity, root/collection indexing, optional metadata copying, all scoped -selection paths, listing/inventory behavior, shared capacity, dependency removal, -and migration contracts. Representative existing tests were inspected as well as -executed; the green suite does not cover the internal label-aliasing probe. - -The package-level ownership probe failed as described in Finding 2. The -construction benchmark and allocation profile support Finding 1. Both probes -used temporary Go test overlays outside the repository; neither changes the -implementation or adds permanent tests. No live provider or real credential -was used. Only this remediation document is changed by the review. diff --git a/docs/roadmap/smoke-tests.md b/docs/roadmap/smoke-tests.md deleted file mode 100644 index c374748..0000000 --- a/docs/roadmap/smoke-tests.md +++ /dev/null @@ -1,592 +0,0 @@ -# Explicit Smoke Test Diagnostics - -Status: accepted target state. Implementation has not started. - -This roadmap replaces the earlier capability proposal and incorporates the -subsequent discussion. It defines the target feature, not a staged implementation -plan or current public API. The companion [implementation plan](implementation.md) -owns the ordered work and delivery mechanics. - -## Purpose And Ownership - -Provide reusable, explicitly invoked diagnostics for endpoint reachability, -authentication, plain-text inference, and structured-output inference. Consumers -can build setup and diagnostic workflows without maintaining their own synthetic -prompts, schemas, and provider probes. - -PromptKit is opinionated about what each test checks and how it operates. It owns -fixed content, provider adapters, evaluation rules, and neutral factual reports. -Consumers select tests, targets, and supported execution settings; they interpret -the results and decide what action to take. There is no overall success, -application-readiness verdict, or configurable criteria framework. - -For structured output, report whether structured output was requested, a response -was received, generated content was valid JSON, and that JSON matched the fixed -test schema. Four positive observations do not prove provider enforcement; no -field attempts to infer it. - -For authentication, distinguish credentials supplied with the dispatched request, -a reachable authentication endpoint, and credentials accepted by that operation. -A reachable endpoint can reject credentials, and a transport failure can leave -acceptance undetermined. Authentication is inapplicable on the maintainer's -credential-free RakestrawHome deployment; that is neither an error nor a failure. - -## Single-Sprint Delivery Objective - -The staged implementation must be completable from start to finish in one sprint, -including built-in defaults, tests, and documentation. It must not require -intermediate human-review approvals, upstream releases, external publication, or -live-provider access to progress or establish completion. Automated checks and -focused local commits remain part of ordinary implementation. - -Built-in diagnostic metadata belongs in the local OpenRouter and RakestrawHome -catalogs, with their validated revisions identified by local candidate tags. -There is no intermediate PromptKit-owned metadata underlay or requirement for -separate loader/catalog publication before integration. - -A temporary, uncommitted Go workspace may support coordinated development. -The completed integration uses normal module requirements and Go-generated -checksums and is validated with `GOWORK=off`, without filesystem replacements. -Exact local tag contents can be supplied through a temporary filesystem module -proxy, without public HTTPS availability or upstream publication. Local tags -alone do not make Go discover adjacent checkouts. -Keep workspace/proxy artifacts outside committed repository contents, following -the [repository-independence policy](../policy/architecture.md#repository-independence). - -PromptKit's completed sprint commits may reference those not-yet-published -catalog versions in `go.mod` and their corresponding Go-generated checksums in -`go.sum`. Implementation completion and local validation do not claim that other -machines can fetch them. Public distribution is separate: consumers eventually -need those exact contents available through their configured module sources. - -Normal release preparation and publication follow implementation separately; -this objective does not bypass the repository's release process or authorize -publication during the sprint. - -## Current Baseline - -At commit `d4676bc`, PromptKit has no dedicated live diagnostic operation. -[`ValidateRequest`](../consumers/pkg-promptkit.md#validate-a-concrete-request) -checks a concrete application request without model generation. `Run` executes -that request and may structurally validate and repair its output. These operations -remain unchanged; smoke tests need not validate an intended application request. - -The [runner](../internal/runner.md) shares captured prompt/profile selection and -execution-target resolution. The [generation coordinator](../internal/generation.md) -handles attempts, retries, usage, and active backend permits; the -[model client](../internal/llm.md) handles transport and decoding. Ordinary `Run` -requires an application prompt and returns no partial result on operational -failure, so calling it directly is not sufficient for the proposed diagnostic API. - -Backend configuration has no diagnostic endpoints, capabilities, or default test -profile. New fields must follow [ADR 0001](../adr/0001-configuration-boundary.md): -ConfigKit owns loading and composition; PromptKit interprets and validates final -effective values. Provenance stays available for explicit inspection and is not -an execution-time resolution or validation mechanism. - -## Public Operation And Request - -One synchronous call runs one independently selected operation: - -```go -func (e *Engine) SmokeTest(ctx context.Context, req SmokeTestRequest) (*SmokeTestReport, error) - -type SmokeTestKind string - -const ( - SmokeTestReachability SmokeTestKind = "reachability" - SmokeTestAuthentication SmokeTestKind = "authentication" - SmokeTestTextInference SmokeTestKind = "text_inference" - SmokeTestStructuredOutput SmokeTestKind = "structured_output" -) - -type SmokeTestRequest struct { - Kind SmokeTestKind - BackendID string - PromptID string - PromptVersion string - ProfileID string - Execution *ExecutionTargetOverride - APIKey string `json:"-"` - TransientRetries *int -} -``` - -The request supplies no variables, inputs, appended messages, schema, or output -contract. An application prompt is a profile-selection aid only: its messages, -session template, input requirements, and output contract are not executed. -Consumers can compose several calls themselves; custom application tests remain -available through ordinary `Run`. - -Reuse existing `ExecutionTargetOverride` semantics. Inference uses its endpoint, -model, numeric, reasoning, timeout, credential-source, and provider-extra controls. -Backend-only probes use endpoint, credential-source, and timeout settings as -applicable; they do not require model completeness or execute generation merely -because a model setting was supplied. Profile settings remain the inference -baseline unless overridden; no silent cheaper-model substitution is proposed. - -Direct credentials take precedence over environment credentials and follow -existing request-scoped ownership and redacted-formatting conventions. Optional -missing/blank credentials are omitted from the upstream request. Authentication -can test the resulting unauthenticated request rather than reject missing keys -locally. If an explicitly required credential prevents inference dispatch, report -that local diagnostic finding without pretending the provider rejected it. -Ordinary execution's required-credential behavior remains unchanged. - -## Selection And Routing - -Preserve the accepted inference selection order: - -| Caller selection | Execution profile | -| --- | --- | -| Prompt ID only | Selected prompt version's default profile | -| Profile ID only | Explicit profile | -| Both | Explicit profile overrides the prompt default | -| Neither | Explicit backend's configured default smoke-test profile | - -Reuse exact identity and omitted-version semantics. An invalid explicit selection -does not fall back to a backend default. Missing backend/default selection is an -error when inference needs it, not a reason to substitute an arbitrary profile. -Backend-only probes require neither a default profile nor unrelated prompt data. - -Whenever inference selects a profile, that profile supplies the resolved endpoint, -credential settings, and backend/capacity identity, subject to ordinary execution -overrides. This applies equally to explicit profiles, prompt defaults, and -backend-default profiles. A supplied `BackendID` is a selection fallback, not a -constraint or a request to rebind the selected profile's backend identity. - -For example, if backend A names a default profile that resolves to backend B, -inference uses B's resolved target and capacity identity. If that default profile -is endpoint-only, inference retains an empty backend identity. Report A as the -default-selection source and B (or the empty identity) as the effective backend. -No recursive backend-default lookup or mismatch rejection is introduced. - -For backend-only probes, an explicit or prompt-selected profile likewise supplies -the target and its backend's diagnostic metadata. Otherwise `BackendID` selects -the backend directly, without consulting its default inference profile. The -profile wins when both selectors are supplied, even when they name different -backends. - -An endpoint-only profile remains usable for inference and reachability without -an inferred backend identity or provider-specific authentication adapter. -Endpoint overrides retain the effective backend's ordinary capacity identity. -Backend-only probes can resolve profile endpoint/credential metadata without -requiring its model. Prompt selection uses captured identity/default-profile -metadata without rendering or loading the application's artifacts or schemas. - -## Report Shapes - -Use factual Boolean observations with presence, rather than a caller-configurable -list of criteria. These Go-facing values need no stable JSON contract initially. - -```go -type SmokeTestReport struct { - Kind SmokeTestKind - Supported bool - Applicable *bool - Completed bool - Reason SmokeTestReason - Target SmokeTestTarget - Facts SmokeTestFacts - Timing SmokeTestTiming - Attempts []SmokeTestAttempt - Usage ObservedTokenUsage - Failure ExecutionFailureKind -} - -type SmokeTestTarget struct { - BackendID, ProfileID, PromptID, PromptVersion string - DefaultSourceBackendID string // backend whose default selected the profile, if any - SelectionSource string // "backend_default", "profile", "prompt", or "backend" - Endpoint string // initially resolved diagnostic destination - EffectiveSettings ExecutionTarget - SettingsPresence ExecutionTargetPresence -} - -type SmokeTestFacts struct { - CredentialsSupplied *bool - EndpointReachable *bool - CredentialsAccepted *bool - ResponseReceived *bool - GenerationCompleted *bool - TextReturned *bool - ExpectedTextMatched *bool - StructuredOutputRequested *bool - ValidJSON *bool - SchemaMatched *bool -} - -type SmokeTestTiming struct { - StartedAt time.Time - Duration time.Duration - CapacityWait *time.Duration - RequestDuration *time.Duration - RetryDelay *time.Duration - EvaluationDuration *time.Duration -} - -type SmokeTestAttempt struct { - Dispatched bool - InitialEndpoint string // resolved destination before redirects - Endpoint string // last observed request destination; facts refer to this destination - HTTPStatus *int - Facts SmokeTestFacts - Timing SmokeTestTiming - Failure ExecutionFailureKind - Usage ObservedTokenUsage -} - -type SmokeTestReason string - -const ( - SmokeTestReasonNone SmokeTestReason = "" - SmokeTestReasonAuthenticationNotApplicable SmokeTestReason = "authentication_not_applicable" - SmokeTestReasonProbeNotConfigured SmokeTestReason = "probe_not_configured" - SmokeTestReasonHTTPDiagnosticsUnavailable SmokeTestReason = "http_diagnostics_unavailable" - SmokeTestReasonRequiredCredentialMissing SmokeTestReason = "required_credential_missing" - SmokeTestReasonAuthenticationResponseUnrecognized SmokeTestReason = "authentication_response_unrecognized" -) -``` - -`Supported` records whether an implementation is available for the target and -selected client. `Applicable=false` records known inapplicability; nil leaves -applicability undetermined, for example when no authentication probe is configured. -Missing probe metadata does not imply that a provider accepts anonymous access. -`Reason` is a named factual code, not application severity or free-form diagnostic -text. The constants distinguish, respectively, no additional reason, explicitly -inapplicable authentication, absent probe metadata, unavailable HTTP diagnostics -for the selected client, a required credential missing before inference dispatch, -and an authentication response whose documented meaning cannot be established. -HTTP status and `Failure` classify ordinary rejection and transport outcomes; -they do not need duplicate reason codes. - -`Completed` means the diagnostic concluded, not that checks passed. A rejected, -failed, unsupported, or inapplicable diagnostic can conclude with nil Go error. -Caller cancellation before completion leaves it false and returns available facts. -**A nil Go error means the diagnostic concluded, not that the tested operation -succeeded.** Make this distinction prominent in GoDoc and consumer examples. - -A fact is true or false only when established; nil means not evaluated, -undetermined, or inapplicable. Operation metadata and failure classification -provide context. In particular: - -- `CredentialsSupplied`: a nonempty credential was attached to the observed - dispatched request for the attempt's `Endpoint`, not merely named in configuration - or attached to an earlier redirect hop. No observed request leaves this fact - absent. For injected clients, do not claim upstream transmission PromptKit cannot - observe. Observation at the transport boundary is not proof of wire transmission. -- `EndpointReachable`: an HTTP response was received from the tested destination, - regardless of status. HTTP 401 therefore yields true. An attempted HTTP operation - ending without a response yields false. An injected generation result alone - does not establish HTTP reachability. -- `CredentialsAccepted`: the designated authentication adapter recognized acceptance - or rejection. A rejection yields false; transport failure, server error, or an - uninterpretable response can leave it nil. Acceptance does not certify balance, - model permissions, or future access. Absence of credentials is not an invalid-key - assertion: a documented 401 with `CredentialsSupplied=false` records rejection - of an unauthenticated request, not rejection of a supplied key. -- `ResponseReceived`: an HTTP response was obtained for built-in operations, or a - response was returned by the injected generation boundary. A non-success or - malformed HTTP response still counts as received. -- `GenerationCompleted`: generation decoded/completed successfully, separately from - generated-content checks. Empty decoded content can complete generation. -- `TextReturned`: generated content is non-whitespace. `ExpectedTextMatched` compares - against the private expected text using trimming only, without case folding or - semantic matching. -- `StructuredOutputRequested`: the fixed native schema constraint was supplied to - the built-in provider request or injected generation boundary. This is not proof - of upstream transmission by an injected client or provider enforcement. -- `ValidJSON`: the generated content, not the HTTP envelope, parsed as JSON. - `SchemaMatched` evaluates parsed JSON against the fixed test schema and remains - absent when parsing or response acquisition prevents evaluation. - -Known inapplicable authentication performs no dispatch and leaves request and -acceptance facts absent. Unknown measurements remain unknown, including phase -measurements for unperformed work. Report-level capacity wait, request duration, -evaluation duration, and retry delay sum the corresponding measured work across -the diagnostic. Request duration excludes capacity waiting, evaluation, and retry -backoff. Retry delay includes completed and interrupted backoff. Per-attempt -timings retain their own measurements; backoff is reported at the operation level. -Total duration is wall time, including selection and other orchestration overhead, -so it need not equal the phase sum. Timings impose no application latency threshold -or universal deadline. - -If retries are enabled, top-level facts describe the most recent dispatched -attempt. Cancellation during backoff or a later capacity wait does not replace -those facts with an undispatched attempt's empty facts. The operation's `Failure` -and `Completed` still describe the interruption; earlier attempt records retain -their own outcomes. A canceled capacity acquisition can add a `Dispatched=false` -record, but no synthetic attempt is added for cancellation during backoff. With -no dispatched attempt, request facts remain absent. Usage reuses presence-aware -accounting without claiming complete provider billing. No facts are inferred from -arbitrary error text. -`Dispatched` means the HTTP/client boundary was invoked, as in existing attempt -observations; it is not proof that packets were sent or billable work occurred. - -Reports omit private prompts, expected text, schemas, generated content, credentials, -raw bodies, and headers. Use content-free failure classifications and HTTP status, -not arbitrary provider messages, to preserve this exposure boundary. Effective -settings, provider-extra values, endpoints, and identities may still be sensitive -operator data; applications own their disclosure. Apply existing credential -exclusion and formatting rules rather than promise that all configuration is safe -for unrestricted logging. - -## Diagnostic Procedures And Capability Configuration - -### Reachability - -Perform one unauthenticated HTTP GET to the configured diagnostic endpoint, or -the effective base endpoint when no separate endpoint is supplied. Any HTTP -response establishes reachability, including 401, 404, or 405. It does not establish -that the completion route, credentials, or a model work. Authentication is tested -separately. Classify transport failure where available without promising exhaustive -DNS/TLS diagnosis. - -### Authentication - -Metadata names a supported provider adapter or explicitly marks authentication -inapplicable. The first adapter is `openrouter_key`, using GET on the -selected base endpoint's `key` path with ordinary bearer credential resolution. -OpenRouter's [current-key API](https://openrouter.ai/docs/api/api-reference/api-keys/get-current-key) -documents this authenticated operation and its 200 and 401 responses. Recognize -the documented successful shape and unauthorized response; leave acceptance -unknown on other outcomes instead of guessing. Returned account/key details -are not exposed. - -Configure RakestrawHome authentication as `none`, pursuant to the maintainer's -stated deployment behavior. Other backends can declare supported adapters. -An arbitrary models-list or health endpoint is not credential-acceptance evidence. - -### Inference - -Plain-text inference uses a small fixed private prompt and expected answer. -Structured-output inference uses a fixed private prompt and self-contained schema, -requests native JSON Schema output through the existing client path, and evaluates -JSON and schema conformance locally. It imports no application output contract. - -Exact content is an implementation detail and does not enter public prompt listings, -inspection, configuration inspection, prepared snapshots, or reports. Injected -clients and providers necessarily receive the content they execute: privacy here -is an API-exposure boundary, not secrecy from executing collaborators. There is -no persistent diagnostic prompt or session state. - -### Backend Metadata - -Extend optional backend configuration through the final-value loader and the -`BackendConfig`/`BackendSource` convenience. Illustrative shape: - -```yaml -backends: - openrouter: - smoke_tests: - default_profile: promptkit-openrouter-standard - authentication: - kind: openrouter_key - rakestrawhome: - smoke_tests: - default_profile: rakestrawhome-gemma-4-31b - authentication: - kind: none -``` - -Omitted smoke-test metadata leaves ordinary execution unchanged. Omitted -reachability metadata uses the effective base for the generic HTTP probe; -omitted authentication metadata makes that operation unsupported, with applicability -undetermined. Missing default-profile selection matters when inference selects it, -not for backend-only checks or an explicit profile. Retain normal effective -presence/null and layering semantics. Validate syntax and recognized configuration -at the final handoff, without contacting providers or validating intermediate layers. - -Provider paths follow the selected endpoint override, not silently the catalog's -original endpoint. Explicit absolute diagnostic endpoints are supported for separate -health/auth routes and take precedence for their operation. Configuration stays -limited to named adapters and endpoint settings, not arbitrary methods or response -predicates. The companion implementation plan specifies convenience types and -presence/null interpretation for these fields. - -Reuse the configured HTTP client and its redirect handling, without adding custom -credential forwarding. `Target.Endpoint` records the resolved starting destination. -Each HTTP attempt records `InitialEndpoint` and the last observed dispatched -request destination as `Endpoint`; its credential, reachability, response, and -acceptance facts describe that destination. If a redirect leads to a failed request, -do not attribute an earlier hop's response or credentials to the later destination. -When a client prevents observing the relevant request, leave the affected evidence -unknown. Injected generation alone supplies no HTTP destination evidence. - -Authentication acceptance is interpreted from the response for the observed -destination under the selected adapter's contract. It does not assert acceptance -at the initial destination or at every redirect hop. These endpoint fields provide -attribution without introducing a public redirect trace. - -Deliver built-in metadata in the local catalog repositories at -`../promptkit-backend-openrouter` and `../promptkit-backend-rakestrawhome`. -Catalog changes and PromptKit adoption are included in the sprint; publication -and remote module availability are not intermediate dependencies. Caller -overlays remain available through ordinary composition. Interpret and validate -only the assembled final configuration. Built-in reachability follows the -effective endpoint rather than pinning a separate catalog endpoint. - -The catalogs expose diagnostic metadata through an additive, opt-in embedded -asset root. Existing `/v3` catalog roots remain usable by older strict loaders; -new PromptKit explicitly selects the additional assets. Provider metadata stays -owned by its catalog rather than being duplicated in PromptKit. - -Local candidate tags identify validated contents; they are not published releases. -The implementation plan owns candidate selection and local proxy mechanics. -The catalogs' publication-date versioning convention still applies to later -publication. If that requires new final version names, update PromptKit's pins -and checksums accordingly; retain the validated catalog contents and never move -a published tag. Selecting the new metadata requires a compatible PromptKit -loader. No intervening PromptKit loader release or external module-download -verification is required to finish implementation. - -Use `promptkit-openrouter-standard` for OpenRouter's default, its `openrouter_key` -adapter, and `rakestrawhome-gemma-4-31b` with RakestrawHome's `none` authentication -setting. Verify each shipped default resolves in the supported assembled effective -configuration, including any PromptKit-owned profiles. This is final-value -integration validation, not a second validation pass over intermediate sources. -A catalog default is replaceable metadata, -not a promise of upstream availability, cheap inference, or a stable model choice. - -## Execution And Error Boundary - -Reuse target resolution, generation coordination, HTTP decoding, structural -validation, and usage accounting internally. A separate diagnostic orchestration -entry point may retain evidence that ordinary `Run` intentionally discards. -Do not duplicate inference serialization or build a second inference stack. - -Add a narrow internal evidence path at the transport/coordinator boundary. Existing -execution events do not carry redirect destinations or dispatched credential -presence, and the coordinator can discard a completed response when cancellation -is observed before return. Capture available facts before that discard and retain -them for diagnostics. Keep ordinary execution's error/result contracts unchanged; -do not expand public execution events just to reconstruct smoke-test evidence. - -Inference participates in existing engine-local admission and active backend -limits. HTTP-only probes need no generation permit or unrelated model selection. -Caller context and configured timeouts govern operations. Inference defaults to -one attempt: nil `TransientRetries` means zero for this operation rather than -inheritance from the engine retry budget. Positive values reuse existing built-in -inference status retries and retain attempt records; injected generation remains -single-dispatch. Reachability and authentication remain single-attempt regardless -of a valid inference retry setting. Structural repair is excluded from the initial -procedure. Otherwise use the selected profile plus explicit overrides. Ordinary -engine defaults and repair behavior do not change. Small test content does not -establish small output, a price estimate, or a cost ceiling. - -Keep `LLMClient` unchanged. Injected clients serve inference smoke tests without -new required methods. HTTP diagnostics must describe an explicitly selected -transport, not secretly probe a different transport in an injected-client engine. -For injected-client engines, the initial delivery reports HTTP diagnostics -unavailable and adds no separate diagnostic HTTP-client option. Inference -diagnostics remain available through the injected client. Known inapplicability, -such as authentication configured as `none`, can be reported without any transport. -Do not silently fall back to a default HTTP client to diagnose an injected -generation client. A future explicit diagnostic transport option remains possible -but is not needed to complete this feature. - -Return conventions: - -- Invalid kinds/settings, nil engines, or invalid target/profile/prompt selections - return ordinary public errors before probe work. Unsupported capability is not - the same as invalid selection. -- Concluded diagnostics return reports with nil Go error, including credential - rejection, provider rejection, transport or configured request-timeout failure, - invalid JSON, schema mismatch, unsupported capability, or inapplicability. - Local capacity exhaustion is a diagnostic execution finding, not provider failure; - request facts remain absent if nothing was dispatched. -- Caller cancellation or caller-context deadline expiration returns accumulated - observations with an error recognizable through `errors.Is` against - `context.Canceled` or `context.DeadlineExceeded`. An already-done context can yield - a report with no observations. Do not dispatch again merely to fill missing facts. - As elsewhere, arbitrary synchronous injected code must return before cancellation - can be observed. - -The cancellation contract applies to this new operation, not to existing -preparation APIs. Internal faults preventing meaningful diagnostic execution may -still return an ordinary error; the implementation plan should identify those -boundaries without promising that every conceivable failure becomes a report. - -## Representative Requests And Results - -Request examples use ordinary exact selections, without private test language: - -```go -// Select the application's default profile, but do not run its prompt. -req := promptkit.SmokeTestRequest{ - Kind: promptkit.SmokeTestTextInference, - PromptID: "meeting.summary", -} -// An explicit profile overrides that default. -req.ProfileID = "promptkit-openrouter-standard" - -// Select a backend's configured default for structured inference. -structured := promptkit.SmokeTestRequest{ - Kind: promptkit.SmokeTestStructuredOutput, - BackendID: "openrouter", -} -``` - -Expected report examples: - -| Operation/outcome | Observations | Go error | -| --- | --- | --- | -| Authentication rejected | Supported/applicable/completed; credentials supplied true, endpoint reachable true, response received true, credentials accepted false; attempt status 401 | nil | -| RakestrawHome authentication | Supported true, applicable false, completed, `authentication_not_applicable`; no dispatch/acceptance facts | nil | -| Unconfigured authentication | Supported false, applicability unknown, completed, `probe_not_configured` | nil | -| HTTP diagnostics unavailable for injected client | Supported false, applicability unknown, completed, `http_diagnostics_unavailable`; no dispatch | nil | -| Authentication without credentials receives documented 401 | Supported/applicable/completed; credentials supplied false, endpoint reachable and response received true, credentials accepted false; rejection describes an unauthenticated request | nil | -| Required credential absent before inference | Supported/applicable/completed; `required_credential_missing`, failure `invalid_request`; no dispatch and no request facts | nil | -| Authentication receives uninterpretable HTTP 200 | Supported/applicable/completed; endpoint reachable and response received true, credential supply recorded, acceptance unknown; `authentication_response_unrecognized`, failure `malformed_response` | nil | -| Text inference returns unexpected nonempty text | Generation completed and text returned true; expected-text match false | nil | -| Structured inference matches schema | Structured output requested, response received, valid JSON, and schema matched true; no provider-enforcement assertion | nil | -| Authentication transport fails | Endpoint reachable/response received false; credential supply recorded when known; acceptance nil | nil | -| Configured request timeout while caller context remains active | Supported/applicable/completed; failure `deadline`; retain observed response facts, or record response/reachability false for observed HTTP dispatch without a response; injected clients do not establish HTTP facts | nil | -| Local capacity exhausted | Supported/applicable/completed; failure `capacity`; no dispatch or request facts | nil | -| Caller cancels during initial capacity wait | Supported/applicable, completed false; failure `canceled`, measured capacity wait, undispatched attempt; request facts absent | `context.Canceled` identity | -| Caller deadline expires during a dispatched request | Supported/applicable, completed false; failure `deadline`; retain available attempt evidence | `context.DeadlineExceeded` identity | -| Caller cancels during retry backoff after HTTP 503 | Completed false, failure `canceled`; top-level facts retain the last dispatched attempt, its 503 remains in attempts, interrupted retry delay recorded, no extra attempt | `context.Canceled` identity | -| Caller cancels | Available observations retained; completed false when interrupted | Recognizable context error | - -Consumers decide whether any of these findings warrant success, a warning, or a -failure in their own application. They own CLI diagnostics, exit statuses, -scheduling, and any combination with `ValidateRequest` or normal execution. - -## Work Needed And Target Completion - -- Add the public request/report operation and GoDoc, keeping business logic internal. -- Extend final backend metadata interpretation and captured lookup for capabilities - and defaults. Preserve explicit configuration inspection without exposing test - content or using source history for validation. -- Connect private inference content to shared selection, execution resolution, - capacity, generation, decoding, and structural evaluation without preparing - application inputs or messages. -- Add narrow HTTP probes, credential handling, evidence retention, factual results, - and partial cancellation reporting through the internal evidence path while - preserving existing client contracts. Specify destination/credential attribution - without exposing headers or a redirect trace. -- Deliver the agreed metadata through both local catalogs with ordinary - catalog/caller precedence and no provider-availability guarantee. Shipped - defaults resolve in the assembled final configuration, and selected versions - work without workspace/replacement substitution. The single-sprint objective - applies to the complete integration. -- Add focused offline tests with controlled transports and injected clients for - selection, rejected/absent credentials, applicability/support, transport/malformed - responses, content/JSON/schema facts, cancellation, shared inference capacity, - and credential/private-content exposure. Assign distinct test owners; do not - repeat existing transport, capacity, or validation matrices or snapshot private - test language. -- Include the report-state matrix, cross-backend and endpoint-only default profiles, - redirect attribution, cancellation after a response, and interrupted retry waits - in focused contract coverage. Reuse existing lower-level checks where sufficient. -- Update canonical consumer, backend-format, integration, and internal documentation - when implementation lands, with release notes explaining the optional addition. - Live diagnostics are explicit consumer operations, not default tests or new - release gates. - -Completion means consumers can select the four kinds, obtain supported factual -observations, distinguish false from unknown/inapplicable, and retain evidence -on cancellation. Existing preparation and execution continue unchanged. Initial -scope excludes application CLI/service work, readiness policy, continuous monitoring, -model-status tests, batch orchestration, customizable test language, cost estimates, -and guarantees about later generation.