Record collection ownership and ConfigKit boundary decisions

docs/roadmap/namespaces.md

@@ -1,10 +1,11 @@
-# Namespaces And Scoped Resource Lookup
+# Namespaces, Collections, And Scoped Resource Lookup
 
 Status: feature roadmap; not implemented. The reserved-slash naming convention,
-reference-resolution rules, and decisions 1–3 (scope shape, validation, and
+reference-resolution rules, decisions 1–3 (scope shape, validation, and integrated
-integrated discovery filtering) are accepted target behavior. Decision 4,
+discovery filtering), and decision 4's move to application/operator-supplied
-catalog migration, remains open. Exact Go names and option types remain delivery
+configuration are accepted target behavior. Collection metadata, inventory, and
-details within the accepted semantics.
+specification-version integration remain pending in decision 5, subject to the
+ConfigKit boundary. Exact Go names and option types remain delivery details.
 
 ## Purpose
 
@@ -25,9 +26,10 @@ Model names here illustrate identity and resolution, not provider availability
 or a promise of maintained catalog entries.
 
 The primary need is organization and convenient lookup within assembled
-libraries. Ansible collections are a conceptual reference for qualified names;
+libraries, with backend, profile, and prompt collections managed independently
-package distribution and relocatable collection imports are not required for
+of PromptKit software releases. Ansible collections are a conceptual reference
-this feature.
+for qualified names; a package manager and relocatable collection imports are
+not required for this feature.
 
 ## Existing Foundation
 
@@ -40,6 +42,12 @@ Existing prompt and profile listings expose assembled identities. The
 [consumer guide](../consumers/pkg-promptkit.md) describes current operations;
 their Go declarations and GoDoc remain authoritative until this feature lands.
 
+Directory-based and embedded configuration loading already exist through
+`DirectoryInputs` and `FilesystemInputs`. The current engine also prepends
+configuration assets from the imported OpenRouter and RakestrawHome Go modules,
+plus PromptKit-owned profiles. Decision 4 removes that automatic inclusion and
+dependency coupling; it does not require a replacement for ConfigKit loading.
+
 [ADR 0001](../adr/0001-configuration-boundary.md) assigns source loading,
 composition, and inheritance to ConfigKit. This proposal adds interpretation
 of consumer-supplied resource names against the final effective configuration,
@@ -167,9 +175,10 @@ Derive namespace membership from the qualified identifier's namespace component:
 `weather` must not include `weatherreporter/...`. Nested slash-separated
 namespaces are outside the accepted grammar.
 
-The initial proposal does not require a collection manifest. Titles,
+Collection namespace/release metadata and an API for reporting effective
-descriptions, and other collection metadata can be considered if consumers
+collection declarations are desired additions. They are distinct from resource
-need more than identity and grouping.
+namespace enumeration. Their configuration shape and specification-version
+handling remain pending in [decision 5](#5-collection-metadata-and-specification-versioning-open).
 
 ## Composition And Compatibility
 
@@ -183,6 +192,19 @@ introduce implicit qualification of definition keys, mounting under caller-chose
 prefixes, or rewriting of stored references as part of consumer scope. Those
 would be separate configuration-design work.
 
+ConfigKit owns all configuration processing before handing off the final,
+fully resolved configuration. PromptKit must not add a collection-manifest
+preflight, source-level uniqueness check, or independent version-checking pass
+around that processing. Its domain interpretation and validation consume final
+effective values. Source history remains available through explicit ConfigKit
+inspection, not as a way to enforce extra assembly rules.
+
+In particular, two documents contributing metadata for the same collection
+namespace are governed by ordinary ConfigKit composition semantics. ConfigKit
+may reject conflicts or combine contributions according to their layers; a
+repeated source declaration is not an additional PromptKit error. The earlier
+proposal to reject duplicate declared namespaces before composition is withdrawn.
+
 Existing unscoped exact lookup remains available for IDs valid under the new
 grammar. Compatibility with arbitrary slash-containing resource IDs is
 intentionally broken. The migration must:
@@ -200,8 +222,9 @@ intentionally broken. The migration must:
 - Document consumer action and coordinate the PromptKit format change with
   catalog adoption under the [release policy](../policy/release.md). Its catalog
   profile-data exception does not cover this general format/API behavior change;
-  account for its explicitly retained built-in profile names when selecting
+  deliberately retire its built-in-profile preservation promise when the
-  migrations.
+  accepted removal lands. Current policy and format documents remain unchanged
+  until implementation.
 
 Renaming a backend changes its identity; duplicating it under another ID can
 separate capacity accounting. Do not use blanket aliases or duplicate catalog
@@ -225,19 +248,24 @@ if a concrete consumer need emerges.
 
 ## Delivery Outline
 
-1. Resolve decision 4 and audit existing identifier usage against the accepted
+1. Resolve decision 5's ConfigKit integration question and audit existing
-   grammar. Record explicit mappings and coordinated migration work. Finalize
+   identifier usage against the accepted grammar. Record migration mappings,
-   Go names and option types that express the accepted decisions 1–3.
+   application provisioning guidance, and any separately required ConfigKit
-2. Implement the smallest public scope surface with shared name qualification
+   work. Finalize Go names and option types for accepted decisions 1–3.
+2. Remove automatic catalog inclusion, catalog-module dependencies, and built-in
+   profiles under decision 4. Use caller-selected configuration sources and
+   update examples/tests to supply their own definitions.
+3. Implement the smallest public scope surface with shared name qualification
    and existing engine collaborators. Cover supported operations consistently.
-3. Add integrated namespace filtering and representative consumer examples.
+4. Add integrated namespace filtering and representative consumer examples.
-4. Verify compatibility and scoped resolution through focused offline tests,
+   Implement collection metadata/inventory only after decision 5 is resolved.
+5. Verify compatibility and scoped resolution through focused offline tests,
    following the [testing policy](../policy/testing.md). Update GoDoc, the
    format reference, consumer guide, and relevant internal documentation as
    behavior lands; retire this roadmap when its work is complete.
 
-Acceptance cases for the settled rules; migration-specific cases follow the
+Acceptance cases for the settled rules; collection metadata/versioning cases
-eventual decision 4:
+follow the eventual decision 5:
 
 - Two prompt scopes resolve the same local name to their respective prompts;
   a missing name never resolves from the other scope or an unqualified entry.
@@ -268,12 +296,18 @@ eventual decision 4:
   through the engine without a corresponding scope.
 - Representative validation, inspection, execution, and smoke-test paths use
   the same qualification rules without duplicating every test at every layer.
+- Engines use caller-selected definitions without automatically contributing
+  provider catalogs or PromptKit-owned profiles. Ordinary directory and embedded
+  configuration sources retain ConfigKit composition semantics.
+- PromptKit does not add source-history-based duplicate-namespace rejection or
+  independently pre-validate collection definitions before the final handoff.
 
-## Decisions And Remaining Recommendation
+## Decisions And Pending Design
 
 Decisions 1–3 are accepted; their alternatives are retained as considered but
-not selected. Decision 4 remains open, with its recommendation and alternative
+not selected. Decision 4 records the accepted change in collection ownership
-unchanged for further discussion.
+and supersedes the earlier catalog-only migration proposal. Decision 5 retains
+the unresolved metadata/versioning question and its alternatives.
 
 ### 1. Public Scope Shape (Accepted)
 
@@ -404,52 +438,121 @@ justify that maintenance. Prefer it if downstream clients routinely need a
 combined catalog view rather than a resource picker or namespace filter. In
 either approach, the full ID remains the authoritative selection value.
 
-### 4. Catalog Migration (Open)
+### 4. Collection Ownership And Built-In Removal (Accepted)
 
-**Recommended: coordinate the grammar change with adoption of provider
+**Accepted: load collections as application/operator-supplied configuration,
-namespaces for ordinary maintained catalog profiles.** Use provider namespaces
+and remove PromptKit's built-in profiles as a breaking change.** Remove the
-such as `openrouter` and `rakestrawhome`, with descriptive slash-free local
+mandatory OpenRouter and RakestrawHome catalog Go-module dependencies and their
-profile names. Use `google.gemma-4-31b-it` for the illustrative model-based
+automatic contribution to engine configuration. PromptKit owns the definition
-profile name; profiles with distinct execution settings need distinct local
+format and execution machinery; collection publishers own their configuration
-names even when they select the same model. Assign every renamed profile an
+repositories, and applications/operators select and provision the collections
-explicit old-to-new mapping after auditing the catalog, including valid old IDs
+they use. One or more explicitly selected directories can supply backends,
-that currently encode a provider prefix differently. Do not infer mappings by
+profiles, and prompts through the existing ConfigKit integration.
-blind punctuation replacement.
 
-Retain the two unqualified built-in profile names explicitly guaranteed by the
+Remove `promptkit-openrouter-standard` and `promptkit-openrouter-auto`, including
-release policy. Keep existing valid backend IDs unless there is a concrete
+their implicit catalog dependencies. Do not replace them with another mandatory
-collision or organizational reason to rename them; provider-qualified profiles
+starter bundle. Applications may embed their own defaults using the same
-can reference those exact backend IDs without duplicating backend identity or
+configuration format and loader, then layer operator configuration as desired.
-capacity accounting. Application-owned prompt libraries choose their own
+Ordinary application overrides remain valid configuration and need not declare
-namespace and migration mappings rather than being silently rewritten by
+a collection release of their own.
-PromptKit.
 
-Prepare catalog data, affected references, consumer migration instructions, and
+This decouples collection updates from PromptKit rebuilds and releases and
-the PromptKit grammar change as coordinated work. Select compatible catalog
+avoids imposing provider inventories on every consumer. The tradeoff is that
-revisions for the PromptKit release and follow each repository's release rules.
+applications and deployments must supply their required configuration and
-Choose a PromptKit version permitted to break the format contract: a minor
+manage compatible collection revisions. PromptKit does not gain a collection
-release before v1, or a major release at/after v1, under the current release
+installer, automatic downloader, or implicit directory search policy.
-policy. Exact versions and actual old-to-new mappings require the catalog audit;
-this recommendation does not claim those mappings have already been established.
 
-This makes the initial feature useful with maintained provider catalogs and
+Retain the captured-engine lifecycle: configuration changes take effect when
-concentrates consumer migration into one planned transition. Retaining the
+the application constructs and validates a replacement engine. A change to an
-explicitly supported unqualified profile names preserves their selected
+installed directory does not mutate an existing engine's captured definitions.
-contract. The cost is a broader coordinated change than grammar enforcement
+Distribution as configuration does not make new provider protocols or diagnostic
-alone, including renames of otherwise valid ordinary profile IDs.
+adapters data-only features; those can still require implementation work.
 
-**Alternative: migrate only IDs that violate the new grammar, and defer broader
+Prepare explicit identifier mappings, replacement configuration examples, and
-catalog naming changes.** Keep all other exact IDs, including valid one-slash
+consumer migration instructions. Update the built-in-profile promise in the
-IDs whose first component would now be their namespace. Ship scope support and
+release policy and its canonical format documentation when removal lands.
-let new libraries adopt provider/application namespaces incrementally. This
+Use a release permitted to break configuration and API contracts under the
-reduces immediate consumer changes and makes the format update easier to review
+current release policy: a minor release before v1, or a major release at/after
-and release independently. It leaves existing catalogs with inconsistent
+v1. Actual catalog mappings, packaging arrangements, and release versions remain
-grouping, may expose model-vendor rather than provider namespaces, and can
+delivery work; independent collection releases need not track future PromptKit
-require another migration to realize the intended provider shorthand. Prefer
+software releases.
-it if the audit reveals that broad catalog renaming would impose substantial
-downstream disruption relative to current demand.
 
-The delimiter, two-component qualified grammar, syntax-based resolution, and
+**Superseded alternatives:** retaining catalog Go modules with renamed profiles,
-decisions 1–3 are settled. The need for a breaking migration is accepted, but
+or retaining a small PromptKit-owned starter bundle, would simplify initial
-decision 4's migration strategy, exact mappings, and release coordination remain
+provisioning but preserve some provider-data coupling. The accepted direction
-open for further discussion.
+places optional defaults in consuming applications and removes those core
+dependencies instead.
+
+### 5. Collection Metadata And Specification Versioning (Open)
+
+**Desired capability:** describe collection namespace and release version in
+configuration and expose an API through which downstream applications can list
+effective collection declarations captured by an engine. A collection release
+version is distinct from an individual prompt version and from the PromptKit
+software version. Use top-level `version: 1` for PromptKit specification version
+1, following ConfigKit's version mechanism, rather than introducing a
+`promptkit_spec` field. The ownership and placement of that version declaration
+across multiple sources remain part of this open decision.
+
+The initially discussed root manifest contained `collection.namespace` and
+`collection.version`. That shape and a standard manifest filename are not
+settled. In particular, the proposal to load/check each manifest separately in
+PromptKit before composing definitions is withdrawn. PromptKit must respect the
+ConfigKit boundary described above, including for version checks and duplicate
+declarations.
+
+ConfigKit v0.7.1's layered-loading contract checks the configured application
+version path on the main input only. Underlay roots cannot claim that path,
+and imported fragments cannot independently declare it. ConfigKit supplies the
+version mechanism while the application defines supported specification values;
+this does not give PromptKit permission to implement an independent collection
+assembly or compatibility-checking pass.
+
+**Recommended starting point, pending decision: ordinary ConfigKit composition
+with collection metadata in final effective values.** For example, the main
+configuration could own `version: 1`, with a namespace-keyed collection mapping:
+
+```yaml
+version: 1
+collections:
+  openrouter:
+    version: "2026.10.13"
+  weatherreporter:
+    version: "2.4.0"
+```
+
+This is a candidate shape, not an implemented or accepted schema. Other sources
+could contribute to `collections` without claiming the main version field.
+ConfigKit would decide whether contributions conflict or combine; PromptKit
+would interpret and validate only the final metadata. A namespace key would
+identify one effective entry without requiring a source-level uniqueness rule.
+It would not automatically qualify definition keys or stored references.
+
+This approach reuses the existing composition contract and avoids a parallel
+manifest loader. Its limitation is that it does not independently check a
+specification declaration for each imported collection. An effective declared
+release version also does not prove that every resource still matches that
+release after operator overrides.
+
+**Alternative: add collection/import compatibility support within ConfigKit
+before adopting independently versioned collection roots.** If every collection
+must declare its own specification version, design that capability in ConfigKit,
+including how versions and metadata interact with imports and layers. PromptKit
+would consume its resulting handoff and diagnostics. This could provide useful
+per-collection compatibility checks without violating ownership, but it expands
+the work into ConfigKit and requires a supported contract there before PromptKit
+can depend on it. No ConfigKit change is selected by this roadmap yet.
+
+Under either approach, the inventory should describe effective collection
+metadata captured at engine construction, not reconstruct a historical list of
+loaded manifests. Source contributions and explanations remain ConfigKit
+inspection concerns. Inventory should not reread installed files or claim that
+declared versions prove unmodified published contents. Namespace enumeration
+from resource IDs remains independent of this metadata API.
+
+**Pending question:** is ordinary composed collection metadata with one main
+specification declaration sufficient, or are independently version-checked
+collection roots essential enough to warrant ConfigKit work? Settle that before
+finalizing metadata schema, root-file conventions, inventory API values, or
+collection compatibility rules. Removal of bundled catalogs and built-in
+profiles remains accepted in either case.