From c7d1b0dca7b21bcc5f7a7c9b146c1347cee0b676 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 14 Oct 2026 01:32:10 +0000 Subject: [PATCH] Propose identity-based collection records for ConfigKit --- ...guration-collections-identified-records.md | 431 ++++++++++++++++++ docs/roadmap/namespaces.md | 6 +- 2 files changed, 436 insertions(+), 1 deletion(-) create mode 100644 docs/roadmap/configuration-collections-identified-records.md diff --git a/docs/roadmap/configuration-collections-identified-records.md b/docs/roadmap/configuration-collections-identified-records.md new file mode 100644 index 0000000..94d1994 --- /dev/null +++ b/docs/roadmap/configuration-collections-identified-records.md @@ -0,0 +1,431 @@ +# Proposal 2: Namespaced Collections As Identified Records + +Status: proposed alternative; not implemented or accepted. This is a middle +ground between [proposal 1](configuration-collections-current-configkit.md), +which uses current ConfigKit, and a future proposal 3 that may change source +and document organization more substantially. The parent +[namespace and collection roadmap](namespaces.md) retains the accepted product +decisions and the pending configuration decision. + +## Recommendation And Scope + +Represent prompts, profiles, and backends as sequences of records with explicit +identity fields. Prefer separate `namespace` and local `id` fields. Ask ConfigKit +to support opt-in, application-declared identity rules at selected configuration +paths, so those records compose and overlay by identity rather than position. + +The extension is one cohesive capability: identified record sequences, including +their strict composition, overlay, inheritance, and provenance behavior. It is +not a general user-defined merge framework. ConfigKit continues to own the +complete assembly process and hands PromptKit one final resolved configuration. + +Keep the root `prompts:`, `profiles:`, and `backends:` keys. Keep collection +metadata as the ordinary namespace-keyed mapping from proposal 1. Keep the main +input's specification declaration and the existing source/layer model. This +proposal does not add bare single-record files, source mounting, implicit +namespace injection, or per-import specification checking. Those are separate +questions for proposal 3. + +## Current Baseline And Required Extension + +The reviewed ConfigKit baseline remains v0.7.1, local checkout commit +`6fffde4af8f31f99d3ee295c426eb3234cb0fb77`. Its +[configuration contract](https://gitea.maximumdirect.net/eric/ConfigKit/src/tag/v0.7.1/docs/config.md) +and the implementation/tests reviewed for proposal 1 establish that: + +- Sequences are atomic: overlapping sequences conflict during strict composition + and replace one another across precedence layers. +- ConfigKit can already decode a single sequence of mappings into application + records. It does not currently use their fields as merge identities. +- `extends` currently identifies immediate mapping siblings. A sequence-element + mapping cannot establish its own sibling relationship, and mapping-valued + `extends` is not a supported reference form. +- Final decoding, application resolution, and semantic validation happen after + configuration assembly and inheritance. + +Consequently, changing PromptKit's document structs alone would not implement +this proposal. Concatenating sequences in PromptKit or converting each source +to keyed mappings before calling ConfigKit would violate the chosen boundary. + +## Proposed Authoring Format + +Each prompt record describes one prompt version, without a nested version-key +mapping. Use explicit strings for identity fields: + +```yaml +collections: + weatherreporter: + version: "2.4.0" +prompts: + - namespace: weatherreporter + id: summarize.alerts + version: "1" + default_profile: openrouter/gemini-flash-latest + messages: + - role: user + content: Summarize the weather alerts. + output: + format: text + validation_mode: none +``` + +A second collection can use the same local ID without colliding: + +```yaml +collections: + homewatch: + version: "3.1.0" +prompts: + - namespace: homewatch + id: summarize.alerts + version: "1" + default_profile: openrouter/gemini-flash-latest + messages: + - role: user + content: Summarize the household alerts. + output: + format: text + validation_mode: none +``` + +The profile name is an illustrative configured reference, not a statement about +current provider inventory. A synthetic provider contribution shows the same +record shape for backends and profiles: + +```yaml +collections: + provider: + version: "2026.10.14" +backends: + - namespace: provider + id: default + endpoint: https://inference.example.invalid/v1 +profiles: + - namespace: provider + id: brief + backend: provider/default + model: vendor/example-model + temperature: 0.2 +``` + +These are independent format examples; the synthetic provider does not satisfy +the earlier examples' OpenRouter reference. An assembled configuration must +supply its selected references through ordinary configuration and existing +PromptKit validation/selection rules. + +Retain the existing `default_profile` field instead of also renaming it to +`profile`. Reducing nesting and making identity explicit are the changes under +evaluation; unrelated field renames are unnecessary. + +### Identity And Public References + +Register these record identities with ConfigKit: + +| Configuration path | Identity fields, in order | +| --- | --- | +| `backends` | `namespace`, `id` | +| `profiles` | `namespace`, `id` | +| `prompts` | `namespace`, `id`, `version` | + +PromptKit turns each final `(namespace, id)` pair into its public +`namespace/id` identifier after handoff; the prompt version remains a separate +selection dimension. The conversion creates runtime indexes, not another merge +or source-processing pass. It must reject invalid final domain identities rather +than silently re-keying them. + +Require identity fields to be explicitly present strings in every record, +including partial override records. A prompt's nested `version` is a prompt +identity component, not the protected top-level specification declaration. +Do not derive any identity component from a typed default, parent, filename, +collection metadata entry, or caller execution scope: matching happens before +those application-level interpretations. + +For unqualified application definitions, use `namespace: ""` explicitly; their +public identifier is the local `id` alone, without a leading slash. +Nonempty namespaces and local IDs follow the accepted namespace grammar; +local IDs cannot contain `/`. Preserve exact spelling. PromptKit owns these +domain checks on the final values; ConfigKit only requires usable structural +identity fields and compares exact string tuples. Distinct tuples must never +be conflated by concatenating fields with an unescaped delimiter internally. + +Stored `default_profile` and `backend` references remain fully qualified or +exact unqualified strings, as already selected in the namespace roadmap. +Provider model values retain their existing syntax, including slashes. + +## Proposed ConfigKit Contract + +### Explicit Registration + +The application's ConfigKit definition declares record-sequence rules at exact +semantic paths, with ordered identity field names. The table above is a +conceptual registration, not a proposed Go API declaration. ConfigKit does not +guess from field names such as `id`, and configuration files cannot silently +switch their own merge policy. No arbitrary merge callbacks are required. + +An initial implementation only needs direct string identity fields at explicitly +registered paths. Wildcard registration, computed keys, key normalization, +implicit identity defaults, and nested identity expressions are not required +by this proposal. + +At a registered path, a non-null contribution must be a sequence of mappings. +Every record must supply all structural identity fields with the declared +string type. Missing/null/non-string identity fields or non-mapping records +are ConfigKit structural errors: the loader cannot assign them a record identity. +These errors cannot be hidden by a later layer. Nonidentity body fields retain +the existing final-value decoding/validation lifecycle. + +For consistency with duplicate mapping-key handling, recommend rejecting two +records with the same tuple within a single source sequence. Across separate +sources, matching tuples participate in normal composition as described below. +This is a proposed explicit upstream rule, not a claim about current ConfigKit +or a PromptKit source-level duplicate scan. + +### Strict Composition And Overlay + +Treat identity fields as the record's structural key. Their repeated equal +values are not overlapping body fields that cause composition conflicts. +Retain the fields in the final record and preserve their origin information. +All remaining fields form the body to which existing mapping rules apply. + +| Contributions at a registered path | Proposed result | +| --- | --- | +| Different identity tuples in one group or different layers | Retain both records | +| Same tuple across peer sources, disjoint body fields | Recursively compose one body | +| Same tuple across peer sources, overlapping non-mapping body values | Composition conflict, even if equal | +| Same tuple in a higher-priority layer | Recursively overlay the higher body on the lower body | +| Same prompt namespace/local ID but different prompt version | Distinct records | +| Identity field changed in a later contribution | A different record; never an implicit rename of the old record | + +Once a tuple is selected for matching, identity fields cannot themselves be +overlaid into a different tuple. This avoids record identity changing during +composition. A higher layer need only repeat the tuple and changed body fields: + +```yaml +version: 1 +profiles: + - namespace: provider + id: brief + temperature: 0.1 +``` + +The provider/backend/model fields inherited through layering remain. A new +record introduced by an override still undergoes ordinary final validation; +matching by identity does not guarantee it is a complete runnable profile. + +For registered sequences, recommend that `[]` contributes no records and does +not clear lower records, like an empty keyed mapping. This differs deliberately +from ordinary sequence replacement. Explicit null at the whole registered path +uses ordinary cross-layer replacement and can clear that whole collection when +the application's final schema permits null. Within strict composition, null +conflicts with another present value as usual. No per-record deletion marker, +rename operation, or general patch language is added. + +Process tiers in precedence-assembly order, lowest first, with deterministic +source order within a tier. Preserve first-contribution order for distinct +records, retain a matched record's position, and append newly encountered +tuples. This makes output deterministic without giving position precedence or +identity meaning; acquisition order need not equal assembly order. PromptKit +can still sort its public inventory independently. + +**Unregistered sequences keep all current semantics.** In particular, `messages` +remains atomic: it conflicts across strict peer definitions and is replaced +as a whole by a later override. Identified record support must not turn every +sequence into a union or keyed merge. + +### Inheritance Within A Registered Sequence + +The extension must give identified records a sibling identity space. For +composite identities, recommend a structural `extends` mapping containing the +complete parent tuple. A profile example is: + +```yaml +profiles: + - namespace: weatherreporter + id: brief + extends: + namespace: provider + id: brief + temperature: 0.4 +``` + +ConfigKit resolves this relationship within the same registered `profiles` +sequence after all composition and precedence. It overlays the child's body +over the parent's resolved body, retains the child's identity fields, and +consumes `extends` before typed decoding. Parent identity fields are never +inherited into the child. References for a three-field identity, such as a +prompt record, must include all three fields; no version guessing or namespace +fallback is introduced. References cannot cross resource paths. + +Preserve existing behavior for omitted relationships, `extends: null`, missing +parents, cycles, and overridden relationships. A partial child contribution +can retain a lower-layer relationship, and only final active relationships are +resolved. Inherited file-backed content keeps the declaring field's source +context, not the child's source directory. + +This structural reference form is only enabled for records at registered paths. +Existing mapping-sibling string inheritance remains unchanged elsewhere. A +future single-field record rule could use that same field in the reference +mapping; no PromptKit-specific interpretation of slash-separated parent strings +is needed inside ConfigKit. + +### Provenance, Inspection, And Typed Handoff + +ConfigKit owns record matching and provenance throughout assembly. A conflict +must identify the record and conflicting source locations, not merely report +that two sequences occupy `prompts`. Preserve ConfigKit's safe diagnostic +contract when representing that identity; do not indiscriminately echo input +field values in error messages. Identity-field declarations retain their +contributing locations even when those equal structural fields do not conflict. + +Keep the final application-facing shape as a sequence of records, suitable for +ordinary slice decoding and effective-config rendering. ConfigKit can use a +private identity index, but should not expose private normalized nodes to +PromptKit. Final body presence/null state, field origins, inheritance history, +redaction, and passive source resource contexts must survive matching. + +No new public identity-selector path language is required for this proposal. +Inspection may address final records by sequence index, with ConfigKit mapping +effective field paths to the appropriate historical source locations internally. +Indexes are snapshot-specific; consumers locate a record by its identity before +constructing an indexed path. Histories must not become attached to another +record merely because its source position differed. Existing positional diff +behavior need not become an identity-aware cross-snapshot diff in this work. + +Rules must apply consistently to all enabled source kinds and composition +entry points, including imports and layered input. Existing explicit CLI +overrides must not leave duplicate or malformed structural identities in the +final registered sequences; ConfigKit owns their integration and consistency +checks. Ordinary application resolution must not trigger another merge or +inheritance pass. Detailed public option declarations and implementation +mechanics belong in the upstream design if this proposal is selected. + +## Collection Metadata, Versions, And Assembly + +Retain proposal 1's ordinary `collections..version` metadata and +effective-descriptor inventory. It is distinct from the generic ConfigKit term +"record sequence." Collection declarations still follow mapping composition; +they are not a source registry or exclusive ownership claim. Repeated metadata +release strings conflict in a strict peer group and can be overridden in a +later layer. PromptKit does not add an independent duplicate-namespace scan. + +Also retain proposal 1's main-owned `version: 1` declaration and its proposed +explicit-version requirement. The value denotes the specification for the +assembled configuration, not each imported package. Prompt record versions and +collection release strings do not change that ownership rule. These proposals +are alternative candidates for a future standard; they are not two simultaneously +supported meanings for a released specification version. + +Combine peer collection inputs in one strict underlay group; use later groups +or main for intentional overrides. Callers continue to choose grouping and +precedence. Directory inputs and explicit imports use existing ConfigKit +selection and resource-context rules. There is no PromptKit pre-parser that +reformats records, removes version fields, or replays the original sources. + +**Independent collection specification checks remain unresolved.** This bounded +extension does not permit imported or underlay roots to declare their own +protected top-level `version`. A publisher must still supply configuration +compatible with the selected main standard. Proposal 3 may consider a broader +source/version contract; proposal 2 must not imply that record identity solves +that separate issue. + +## Ergonomic Alternatives And Tradeoffs + +The first syntax discussed is also compatible with the generic extension: + +```yaml +prompts: + - id: weatherreporter/summarize.alerts + version: "1" + default_profile: openrouter/gemini-flash-latest + messages: + - role: user + content: Summarize the weather alerts. + output: + format: text + validation_mode: none +``` + +That alternative registers prompt identity as `(id, version)` and profile/backend +identity as `(id)`. It is simpler to align with public reference strings and +requires fewer identity fields in partial overrides. It repeats the namespace +inside each full ID and gives authors no separate namespace field. + +Prefer the composite form for explicit author-facing fields and easier editing +of local names. Do not support both forms opportunistically in the same standard: +that would require aliasing or normalization before identity matching. Select +one representation for the adopted specification. The first form remains a +reasonable alternative if the extra namespace line proves more cumbersome than +the qualified string. + +Retaining keyed mappings only for profiles could avoid extending inheritance +to records, but would give authors different identity conventions by resource +type. The recommendation favors a consistent record model and includes the +necessary generic inheritance work explicitly. + +| Concern | Proposal 1 | Proposal 2 | +| --- | --- | --- | +| Author identity | Qualified YAML mapping key | Explicit namespace/local-ID fields | +| Prompt version | Nested mapping key | Field in each prompt record | +| Cross-source composition | Current ConfigKit | Opt-in record identity extension | +| Body conflicts and precedence | Mapping rules | Same mapping rules after tuple matching | +| Root `prompts:` wrapper | Required | Required | +| Provider model strings and execution references | Unchanged | Unchanged | +| Independent imported specification checks | Unavailable | Still unavailable | +| Main maintenance cost | PromptKit schema/API work | PromptKit work plus ConfigKit record lifecycle support | + +This is more than a syntax-only change, but remains bounded: it introduces one +generic representation for identified entries while preserving configuration +source ownership and ordinary merge policy. It does not introduce collection +installation, dependency resolution, namespace ownership, mounted sources, +arbitrary transforms, or a general-purpose merge strategy system. + +## Delivery And Validation + +If selected, agree the upstream ConfigKit record contract first, implement and +test it there, then adopt that capability in PromptKit's prospective schema. +PromptKit would decode final records, validate domain identities/settings, build +its ordinary execution indexes, and implement the already accepted scope and +resource-discovery behavior plus the proposed collection inventory. Bundled +catalog/profile removal remains the accepted direction. + +Migration must convert each old keyed resource into its chosen record identity, +flatten prompt-version entries, update any inheritance reference syntax, and +preserve provider model strings. Choose specification and software release +versions deliberately; do not silently reinterpret deployed configuration. + +The upstream acceptance cases should include: + +- Different tuples coexist; same local prompt names in different namespaces + never collide; different versions of one prompt remain separate. +- Matching tuples compose disjoint bodies but reject overlapping scalar/null/ + ordinary-sequence body fields, including equal values. +- Matching tuples overlay across tiers without deleting unrelated records. + Changing a tuple adds another record; `[]`, whole-path null, and ordinary + message sequences follow their distinct documented semantics. +- Missing or malformed identity fields and repeated tuples in one source fail + structurally; identity comparisons cannot collide through delimiter encoding. +- Composite inheritance resolves final relationships, preserves child identity, + rejects missing/cyclic parents, and retains declaring resource contexts. +- Matching, effective indexes, source positions, sensitivity, and provenance + remain consistent through imports, overrides, inheritance, and rendering. +- Unregistered configuration loads and ordinary sequence behavior are unchanged. +- Existing version-path protections remain intact; record `version` fields are + not mistaken for the protected root field. + +PromptKit acceptance should exercise the authoring examples, final identity +conversion, scoped selection, metadata inventory, and migrated references with +offline collaborators. The record composition and inheritance examples require +the proposed upstream extension; they are not executable supported workflows +on ConfigKit v0.7.1 today. Syntax checks and current-behavior probes can establish +the starting point, but cannot validate an unimplemented extension. + +Review verification on 2026-10-14: a temporary offline consumer of the local +ConfigKit checkout decoded the five examples without the new inheritance form, +confirmed that the composite `extends` example is rejected today, and confirmed +current sequence conflict/replacement behavior across peer and precedence layers. +Local documentation links and whitespace checks passed. The temporary consumer +and its module replacement are outside both repositories. + +This proposal remains for evaluation alongside proposal 1. The parent roadmap's +configuration decision remains open, and broader source-placement/versioning +changes are reserved for the later proposal 3 discussion. diff --git a/docs/roadmap/namespaces.md b/docs/roadmap/namespaces.md index 015021f..daaa2dd 100644 --- a/docs/roadmap/namespaces.md +++ b/docs/roadmap/namespaces.md @@ -487,7 +487,11 @@ dependencies instead. [Proposal 1: current ConfigKit](configuration-collections-current-configkit.md) develops a concrete configuration standard for the existing composition rules. -It is a proposal for comparison; this decision remains open. +[Proposal 2: identified records](configuration-collections-identified-records.md) +uses an opt-in upstream extension for identity-based record composition and +inheritance. Both retain main-owned specification versioning; broader source +and version changes can be evaluated in a later proposal 3. These are proposals +for comparison; this decision remains open. **Desired capability:** describe collection namespace and release version in configuration and expose an API through which downstream applications can list