From d3d468a8b47d780787fca329ff65b107d98ca59f Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 14 Oct 2026 12:48:08 +0000 Subject: [PATCH] Plan configuration input destination implementation --- docs/roadmap/implementation.md | 432 ++++++++++++++++++++++ docs/roadmap/input-destinations.md | 133 +------ docs/roadmap/promptkit-feature-request.md | 319 ++++++++++++++++ 3 files changed, 761 insertions(+), 123 deletions(-) create mode 100644 docs/roadmap/implementation.md create mode 100644 docs/roadmap/promptkit-feature-request.md diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..091e3d8 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,432 @@ +# Implement configuration input destinations + +Status: planned. Execute Stage 1 through Stage 12 in order, one focused stage +per Luna implementation prompt. The [feature roadmap](input-destinations.md) +owns the approved behavior and rationale; the +[consumer request](promptkit-feature-request.md#public-acceptance-scenarios) +supplies the public acceptance scenarios. This document owns execution and +completion evidence. Planning baseline: `6353598`. + +## Working instructions + +- Start each stage with `AGENTS.md`, [development.md](../development.md), the + applicable policies, and working-tree inspection. Use graph-first code + discovery. Preserve unrelated changes and previously completed stages. +- Implement only that stage and its supporting tests/documentation. Mark it + complete here with actual commands and evidence, then commit the nonempty + validated unit with a short plain-English message. Subsequent stages remain + planned. A stage is not complete solely because its narrow tests pass. +- Keep early mechanics private until the complete engine behavior is ready. + Root destination fields land in Stage 10. Ordinary root loading must remain + working throughout. Do not publish a partially supported public placement API. +- Follow the existing engine/document/YAML boundaries. Reuse strict composition, + recursive overlay, sibling inheritance, paths, history, and sensitivity. + No dependencies, toolchain changes, alternate mergers, mount registry, + per-fragment lifecycle, import-destination syntax, resource discovery, or new + release gates are needed. These are scope boundaries, not new prohibitions + on separately approved future work. +- Update current-behavior documentation when public behavior lands; keep future + claims here or in the roadmap until then. Add the ADR in Stage 1 under the + documentation policy's accepted-decision exception. +- Use focused offline tests while iterating; run `./scripts/check.sh` before + committing production changes. Documentation-only stages need whitespace and + local-link checks. Stage 12 adds full ownership/race validation. Tests protect + observable behavior, not wrapper allocation or helper-call counts. Reuse + coverage rather than repeat an entire loading matrix at every layer. +- No benchmark campaign or public complexity guarantee is required. Prefer + straightforward bounded walks and wrapping each acquired contribution once; + avoid rescanning all preceding contributions for every new input. +- Do not push, tag, publish, update downstream dependencies, or retire unfinished + records. Release compatibility review is separate from publication. + +## Resolved implementation decisions + +1. Add root-owned `Input.Destination Path` and `SourceInfo.Destination Path`, + mirrored privately in engine input/source records. Extend `loadInput` and + explicit root/engine conversion and cloning. No new loading entry point, + public wrapper type, option, or origin field is needed. Ordinary `Load` + continues to supply root placement. +2. The zero path means root. Validate `Path.valid()` and reject every explicit + index segment, including a negative index. Normalize parsed tokens to literal + mapping-key segments: `/0` means the key `0`, not sequence element zero. + Empty, dot, dot-dot, slash-containing, tilde-containing, and numeric keys keep + existing semantic-path meaning. Do not split dots, interpret filesystem + traversal, enforce identifier grammar, or add namespace uniqueness. +3. Reject destination segments equal to `configkit` or `extends`: placement + would introduce a reserved mapping key with a mapping value. Substrings such + as `myextends` are ordinary keys. This preserves the approved structural + rules rather than adding an application naming policy. Source-root import + directives are still processed before placement. +4. Validate destination shape/reservations for all declared inputs during + grouped option preflight, before reading any source. Reuse + `CodeOptionsInvalid` for invalid destination declarations; include the + offending semantic path when representable, with no fabricated physical + location. An invalid source still follows existing source-option validation. + A valid destination on `NoSource` contributes no nodes, source record, or + budget charges. No extra error type or exact message contract is introduced. +5. Separate acquired source syntax from application placement. Retain original + parsed source metadata for filesystem/directive handling. Strip source-root + directives, then wrap each application body at the normalized destination + before fragment version checks and strict composition. Imports inherit the + same destination transitively. Do not wrap both a source application and its + recomposed result; every retained application must already use effective + coordinates. Content roots remain mappings under existing source contracts. +6. Wrapping a body at `Keys("a", "b")` creates `{a: {b: body}}` as private + document mappings. Actual body nodes and entries keep their physical source + coordinates. Introduced mappings/entries use the contributing document's root + location as a documented source fallback, not purported authored coordinates. + They have the same contributing source ID and create no synthetic source. + Retained history trees remain immutable and separate from assembly mutation. +7. A present empty body creates explicit destination mappings, including an + import-only source whose application body is empty. It still cannot clear + lower mappings. A missing optional import and `NoSource` remain absent. Empty + source text remains the existing invalid document, not an empty mapping. +8. `VersionRule.Path` stays an effective application path. Check main once after + its own imports compose and before underlays/overlay/CLI, at the existing + phase. Imported and underlay claims are checked against placed application + content, including scalar/null/sequence ancestors replacing a protected path. + A mapping ancestor with no protected descendant is not a version claim. + Mounted domain `version` fields are ordinary data. Existing inheritance, + selected-overlay, CLI protection and exact-path numeric conversion remain; + no nested blanket conversion or additional fragment checker is introduced. +9. Placement does not alter physical source identity, filesystem namespace, + source IDs, acquisition order, imported-from locations, or resource contexts. + Repeated physical sources in one strict group still fail even at different + destinations. The same physical source in separate tiers retains the existing + independent source applications/records and each record's own destination. + Overlay and CLI source records have root `Destination`; they are not mounted + merely because main is mounted. Active layered overlays retain their explicit + independent filesystem selection and application-root meaning. +10. Value-level paths use effective coordinates. Keep genuine source-local + import-directive diagnostics and parse/read failures at their original + locations; do not invent a logical child path when parsing cannot identify + one. Prefix existing application-relative diagnostic paths exactly once + where available. Preserve structured codes, related physical locations, + deterministic ordering, and value-free messages. +11. Use current depth/node limits, not a new destination cap. With root depth + zero, each destination segment adds one level and, under `countInputNodes`, + one mapping node plus one key. Retain current raw input accounting (including + directives); charge only the additional `2 * segmentCount` synthetic nodes + per acquired body, once, to the shared input budget. Placement adds no bytes, + acquired sources, import nesting, or glob visits. Validate placed application + depth and per-document node bounds before composition; share accounting + across inputs and imports. Precheck wrapper depth/count before allocating + wrappers. Use existing source/import limit diagnostics in their acquisition + contexts; retain projection, expansion, and history bounds independently. + Root placement adds no charge or new rejection. NoSource adds none either. +12. Cloned source records include independently owned destination paths at every + existing publication boundary: results, load errors, metadata/inspection, + callback input information, and root conversion. Passive resource contexts + remain keyed by the real source identity; inspection must work without an + open configuration handle after load. This does not extend caller-owned + filesystem or callback resource lifetimes. + +## Stage 1 — Record the decision and capture the baseline + +Status: planned. Scope: decision record and evidence, no feature code. + +1. Read the roadmap, request, relevant policies, and ADRs 0001, 0003, 0006, + 0008, and 0009. Inspect the owners identified in the roadmap and the function + map below. Confirm the baseline and worktree before starting. +2. Run `./scripts/check.sh`; record the commit, Go version, and results here. + Locate existing coverage for root loading, strict underlays, version ownership, + repeated sources, history, resource contexts, and detached records. +3. Write `docs/adr/0011-input-destinations.md` after confirming the number is + unused. Record the approved input-placement decision as Accepted, with the + roadmap owning implementation status. Follow the repository ADR format. + Focus on ConfigKit, put rejected options only in Alternatives Considered, + and avoid negative parallelism elsewhere. No PromptKit-specific semantics. +4. Record the compatibility assessment under the release policy: adding fields + preserves zero-value/keyed-literal use, but callers using unkeyed literals + for `Input` or `SourceInfo` must migrate to keyed literals. Treat the feature + as minor-release work; select a version and write its note only during + separately authorized release preparation. + +Validate: local-link test and whitespace checks. Done: baseline evidence and +the decision are recorded; implemented contracts remain unchanged. + +## Stage 2 — Add private destination normalization and wrapping + +Status: planned. Scope: pure private mechanics with bounded unit tests. + +1. Add a small engine helper file for normalized mapping-only destinations and + constructing placed document bodies. Follow decisions 2–7; keep helpers + private and avoid introducing a second path abstraction. +2. Validate invalid paths, explicit indices, and exact reserved tokens. Normalize + parsed numeric tokens into keys, preserving literal text and pointer escaping. +3. Build wrappers without mutating the supplied tree. Preserve body locations; + give introduced entries/mappings the source-root fallback. Implement depth and + node measurement/prechecks usable with existing limits, including the extra + synthetic-node count; aggregate charging is wired in Stage 7. +4. Cover root identity/equivalence, nested empty mapping, literal/escaped/numeric + keys, invalid index/path, exact versus substring reserved keys, owned paths, + unchanged body coordinates, and bounded wrapper checks. Do not count helper + invocations or fix an allocation strategy in tests. + +Validate: focused destination/path tests and the everyday check. Done: the +mechanism is usable privately; no loader accepts a new public field yet. + +## Stage 3 — Integrate private non-import input placement + +Status: planned. Scope: engine request plumbing and ordinary acquired bodies. + +1. Add destination fields to engine `Input`, `loadInput`, and `SourceInfo`. + Preserve ownership through engine `LoadLayered` request construction, + `cloneSourceInfo`, and source-record copies. Keep the root facade unchanged. +2. Extend grouped preflight to validate/normalize destinations before source + reads. Use the existing diagnostic accumulator and options-invalid category. + Ordinary `Load` supplies root placement through its existing request adapter. +3. Thread placement into `assembleBaseInputTracked` without confusing raw + `parsedSource` with effective `assembledInput.document`. Preserve the root + path through existing wrappers/callers. For imports-disabled acquisition, + validate disabled directives on raw content, place the body, and retain the + destination on the acquired source record. NoSource still short-circuits. +4. Use engine loads to cover mounted main and underlay bodies, omitted versus + explicit root destinations, empty mapping versus NoSource, and preflight + rejection without consuming a test-owned reader. Reuse current source-kind + acquisition tests; full source equivalence is verified publicly in Stage 10. + +Validate: destination, grouped-load, source, and existing ordinary-load tests; +the everyday check. Done: private non-import placement reaches assembly, with +root behavior unchanged. Imports are the next private integration step. + +## Stage 4 — Carry placement through import acquisition + +Status: planned. Scope: source-local directives and placed imported applications. + +1. Thread a normalized destination through `loadImportedDocumentsTracked`, + `loadImportedWithAccessTracked`, and `importTraversal`, retaining root defaults + for current private callers. Place the selected body's application content + after `splitImportDirectives`; do the same for every imported body. +2. Set placement on selected/imported source records, including error records + once a source has been acquired. Keep `ImportedFrom` at the physical directive. + Run imported `fragmentVersionClaim` against placed bodies and retain placed + bodies in `state.documents` before `composeDocuments` builds applications. +3. Leave filesystem joins, containment, glob traversal, optional missing inputs, + active cycles, visited identities, and acquisition IDs/order unchanged. + Destination must not become part of the repeated-source identity key. +4. Test native and host transitive imports, nested source-root directives, + import-only bodies, and mounted imported protected versus ordinary version + paths. Verify a repeated file at two destinations in one group still fails, + including an import/direct-input alias. Reuse cycle/containment regressions. + +Validate: focused destination/import/glob/grouped tests and everyday check. +Done: every retained imported contribution has effective placement and physical +source identity, with no double wrapping or application-level directive leakage. + +## Stage 5 — Complete strict tiers and main version ownership + +Status: planned. Scope: assembly and version interactions, not a new merger. + +1. Review `acquireUnderlayGroup`: version checks must use placed content, and + documents taken from `acquired.composition.applications` must already be + placed. Recompose strict peers with ordinary `composeDocuments`; preserve + per-source applications, origins, UnderlayIndex, and input accounting. +2. In `executeLoadRequest`, main's version check uses its placed main/import + composition at the existing point. `assembleLayeredInput` recursively overlays + completed underlay groups beneath main, selected overlay, and CLI as before. + Do not mount the final assembly or prepend main's destination to upper tiers. +3. Test disjoint same-destination peers, different destinations, equal and unequal + conflicts, ancestor/descendant destinations, and a strict conflict despite an + upper override. Verify recursive mapping overlays, atomic sequences, null, + explicit false/zero, and version-only domain overrides retaining lower fields. +4. Exercise mounted main with a nested VersionRule, absent root declarations, + imported/underlay direct and ancestor claims, and nonclaiming mapping ancestors. + Check the hook runs once and does not inspect each fragment independently. + Preserve exact configured-path integer conversion and ordinary nested typing. + +Validate: destination/composition/overlay/version/grouped tests and everyday +check. Done: placed and authored equivalents agree on values and conflicts; +main ownership and lifecycle timing are preserved. + +## Stage 6 — Reconcile diagnostic coordinate systems + +Status: planned. Scope: useful paths without fictional source locations. + +1. Audit early acquisition, import, fragment-version, and composition failures. + Use one narrow path-prefix translation where diagnostics genuinely identify + application content; do not blanket-prefix every diagnostic or double-prefix + diagnostics already produced by placed trees. +2. Preserve syntax/read/directive locations and related import sites. Value + conflicts and later typed/resolver/validation failures use effective paths + but original content coordinates. Invalid introduced structure identifies + the destination declaration rather than a nonexistent source wrapper line. +3. Add focused assertions for a mounted conflict with two real sources, a mounted + decode or validation error, an imported version error, invalid destination, + and malformed source-local directives. Test stable codes/paths/locations and + nil results, not whole messages or callback choreography. + +Validate: destination/diagnostic/import/version/decode tests and everyday check. +Done: diagnostics consistently distinguish application paths from source syntax. + +## Stage 7 — Complete shared placement accounting + +Status: planned. Scope: resource bounds with no new public limit system. + +1. Wire Stage 2's prechecks and synthetic-node charges into both tracked ordinary + acquisition and import traversal, using `importAccounting` and current limit + structures. Charge each source body once, including empty import bodies. +2. Preserve existing raw byte/source/node accounting and imported directive + inspection. Additional wrappers count in per-body depth/node checks and + shared aggregate nodes; recomposition must not charge them again. Ensure + selected overlays and CLI start from totals including placement charges. +3. Test below/at/above relevant bounds using small private limit fixtures where + existing seams permit; add a narrow private seam if necessary. Cover deep + body plus destination, many shallow mounted inputs, imported wrappers, and + empty mappings. Verify root placement and NoSource do not add charges. + Preserve history/expansion limits; do not turn fixture values into defaults. + +Validate: destination/limit/import/override tests and everyday check. Done: +placement cannot bypass current budgets and introduces no unrelated caps. + +## Stage 8 — Preserve provenance, history, and source resource contexts + +Status: planned. Scope: engine inspection and retained evidence. + +1. Audit `compositionOrigins`, `completeCompositionOrigins`, application replay, + metadata construction/finalization, and source-record cloning against placed + applications. Prefer feeding placed trees into existing processing rather + than adding another path-offset index or rewriting final metadata afterward. +2. Verify explicit destination containers, actual leaves, composite origins, + superseded/removed descendants, and root/source fallbacks. Historical and + effective indexes keep independently owned paths and existing shape semantics. +3. Preserve source-ID resource contexts through imports, surviving lower-tier + values, and inheritance. Test completed-result and resolution-time context + queries against real relative files, without making ConfigKit open the + referenced application resource or retain loading handles. +4. Exercise the same physical source in separate tiers with different placement, + inspect each source record, and verify independent destination paths. Repeat + inspection after comparison and independent loads to detect retained mutation. + +Validate: destination/metadata/provenance/history/resource tests, +`go test -race ./internal/engine`, and everyday check. Done: inspection explains +effective placement and physical origin without consumer reconstruction. + +## Stage 9 — Verify final inheritance, sensitivity, and lifecycle integration + +Status: planned. Scope: complete private engine behavior before facade exposure. + +1. Resolve sibling inheritance only after full source precedence using the + existing expansion path. Cover a child and parent supplied by separate peers, + nested destination-local siblings, cancellation, and an inherited relative + resource retaining its declaring context. Cross-collection search is unchanged. +2. Reuse existing inheritance-version, overlay, and CLI guards; add targeted + mounted cases for actual protected paths and their ancestors. Do not reject + unrelated inheritance merely because it shares a mapping ancestor. +3. Test effective-path secret predicates, field tags and recorded dependencies, + superseded classified values, safe effective output/history/explanation, and + bilateral diff redaction using disposable canaries. Compare mounted versus + explicitly wrapped loads for equal effective values and no differences. +4. Verify defaults, decoding, resolution, secret lookup, and validation execute + once on the final shape; failures return no usable result. Reuse existing + ordinary-load lifetime/ownership coverage and add mounted protection where + placement creates a distinct risk. + +Validate: destination/inheritance/secret/sensitivity/explain/diff/lifecycle tests +and everyday check. Done: all approved engine semantics work together without +per-fragment loading or a post-return processing pass. + +## Stage 10 — Expose the root-owned facade and public acceptance tests + +Status: planned. Scope: the two public fields and complete consumer behavior. + +1. Add `Input.Destination Path` and `SourceInfo.Destination Path` to root types. + Extend `toEngineInput`, `fromEngineSourceInfo`, `toEngineSourceInfo`, and all + source publication paths with owned conversion. Run existing public/import + boundary tests; expose no engine nodes, aliases, or bridge methods. +2. Document zero/root behavior, mapping-only paths, main/underlay parity, + inherited import placement, source-coordinate fallback, and source ownership + beside declarations. Update the configuration contract for implemented input + destinations, preserving canonical ownership of shared semantics. +3. Add external-package public-load coverage for equivalent host file, native + fs.FS file, byte, reader, and Mapping sources. Cover default/root destinations, + literal escaped/numeric keys, empty mapping, NoSource, and invalid declarations. + Include one public mounted main case, not just underlays. +4. Add representative public tests tying final configuration to metadata, + source placement, history, resources, callbacks, and safe diffs. Mutate returned + record copies and repeat queries/loads to confirm detachment. Reuse engine + matrices; public tests demonstrate supported integration rather than every + private branch. Confirm failed loads expose detached source destinations. + +Validate: `go test . -run 'Destination|Facade|PublicFacadeBoundary|DependencyImportBoundary'`, +`go test -race ./...`, and everyday check. Done: the requested workflow is usable +entirely through ConfigKit's supported import path and documented surface. + +## Stage 11 — Add maintained simple and collection-style consumer examples + +Status: planned. Scope: executable adoption examples with consumer-owned types. + +1. Add a minimal mounted-fragment example using `Input.Destination` in an + ordinary layered load. Show literal keys and root-default compatibility + without introducing consumer parsing or generic merge helpers. +2. Add one representative collection assembly under `examples/consumers`: peer + prompt/profile fragments, collection metadata with ordinary `version`, a + source-root import, local sibling inheritance, and an intentional higher-tier + version-only override retaining the body. Keep the application VersionRule + main-owned. Use a controlled native filesystem or temporary files. +3. Demonstrate effective-path inspection plus source-relative resource lookup + during resolution and from the result, including an inherited/surviving field. + Application code owns opening/interpreting the resource and any qualified IDs. +4. Add tests for final values, presence, actual source placement/coordinates, + resource context, and expected conflict behavior. Link maintained examples + from the consumer guide; add a small executable root example if useful for + discovery without duplicating the complete scenario. + +Validate: `go test ./examples/consumers`, applicable `Example` tests, local links, +and everyday check. Done: simple and representative consumers substantiate the +same configuration model and need no compensating structural pass. + +## Stage 12 — Reconcile documentation and audit the complete implementation + +Status: planned. Scope: final integration audit and evidence, not release work. + +1. Update README navigation, consumer guidance, the component overview, and + development reading/test maps. Add ADR 0011 to navigation. Update architecture + policy only as needed to recognize placement within existing assembly and + source ownership, linking precise behavior to the configuration contract. + Do not change unrelated policies, deferred boundaries, or published notes. +2. Audit every resolved decision, roadmap behavior, and request acceptance row + against implementation and tests. Record the corresponding test names here; + for reused coverage explain the behavior it proves. Do not mark complete + based on green tests that do not exercise mounting. +3. Review the complete baseline-to-candidate diff for accidental strengthening: + extra naming rules, namespace uniqueness, main-only restrictions, changed + merge/version semantics, invented source-read ordering guarantees, new limits, + or expanded secret policies. Remove temporary helpers/unsupported branches + left by staged integration. Confirm original source coordinates remain real. +4. Run `./scripts/check.sh` and + `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`. Review source-kind and + placement ownership coverage, error disclosure, complete-on-return handling, + and root/private boundaries. Record commit, Go version, results, and any + limitations. No release command, CI waiting, benchmark, or downstream upgrade. +5. Mark this plan and the roadmap implemented only when code, public examples, + and owning contracts agree. Retain completed planning evidence until separately + authorized release preparation. + +Done: one complete layered load supports input destinations with coherent values, +diagnostics, inspection, limits, and source ownership across the approved surface. + +## Implementation owner map + +Use graph tools to locate these symbols rather than assume line numbers: + +- Public input forwarding: `layered.go`, `records.go`, `conversion.go` + (source conversions), and existing facade tests. +- Private request and lifecycle: `internal/engine/layered.go`, `load_request.go` + (`loadInput`, `acquireUnderlayGroup`, `assembleLayeredInput`), `load.go` + (`executeLoadRequest`), and `load_input.go` (`assembleBaseInputTracked`). +- Directives and imports: `directives.go` (`splitImportDirectives`, + `fragmentVersionClaim`), and `imports.go` (`visitPath`). +- Shared document semantics: `compose.go`, `document_walk.go` + (`countInputNodes`), `path.go`, `parse.go`, and `value_operation.go` limits. +- Inspection: `metadata.go`, `history.go`, `explain.go`, `resource_context.go`, + `inheritance_origin.go`, `render.go`, and `diff.go`; source copies live in + engine `records.go`. Confirm current filenames through the component overview. + +## Open Questions + +None required for implementation. The four policy choices are approved; this +plan selects API spelling, validation timing/error category, coordinate handling, +and accounting within them. Surface a newly discovered incompatibility or scope +change with evidence rather than resolving it by silently narrowing the feature +or strengthening an unrelated contract. diff --git a/docs/roadmap/input-destinations.md b/docs/roadmap/input-destinations.md index 4ac3216..afea488 100644 --- a/docs/roadmap/input-destinations.md +++ b/docs/roadmap/input-destinations.md @@ -1,10 +1,10 @@ # Configuration Input Destinations Roadmap -Status: design decisions approved; implementation has not started. Execute the -stages below in order. This roadmap responds to the -[PromptKit feature request](promptkit-feature-request.md) and owns the target -state, delivery sequence, and selected design decisions. Public API spelling and -diagnostic details remain to be finalized during implementation planning. +Status: design decisions approved; implementation has not started. This roadmap +responds to the [PromptKit feature request](promptkit-feature-request.md) and owns +the target state and selected design decisions. The +[implementation plan](implementation.md) owns sequencing, implementation details, +and completion evidence. ## Targeted end state @@ -20,9 +20,9 @@ For example, a fragment defining `summarize.alerts` can contribute beneath local key can contribute beneath another collection without conflict. Collection and prompt names remain ordinary application keys, not ConfigKit namespaces. -The intended public surface is per-input placement, illustratively an optional -`Destination Path` field on `Input`, not a new loading entry point or an operation -on completed results. Content-bearing main inputs and underlay inputs support +The public surface adds an optional `Destination Path` field on `Input`, not a +new loading entry point or an operation on completed results. Content-bearing +main inputs and underlay inputs support the same destination model; root placement remains the default. The following behavior defines the target: @@ -120,119 +120,6 @@ or a second configuration model. The consistency constraints; [deferred scope](deferred.md) continues to govern the unrequested features above. -## Ordered implementation plan - -### 1. Settle placement semantics and the public contract - -Finalize public API spelling using the Selected Design Decisions below for -main/NoSource behavior, placement inspection, reserved-key handling, and -empty-contribution semantics. Define destination validation and -structured errors, including when validation occurs relative to source access. -Use existing error categories where suitable rather than inventing a parallel -error system. - -Record the significant assembly decision in an ADR. Review compatibility under -the release policy, including exported struct changes and unchanged zero-value -behavior. Keep unimplemented claims in the roadmap; update canonical contracts -when implementation lands. Do not turn unrelated deferred features into work. - -Completion: selected semantics cover the request's public acceptance scenarios -and a simple consumer as well as PromptKit's multi-collection workflow. - -### 2. Carry destinations through acquisition and imports - -Thread the selected destination through root-to-engine conversion and private -load requests. Preserve its ownership and semantic key representation. Separate -source-local directive processing from application placement: extract directives -at their original root, inherit placement through traversal, and place application -content before checks and composition that require effective paths. - -Make version protection destination-aware for underlay roots and imported -fragments, preserving the main check's lifecycle position. Ensure diagnostics -produced before final composition identify effective application paths where -available while preserving actual source coordinates. Structural directive -errors still need their genuine source-local declaration locations. - -Retain repeated-source detection across the whole strict group, independent of -destination. Preserve filesystem namespace, resource contexts, and traversal -ordering. Account for destination depth and introduced nodes using existing -budgets; distinguish them from acquired source bytes and physical source count. - -Completion: all content-bearing source kinds and transitive imports contribute -at the intended destination, with correct version protection and source identity. - -### 3. Integrate composition, provenance, and the final lifecycle - -Apply ordinary composition to placed contributions and ordinary overlay between -tiers. Retain each source application at its effective placement for history; -carry real source attribution and the selected synthetic-container convention -through origin construction. Avoid a separate merge algorithm or a mount registry. - -Exercise overlapping destinations, empty mappings, and a version-only domain -override. Resolve sibling inheritance after the full precedence assembly, then -use existing defaults, typed decoding, resolution, secrets, and validation. -Check protection of the specification path through inheritance, selected -overlays, and CLI overrides as well as direct contributions. - -Completion: mounted and explicitly wrapped equivalents agree on values, -conflicts, presence, and lifecycle results; physical locations remain accurate. - -### 4. Complete inspection and public consumer workflows - -Expose placement through the selected existing inspection surface, with detached -records. Verify effective paths throughout metadata, explanation replay, history, -rendering, and diffs. Apply the existing sensitivity rules to mounted and inherited -fields, including superseded values. Do not fabricate coordinates for wrappers -that were never written in a source file. - -Demonstrate source-relative resource lookup from both resolution and completed -results, including inherited fields and lower-tier fields surviving an override. -Add executable public examples for one mounted fragment and a representative -multi-collection assembly with peer composition and an intentional override. -Applications select sources and destinations; the examples should require no -consumer-side parsing, generic merging, or provenance reconstruction. - -Completion: the supported root API is sufficient for the requested end-to-end -workflow, including accurate placement and source-relative resource inspection. - -### 5. Validate the full feature and update owning documentation - -Use engine tests for placement mechanics and public tests for observable loading -and inspection. Map every scenario in the request's acceptance table to a test -or executable example, reusing existing coverage where it proves the same -behavior. Include focused cases for: - -- omitted/empty destinations, literal escaped and numeric keys, invalid index - destinations, and the approved reserved-key and main-input rules; -- different destinations, shared destinations, ancestor overlaps, equal and - unequal conflicts, explicit null, omitted fields, and empty contributions; -- recursive precedence, atomic sequences, and domain-version-only overrides; -- inherited import destinations, source-root directives, repeated physical - sources at different destinations, and unchanged cycle/containment behavior; -- nested ordinary version fields versus actual protected-path and ancestor - claims, including mounted imported content; -- local inheritance across peers and source contexts after inheritance/overlay; -- effective diagnostic paths with real locations, callback paths, history, - rendering, diffs, and classified values protected with disposable canaries; -- equivalent content across every supported source kind, invalid loads with no - usable result, introduced depth/node limits, and detached inspection ownership. - -Follow the [testing policy](../policy/testing.md): protect observable contracts -without fixing incidental wrapper allocation or helper-call sequences. Run -focused tests, then the everyday check and ownership race validation in the -[development guide](../development.md). Validate local documentation links. -No independent performance benchmark runs are required for this roadmap. - -Update declaration documentation, the configuration contract, consumer guidance, -and the component overview as their behavior lands. Document how source-local -coordinates relate to effective paths and how placement is inspected. Review -release compatibility separately from publication; feature completion does not -authorize a release. - -Completion: all requested public scenarios pass, owning documentation describes -implemented behavior, and consumers can perform one complete layered load with -no compensating source-rewriting or post-load structural pass. - ## Selected Design Decisions ### Main-input destinations and absent inputs @@ -246,8 +133,8 @@ across its roles and preserves absence without per-fragment version checks. ### Public placement inspection and synthetic ancestors -Expose effective placement on existing source records, -illustratively `SourceInfo.Destination`, including inherited import placement. +Expose effective placement as `SourceInfo.Destination Path` on existing source +records, including inherited import placement. Use original source identity for introduced ancestor mappings, documenting that any root-location fallback identifies the contributing document rather than a physically authored wrapper. Actual content keeps its real coordinates. This diff --git a/docs/roadmap/promptkit-feature-request.md b/docs/roadmap/promptkit-feature-request.md new file mode 100644 index 0000000..7813d0e --- /dev/null +++ b/docs/roadmap/promptkit-feature-request.md @@ -0,0 +1,319 @@ +# PromptKit Feature Request: Configuration Input Destinations + +Status: consumer feature request; not accepted or implemented. + +## Requested Capability + +Provide a public API through which a consumer can select a destination mapping +path for each configuration input. The input's application content contributes +beneath that path while retaining ConfigKit's normal composition, precedence, +inheritance, validation, diagnostics, and inspection behavior. + +PromptKit needs this capability for its proposal 3, **Collections With Mounted +Configuration Fragments**, recorded in PromptKit's +`docs/roadmap/configuration-collections-mounted-fragments.md`. This request +describes the public API needs and observable acceptance criteria. It does not +prescribe ConfigKit's internal design or an implementation plan. + +The reviewed public baseline is ConfigKit v0.7.1. The current +[layered-loading contract](../config.md#layered-loading) lets consumers choose +sources, import policy, filesystem context, and precedence groups, but does not +provide an input destination. Existing behavior remains authoritative until an +extension is adopted. + +## Consumer Problem + +PromptKit applications assemble independently maintained collections of prompts, +inference profiles, and backends. Collections can contain identical local names, +such as `summarize.alerts`, without identifying the same resource. PromptKit wants +an effective configuration organized by collection, while letting authors omit +repeated collection and resource-container wrappers from individual files. + +A file selected from a collection's `prompts` directory would contain: + +```yaml +summarize.alerts: + version: "1" + default_profile: provider/brief + messages: + - role: user + content: Summarize the weather alerts. + output: + format: text + validation_mode: none +``` + +Given destination `[collections, weatherreporter, prompts]`, its contribution +would appear in the final application configuration as: + +```yaml +collections: + weatherreporter: + prompts: + summarize.alerts: + version: "1" + default_profile: provider/brief + messages: + - role: user + content: Summarize the weather alerts. + output: + format: text + validation_mode: none +``` + +Another collection can contribute `summarize.alerts` beneath +`[collections, homewatch, prompts]` without conflicting. After final handoff, +PromptKit interprets these as `weatherreporter/summarize.alerts` and +`homewatch/summarize.alerts`. The profile reference is illustrative and must be +supplied separately by the application configuration. + +PromptKit will support one installed definition per prompt. Its optional flat +`version` is ordinary domain metadata, not a merge identity. This request does +not need identified-record sequences, sequence-root documents, or a custom +merge policy. + +Authors could write the complete wrappers today, but that repeats assembly +details in every fragment. Consumer-side parsing and wrapping would put generic +source processing back into PromptKit and complicate source attribution. +PromptKit instead wants to select sources and destinations, then receive one +final configuration through ConfigKit's supported loading lifecycle. + +## Minimum Public API + +### Per-Input Destination + +Allow a destination to be attached to each `Input` passed to `LoadLayered`. +One possible public spelling is an optional `Destination Path` field on `Input`. +The name and exact signature are suggestions; equivalent public API designs +are welcome. + +Illustrative consumer code using the proposed field, which does not exist in +v0.7.1: + +```go +promptInput := configkit.Input{ + Source: configkit.File("/usr/local/share/collections/weatherreporter/prompts/alerts.yml"), + Destination: configkit.Keys("collections", "weatherreporter", "prompts"), +} + +peers := configkit.Underlay{ + Inputs: []configkit.Input{promptInput}, +} +``` + +The consumer would include `peers` in its existing `LayeredOptions` and invoke +`LoadLayered` with its normal root main input and application `Definition`. +This is a partial API sketch, not an executable example or a proposed new +loading entry point. A single public layered load is sufficient for PromptKit; +it does not require a separate mounting operation on completed results. + +The API must let the consumer: + +- Express a destination relative to the application configuration root using + literal mapping-key segments. Dots, slashes, and numeric-looking text inside + a key are not implicit path separators or array indexes. PromptKit's namespace + grammar remains its own domain rule. +- Omit the destination to retain existing root placement. An explicitly empty + destination should have the same meaning. Mapping destinations are sufficient; + sequence-index destinations are not needed by this request. +- Assign different destinations to inputs in the same strict composition group, + and assign the same destination to inputs in separate precedence groups. +- Use the destination with existing content-bearing source kinds: host files, + native filesystem files, bytes, readers, and mapping sources, subject to their + existing acquisition and ownership contracts. +- Retain the existing import opt-in and filesystem context independently of the + destination. A configuration destination is not a filesystem root or base + directory. + +PromptKit can continue to enumerate an explicitly selected directory and attach +the same destination to its returned inputs. ConfigKit does not need a new +directory scanner, namespace registry, filename convention, or PromptKit schema. +The namespace comes from the application's assembly declaration, not from an +implicit interpretation of a directory basename. + +### Representative Assembly + +These are application choices, not ConfigKit-specific reserved paths: + +| Selected inputs | Destination key segments | +| --- | --- | +| Application main document | Root | +| `weatherreporter/collection.yml` | `collections`, `weatherreporter` | +| YAML files under `weatherreporter/prompts` | `collections`, `weatherreporter`, `prompts` | +| YAML files under `homewatch/prompts` | `collections`, `homewatch`, `prompts` | +| YAML files under `provider/profiles` | `collections`, `provider`, `profiles` | +| YAML files under `provider/backends` | `collections`, `provider`, `backends` | + +Peer collection contributions belong to a single strict underlay group when the +application wants conflict detection among them. Intentional operator overrides +use later groups or main. Destination selection must not establish precedence +or make source traversal order authorize replacement. + +## Required Observable Behavior + +### Composition And Precedence + +Mounted and explicitly wrapped contributions must have equivalent application +values and conflict behavior at their effective destinations. Actual source +locations naturally differ and must remain accurate. + +Retain the [strict composition](../config.md#strict-document-composition) and +[layering](../config.md#layered-loading) rules: + +- Disjoint mapping entries at the same destination combine. +- Overlapping non-mapping values within a strict group conflict, including equal + scalars, nulls, and sequences. Different destinations do not collide merely + because the source documents share a local key. +- Later precedence layers recursively override mappings and replace other + values. A strict conflict cannot be repaired by an upper layer. +- Empty mappings do not clear lower definitions; ordinary arrays remain atomic. + In particular, `messages` does not acquire sequence-merging behavior. +- Overlapping destination paths and contributions at their ancestors follow + those same rules. No namespace-level uniqueness restriction is requested. + +Changing only a prompt's `version` in a later layer changes that field while +retaining omitted lower-layer fields. ConfigKit must not treat a domain version +label as a request to replace an entire definition. + +### Imports And Existing Structural Directives + +An input with imports enabled must be able to use its existing source-root +`configkit.imports` directive while contributing application data at a nested +destination. The directive remains structural configuration, not an application +field below that destination. + +For the minimum API, imports inherit their importing input's destination, +including transitive imports. Import paths continue to resolve relative to +their declaring source's filesystem context. A mounted document must not send +its imported application fields back to the configuration root accidentally. + +Existing opt-in, containment, missing-input, cycle, and repeated-source behavior +must remain applicable. A different destination is not a request to bypass the +current rejection of a repeated physical source within a strict group. + +Sibling `extends` must resolve against mappings at their effective destinations. +For example, two independently supplied profile fragments mounted beneath the +same collection's `profiles` mapping can establish `extends: brief` once they +compose. Inheritance remains ConfigKit-owned; PromptKit does not perform another +relationship-resolution pass. + +### Protected Specification Paths + +The distinction between source-local field names and effective application +paths is essential. PromptKit's main document will declare `version: 1` at the +configuration root. A separate collection metadata file can contain: + +```yaml +version: "2.4.0" +``` + +Mounted at `[collections, weatherreporter]`, that field is an ordinary collection +release label. It must not be rejected as an attempt to contribute the protected +root specification field. The same applies to a prompt's nested optional +version. Only the main-owned specification declaration uses the application's +configured [VersionRule](../config.md#application-schema-versions). + +Conversely, placement must not permit an underlay or imported fragment to claim +or replace the actual protected specification path or an ancestor that would +replace it. The public contract must define protection in terms of effective +application paths, rather than reserving the spelling `version` in every source. +Protection must remain valid with imports, inheritance, overlays, and explicit +CLI overrides. + +PromptKit needs one main-owned specification check with the existing lifecycle. +It does not need main mounted away from the root, integer-to-string conversion +for every nested `version`, or separate compatibility checks for every fragment. + +### Typed Loading, Diagnostics, And Inspection + +The destination must be visible consistently through the ordinary public +loading and inspection surface: + +- Final typed configuration and application callbacks use effective destination + paths. Defaults, resolution, and validation retain the existing single-load + lifecycle; fragments are not independently validated as whole applications. +- Metadata and presence queries distinguish omission, explicit null, and other + explicit values at the mounted paths. +- Diagnostics identify the effective conflicting or invalid path and the real + contributing source locations. Consumers can determine which source and + placement produced a value without recreating that relationship themselves. +- Effective rendering, explanations, history, and configuration comparison agree + with the final configuration. Existing sensitivity and disclosure rules apply + to mounted fields, including application path-based secret classification. +- Unsupported destination forms and destination-related loading failures use + the normal structured error boundary. Exact new diagnostic codes, if needed, + are for ConfigKit to define. + +PromptKit does not require a separate mount registry or new inspection facade. +Existing inspection APIs are sufficient if they expose accurate effective paths, +source attribution, and placement information. ConfigKit should document how +source-local locations relate to effective paths. + +### Source-Relative Resources And Lifecycle + +For a mounted field such as `content_file: ./summary.md`, obtaining its effective +origin and calling `Result.SourceContext` or `Resolution.SourceContext` must +still identify the declaring source's filesystem and base directory. The same +holds when that field survives an override or is inherited into another mapping. +Its configuration destination must not change the meaning of the relative file +reference. + +Retain existing source ownership, load completion, result lifetime, containment, +and resource-access behavior. Destinations must participate in documented load +limits without creating a way to bypass them; this request does not prescribe +new limit values or their accounting machinery. + +## Optional Follow-Ups, Not Prerequisites + +**Import destinations in YAML.** A destination on an import directive could let +publishers describe assembly in configuration rather than Go. Proposal 3 +recommends destinations relative to the importing document's destination, with +omission retaining that destination. This is useful, but per-input placement +through the public API is sufficient for initial PromptKit adoption. No new +directive spelling is selected by this request. + +**Independent specification headers.** A collection entry document might later +declare its own supported specification version. That needs a distinct contract +for checking the header and determining its role in the composed configuration. +Mounting collection release metadata does not supply this capability. Keep it a +separate request if PromptKit decides it is necessary. + +**Cross-collection inheritance.** Nested collection profiles are not immediate +siblings across collections. Current sibling inheritance remains sufficient for +the initial proposal, with authored definitions flattened where necessary. +Generic cross-path inheritance could be considered separately if consumer needs +justify revisiting the [deferred scope](deferred.md). Mounting must not silently +reinterpret `extends: provider/brief` as a cross-collection reference. + +PromptKit also does not request collection installation, automatic directory +discovery, remote fetching, namespace ownership enforcement, dependency solving, +live reload, filename-derived identity, or arbitrary document transformations. +Its qualified resource IDs, optional prompt version semantics, collection +inventory, and execution APIs remain PromptKit responsibilities. + +## Public Acceptance Scenarios + +Evaluate the extension through supported public loading and inspection APIs: + +| Scenario | Expected consumer-visible result | +| --- | --- | +| Destination omitted or empty | Existing root-placement behavior | +| Same fragment shape mounted under two distinct collection keys from independent sources | Two separate effective resources | +| Disjoint inputs at the same destination in one group | Mapping contributions combine | +| Equal or unequal overlapping scalar fields in one group | Structured composition conflict with both source locations | +| Same destination across deliberate precedence tiers | Existing recursive overlay behavior | +| Version-only prompt override | Label changes; omitted prompt body fields survive | +| Nested mounted import without a destination directive | Imported fields contribute at the inherited destination | +| Local profile inheritance across mounted peer fragments | Parent resolves at the effective sibling scope | +| Collection metadata containing `version` mounted below root | Ordinary nested field, independent of root VersionRule | +| Mounted/imported contribution actually targeting the protected version path | Rejected under version ownership rules | +| Mounted field using source-relative content, including inheritance | Original declaring resource context remains available | +| Mounted explicit null and omitted field | Distinct effective presence information | +| Mounted validation error or secret field | Effective paths, physical locations, and existing disclosure rules agree | +| File, native filesystem, byte, reader, or mapping input with equivalent content | Equivalent application values, subject to existing source contracts | +| Unsupported destination or failed load | Normal structured failure, no usable partial result | + +The minimum usable outcome is a consumer that selects collection sources and +destinations, performs one ordinary layered load, and receives final typed +configuration with reliable diagnostics and inspection. It should need no +consumer-side source rewriting, generic merging, or provenance reconstruction.