Plan configuration input destinations

docs/roadmap/input-destinations.md

@@ -0,0 +1,292 @@
+# Configuration Input Destinations Roadmap
+
+Status: proposed; implementation has not started. Execute the stages below in
+order. This roadmap responds to the
+[PromptKit feature request](promptkit-feature-request.md) and owns the target
+state, delivery sequence, and unresolved design choices. Public spelling and
+recommendations below are proposals until selected explicitly.
+
+## Targeted end state
+
+A consumer can assign a mapping destination to a configuration input and perform
+one ordinary `LoadLayered` call. The input's application content contributes
+beneath that destination while retaining normal composition, precedence,
+inheritance, typed loading, diagnostics, and inspection. PromptKit can assemble
+collections without parsing and rewriting sources, merging values, or rebuilding
+provenance itself.
+
+For example, a fragment defining `summarize.alerts` can contribute beneath
+`collections`, `weatherreporter`, `prompts`. An independent source with the same
+local key can contribute beneath another collection without conflict. Collection
+and prompt names remain ordinary application keys, not ConfigKit namespaces.
+
+The intended public surface is per-input placement, illustratively an optional
+`Destination Path` field on `Input`, not a new loading entry point or an operation
+on completed results. The minimum supported workflow has a root main input and
+mounted underlay inputs; broader main placement is an open decision below.
+
+The following behavior defines the target:
+
+- Omitted and empty destinations retain root placement. Destinations consist of
+  literal mapping keys, including dots, slashes, and numeric-looking text.
+  Explicit sequence-index destinations are outside the requested capability.
+- Host files, native filesystem files, bytes, readers, and mapping sources all
+  support placement under their existing acquisition and ownership contracts.
+  Filesystem context, import opt-in, and precedence remain independent choices.
+- Mounted contributions have the same application values and conflict behavior
+  as explicitly wrapped contributions, with accurate original source locations.
+  Disjoint mappings combine; overlapping non-mapping values within a group
+  conflict even when equal. A later tier cannot repair a strict-group conflict.
+- Different destinations can coexist in one group, and the same destination can
+  occur in different tiers. Ancestor/descendant destinations follow ordinary
+  composition and overlay rules without namespace uniqueness restrictions.
+  Later tiers recursively overlay mappings and replace other values. Empty
+  mappings do not clear lower values; sequences remain atomic.
+- A prompt's ordinary `version` override changes that field and retains omitted
+  lower fields. It neither identifies a merge record nor replaces a whole prompt.
+- Source-root `configkit.imports` remains structural. Imported application data
+  inherits the input destination transitively. Import paths still resolve from
+  their declaring source; containment, optionality, cycles, and repeated-source
+  rules remain applicable. Placement does not change physical source identity.
+- Sibling inheritance runs after assembly in the effective destination scope.
+  Peer fragments can supply a child and its sibling parent. Qualified PromptKit
+  references do not acquire cross-collection inheritance meaning.
+- Version ownership is enforced at effective application paths. A collection's
+  mounted `version` is ordinary data, while underlay/import contributions to the
+  actual protected path, or replacements of its ancestors, remain invalid.
+  There is one main-owned version check and no blanket nested numeric conversion.
+- Defaults, decoding, resolution, secret lookup, and validation run once on the
+  assembled application, not independently on fragments. Callbacks and value
+  diagnostics use effective paths. Failures use structured load errors and yield
+  no usable partial result.
+- Presence, provenance, history, explanations, effective rendering, and diffs
+  agree with the final values. Sensitivity is evaluated at effective application
+  paths under the existing historical and dependency rules.
+- Original source locations and passive resource contexts survive placement,
+  overrides, and inheritance. `Result.SourceContext` and
+  `Resolution.SourceContext` still identify the declaring filesystem and base
+  directory for a field such as `content_file: ./summary.md`.
+- Destinations and the structure they introduce participate in existing load
+  limits; placement cannot bypass depth or node accounting. Source ownership,
+  complete-on-return loading, and detached result behavior remain intact.
+
+Import-destination syntax, independent collection specification headers, and
+cross-collection inheritance are separate possible follow-ups. This work does
+not add directory discovery, collection installation, remote acquisition,
+namespace ownership enforcement, dependency solving, live reload, filename-based
+identity, arbitrary transformations, or new merge strategies. PromptKit owns its
+collection inventory, qualified IDs, and execution semantics.
+
+## Current baseline
+
+The request reviewed v0.7.1. The current worktree has these relevant boundaries:
+
+- [Input](../../layered.go) holds a source, import opt-in, and filesystem context,
+  but no destination. Root types delegate to the engine through the existing
+  [facade boundary](../internal/overview.md).
+- [Layered loading](../config.md#layered-loading) already provides strict groups,
+  ordered precedence, shared acquisition budgets, and one final lifecycle.
+  [Underlay acquisition](../../internal/engine/load_request.go) gathers source
+  applications and recomposes them into their strict group.
+- [Input acquisition](../../internal/engine/load_input.go) and
+  [import traversal](../../internal/engine/imports.go) process source-root
+  directives and source identity before final assembly. Imported version claims
+  are checked during traversal; underlay claims are checked before group
+  composition. These currently operate without a destination.
+- [Version claim checks](../../internal/engine/directives.go) inspect a document
+  using the configured application path. Wrapping only an already acquired
+  result would leave early fragment checks and diagnostic paths incorrect.
+- [Sibling inheritance](../adr/0009-sibling-inheritance.md) already runs after
+  source precedence and preserves source-to-destination derivation. Placement
+  must feed that existing mechanism rather than resolve fragments independently.
+- Existing source records identify acquisition and precedence but do not expose
+  input placement. Existing history retains source applications, so moving only
+  the final tree would be insufficient for coherent inspection.
+
+The architectural fit is an extension of input assembly, not a parser extension
+or a second configuration model. The
+[architecture policy](../policy/architecture.md) supplies the ownership and
+consistency constraints; [deferred scope](deferred.md) continues to govern the
+unrequested features above.
+
+## Ordered implementation plan
+
+### 1. Settle placement semantics and the public contract
+
+Resolve the Open Decisions before their dependent implementation. Select the
+public API spelling, main/NoSource behavior, placement inspection, reserved-key
+handling, and empty-contribution semantics. Define destination validation and
+structured errors, including when validation occurs relative to source access.
+Use existing error categories where suitable rather than inventing a parallel
+error system.
+
+Record the significant assembly decision in an ADR. Review compatibility under
+the release policy, including exported struct changes and unchanged zero-value
+behavior. Keep unimplemented claims in the roadmap; update canonical contracts
+when implementation lands. Do not turn unrelated deferred features into work.
+
+Completion: selected semantics cover the request's public acceptance scenarios
+and a simple consumer as well as PromptKit's multi-collection workflow.
+
+### 2. Carry destinations through acquisition and imports
+
+Thread the selected destination through root-to-engine conversion and private
+load requests. Preserve its ownership and semantic key representation. Separate
+source-local directive processing from application placement: extract directives
+at their original root, inherit placement through traversal, and place application
+content before checks and composition that require effective paths.
+
+Make version protection destination-aware for underlay roots and imported
+fragments, preserving the main check's lifecycle position. Ensure diagnostics
+produced before final composition identify effective application paths where
+available while preserving actual source coordinates. Structural directive
+errors still need their genuine source-local declaration locations.
+
+Retain repeated-source detection across the whole strict group, independent of
+destination. Preserve filesystem namespace, resource contexts, and traversal
+ordering. Account for destination depth and introduced nodes using existing
+budgets; distinguish them from acquired source bytes and physical source count.
+
+Completion: all content-bearing source kinds and transitive imports contribute
+at the intended destination, with correct version protection and source identity.
+
+### 3. Integrate composition, provenance, and the final lifecycle
+
+Apply ordinary composition to placed contributions and ordinary overlay between
+tiers. Retain each source application at its effective placement for history;
+carry real source attribution and the selected synthetic-container convention
+through origin construction. Avoid a separate merge algorithm or a mount registry.
+
+Exercise overlapping destinations, empty mappings, and a version-only domain
+override. Resolve sibling inheritance after the full precedence assembly, then
+use existing defaults, typed decoding, resolution, secrets, and validation.
+Check protection of the specification path through inheritance, selected
+overlays, and CLI overrides as well as direct contributions.
+
+Completion: mounted and explicitly wrapped equivalents agree on values,
+conflicts, presence, and lifecycle results; physical locations remain accurate.
+
+### 4. Complete inspection and public consumer workflows
+
+Expose placement through the selected existing inspection surface, with detached
+records. Verify effective paths throughout metadata, explanation replay, history,
+rendering, and diffs. Apply the existing sensitivity rules to mounted and inherited
+fields, including superseded values. Do not fabricate coordinates for wrappers
+that were never written in a source file.
+
+Demonstrate source-relative resource lookup from both resolution and completed
+results, including inherited fields and lower-tier fields surviving an override.
+Add executable public examples for one mounted fragment and a representative
+multi-collection assembly with peer composition and an intentional override.
+Applications select sources and destinations; the examples should require no
+consumer-side parsing, generic merging, or provenance reconstruction.
+
+Completion: the supported root API is sufficient for the requested end-to-end
+workflow, including accurate placement and source-relative resource inspection.
+
+### 5. Validate the full feature and update owning documentation
+
+Use engine tests for placement mechanics and public tests for observable loading
+and inspection. Map every scenario in the request's acceptance table to a test
+or executable example, reusing existing coverage where it proves the same
+behavior. Include focused cases for:
+
+- omitted/empty destinations, literal escaped and numeric keys, invalid index
+  destinations, and the selected reserved-key and main-input rules;
+- different destinations, shared destinations, ancestor overlaps, equal and
+  unequal conflicts, explicit null, omitted fields, and empty contributions;
+- recursive precedence, atomic sequences, and domain-version-only overrides;
+- inherited import destinations, source-root directives, repeated physical
+  sources at different destinations, and unchanged cycle/containment behavior;
+- nested ordinary version fields versus actual protected-path and ancestor
+  claims, including mounted imported content;
+- local inheritance across peers and source contexts after inheritance/overlay;
+- effective diagnostic paths with real locations, callback paths, history,
+  rendering, diffs, and classified values protected with disposable canaries;
+- equivalent content across every supported source kind, invalid loads with no
+  usable result, introduced depth/node limits, and detached inspection ownership.
+
+Follow the [testing policy](../policy/testing.md): protect observable contracts
+without fixing incidental wrapper allocation or helper-call sequences. Run
+focused tests, then the everyday check and ownership race validation in the
+[development guide](../development.md). Validate local documentation links.
+No independent performance benchmark runs are required for this roadmap.
+
+Update declaration documentation, the configuration contract, consumer guidance,
+and the component overview as their behavior lands. Document how source-local
+coordinates relate to effective paths and how placement is inspected. Review
+release compatibility separately from publication; feature completion does not
+authorize a release.
+
+Completion: all requested public scenarios pass, owning documentation describes
+implemented behavior, and consumers can perform one complete layered load with
+no compensating source-rewriting or post-load structural pass.
+
+## Open Decisions
+
+### Main-input destinations and absent inputs
+
+Recommendation: allow the same destination model for content-bearing main inputs
+as for underlays, with `VersionRule.Path` remaining an effective application path
+and any specification declaration still owned by main. The checker retains its
+existing handling of an absent declaration. Treat `NoSource` as no contribution,
+even with a valid destination. This keeps `Input` coherent and avoids a role-only
+restriction. PromptKit can continue using root main; broader main placement must
+not add per-fragment checks or move the existing version-check phase.
+
+Alternative: initially support non-root destinations only for underlay inputs
+and reject non-root main placement explicitly. This meets PromptKit's minimum
+need and reduces the first implementation's version-check interactions, but adds
+a role-dependent rule to a shared public type. Adopt that restriction only as an
+explicit scope choice, not because threading main placement is inconvenient.
+For `NoSource`, rejecting a nonempty destination could catch mistaken assembly
+declarations, but makes an otherwise absent contribution less composable.
+
+### Public placement inspection and synthetic ancestors
+
+Recommendation: expose effective placement on existing source records,
+illustratively `SourceInfo.Destination`, including inherited import placement.
+Use original source identity for introduced ancestor mappings, documenting that
+any root-location fallback identifies the contributing document rather than a
+physically authored wrapper. Actual content keeps its real coordinates. This
+allows consumers to connect placement to sources without a new query system.
+
+Alternative: add placement to origin records or a dedicated lookup, and represent
+synthetic ancestors without a physical location. This can distinguish authored
+and synthetic structure more precisely, but expands per-value metadata or adds
+an API when placement is fundamentally per source application. Evaluate the
+choice against inherited fields, composite origins, and the same source selected
+in separate tiers; do not infer placement from a final path by guesswork.
+
+### Reserved structural keys in destinations
+
+Recommendation: use existing `Path` conventions for literal keys and parsed
+numeric tokens, reject explicit index segments, and preserve current structural
+reservations in the effective tree. Reject destination segments that would
+materialize `configkit` or a mapping-valued `extends` contrary to those contracts,
+with a clear structured destination error. Ordinary keys containing those words,
+dots, or slashes remain ordinary keys. Source-root directives are processed before
+placement and are not rejected simply because their application body is mounted.
+
+Alternative: permit structural spellings as literal synthetic wrapper keys by
+distinguishing them from authored directives. This enables additional application
+paths but introduces an exception where mounted and explicitly wrapped inputs
+can have different meanings. It would require an explicit change to reserved-key
+semantics and is not needed for the requested collection paths. Running existing
+directive validation on the placed tree is another viable error mechanism, but
+must still identify that the invalid structure came from destination selection.
+
+### Empty mappings and presence of introduced containers
+
+Recommendation: mounting an actual empty application mapping materializes its
+destination mappings as explicit contributions, matching authored wrappers.
+This preserves the request's equivalence principle and lets consumers inspect
+an intentionally empty collection. Such mappings still do not clear lower-tier
+children. An absent source contributes nothing.
+
+Alternative: treat an empty mounted body as no contribution. This avoids empty
+containers for fragments containing only imports or no application fields, but
+loses equivalence with explicitly wrapped empty mappings and erases an observable
+presence distinction. If selected, document that exception and its effect on
+history and empty imported bodies rather than deriving it accidentally from
+whether the implementation allocated wrapper nodes.