ConfigKit/docs/roadmap/implementation.md

# Implement configuration input destinations

Status: planned. Execute Stage 1 through Stage 12 in order, one focused stage
per Luna implementation prompt. The [feature roadmap](input-destinations.md)
owns the approved behavior and rationale; the
[consumer request](promptkit-feature-request.md#public-acceptance-scenarios)
supplies the public acceptance scenarios. This document owns execution and
completion evidence. Planning baseline: `6353598`.

## Working instructions

- Start each stage with `AGENTS.md`, [development.md](../development.md), the
  applicable policies, and working-tree inspection. Use graph-first code
  discovery. Preserve unrelated changes and previously completed stages.
- Implement only that stage and its supporting tests/documentation. Mark it
  complete here with actual commands and evidence, then commit the nonempty
  validated unit with a short plain-English message. Subsequent stages remain
  planned. A stage is not complete solely because its narrow tests pass.
- Keep early mechanics private until the complete engine behavior is ready.
  Root destination fields land in Stage 10. Ordinary root loading must remain
  working throughout. Do not publish a partially supported public placement API.
- Follow the existing engine/document/YAML boundaries. Reuse strict composition,
  recursive overlay, sibling inheritance, paths, history, and sensitivity.
  No dependencies, toolchain changes, alternate mergers, mount registry,
  per-fragment lifecycle, import-destination syntax, resource discovery, or new
  release gates are needed. These are scope boundaries, not new prohibitions
  on separately approved future work.
- Update current-behavior documentation when public behavior lands; keep future
  claims here or in the roadmap until then. Add the ADR in Stage 1 under the
  documentation policy's accepted-decision exception.
- Use focused offline tests while iterating; run `./scripts/check.sh` before
  committing production changes. Documentation-only stages need whitespace and
  local-link checks. Stage 12 adds full ownership/race validation. Tests protect
  observable behavior, not wrapper allocation or helper-call counts. Reuse
  coverage rather than repeat an entire loading matrix at every layer.
- No benchmark campaign or public complexity guarantee is required. Prefer
  straightforward bounded walks and wrapping each acquired contribution once;
  avoid rescanning all preceding contributions for every new input.
- Do not push, tag, publish, update downstream dependencies, or retire unfinished
  records. Release compatibility review is separate from publication.

## Resolved implementation decisions

1. Add root-owned `Input.Destination Path` and `SourceInfo.Destination Path`,
   mirrored privately in engine input/source records. Extend `loadInput` and
   explicit root/engine conversion and cloning. No new loading entry point,
   public wrapper type, option, or origin field is needed. Ordinary `Load`
   continues to supply root placement.
2. The zero path means root. Validate `Path.valid()` and reject every explicit
   index segment, including a negative index. Normalize parsed tokens to literal
   mapping-key segments: `/0` means the key `0`, not sequence element zero.
   Empty, dot, dot-dot, slash-containing, tilde-containing, and numeric keys keep
   existing semantic-path meaning. Do not split dots, interpret filesystem
   traversal, enforce identifier grammar, or add namespace uniqueness.
3. Reject destination segments equal to `configkit` or `extends`: placement
   would introduce a reserved mapping key with a mapping value. Substrings such
   as `myextends` are ordinary keys. This preserves the approved structural
   rules rather than adding an application naming policy. Source-root import
   directives are still processed before placement.
4. Validate destination shape/reservations for all declared inputs during
   grouped option preflight, before reading any source. Reuse
   `CodeOptionsInvalid` for invalid destination declarations; include the
   offending semantic path when representable, with no fabricated physical
   location. An invalid source still follows existing source-option validation.
   A valid destination on `NoSource` contributes no nodes, source record, or
   budget charges. No extra error type or exact message contract is introduced.
5. Separate acquired source syntax from application placement. Retain original
   parsed source metadata for filesystem/directive handling. Strip source-root
   directives, then wrap each application body at the normalized destination
   before fragment version checks and strict composition. Imports inherit the
   same destination transitively. Do not wrap both a source application and its
   recomposed result; every retained application must already use effective
   coordinates. Content roots remain mappings under existing source contracts.
6. Wrapping a body at `Keys("a", "b")` creates `{a: {b: body}}` as private
   document mappings. Actual body nodes and entries keep their physical source
   coordinates. Introduced mappings/entries use the contributing document's root
   location as a documented source fallback, not purported authored coordinates.
   They have the same contributing source ID and create no synthetic source.
   Retained history trees remain immutable and separate from assembly mutation.
7. A present empty body creates explicit destination mappings, including an
   import-only source whose application body is empty. It still cannot clear
   lower mappings. A missing optional import and `NoSource` remain absent. Empty
   source text remains the existing invalid document, not an empty mapping.
8. `VersionRule.Path` stays an effective application path. Check main once after
   its own imports compose and before underlays/overlay/CLI, at the existing
   phase. Imported and underlay claims are checked against placed application
   content, including scalar/null/sequence ancestors replacing a protected path.
   A mapping ancestor with no protected descendant is not a version claim.
   Mounted domain `version` fields are ordinary data. Existing inheritance,
   selected-overlay, CLI protection and exact-path numeric conversion remain;
   no nested blanket conversion or additional fragment checker is introduced.
9. Placement does not alter physical source identity, filesystem namespace,
   source IDs, acquisition order, imported-from locations, or resource contexts.
   Repeated physical sources in one strict group still fail even at different
   destinations. The same physical source in separate tiers retains the existing
   independent source applications/records and each record's own destination.
   Overlay and CLI source records have root `Destination`; they are not mounted
   merely because main is mounted. Active layered overlays retain their explicit
   independent filesystem selection and application-root meaning.
10. Value-level paths use effective coordinates. Keep genuine source-local
    import-directive diagnostics and parse/read failures at their original
    locations; do not invent a logical child path when parsing cannot identify
    one. Prefix existing application-relative diagnostic paths exactly once
    where available. Preserve structured codes, related physical locations,
    deterministic ordering, and value-free messages.
11. Use current depth/node limits, not a new destination cap. With root depth
    zero, each destination segment adds one level and, under `countInputNodes`,
    one mapping node plus one key. Retain current raw input accounting (including
    directives); charge only the additional `2 * segmentCount` synthetic nodes
    per acquired body, once, to the shared input budget. Placement adds no bytes,
    acquired sources, import nesting, or glob visits. Validate placed application
    depth and per-document node bounds before composition; share accounting
    across inputs and imports. Precheck wrapper depth/count before allocating
    wrappers. Use existing source/import limit diagnostics in their acquisition
    contexts; retain projection, expansion, and history bounds independently.
    Root placement adds no charge or new rejection. NoSource adds none either.
12. Cloned source records include independently owned destination paths at every
    existing publication boundary: results, load errors, metadata/inspection,
    callback input information, and root conversion. Passive resource contexts
    remain keyed by the real source identity; inspection must work without an
    open configuration handle after load. This does not extend caller-owned
    filesystem or callback resource lifetimes.

## Stage 1 — Record the decision and capture the baseline

Status: complete. Scope: decision record and evidence, no feature code.

1. Read the roadmap, request, relevant policies, and ADRs 0001, 0003, 0006,
   0008, and 0009. Inspect the owners identified in the roadmap and the function
   map below. Confirm the baseline and worktree before starting.
2. Run `./scripts/check.sh`; record the commit, Go version, and results here.
   Locate existing coverage for root loading, strict underlays, version ownership,
   repeated sources, history, resource contexts, and detached records.
3. Write `docs/adr/0011-input-destinations.md` after confirming the number is
   unused. Record the approved input-placement decision as Accepted, with the
   roadmap owning implementation status. Follow the repository ADR format.
   Focus on ConfigKit, put rejected options only in Alternatives Considered,
   and avoid negative parallelism elsewhere. No PromptKit-specific semantics.
4. Record the compatibility assessment under the release policy: adding fields
   preserves zero-value/keyed-literal use, but callers using unkeyed literals
   for `Input` or `SourceInfo` must migrate to keyed literals. Treat the feature
   as minor-release work; select a version and write its note only during
   separately authorized release preparation.

Validated: `./scripts/check.sh` (go1.26.8 linux/amd64) passed on candidate
`d3d468a8b47d780787fca329ff65b107d98ca59f`; this includes formatting,
whitespace, local documentation links, tests, vet, and build. The planning
baseline `6353598bd6cce2eb78a5fe990e144cc0cadf1280` is present in history. The
worktree was clean before this stage.

Coverage located for later destination work:

- Root and equivalent source loading: `TestEquivalentSourceFormsPreserveInputSemantics`
  (`facade_equivalence_test.go`) and
  `TestFacadeLayeredZeroUnderlaysMatchesOrdinaryLoad` (`facade_layered_test.go`).
- Strict underlays and group budgets: `TestGroupedLoadStrictConflictsRepetitionAndBudgets`
  and `TestGroupedLoadPrecedenceSourceOrderAndSingleLifecycle`
  (`internal/engine/load_request_test.go`).
- Version ownership and imported claims: `TestLoadRejectsImportConflictsAndFragmentVersions`
  (`internal/engine/imports_test.go`) and version-path coverage in
  `internal/engine/load_test.go`.
- Repeated physical sources: `TestRepeatedImportsAcrossTiersAndReaderConsumption`
  (`facade_equivalence_test.go`).
- History and source contexts: `TestUnderlayHistoryRetainsLaterMappingContributors`
  (`facade_underlay_origins_test.go`) and
  `TestFacadeSourceContextsFollowOriginsWithoutQueryIO`
  (`facade_resource_context_test.go`).
- Detached records: `TestLoadMetadataReflectsDefaultsInputAndDetachedRecords`
  (`internal/engine/load_test.go`) and
  `TestMetadataQueriesReturnDetachedRecords` (`internal/engine/metadata_test.go`).

The decision is recorded in [ADR 0011](../adr/0011-input-destinations.md).
Compatibility assessment: adding the two public fields breaks callers using
unkeyed literals for `Input` or `SourceInfo`; keyed and zero-value use remains
compatible. The release policy permits the intentional pre-1.0 contract change
in a minor release when the release note states impact and migration. Version
selection and release-note preparation remain out of scope until separately
authorized. No feature code or current behavior contract changed.

## Stage 2 — Add private destination normalization and wrapping

Status: complete. Scope: pure private mechanics with bounded unit tests.

1. Add a small engine helper file for normalized mapping-only destinations and
   constructing placed document bodies. Follow decisions 2–7; keep helpers
   private and avoid introducing a second path abstraction.
2. Validate invalid paths, explicit indices, and exact reserved tokens. Normalize
   parsed numeric tokens into keys, preserving literal text and pointer escaping.
3. Build wrappers without mutating the supplied tree. Preserve body locations;
   give introduced entries/mappings the source-root fallback. Implement depth and
   node measurement/prechecks usable with existing limits, including the extra
   synthetic-node count; aggregate charging is wired in Stage 7.
4. Cover root identity/equivalence, nested empty mapping, literal/escaped/numeric
   keys, invalid index/path, exact versus substring reserved keys, owned paths,
   unchanged body coordinates, and bounded wrapper checks. Do not count helper
   invocations or fix an allocation strategy in tests.

Validated with `go test ./internal/engine -run 'Destination|PlaceDocument'`
and `./scripts/check.sh` (go1.26.8 linux/amd64); both passed. The private
helpers normalize mapping-only destinations, preserve authored nodes and source
coordinates, and check per-document depth/node bounds before allocating wrappers.
No loader accepts a new public field yet.

## Stage 3 — Integrate private non-import input placement

Status: planned. Scope: engine request plumbing and ordinary acquired bodies.

1. Add destination fields to engine `Input`, `loadInput`, and `SourceInfo`.
   Preserve ownership through engine `LoadLayered` request construction,
   `cloneSourceInfo`, and source-record copies. Keep the root facade unchanged.
2. Extend grouped preflight to validate/normalize destinations before source
   reads. Use the existing diagnostic accumulator and options-invalid category.
   Ordinary `Load` supplies root placement through its existing request adapter.
3. Thread placement into `assembleBaseInputTracked` without confusing raw
   `parsedSource` with effective `assembledInput.document`. Preserve the root
   path through existing wrappers/callers. For imports-disabled acquisition,
   validate disabled directives on raw content, place the body, and retain the
   destination on the acquired source record. NoSource still short-circuits.
4. Use engine loads to cover mounted main and underlay bodies, omitted versus
   explicit root destinations, empty mapping versus NoSource, and preflight
   rejection without consuming a test-owned reader. Reuse current source-kind
   acquisition tests; full source equivalence is verified publicly in Stage 10.

Validate: destination, grouped-load, source, and existing ordinary-load tests;
the everyday check. Done: private non-import placement reaches assembly, with
root behavior unchanged. Imports are the next private integration step.

## Stage 4 — Carry placement through import acquisition

Status: planned. Scope: source-local directives and placed imported applications.

1. Thread a normalized destination through `loadImportedDocumentsTracked`,
   `loadImportedWithAccessTracked`, and `importTraversal`, retaining root defaults
   for current private callers. Place the selected body's application content
   after `splitImportDirectives`; do the same for every imported body.
2. Set placement on selected/imported source records, including error records
   once a source has been acquired. Keep `ImportedFrom` at the physical directive.
   Run imported `fragmentVersionClaim` against placed bodies and retain placed
   bodies in `state.documents` before `composeDocuments` builds applications.
3. Leave filesystem joins, containment, glob traversal, optional missing inputs,
   active cycles, visited identities, and acquisition IDs/order unchanged.
   Destination must not become part of the repeated-source identity key.
4. Test native and host transitive imports, nested source-root directives,
   import-only bodies, and mounted imported protected versus ordinary version
   paths. Verify a repeated file at two destinations in one group still fails,
   including an import/direct-input alias. Reuse cycle/containment regressions.

Validate: focused destination/import/glob/grouped tests and everyday check.
Done: every retained imported contribution has effective placement and physical
source identity, with no double wrapping or application-level directive leakage.

## Stage 5 — Complete strict tiers and main version ownership

Status: planned. Scope: assembly and version interactions, not a new merger.

1. Review `acquireUnderlayGroup`: version checks must use placed content, and
   documents taken from `acquired.composition.applications` must already be
   placed. Recompose strict peers with ordinary `composeDocuments`; preserve
   per-source applications, origins, UnderlayIndex, and input accounting.
2. In `executeLoadRequest`, main's version check uses its placed main/import
   composition at the existing point. `assembleLayeredInput` recursively overlays
   completed underlay groups beneath main, selected overlay, and CLI as before.
   Do not mount the final assembly or prepend main's destination to upper tiers.
3. Test disjoint same-destination peers, different destinations, equal and unequal
   conflicts, ancestor/descendant destinations, and a strict conflict despite an
   upper override. Verify recursive mapping overlays, atomic sequences, null,
   explicit false/zero, and version-only domain overrides retaining lower fields.
4. Exercise mounted main with a nested VersionRule, absent root declarations,
   imported/underlay direct and ancestor claims, and nonclaiming mapping ancestors.
   Check the hook runs once and does not inspect each fragment independently.
   Preserve exact configured-path integer conversion and ordinary nested typing.

Validate: destination/composition/overlay/version/grouped tests and everyday
check. Done: placed and authored equivalents agree on values and conflicts;
main ownership and lifecycle timing are preserved.

## Stage 6 — Reconcile diagnostic coordinate systems

Status: planned. Scope: useful paths without fictional source locations.

1. Audit early acquisition, import, fragment-version, and composition failures.
   Use one narrow path-prefix translation where diagnostics genuinely identify
   application content; do not blanket-prefix every diagnostic or double-prefix
   diagnostics already produced by placed trees.
2. Preserve syntax/read/directive locations and related import sites. Value
   conflicts and later typed/resolver/validation failures use effective paths
   but original content coordinates. Invalid introduced structure identifies
   the destination declaration rather than a nonexistent source wrapper line.
3. Add focused assertions for a mounted conflict with two real sources, a mounted
   decode or validation error, an imported version error, invalid destination,
   and malformed source-local directives. Test stable codes/paths/locations and
   nil results, not whole messages or callback choreography.

Validate: destination/diagnostic/import/version/decode tests and everyday check.
Done: diagnostics consistently distinguish application paths from source syntax.

## Stage 7 — Complete shared placement accounting

Status: planned. Scope: resource bounds with no new public limit system.

1. Wire Stage 2's prechecks and synthetic-node charges into both tracked ordinary
   acquisition and import traversal, using `importAccounting` and current limit
   structures. Charge each source body once, including empty import bodies.
2. Preserve existing raw byte/source/node accounting and imported directive
   inspection. Additional wrappers count in per-body depth/node checks and
   shared aggregate nodes; recomposition must not charge them again. Ensure
   selected overlays and CLI start from totals including placement charges.
3. Test below/at/above relevant bounds using small private limit fixtures where
   existing seams permit; add a narrow private seam if necessary. Cover deep
   body plus destination, many shallow mounted inputs, imported wrappers, and
   empty mappings. Verify root placement and NoSource do not add charges.
   Preserve history/expansion limits; do not turn fixture values into defaults.

Validate: destination/limit/import/override tests and everyday check. Done:
placement cannot bypass current budgets and introduces no unrelated caps.

## Stage 8 — Preserve provenance, history, and source resource contexts

Status: planned. Scope: engine inspection and retained evidence.

1. Audit `compositionOrigins`, `completeCompositionOrigins`, application replay,
   metadata construction/finalization, and source-record cloning against placed
   applications. Prefer feeding placed trees into existing processing rather
   than adding another path-offset index or rewriting final metadata afterward.
2. Verify explicit destination containers, actual leaves, composite origins,
   superseded/removed descendants, and root/source fallbacks. Historical and
   effective indexes keep independently owned paths and existing shape semantics.
3. Preserve source-ID resource contexts through imports, surviving lower-tier
   values, and inheritance. Test completed-result and resolution-time context
   queries against real relative files, without making ConfigKit open the
   referenced application resource or retain loading handles.
4. Exercise the same physical source in separate tiers with different placement,
   inspect each source record, and verify independent destination paths. Repeat
   inspection after comparison and independent loads to detect retained mutation.

Validate: destination/metadata/provenance/history/resource tests,
`go test -race ./internal/engine`, and everyday check. Done: inspection explains
effective placement and physical origin without consumer reconstruction.

## Stage 9 — Verify final inheritance, sensitivity, and lifecycle integration

Status: planned. Scope: complete private engine behavior before facade exposure.

1. Resolve sibling inheritance only after full source precedence using the
   existing expansion path. Cover a child and parent supplied by separate peers,
   nested destination-local siblings, cancellation, and an inherited relative
   resource retaining its declaring context. Cross-collection search is unchanged.
2. Reuse existing inheritance-version, overlay, and CLI guards; add targeted
   mounted cases for actual protected paths and their ancestors. Do not reject
   unrelated inheritance merely because it shares a mapping ancestor.
3. Test effective-path secret predicates, field tags and recorded dependencies,
   superseded classified values, safe effective output/history/explanation, and
   bilateral diff redaction using disposable canaries. Compare mounted versus
   explicitly wrapped loads for equal effective values and no differences.
4. Verify defaults, decoding, resolution, secret lookup, and validation execute
   once on the final shape; failures return no usable result. Reuse existing
   ordinary-load lifetime/ownership coverage and add mounted protection where
   placement creates a distinct risk.

Validate: destination/inheritance/secret/sensitivity/explain/diff/lifecycle tests
and everyday check. Done: all approved engine semantics work together without
per-fragment loading or a post-return processing pass.

## Stage 10 — Expose the root-owned facade and public acceptance tests

Status: planned. Scope: the two public fields and complete consumer behavior.

1. Add `Input.Destination Path` and `SourceInfo.Destination Path` to root types.
   Extend `toEngineInput`, `fromEngineSourceInfo`, `toEngineSourceInfo`, and all
   source publication paths with owned conversion. Run existing public/import
   boundary tests; expose no engine nodes, aliases, or bridge methods.
2. Document zero/root behavior, mapping-only paths, main/underlay parity,
   inherited import placement, source-coordinate fallback, and source ownership
   beside declarations. Update the configuration contract for implemented input
   destinations, preserving canonical ownership of shared semantics.
3. Add external-package public-load coverage for equivalent host file, native
   fs.FS file, byte, reader, and Mapping sources. Cover default/root destinations,
   literal escaped/numeric keys, empty mapping, NoSource, and invalid declarations.
   Include one public mounted main case, not just underlays.
4. Add representative public tests tying final configuration to metadata,
   source placement, history, resources, callbacks, and safe diffs. Mutate returned
   record copies and repeat queries/loads to confirm detachment. Reuse engine
   matrices; public tests demonstrate supported integration rather than every
   private branch. Confirm failed loads expose detached source destinations.

Validate: `go test . -run 'Destination|Facade|PublicFacadeBoundary|DependencyImportBoundary'`,
`go test -race ./...`, and everyday check. Done: the requested workflow is usable
entirely through ConfigKit's supported import path and documented surface.

## Stage 11 — Add maintained simple and collection-style consumer examples

Status: planned. Scope: executable adoption examples with consumer-owned types.

1. Add a minimal mounted-fragment example using `Input.Destination` in an
   ordinary layered load. Show literal keys and root-default compatibility
   without introducing consumer parsing or generic merge helpers.
2. Add one representative collection assembly under `examples/consumers`: peer
   prompt/profile fragments, collection metadata with ordinary `version`, a
   source-root import, local sibling inheritance, and an intentional higher-tier
   version-only override retaining the body. Keep the application VersionRule
   main-owned. Use a controlled native filesystem or temporary files.
3. Demonstrate effective-path inspection plus source-relative resource lookup
   during resolution and from the result, including an inherited/surviving field.
   Application code owns opening/interpreting the resource and any qualified IDs.
4. Add tests for final values, presence, actual source placement/coordinates,
   resource context, and expected conflict behavior. Link maintained examples
   from the consumer guide; add a small executable root example if useful for
   discovery without duplicating the complete scenario.

Validate: `go test ./examples/consumers`, applicable `Example` tests, local links,
and everyday check. Done: simple and representative consumers substantiate the
same configuration model and need no compensating structural pass.

## Stage 12 — Reconcile documentation and audit the complete implementation

Status: planned. Scope: final integration audit and evidence, not release work.

1. Update README navigation, consumer guidance, the component overview, and
   development reading/test maps. Add ADR 0011 to navigation. Update architecture
   policy only as needed to recognize placement within existing assembly and
   source ownership, linking precise behavior to the configuration contract.
   Do not change unrelated policies, deferred boundaries, or published notes.
2. Audit every resolved decision, roadmap behavior, and request acceptance row
   against implementation and tests. Record the corresponding test names here;
   for reused coverage explain the behavior it proves. Do not mark complete
   based on green tests that do not exercise mounting.
3. Review the complete baseline-to-candidate diff for accidental strengthening:
   extra naming rules, namespace uniqueness, main-only restrictions, changed
   merge/version semantics, invented source-read ordering guarantees, new limits,
   or expanded secret policies. Remove temporary helpers/unsupported branches
   left by staged integration. Confirm original source coordinates remain real.
4. Run `./scripts/check.sh` and
   `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`. Review source-kind and
   placement ownership coverage, error disclosure, complete-on-return handling,
   and root/private boundaries. Record commit, Go version, results, and any
   limitations. No release command, CI waiting, benchmark, or downstream upgrade.
5. Mark this plan and the roadmap implemented only when code, public examples,
   and owning contracts agree. Retain completed planning evidence until separately
   authorized release preparation.

Done: one complete layered load supports input destinations with coherent values,
diagnostics, inspection, limits, and source ownership across the approved surface.

## Implementation owner map

Use graph tools to locate these symbols rather than assume line numbers:

- Public input forwarding: `layered.go`, `records.go`, `conversion.go`
  (source conversions), and existing facade tests.
- Private request and lifecycle: `internal/engine/layered.go`, `load_request.go`
  (`loadInput`, `acquireUnderlayGroup`, `assembleLayeredInput`), `load.go`
  (`executeLoadRequest`), and `load_input.go` (`assembleBaseInputTracked`).
- Directives and imports: `directives.go` (`splitImportDirectives`,
  `fragmentVersionClaim`), and `imports.go` (`visitPath`).
- Shared document semantics: `compose.go`, `document_walk.go`
  (`countInputNodes`), `path.go`, `parse.go`, and `value_operation.go` limits.
- Inspection: `metadata.go`, `history.go`, `explain.go`, `resource_context.go`,
  `inheritance_origin.go`, `render.go`, and `diff.go`; source copies live in
  engine `records.go`. Confirm current filenames through the component overview.

## Open Questions

None required for implementation. The four policy choices are approved; this
plan selects API spelling, validation timing/error category, coordinate handling,
and accounting within them. Surface a newly discovered incompatibility or scope
change with evidence rather than resolving it by silently narrowing the feature
or strengthening an unrelated contract.