ConfigKit/docs/roadmap/implementation.md

# Implement shared source version checks

Status: Stage 1 complete; runtime implementation has not started.

Audience: a Luna coding agent implementing one numbered stage per prompt, in
order. The [feature roadmap](source-version-checks.md) owns target behavior and
accepted policy; this plan owns concrete implementation decisions and execution.
Planning baseline: `d9f08ff7198ad5a8fb5c49a43e7ccdad2ad0e707`.

The completed relative-import plan and evidence replaced here remain at
`d9f08ff:docs/roadmap/implementation.md`. Its roadmap and remediation are
complete, not instructions for this sprint.

## Working instructions

- Read `AGENTS.md`, [development.md](../development.md), and applicable policies.
  This work involves [architecture](../policy/architecture.md),
  [testing](../policy/testing.md), and [documentation](../policy/documentation.md).
  Use graph-first discovery, verify live sources, and preserve unrelated work.
- Implement one stage at a time. Record completion, actual commands, toolchain,
  results, limitations, and implementation revisions here. Commit each validated
  nonempty unit with a short plain-English message.
- Keep public types root-owned with explicit engine conversions. Reuse semantic
  Paths, documents, composition, overlay, version validation, sensitivity, and
  inspection. No second loader, public parser nodes, generic transformation
  hooks, or consumer-specific policy.
- Default mode preserves current behavior, including effective-path lookup,
  main checker timing, and protected-path failures. Enabled modes deliberately
  select source-local declarations and non-main consumption. Do not broaden
  these changes into unrelated default-mode fixes.
- Do not add equality/SemVer rules, non-null requirements, numeric coercion,
  path-form restrictions, source uniqueness policies, limits, dependencies,
  toolchain changes, or release gates. Declaration acceptance is not fragment
  schema conformance. Exclusions describe this delivery, not new permanent bans.
- Internal stages may test modes before exposing them publicly. Keep Go/current
  documentation accurate to each delivered boundary; the roadmap owns the target.
- Use focused offline tests while iterating and `./scripts/check.sh` for each
  production-code unit. Documentation-only units need local-link and whitespace
  checks. Reuse existing tests; avoid a full mode/source/tier/path cross-product,
  helper-call assertions, allocation identity, or exact message snapshots.
  No independent benchmarks or performance thresholds are required.
- Do not rewrite accepted ADR decisions, publish, select a release version,
  upgrade consumers, or retire completed roadmap records. Public struct additions
  and enabled-mode path interpretation need minor-release compatibility guidance
  during separately authorized release preparation.

## Concrete implementation decisions

### Public options and evidence

1. Add root-owned `type SourceVersionMode uint8` with consecutive constants
   `SourceVersionModeDefault` (zero), `SourceVersionModeIfPresent`, and
   `SourceVersionModeRequired`. Add `SourceVersionMode SourceVersionMode`
   to `Options[T]` and `LayeredOptions[T]`, not Definition, Input, or YAML.
   Mirror privately and convert explicitly. Unknown modes, or enabled modes
   without a usable VersionRule and checker, produce `CodeOptionsInvalid`
   before source access or callbacks. Keep existing rule/path validation.
2. Add root-owned `SourceVersionCheck` with `Path Path`, `Version Version`,
   `Location *Location`, and `Redacted bool`. Add
   `VersionCheck *SourceVersionCheck` to SourceInfo. Nil means no enabled
   declaration processing occurred. A non-nil record with Present false records
   observed absence, including optional omissions whose checker was skipped; it
   does not claim a successful callback or acceptance. On a malformed selected
   value or overlap failure, retain only known presence/null, path, and location
   with conservative redaction; never expose arbitrary scalar/container content.
   Read/parse failures before this phase leave the record nil.
   Reuse Version's absent/null/text model.
   Redaction preserves presence/null, empties classified Text, and sets Redacted.
3. Evidence Path is source-local, with semantic segment intent resolved against
   original source shape where possible. Present declarations retain their actual
   authored location; absence uses the nearest available source location, not an
   invented header coordinate. Destination and source identity already belong
   to SourceInfo. Add no new query API, raw-node field, expected-version value,
   or string/integer discriminator.
4. Clone the record, Path, and Location at all query/conversion/error boundaries.
   Callback-time source queries and failed-load source records expose conservative
   text-redacted evidence until final classification is available. Successful
   final inspection may expose unclassified text; previous copies stay detached.
   Keep raw evidence separate from public-safe source records. Do not extend a
   failed load to finish classification or introduce a failed-load evidence API.

### Lookup, checking, and diagnostics

5. Carry the configured local rule and an effective main-owned rule separately.
   Enabled modes derive the latter once by prefixing main's Destination using
   semantic segments. Use it consistently for remaining-payload protection,
   inheritance/CLI checks, decoding's exact-path integer exception, and application
   metadata. Default mode uses the configured effective path unchanged. Never
   mutate caller rules or repeatedly prefix a path.
6. Read each declaration from its original source document before mounting or
   consumption. Keep contextual numeric-key/index resolution, empty/escaped keys,
   parsed tokens, and explicit indices. No mapping-only/root-only restriction;
   selecting a root mapping simply fails ordinary version-type validation.
7. Missing members, unavailable indices, and obstructing non-containers mean the
   requested declaration is absent, as with current document lookup. Distinguish
   causes privately for accurate nearest locations, not to invent rejection of
   an ancestor's domain value. If-present non-main absence skips the checker and
   records absence; required absence is `CodeVersionError` before callback.
   A present unsupported selected value is `CodeVersionType`, not absence.
   Actual main absence in if-present mode still reaches its checker once;
   required mode rejects it first. NoSource is not a document and preserves the
   existing single absent-main callback in both enabled modes.
8. Reuse strict decimal recognition and exact Version.Text. No machine-integer
   conversion or comparison with main. Required null reaches the checker and can
   be accepted. Warnings alone succeed and accepted non-main headers are consumed.
9. Enabled declaration checkers and their Validation paths use the current
   source's local coordinates for every role, including mounted main. Resolve
   primary/related locations against that original source. Generated header
   diagnostics also use source-local paths; do not rebase them to phantom payload
   fields. Ordinary payload diagnostics remain effective-path diagnostics.
   Default version diagnostics are unchanged. Reuse `CodeVersionError` for
   missing-required/structural-overlap failures, `CodeVersionType` for malformed
   values, and existing checker error/warning/hook codes. No new code is needed.
10. Keep callback lifetime, safe generic returned-error handling, and synchronous
    execution. Check main once from its own declaration, never a composed value.
    Check other documents as acquired, before composition can hide rejection.
    Do not promise a new inter-source callback order or exhaustive acquisition
    after failure. Build schema successfully before callbacks; enabled mode may
    prepare it before acquisition, without changing the default pipeline.
    Defaults, decoding, resolution, secret lookup, and validation still run once
    on the assembled application.

### Consumption and structural boundaries

11. Validate source-local controls before consumption can erase malformed
    directives. Reuse import/overlay validation and inheritance-control
    recognition, not per-fragment inheritance expansion. Reject actual overlap
    with configkit directive structure or an extends control, including headers
    inside import declarations or invalid control subtrees. Unrelated siblings
    and ordinary keys containing those spellings remain allowed. Missing headers
    follow their mode; they do not suppress independently invalid directives.
    Add no error-order guarantee.
12. Retain main's node. Remove exactly an accepted non-main mapping member or
    sequence element in an owned payload tree. Preserve authored ancestors,
    newly empty mappings/sequences, sibling order, and physical node locations.
    Sequence deletion shifts later payload indices; evidence keeps original
    coordinates. Removing a member inside a sequence element keeps that element,
    even when empty. No null placeholders, ancestor pruning, or Source mutation.
13. Extract source-root imports from original content; consume headers in its
    application body before placement/composition. Underlay roots and all imports
    are non-main; selected overlays are also non-main. Check remaining placed
    payload against effective main ownership. Consumption does not permit
    forbidden overlay controls, sequence alignment, or CLI ownership bypass.
14. Charge original acquired bytes/nodes once, including consumed headers. Apply
    existing placed-body bounds and full-destination wrapper charges to remaining
    payload. Keep source/depth/glob accounting, records, resource contexts, and
    physical repetition unchanged. Acquired header-only mappings retain their
    empty body/ancestors and placement; NoSource/optional absent sources add no
    document or evidence. No new cap or full-source clone is required.

### Evidence and sensitivity

15. Carry private evidence per acquired SourceID in per-load acquisition/assembly
    state, separate from application history. Retain an owned scalar/absence,
    original local and mounted semantic paths, location, and the ancestor-shape
    context needed for classification. Capture key/index intent before deletion.
    Retain only needed context, not entire source-tree clones or loading handles.
16. At publication classify evidence at its original mounted path and ancestors
    with existing schema tags, predicate, historical projected sensitivity,
    inheritance, and recorded-dependency propagation. Include evidence-only
    candidates in classification without adding them to Metadata paths, payload
    applications, rendering, or Diff. Do not infer original sequence shape from
    shifted payload or round-trip through pointer strings. Reuse dependency
    closure rather than building a weaker independent redactor. Retained header
    text never triggers credential lookup.
17. Sanitize source records before public conversion; early records remain
    conservatively redacted. Final successful evidence uses completed sensitivity,
    retaining historical classification despite replacement. Main's normal
    payload/history remains intact. Consumed non-main headers are only source
    evidence: equal payloads with different headers must not acquire effective
    differences merely from that evidence.

## Implementation owner map

| Responsibility | Existing owners to inspect and extend |
| --- | --- |
| Public options, callbacks, records, conversions | `lifecycle.go`, `layered.go`, `records.go`, `conversion.go`, `diagnostic.go` |
| Options and per-load coordination | `internal/engine/options.go`, `load.go`, `load_request.go`, `layered.go` |
| Acquisition, directives, placement, overlay | `internal/engine/load_input.go`, `imports.go`, `directives.go`, `destination.go`, `selected_overlay.go` |
| Checking and semantic paths | `internal/engine/validation.go`, `document_walk.go`, `version_path_test.go` |
| Evidence copies, errors, publication | `internal/engine/records.go`, `diagnostic.go`, `load_result.go` |
| Classification and propagation | `internal/engine/secret.go`, schema secret-path helpers, `secret_propagation_test.go`, `render_test.go` |
| Compatibility tests | Engine `load_test.go`, `imports_test.go`, `selected_overlay_test.go`, `load_request_test.go`, `destination_*_test.go`; root `facade_version_test.go`, `facade_layered_test.go`, `facade_destination_test.go`, `records_test.go` |

Private helper names/file splits may follow the implementation while preserving
these decisions. A new internal package solely for headers is unnecessary.

## Validation commands

Use the applicable subset during each stage; add new tests under the
SourceVersion name family so focused checks include them. Existing test owners
remain authoritative. Commands run from the repository root with dependencies
already available; these are development checks, not new release gates.

```sh
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SourceVersion|Version|Import|Underlay|Overlay|Destination'
GOTOOLCHAIN=local GOPROXY=off go test ./internal/engine -run 'SourceVersion|Secret|Sensitivity|History|Inheritance|Resolution'
GOTOOLCHAIN=local GOPROXY=off go test . -run 'SourceVersion|Version|Layered|SourceInfo|Destination|PublicFacadeBoundary|DependencyImportBoundary'
GOTOOLCHAIN=local GOPROXY=off go test ./examples/consumers
GOTOOLCHAIN=local GOPROXY=off go test . -run '^TestLocalDocumentationLinks$'
GOTOOLCHAIN=local ./scripts/check.sh
GOTOOLCHAIN=local GOPROXY=off go test -race ./...
git diff --check
```

## Stage 1 — Capture the baseline and record the extension

1. Confirm the worktree/baseline; read the roadmap, plan, applicable policies,
   contract, and relevant ADRs. Run the everyday check and record revision,
   toolchain, results, and environmental limitations.
2. Write Accepted ADR `0013-shared-source-version-checks.md` after checking the
   number is unused. Record one policy, modes, source-local checking, consumption,
   evidence, and retained main protection. Focus on ConfigKit's selected choice;
   rejected choices belong only in Alternatives Considered. Keep sequencing here.
3. Preserve prior accepted decision text. Add status/navigation links where
   useful. Note struct additions and enabled mounted-main interpretation as
   minor-release considerations; do not select a version or write release notes.

Validate: local-link test and whitespace. Done: baseline and accepted decision
recorded, with no runtime changes.

Completion evidence: reviewed baseline `d9f08ff7198ad5a8fb5c49a43e7ccdad2ad0e707`
and the clean implementation worktree at `70c95bf660794809e58182d736c67c9085238705`;
confirmed ADR 0013 was unused. The everyday check passed with Go
`go1.26.8 linux/amd64` using
`GOTOOLCHAIN=local GOPROXY=off ./scripts/check.sh`. Documentation links and
whitespace are validated for this unit. No environmental limitation affected
these checks; validation was performed on Linux/amd64 only. No runtime changes
were made.

## Stage 2 — Share version interpretation without changing defaults

1. Separate reusable value interpretation and callback reporting from checkVersion,
   retaining its default entry point, absence/null/text, exact integers, warnings,
   safe errors, primary/related locations, and context closure.
2. Reuse existing scalar/path/hook/lifetime tests; add only missing protection for
   the refactor. Introduce a small private result separating checked value and
   diagnostics. Failure decisions must use severity, not slice length.
3. Do not globally rewrite unrelated diagnostic plumbing or public behavior.

Validate: engine version/validation and facade version tests; everyday check.
Done: shared mechanisms exist with unchanged default behavior.

## Stage 3 — Implement source-local extraction and exact consumption

1. Add a private processor receiving original document, local rule, source role,
   destination, and mode. Implement decisions 6–12, including missing/obstructed
   lookup and pre-consumption structural overlap.
2. Return owned remaining payload, original import declarations, and private
   evidence with original segment/ancestor context. No second parser, fragment
   decoding, or mutation of reusable input.
3. Add focused tests for vocabulary, missing/obstructed locations, malformed
   selected nodes, literal/parsed numeric keys and indices, empty ancestors,
   sequence deletion, and actual versus unrelated structural overlap.

Validate: helper/version-path/directive/inheritance-control tests and everyday
check. Done: the processor is tested independently of public loading.

## Stage 4 — Add private mode state and main-document integration

1. Add the engine mode to ordinary/layered options and loadRequest. Validate it at
   both entry points before source access. Underlay helper options must not be
   treated as standalone loads missing the shared rule.
2. Carry local/effective rules, source roles, warnings, and evidence in per-load
   state. Let acquired inputs return evidence and nonfatal diagnostics.
3. In enabled mode, check actual main before imports/composition, retain its node,
   and prepare schema before callbacks. Skip the old later main check only in
   enabled mode. Preserve NoSource's absent callback and default timing.
4. Thread effective protection through typed conversion, inheritance, CLI, and
   application inspection while keeping declaration callbacks local. Test mounted
   main, absence/NoSource, null acceptance, invalid options before reading, and
   exactly one main callback.

Validate: load/grouped/version/destination/hook tests and everyday check. Done:
private enabled main behavior works without changing old callers.

## Stage 5 — Integrate recursive imports, warnings, and accounting

1. Thread shared state through import entry/traversal. Check each acquired child
   once against its own local header, then place and compose remaining payload.
   Keep nested/glob effective destinations and original directive discovery.
2. Apply effective main protection afterward. Preserve physical identity,
   cycle/repetition, optional absence, source records, resource contexts, and raw
   node/byte charges. Header-only acquired mappings still contribute empty bodies.
3. Propagate warnings to load diagnostics without aborting traversal. Update only
   relevant callers currently treating any diagnostic as failure.
4. Cover nested/glob imports, mounted relative destinations, accepted unequal
   declarations, rejected hidden declarations, required omissions, safe locations,
   optional missing sources, raw accounting, and default import.version failures.
   Reuse existing containment/glob/source-kind tests.

Validate: import/glob/destination/accounting/version tests and everyday check.
Done: imports use one policy without lost warnings or changed physical traversal.

## Stage 6 — Integrate direct underlays and their strict groups

1. Thread state through acquireUnderlayGroup/grouped assembly. Direct underlays
   are non-main; imports reuse Stage 5. Preserve layers, indices, contexts,
   evidence, and warnings across groups.
2. Consume local headers before strict composition, then protect remaining placed
   payload. Preserve tier precedence, empty containers, raw budgets, and reusable
   sources in allowed separate tiers.
3. Cover accepted distinct peer declarations, a rejected underlay hidden by main,
   required absence, empty/NoSource groups, mounted nested headers, retained
   payload conflicts, and unchanged default underlay.version failures.

Validate: grouped/underlay/destination/version tests and everyday check. Done:
direct and imported underlays follow the same checking policy.

## Stage 7 — Integrate selected overlays and final ownership checks

1. Process overlay headers after parse/structural validation, before remaining
   payload protection and application. Reuse shared state in ordinary and layered
   overlay loading; preserve source identity, namespaces, close behavior, and raw
   node/byte charges.
2. Carry accepted warnings through assembly. Keep overlay-control restrictions,
   and effective main protection through CLI and inheritance.
3. Cover accepted/required/rejected overlay headers, warnings, unchanged default
   behavior, ordinary CLI fields without header requirements, mounted-main
   protection, and protected payload after nested/sequence consumption. Sequence
   evidence stays original while remaining indices shift.

Validate: overlay/override/inheritance/version tests and everyday check. Done:
all document roles are integrated privately; synthetic operations are exempt
from header checks, not from existing protection.

## Stage 8 — Expose the public load-wide mode

1. Add root mode/constants/option fields and both conversions from decision 1.
   Preserve zero-value behavior and root-owned boundaries.
2. Add external-package tests for ordinary/layered enabled loads, zero-mode
   compatibility, invalid mode/rule before reading, and reuse of one Definition
   with different modes. Use representative source-kind coverage.
3. Document implemented mode behavior in Go declarations, including local versus
   effective interpretation. Evidence remains private until Stage 11; do not
   claim it is publicly available yet.

Validate: facade version/layered/source and boundary tests; everyday check.
Done: both supported public load APIs expose the implemented modes.

## Stage 9 — Classify evidence at original paths and ancestors

1. Include declaration paths/ancestors in final classification candidates,
   independently of effective/historical Metadata indexes. Do not add consumed
   nodes to applications or explanation replay.
2. Apply schema-tag and predicate rules using original mounted semantic context,
   including tags beneath dynamic containers and ancestor classification.
   Preserve numeric-key/index intent before sequence shifts; consumed paths need
   not survive final decoding. Reuse existing redaction mechanisms.
3. Test private evidence with disposable canaries: unclassified text, direct/
   ancestor tags and predicates, original sequence shape versus shifted payload,
   and null/absence. Keep raw evidence private throughout this stage.

Validate: source-version/sensitivity/rendering tests and everyday check. Done:
direct/ancestor classification is correct without making headers payload history.

## Stage 10 — Preserve historical and dependency sensitivity

1. Connect evidence candidates to existing historical, inheritance, and recorded
   dependency propagation. Later replacement or consumption must not erase
   applicable classification; do not invent new propagation policy.
2. Consider evidence-only paths even when absent from final metadata. Keep
   Resolution.Record's existing schema-path validity; do not invent dependencies
   on headers solely to create tests.
3. Cover displaced historical classification and legitimate recorded dependencies,
   with inheritance where applicable. Ensure neighboring unclassified evidence
   remains visible and no synthetic effective/history/diff paths are introduced.

Validate: source-version/history/inheritance/resolution/sensitivity tests,
`go test -race ./internal/engine`, and everyday check. Done: established
sensitivity closure applies without polluting payload history.

## Stage 11 — Publish detached sanitized source evidence

1. Add private/public SourceVersionCheck and SourceInfo.VersionCheck records.
   Extend both conversions and all clone paths, including callback source queries,
   successful inspection, and existing LoadError source access.
2. Before final classification, expose conservative sanitized records; after
   successful classification, freeze final sanitized evidence. Keep local paths,
   physical locations, absence/null, and unchecked distinction. Raw evidence
   stays in private state; no loading handle is retained.
3. Add facade/copy tests mutating nested paths/locations and records. Repeated
   queries and independent loads remain unaffected. Delete sources after load
   and verify evidence/resource metadata without reopening files.
4. Cover early acquisition and later resolver/validator failures plus callback
   source queries with canaries. Default and synthetic CLI records stay nil.

Validate: record/conversion/source/version/ownership and public boundary tests,
root/engine race checks, and everyday check. Done: existing public source
inspection surfaces safely expose detached evidence.

## Stage 12 — Audit lifecycle agreement and compatibility

1. Map tests to every roadmap participation row/edge decision; add only missing
   cross-boundary regressions with one primary owner per behavior.
2. Verify final values, zero/false/null presence, defaults, exact-path integer
   conversion, resolution, secret lookup, and validation after consumption.
   Hooks see one assembled configuration. Accepted declarations need not match.
3. Compare equal payloads with differing consumed headers: evidence changes but
   effective rendering/history/Diff gain no consumed fields. Main history stays
   normal. Cover empty ancestors and ordinary versions outside the header path.
4. Check physical contexts after placement/inheritance/precedence, existing
   accounting boundaries, all supported document source kinds, and default
   import/underlay/overlay/CLI/version compatibility.
5. Run `./scripts/check.sh` and
   `GOTOOLCHAIN=local GOPROXY=off go test -race ./...`. Record actual evidence;
   do not claim untested platforms or consumer adoption. Consumer checks are not
   new release gates.

Done: all target behavior has public-boundary evidence and deterministic checks
pass, without extra contract restrictions or redundant coverage.

## Stage 13 — Add maintained adoption examples

1. Add a small executable example showing if-present and required adoption with
   one Definition.Version policy, accepted distinct declarations, main/import/
   underlay or overlay sources, and safe evidence inspection.
2. Include a mounted input illustrating local lookup and effective main
   protection. Keep ordinary release metadata at another path; no consumer-side
   second load/merge, collection checker, or implicit schema migration.
3. Keep copyable assets in examples with tests for final payload and safe output.
   Reuse existing examples for remaining source plumbing; do not print canaries
   or promise callback ordering.

Validate: example/consumer tests, links, and everyday check. Done: tested examples
demonstrate the coherent shared policy with reduced consumer plumbing.

## Stage 14 — Reconcile documentation and finish the audit

1. Update Go docs and docs/config.md as canonical owners for modes, participation,
   local diagnostics, consumption, effective protection, mounted main, evidence
   states/disclosure, and default behavior. Explain conservative callback-time
   redaction and declared compatibility versus final validation. Link from
   consumer guidance, development, and internal overview.
2. Update the architecture ownership summary where needed, linking to contract
   and ADR rather than duplicating mechanics. Preserve previous accepted ADR
   decisions. Reconcile obsolete design/consumption notes in the source request.
   Repair completed relative-import roadmap/remediation links that meant the old
   overwritten plan, using the Git-history pointer above.
3. Audit for accidental default checks, equality/non-null rules, path grammar,
   fragment decoding, ordering promises, extra uniqueness/caps, unsafe evidence
   queries, and new release gates. Match every roadmap outcome to test evidence.
4. Mark plan/roadmap complete only after all stages pass. Run everyday checks,
   full race suite after code corrections, and final link/whitespace checks.
   Commit the validated unit; leave a clean workspace. Do not publish or retire
   completed records in this stage.

Done: implementation, coverage, documentation, and approved scope agree. No
product questions remain open under the roadmap and concrete decisions above.