ConfigKit/docs/roadmap/implementation.md

# Configuration Discovery Implementation Plan

Status: in progress. Stage 1 is complete.

Implement the target behavior in [the discovery roadmap](discovery.md). That
document owns the selected conventions and OD-1 through OD-5; this plan owns the
API decisions, implementation sequence, and validation. Each numbered stage is a
separate Luna-sized unit. Complete stages in order and keep each commit buildable.

## Baseline And Working Rules

- Start from the released v0.9.0 implementation, with the roadmap clarification
  at `92309ddcc3495c88c06d3a4dab414912a83acfae`. Inspect the current worktree and
  preserve unrelated changes. The sequence integration and remediation are done.
- Follow [development guidance](../development.md) and the applicable
  [agent](../policy/agents.md), [architecture](../policy/architecture.md),
  [testing](../policy/testing.md), and [documentation](../policy/documentation.md)
  policies. Use graph tools for code discovery. Read only the relevant owners.
- Keep the public API root-owned, with explicit delegation to `internal/engine`.
  Add cohesive discovery files rather than growing the existing large lifecycle
  functions. Reuse file acquisition, composition, and inspection machinery.
- Discovery chooses one source. Keep strict composition within tiers, precedence
  between tiers, collection rules, inheritance, version policy, CLI ordering,
  secrets, and complete-on-return ownership unchanged after selection.
- The intentional lifecycle extension is shared absent-main behavior for
  `NoSource()` and optional discovery without a match. Do not invent mandatory
  files, new limits, retries, background activity, naming styles, alternate merge
  behavior, or release gates. Do not implement the separate complexity roadmap.
- Add tests at the owning boundary. Update current contracts when their behavior
  lands, rather than leaving contradictory documentation until the last stage.
  Commit each validated, nonempty stage with a plain English message and mark its
  completion here. Do not publish a release or update downstream dependencies.

## Decisions Used Throughout The Stages

### Public API

Use these names and signatures. The two opaque types below have private state;
their actual representation must satisfy the existing facade boundary tests.

```go
type Search struct { /* private state */ }
type FilenameGroup struct { /* private state */ }

type SearchOptions struct {
    Required bool
}

type DirectorySearchOptions struct {
    Filenames []FilenameGroup
}

func NamedFilenames() FilenameGroup
func GenericFilenames() FilenameGroup
func CustomFilenames(names ...string) FilenameGroup

func SearchDirectory(directory string, supplied ...DirectorySearchOptions) Search
func Paths(paths ...string) Search
func Searches(searches ...Search) Search

func UnixSystemDirectories() Search
func UserConfigDirectory() Search
func HomeDirectory() Search
func WorkingDirectory() Search

func SearchConfig(application string, search Search, supplied ...SearchOptions) Source
```

`Searches` is ordered concatenation, not set union. It permits the simple
`SearchConfig("myapp", UnixSystemDirectories())` and a mixed search such as:

```go
source := configkit.SearchConfig("myapp", configkit.Searches(
    configkit.Paths("/run/myapp/config.yml"),
    configkit.WorkingDirectory(),
    configkit.UserConfigDirectory(),
    configkit.UnixSystemDirectories(),
), configkit.SearchOptions{Required: true})
```

This uses the existing convention of zero or one options value without mixing
search-wide controls into the candidate list. More than one options value is a
deferred option error. `SearchConfig` returns the existing `Source`, which works
through ordinary loading and `Input`; do not introduce a main-only source ban.
An underlay input retains its existing tier and contribution-version rules.

For directory searches, omitted options or `Filenames == nil` use Named followed
by Generic. A non-nil empty `Filenames` slice means no names. Groups concatenate
in the supplied order; `CustomFilenames` is one group of literal names. An empty
custom group contributes none. A directory whose expanded filename list is empty
does not resolve its base. Presets retain the roadmap's fixed group selections.

The zero `Search` and `Searches()` describe an empty candidate list; zero
`FilenameGroup` is an empty group. `Paths()` and `CustomFilenames()` are empty.
An explicitly supplied empty path or filename string is invalid, as opposed to
an empty list. The zero `Source` remains invalid.

Snapshot caller-owned slices, including nested filename groups and option
slices, during construction. Descriptions remain immutable and reusable; dynamic
directories and candidate availability are observed again on each consumption.
Constructor failures are retained and reported during load, following opaque
source conventions. Validate all static search arguments before acquisition,
including invalid later components that an earlier match would otherwise hide.

### Literal Components And Platform Behavior

Apply the roadmap's component rules to application slugs and custom filenames.
A needed slug and each literal filename must be nonempty, distinct from `.` and
`..`, and one usable host filename component: no NUL, absolute/volume prefix, or
host path separator. Use standard path helpers; account for Windows reserved
names and invalid filename characters on Windows. Apply platform-specific rules
only on the relevant platform. Preserve case and spelling, including spaces and
non-ASCII names that are valid there. Do not impose a cross-platform ASCII or
kebab-case subset. Do not add speculative length limits; actual filesystem
failures follow normal diagnostics.

Require a slug only if the description uses it in a directory or a Named group.
Explicit paths and Generic-only/custom-only literal directory searches need none.
Do not substitute placeholders in custom names or expand shell syntax in paths.
Resolve relative custom paths against the working directory at consumption.

The Unix system preset is available on Go's `unix` platform set: aix, android,
darwin, dragonfly, freebsd, hurd, illumos, ios, linux, netbsd, openbsd, and solaris.
Use a small private platform predicate, preferably the standard `unix` build
constraint with its complement. Reaching it elsewhere reports an option error;
an earlier selected candidate prevents reaching it. This describes preset
applicability, not an expansion of ConfigKit's platform support or release gates.
Other presets call the three Go directory helpers specified in the roadmap.

### Selection, Diagnostics, And Resource Ownership

Resolve a search to a concrete host `File` or `NoSource` before parsing it.
Visit candidates incrementally in directory-first order. Resolve each reached
directory description once for its filenames; do not eagerly resolve later
bases, list directories, or globally deduplicate/reorder descriptions.

Use the same absolute-path and symlink canonicalization rules as host files.
During candidate probing, only errors matching `fs.ErrNotExist` mean absence.
This includes a missing target reached through ordinary symlink resolution.
Other path-resolution/stat errors terminate selection. An existing target is
selected; its regular-file preflight, open, read, and parse follow existing file
rules. Any failure after selection is terminal, including a file disappearing
before open. Do not decide fallback by inspecting a completed load's diagnostics.

A small stat/canonicalization probe followed by the existing file reader is
sufficient. Read and parse the selected file through the ordinary path once;
do not read it to probe and then reopen it for parsing. Existing preflight and
post-open checks remain. No stronger filesystem snapshot or race guarantee is
introduced. Preserve existing symlink and containment contracts.

Use existing diagnostic codes: `CodeOptionsInvalid` for invalid descriptions and
reached unsupported presets, `CodeSourceMissing` for required exhaustion, and
normal source access/read codes for candidate and directory-helper failures.
Required exhaustion has a search-specific, value-free explanation and no
fabricated source location. Directory-helper failures have no acquired source;
retain normal selected-candidate records for file failures. Do not expose raw
OS errors, environment contents, or YAML values in messages. No new public
attempt trace, source kind, or per-candidate history is needed.

### Absent Input And Filesystem Context

After resolution, selected files use existing host-file context rules: derive
the base from the actual file, reject explicit `BaseDir` or a native `FS`, and
enforce any supplied root. Such errors fail without resuming discovery.

For absence, imports are permitted but have no main declarations to traverse.
Retain a nil main document and no main source record. Version checking receives
the existing absent-main input; required source-version mode does not fabricate
a participating document. Actual underlays, imports, and overlays keep their
normal checks and accounting.

An absent input may have an explicit filesystem context. A host `Root` must be
an absolute existing directory; `BaseDir` may be omitted, in which case use that
root only as internal filesystem context. If provided, validate `BaseDir` using
the existing absolute-directory and containment rules. Complete native contexts
continue to use the existing namespace, `BaseDir`, and `Root` rules. The required
new shorthand is root-only host context, not a new native-context shorthand.
Validate supplied contexts even without imports or an active overlay.

For a search, validate static context syntax before acquisition where it does
not depend on the outcome; apply selected-file versus absent-input compatibility
checks after resolution. Do not reject an otherwise valid absent-input context
merely because selection could have produced a file. An invalid context never
causes fallback.

An ordinary active overlay with absent main requires explicit context. Load it
from that context's normal root-relative overlay location. Never infer context
from attempted directories. Layered overlays still require their independent
`OverlayFilesystem` with its existing full context rules. Internal empty assembly
or context descriptors are acceptable; they must not become an acquired main
document, source ID, or explicit-presence contribution.

## Stage 1 — Define Immutable Search Descriptions And Validation (Complete)

Read [the facade ADR](../adr/0006-public-facade.md), the roadmap, and the source,
options, filesystem, and lifecycle owners named in the component overview.

1. Add engine-owned descriptions and search/group constructors corresponding to
   the chosen API in a focused `internal/engine/discovery.go` file. Leave the
   source-producing `SearchConfig` constructor for Stage 6. Store flat ordered
   search entries rather than a recursively interpreted concatenation tree.
   Snapshot nested lists and retain constructor errors without filesystem access.
2. Implement pure description validation, default/empty filename handling, and
   slug/component validation. Distinguish a zero/empty description from an
   explicitly invalid supplied argument. Ensure concatenation preserves errors
   as well as entries; do not validate irrelevant, unused slug text.
3. Add engine tests for supplied-option arity, snapshots, zero/empty components,
   omitted versus empty filenames, reordered groups, slug-independent searches,
   and platform-appropriate component validation. Do not duplicate standard
   library path tests or introduce a large filename-validation framework.
4. Write ADR 0019, `0019-explicit-configuration-discovery.md`, recording the
   accepted ownership decision: consumers select conventions and order;
   ConfigKit selects one real source within synchronous loading; optional
   exhaustion shares the absent-main lifecycle. Use the repository ADR format.
   Explain the chosen decision and reasoning directly; keep alternatives in
   Alternatives Considered. Leave implementation status in these roadmap files
   and filename inventories/API mechanics in the eventual canonical contract.

Acceptance: descriptions and validation are testable without acquisition; the
public loading behavior is unchanged. Run the new focused engine tests and the
everyday check. Mark and commit the stage.

## Stage 2 — Expand Ordered Candidates And Resolve Preset Bases

1. Implement incremental ordered expansion for literal paths, directory entries,
   groups, and presets. Preserve component positions and filename order exactly.
   Keep requiredness on the whole search. Do not allocate a complete candidate
   product merely to consume its first member.
2. Implement presets exactly as the roadmap tables specify, including all 14
   Unix system candidates and the Named-only `/etc` search. Implement user,
   home, and working-directory bases with the Go helpers. Resolve a base only
   when its nonempty directory search is reached.
3. Provide only a narrow private test seam for directory helpers/platform
   applicability so failure and short-circuit behavior are deterministic. Bind
   normal loads to the standard library; do not expose environment providers or
   mutable process-global function replacements as public API.
4. Test expansion order and reversed groups, preset composition, custom paths,
   empty entries followed by nonempty entries, and unsupported/unavailable bases.
   Assert the contractual preset list at this owner. Use controlled helper
   results for pure preset tests; do not create files under real `/etc` or home.
   Demonstrate that stopping consumption prevents later base resolution and
   that empty filename selections never resolve a base.

Acceptance: deterministic ordered candidates, with no load integration yet.
Work is linear in description size plus candidates visited, excluding OS path
operations and selected-file processing. No benchmark gate is needed. Run focused
discovery tests and the everyday check; mark and commit.

## Stage 3 — Select A Concrete File Through Host Acquisition Rules

1. Add a private resolver taking an immutable search description and returning
   the concrete source or safe diagnostics. Non-search sources pass through.
   Keep the selection state local to a consumption, never cached in `Source`.
2. Reuse or narrowly extract existing host-path canonicalization/probing from
   `internal/engine/source.go`. Delegate selected-file reading to `readSource`
   and `readRegularFile`; do not add a parallel reader or parser.
3. Return `NoSource()` on optional exhaustion and a required-search diagnostic on
   required exhaustion, including a completely empty search. Missing attempts
   create no source evidence and consume no acquisition budget.
4. Test with real temporary files: first match, later match, both extensions,
   directory/nonregular target, contained symlink, required/optional exhaustion,
   and a reached error preventing selection of a later good file. Use the
   existing narrow filesystem failure seams for permission/read failures that
   cannot reliably be reproduced when tests run as a privileged user. Reuse
   existing FIFO and bounded-reader tests rather than duplicating them.
5. Check relative paths and literal `~`, variable, and wildcard characters in
   controlled directories. Verify that a later base is untouched after success.

Acceptance: selection returns one ordinary file or absence, without parsing or
running defaults/resolution. A selected source's later read failure is not a
fallback signal. Run focused discovery/source tests and the everyday check;
mark and commit.

## Stage 4 — Unify Absent Input Acquisition And Context Validation

This stage changes `NoSource()` deliberately, before attaching discovery to loads.

1. In `options.go`, remove the blanket rejection of imports for `sourceNone`.
   In `filesystem.go`, replace blanket context rejection with the absent-context
   rules above, separating static checks from directory access as needed.
2. In `assembleBaseInputTrackedForRole` and its collaborators, handle absent
   acquisition without calling the import traversal that requires a main
   document. Validate any supplied context separately. Preserve nil main state,
   zero main source IDs, zero input accounting, and existing version callbacks.
3. Keep actual underlay and transitive-source imports unchanged. Absence in an
   underlay contributes nothing and does not add an absent-main version callback.
4. Adjust tests that asserted the intentionally retired rejections; add focused
   cases for `NoSource` with imports, valid root-only context, invalid context,
   ordinary defaults/explicit zero and false CLI values, and layered underlays
   whose imports still load. Cover absent-main version acceptance and rejection
   at the existing version-test owner, including enabled source-version modes.
5. Update the `NoSource`/context/import portions of declaration comments and
   `docs/config.md` for the behavior now implemented. Keep the active ordinary
   overlay restriction until Stage 5, so this intermediate commit is accurate.

Acceptance: imports can be enabled on absent input without synthetic sources,
while invalid supplied contexts still fail. Run focused load, filesystem,
version, and grouped tests, then the everyday check; mark and commit.

## Stage 5 — Load Ordinary Overlays With An Absent Main

1. Replace the ordinary `NoSource` active-overlay rejection with the explicit
   context requirement. Resolve absent-input overlay access from its context;
   do not invent a selected main file to satisfy `importFilesystem`.
2. Adapt the smallest shared context/overlay helpers needed in
   `filesystem.go`, `selected_overlay.go`, `load_input.go`, and
   `source_context.go`. Keep layered overlay selection's independent context
   contract unchanged. Reuse the existing overlay reader and accumulator.
3. Preserve source records and contexts for the actual overlay, including its
   physical path, layer, ID, and resource base. With no other acquired sources
   it is `source-1`. Missing main contributes no source or presence. Missing
   selected overlay and invalid/outside-root contexts remain errors.
4. Add ordinary tests for absent-main overlay success, missing explicit context,
   missing overlay, and overlay resource context. Exercise a complete native
   context through existing filesystem machinery and root-only host context.
   Reuse layered overlay tests to protect its independent root behavior.
5. Update owning declarations and contract text for the completed absent-main
   overlay behavior. Review cleanup on success and each new failure path;
   acquired file/root handles remain confined to the load call.

Acceptance: shared absent-main behavior is complete for explicit `NoSource`.
Run focused overlay/context/load tests, `go test -race ./internal/engine`, and
the everyday check. Mark and commit.

## Stage 6 — Integrate Search Resolution Into Engine Loading

1. Add a private discovery source kind/state to engine `Source` and have the
   engine `SearchConfig` constructor capture the validated description. Add
   static search validation to `selectOptions`; layered validation must inspect
   every supplied input before acquisition, as it does for other options.
2. Resolve each discovery source once at the shared input-acquisition boundary,
   before `parseSource`. Carry the concrete selection in a load-local field of
   `assembledInput`, initialized for every source kind. Use it for import access,
   source admission, and contexts within acquisition. Update the ordinary main
   request to that concrete source before overlay application and context
   reconstruction in `executeLoadRequest`.
3. Apply outcome-specific context checks after selection, without rerunning the
   search. Audit remaining kind switches and callers so no downstream access
   helper receives an unresolved discovery descriptor. Keep all existing early
   validation that is independent of selection; do not change schema/version
   callback ordering to implement discovery.
4. Use the same path for layered main and underlay inputs. Preserve per-tier
   repetition checks, main-first acquisition order, placement, and budgets. A
   selected source is `SourceKindFile`, not a new public discovery kind.
5. Add engine lifecycle cases comparing a searched source to its explicit `File`
   counterpart, and exhausted optional search to `NoSource`, with the same
   options. Compare actual paths, destinations, source IDs/layers, presence, and
   context rather than internal descriptor shapes. Assert malformed selected
   YAML and version/decode failures never fall back to a later good file.
6. Cover reuse after file/env/working-directory changes using isolated tests;
   do not run environment-mutating tests in parallel. Verify static invalid
   later components fail before probing, whereas dynamic later base failures
   are skipped after an earlier selection.

Acceptance: engine ordinary and layered loading consume searches through one
lifecycle and preserve real-source evidence. Run focused discovery/load/import/
version/overlay tests, engine race tests, and the everyday check; mark and commit.

## Stage 7 — Expose The Root Facade And Verify Public Integration

1. Add the specified root-owned discovery types, functions, comments, and
   explicit conversions, preferably in root `discovery.go`. Keep engine types
   out of exported fields/signatures and do not add bridge methods or aliases.
   Convert nested lists without losing nil versus non-nil empty filenames.
2. Update `Source` constructor guidance. Add the implemented discovery contract
   to `docs/config.md`: public declarations own symbol details; the contract
   owns ordered selection, failure behavior, absence, context, and metadata.
3. Add root facade tests for the simple preset/custom-directory case, mixed
   ordered searches with requiredness, absent-main parity, slice ownership,
   and ordinary/layered use. Exercise custom Generic-only names with an empty
   slug and explicit empty filenames through the public conversion boundary.
4. Test the important selected/absent context pair with one root-only host
   context: selected file derives its base, absence permits an overlay at the
   supplied root. Test a selected file outside that root fails without fallback.
   Retain errors for selected files with explicit `BaseDir` or native `FS`.
5. Reuse existing public boundary and dependency tests; update an explicit symbol
   inventory only if needed to recognize the deliberately added API.

Acceptance: consumers can use the feature entirely through the root package.
Run focused facade discovery/context tests, public/dependency boundary tests,
and the everyday check. Mark and commit.

## Stage 8 — Complete Cross-Feature Coverage And Consumer Examples

1. Add a small maintained consumer example/test for a simple explicitly selected
   preset with test-controlled directories, and another for custom-first search
   used with underlays/imports/overlay/CLI. Keep illustrative snippets short and
   link to the maintained examples. No example should depend on real host config.
2. Review the integration coverage against the roadmap. Add only missing risk
   cases: selected relative imports/resources; placement and sequence-item
   placement; absence creating no item; actual collection/inheritance semantics;
   defaults and callbacks running once; invalid chosen content preventing later
   fallback; final metadata/rendering agreeing with the selected source.
   Reuse existing structural, secret, removed-history, and lifetime tests where
   discovery introduces no new path through them.
3. Review all new branches for independently owned descriptions/results and
   cleanup. Use a narrow controlled acquisition failure test if an introduced
   failure branch's closure behavior is not already covered. Do not turn this
   into a cross-platform handle-counting framework or a new support gate.

Acceptance: representative consumers need no application-side fallback loader
or provenance reconstruction. Run maintained consumer tests, relevant executable
examples, and the everyday check. Mark and commit.

## Stage 9 — Align Architecture And Contributor Documentation

1. Update the architecture policy's responsibility boundary narrowly: applications
   choose main-file conventions; explicit library-provided searches execute
   inside loading. Preserve the exclusion of implicit discovery and network
   acquisition. Link ADR 0019 and the canonical discovery contract.
2. Update the internal overview for discovery ownership and resolved-source
   propagation, the development guide's focused test map/ADR list, and consumer
   guidance. Add only concise README navigation or an example if useful; retain
   the current minimal explicit-source example. Explain ordered concatenation
   and the directory-first Cartesian product where applicable in the contract.
3. Review the declaration comments and discovery contract added in earlier
   stages against the completed feature. Keep one canonical owner per fact and
   fix stale claims about `NoSource`, implicit discovery, or overlay context.

Acceptance: documentation describes implemented behavior and links the API,
contract, ADR, and maintained examples. Run the documentation link test and
whitespace review. Execute any newly changed behavior-bearing examples; reuse
Stage 8 results for unchanged examples. Mark and commit.

## Stage 10 — Review The Complete Feature And Validate The Candidate

1. Compare the full implementation with the roadmap and this plan. Check every
   OD, the API/default/empty distinctions, preset lists, lazy failures, context
   outcome checks, source admission, callback execution, and complete-on-return
   ownership. Ensure explicit `File` behavior and all existing precedence rules
   remain intact.
2. Look specifically for accidental contract expansion: implicit presets,
   global canonical deduplication, application-name normalization, extra file
   restrictions, suppressed access errors, no-match version bypass, fallback
   after selected-content errors, or new human/CI validation requirements.
3. Run `go test -race ./...` and `./scripts/check.sh` under the installed compatible
   toolchain with dependencies already available, following the development
   guide. Reuse successful checks for unchanged content; investigate failures
   rather than weakening assertions. No timing/benchmark threshold, CI wait,
   native macOS run, or release publication is part of this implementation.
4. Review status and staged diff, mark this plan complete only after the
   acceptance criteria are satisfied, and commit any final nonempty unit.
   Report validation and remaining limitations. Keep pending roadmap documents;
   completed-plan retirement belongs to later release preparation.

## Open Questions

None. OD-1 through OD-5 are settled in the feature roadmap; this plan fixes the
remaining API and integration details without changing those decisions.