ConfigKit/docs/roadmap/implementation.md

Configuration Discovery Implementation Plan

Status: in progress. Stages 1, 2, 3, and 4 are complete.

Implement the target behavior in the discovery roadmap. That document owns the selected conventions and OD-1 through OD-5; this plan owns the API decisions, implementation sequence, and validation. Each numbered stage is a separate Luna-sized unit. Complete stages in order and keep each commit buildable.

Baseline And Working Rules

• Start from the released v0.9.0 implementation, with the roadmap clarification at 92309ddcc3495c88c06d3a4dab414912a83acfae. Inspect the current worktree and preserve unrelated changes. The sequence integration and remediation are done.
• Follow development guidance and the applicable agent, architecture, testing, and documentation policies. Use graph tools for code discovery. Read only the relevant owners.
• Keep the public API root-owned, with explicit delegation to internal/engine. Add cohesive discovery files rather than growing the existing large lifecycle functions. Reuse file acquisition, composition, and inspection machinery.
• Discovery chooses one source. Keep strict composition within tiers, precedence between tiers, collection rules, inheritance, version policy, CLI ordering, secrets, and complete-on-return ownership unchanged after selection.
• The intentional lifecycle extension is shared absent-main behavior for NoSource() and optional discovery without a match. Do not invent mandatory files, new limits, retries, background activity, naming styles, alternate merge behavior, or release gates. Do not implement the separate complexity roadmap.
• Add tests at the owning boundary. Update current contracts when their behavior lands, rather than leaving contradictory documentation until the last stage. Commit each validated, nonempty stage with a plain English message and mark its completion here. Do not publish a release or update downstream dependencies.

Decisions Used Throughout The Stages

Public API

Use these names and signatures. The two opaque types below have private state; their actual representation must satisfy the existing facade boundary tests.

type Search struct { /* private state */ }
type FilenameGroup struct { /* private state */ }

type SearchOptions struct {
    Required bool
}

type DirectorySearchOptions struct {
    Filenames []FilenameGroup
}

func NamedFilenames() FilenameGroup
func GenericFilenames() FilenameGroup
func CustomFilenames(names ...string) FilenameGroup

func SearchDirectory(directory string, supplied ...DirectorySearchOptions) Search
func Paths(paths ...string) Search
func Searches(searches ...Search) Search

func UnixSystemDirectories() Search
func UserConfigDirectory() Search
func HomeDirectory() Search
func WorkingDirectory() Search

func SearchConfig(application string, search Search, supplied ...SearchOptions) Source

Searches is ordered concatenation, not set union. It permits the simple SearchConfig("myapp", UnixSystemDirectories()) and a mixed search such as:

source := configkit.SearchConfig("myapp", configkit.Searches(
    configkit.Paths("/run/myapp/config.yml"),
    configkit.WorkingDirectory(),
    configkit.UserConfigDirectory(),
    configkit.UnixSystemDirectories(),
), configkit.SearchOptions{Required: true})

This uses the existing convention of zero or one options value without mixing search-wide controls into the candidate list. More than one options value is a deferred option error. SearchConfig returns the existing Source, which works through ordinary loading and Input; do not introduce a main-only source ban. An underlay input retains its existing tier and contribution-version rules.

For directory searches, omitted options or Filenames == nil use Named followed by Generic. A non-nil empty Filenames slice means no names. Groups concatenate in the supplied order; CustomFilenames is one group of literal names. An empty custom group contributes none. A directory whose expanded filename list is empty does not resolve its base. Presets retain the roadmap's fixed group selections.

The zero Search and Searches() describe an empty candidate list; zero FilenameGroup is an empty group. Paths() and CustomFilenames() are empty. An explicitly supplied empty path or filename string is invalid, as opposed to an empty list. The zero Source remains invalid.

Snapshot caller-owned slices, including nested filename groups and option slices, during construction. Descriptions remain immutable and reusable; dynamic directories and candidate availability are observed again on each consumption. Constructor failures are retained and reported during load, following opaque source conventions. Validate all static search arguments before acquisition, including invalid later components that an earlier match would otherwise hide.

Literal Components And Platform Behavior

Apply the roadmap's component rules to application slugs and custom filenames. A needed slug and each literal filename must be nonempty, distinct from . and .., and one usable host filename component: no NUL, absolute/volume prefix, or host path separator. Use standard path helpers; account for Windows reserved names and invalid filename characters on Windows. Apply platform-specific rules only on the relevant platform. Preserve case and spelling, including spaces and non-ASCII names that are valid there. Do not impose a cross-platform ASCII or kebab-case subset. Do not add speculative length limits; actual filesystem failures follow normal diagnostics.

Require a slug only if the description uses it in a directory or a Named group. Explicit paths and Generic-only/custom-only literal directory searches need none. Do not substitute placeholders in custom names or expand shell syntax in paths. Resolve relative custom paths against the working directory at consumption.

The Unix system preset is available on Go's unix platform set: aix, android, darwin, dragonfly, freebsd, hurd, illumos, ios, linux, netbsd, openbsd, and solaris. Use a small private platform predicate, preferably the standard unix build constraint with its complement. Reaching it elsewhere reports an option error; an earlier selected candidate prevents reaching it. This describes preset applicability, not an expansion of ConfigKit's platform support or release gates. Other presets call the three Go directory helpers specified in the roadmap.

Selection, Diagnostics, And Resource Ownership

Resolve a search to a concrete host File or NoSource before parsing it. Visit candidates incrementally in directory-first order. Resolve each reached directory description once for its filenames; do not eagerly resolve later bases, list directories, or globally deduplicate/reorder descriptions.

Use the same absolute-path and symlink canonicalization rules as host files. During candidate probing, only errors matching fs.ErrNotExist mean absence. This includes a missing target reached through ordinary symlink resolution. Other path-resolution/stat errors terminate selection. An existing target is selected; its regular-file preflight, open, read, and parse follow existing file rules. Any failure after selection is terminal, including a file disappearing before open. Do not decide fallback by inspecting a completed load's diagnostics.

A small stat/canonicalization probe followed by the existing file reader is sufficient. Read and parse the selected file through the ordinary path once; do not read it to probe and then reopen it for parsing. Existing preflight and post-open checks remain. No stronger filesystem snapshot or race guarantee is introduced. Preserve existing symlink and containment contracts.

Use existing diagnostic codes: CodeOptionsInvalid for invalid descriptions and reached unsupported presets, CodeSourceMissing for required exhaustion, and normal source access/read codes for candidate and directory-helper failures. Required exhaustion has a search-specific, value-free explanation and no fabricated source location. Directory-helper failures have no acquired source; retain normal selected-candidate records for file failures. Do not expose raw OS errors, environment contents, or YAML values in messages. No new public attempt trace, source kind, or per-candidate history is needed.

Absent Input And Filesystem Context

After resolution, selected files use existing host-file context rules: derive the base from the actual file, reject explicit BaseDir or a native FS, and enforce any supplied root. Such errors fail without resuming discovery.

For absence, imports are permitted but have no main declarations to traverse. Retain a nil main document and no main source record. Version checking receives the existing absent-main input; required source-version mode does not fabricate a participating document. Actual underlays, imports, and overlays keep their normal checks and accounting.

An absent input may have an explicit filesystem context. A host Root must be an absolute existing directory; BaseDir may be omitted, in which case use that root only as internal filesystem context. If provided, validate BaseDir using the existing absolute-directory and containment rules. Complete native contexts continue to use the existing namespace, BaseDir, and Root rules. The required new shorthand is root-only host context, not a new native-context shorthand. Validate supplied contexts even without imports or an active overlay.

For a search, validate static context syntax before acquisition where it does not depend on the outcome; apply selected-file versus absent-input compatibility checks after resolution. Do not reject an otherwise valid absent-input context merely because selection could have produced a file. An invalid context never causes fallback.

An ordinary active overlay with absent main requires explicit context. Load it from that context's normal root-relative overlay location. Never infer context from attempted directories. Layered overlays still require their independent OverlayFilesystem with its existing full context rules. Internal empty assembly or context descriptors are acceptable; they must not become an acquired main document, source ID, or explicit-presence contribution.

Stage 1 — Define Immutable Search Descriptions And Validation (Complete)

Read the facade ADR, the roadmap, and the source, options, filesystem, and lifecycle owners named in the component overview.

1. Add engine-owned descriptions and search/group constructors corresponding to the chosen API in a focused internal/engine/discovery.go file. Leave the source-producing SearchConfig constructor for Stage 6. Store flat ordered search entries rather than a recursively interpreted concatenation tree. Snapshot nested lists and retain constructor errors without filesystem access.
2. Implement pure description validation, default/empty filename handling, and slug/component validation. Distinguish a zero/empty description from an explicitly invalid supplied argument. Ensure concatenation preserves errors as well as entries; do not validate irrelevant, unused slug text.
3. Add engine tests for supplied-option arity, snapshots, zero/empty components, omitted versus empty filenames, reordered groups, slug-independent searches, and platform-appropriate component validation. Do not duplicate standard library path tests or introduce a large filename-validation framework.
4. Write ADR 0019, 0019-explicit-configuration-discovery.md, recording the accepted ownership decision: consumers select conventions and order; ConfigKit selects one real source within synchronous loading; optional exhaustion shares the absent-main lifecycle. Use the repository ADR format. Explain the chosen decision and reasoning directly; keep alternatives in Alternatives Considered. Leave implementation status in these roadmap files and filename inventories/API mechanics in the eventual canonical contract.

Acceptance: descriptions and validation are testable without acquisition; the public loading behavior is unchanged. Run the new focused engine tests and the everyday check. Mark and commit the stage.

Stage 2 — Expand Ordered Candidates And Resolve Preset Bases (Complete)

1. Implement incremental ordered expansion for literal paths, directory entries, groups, and presets. Preserve component positions and filename order exactly. Keep requiredness on the whole search. Do not allocate a complete candidate product merely to consume its first member.
2. Implement presets exactly as the roadmap tables specify, including all 14 Unix system candidates and the Named-only /etc search. Implement user, home, and working-directory bases with the Go helpers. Resolve a base only when its nonempty directory search is reached.
3. Provide only a narrow private test seam for directory helpers/platform applicability so failure and short-circuit behavior are deterministic. Bind normal loads to the standard library; do not expose environment providers or mutable process-global function replacements as public API.
4. Test expansion order and reversed groups, preset composition, custom paths, empty entries followed by nonempty entries, and unsupported/unavailable bases. Assert the contractual preset list at this owner. Use controlled helper results for pure preset tests; do not create files under real /etc or home. Demonstrate that stopping consumption prevents later base resolution and that empty filename selections never resolve a base.

Acceptance: deterministic ordered candidates, with no load integration yet. Work is linear in description size plus candidates visited, excluding OS path operations and selected-file processing. No benchmark gate is needed. Run focused discovery tests and the everyday check; mark and commit.

Stage 3 — Select A Concrete File Through Host Acquisition Rules (Complete)

1. Add a private resolver taking an immutable search description and returning the concrete source or safe diagnostics. Non-search sources pass through. Keep the selection state local to a consumption, never cached in Source.
2. Reuse or narrowly extract existing host-path canonicalization/probing from internal/engine/source.go. Delegate selected-file reading to readSource and readRegularFile; do not add a parallel reader or parser.
3. Return NoSource() on optional exhaustion and a required-search diagnostic on required exhaustion, including a completely empty search. Missing attempts create no source evidence and consume no acquisition budget.
4. Test with real temporary files: first match, later match, both extensions, directory/nonregular target, contained symlink, required/optional exhaustion, and a reached error preventing selection of a later good file. Use the existing narrow filesystem failure seams for permission/read failures that cannot reliably be reproduced when tests run as a privileged user. Reuse existing FIFO and bounded-reader tests rather than duplicating them.
5. Check relative paths and literal ~, variable, and wildcard characters in controlled directories. Verify that a later base is untouched after success.

Acceptance: selection returns one ordinary file or absence, without parsing or running defaults/resolution. A selected source's later read failure is not a fallback signal. Run focused discovery/source tests and the everyday check; mark and commit.

Stage 4 — Unify Absent Input Acquisition And Context Validation (Complete)

This stage changes NoSource() deliberately, before attaching discovery to loads.

1. In options.go, remove the blanket rejection of imports for sourceNone. In filesystem.go, replace blanket context rejection with the absent-context rules above, separating static checks from directory access as needed.
2. In assembleBaseInputTrackedForRole and its collaborators, handle absent acquisition without calling the import traversal that requires a main document. Validate any supplied context separately. Preserve nil main state, zero main source IDs, zero input accounting, and existing version callbacks.
3. Keep actual underlay and transitive-source imports unchanged. Absence in an underlay contributes nothing and does not add an absent-main version callback.
4. Adjust tests that asserted the intentionally retired rejections; add focused cases for NoSource with imports, valid root-only context, invalid context, ordinary defaults/explicit zero and false CLI values, and layered underlays whose imports still load. Cover absent-main version acceptance and rejection at the existing version-test owner, including enabled source-version modes.
5. Update the NoSource/context/import portions of declaration comments and docs/config.md for the behavior now implemented. Keep the active ordinary overlay restriction until Stage 5, so this intermediate commit is accurate.

Acceptance: imports can be enabled on absent input without synthetic sources, while invalid supplied contexts still fail. Run focused load, filesystem, version, and grouped tests, then the everyday check; mark and commit.

Stage 5 — Load Ordinary Overlays With An Absent Main

1. Replace the ordinary NoSource active-overlay rejection with the explicit context requirement. Resolve absent-input overlay access from its context; do not invent a selected main file to satisfy importFilesystem.
2. Adapt the smallest shared context/overlay helpers needed in filesystem.go, selected_overlay.go, load_input.go, and source_context.go. Keep layered overlay selection's independent context contract unchanged. Reuse the existing overlay reader and accumulator.
3. Preserve source records and contexts for the actual overlay, including its physical path, layer, ID, and resource base. With no other acquired sources it is source-1. Missing main contributes no source or presence. Missing selected overlay and invalid/outside-root contexts remain errors.
4. Add ordinary tests for absent-main overlay success, missing explicit context, missing overlay, and overlay resource context. Exercise a complete native context through existing filesystem machinery and root-only host context. Reuse layered overlay tests to protect its independent root behavior.
5. Update owning declarations and contract text for the completed absent-main overlay behavior. Review cleanup on success and each new failure path; acquired file/root handles remain confined to the load call.

Acceptance: shared absent-main behavior is complete for explicit NoSource. Run focused overlay/context/load tests, go test -race ./internal/engine, and the everyday check. Mark and commit.

Stage 6 — Integrate Search Resolution Into Engine Loading

1. Add a private discovery source kind/state to engine Source and have the engine SearchConfig constructor capture the validated description. Add static search validation to selectOptions; layered validation must inspect every supplied input before acquisition, as it does for other options.
2. Resolve each discovery source once at the shared input-acquisition boundary, before parseSource. Carry the concrete selection in a load-local field of assembledInput, initialized for every source kind. Use it for import access, source admission, and contexts within acquisition. Update the ordinary main request to that concrete source before overlay application and context reconstruction in executeLoadRequest.
3. Apply outcome-specific context checks after selection, without rerunning the search. Audit remaining kind switches and callers so no downstream access helper receives an unresolved discovery descriptor. Keep all existing early validation that is independent of selection; do not change schema/version callback ordering to implement discovery.
4. Use the same path for layered main and underlay inputs. Preserve per-tier repetition checks, main-first acquisition order, placement, and budgets. A selected source is SourceKindFile, not a new public discovery kind.
5. Add engine lifecycle cases comparing a searched source to its explicit File counterpart, and exhausted optional search to NoSource, with the same options. Compare actual paths, destinations, source IDs/layers, presence, and context rather than internal descriptor shapes. Assert malformed selected YAML and version/decode failures never fall back to a later good file.
6. Cover reuse after file/env/working-directory changes using isolated tests; do not run environment-mutating tests in parallel. Verify static invalid later components fail before probing, whereas dynamic later base failures are skipped after an earlier selection.

Acceptance: engine ordinary and layered loading consume searches through one lifecycle and preserve real-source evidence. Run focused discovery/load/import/ version/overlay tests, engine race tests, and the everyday check; mark and commit.

Stage 7 — Expose The Root Facade And Verify Public Integration

1. Add the specified root-owned discovery types, functions, comments, and explicit conversions, preferably in root discovery.go. Keep engine types out of exported fields/signatures and do not add bridge methods or aliases. Convert nested lists without losing nil versus non-nil empty filenames.
2. Update Source constructor guidance. Add the implemented discovery contract to docs/config.md: public declarations own symbol details; the contract owns ordered selection, failure behavior, absence, context, and metadata.
3. Add root facade tests for the simple preset/custom-directory case, mixed ordered searches with requiredness, absent-main parity, slice ownership, and ordinary/layered use. Exercise custom Generic-only names with an empty slug and explicit empty filenames through the public conversion boundary.
4. Test the important selected/absent context pair with one root-only host context: selected file derives its base, absence permits an overlay at the supplied root. Test a selected file outside that root fails without fallback. Retain errors for selected files with explicit BaseDir or native FS.
5. Reuse existing public boundary and dependency tests; update an explicit symbol inventory only if needed to recognize the deliberately added API.

Acceptance: consumers can use the feature entirely through the root package. Run focused facade discovery/context tests, public/dependency boundary tests, and the everyday check. Mark and commit.

Stage 8 — Complete Cross-Feature Coverage And Consumer Examples

1. Add a small maintained consumer example/test for a simple explicitly selected preset with test-controlled directories, and another for custom-first search used with underlays/imports/overlay/CLI. Keep illustrative snippets short and link to the maintained examples. No example should depend on real host config.
2. Review the integration coverage against the roadmap. Add only missing risk cases: selected relative imports/resources; placement and sequence-item placement; absence creating no item; actual collection/inheritance semantics; defaults and callbacks running once; invalid chosen content preventing later fallback; final metadata/rendering agreeing with the selected source. Reuse existing structural, secret, removed-history, and lifetime tests where discovery introduces no new path through them.
3. Review all new branches for independently owned descriptions/results and cleanup. Use a narrow controlled acquisition failure test if an introduced failure branch's closure behavior is not already covered. Do not turn this into a cross-platform handle-counting framework or a new support gate.

Acceptance: representative consumers need no application-side fallback loader or provenance reconstruction. Run maintained consumer tests, relevant executable examples, and the everyday check. Mark and commit.

Stage 9 — Align Architecture And Contributor Documentation

1. Update the architecture policy's responsibility boundary narrowly: applications choose main-file conventions; explicit library-provided searches execute inside loading. Preserve the exclusion of implicit discovery and network acquisition. Link ADR 0019 and the canonical discovery contract.
2. Update the internal overview for discovery ownership and resolved-source propagation, the development guide's focused test map/ADR list, and consumer guidance. Add only concise README navigation or an example if useful; retain the current minimal explicit-source example. Explain ordered concatenation and the directory-first Cartesian product where applicable in the contract.
3. Review the declaration comments and discovery contract added in earlier stages against the completed feature. Keep one canonical owner per fact and fix stale claims about NoSource, implicit discovery, or overlay context.

Acceptance: documentation describes implemented behavior and links the API, contract, ADR, and maintained examples. Run the documentation link test and whitespace review. Execute any newly changed behavior-bearing examples; reuse Stage 8 results for unchanged examples. Mark and commit.

Stage 10 — Review The Complete Feature And Validate The Candidate

1. Compare the full implementation with the roadmap and this plan. Check every OD, the API/default/empty distinctions, preset lists, lazy failures, context outcome checks, source admission, callback execution, and complete-on-return ownership. Ensure explicit File behavior and all existing precedence rules remain intact.
2. Look specifically for accidental contract expansion: implicit presets, global canonical deduplication, application-name normalization, extra file restrictions, suppressed access errors, no-match version bypass, fallback after selected-content errors, or new human/CI validation requirements.
3. Run go test -race ./... and ./scripts/check.sh under the installed compatible toolchain with dependencies already available, following the development guide. Reuse successful checks for unchanged content; investigate failures rather than weakening assertions. No timing/benchmark threshold, CI wait, native macOS run, or release publication is part of this implementation.
4. Review status and staged diff, mark this plan complete only after the acceptance criteria are satisfied, and commit any final nonempty unit. Report validation and remaining limitations. Keep pending roadmap documents; completed-plan retirement belongs to later release preparation.

Open Questions

None. OD-1 through OD-5 are settled in the feature roadmap; this plan fixes the remaining API and integration details without changing those decisions.