ConfigKit/docs/roadmap/discovery.md

# Composable Configuration Discovery Roadmap

Status: proposal with the agreed direction captured below. Preset contents and
the remaining behavioral choices are under [Open Decisions](#open-decisions).
This document describes future work, not an implemented contract. A staged
implementation plan will follow once the decisions are settled.

## Purpose And Ownership

Make ordinary file-based loading concise for applications that want conventional
configuration locations. ConfigKit supplies useful built-in candidate lists and
an ordered selection operation. Consumers continue to own main-file search
conventions: they choose which lists apply, their order, custom candidates, and
whether a configuration file is required.

A consumer can select one preset for a simple application or compose several
presets and custom path lists. The application supplies an explicit slug where
needed to construct conventional paths. ConfigKit does not infer application
identity from the executable name or enable discovery for existing explicit
sources.

Discovery selects one source for the existing loading lifecycle. It does not
create additional configuration tiers or merge every matching file. Existing
layered loading remains available when consumers intentionally combine sources.

## Current Baseline

The [source contract](../config.md#sources) provides explicit host files, native
filesystem files, bytes, readers, mappings, and `NoSource()`. A missing explicit
file fails. The zero source is invalid. There is no conventional main-file search.

[Ordinary loading](../config.md#single-source-loading) infers a selected overlay's
filesystem context from the main source. `NoSource()` currently rejects enabled
imports, active overlays, and filesystem context. In
[layered loading](../config.md#layered-loading), an absent main input can coexist
with underlays; a selected overlay requires an explicit `OverlayFilesystem`.
These distinctions matter when an optional search finds nothing.

The [architecture policy](../policy/architecture.md#responsibility-boundaries)
assigns search conventions to consumers and currently excludes implicit main-file
discovery. The addition should clarify that consumers may explicitly select
library-provided search conventions. Consumer ownership remains unchanged.

## Agreed Direction

### Ordered Candidate Groups

Provide reusable descriptions of built-in search lists and custom ordered paths.
Consumers select one or more groups in the desired order. Conceptually, concatenate
the groups into one ordered candidate sequence, preserving order within each
group. This is ordered concatenation followed by first-match selection, not a
set union or a precedence merge.

Names below are illustrative and are not implemented Go API:

```go
// A simple system-wide application.
source := configkit.SearchConfig("myapp", configkit.UnixSystemDirectories())
```

```go
// A consumer-selected working-directory, user, then system search.
source := configkit.SearchConfig(
    "myapp",
    configkit.WorkingDirectory(),
    configkit.UserConfigDirectory(),
    configkit.UnixSystemDirectories(),
)
```

```go
// Explicit custom candidates followed by a selected conventional fallback.
source := configkit.SearchConfig(
    "myapp",
    configkit.Paths("/run/myapp/config.yml", "/opt/myapp/config.yaml"),
    configkit.UnixSystemDirectories(),
)
```

Use the resulting source with ordinary loading or as a layered main input. Custom
lists participate at their specified position; they do not silently append
unselected built-in defaults. Reordering groups changes search priority.

### Selection And Failure Behavior

| Situation | Selected behavior |
| --- | --- |
| Candidate is absent | Continue to the next candidate. |
| A candidate file is selected | Stop searching and load that file. |
| Candidate is inaccessible, unreadable, or not an accepted file type | Fail using normal source diagnostics. |
| Selected file fails parsing, version checks, composition, decoding, or validation | Fail without trying another candidate. |
| Entire search has no match | Absence is permitted by default; continue through applicable defaults and normal validation. The interaction with other inputs is addressed under OD-3. |
| Entire search has no match and is required | Fail with a diagnostic explaining the unsuccessful search. |
| Caller explicitly uses `File(path)` | Retain existing exclusive-file behavior, including errors for a missing file. |

Requiredness applies to the complete search, not to each group. Permitting an
absent file does not waive application version, schema, or domain requirements;
the remaining lifecycle may still reject an incomplete configuration.

Perform discovery before parsing the chosen main file. Do not implement fallback
by repeatedly running complete loads. Defaults, application resolution, secret
lookup, and final validation retain their existing lifecycle execution rules.
Once a file has been selected, later failure does not restart discovery.

### Construction And Filesystem Behavior

Constructors describe the search without filesystem access. Resolve dynamic base
directories and access candidate files during loading, so reusable descriptions
can observe changes on subsequent loads. Do not change the process working
directory or environment.

Reuse the host-file acquisition rules for selected files, including regular-file
checks, symlink handling, resource ownership, and source limits. The feature does
not require a new filesystem traversal engine, recursive directory search, or
new file-size limits. Candidate filenames and groups are bounded by their
explicit descriptions, rather than by scanning unrelated directories.

The exact Go names, constructors, and representation of search-wide options
belong in the implementation plan. Keep the public surface root-owned and keep
the single-preset case concise.

### Placement, Provenance, And Existing Features

The selected file participates as the actual file source. Preserve its physical
identity, effective placement, source origin, and passive resource context.
Relative imports and relative application resources use the selected file's
normal context. Discovery does not relocate overlay or CLI coordinates.

Missing candidates are not acquired sources and do not create source IDs,
configuration history, or explicit input presence. An entirely absent search
contributes no synthetic empty configuration file. Diagnostics should distinguish
an exhausted required search from a failed candidate. Existing selected-source
inspection remains the foundation; a new public per-attempt trace is not needed
for this proposal.

Preserve collection policies, inheritance, source-version rules, CLI ordering,
final validation, and frozen inspection after selection. Search-list order does
not change precedence among underlays, the main input, an overlay, and CLI values.
An explicitly selected overlay is not another main-file fallback candidate.

## Work Needed From The Baseline

The subsequent implementation plan should cover:

- Define candidate-group and search-source representations, slug validation, and
  a search-wide requiredness option. Keep construction separate from acquisition.
- Implement the selected preset expansions, ordered custom paths, and first-match
  selection through existing file access. Resolve each source once per consumption
  before entering its parse and composition lifecycle.
- Integrate selected and absent outcomes with ordinary and layered main inputs,
  placement, imports, application version checking, and overlay contexts according
  to OD-3. Preserve ownership and close acquired handles on success and failure.
- Preserve real selected-source metadata and safe diagnostics without treating
  absent candidates as configuration contributions.
- Exercise candidate and group ordering, custom-list composition, both filename
  extensions, no-match behavior, required searches, unavailable preset bases,
  errors that must stop fallback, repeated loads, and untouched explicit `File`
  behavior. Cover imports, overlays, underlays, defaults, CLI, source versions,
  source contexts, and source-list/inspection evidence at their boundaries.
- Clarify the architecture policy's consumer-selected discovery boundary. Record
  any warranted durable ownership decision in an ADR; keep preset filenames and
  API mechanics in their canonical contract when implemented. Update consumer
  examples and the configuration contract when the feature lands.

This roadmap does not change implementation, policy, or ADRs by itself. Coordinate
implementation with the ongoing sequence integration audit rather than assuming
its in-progress changes are part of a fixed implementation baseline.

## Open Decisions

### OD-1 — Preset Contents And Filename Order

**Recommendation:** begin with three explicitly selected presets. Try `.yml`
before `.yaml` at each location, completing both alternatives there before moving
to the next location:

| Preset | Candidate locations, in order |
| --- | --- |
| Unix system | `/usr/local/etc/<slug>/config.yml`, `/usr/local/etc/<slug>/config.yaml`, `/etc/<slug>/config.yml`, `/etc/<slug>/config.yaml`, `/etc/<slug>.yml`, `/etc/<slug>.yaml` |
| User configuration | `<user-config-dir>/<slug>/config.yml`, then `config.yaml` in the same directory |
| Working directory | `./<slug>.yml`, then `./<slug>.yaml` |

For the user preset, use Go's platform-specific
[`os.UserConfigDir`](https://pkg.go.dev/os#UserConfigDir) as the base. This supplies
a conventional user location without inventing separate home-directory rules.
The system list retains the originally proposed location order. Application-named
working-directory files make the selected application's convention explicit.
No preset walks ancestor directories.

**Alternative:** use `./config.yml` and `./config.yaml` for the working-directory
preset, or add them after the application-named files. Generic names are concise
for project-local applications, while application-named files distinguish
applications sharing a working directory. Adding both is convenient but expands
the candidate list and makes a generic file a fallback even when the consumer
intended an application-specific convention.

An additional viable user convention is a home-directory dotfile or directory,
such as `~/.<slug>.yml` or `~/.<slug>/config.yml`. It is familiar for some tools,
but would add a second user convention and ordering to explain. Prefer the
user-config-directory preset initially; custom paths can express other layouts.

### OD-2 — Unavailable Bases And Platform Applicability

**Recommendation:** resolve group bases as the ordered search reaches them.
If a selected group's base cannot be resolved, report an error. Treat using the
Unix system preset on a platform where it is unsupported as an explicit
configuration error when that group is reached. Specify the supported platforms
in the implementation plan. Do not silently translate the Unix preset into a
different platform's convention. User and working-directory presets use their
platform-native bases.

An unavailable base is different from a candidate file being absent. Failure to
determine the user's configuration directory should not silently redirect loading
to a different group. Once an earlier candidate is selected, later groups need
not resolve their bases. Static invalid search arguments still fail ordinary
option validation before source acquisition.

**Alternative:** treat unsupported presets or unavailable bases as empty groups
and continue. This makes one preset composition usable across more environments,
but can unexpectedly select a lower-priority configuration. A selectable
skip-unavailable behavior could serve that need later; it adds a distinct policy
from the search-wide requirement that at least one file exist.

### OD-3 — No Match, Imports, And Selected Overlays

**Recommendation:** use a shared absent-main lifecycle for optional no-match and
explicit `NoSource()`. Allow import enablement when the main file is absent;
there are no main-file declarations to traverse. Other actual sources retain
their normal import behavior. In layered loads, retain underlays and the existing
explicit overlay-filesystem requirement.

For ordinary loads with an active overlay and no main file, require an explicit
filesystem context. Do not infer an overlay root from the first or last searched
directory. A missing selected overlay remains an error. Defaults, CLI overrides,
resolution, and validation continue normally, and main-version checking receives
absence rather than a fabricated document.

This would deliberately relax the current `NoSource()` restrictions on imports
and on overlays with explicit context. It keeps one meaning for absent main input
and lets optional configuration remain optional without silently ignoring a
requested overlay. It requires lifecycle work beyond candidate selection.

**Alternative:** map no-match to the current `NoSource()` contract and retain its
restrictions. This minimizes lifecycle changes, but an otherwise valid optional
search with imports enabled can fail solely because no file was found, and
ordinary overlay-only loading would remain unavailable. A special absent-search
case could avoid changing `NoSource()`, at the cost of two different absence rules.

### OD-4 — Empty Composition And Implicit Preset Selection

**Recommendation:** require explicit group selection to activate a convention.
An empty group composition has no candidates: optional search contributes no
source, and required search fails as exhausted. Empty custom lists follow the
same rule. This supports programmatic list construction and introduces no hidden
system, user, or working-directory fallback. One explicitly selected preset
still gives the simple application a short call.

**Alternative:** let omission of groups select a documented default preset.
This shortens the simplest call, but omission and an intentionally empty computed
list would need distinguishable representations. It also requires choosing a
default convention across platforms and application styles. Another alternative
is to reject an empty composition as invalid; that catches accidental empty lists
but adds a special case beyond the search-wide requiredness rule.

### OD-5 — Slugs And Custom Path Interpretation

**Recommendation:** require a nonempty single-component slug when a selected
preset uses it to construct filenames. Reject path separators, `.` and `..`,
and other forms that cannot be used as that component on the target platform.
Preserve spelling; do not silently lowercase, trim, or impose a kebab-case naming
scheme. A composition made only of explicit custom paths need not require an
application slug.

Treat custom paths as literal host file paths. Resolve relative paths against
the working directory at load time; do not expand shell variables, `~`, or glob
syntax. Consumers can construct expanded paths themselves. Selected presets
provide the convenience for user and working-directory bases.

**Alternative:** require a restricted portable slug grammar, such as lowercase
ASCII letters, digits, and hyphens. This is easy to document and portable, but
unnecessarily excludes otherwise usable application names. Automatic shell-like
custom-path expansion is also viable, but adds escaping and environment semantics
that ordinary `File(path)` does not supply. Literal paths keep both interfaces
consistent.