configkit/docs/roadmap/main-relative-inputs.md

# Main-relative companion imports and overlays

Status: accepted direction and public API structure; absent-main and remaining
filesystem-policy choices are open. Not implemented. Implementation baseline:
`191015e`.

## Purpose

Make the common workflow straightforward: discover the main configuration file,
load optional companion fragments beside it, and apply a selected native overlay
in one configuration load. Applications choose their conventions; operators can
drop resource definitions into the declared locations without adding import
boilerplate to the main file or creating a manifest on disk.

The central addition is deferred filesystem-context binding for application-owned
underlay inputs, including `Mapping` and `Bytes` sources containing native import
declarations. Bind their imports to the selected main directory during loading.
Reuse existing optional-file, directory, recursion, pattern, destination, and
composition behavior. A new file constructor alone would not cover this workflow,
and a second fragment-discovery implementation is unnecessary.

## Current baseline

The [discovery contract](../config.md#configuration-discovery) resolves
`SearchConfig` inside loading to one host file or absent input. Selected-file
failures do not resume discovery. The constructor returns a source description,
not a selected filename.

Ordinary `Load` already infers overlay context from a file source. Without a
root override, `/srv/campaign/audita.yml` supplies the location
`/srv/campaign/overlays/production.yml` for overlay `production`. Existing overlay
lookup uses the configuration root, which can differ from the main directory.
Optional discovery exhaustion provides no inferred context.

The [layered API](../config.md#layered-loading) has the coordination gap:

- An active overlay requires `LayeredOptions.OverlayFilesystem`. Preflight
  rejects its absence before discovery runs.
- Underlay sources and contexts are supplied independently. Application-owned
  mappings or bytes can already contain import declarations, but cannot obtain
  their import context from the main file that discovery will select.
- Native imports already support optional named files and recursive directory
  searches with filename filtering and placement. They compose as peers of
  their declaring input, so putting declarations in an underlay gives them
  underlay priority. Putting the same declarations in the main file does not.
- Main acquisition already precedes underlay acquisition. Existing composition
  subsequently applies underlay groups from lowest to highest beneath the main.

This feature extends context selection and binding within that lifecycle.

## Audita workflow

Audita supplies the companion import declarations in application-owned mappings
or bytes. Their filesystem bases follow the selected main file, whether selected
explicitly or through ConfigKit's built-in searches. For a main file at
`/srv/campaign/audita.yml`, running from `/tmp` must not relocate these companions:

| Resource | Application-declared locations | Role |
| --- | --- | --- |
| Glossaries | Optional `glossary.yml` and recursive `glossary/` | Underlay contributions mounted at `glossary_collections` using the appropriate existing destination and placement controls |
| Prompt definitions | Optional `prompts.yml` and recursive `prompts/` | Operator contributions above the bundled PromptKit catalog and Audita's embedded prompts |
| Selected native overlay | `overlays/<name>.yml` | The one named overlay applied after main composition |

Both resource-directory searches select `*.yml` and `*.yaml`. The file and
its corresponding directory contribute together if both exist; neither suppresses
the other. Each pair composes strictly as peers within its tier. Missing optional
resources contribute nothing; unreadable selected files, invalid definitions,
and composition conflicts remain errors.

Prompt overrides rely on existing tier and identity rules. The operator prompt
tier sits above embedded prompt tiers, allowing whole-record upsert by registered
prompt ID under the applicable collection policy. Two operator fragments defining
the same ID in that tier conflict; filename ordering does not choose a winner.
The consuming application selects each fragment's schema and placement.

This does not require combining Audita's and PromptKit's schemas. If PromptKit
uses a separate engine, its catalog load establishes the prompt tiers and can
use the selected main's recorded context after Audita's application load. That
is a separate domain load, not a preliminary load to repeat main discovery.
This feature's acceptance does not require changing another repository.

The overlay directory is not imported wholesale as an underlay: its files are
alternative named configurations. No selection means no overlay. Selecting a
name retains the existing missing-selected-overlay error. An aggregate
`overlays.yml` would need its own named-definition and conflict rules and is
outside this feature; it is not a second spelling for current overlay lookup.

## Targeted end state

### Companion underlays

The application declares an underlay `Input` containing native imports and
selects main-derived filesystem binding for that input. `Mapping` and `Bytes`
are first-class supported sources for this use. OD-1 settles the public API
structure below; no new operator-facing YAML syntax is needed.

Once the main is selected, ConfigKit supplies the underlay import base from
that file's directory and binds the namespace and containment policy. Existing
import selection then acquires the companion files. Every imported file uses
its own directory for its further relative imports. For example, an import of
`extra.yml` from `glossary/terms.yml` resolves to `glossary/extra.yml`.

Applications choose underlay groups and their order. A group composes strictly;
separate groups establish precedence. Existing destinations, sequence placement,
source-version policies, and load-wide import defaults still apply. Binding a
context does not itself enable imports or change collection operations.

The relationship is explicit on companion inputs. Independently located and
embedded inputs keep their own contexts and can coexist in the same load.
Providing a concrete context selects that complete context over derived
binding, without field-by-field fallback or recovery from an invalid explicit
context. OD-1 records the representation and precedence rules.

There is no requirement to add a dedicated main-relative file constructor.
Optional named-file acquisition is already expressible through the underlay's
native imports. Ordinary `File` paths do not silently gain new relative semantics.

### Layered overlays

For an active layered overlay, use an explicit `OverlayFilesystem` when supplied.
Otherwise derive the overlay context automatically from the available main-file
information; the common workflow requires no `OverlayFromMain` flag. An invalid
explicit context fails rather than falling back. Inactive overlays need no
context. Missing-main and fallback cases are addressed below and in OD-2.

For this newly derived layered context, the lookup location is the main file's
directory, independently of a wider containment root. If the main is
`/srv/campaign/audita.yml` and its permitted root is `/srv`, the selected file
is `/srv/campaign/overlays/production.yml`, not `/srv/overlays/production.yml`.
Widening containment must not relocate the automatically selected companions.

Keep existing explicit-overlay-context lookup behavior and ordinary `Load`
behavior. The implementation must represent the derived lookup base separately
from its containment root where necessary, rather than simply copying today's
root-relative overlay context. Native overlay names, filenames, selection,
precedence, and validation retain their existing semantics.

### Absent main and optional companions

Optional discovery may find no main file while defaults and CLI settings still
permit a successful application load. Support a declared absent-main policy for
dependent companion imports. Skipping optional companions is the intended default
for this workflow; applications can instead require a base or supply an explicit
fallback directory/context. The representation and exact skip scope remain in
OD-2, including inputs with required imports or additional application payload.

Do not infer a directory from unsuccessful candidates or silently use the working
directory. An application may explicitly choose the working directory as its
fallback. An active selected overlay still needs a usable context and the selected
file; optional companion imports do not make that overlay optional. Independent
underlays and explicitly located overlays retain their absent-main behavior.
No fallback applies to a failure reading or processing an actually selected main.

### Lifecycle, evidence, and compatibility

Bind dependent contexts after the main information is available and before
traversing their imports or loading the overlay. Preserve main-first acquisition,
underlay precedence, version processing, selected-overlay timing, and the existing
defaults, CLI, resolution, secret, and validation phases. No application context
callback, second discovery API, or preliminary full load is required.

Check static declarations during preflight where possible; defer checks requiring
the selected directory. Do not introduce a global ordering guarantee for unrelated
diagnostics or filesystem checks. Selected-file and companion acquisition errors
remain terminal rather than restarting discovery.

Application-supplied mappings or bytes remain ordinary input sources with their
own labels, evidence, and existing validation obligations. Binding a context does
not manufacture a main document, exempt these inputs from source-version policy,
or give them a fabricated physical file path. Imported files retain their actual
source identity, locations, precedence layer, underlay index, and passive resource
context. Preserve namespace identity, repetition scope, shared budgets, detached
inspection, and acquisition-handle cleanup.

Successful existing ordinary and explicitly configured layered loads should retain
their behavior. Automatically deriving a layered overlay context intentionally
relaxes the former missing-context error. Companion directory conventions are
selected by applications; ConfigKit does not universally scan `glossary/` or
`prompts/` without those declarations.

This work adds no general resource-path resolver, remote acquisition, mutation of
the process working directory, filesystem race-protection guarantee, or general
source-provider callback framework. These are scope boundaries, not new permanent
architectural exclusions. The design follows
[ADR 0019](../adr/0019-explicit-configuration-discovery.md). Review ADR and policy
follow-up for deferred context ownership without rewriting accepted decisions.
No release version or additional release gate is selected here.

## Work from the baseline

1. **Finalize context declarations and absence policy.** Resolve the remaining
   choices below and prepare Go examples for Audita's mapping/bytes workflow,
   explicit context overrides, and independently located embedded inputs.
2. **Carry binding intent through the public facade and load request.** Update
   the relevant input/options declarations and conversions in `layered.go` and
   `lifecycle.go`, retaining root-owned public types and normal ownership rules.
   Distinguish a deferred relationship from a concrete filesystem context.
3. **Bind underlay import contexts after main selection.** Extend preparation
   and acquisition in `internal/engine/load_request.go`, `load_input.go`, and
   `acquisition.go` as needed. Reuse the acquired main's directory and namespace,
   apply the chosen missing-base policy, and delegate selection to existing import
   machinery. Preserve independently supplied contexts.
4. **Derive overlay context and separate lookup from containment.** Remove the
   unconditional explicit-context requirement for the new layered default.
   Preserve explicit and ordinary lookup while enabling a main-directory lookup
   base inside a potentially wider root. Reuse selected-overlay loading and its
   failure and resource-ownership behavior.
5. **Verify the complete workflow and document it.** Add maintained consumer
   examples, public integration coverage, and focused engine tests. Update the
   discovery, layered-loading, import-context, and selected-overlay contracts,
   declaration comments, and internal ownership documentation when implemented.

This is dependency order for a later implementation plan, not a required commit
layout or a requirement to expose intermediate stages to consumers.

## Validation and completion

Validate behavior through the public loading boundary, including:

- Explicit and discovered main files, early and later search candidates, and an
  unrelated invocation directory. Companions follow the same selected main.
- Mapping and bytes underlays containing optional named-file and recursive
  directory imports with `*.yml`/`*.yaml` filtering and mounted destinations.
- File only, directory only, both, neither, empty directories, and invalid or
  unreadable selected fragments. Both forms compose as peers when present.
- Operator prompt overrides above embedded catalogs using registered identity;
  same-tier duplicate IDs remain conflicts. Main and overlay precedence survive.
- Mixed embedded and companion inputs, multiple underlay tiers, transitive
  imports, placement, source-version checks, and load-wide import defaults.
- Automatic derived overlays and explicit overrides, active and inactive names,
  broader containment roots, and missing selected overlays.
- Optional discovery exhaustion, required discovery, fallback and skip policies,
  mixed payload/required-import cases from OD-2, and terminal selected-main errors.
- Symlinks, eligible namespaces, parent-relative paths within the selected scope,
  context validity, and namespace identity under OD-3 and OD-4.
- Normal source evidence, diagnostics, passive resource contexts, cleanup, and
  budget accounting; no duplicate lifecycle callbacks or preliminary full load.
- Existing ordinary and explicit layered workflows and description reuse.

Run focused checks during implementation and applicable normal repository checks
before completion. No benchmark campaign is required. Completion means the
application can discover its main, bind companion imports and the selected
overlay, and receive normal resolved configuration and inspection in one load,
without an application directory walker, parser, or extra discovery pass.

## Settled Decisions

### OD-1: Public representation of deferred context binding — settled

Add a small, optional main-relative binding description to `Input`, retaining
the concrete `Input.Filesystem` field. A helper supplies the common default
description; explicit settings group any absent-main controls decided in OD-2.
Companion `Mapping` and `Bytes` inputs use native imports for fragment selection.
Preserve the input's source, placement, import enablement, and tier.

The proposed shape is an `Input.MainRelative` field of type
`*MainRelativeContext`, with a `MainRelative()` convenience helper. These names
illustrate the settled structure; final spelling and OD-2's settings can be
finished during API design. For example, this unimplemented API sketch assumes
`companionDeclarations` contains ordinary native import declarations:

```go
configkit.Input{
    Source:       configkit.Mapping("companions", companionDeclarations),
    Imports:      true,
    MainRelative: configkit.MainRelative(),
}
```

A nil binding preserves existing independent context behavior. A non-nil binding
requests context derivation during loading. If `Input.Filesystem` is also supplied,
that complete concrete context wins. Do not merge fields or fall back to derivation
when the explicit context is invalid. This permits overriding reusable companion
setup without clearing its binding declaration.

Layered overlays derive automatically when `OverlayFilesystem` is absent; no
additional opt-in flag is needed. An explicit overlay context wins as a whole.
If OD-2 calls for configurable overlay fallback behavior, reuse the small binding
settings where appropriate without requiring a general context-selection API.

Keep the binding on `Input`: `Source` still describes the supplied content and
`FilesystemContext` still describes a concrete location. This does not reinterpret
a file source's own path. Grouping related settings avoids scattering them across
a boolean and sibling fields, while keeping the addition smaller than a general
context selector. No source wrapper, deferred fields on `FilesystemContext`,
callback, or second file-selection API is needed for this design.

## Open Decisions

### OD-2: Scope and representation of absent-main handling

**Settled:** optional companions should support defaults-only startup; a declared
fallback is useful, and guessed candidate directories or implicit working-directory
fallback are inappropriate. Selected-main failures remain terminal.

**Recommendation:** put absence behavior on each derived binding. Default to
skipping optional companion import expansion when no main base exists. Provide
an explicit require-base choice and a concrete fallback context; use a fallback
only when no main base is available, never to recover from a selected-file error.
Keep any independent payload in the underlay input, and do not silently waive
required import declarations. A supplied fallback for the overlay can likewise
provide its lookup base; an active overlay without any usable context fails.
This preserves useful absent-main behavior without silently discarding unrelated
configuration or conflating required imports with optional ones. Finalize how
these choices are represented and how a fallback declares lookup base and root.

**Alternative:** scope skip to the entire explicitly designated companion input.
This is simpler for the import-only mappings Audita describes and avoids partial
input processing, but also omits any payload or required imports in that input.
It needs clear semantics for that deliberate condition. A single load-wide
absence policy is another viable convenience, but cannot express different
requirements for separate companion inputs as directly.

### OD-3: Containment inheritance and symlink convention

**Settled:** derived companion imports and derived layered overlays use the main
file's directory for lookup, even with a wider configured containment root.
Explicit overlay contexts retain their current lookup semantics.

**Recommendation:** use the selected main's canonical directory, and inherit its
resolved namespace and containment root for derived bindings. Existing import
rules then provide file-local bases for transitive imports. Permit parent-relative
paths when existing containment rules allow them. This reuses current canonical
file behavior and preserves intentional wider roots without relocating companions.
A concrete underlay context override can select a different scope under its
normal rules. Document the symlink-target-directory convention explicitly.

**Alternative:** create a fresh containment root at the selected main directory.
This keeps companions locally scoped but discards intentional wider roots and
prevents some sibling-directory layouts. Anchoring to the lexical symlink directory
is also viable for deployment bundles; it differs from current file/import
behavior and would need distinct lookup and containment handling.

### OD-4: Main-source eligibility beyond discovered host files

**Settled:** both explicitly selected and discovered host mains support the
workflow. Non-file companion inputs, especially mappings and bytes, are in scope;
this decision concerns the main input that supplies their base.

**Recommendation:** also support native `Filesystem.File` mains using their
namespace's path and ownership rules. For an initial file-based scope, non-file
mains use independently supplied or fallback contexts rather than implicitly
providing a main-file directory. This covers the requested workflow and avoids
making discovery a prerequisite for equivalent file inputs.

**Alternative:** derive from any usable main context, including `BaseDir` on a
byte, reader, or mapping main. This is more uniform with context-based loading
and avoids excluding a known base, but requires explaining main-relative as
main-context-relative and deciding how root-only or absent-main contexts behave.
A host-only initial scope is smaller but leaves native-file consumers with the
same coordination gap.