Defer single-file overlays and retire the roadmap
This commit is contained in:
@@ -102,3 +102,29 @@ string equality must not be presented as semantic equality.
|
||||
Keeping `Path` solely as an opaque query/building value remains an alternative.
|
||||
No current contract changes or implementation commitment follow from this entry;
|
||||
a comparison operation can be added compatibly later.
|
||||
|
||||
## 5. Single-file named overlays
|
||||
|
||||
Status: deferred. Consider supporting a single `overlays.yml` containing named
|
||||
overlay bodies alongside the existing
|
||||
[selected-overlay directory convention](../config.md#selected-overlays).
|
||||
This would let small applications keep their overlays in one file. Planning was
|
||||
tabled because the added semantics outweighed the immediate convenience; the
|
||||
draft is preserved at `3e50388:docs/roadmap/overlays.md`.
|
||||
|
||||
The draft selected a direct name-to-body mapping, strict peer composition when
|
||||
both storage forms contribute to the selected name, and file-level version
|
||||
checks using the application's existing rule and mode. These provide a starting
|
||||
point for reconsideration, not an implementation commitment.
|
||||
|
||||
### Questions to resolve before implementation
|
||||
|
||||
- Where should collection controls live: within each named body or at catalog
|
||||
level, and how should that interact with document-root control ownership?
|
||||
- How should configured version metadata paths coexist with overlay names,
|
||||
including nested paths and modes that do not check overlay versions?
|
||||
- Which semantic checks should apply to unselected bodies?
|
||||
- Should `.yaml` also be recognized, and how should multiple matching files
|
||||
interact?
|
||||
- How should catalog lookup be enabled without disrupting successful loads that
|
||||
previously ignored an adjacent `overlays.yml`?
|
||||
|
||||
@@ -1,357 +0,0 @@
|
||||
# Single-file named overlays
|
||||
|
||||
Status: strict composition, direct catalog structure, and file-level version
|
||||
checking are settled. Remaining decisions are listed below. Not implemented.
|
||||
Implementation baseline: `ce6005c`.
|
||||
|
||||
## Purpose
|
||||
|
||||
Support a single `overlays.yml` containing named overlay definitions alongside
|
||||
the existing `overlays/<name>.yml` convention. Small applications can keep their
|
||||
profiles together, while applications with larger configurations can split them
|
||||
into individual files or contribute through both forms.
|
||||
|
||||
Applications continue to select one overlay name. ConfigKit selects the matching
|
||||
contributions, strictly composes them, and applies the resulting overlay once.
|
||||
This follows [ADR 0001](../adr/0001-separate-composition-and-overlay.md): peer
|
||||
composition and intentional precedence remain separate operations.
|
||||
|
||||
This roadmap extends file-based selected overlays. The selected contributions
|
||||
compose as peers at the overlay stage; overlay definitions do not acquire
|
||||
underlay or main-input precedence tiers. A configuration-wide catalog populated
|
||||
through inline `config.overlays`, ordinary imports, or application underlays is
|
||||
outside this feature's scope. No new application or source-control key is reserved.
|
||||
|
||||
## Current baseline
|
||||
|
||||
The [selected-overlay contract](../config.md#selected-overlays) currently loads
|
||||
one YAML mapping from `overlays/<name>.yml`. Overlay names are not paths. Empty
|
||||
selection and the exact lowercase name `default` select no overlay. A selected
|
||||
missing file fails; there is no extension fallback or overlay stack.
|
||||
|
||||
Ordinary loads and explicit layered overlay contexts locate that file beneath
|
||||
the configured root. Layered main-derived and fallback contexts use their
|
||||
resolved base for lookup, with a separate containment root. The implemented
|
||||
[main-relative context decision](../adr/0022-main-relative-context.md) supplies
|
||||
the context needed by this feature; it does not implement aggregate overlays.
|
||||
|
||||
The existing loader in `internal/engine/selected_overlay.go` acquires and parses
|
||||
the file, checks overlay directives and source-version policy, and applies its
|
||||
payload with the effective collection policy. Overlays apply after base
|
||||
composition and main-version checking, before inheritance, defaults, decoding,
|
||||
CLI overrides, application resolution, and final validation. Source records,
|
||||
locations, resource contexts, history, repetition checks, limits, and handle
|
||||
cleanup already cover the individual-file form.
|
||||
|
||||
## Targeted end state
|
||||
|
||||
### File layout and selection
|
||||
|
||||
The catalog maps overlay names directly to overlay bodies, without an enclosing
|
||||
`overlays` key. It may also carry file-level version metadata under the application's
|
||||
configured version path. The following example omits optional version metadata:
|
||||
|
||||
```yaml
|
||||
# overlays.yml — planned format
|
||||
production:
|
||||
service:
|
||||
workers: 8
|
||||
development:
|
||||
service:
|
||||
workers: 1
|
||||
log_level: debug
|
||||
```
|
||||
|
||||
Selecting `production` takes only that entry as the catalog's contribution.
|
||||
The same selection may also acquire this existing-format file:
|
||||
|
||||
```yaml
|
||||
# overlays/production.yml
|
||||
service:
|
||||
log_level: warning
|
||||
```
|
||||
|
||||
Strict composition produces this one overlay payload:
|
||||
|
||||
```yaml
|
||||
service:
|
||||
workers: 8
|
||||
log_level: warning
|
||||
```
|
||||
|
||||
If the individual file instead also defines `service.workers: 12`, ordinary
|
||||
strict-composition conflict rules apply. File order grants no precedence.
|
||||
The other catalog entries do not contribute to the selected configuration.
|
||||
OD-1 settles the direct catalog structure. Version checks are file-scoped as
|
||||
described below; metadata-name collisions, collection-control placement, and
|
||||
the validation boundary remain in OD-2 and OD-3.
|
||||
|
||||
Use the same lookup directory for the catalog and the existing `overlays/`
|
||||
directory. Preserve ordinary versus layered lookup semantics, explicit context
|
||||
precedence, main-relative and fallback behavior, namespace identity, symlink
|
||||
handling, and containment. The new filename variants and activation behavior
|
||||
remain in OD-4 and OD-5.
|
||||
|
||||
No selected overlay means neither form is read. With a selection, either form
|
||||
may supply the named overlay. An absent catalog entry is not an error when the
|
||||
individual file supplies the selection, and a missing individual file is not an
|
||||
error when the catalog supplies it. If neither contributes, report the existing
|
||||
missing-overlay condition. A selected empty mapping is a present contribution,
|
||||
not a missing overlay. Acquisition or parsing failures are not a reason to
|
||||
silently prefer the other form.
|
||||
|
||||
### File-level version checks
|
||||
|
||||
**Settled:** a version declaration checks whether a physical configuration file
|
||||
declares an acceptable specification version. It does not independently version
|
||||
each overlay body. Use the existing application-configured `Definition.Version`
|
||||
path and source-version mode; do not hard-code the spelling `version`.
|
||||
|
||||
For an application using the root `version` key, a catalog can contain:
|
||||
|
||||
```yaml
|
||||
version: 2
|
||||
production:
|
||||
service:
|
||||
workers: 8
|
||||
development:
|
||||
service:
|
||||
workers: 1
|
||||
```
|
||||
|
||||
Apply the configured source-version policy once to each acquired physical file,
|
||||
before its contribution enters strict composition. The catalog check is independent
|
||||
of which name is selected and also applies when the acquired catalog lacks that
|
||||
name. No selected overlay still means neither form is read or checked. In enabled
|
||||
modes, evaluate the catalog declaration at its physical document root, not inside
|
||||
the selected body. Preserve the existing distinction between check-when-present
|
||||
and required modes and the application's authority to accept the declared value.
|
||||
|
||||
An acquired individual overlay file has its own declaration and existing version
|
||||
processing. When both files contribute, each is checked independently; one file
|
||||
cannot satisfy another's required declaration. Do not check or require separate
|
||||
version declarations in the catalog's named bodies. Keep catalog version metadata
|
||||
out of the applied overlay, and keep its evidence attached to the actual catalog
|
||||
source and authored location. Body values remain subject to normal protection of
|
||||
the application's version path when applying the overlay.
|
||||
|
||||
This decision concerns stale-file detection. Collection controls describe overlay
|
||||
operations and need not share the version declaration's scope. OD-2 separates the
|
||||
remaining control-placement and catalog-metadata recognition decisions.
|
||||
|
||||
### Strict composition, then one overlay application
|
||||
|
||||
**Settled:** contributions for the selected name from both forms are peers.
|
||||
Strictly compose them before applying their combined result to the lower
|
||||
configuration. Do not apply them sequentially as precedence layers, suppress
|
||||
one form, or reject a shared name merely because both forms contribute.
|
||||
|
||||
Reuse existing composition and collection-policy rules. Effective lower-tier
|
||||
identity and applicable selected-overlay declarations must be available when
|
||||
composing the peers. Disjoint identified records can compose; conflicting
|
||||
identities among peers fail. The combined collection then uses its applicable
|
||||
`replace`, `append`, or `upsert` operation against the lower value once.
|
||||
Unidentified sequences retain existing strict-composition and overlay behavior.
|
||||
|
||||
Resolve collection declarations across the contributions using the existing
|
||||
same-tier rules; do not independently apply each contribution's mode to the
|
||||
lower configuration. Keep the existing rules for version-path protection and
|
||||
forbidding overlay imports. This feature adds neither overlay inheritance nor
|
||||
selection of several overlay names.
|
||||
|
||||
### Processing and source evidence
|
||||
|
||||
Acquire selected contributions within the existing load, after base processing
|
||||
reaches the overlay phase. Preserve subsequent lifecycle order and normal
|
||||
failure behavior. Reuse parser and composition representations rather than
|
||||
decoding the catalog into application structs and reconstructing a document.
|
||||
|
||||
The selected catalog body's application paths are relative to that body, while
|
||||
its locations still identify the real catalog file and authored line/column.
|
||||
Diagnostics must distinguish application coordinates from the catalog wrapper;
|
||||
never fabricate a physical file such as `overlays.yml/production`. Relative
|
||||
resource contexts follow the physical declaring file, so resources beside
|
||||
`overlays.yml` and beside `overlays/production.yml` have different bases.
|
||||
Preserve both contributors when mappings combine, and normal field origins and
|
||||
history when the completed overlay is applied.
|
||||
|
||||
Keep real acquisition accounting: parse the acquired catalog under existing
|
||||
YAML limits and charge its complete bytes and parsed nodes, including unselected
|
||||
bodies. Each acquired physical file contributes a source record; unused names
|
||||
are not separately acquired sources. Preserve repetition/containment checks,
|
||||
detached inspection, and cleanup on every exit. Do not add discovery-attempt
|
||||
history or a new inspection API solely for this feature.
|
||||
|
||||
Use deterministic acquisition order for source IDs and sequence assembly. The
|
||||
recommended order is catalog contribution, then individual-file contribution;
|
||||
this orders evidence and composed records, not precedence. Reuse existing
|
||||
diagnostic classifications where they fit. Record file-level version evidence
|
||||
once per acquired source; collection-control scope remains open in OD-2.
|
||||
|
||||
## Work from the baseline
|
||||
|
||||
1. **Settle the remaining contract choices.** Resolve the decisions below,
|
||||
including activation compatibility, and prepare examples for each source
|
||||
form, combined contributions, source-version modes, and collection controls.
|
||||
2. **Separate overlay acquisition from application.** Adapt the selected-overlay
|
||||
loader to acquire optional contributions and distinguish missing selection
|
||||
from acquisition failure. Retain existing context selection and access rules.
|
||||
3. **Check files and select catalog bodies without losing source evidence.** Parse
|
||||
and apply file-level source-version policy to each acquired file. Separate
|
||||
catalog metadata from named definitions, select the requested body, and retain
|
||||
actual locations. Apply the chosen collection-control scope and account for
|
||||
the complete acquired file.
|
||||
4. **Compose peers and apply once.** Reuse strict composition with the lower
|
||||
collection policy and selected declarations, then feed the composed result
|
||||
into the existing overlay operation. Preserve contributor origins, history,
|
||||
sensitivity, version protection, and resource contexts.
|
||||
5. **Verify and document.** Add public integration coverage and maintained
|
||||
consumer examples; update selected-overlay contracts, public declaration
|
||||
comments as needed, and internal ownership documentation when implemented.
|
||||
|
||||
This is a dependency order for a later implementation plan, not a prescribed
|
||||
commit layout. Do not change existing ADR decisions merely to document a new
|
||||
storage form. Review any durable ownership change separately if one emerges.
|
||||
|
||||
## Validation and completion
|
||||
|
||||
Cover the public loading boundary with:
|
||||
|
||||
- Catalog only, individual file only, both, neither, a catalog without the
|
||||
selected name, selected empty mappings, and inactive/default selections.
|
||||
- Complementary mappings, conflicting fields, identified-sequence composition,
|
||||
duplicate identities, mode declarations, and one final collection operation.
|
||||
- Ordinary and layered loading, discovered/explicit/non-file/absent mains,
|
||||
explicit overlay contexts, main-derived contexts, fallback contexts, wider
|
||||
roots, host and native filesystems, and existing link-handling capabilities.
|
||||
- Invalid YAML, catalog shape, selected/unselected body validation, source-version
|
||||
modes, protected version paths, and forbidden directives under the settled rules.
|
||||
- One file-level version evaluation per acquired catalog, independent of the
|
||||
selected name, including a catalog without that name. Cover both source forms
|
||||
under required and check-when-present modes, configured metadata paths and name
|
||||
collisions, metadata exclusion from payloads, and actual declaration locations.
|
||||
- Actual source paths and locations, origins and contributor history, sensitive
|
||||
values, distinct resource bases, source IDs, and detached snapshots.
|
||||
- Missing versus unreadable sources, repetition, complete catalog accounting,
|
||||
limits, and cleanup on failure. No duplicate load lifecycle callbacks.
|
||||
- Existing individual-file workflows and the activation compatibility cases
|
||||
settled in OD-5.
|
||||
|
||||
Use focused tests and normal repository checks during implementation. No new
|
||||
benchmark campaign or release gate is required. Completion means applications
|
||||
can select one named overlay from either or both forms with the same composition,
|
||||
precedence, and inspection model.
|
||||
|
||||
## Settled Decisions
|
||||
|
||||
### OD-1: Catalog structure — settled
|
||||
|
||||
Use the direct name-to-body mapping illustrated above, with no enclosing
|
||||
`overlays` mapping. The filename identifies the document's role; each named body
|
||||
remains recognizable as an individual overlay file's contents. The top-level
|
||||
mapping is a selection container and is not passed to application decoding.
|
||||
Overlay names retain existing spelling and selection rules, including empty
|
||||
selection and exact lowercase `default` meaning no overlay.
|
||||
|
||||
This settles the storage structure within the narrower file-based feature.
|
||||
File-level version metadata coexists with the named entries. Its recognition and
|
||||
name-collision rules, and collection-control placement, remain subject to OD-2.
|
||||
|
||||
## Open Decisions
|
||||
|
||||
### OD-2: File-level versions and overlay controls — partially settled
|
||||
|
||||
**Settled:** source-version checking operates once per acquired physical file,
|
||||
using the existing application rule and mode. The catalog's declaration belongs
|
||||
at its configured file-level path, not inside every named body. Collection-control
|
||||
placement is a separate decision. The targeted end state above records the
|
||||
version behavior; the remaining questions follow.
|
||||
|
||||
#### Collection-control placement
|
||||
|
||||
**Recommendation:** interpret `config.sequences` inside the selected body, using
|
||||
application-relative destinations such as `/prompts`. Process each selected
|
||||
contribution's controls before strict composition under existing same-tier rules.
|
||||
This keeps an overlay's collection operations beside its payload and preserves
|
||||
their meaning when moving that payload between forms. The file-level version
|
||||
declaration stays separate. Imports remain forbidden under existing overlay rules;
|
||||
nested application data does not activate controls.
|
||||
|
||||
**Alternative:** place shared collection controls at catalog level and apply them
|
||||
to every selected body. This reduces repetition where profiles use identical
|
||||
operations, but needs metadata-name rules, makes operations less local to their
|
||||
payloads, and differs from the structure of individual overlay files.
|
||||
|
||||
#### Version metadata recognition and name collisions
|
||||
|
||||
**Recommendation:** when the application configures a version path, give that
|
||||
path a defined metadata role in the catalog and reserve its enclosing top-level
|
||||
entry from overlay-name selection. For a root `version` path this reserves only
|
||||
the catalog name `version`; for a nested path it makes the enclosing entry a
|
||||
metadata container. This avoids guessing whether a value is a header or a profile.
|
||||
Do not reserve a hard-coded version name when no version rule is configured.
|
||||
|
||||
Specify how recognition works in main-only mode separately from whether a checker
|
||||
runs, keeping metadata out of overlay payloads. Retain support for existing version
|
||||
path forms; concrete examples must cover nested paths and retained empty ancestors
|
||||
after header consumption. These rules are still open, not implicit additions to
|
||||
the settled file-level check.
|
||||
|
||||
**Alternative:** reserve only the exact version path and allow other fields in
|
||||
its enclosing entry to form an overlay body. This permits more names and avoids
|
||||
reserving a whole metadata container, but gives one mapping both metadata and
|
||||
profile roles. Selection, header removal, and empty-body behavior become harder
|
||||
to explain and need explicit rules.
|
||||
|
||||
### OD-3: Validation of unselected bodies
|
||||
|
||||
**Recommendation:** require a valid YAML mapping catalog and look up the selected
|
||||
name. Preserve parser-level validation of the whole file, including duplicate
|
||||
mapping-key rules and limits. Require the selected body to satisfy normal overlay
|
||||
mapping and directive rules, but do not run those semantic checks or application
|
||||
callbacks for unselected bodies. File-level version policy still applies to the
|
||||
acquired catalog independently of selection. Final application validation assesses
|
||||
the resulting effective configuration, not each profile in isolation.
|
||||
|
||||
This keeps a load focused on the requested configuration while acknowledging
|
||||
that malformed YAML anywhere prevents reliable parsing. An empty mapping catalog
|
||||
contains no definitions; selected null/scalar/sequence bodies follow the existing
|
||||
mapping-root rules rather than becoming shorthand for absence.
|
||||
|
||||
**Alternative:** validate every body's shape, name, and directives when
|
||||
reading the catalog. This catches unused-profile defects earlier, but makes one
|
||||
profile's load depend on unrelated definitions and expands semantic validation.
|
||||
A whole-catalog lint facility could serve that purpose separately if requested.
|
||||
|
||||
### OD-4: Filename variants
|
||||
|
||||
**Recommendation:** initially recognize exactly `overlays.yml` and the existing
|
||||
`overlays/<name>.yml`. This matches current overlay spelling without adding a
|
||||
separate extension-precedence decision. Both recognized forms compose as peers.
|
||||
|
||||
**Alternative:** also recognize `.yaml`. This accommodates common YAML filenames,
|
||||
but requires deciding whether multiple extensions are additional peers or fallback
|
||||
alternatives. If adopted, prefer strict peer composition over hidden precedence;
|
||||
document stable acquisition order and collisions consistently for both forms.
|
||||
|
||||
### OD-5: Activation and compatibility
|
||||
|
||||
**Recommendation:** use the existing selected overlay name to activate lookup of
|
||||
both forms, without another option in the common workflow. The storage layout
|
||||
should not require consumers to implement different selection APIs. Document
|
||||
that an acquired catalog participates in parsing even if its selected name is
|
||||
absent, and that conflicting selected contributions fail strict composition.
|
||||
|
||||
Before committing to release compatibility, assess existing deployments that
|
||||
have an `overlays.yml` previously ignored by ConfigKit: automatic participation
|
||||
can expose invalid content or conflicts alongside an otherwise successful
|
||||
individual-file load. The [release policy](../policy/release.md#versioning-and-compatibility)
|
||||
preserves successful valid configurations; merely accepting additional layouts
|
||||
does not resolve this interaction. No release version is selected here.
|
||||
|
||||
**Alternative:** add explicit catalog enablement, preserving existing directory-only
|
||||
lookup until selected by the application. This provides a clear compatibility
|
||||
boundary and lets consumers adopt the feature deliberately, but introduces an
|
||||
extra setting for the common layout. Once enabled, contributions from both forms
|
||||
must still compose strictly; this alternative does not reopen the settled
|
||||
coexistence decision.
|
||||
Reference in New Issue
Block a user