ConfigKit/docs/roadmap/source-version-checks.md

# Shared Source Version Checks Roadmap

Status: design decisions settled; implementation has not started. Execute the
stages below in order. This roadmap develops the partially accepted source-version request in
[PromptKit's proposal](promptkit-feature-request.md#partially-accepted-addition-shared-source-specification-checks)
using the consumer's single `Definition.Version` policy. The design direction
below incorporates the maintainer's subsequent decisions on lookup, consumption,
inspection, and declaration handling; it supersedes the proposal's earlier
open-ended discussion of those subjects for this work.

## Targeted end state

A consumer can require that source documents contributing to a configuration
declare a version accepted by its existing `Definition.Version` checker. There
is one consumer-owned configuration specification and one acceptance policy.
Different accepted declarations may coexist: ConfigKit does not require exact
equality with the main declaration, impose SemVer, or invent an expected value.

The consumer selects a load-wide enum on ordinary and layered options. Its zero
value retains current behavior; exact exported names remain implementation design
work. The same `Definition.Version` can be reused across loads with different
strictness. The three modes are:

| Mode | Missing non-main declaration | Present non-main declaration |
| --- | --- | --- |
| Current behavior, the default | Existing behavior | Existing behavior, including protected-path restrictions |
| Check when present | Proceed without a source-version callback for that omission | Apply the shared version vocabulary and checker |
| Require declarations | Structured load error | Apply the shared version vocabulary and checker |

The main declaration remains part of the configuration and retains its existing
checker policy. The treatment of missing main input and mounted main paths is
addressed explicitly below rather than silently replacing existing behavior.
An enabled mode requires a configured version rule and checker; select a normal
options error for an unusable rule, rather than silently ignoring the mode.

### Source-local lookup and shared policy

In enabled modes, interpret the configured header path relative to each source
document before mounting. Read that document's own declaration, not a composed
or overlaid value. An import cannot supply its parent's missing declaration,
and another source or later tier cannot hide a rejected declaration. Check
acquired participating documents even if their application payload is later
superseded. A rejection fails the load with no usable partial result.

Reuse the existing version vocabulary: absence, explicit null, strings, and
strict decimal integers. Null is present and reaches the checker as null; it is
not an omission. A required declaration need not be non-null if the consumer's
checker accepts null. Unsupported scalar/container kinds remain present but
fail the existing kind of version-type validation before invoking the checker.
Do not coerce malformed values into strings or treat them as missing. Preserve
exact string/integer text and the existing absence of a machine-integer range
requirement. No new string-versus-integer discriminator is required by this scope.

This checks declared compatibility. Final decoding and domain validation still
validate the assembled application; fragments are not independently decoded as
complete applications, and declarations do not prove schema conformance.

### Header consumption and evidence

After a non-main declaration passes its check, consume that declaration as
structural source metadata before its application content participates in
composition. Preserve the main declaration and existing main value semantics.
Do not consume unrelated nested versions, imported payload fields at other
paths, or declarations in the default current-behavior mode.

For example, with a source-local header path `version`, a non-main document's
`version: "1"` is checked and consumed even when that document will be mounted
under `collections/weather`. A local release label at that same path cannot
simultaneously remain ordinary payload in an enabled mode. Applications must
choose a distinct location for release metadata or for their specification
header. A prompt's `version` below a separate prompt key is unaffected unless
that is the configured header path.

Retain each checked declaration, its source-local path, and its physical source
location as inspectable evidence. Keep this evidence separate from effective
domain values and their composition history; consumed headers should not appear
as surviving fields or create effective-value diff changes. Source destination
and resource context remain available through existing inspection. Missing
declarations need an explicit absence state and an appropriate source-location
fallback for diagnostics, not an invented authored header coordinate.

Preserve normal safe diagnostics, warning behavior, callback lifetime, and
disclosure rules. Evidence will be attached to existing source inspection as a
detached declaration record, distinguishing unchecked sources, checked absence,
present null/text, and redaction, with local path and physical location. The
agreed sensitivity rules below govern disclosure; retaining evidence does not
authorize unredacted exposure of sensitive values or arbitrary malformed input.

### Participation and unchanged boundaries

Document participation must be explicit, independent of filesystem naming or
collection identity. The agreed scope is:

| Contribution | Treatment |
| --- | --- |
| Selected main document | Shared checker; declaration retained; no duplicate main callback |
| Direct underlay documents | Check under the selected mode; accepted headers consumed |
| Imported documents, including transitive imports and glob matches | Same policy and mode; accepted headers consumed |
| Selected overlay document | Same policy and mode; accepted header consumed before overlay application |
| Host files, native filesystem files, bytes, readers, and mapping sources | Equivalent document participation under their existing source contracts |
| CLI field overrides | Synthetic field operations; no document header requirement |
| Typed defaults, resolution operations, inheritance copies, and placement wrappers | Not newly acquired documents; no additional header checks |
| `NoSource`, unmatched optional imports, and empty groups | No acquired document and no new header requirement |

A present empty document mapping is still a document. Reuse of a source in
allowed separate tiers retains existing acquisition semantics; do not introduce
a global once-only check or a new uniqueness policy.

Existing source acquisition, containment, import destinations, strict composition,
precedence, inheritance, and complete-on-return loading continue to apply.
Consumption permits the checked header to serve as metadata; it must not allow
remaining payload to bypass main-version ownership. Synthetic overrides remain
subject to their existing protected-path restrictions.

This feature does not add independently versioned collections or subtrees,
per-input compatibility policies, automatic schema migration, YAML-selected
callbacks, or general source-validation hooks. These are outside this delivery,
not new permanent prohibitions on future work.

### Agreed main-path and consumption behavior

In enabled modes, locate main's header source-locally, retain it, and derive its
effective protected path by prefixing main's destination. Use that effective
path consistently for protection, existing typed version conversion, and
inspection. Invoke the shared checker once for main. Preserve its existing
acceptance policy for absence; required mode additionally rejects a missing
header when an actual main document participates. `NoSource` retains the existing
absent-main checker behavior because it is not a document.

Default mode keeps the existing effective-path interpretation. Root-mounted main
inputs need no path adjustment; mounted-main callers opting in must account for
the enabled mode's source-local interpretation. No separate main-version path or
root-only main restriction is introduced.

For nested mapping headers, consume only the selected non-main declaration and
retain authored ancestor containers, including newly empty mappings. This avoids
removing additional input presence or changing unrelated default/composition
behavior. Sequence-element and structural-directive behavior is settled below.

## Current baseline

Reviewed worktree baseline: `b3fc6a5`. Input destinations and relative import
destinations are implemented; the
[relative import roadmap](relative-import-destinations.md) records completion.

- [VersionRule and Version](../../lifecycle.go) provide a path, a checker, and
  absent/null/text states. There is no expected-version string or source-check
  mode. Root facade conversions preserve the public/internal boundary.
- The [version contract](../config.md#application-schema-versions) checks the
  main version before defaults and typed decoding. The current path is an
  effective application path; changing its interpretation in enabled modes
  needs explicit handling for mounted main documents.
- [Loading](../../internal/engine/load.go) invokes the main check after main
  acquisition/composition. [Version validation](../../internal/engine/validation.go)
  already handles scalar vocabulary, checker diagnostics, and safe callback
  errors. It is a useful shared mechanism, not a ready-made source-check API.
- [Imports](../../internal/engine/imports.go),
  [underlays](../../internal/engine/load_request.go), and
  [selected overlays](../../internal/engine/selected_overlay.go) currently reject
  contributions to the protected version path. Enabled source checks need to
  extract and consume headers before payload protection and composition, without
  weakening the default behavior or checking only the eventual winning values.
- Existing source records carry identity, locations, and destination, but no
  dedicated checked-version evidence. Existing sensitivity and inspection are
  primarily organized around application paths and retained application trees.

## Ordered implementation plan

### 1. Finalize the contract and integration design

Select exact public mode and evidence type names within the agreed load-wide
API, participation, main-path, inspection, and edge-case decisions. Define header
lookup for nested paths and distinguish a missing member from an obstructing
non-container ancestor. Specify consumption without broadening it into a generic transformation
API. Preserve authored ancestors when consuming nested mapping headers as agreed;
apply the agreed sequence-element removal and structural-overlap rejection rules.

Record the change to main-only version ownership in an ADR and review public API
compatibility under repository policy. Do not rewrite accepted ADR decisions or
describe the feature as implemented before it lands. Existing default behavior
must remain covered; tests must not turn incidental acquisition ordering or
allocation choices into new guarantees.

Completion: the three modes and every participating source role have defined
lookup, callback, consumption, failure, and inspection semantics.

### 2. Share declaration checking and integrate source acquisition

Separate reusable version-value interpretation and checker reporting from the
current main-only invocation. Carry the selected mode through ordinary and
layered root APIs into the engine. Resolve source-local locations before mounting
and perform non-main checks before composition or precedence can mask them.

Consume only accepted non-main headers, retain their evidence, and apply existing
placement to the remaining body. Integrate direct inputs, recursive imports, and
selected overlays according to the selected participation contract. Continue
checking remaining payload for protected-path claims. Preserve source identity,
resource contexts, and applicable load accounting; consumed input still counts
as acquired input rather than bypassing source limits.

Avoid duplicate main checks or a second load of any document. Preserve the main
declaration's final value and typed conversion behavior. Ensure warnings do not
become failures merely because acquisition helpers previously treated any returned
diagnostic as fatal. Keep returned hook errors safe and callback contexts closed.

Completion: optional and required checks work across participating sources,
incompatible superseded input fails, and disabled-mode loads remain unchanged.

### 3. Expose evidence and verify lifecycle agreement

Add the selected root-owned inspection records and explicit conversions. Retain
source-local declarations and physical locations without fabricating effective
payload paths. Apply the agreed disclosure treatment to retained evidence,
including declarations consumed before final sensitivity evaluation. Return
detached records and preserve evidence when source files subsequently disappear.

Check agreement among configuration, provenance, history, rendering, and diff:
main remains represented normally; consumed headers remain source evidence;
ordinary version payload remains ordinary. Verify diagnostics identify the
actual source even when several documents use the same local header path.
Follow existing load-error reporting rather than promising exhaustive evidence
for documents never acquired after an earlier failure.

Completion: consumers can inspect checked declarations without reopening files,
and evidence does not leak sensitive values or alter effective-value semantics.

### 4. Add public acceptance coverage and update documentation

Use focused engine tests for vocabulary, extraction, and consumption; root tests
for public options and detached evidence; and executable consumer examples for
multi-file loading. Cover these distinct behaviors without duplicating all
existing import and composition tests:

- all modes, with absent, null, string, decimal-integer, and malformed values;
- multiple versions accepted by one checker without equality enforcement;
- required missing declarations and incompatible inputs hidden by later layers;
- main preservation and no duplicate callback, including the chosen mounted-main
  interpretation and `NoSource` behavior;
- direct underlays, transitive/glob imports, overlays, and every document source
  kind; no header requirement for synthetic field overrides;
- source-local lookup before input and relative-import destination placement;
- nested-header consumption and retained empty containers; sequence-element
  removal with shifted payload indexes and original evidence coordinates;
- structural-directive overlap rejection without rejecting unrelated header paths
  or ordinary keys containing directive spellings; remaining payload protection;
- ordinary release/prompt version fields outside the header path;
- warning propagation, safe checker failures, real source locations, and absent
  header location fallbacks;
- inspectable detached evidence and disclosure using disposable canaries, including
  classified header/ancestor paths, historical classification, and safe failures;
- consumed-header exclusion from effective rendering/diffs, with retained main
  history and unchanged resource-relative behavior;
- default-mode compatibility, including existing fragment version rejection.

Demonstrate gradual adoption with optional checks and full declaration coverage
with required checks, using one policy. Do not include independent collection
checkers in examples. Update Go declaration documentation, the configuration
contract, consumer guidance, and internal overview as behavior lands; reconcile
the source-header request's earlier remaining-design notes then.

Follow the [development guide](../development.md) and
[testing policy](../policy/testing.md): focused checks during implementation,
the everyday check before handoff, ownership race checks where applicable, and
documentation-link validation. No independent performance benchmarks are part of
this roadmap. Release preparation and publication are separate work.

Completion: the agreed source coverage and three modes work through supported
public APIs, inspection and disclosure agree with the contract, and validation
and documentation are complete. Independent specification policies are not
required to complete this feature.

## Agreed Edge-Case Decisions

The maintainer has accepted all three decisions below. Together with the mode
API location, source participation, main-path handling, source-record evidence,
and nested mapping ancestor preservation above, these settle the discussed
product decisions. Exact public names and internal mechanics remain implementation
work; the feature is not yet implemented.

### Sensitivity of consumed declaration evidence

Classify declaration evidence using its original source shape
at its mounted application path, including classified ancestors. Preserve the
header as private evidence long enough to apply applicable existing field-tag,
predicate, historical, and recorded-dependency sensitivity rules; consumption
must not make a classified declaration public. Do not require the consumed path
to survive in final effective metadata. Use explicit redaction state and omit the
classified text from public evidence while retaining presence, source attribution,
and the fact that it was checked. The checker still receives the declaration as
trusted application code, as the existing version checker does.

For example, a header `meta.version` mounted at `collections.weather` is masked
if either that header path or an ancestor such as `collections.weather.meta` is
classified. An unclassified header remains inspectable. A later override must
not erase applicable historical classification. Sequence evidence uses original
source coordinates and segment kinds, not indexes shifted by consumption.

On failed loads, retain the existing safe diagnostic boundary: do not include raw
header text in generated failure messages. If checked evidence is exposed through
load errors before full sensitivity is available, mask its text unless applicable
classification can already be established safely; this does not require a new
failed-load evidence API or continuing a failed load to finish classification.

### Headers that are sequence elements

Support consumption of a selected sequence element by deleting
that element, retaining the sequence and its order among remaining elements.
Later indexes shift in the remaining payload; an emptied sequence remains `[]`.
This is ordinary removal, not a merge strategy. Retain the header's original
index and physical location in source evidence; downstream payload paths reflect
the post-consumption sequence. Main's retained declaration is not deleted.
Removing a mapping member inside a sequence element instead retains that element,
including an empty mapping, under the agreed ancestor-preservation rule.

This gives exact-node consumption a concrete meaning without restricting existing
path forms. Verify protected-path and typed behavior where main retains an element
but non-main payloads have shifted indexes; do not invent alignment or bypass
normal sequence replacement, decoding, or ownership rules to make such inputs fit.

### Overlap with structural directives

Reject an actual overlap between a checked header and an existing
structural directive node with a structured error in enabled modes. A value must
not simultaneously serve as a consumed specification declaration and as import
or inheritance control. Include a header selected inside an import declaration
or at an `extends` control; preserve ordinary structural validation so malformed
controls cannot disappear through consumption. Use source-local paths and real
locations for these errors. This is an explicitly accepted, narrow restriction
for enabled source checks, not a general prohibition on nested header paths.

Limit the rule to structural overlap, not substrings in ordinary keys. A header
at `metadata.version` is not blocked merely because its source also contains
`configkit.imports`, and a key such as `extends_version` is not an `extends`
control. Missing headers remain governed by the selected mode; existing invalid
directives still fail independently.