ConfigKit/docs/roadmap/source-version-checks.md

Shared Source Version Checks Roadmap

Status: planned; implementation has not started. Execute the stages below in order. This roadmap develops the partially accepted source-version request in PromptKit's proposal using the consumer's single Definition.Version policy. The design direction below incorporates the maintainer's subsequent decisions on lookup, consumption, inspection, and declaration handling; it supersedes the proposal's earlier open-ended discussion of those subjects for this work.

Targeted end state

A consumer can require that source documents contributing to a configuration declare a version accepted by its existing Definition.Version checker. There is one consumer-owned configuration specification and one acceptance policy. Different accepted declarations may coexist: ConfigKit does not require exact equality with the main declaration, impose SemVer, or invent an expected value.

The consumer selects a load-wide enum on ordinary and layered options. Its zero value retains current behavior; exact exported names remain implementation design work. The same Definition.Version can be reused across loads with different strictness. The three modes are:

Mode	Missing non-main declaration	Present non-main declaration
Current behavior, the default	Existing behavior	Existing behavior, including protected-path restrictions
Check when present	Proceed without a source-version callback for that omission	Apply the shared version vocabulary and checker
Require declarations	Structured load error	Apply the shared version vocabulary and checker

The main declaration remains part of the configuration and retains its existing checker policy. The treatment of missing main input and mounted main paths is addressed explicitly below rather than silently replacing existing behavior. An enabled mode requires a configured version rule and checker; select a normal options error for an unusable rule, rather than silently ignoring the mode.

Source-local lookup and shared policy

In enabled modes, interpret the configured header path relative to each source document before mounting. Read that document's own declaration, not a composed or overlaid value. An import cannot supply its parent's missing declaration, and another source or later tier cannot hide a rejected declaration. Check acquired participating documents even if their application payload is later superseded. A rejection fails the load with no usable partial result.

Reuse the existing version vocabulary: absence, explicit null, strings, and strict decimal integers. Null is present and reaches the checker as null; it is not an omission. A required declaration need not be non-null if the consumer's checker accepts null. Unsupported scalar/container kinds remain present but fail the existing kind of version-type validation before invoking the checker. Do not coerce malformed values into strings or treat them as missing. Preserve exact string/integer text and the existing absence of a machine-integer range requirement. No new string-versus-integer discriminator is required by this scope.

This checks declared compatibility. Final decoding and domain validation still validate the assembled application; fragments are not independently decoded as complete applications, and declarations do not prove schema conformance.

Header consumption and evidence

After a non-main declaration passes its check, consume that declaration as structural source metadata before its application content participates in composition. Preserve the main declaration and existing main value semantics. Do not consume unrelated nested versions, imported payload fields at other paths, or declarations in the default current-behavior mode.

For example, with a source-local header path version, a non-main document's version: "1" is checked and consumed even when that document will be mounted under collections/weather. A local release label at that same path cannot simultaneously remain ordinary payload in an enabled mode. Applications must choose a distinct location for release metadata or for their specification header. A prompt's version below a separate prompt key is unaffected unless that is the configured header path.

Retain each checked declaration, its source-local path, and its physical source location as inspectable evidence. Keep this evidence separate from effective domain values and their composition history; consumed headers should not appear as surviving fields or create effective-value diff changes. Source destination and resource context remain available through existing inspection. Missing declarations need an explicit absence state and an appropriate source-location fallback for diagnostics, not an invented authored header coordinate.

Preserve normal safe diagnostics, warning behavior, callback lifetime, and disclosure rules. Evidence will be attached to existing source inspection as a detached declaration record, distinguishing unchecked sources, checked absence, present null/text, and redaction, with local path and physical location. The remaining sensitivity decision below governs disclosure; retaining evidence does not authorize unredacted exposure of sensitive values or arbitrary malformed input.

Participation and unchanged boundaries

Document participation must be explicit, independent of filesystem naming or collection identity. The agreed scope is:

Contribution	Treatment
Selected main document	Shared checker; declaration retained; no duplicate main callback
Direct underlay documents	Check under the selected mode; accepted headers consumed
Imported documents, including transitive imports and glob matches	Same policy and mode; accepted headers consumed
Selected overlay document	Same policy and mode; accepted header consumed before overlay application
Host files, native filesystem files, bytes, readers, and mapping sources	Equivalent document participation under their existing source contracts
CLI field overrides	Synthetic field operations; no document header requirement
Typed defaults, resolution operations, inheritance copies, and placement wrappers	Not newly acquired documents; no additional header checks
NoSource, unmatched optional imports, and empty groups	No acquired document and no new header requirement

A present empty document mapping is still a document. Reuse of a source in allowed separate tiers retains existing acquisition semantics; do not introduce a global once-only check or a new uniqueness policy.

Existing source acquisition, containment, import destinations, strict composition, precedence, inheritance, and complete-on-return loading continue to apply. Consumption permits the checked header to serve as metadata; it must not allow remaining payload to bypass main-version ownership. Synthetic overrides remain subject to their existing protected-path restrictions.

This feature does not add independently versioned collections or subtrees, per-input compatibility policies, automatic schema migration, YAML-selected callbacks, or general source-validation hooks. These are outside this delivery, not new permanent prohibitions on future work.

Agreed main-path and consumption behavior

In enabled modes, locate main's header source-locally, retain it, and derive its effective protected path by prefixing main's destination. Use that effective path consistently for protection, existing typed version conversion, and inspection. Invoke the shared checker once for main. Preserve its existing acceptance policy for absence; required mode additionally rejects a missing header when an actual main document participates. NoSource retains the existing absent-main checker behavior because it is not a document.

Default mode keeps the existing effective-path interpretation. Root-mounted main inputs need no path adjustment; mounted-main callers opting in must account for the enabled mode's source-local interpretation. No separate main-version path or root-only main restriction is introduced.

For nested mapping headers, consume only the selected non-main declaration and retain authored ancestor containers, including newly empty mappings. This avoids removing additional input presence or changing unrelated default/composition behavior. Sequence-element and structural-directive cases remain open below.

Current baseline

Reviewed worktree baseline: b3fc6a5. Input destinations and relative import destinations are implemented; the relative import roadmap records completion.

• VersionRule and Version provide a path, a checker, and absent/null/text states. There is no expected-version string or source-check mode. Root facade conversions preserve the public/internal boundary.
• The version contract checks the main version before defaults and typed decoding. The current path is an effective application path; changing its interpretation in enabled modes needs explicit handling for mounted main documents.
• Loading invokes the main check after main acquisition/composition. Version validation already handles scalar vocabulary, checker diagnostics, and safe callback errors. It is a useful shared mechanism, not a ready-made source-check API.
• Imports, underlays, and selected overlays currently reject contributions to the protected version path. Enabled source checks need to extract and consume headers before payload protection and composition, without weakening the default behavior or checking only the eventual winning values.
• Existing source records carry identity, locations, and destination, but no dedicated checked-version evidence. Existing sensitivity and inspection are primarily organized around application paths and retained application trees.

Ordered implementation plan

1. Finalize the contract and integration design

Resolve the Open Decisions below. Select exact public mode and evidence type names within the agreed load-wide API, participation, main-path, and inspection design. Define header lookup for nested paths and distinguish a missing member from an obstructing non-container ancestor. Specify consumption without broadening it into a generic transformation API. Preserve authored ancestors when consuming nested mapping headers as agreed; settle sequence-element consumption and structural-directive overlaps below.

Record the change to main-only version ownership in an ADR and review public API compatibility under repository policy. Do not rewrite accepted ADR decisions or describe the feature as implemented before it lands. Existing default behavior must remain covered; tests must not turn incidental acquisition ordering or allocation choices into new guarantees.

Completion: the three modes and every participating source role have defined lookup, callback, consumption, failure, and inspection semantics.

2. Share declaration checking and integrate source acquisition

Separate reusable version-value interpretation and checker reporting from the current main-only invocation. Carry the selected mode through ordinary and layered root APIs into the engine. Resolve source-local locations before mounting and perform non-main checks before composition or precedence can mask them.

Consume only accepted non-main headers, retain their evidence, and apply existing placement to the remaining body. Integrate direct inputs, recursive imports, and selected overlays according to the selected participation contract. Continue checking remaining payload for protected-path claims. Preserve source identity, resource contexts, and applicable load accounting; consumed input still counts as acquired input rather than bypassing source limits.

Avoid duplicate main checks or a second load of any document. Preserve the main declaration's final value and typed conversion behavior. Ensure warnings do not become failures merely because acquisition helpers previously treated any returned diagnostic as fatal. Keep returned hook errors safe and callback contexts closed.

Completion: optional and required checks work across participating sources, incompatible superseded input fails, and disabled-mode loads remain unchanged.

3. Expose evidence and verify lifecycle agreement

Add the selected root-owned inspection records and explicit conversions. Retain source-local declarations and physical locations without fabricating effective payload paths. Apply the agreed disclosure treatment to retained evidence, including declarations consumed before final sensitivity evaluation. Return detached records and preserve evidence when source files subsequently disappear.

Check agreement among configuration, provenance, history, rendering, and diff: main remains represented normally; consumed headers remain source evidence; ordinary version payload remains ordinary. Verify diagnostics identify the actual source even when several documents use the same local header path. Follow existing load-error reporting rather than promising exhaustive evidence for documents never acquired after an earlier failure.

Completion: consumers can inspect checked declarations without reopening files, and evidence does not leak sensitive values or alter effective-value semantics.

4. Add public acceptance coverage and update documentation

Use focused engine tests for vocabulary, extraction, and consumption; root tests for public options and detached evidence; and executable consumer examples for multi-file loading. Cover these distinct behaviors without duplicating all existing import and composition tests:

• all modes, with absent, null, string, decimal-integer, and malformed values;
• multiple versions accepted by one checker without equality enforcement;
• required missing declarations and incompatible inputs hidden by later layers;
• main preservation and no duplicate callback, including the chosen mounted-main interpretation and NoSource behavior;
• direct underlays, transitive/glob imports, overlays, and every document source kind; no header requirement for synthetic field overrides;
• source-local lookup before input and relative-import destination placement;
• nested-header consumption, empty containers, structural-path interactions, and remaining payload protection;
• ordinary release/prompt version fields outside the header path;
• warning propagation, safe checker failures, real source locations, and absent header location fallbacks;
• inspectable detached evidence and disclosure using disposable canaries;
• consumed-header exclusion from effective rendering/diffs, with retained main history and unchanged resource-relative behavior;
• default-mode compatibility, including existing fragment version rejection.

Demonstrate gradual adoption with optional checks and full declaration coverage with required checks, using one policy. Do not include independent collection checkers in examples. Update Go declaration documentation, the configuration contract, consumer guidance, and internal overview as behavior lands; reconcile the source-header request's earlier remaining-design notes then.

Follow the development guide and testing policy: focused checks during implementation, the everyday check before handoff, ownership race checks where applicable, and documentation-link validation. No independent performance benchmarks are part of this roadmap. Release preparation and publication are separate work.

Completion: the agreed source coverage and three modes work through supported public APIs, inspection and disclosure agree with the contract, and validation and documentation are complete. Independent specification policies are not required to complete this feature.

Open Decisions

These are recommendations for further discussion, not accepted restrictions or implemented behavior. The mode API location, source participation, main-path handling, source-record evidence, and nested mapping ancestor preservation above have been agreed.

Sensitivity of consumed declaration evidence

Recommendation: classify declaration evidence using its original source shape at its mounted application path, including classified ancestors. Preserve the header as private evidence long enough to apply applicable existing field-tag, predicate, historical, and recorded-dependency sensitivity rules; consumption must not make a classified declaration public. Do not require the consumed path to survive in final effective metadata. Use explicit redaction state and omit the classified text from public evidence while retaining presence, source attribution, and the fact that it was checked. The checker still receives the declaration as trusted application code, as the existing version checker does.

For example, a header meta.version mounted at collections.weather is masked if either that header path or an ancestor such as collections.weather.meta is classified. An unclassified header remains inspectable. A later override must not erase applicable historical classification. Sequence evidence uses original source coordinates and segment kinds, not indexes shifted by consumption.

On failed loads, retain the existing safe diagnostic boundary: do not include raw header text in generated failure messages. If checked evidence is exposed through load errors before full sensitivity is available, mask its text unless applicable classification can already be established safely; this does not require a new failed-load evidence API or continuing a failed load to finish classification.

Alternative: always redact declaration text in public source evidence. This simplifies classification and prevents disclosure, but hides ordinary version identifiers that the consumer explicitly wants to inspect. Conversely, treating headers as inherently public would be simpler but would bypass existing classification where the configured header path or its ancestor is sensitive. Prefer normal classification rather than either blanket rule.

Headers that are sequence elements

Recommendation: support consumption of a selected sequence element by deleting that element, retaining the sequence and its order among remaining elements. Later indexes shift in the remaining payload; an emptied sequence remains []. This is ordinary removal, not a merge strategy. Retain the header's original index and physical location in source evidence; downstream payload paths reflect the post-consumption sequence. Main's retained declaration is not deleted. Removing a mapping member inside a sequence element instead retains that element, including an empty mapping, under the agreed ancestor-preservation rule.

This gives exact-node consumption a concrete meaning without restricting existing path forms. Verify protected-path and typed behavior where main retains an element but non-main payloads have shifted indexes; do not invent alignment or bypass normal sequence replacement, decoding, or ownership rules to make such inputs fit.

Alternative: reject headers whose selected node is a sequence element in enabled modes, while permitting mapping-member headers nested inside sequences. This avoids index shifts and may be easier for consumers to reason about, but narrows supported header locations and is not required for correctness if deletion is specified. Replacing the element with null would preserve indexes but fabricate a payload value and could change decoding; it is not the recommended interpretation of consumption.

Overlap with structural directives

Recommendation: reject an actual overlap between a checked header and an existing structural directive node with a structured error in enabled modes. A value must not simultaneously serve as a consumed specification declaration and as import or inheritance control. Include a header selected inside an import declaration or at an extends control; preserve ordinary structural validation so malformed controls cannot disappear through consumption. Use source-local paths and real locations for these errors. This is a proposed semantic restriction to approve, not an existing general prohibition on nested header paths.

Limit the rule to structural overlap, not substrings in ordinary keys. A header at metadata.version is not blocked merely because its source also contains configkit.imports, and a key such as extends_version is not an extends control. Missing headers remain governed by the selected mode; existing invalid directives still fail independently.

Alternative: interpret structural directives first, then allow the same node to serve as a checked and consumed header. This can be defined and avoids a new path restriction, but gives one value two unrelated roles: for example, an accepted version string could also name an import file or inheritance parent. It also needs extra ordering and retained-control rules to keep consumption from canceling behavior. Prefer an explicit overlap error over that dual interpretation; there is no identified consumer need for sharing these roles.