From e3860036f899ae8e57926504828382d20907ff92 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Wed, 14 Oct 2026 19:03:25 +0000 Subject: [PATCH] Plan shared source version checks --- docs/roadmap/source-version-checks.md | 318 ++++++++++++++++++++++++++ 1 file changed, 318 insertions(+) create mode 100644 docs/roadmap/source-version-checks.md diff --git a/docs/roadmap/source-version-checks.md b/docs/roadmap/source-version-checks.md new file mode 100644 index 0000000..19bbb6b --- /dev/null +++ b/docs/roadmap/source-version-checks.md @@ -0,0 +1,318 @@ +# Shared Source Version Checks Roadmap + +Status: planned; implementation has not started. Execute the stages below in +order. This roadmap develops the partially accepted source-version request in +[PromptKit's proposal](promptkit-feature-request.md#partially-accepted-addition-shared-source-specification-checks) +using the consumer's single `Definition.Version` policy. The design direction +below incorporates the maintainer's subsequent decisions on lookup, consumption, +inspection, and declaration handling; it supersedes the proposal's earlier +open-ended discussion of those subjects for this work. + +## Targeted end state + +A consumer can require that source documents contributing to a configuration +declare a version accepted by its existing `Definition.Version` checker. There +is one consumer-owned configuration specification and one acceptance policy. +Different accepted declarations may coexist: ConfigKit does not require exact +equality with the main declaration, impose SemVer, or invent an expected value. + +The consumer selects one of three modes through the supported loading API: + +| Mode | Missing non-main declaration | Present non-main declaration | +| --- | --- | --- | +| Current behavior, the default | Existing behavior | Existing behavior, including protected-path restrictions | +| Check when present | Proceed without a source-version callback for that omission | Apply the shared version vocabulary and checker | +| Require declarations | Structured load error | Apply the shared version vocabulary and checker | + +The main declaration remains part of the configuration and retains its existing +checker policy. The treatment of missing main input and mounted main paths is +addressed explicitly below rather than silently replacing existing behavior. +An enabled mode requires a configured version rule and checker; select a normal +options error for an unusable rule, rather than silently ignoring the mode. + +### Source-local lookup and shared policy + +In enabled modes, interpret the configured header path relative to each source +document before mounting. Read that document's own declaration, not a composed +or overlaid value. An import cannot supply its parent's missing declaration, +and another source or later tier cannot hide a rejected declaration. Check +acquired participating documents even if their application payload is later +superseded. A rejection fails the load with no usable partial result. + +Reuse the existing version vocabulary: absence, explicit null, strings, and +strict decimal integers. Null is present and reaches the checker as null; it is +not an omission. A required declaration need not be non-null if the consumer's +checker accepts null. Unsupported scalar/container kinds remain present but +fail the existing kind of version-type validation before invoking the checker. +Do not coerce malformed values into strings or treat them as missing. Preserve +exact string/integer text and the existing absence of a machine-integer range +requirement. No new string-versus-integer discriminator is required by this scope. + +This checks declared compatibility. Final decoding and domain validation still +validate the assembled application; fragments are not independently decoded as +complete applications, and declarations do not prove schema conformance. + +### Header consumption and evidence + +After a non-main declaration passes its check, consume that declaration as +structural source metadata before its application content participates in +composition. Preserve the main declaration and existing main value semantics. +Do not consume unrelated nested versions, imported payload fields at other +paths, or declarations in the default current-behavior mode. + +For example, with a source-local header path `version`, a non-main document's +`version: "1"` is checked and consumed even when that document will be mounted +under `collections/weather`. A local release label at that same path cannot +simultaneously remain ordinary payload in an enabled mode. Applications must +choose a distinct location for release metadata or for their specification +header. A prompt's `version` below a separate prompt key is unaffected unless +that is the configured header path. + +Retain each checked declaration, its source-local path, and its physical source +location as inspectable evidence. Keep this evidence separate from effective +domain values and their composition history; consumed headers should not appear +as surviving fields or create effective-value diff changes. Source destination +and resource context remain available through existing inspection. Missing +declarations need an explicit absence state and an appropriate source-location +fallback for diagnostics, not an invented authored header coordinate. + +Preserve normal safe diagnostics, warning behavior, callback lifetime, and +disclosure rules. The public evidence representation and treatment of classified +declarations need the design decision below; retaining evidence does not authorize +unredacted exposure of sensitive values or arbitrary malformed input. + +### Participation and unchanged boundaries + +Document participation must be explicit, independent of filesystem naming or +collection identity. The recommended scope is: + +| Contribution | Recommended treatment | +| --- | --- | +| Selected main document | Shared checker; declaration retained; no duplicate main callback | +| Direct underlay documents | Check under the selected mode; accepted headers consumed | +| Imported documents, including transitive imports and glob matches | Same policy and mode; accepted headers consumed | +| Selected overlay document | Same policy and mode; accepted header consumed before overlay application | +| Host files, native filesystem files, bytes, readers, and mapping sources | Equivalent document participation under their existing source contracts | +| CLI field overrides | Synthetic field operations; no document header requirement | +| Typed defaults, resolution operations, inheritance copies, and placement wrappers | Not newly acquired documents; no additional header checks | +| `NoSource`, unmatched optional imports, and empty groups | No acquired document and no new header requirement | + +A present empty document mapping is still a document. Reuse of a source in +allowed separate tiers retains existing acquisition semantics; do not introduce +a global once-only check or a new uniqueness policy. The participation boundary +and mode API remain recommendations to settle in stage 1. + +Existing source acquisition, containment, import destinations, strict composition, +precedence, inheritance, and complete-on-return loading continue to apply. +Consumption permits the checked header to serve as metadata; it must not allow +remaining payload to bypass main-version ownership. Synthetic overrides remain +subject to their existing protected-path restrictions. + +This feature does not add independently versioned collections or subtrees, +per-input compatibility policies, automatic schema migration, YAML-selected +callbacks, or general source-validation hooks. These are outside this delivery, +not new permanent prohibitions on future work. + +## Current baseline + +Reviewed worktree baseline: `b3fc6a5`. Input destinations and relative import +destinations are implemented; the +[relative import roadmap](relative-import-destinations.md) records completion. + +- [VersionRule and Version](../../lifecycle.go) provide a path, a checker, and + absent/null/text states. There is no expected-version string or source-check + mode. Root facade conversions preserve the public/internal boundary. +- The [version contract](../config.md#application-schema-versions) checks the + main version before defaults and typed decoding. The current path is an + effective application path; changing its interpretation in enabled modes + needs explicit handling for mounted main documents. +- [Loading](../../internal/engine/load.go) invokes the main check after main + acquisition/composition. [Version validation](../../internal/engine/validation.go) + already handles scalar vocabulary, checker diagnostics, and safe callback + errors. It is a useful shared mechanism, not a ready-made source-check API. +- [Imports](../../internal/engine/imports.go), + [underlays](../../internal/engine/load_request.go), and + [selected overlays](../../internal/engine/selected_overlay.go) currently reject + contributions to the protected version path. Enabled source checks need to + extract and consume headers before payload protection and composition, without + weakening the default behavior or checking only the eventual winning values. +- Existing source records carry identity, locations, and destination, but no + dedicated checked-version evidence. Existing sensitivity and inspection are + primarily organized around application paths and retained application trees. + +## Ordered implementation plan + +### 1. Finalize the contract and integration design + +Resolve the Open Decisions below. Select public mode names and placement, source +participation, main-path handling, and evidence types. Define header lookup for +nested paths and distinguish a missing member from an obstructing non-container +ancestor. Specify consumption without broadening it into a generic transformation +API. Recommend removing only the selected header node and preserving its authored +ancestor containers, including newly empty mappings; decide handling of overlaps +with structural directives before implementation. + +Record the change to main-only version ownership in an ADR and review public API +compatibility under repository policy. Do not rewrite accepted ADR decisions or +describe the feature as implemented before it lands. Existing default behavior +must remain covered; tests must not turn incidental acquisition ordering or +allocation choices into new guarantees. + +Completion: the three modes and every participating source role have defined +lookup, callback, consumption, failure, and inspection semantics. + +### 2. Share declaration checking and integrate source acquisition + +Separate reusable version-value interpretation and checker reporting from the +current main-only invocation. Carry the selected mode through ordinary and +layered root APIs into the engine. Resolve source-local locations before mounting +and perform non-main checks before composition or precedence can mask them. + +Consume only accepted non-main headers, retain their evidence, and apply existing +placement to the remaining body. Integrate direct inputs, recursive imports, and +selected overlays according to the selected participation contract. Continue +checking remaining payload for protected-path claims. Preserve source identity, +resource contexts, and applicable load accounting; consumed input still counts +as acquired input rather than bypassing source limits. + +Avoid duplicate main checks or a second load of any document. Preserve the main +declaration's final value and typed conversion behavior. Ensure warnings do not +become failures merely because acquisition helpers previously treated any returned +diagnostic as fatal. Keep returned hook errors safe and callback contexts closed. + +Completion: optional and required checks work across participating sources, +incompatible superseded input fails, and disabled-mode loads remain unchanged. + +### 3. Expose evidence and verify lifecycle agreement + +Add the selected root-owned inspection records and explicit conversions. Retain +source-local declarations and physical locations without fabricating effective +payload paths. Apply the agreed disclosure treatment to retained evidence, +including declarations consumed before final sensitivity evaluation. Return +detached records and preserve evidence when source files subsequently disappear. + +Check agreement among configuration, provenance, history, rendering, and diff: +main remains represented normally; consumed headers remain source evidence; +ordinary version payload remains ordinary. Verify diagnostics identify the +actual source even when several documents use the same local header path. +Follow existing load-error reporting rather than promising exhaustive evidence +for documents never acquired after an earlier failure. + +Completion: consumers can inspect checked declarations without reopening files, +and evidence does not leak sensitive values or alter effective-value semantics. + +### 4. Add public acceptance coverage and update documentation + +Use focused engine tests for vocabulary, extraction, and consumption; root tests +for public options and detached evidence; and executable consumer examples for +multi-file loading. Cover these distinct behaviors without duplicating all +existing import and composition tests: + +- all modes, with absent, null, string, decimal-integer, and malformed values; +- multiple versions accepted by one checker without equality enforcement; +- required missing declarations and incompatible inputs hidden by later layers; +- main preservation and no duplicate callback, including the chosen mounted-main + interpretation and `NoSource` behavior; +- direct underlays, transitive/glob imports, overlays, and every document source + kind; no header requirement for synthetic field overrides; +- source-local lookup before input and relative-import destination placement; +- nested-header consumption, empty containers, structural-path interactions, + and remaining payload protection; +- ordinary release/prompt version fields outside the header path; +- warning propagation, safe checker failures, real source locations, and absent + header location fallbacks; +- inspectable detached evidence and disclosure using disposable canaries; +- consumed-header exclusion from effective rendering/diffs, with retained main + history and unchanged resource-relative behavior; +- default-mode compatibility, including existing fragment version rejection. + +Demonstrate gradual adoption with optional checks and full declaration coverage +with required checks, using one policy. Do not include independent collection +checkers in examples. Update Go declaration documentation, the configuration +contract, consumer guidance, and internal overview as behavior lands; reconcile +the source-header request's earlier remaining-design notes then. + +Follow the [development guide](../development.md) and +[testing policy](../policy/testing.md): focused checks during implementation, +the everyday check before handoff, ownership race checks where applicable, and +documentation-link validation. No independent performance benchmarks are part of +this roadmap. Release preparation and publication are separate work. + +Completion: the agreed source coverage and three modes work through supported +public APIs, inspection and disclosure agree with the contract, and validation +and documentation are complete. Independent specification policies are not +required to complete this feature. + +## Open Decisions + +### Mode API and document participation + +Recommendation: expose a load-wide enum on ordinary and layered options, with +zero retaining current behavior, and use the participation table above. This +allows one reusable `Definition.Version` policy to be used with different adoption +strictness on different loads. Include mapping sources and selected overlays: +they contribute documents even though they are not necessarily imported files. +Exclude synthetic field operations and absent sources as described above. + +Alternative: place the mode on `VersionRule` so policy and enforcement travel +together. This reduces option plumbing but requires changing/copying the rule +to vary enforcement. A narrower first implementation checking imports and +underlays but exempting overlays is also viable; it reduces integration work but +offers weaker assurance about all contributing documents and needs an explicit +documented exception. Neither choice should imply per-source independent policies. + +### Mounted main documents and the protected application path + +Recommendation: in enabled modes, locate main's header source-locally as requested, +retain it, and derive its effective protected path by prefixing main's destination. +Use that derived path consistently for protection, existing typed version +conversion, and inspection. Invoke the shared checker once for main, preserving +its existing acceptance policy for absence; required mode additionally rejects a +missing declaration when an actual main document participates. `NoSource` retains +the existing absent-main checker behavior, since it is not a document. +Default mode keeps today's effective-path interpretation unchanged. This supports +mounted main input without contradicting source-local lookup or losing protection. + +Alternative: separate the public source-header path from the existing effective +main-version path while still using one checker. It makes both coordinate systems +explicit and can ease migration for mounted main callers, but adds configuration +and two paths that must be kept coherent. Root-only main input would simplify the +problem but would restrict an existing capability; do not impose that restriction +as an implementation shortcut. + +### Evidence representation and sensitivity + +Recommendation: attach a detached declaration record to existing source +inspection, with local path, presence/null/text state, physical location, and +explicit redaction state. Reuse existing sensitivity machinery where applicable +to the source's mounted header path, retaining classification evidence before +consumption so removing a node does not erase its sensitivity. Main evidence and +source evidence should use consistent disclosure. Retain actual declaration text +privately when needed; inspectable sensitive evidence can be redacted without +denying that a declaration was checked. Specify handling for historical tags and +dependency classification before choosing storage, rather than exposing raw +`Version.Text` unconditionally. + +Alternative: expose declarations through a dedicated source query returning safe +snapshots instead of extending source records. This better accommodates on-demand +redaction and richer evidence, but adds an API and lookup semantics. Both choices +must preserve physical attribution and the requested declaration evidence; neither +should expose arbitrary malformed input or require reopening the source. + +### Nested consumption and structural directives + +Recommendation: remove only the checked non-main declaration and retain authored +ancestor containers, even when empty. Validate source structure and interpret +existing directives so header consumption cannot conceal an invalid directive or +silently cancel an import. Preserve the existing path vocabulary where its +semantics apply; explicitly work through sequence-index and structural-path +interactions before implementation instead of adding a blanket top-level-only +header restriction. Retaining ancestors best preserves input presence outside +the requested consumed value. + +Alternative: prune ancestor containers emptied by consumption. This can make a +nested metadata-only header disappear completely from application payload, but +also removes authored empty-container presence and can affect defaults and +composition. A dedicated top-level header location would simplify these cases, +but narrows the existing configurable-path model and would need an explicit +consumer requirement or design decision rather than an implementation shortcut.