diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index c0e69fe..56e8e14 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,379 +1,388 @@ -# Source origins during resolution: implementation plan +# Public facade: implementation plan -Status: implementation complete (2026-09-12). All stages were completed in order. -The [feature roadmap](source-origins-during-resolution.md) owns the target contract -and consumer intent. This document owns implementation steps, validation, and -handoffs. It is written for one gpt-5.6-terra coding agent per stage; no stage -requires parallel agents or changes to a downstream repository. +Status: not started. Baseline: `28e7fc5f690b36c60e4d03586afcff00fb1dee90`. +The [feature roadmap](public.md) owns the target architecture and user intent. +This plan owns implementation decisions, ordered work, validation, and handoffs. +Each numbered stage is sized for one gpt-5.6-terra coding-agent assignment; +no stage requires subagents or changes to downstream repositories. -## Required reading and working rules +Historical navigation: this filename previously held the completed source-origins +plan, including its Stage 5 completion record. That record is preserved at the +baseline above: `git show 28e7fc5:docs/roadmap/implementation.md`. Existing incoming +links from the source-origins roadmap, remediation record, and development guide +refer to that historical plan, not the stages below. Stage 1 repairs those links. -Before each stage, inspect the worktree and read the preceding stage's completion -record. Follow [development](../development.md) and all policies: +## Working rules + +Before each stage read [development](../development.md), the feature roadmap, +the preceding completion records, and all applicable policies: [architecture](../policy/architecture.md), [documentation](../policy/documentation.md), [testing](../policy/testing.md), and [release](../policy/release.md). -Read the roadmap, the configuration contract's resolution and metadata sections, -and the relevant code before editing. Use the code graph for discovery and verify -current definitions rather than assuming line numbers or private signatures. +Read the owning configuration/API contracts and implementation before editing. +Prefer the code graph for discovery, verifying its results against current code. +Preserve unrelated work and the baseline's already-fixed resolution-history copy. -Preserve unrelated work. Keep each stage independently buildable with passing -tests; do not introduce stub public methods or defer known correctness failures. -Run the focused commands listed for the stage and `./scripts/check.sh` from the -repository root. Acquire required toolchains/dependencies explicitly before offline -checks. Document unsupported filesystem fixtures rather than silently omitting them. +Complete stages in order. Every handoff must build and pass its relevant tests; +do not leave placeholder public methods, skipped regressions, or known failures +for a later stage. Run `./scripts/check.sh` at every code-stage handoff, in addition +to the focused gates below. Keep default tests deterministic and offline. Follow +the development guide for dependency acquisition and supported toolchains. -Use real collaborators and temporary files for provenance. Keep default tests -offline, deterministic, and credential-free. Prefer codes, paths, layers, locations, -and observable snapshots over private call counts or incidental error messages. -Only use a narrow I/O or credential seam where it proves the no-extra-access rule. +Record completion beneath each stage: changed owners, tests and actual toolchain, +results, deviations, remaining risks, and the next handoff. Update current +documentation with actual changes, not promised completion. Commit stage work +when authorized; this plan does not authorize pushes, tags, release selection, +or publication. Release notes and native-platform release evidence remain part +of the separate release procedure. -At each handoff, update that stage's status here with changed files, tests/results, -actual toolchains, deviations, and remaining risks. Keep public reference and current -contract documentation synchronized with behavior as it lands. Commit coherent -stage work when authorized, but do not push, tag, select a version, or prepare a -speculative release note as part of this implementation sprint. +## Fixed implementation decisions -## Implementation decisions +### Package and migration strategy -The roadmap's two method signatures are fixed for this sprint: +Final production packages are the public root, `internal/engine`, and the existing +`internal/document` and `internal/yaml`. Keep the engine cohesive; do not create +separate packages for every phase. Root imports engine; engine imports its +supporting packages; no internal production package imports root. Only the YAML +adapter imports the upstream YAML dependency. -```go -func (r *Resolution) InputInfo(path Path) (FieldInfo, bool) -func (r *Resolution) Source(id string) (SourceInfo, bool) -``` +Public named types remain defined in root. Use named private fields for delegated +state, explicit forwarding methods, and ordinary public record declarations. +Do not alias public types to engine types, anonymously embed engine types, expose +engine types in public signatures/fields, or add public bridge methods. +Internal type aliases used only inside implementation are not public aliases. -The required behavior is a stable entry-state view and detached public return -records, not a particular internal representation. The recommended implementation -is a private `Metadata` value containing effective fields and path shapes, together -with a source-ID index. Reuse `Metadata.Lookup` semantics. A historical view is not -needed for this feature, and exposing one is outside this sprint's scope. +Because private-state references span most root implementation files, use a +bounded relocation candidate rather than attempting a live package split in +every stage. Stage 2 copies the existing implementation into `internal/engine`; +Stages 4–7 build the eventual root facade in temporary `internal/facade` (package +name `configkit`). Root remains the sole consumer entry point until Stage 8. +The temporary facade is staging space, not a second supported API/package. -Build the view after `metadataFromDecoded` and `applyOverlayMetadata` in -`resolveLoadInput`, immediately before `runResolution`, only when a resolver exists. -Its path shapes come from `decoded.final`, not the raw document or mutable Go -configuration. Separate copies of field/source records are one straightforward way -to isolate the view; immutable sharing or ownership transfer is equally acceptable -when it preserves query stability and detached returns. Source records come from -`assembledInput.sources`. Avoid aliases that observable consumer operations or -subsequent metadata updates could mutate while the view remains usable. -Missing or duplicate internally generated IDs are implementation-invariant failures -to fix, not new consumer-facing diagnostic categories. +This temporary duplication is a migration mechanism only: no feature development, +runtime selector, fallback engine, or new consumer import is introduced. Baseline +semantics remain fixed; necessary corrections must be reconciled explicitly. +Stage 8 moves the facade to root and removes the old implementation and temporary +directory in the same handoff. There is one production implementation at completion. +This makes the large textual move mechanical while separating it from the more +demanding ownership and callback work. Do not keep two engines as a final design. -Prefer passing the private view explicitly to `runResolution` and retaining it in -`resolutionState`. Update all private callers/tests deliberately: construct their -entry view from their decoded fixtures and supplied sources where they test these -queries. Every production resolution callback must receive complete provenance. -Private signatures and representation may change when an idiomatic alternative -better serves that requirement; record the reasoning in the stage handoff. +### Type and state ownership -Queries first check callback activity through the existing context guard. Only -then may invalid paths or unknown IDs return misses. `InputInfo` delegates to the -entry metadata lookup; `Source` clones the matched source on return. Preserve -existing `IsExplicit`, diagnostics, resolution recording, and secret timing. +| Public family | Selected representation and adaptation | +| --- | --- | +| `Path` | Root struct with named private `engine.Path` value; constructors and methods delegate. Preserve semantic segment kinds and invalid state by direct private wrapping, never stringify/reparse. | +| `Layer`, `ChangeKind`, `Severity`, `Code`, `SecretFileStatus` | Root-defined string types and explicitly typed constants with unchanged values. Convert to engine equivalents at the boundary; engine constants may supply values without making public aliases. | +| `Location`, `SourceInfo`, `Diagnostic`, `Origin`, `FieldInfo`, `Change` | Root-defined records with unchanged exported fields. Explicit private conversions cover nested pointers, slices, and paths. | +| `ValueSnapshot`, `ValueChange`, `SecretReferenceInfo`, `Explanation`, `SecretFileResult` | Root-defined output records. Convert detached engine records; preserve nil/empty distinctions and field semantics. | +| `Source` | Root struct with private `engine.Source` value. Engine owns bytes, constructor validity, and the shared reader claim; wrapper copies must not create new claims or re-read/copy source input. | +| `Metadata` | Root struct with private engine metadata value; explicit query methods convert detached results. Engine retains effective and historical indexes. | +| `Resolution`, `Validation` | Root structs with private pointers to their engine contexts. Engine alone owns activity and audit state; wrappers never recreate that state. | +| `Definition[T]`, `Options[T]`, `FilesystemContext`, `Override`, `VersionRule`, `Version` | Root records retaining all current fields and signatures. Private option/callback conversion into engine records; no extra configuration clone or reflection-based generic record mapper. | +| `Result[T]` | Root struct retaining `Config T`, plus private pointer to a non-generic engine inspection result. Engine load returns configuration and inspection separately; facade stores configuration exactly once. | +| `Secret` | Root-owned opaque scalar retaining value-copy semantics and existing public methods. Private exact-type adapter lets engine read/build it; no public accessor for plumbing. | +| `LoadError`, `SecretsError`, `SecretsReport` | Root structs with private engine backing state and explicit methods. Translate engine typed failures at entry points; never return or unwrap an engine error as the public structured error. | -Queries must not reload input, read files, resolve symlinks, access credentials, -or re-project mutable configuration instead of reading entry-state information. -Prefer reusing existing indexes and avoiding unnecessary whole-tree copies. -Per-load source indexing is a recommendation, not a complexity guarantee: a simple -scan may be appropriate for the existing bounded source collection. Assess costs -in context, including path lengths and nested record copies; this feature adds no -performance guarantees, resource caps, or benchmark gates. +The inventory in Stage 1 verifies every baseline exported declaration, not just +these grouped examples. Public type identity stays rooted in `configkit`. +Preserve receivers, generics, supported zero/nil behavior, interface satisfaction, +error assertions, and supported construction/copy behavior. Private field layout +and incidental formatting are not new compatibility promises. -## Stage 1 — Factor reusable effective-path indexing +Use handwritten, local conversion helpers. For detached outputs, transfer already +detached data where safe instead of deep-copying everything again. Never expose +engine-owned mutable slices/maps through public records. Preserve order and +missing/null distinctions. Do not round-trip records through JSON/YAML or use +`unsafe`, registration globals, or reflection to bypass private fields. -Status: complete (2026-09-12). Dependencies: none. +### Exact-type secret bridge -Scope: `metadata.go`, `load_metadata.go`, and their focused test owners. +Introduce an engine-internal `SecretBinding` containing `Type reflect.Type`, +`Read func(reflect.Value) SecretState`, and +`Make func(SecretState) reflect.Value`. `SecretState` carries the existing name, +value, specified, and resolved fields. It is operation data, never inspection +data. The root constructs the binding privately for exactly `reflect.TypeFor[Secret]()`; +Read and Make use ordinary type assertions/construction in the owning package. -1. Prepare reusable effective-path lookup for the entry view. The recommended - approach extracts indexing from `finalizeMetadataPaths`; an equivalent design - may reuse existing traversal directly. Keep historical indexing and result - publication behavior unchanged, and preserve one set of semantic-path rules. -2. Add the private entry-view construction described above using pre-resolution - metadata, the decoded final tree, and acquired source records. Share extracted - machinery with result finalization where useful. Do not expose public methods yet. -3. Establish and document internal ownership sufficient for stable queries and - detached returns. Reuse `cloneFieldInfo` and `cloneSourceInfo` where copying is - needed; do not require eager copies or duplicate history solely to match this - plan's recommended representation. -4. Add focused coverage proving the private view recognizes defaulted fields, - missing descendants, root, escaped keys, explicit key/index mismatches, and - contextual canonical sequence indices. Test stability across actual lifecycle - operations and mutation of returned records, not arbitrary mutation of backing - data that the implementation owns as immutable. Reuse existing shape tests for - exhaustive syntax cases rather than duplicating them. +Supply the binding explicitly to the engine load and schema construction. Store +it in the per-operation schema/value-operation context, available to recursive +decoding, projection, cloning, dynamic-value checks, resolution, and validation. +Keep the application configuration type `T` unchanged: no mirror configuration +structs and no conversion of arbitrary application objects. -Acceptance: all existing final metadata/history behavior is unchanged; a complete -entry view can be constructed before final publication without a historical index. -No public API or configuration contract changes are claimed by this stage. +Replace all direct engine assertions/constructions of the old `Secret`, not just +schema recognition. Include decoding, unresolved versus validated clone modes, +projection, required/optional references, lookup caching, and validator mutation +audit. Recognition remains exact-type recognition, not acceptance of any type +implementing an interface. Bindings are fixed by the facade, not configurable by +consumers, and perform no environment access themselves. Environment lookup +remains in its existing lifecycle phase and its existing private test seam. -Validation: `go test . -run 'Metadata|Path|Provenance|Historical'`, then the everyday -checker. Handoff should identify the helper and constructor signatures for Stage 2. +Engine unit tests use a test-local secret type and binding. Root integration +tests exercise the actual public `Secret`. There is no production engine-owned +replacement secret type that consumers must put in their configuration. -Completion record (2026-09-12): extracted `effectivePathIndex(final)` from final -metadata publication and introduced `newResolutionInput(metadata, final, sources)`. -The private view owns cloned metadata/source records, indexes source IDs, and exposes -private detached lookup helpers for the next step; it intentionally has no public API -or load-lifecycle wiring yet. `TestResolutionInputKeepsDetachedEntryMetadata` covers -root, defaults, explicit null, missing descendants, escaped keys, contextual parsed -sequence indices, key/index mismatches, detached records, and stability after real -recorded resolution. `go test . -run 'Metadata|Path|Provenance|Historical'` and -`./scripts/check.sh` passed with Go 1.26.7 on Linux. No deviations or remaining -Stage 1 risks were identified. +### Callback, option, result, and error bridges -## Stage 2 — Wire the callback API and enforce lifetime +Keep the engine lifecycle generic in the real `T`. Convert each supplied options +value without selecting/validating options in the facade: engine retains original +validation order and diagnostics, including multiple-options rejection. Preserve +nil callbacks, version pointers, defaults behavior, and override values without +eagerly invoking callbacks or evaluating `any` values. -Status: complete (2026-09-12). Dependencies: Stage 1. +Adapt Resolve/Validate/Version.Check by closures wrapping the exact engine context +pointer and passing the same `*T` received from engine. Convert version records and +semantic paths at those boundaries. The secret-classification predicate similarly +wraps the engine path. Do not recover, retry, close, or extend callbacks in the +facade; engine retains the existing lifetime and failure behavior. -Scope: `load_result.go`, `resolution.go`, necessary private callers, reference -comments, and resolution/callback ownership tests. +Generic public Default/Normalize forward to generic engine helper functions with +the original target pointer and value. Do not implement them as just assignment +plus Record: existing target validation and projected expected-value audit matter. +Nil wrapper handling must preserve current inactive-context behavior. -1. Build and pass the entry view at the existing pre-resolution boundary. Use - `input.sources`, which includes sources acquired by import, profile, and CLI - processing; do not reconstruct source identities from labels or field locations. -2. Implement both public methods with the existing activity guard and detached - returns. Keep lookup state frozen when recording changes or mutating configuration. -3. Add public-load tests for a file-authored scalar, a defaulted field, explicit - null, a missing path, empty/unknown source IDs, and `NoSource`. Assert exact - `InputInfo.Explicit` agreement with `IsExplicit` for found paths. -4. Exercise nil/zero contexts and retained contexts after successful and failed - callbacks. Both methods must use the existing fixed closed-context panic behavior. - Saved return records remain usable after callback exit. Do not broaden this work - into a new concurrency contract or a general callback framework refactor. -5. Add Go method documentation and update the configuration contract's resolution - section to describe the implemented entry-state semantics, misses, lifetime, - and source distinction. Fix the development guide's stale statement that this - implementation plan has not yet been written. +Refactor engine publication to return a short-lived generic loaded value containing +`Config T` and an inspection pointer. Inspection owns frozen trees, metadata, +history, sources, diagnostics, and sensitivity; it does not retain `T` or callback +bindings. Public `Result[T]` holds `Config` and that pointer, not another complete +generic engine result. Result inspection must not reread caller-mutated Config. +Existing caller-constructed zero results and nil result receivers retain behavior. -Acceptance: both methods work during an ordinary public `Load`; calls cannot access -live mutable metadata or bypass callback scope. No-rule/no-resolver behavior and -existing mutation auditing remain unchanged. +Translate engine load/bootstrap failures into root `*LoadError`/`*SecretsError`, +respectively. Their backing state can be delegated privately, but no public +Unwrap exposes internal failures. Retain bootstrap partial reports and ordinary +writer errors/short-write behavior. Do not wrap arbitrary render/write errors into +new structured load errors or add error-disclosure guarantees. -Validation: `go test . -run 'Resolution|Callback|Metadata|Load'`, then the everyday -checker. Do not mark complete with public methods that work only in private tests. +## Stage 1 — Record the boundary and establish compatibility evidence -Completion record (2026-09-12): `resolveLoadInput` now creates a -`resolutionInput` after decoded/overlay metadata and passes it explicitly into -`runResolution`; the private state retains it for the callback. `Resolution.InputInfo` -and `Resolution.Source` use the existing activity guard before exact entry lookup or -source lookup, and return detached records. Public load tests cover a canonical -file-authored value, a retained default, explicit null, misses, empty/unknown IDs, -and defaults-only input; they also cover nil, zero, successful-retained, and -failed-retained contexts plus saved record independence. Go reference comments, -the configuration contract, and the development guide now describe the active work -accurately. `go test . -run 'Resolution|Callback|Metadata|Load'` and -`./scripts/check.sh` passed with Go 1.26.7 on Linux. No deviations or remaining -Stage 2 risks were identified. +Status: not started. -## Stage 3 — Verify layered provenance and snapshot independence +1. Run baseline everyday checks. Inventory the root's exported types, constants, + fields, functions, receiver method sets, generics, and interface implementations + with Go tooling. Record the baseline revision and concise ownership mapping in + the stage handoff; do not create a second normative API specification. +2. Add the next available ADR (currently `0006-public-facade.md`) recording wrappers, + private delegation, the dependency direction, and the alias-first alternative. + Record the temporary migration mechanism here, not as durable architecture policy. +3. Update architecture policy's public API/dependency section for the agreed + boundary, preserving proportionality and existing behavior contracts. Explain + that this is accepted design, not completed implementation. Add ADR navigation. +4. Repair the three historical incoming references described at the top of this + plan to identify the old plan by Git revision. Preserve the completed provenance + evidence; do not label this facade plan as its completion record. +5. Retain/add a compact external-package facade compatibility test covering the + realistic boundary risks: root named-type identity, exported construction, + generic helper use, structured error assertions, and relevant interfaces. + Use existing tests for behavior already protected. Inspect the complete exported + surface separately; do not add golden dumps of private struct layouts. -Status: complete (2026-09-12). Dependencies: Stage 2. +Acceptance: baseline passes, public surface is understood, ADR/policy accurately +distinguish decision from implementation, and documentation links pass. +Run `go test . -run 'LocalDocumentationLinks|DependencyImportBoundary|Example|Facade'` +and the everyday check. No engine migration in this stage. -Scope: public lifecycle/provenance, import/profile/override, and ownership tests; -production fixes only for gaps exposed at those boundaries. +## Stage 2 — Establish the internal relocation candidate -1. Build a real temporary import graph with path-bearing scalars and a sequence - element. Include a composite mapping with contributions from distinct files. - During resolution, assert scalar ownership, all container contributors, and - successful source lookup for every origin location returned by the view. -2. Apply a profile replacing one scalar and an equal-text value; verify replacing - origins and untouched siblings. Exercise equal-text CLI replacement separately. - Keep source IDs derived from returned records, not inferred from value equality. -3. Exercise byte and reader roots with filesystem context and file imports: the - memory root has no file path while imported files do. Also cover non-file input - without imports, default-only fields, and unknown source IDs. Retain default - descendants reintroduced by normal decoding instead of treating them as raw - input declarations. -4. Extend shape coverage through inlined structs/maps, typed arrays, and dynamic - structures. Assert misses for invalid indices, signed index tokens in sequences, - and explicit key/index mismatches; signed mapping keys remain literal keys. -5. Normalize, delete, add, and move fields within a callback using valid records. - Queries for original paths must stay fixed; absent destinations stay absent, - and pre-existing destinations retain their entry information. Final metadata - must still reflect resolution rather than that fixed entry view. -6. Mutate returned `Origin.Locations` and `SourceInfo.ImportedFrom`; verify repeated - queries, saved records, final results, and another load are independent. Exercise - independent loads under the race detector without sharing an active context. +Status: not started. -Acceptance: roadmap criteria 2–8 and 11 are covered at their owning boundaries. -No duplicate provenance model, nearest-ancestor fallback, or source-order heuristic -is introduced. Any discovered defect is fixed before the stage is marked complete. +1. Copy root production implementation except public package orientation into + `internal/engine`, changing package declarations/import references mechanically. + Preserve file divisions and existing document/YAML packages. Keep internal + functions private unless a later facade connection actually needs an export. +2. Copy behavior/mechanism tests and needed fixtures for engine validation, excluding + repository documentation, release-script, import-boundary, public examples, + and consumer integration tests. Keep root unchanged and passing. Repair fixture + paths explicitly; do not make tests depend on arbitrary working directories. +3. Avoid blind replacements inside user-facing strings, diagnostic codes, fixture + YAML, or documentation comments. The package rename is not permission to rename + messages or behavior. Record the candidate's source baseline. +4. Update the component overview with an explicitly temporary candidate entry. + Do not claim root loads delegate yet. -Validation: `go test . -run 'Resolution|Provenance|Import|Profile|Override|Metadata|Path'`, -`go test -race ./...`, then the everyday checker. +Acceptance: `go test ./internal/engine` and the everyday check pass. Engine imports +no root code, existing internal/parser boundaries remain intact, and root consumers +still execute the unchanged original implementation. This stage is deliberately +mechanical; do not mix ownership redesign into the relocation diff. -Completion record (2026-09-12): public lifecycle tests now build real temporary -main/import/profile graphs and prove scalar, sequence, composite-container, -profile, and equal-text CLI provenance through `InputInfo`/`Source`, deriving source -IDs from returned locations. Coverage also exercises byte and reader roots with -filesystem imports, a standalone non-file root, retained defaults, unknown IDs, -typed/inlined/map/array/dynamic shape matching, signed mapping keys, and invalid -sequence shapes. Recorded normalization, deletion, addition, and move operations -leave entry queries fixed while final metadata reflects resolution. Returned records, -completed results, another load, and concurrent independent loads remain detached. -No production gap was exposed. `go test . -run -'Resolution|Provenance|Import|Profile|Override|Metadata|Path'`, `go test -race -./...`, and `./scripts/check.sh` passed with Go 1.26.7 on Linux. No deviations or -remaining Stage 3 risks were identified. +## Stage 3 — Decouple reflection from the public secret type -## Stage 4 — Demonstrate source-aware normalization and safe inspection +Status: not started. -Status: complete (2026-09-12). Dependencies: Stage 3. +1. Implement the exact-type SecretBinding/SecretState design above in engine. + Thread it through schema and recursive value operations explicitly; do not use + a global binding or make it an application option. +2. Replace concrete-secret assumptions in `schema.go`, `decode.go`, `clone.go`, + `projection.go`, `value_operation.go`, `secret_reference.go`, and their callers. + Find all additional references using the graph/text search; the list is not a + reason to leave a direct assertion behind. +3. Move engine tests to their local secret type/binding and remove the candidate's + production concrete Secret and public-style constructor. Preserve named-type + rejection/acceptance semantics and dynamic-value checks from the baseline. +4. Exercise unresolved/default secrets, pointer and container shapes, borrowed + callback values, lookup-once behavior, and validator replacements without + disclosing canary credentials in diagnostic expectations. -Scope: maintained Notarius-shaped example code/fixtures, executable examples, -explanation/secret integration tests, and consumer guidance. +Acceptance: engine secret/schema/decode/clone/projection/validation tests and race +tests pass; binding instances cannot affect each other. Run +`go test ./internal/engine -run 'Secret|Schema|Decode|Clone|Projection|Validation|Callback'`, +`go test -race ./internal/engine`, and the everyday check. -1. Add a small application-owned path-bearing configuration example in - `examples/consumers`. Keep existing catalog/version examples working. Supply the - invocation directory explicitly to the example loader rather than calling - `os.Chdir` or adding process-global state to tests. -2. Implement the example's policy using the two public queries: absolute paths - unchanged; CLI-relative paths based on the supplied invocation directory; - default-relative paths based on the canonical main source; file-authored paths - based on the scalar's declaring source. Use the existing primary-source identity - convention for main-source lookup, not a guess about contributor ordering. -3. Demonstrate file-relative normalization with fixtures that have an unambiguous - declaring file. If the example handles absent or ambiguous file provenance, - make that behavior an explicit example policy, not a correctness requirement - or prescribed Notarius policy. Rejection, a caller-supplied fallback directory, - retaining a relative value, or warning are application choices. The example's - selected behavior must be documented and tested; ConfigKit must continue to - expose the distinction rather than fabricate a file from a diagnostic label. -4. Record every changed path through `Normalize` or `Record`, including valid - dependencies where derived values depend on other configuration paths. Prove - runtime values, effective YAML, and explanation agree; retain the original - relative declaration and normalization reason in history. No existence or - confinement checks are part of the example. -5. Add a symlink fixture proving canonical target-directory semantics and an - invocation-directory mismatch proving resolution does not depend on process CWD. - Make unsupported symlink capability explicit under the testing policy. -6. Use synthetic sensitive fields to verify original/effective redaction remains - intact. With existing reader/secret lookup seams, prove queries do not reread - source data or resolve credentials during the resolver; normal later lookup - remains unchanged. Do not create credentials or environment-dependent tests. -7. Document the public workflow and canonical symlink choice in the consumer guide; - point to executable artifacts instead of duplicating large code examples. +## Stage 4 — Build candidate value, source, and diagnostic wrappers -Acceptance: roadmap criteria 1, 9, 10, and 12 are demonstrated in one-load public -workflows. Filesystem policy stays in the example application, not the root library. +Status: not started. -Validation: `go test ./examples/consumers`, -`go test . -run 'Resolution|Explain|Secret|Example'`, then the everyday checker. +1. Create temporary `internal/facade`, package `configkit`, importing engine. + Implement root-owned Path, Source, Secret, scalar enums/constants, and provenance + records according to the ownership table. Preserve declaration comments for + their eventual root placement. There must be no aliases or embedded engine types. +2. Implement private path/record conversions and the real facade secret binding. + Keep public Secret's small access/format/JSON methods local; its private scalar + state remains value-based, not a shared mutable reference. +3. Add LoadError and WriteDiagnostics delegation/conversion. Candidate engine may + export narrow constructors/accessors as needed internally, not public bridge + functions on the candidate facade. Preserve nil handling and typed errors. +4. Test candidate public operations from its external test package. Verify that + numeric-key versus index versus parsed paths are not collapsed; source copies + share the same reader claim; bytes are snapshotted at construction; formatting + and JSON do not expose resolved secret values. -Completion record (2026-09-12): added the maintained `NotariusPathConfig` example -and fixtures. Its application-owned resolver uses `InputInfo`/`Source` only: file -values use their exact declaring file, defaults use `source-1`, CLI values use the -explicit invocation directory, and absolute values remain unchanged. Missing CLI -bases or missing/composite file provenance retain the relative value and report an -application warning. Tests prove imported-file, default, CLI, absolute, and canonical -symlink-target bases without changing process CWD; the symlink test explicitly skips -when the platform cannot create it. They also prove effective/explanation agreement, -retained original declaration and reason, sensitive redaction, and that provenance -queries do not re-read a reader source or resolve credentials before normal lookup. -The consumer guide links to the executable artifact. `go test ./examples/consumers`, -`go test . -run 'Resolution|Explain|Secret|Example'`, and `./scripts/check.sh` passed -with Go 1.26.7 on Linux. No deviations or remaining Stage 4 risks were identified. +Acceptance: candidate wrapper tests and engine tests pass. Run +`go test ./internal/facade ./internal/engine` and the everyday check. Lifecycle +entry points are added in Stage 6; do not add nonfunctional stubs here. -## Stage 5 — Audit the complete contract and hand off release readiness +## Stage 5 — Separate engine publication and inspection -Status: complete (2026-09-12). Dependencies: Stage 4. +Status: not started. -Scope: integration review, canonical documentation, roadmap completion records, -and proportionate fixes for remaining acceptance failures. No feature expansion. +1. Split generic engine Result into the short-lived loaded configuration carrier + and non-generic inspection state specified above. Move inspection methods to + the inspection receiver without changing their snapshots or algorithms. +2. Adapt `load_result.go`, `load_metadata.go`, `metadata.go`, `explain.go`, + `explain_render.go`, `render.go`, `history.go`, and `snapshot.go` as necessary. + Keep the fixed entry-state view independent of final history, including the + baseline fix that avoids copying unused resolution history. +3. Update engine tests for the internal receiver/return changes while preserving + observable expectations. Do not broaden nil/result-construction contracts. -1. Check all twelve roadmap acceptance criteria against named tests and current - implementation. Record the mapping in this stage's completion entry. Passing - broad suites alone is not evidence that a missing scenario was exercised. -2. Inspect every new query path for environment/filesystem access, raw values, - caller-mutable aliases, and accidental callback-lifetime extension. Review - allocation and traversal costs for proportionality without turning the chosen - mechanism into a public guarantee or adding unsupported performance gates. - Avoid unnecessary history duplication and unused work in no-resolver loads. -3. Review public signatures and current configuration behavior against the previous - release. Confirm the capability is additive and no parser, path, composition, - credential, or existing callback semantics changed unintentionally. -4. Update the internal overview and finish canonical Go/configuration/consumer - documentation. Transfer implemented contract prose from the feature roadmap - to its proper owners and retain a brief completion record with links. Preserve - historical ADR rationale and published notes. No new ADR is required for this - information-access extension unless implementation actually changes a governing - decision; stop and obtain review rather than silently widening scope. -5. Run the full source checker with qualifying acquired toolchains, recording the - actual versions and host. The following are the repository's recorded versions; - follow the release runbook if selecting updated toolchains: +Acceptance: engine loading, historical provenance, detached metadata, explanation, +and rendering behavior pass. Verify publication still performs its existing +configuration detachment but does not store Config in inspection. Run +`go test ./internal/engine -run 'Load|Metadata|Provenance|Historical|Explain|Effective|Render|Independence'`, +`go test -race ./internal/engine`, and the everyday check. - ```sh - CONFIGKIT_BASELINE_TOOLCHAIN=go1.25.5 \ - CONFIGKIT_NEWER_TOOLCHAIN=go1.26.7 \ - ./scripts/release-check.sh - ``` +## Stage 6 — Wire candidate lifecycle and callback wrappers -6. Hand off remaining publication work separately: maintainer version choice, - dependency reassessment, version-matched release notes, completed-roadmap - retirement and link repair, exact-commit Linux/macOS validation, publication - guards, and clean consumer retrieval. Follow [the runbook](../release.md). - Historical one-release exceptions do not carry forward to this feature. +Status: not started. -Acceptance: all target requirements have concrete evidence; available automated -checks pass; documentation distinguishes implemented behavior from release status. -Implementation completion is not publication approval. Do not create a release -tag or mark unavailable native macOS evidence as passed. +1. Add candidate Definition, Options, VersionRule, Version, FilesystemContext, + Override, Resolution, and Validation with unchanged public declarations. + Implement the private option and callback bridges specified above. +2. Add candidate Load using the engine and facade secret binding. Add Result[T] + with Config and private inspection, initially with Metadata, Sources, and + Diagnostics. Implement Metadata's complete public query methods. +3. Forward all resolution methods, including InputInfo and Source, and the generic + Default/Normalize helpers. Engine owns context activity, helper verification, + resolution snapshots, and validation mutation detection. +4. Add focused external candidate tests using actual public wrapper types inside + application configuration. Exercise version callbacks, defaulting/normalization, + retained/copy contexts, root typed error assertions, and public Secret recognition. + Private engine secret tests alone are insufficient evidence for this bridge. -Validation: `./scripts/check.sh`, `go test -race ./...`, and the full checker above. +Acceptance: real candidate loads with callbacks succeed/fail identically to the +baseline, no callback is called by option conversion, no extra lookup occurs, +and detached entry provenance stays fixed. Run +`go test ./internal/facade ./internal/engine -run 'Load|Resolution|Validation|Version|Metadata|Secret|Callback|Facade'`, +`go test -race ./internal/facade ./internal/engine`, and the everyday check. -Completion record (2026-09-12): all acceptance criteria have named evidence: +## Stage 7 — Complete candidate inspection and bootstrap surface -1. `TestNotariusPathExampleUsesEntryProvenance` covers canonical main-file and - symlink target bases without process-CWD dependence. -2. `TestResolutionInputKeepsLayeredOriginsAndSources` covers imported scalars and - sequence elements. -3. The same layered test covers profile replacement and untouched base/import - siblings. -4. The layered test covers equal-text profile and CLI replacement. -5. `TestResolutionEntryQueriesExposeDetachedLoadInput` and - `TestResolutionInputDistinguishesNonFileRootsAndImportedFiles` cover defaults, - nulls, misses, CLI/file identities, `NoSource`, bytes, and readers. -6. The layered test verifies every composite location has a source; - `TestResolutionEntryQueriesExposeDetachedLoadInput` verifies unknown IDs miss - safely. -7. `TestResolutionInputRemainsFixedAcrossRecordedMutations` covers normalization, - deletion, addition, and move stability. -8. `TestResolutionInputUsesEntryShapesAcrossTypedAndDynamicValues` covers inline, - map, array, dynamic, escaped/signed keys, and invalid shape cases. -9. `TestNotariusPathExampleUsesEntryProvenance` verifies runtime, effective YAML, - and explanation history together. -10. `TestResolutionInputQueriesDoNotReadOrResolveSecrets` plus the Notarius test - verify no extra reader/credential access and preserved redaction. -11. `TestResolutionInputKeepsLayeredOriginsAndSources` and - `TestResolutionEntryQueriesExposeDetachedLoadInput` verify detached records. -12. `NotariusPathConfig`, its fixtures, executable example, and consumer-guide - link provide the maintained public-API/standard-library workflow. +Status: not started. -The final review found that queries use only the callback guard and detached -metadata/source indexes; they do not extend callback lifetime, expose raw values, or -perform environment/filesystem work. The view is created only for resolver loads and -now retains only effective field/path metadata and source records. Post-implementation -review found that the original constructor unnecessarily cloned history; remediation -replaced that clone with detached field-only construction, leaving final result -history unchanged. The comparison with `v0.2.0` found only additive exported -methods; loading, parser, path, composition, credential, and existing callback -semantics remain unchanged. Canonical Go, configuration, consumer, internal, and -roadmap documentation are synchronized; the feature roadmap is retained only until -release preparation. `./scripts/check.sh`, `go test -race ./...`, documentation and -import-boundary checks, and `CONFIGKIT_BASELINE_TOOLCHAIN=go1.25.5 -CONFIGKIT_NEWER_TOOLCHAIN=go1.26.7 ./scripts/release-check.sh` passed on Linux with -Go 1.25.5 and Go 1.26.7. Publication remains unapproved: a maintainer must choose a -pre-v1 minor version, write its note, retire completed roadmaps, obtain exact -candidate native Linux/macOS evidence, and follow the release procedure. No version, -release note, candidate, tag, or push was created. +1. Add the public explanation/value records and Result Explain, WriteEffective, + and WriteExplanation delegation. Inspection uses the frozen engine state, + not the mutable public Config field. Complete every remaining public method. +2. Add SecretsReport, SecretsError, bootstrap records/status constants, and + LoadSecretsDir. Preserve opt-in writes, partial reports, detached entries, + writer counts/errors, and non-unwrapping typed errors. +3. Exercise complete load-to-inspection workflows through the candidate. Prove + that caller mutation of Config or returned records does not rewrite inspection + history or affect a later query/load. Include malformed paths and nil receivers + where the baseline already defines behavior. +4. Compare the complete candidate exported surface with the baseline inventory, + normalizing only the temporary import path. Resolve all signature/method/field + omissions before cutover. No placeholder methods or ignored API mismatches. + +Acceptance: the candidate implements the entire supported API and has real +integration evidence for inspection, credentials, bootstrap, and writer failures. +Run `go test ./internal/facade ./internal/engine`, race tests for both packages, +and the everyday check. Root remains unchanged until Stage 8. + +## Stage 8 — Cut over the root and remove migration duplication + +Status: not started. + +1. Replace root production plumbing with the tested candidate facade, retaining + root package documentation and the root named types. Move files; do not leave + forwarding aliases to the temporary facade. Remove `internal/facade` entirely. +2. Root-owned unit/integration tests remain at root; engine mechanism tests remain + with engine. Remove duplicated baseline copies after confirming coverage owners. + Split mixed tests by boundary: assertions requiring private engine state move + internally, while public lifecycle/consumer assertions still exercise root. + Do not move all tests inward simply to avoid adapting public wrappers. +3. Run existing executable API examples and maintained consumers unchanged in + their imports/configuration types. Fix wrapper defects rather than migrating + consumers to a different API. Test the actual root Secret binding after the move. +4. Keep repository check tests at root. Repair documentation/fixture references, + fuzz target locations, and development task commands as files move. For links + formerly pointing at mixed implementation files, point API references at root + and implementation references at engine as appropriate. +5. Remove old root algorithms, temporary candidate tests superseded by root tests, + and staging documentation. Preserve useful narrow private test seams, but no + root accessors solely exposing engine internals to tests. + +Acceptance: one engine serves the sole root facade; no candidate directory, +parallel implementation, or root import from internal production code remains. +Run the everyday check, `go test ./examples/consumers`, root examples, and +`go test -race ./...`. All gates must pass in this handoff; do not leave a broken +root suite for the final audit. + +## Stage 9 — Audit the public boundary and finalize documentation + +Status: not started. + +1. Compare the final complete exported API against the baseline. Audit public + signatures, fields, aliases, embedded types, method sets, and error returns + for internal-type exposure. Review root code for remaining substantive + algorithms; explain small value operations deliberately retained there. +2. Extend the existing import-boundary check for inward production dependencies. + Add a narrowly scoped public-boundary check that detects exported aliases to, + embedded exposure of, and exported signatures/fields involving internal types. + Use standard Go tooling and avoid private-field/layout snapshots. Account for + generic declarations and public records referencing other public records. + Keep behavior tests as the evidence for adapter correctness. +3. Verify source ownership, secret binding isolation, zero/nil handling, helper + audit semantics, typed errors, final Config versus frozen inspection, and the + baseline resolution-entry optimization. Add tests only for uncovered risks. +4. Finalize architecture-policy wording, ADR navigation, component overview, + development test/fuzz commands, and public declaration comments. Add a concise + internal facade mechanism document if needed for the secret/callback/record + connections; do not spread implementation recipes across consumer docs. +5. Update consumer orientation only where useful. Keep YAML behavior in its + existing contract. Convert the feature roadmap to an implemented summary with + canonical links and evidence; record remaining release preparation separately. + +Acceptance: documented target matches code and no unapproved API/behavior change +is hidden by the refactor. Run the everyday check, full race suite, consumer +examples, documentation/import/public-boundary checks, and tests/builds using the +supported baseline and newer Go toolchains recorded in the release procedure. +Record actual versions/platforms; do not treat Linux tests or cross-builds as +native macOS release evidence. Do not publish or select a version in this stage. ## Open Questions -None blocking implementation. The API, snapshot boundary, missing-value semantics, -source identity handling, and ownership are specified by the roadmap and decisions -above. Example fallback behavior is illustrative application policy, not a missing -library contract; record any selected example behavior without prescribing it to -consumers. Release version and publication timing are later -maintainer decisions, not unresolved implementation choices. If new evidence -contradicts a target contract, document the conflict, a recommended correction and -its rationale, and at least one viable alternative before proceeding past it. +None requiring a user decision were identified. Wrappers, public type ownership, +one final engine, per-operation exact-type secret adaptation, and a single cutover +are specified above. Internal helper names and file grouping remain ordinary +implementation choices. A newly discovered conflict with an existing contract +must be surfaced with evidence and alternatives rather than resolved by changing +consumer behavior or reverting to public aliases. diff --git a/docs/roadmap/public.md b/docs/roadmap/public.md new file mode 100644 index 0000000..b1312b0 --- /dev/null +++ b/docs/roadmap/public.md @@ -0,0 +1,187 @@ +# Public facade and internal implementation + +Status: agreed direction; not implemented. This roadmap defines the desired end +state and user intent. The [implementation plan](implementation.md) owns the +ordered work, concrete adapter design, validation commands, and handoffs. + +## Intent + +Make ConfigKit easier for humans and coding agents to navigate by separating its +consumer-facing API from its implementation machinery. Consumers should find a +readable, documented public surface in the root `configkit` package; maintainers +should find the loading and inspection machinery under `internal/`. + +Long-term maintainability is the primary objective. The public API should remain +stable while internal implementation can evolve without becoming a consumer +commitment. An internal implementation change must not expand or alter the public +API merely because the facade exposes the same underlying type. Repository +aesthetics and easier navigation are useful secondary benefits, not the reason +to minimize public-package code. + +This is an organizational and API-boundary refactor, not a new configuration +feature or security boundary. Unexported root-package details are already private +under ordinary Go access rules. The concern is accidentally exposing internal +types or exported methods that consumers can depend on even without documentation. +Wrappers provide structural separation; they do not make internal changes that +alter observable public behavior automatically compatible. + +The [architecture policy](../policy/architecture.md) and +[implementation discipline](../../AGENTS.md#implementation-discipline) govern the +design. Existing [configuration contracts](../config.md) and exported Go +declaration comments remain authoritative. The user-approved wrapper boundary is +a contributor design choice, not a strengthening of consumer behavior contracts. + +## Current architecture + +The root currently owns both public declarations and most private implementation: +source acquisition, imports, composition and overlays, reflection descriptors, +decoding, resolution and validation coordination, provenance, history, secret +handling, and rendering. The [component overview](../internal/overview.md) owns +the detailed inventory. + +Two established internal boundaries already exist: `internal/document` owns the +parser-independent tree, and `internal/yaml` owns upstream YAML integration. +[ADR 0002](../adr/0002-own-yaml-semantics.md) remains applicable. + +Public types are not merely passive records. For example, `Result[T]` combines +its public `Config` field with private inspection snapshots; `Resolution` holds +callback state; `Path` holds semantic segments; and `Secret` holds private +reference and resolved-value state. Moving implementation requires deliberate +type ownership, not just file relocation. + +## Target architecture + +### Public root package + +The root remains the single supported consumer import path. It contains: + +- package and API reference documentation; +- public-owned type declarations, including wrappers and ordinary public records; +- public constants, constructors, entry points, and lightweight delegation; +- public contract tests and executable API examples where appropriate. + +Substantial source processing, decoding, lifecycle bookkeeping, history replay, +and rendering mechanics live internally. Small value operations or adapters may +remain in the root when moving them would introduce disproportionate machinery. +Public API documentation remains beside public declarations and must be useful +through ordinary Go documentation and editor navigation. + +### Public-owned types and private delegation + +The agreed approach is root-defined public types, with private delegation to +internal implementation where needed. Public wrappers expose deliberately chosen +methods; internal state is held privately rather than exported or embedded in a +way that promotes internal methods into the public surface. Public signatures +and fields do not expose internal plumbing types. Adding an internal exported +method must not add a public method automatically. + +Simple public records and small value operations can remain directly implemented +in the root. There is no requirement to wrap every value or move every method. +Aliases to internally owned implementation types are not the facade strategy: +they would couple the public method set to the implementation this work is meant +to separate. + +Straightforward forwarding code is an acceptable maintenance cost. Adapters must +avoid competing mutable state and unnecessary conversions, particularly around +`Result[T].Config`, frozen inspection data, callback lifetime, source ownership, +and reflection-sensitive secret handling. These connections are private +implementation details, not public bridge APIs or a generic registration framework. + +### Internal engine and supporting packages + +A cohesive internal engine owns the loading lifecycle and the closely coupled +schema, decoding, resolution, provenance, and inspection implementation. The +selected package is `internal/engine`. + +The existing document and YAML packages retain their responsibilities. Additional +packages are appropriate only where a cohesive responsibility and practical +dependency boundary justify them. There is no requirement for one package per +phase, file category, or component in the current inventory. + +Dependencies run from the facade into internal implementation and from the +engine into its supporting packages. Internal packages do not import the public +root. Public types remain root-owned; internal collaborators use internally owned +state and narrow implementation connections. Internal code does not gain direct +access to private fields of public types across packages. + +The engine may initially remain fairly large. A smaller root alone does not +justify a network of adapters, generic provider interfaces, or a framework. +Internal exports needed between packages are implementation surfaces, not new +supported consumer APIs. Existing parser isolation remains intact. + +## Compatibility and scope + +The target is a behavior-preserving refactor requiring no consumer migration: + +- Preserve the module/import path, exported names, function and method + signatures, generic use, public fields, and supported construction patterns. +- Preserve loading order, source ownership, defaults, composition and overlay + behavior, resolution and validation semantics, and structured diagnostics. +- Preserve callback lifetime, fixed entry provenance, detached query results, + final inspection snapshots, and independence across loads. +- Preserve secret recognition, credential timing, explicit access, redaction, + and opt-in bootstrap behavior. +- Preserve the supported Go and platform baseline and the existing dependency + isolation; this work does not require a toolchain or third-party dependency + change. + +Type restructuring can affect more than compilation. The implementation must +assess method sets, interface satisfaction, zero values, copying and equality +where supported, error assertions, reflection-based special-type recognition, +and public documentation. Keeping public named types in the root preserves their +defining package identity, but does not by itself prove compatibility of their +state or behavior. Observable effects must be evaluated and reported, not silently +declared compatible or newly guaranteed forever. + +No new input restrictions, resource limits, lifecycle restrictions, security +guarantees, configuration semantics, or public extension hooks are in scope. +Do not expose plumbing publicly merely to solve an import cycle. A discovered +need to change an established contract requires a separate explicit decision. +Version selection and publication remain governed by the +[release policy](../policy/release.md), not by this roadmap. + +## Completion criteria + +Completion means the root is a legible public surface, substantive plumbing is +internally owned, consumers require no migration, and representative +load-to-inspection workflows still pass through the facade. Public exported +members are deliberate declarations, not incidental exposure of engine methods +or state. The completed architecture has one engine, no parallel implementation, +and no competing mutable configuration state. Small operations deliberately +retained in the root are justified by maintainability, not a numerical size gate. + +Public behavior and maintained consumer examples exercise the root facade; +mechanism tests live with their implementation owners. Test-only access does not +expand the public API. Repository checks preserve parser isolation and the public +boundary. Validation follows the [testing policy](../policy/testing.md); +release-candidate validation remains separate under the +[release procedure](../release.md). + +Tests should protect behavior and intentional dependency boundaries, not exact +private layouts, forwarding call sequences, or an arbitrary package count. +Implementation completion must be backed by evidence; this proposal is not +evidence that the target architecture already exists. Completed roadmap material +is retired during release preparation under the documentation policy. + +## Documentation ownership in the target architecture + +Under the [documentation policy](../policy/documentation.md), architecture policy +owns the public-wrapper and inward-dependency rules, preserving proportionality. +An ADR records the accepted decision and why an alias-first facade was rejected: +it reduces forwarding but couples public methods to internal implementation. +The additional wrapper code is justified by long-term API independence. + +The component overview describes actual facade, engine, supporting packages, and +state ownership. Development navigation locates the tests and mechanisms at their +new owners. Internal documentation explains the private adapters. Exported API +reference comments remain in the public package; consumer guidance identifies +the supported root import without requiring knowledge of the engine. + +Configuration semantics remain in the existing contract, not in package-layout +documentation. Internal layouts do not become public stability guarantees. +Release documentation describes the behavior-preserving refactor without +replacing canonical API or architecture documentation. + +The alias-versus-wrapper decision is resolved by the user's explicit selection +of wrappers. An implementation convenience is not authority to change that +decision or an established consumer contract.