diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..0cc3d94 --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,378 @@ +# Configured secrets bootstrap implementation plan + +## Scope and execution + +Implement the [secrets-directory roadmap](secrets-directory.md) and accepted +[ADR 0023](../adr/0023-explicit-configured-secrets-bootstrap.md). The starting +revision is `fb0560d`. Read [development](../development.md), the +[agent policy](../policy/agents.md), and the applicable architecture, testing, +and documentation policies before implementing a stage. + +The stages run sequentially in one sprint. Intermediate stages may leave callers +temporarily uncompilable while an internal handoff changes. Finish the shared +integration before final validation. Keep tests with their behavior's owning +stage and reuse existing fixtures and failure seams. Each stage is intended to +fit one Luna implementation prompt. + +The completed feature provides explicit bootstrap before ordinary configuration +loading. Discovery and the captured main document are shared between those two +calls. Ordinary secret resolution continues to read the environment at its +existing lifecycle point. Filesystem access and bootstrap finish synchronously. + +## Decisions for implementation + +### Public API + +Keep the existing function and its exact function type. Add these root-owned +public declarations, following the existing facade-to-engine conversion pattern. + +```go +func LoadSecretsIntoEnvironment(directory string) (*SecretsReport, error) + +type SecretsOptions struct { + Overwrite bool +} + +func LoadSecretsIntoEnvironmentWithOptions( + directory string, options SecretsOptions, +) (*SecretsReport, error) + +type SecretsFromConfigOptions struct { + Filesystem *FilesystemContext + Overwrite *bool +} + +func LoadSecretsIntoEnvironmentFromConfig( + main Source, supplied ...SecretsFromConfigOptions, +) (*SecretsReport, Source, error) +``` + +The companion accepts zero or one options value, matching the load entry points. +Multiple values produce `LoadError` with `CodeOptionsInvalid`, an empty non-nil +report, and zero `Source` before acquisition or writes. Snapshot the supplied +context and Boolean value at call entry using existing conversion conventions. +Nil overwrite uses YAML, then false. A supplied false or true wins over YAML. + +The companion accepts host `File` and `SearchConfig` sources and `NoSource`. +A prepared host source can also be supplied to an explicit companion call using +its captured main. Reject other source kinds and native filesystem contexts as +`LoadError` option failures before consuming them. Preserve constructor and search +validation diagnostics through the existing helpers. NoSource and optional search +exhaustion succeed with an empty report and `NoSource()`. Explicit context on an +absent input follows existing absent-host-context validation; subsequent absent +loading uses the ordinary caller-supplied context rules. + +Every failure returns a non-nil report and zero `Source`. Discovery, acquisition, +parse, control, and preparation-context failures use `LoadError`. Directory +resolution/inspection, credential reads, and environment writes use +`SecretsError`. Keep both error wrappers and their existing safe rendering and +detached records. Successful preparation with no directory returns the prepared +main and an empty report. + +### Controls and source roles + +Recognize root `config.secrets_dir` as a nonempty string containing a relative +host path and root `config.secrets_overwrite` as a Boolean. Accept ordinary YAML +Boolean spellings through the parser's existing type handling. Nulls, wrong +types, empty directory strings, absolute paths, and host volume-qualified paths +are invalid. Treat path text literally using host path operations. Parent-relative +paths are usable when their canonical targets stay inside the selected root. + +Only the main input's document root owns these controls. Imports, underlays, +and selected overlays containing them produce a source-local control diagnostic. +Nested application `config` mappings and root application data introduced by +placement, defaults, CLI, or resolution retain their existing meaning under +[ADR 0020](../adr/0020-document-root-controls.md). + +Normal loading checks and consumes the main controls, with imports enabled or +disabled. Filesystem checks for the declared secrets directory happen in the +companion helper. Both ordinary loading entry points perform their usual secret +lookup. Preparation validates the root control mapping and the bootstrap fields, +while other control grammars and import enablement are checked by the later load. +Retain the original main tree so imports and collection declarations are still +available to that load. A valid overwrite declaration without a directory is a +no-op. Invalid bootstrap fields fail even if a Go overwrite choice is supplied. + +Use existing diagnostic codes. Use `CodeSecretsInvalidPath` for invalid directory +syntax, `CodeSecretsInvalidValue` for a non-Boolean overwrite setting, and the +existing directive-context codes for misplaced controls (`CodeImportInvalid` or +`CodeOverlayDirective`). Root mapping errors keep their current classification. +Bootstrap-field diagnostics carry `/config/secrets_dir` or +`/config/secrets_overwrite` and source-local locations, inside a `LoadError`. +Avoid raw input, parser errors, or credential contents in messages. + +### Captured-source ownership and context + +Represent preparation privately as a captured host-file payload attached to an +engine `sourceFile`. Keep public `Source` opaque and report `SourceKindFile`. +Capture the original parsed document, original byte count, canonical main path, +canonical base, and canonical containment root. Retain the unconsumed controls. +Keep captured data read-only and give each load an owned document clone with +fresh source IDs and accounting. Namespace tokens and acquisition state belong +to each load. Retain passive path facts with the source. + +Normal acquisition consumes the captured main without discovery, file reading, +or bootstrap. Apply the usual byte, document, placement, and aggregate accounting +to the captured input in every load. A preparation-time count never substitutes +for load-wide admission. Imports and resources continue to be acquired when their +normal operations run. Main deletion or modification does not change captured +contents; later filesystem operations may fail under their existing contracts. + +Use the captured context when later loading supplies none. Resolve an explicit +later host context against the captured canonical main path. It is compatible +when its canonical root, base, and main path match the captured facts. Compare +path facts using the existing host path rules, allowing equivalent spellings +and root symlink aliases. Native contexts, a supplied file BaseDir, and genuinely +different roots fail with `CodeOptionsInvalid` in `LoadError`. Avoid re-resolving +the original main filename to compare contexts, since it may now point elsewhere. +Use the same check for ordinary options and layered main input contexts before +import or overlay handling can substitute a context. + +### Directory bootstrap semantics + +Reuse `loadSecretsDir` and its environment/access seams for all three public +helpers. Preservation mode retains both existing-variable checks, including +empty values and a variable created between preflight and application. +Overwrite mode reads existing names' eligible files and applies them regardless +of environment presence. Unrelated environment variables retain their values. +Use the existing loaded/skipped/failed/not-applied statuses. `ExistingEmpty` +continues to describe skipped existing values. + +Configured directory resolution uses the captured base and root. Canonicalize +the selected directory with the existing filesystem helpers and check containment +of that canonical target before reads or writes. A directory symlink can resolve +to a contained target, following the existing bootstrap directory convention. +Return `SecretsError` with `CodeSecretsInvalidPath` for an outside-root target; +use existing filesystem classifications for missing or unreadable targets. +The direct-directory helpers keep their existing path resolution behavior. + +Bootstrap includes all eligible directory entries, independently of the later +typed schema. Reuse dot-entry exclusion, regular-file checks, limits, sorted +order, newline handling, empty-file handling, and NUL rejection. Complete +metadata checks and required reads before writes. A write failure retains prior +writes and reports remaining entries as not applied. Later load failures leave +bootstrap effects in place. Reports and prepared sources retain no credential +bytes from secret files. Applications own startup ordering and concurrent +process-environment mutation under the existing bootstrap contract. + +## Stage 1 — Extend shared directory bootstrap with overwrite + +Work in `internal/engine/secrets.go` and `internal/engine/secrets_test.go`. + +1. Add a small engine options value for overwrite and keep `LoadSecretsDir` + delegating with preservation enabled. Add its options-bearing counterpart. + Extend the shared private worker and existing test seams accordingly. +2. Make overwrite affect only staging and applying values. Preserve sorted + enumeration, all metadata validation before reads, bounded reads before + writes, skipped-file behavior in preservation mode, and failure reporting. +3. Add focused behavioral cases for overwriting nonempty and empty existing + variables, overwriting with empty file content, and leaving names without + files alone. Extend the existing staged-presence-race and partial-apply-failure + cases to cover overwrite. Reuse current read-limit, dot-file, NUL, and safe + error coverage. Process-mutating tests restore state and run serially. + +Stage completion means both policies work through one bootstrap worker and +existing preservation behavior remains covered. Run the relevant existing and +new `LoadSecretsDir` tests once callers compile. + +## Stage 2 — Add main-scoped bootstrap control parsing + +Work primarily in `internal/engine/directives.go`, `acquisition.go`, and the +directive/version parsing handoffs in `source_version.go`. + +1. Introduce a small private bootstrap-settings value with declaration presence, + directory text, overwrite value, and source-local diagnostic locations. Share + its field validation between preparation and normal control extraction. +2. Give extraction enough source-role information to allow these controls only + on the actual main document. `rootDirectiveContextSource` and + `rootDirectiveContextLayered` alone cannot identify that role because those + contexts also occur on contributions. Carry the role through the relevant + acquisition/version and import traversal call sites. +3. Keep bootstrap-only extraction usable before normal imports/sequence grammar + validation. The companion can prepare a main that declares imports without + enabling or acquiring them. Normal extraction checks the complete control + block and removes it before placement and application decoding. +4. Extend directive-scope and facade-control tests with main declarations, + disabled imports, wrong types, explicit false, misplaced controls, and nested + ordinary data. Verify source-version control-overlap checks recognize the + new fields and that version acceptance itself stays in the load. + +Stage completion means the normal load accepts and consumes valid main settings +while leaving environment state untouched. Other source-root controls retain +their existing grammar, enablement, and error precedence. + +## Stage 3 — Share host-main selection and capture + +Work in `internal/engine/source.go`, `parse.go`, `discovery.go`, and the initial +selection/read portion of `acquisition.go`. A focused `prepared_source.go` may +hold the new private payload and preparation helpers. + +1. Factor the existing selection and bounded main parsing handoff for reuse by + normal acquisition and preparation. Reuse `resolveSearchInput`, search option + checks, `readSource`, and YAML adapter diagnostics. Keep candidate ordering, + optional exhaustion, selected-file failure, and regular-file semantics. +2. Add the captured host payload specified above. Preserve original source line + locations and all controls. Construct independent documents for subsequent + loads with `document.Clone` and `setDocumentSourceID`. Count captured bytes + under the current read limits before returning a document to acquisition. +3. For capture, resolve and validate the host base/root through existing + `filesystemOptionDiagnostics`, `resolveFilesystemContext`, and directory + validation. Store path facts without access handles or a preparation token. +4. Add internal cases proving captured data stays fixed after the original file + changes, becomes absent, or another search candidate appears. Retain ordinary + File/SearchConfig reread behavior. Cover parser/read failures and limits with + existing narrow test fixtures. Keep sources and payloads private until the + public wrapper stage. + +Stage completion means the acquisition front end can consume an ordinary host +file or its captured payload through the same parsed-source handoff. + +## Stage 4 — Integrate captured contexts with ordinary loading + +Work in `options.go`, `filesystem_context.go`, `filesystem.go`, `acquisition.go`, +`load_input.go`, `selected_overlay.go`, and `source_context.go` as needed. + +1. Make context resolution select captured path facts and enforce the explicit + context equivalence rule at the shared acquisition boundary. Ensure both + imported and standalone branches receive the selected resolved context. +2. Carry the captured base/root into standalone main acquisition, ordinary + overlay resolution, and passive source-context completion. Ordinary overlays + keep their established root-relative lookup convention. +3. Preserve file-kind records, canonical paths, source-local diagnostics, and + placement projection. Reuse normal source registration and accounting. +4. Add focused internal cases for omitted/equivalent/incompatible later contexts, + explicit wider roots, a symlinked selected main, main removal, destinations, + and ordinary overlay/resource-context behavior. Compare relevant records and + values with a normal load of the same unmodified file. + +Stage completion means an ordinary load uses captured contents and context +through all its normal paths. Real filesystem failures are handled at the +existing access boundaries without retaining an open handle in the source. + +## Stage 5 — Integrate imports and layered main-relative consumers + +Work in `import_acquisition.go`, `load_request.go`, and layered/context owners +identified by the shared acquisition handoff. + +1. Feed the captured main document and resolved context into imported acquisition. + Charge its raw bytes and nodes through existing accounting exactly once per + load. Keep placement wrapper accounting and per-tier repetition tracking. +2. Treat the captured main as the same canonical host file for repeated-file and + cycle checks. A fresh load gets its own host namespace and source IDs. +3. Ensure layered acquisition propagates the captured main context to declared + MainRelative underlays and selected overlays, with existing explicit context + and fallback precedence. Preserve main-first acquisition and tier precedence. +4. Verify live imports after main capture, repeat/cycle detection involving the + captured main, source/version evidence with imports, an explicitly wider root + permitting contained parent-relative imports, and independent repeated loads. + Include one integrated layered case with a main-relative underlay and overlay. + Use existing import/placement/version suites for unchanged semantics. + +Stage completion means prepared main sources behave as host main inputs across +ordinary and layered composition, metadata, and main-relative bindings. + +## Stage 6 — Implement configured bootstrap orchestration + +Add a focused engine companion in `secrets_from_config.go`, using the capture +and bootstrap helpers from earlier stages. + +1. Validate and snapshot options, resolve the main, capture its document and + context, and extract bootstrap settings. Return the no-directory and absent + cases under the public return contract. A valid Go overwrite override does + not bypass validation of malformed YAML bootstrap fields. +2. Join the declared directory to the captured base. Add an optional internal + containment-root constraint to shared directory preparation so canonical + resolution and containment validation occur before the worker uses that + canonical directory. Direct-directory calls supply no such constraint. +3. Select overwrite using Go, YAML, then false, and invoke the shared worker. + Return the prepared source only after successful bootstrap. All branches + return the non-nil report and appropriate existing error family. +4. Test no declaration, overwrite without a directory, optional/required missing + main, unsupported source/context, malformed controls, missing or outside-root + directory, and directory symlink containment. Assert empty/partial reports and + zero prepared sources on errors. Use the existing environment seam for + deterministic apply failures and verify safe error/report output. + +Stage completion means configured bootstrap can finish independently of schema, +imports, version hooks, resolution, and validation, returning a reusable main +only when its own operation succeeds. + +## Stage 7 — Expose root-owned APIs and convert both error families + +Work in root `secrets.go`, `source.go`, and existing facade conversion owners. + +1. Add the two options types and new functions with the signatures above. Keep + the existing function type intact and route its default behavior through the + shared bootstrap implementation. +2. Convert the companion's report, prepared engine Source, and errors to the + existing public wrappers. Preserve `LoadError` source records as well as + diagnostics. Preserve `SecretsError` without exposing underlying filesystem + or environment errors. Add only the small conversion logic this boundary + needs, reusing existing converters. +3. Document signatures, options precedence, reports, error types, and source + ownership on their declarations. Explain captured reuse versus File/SearchConfig + reuse in Source's documentation. Keep implementation types private. +4. Add facade tests for both new functions and assignment of the old function to + `func(string) (*SecretsReport, error)`. Exercise YAML true with explicit Go + false, YAML false with Go true, omitted choices, and multiple options. Verify + public `errors.As` behavior for preparation and bootstrap failures. + +Stage completion means applications can use the complete two-call startup flow +through ConfigKit's public package with the documented ownership and errors. + +## Stage 8 — Verify the complete startup handoff + +Extend public integration coverage and add one maintained consumer startup +example with ordinary and layered use explained from the same helper. + +1. Start in a different working directory, discover a main with relative secrets, + bootstrap all eligible entries including an SDK-only name, and load the + prepared source. Verify configured overwrite and existing empty-variable + behavior with disposable environment names. +2. Change the main, discovery candidates, an import, and an environment value + after preparation. Verify captured main values, live import values, fresh + environment lookup, and independent metadata/results across subsequent loads. + Include selected-main provenance and source resource context assertions. +3. Exercise the layered handoff with explicit root preparation and companion + inputs. Confirm compatible later context succeeds and incompatible context + fails safely after the already-completed bootstrap. +4. Show a later version/decoding or validation failure preserves the bootstrap + report and environment effects. Separately verify ordinary Load/LoadLayered + on a main containing the controls perform no environment writes, even with + YAML overwrite true. Existing secret-resolution tests continue to own + credential lookup and redaction behavior. +5. Verify captured sources retain no handles through successful and failed + preparation. Use existing resource-lifetime techniques and narrow seams, + avoiding a new filesystem framework. Exercise independent captured-source + loads under the race detector without concurrent process mutation. + +Stage completion means the feature works through normal entry points and its +integration risks are covered without duplicating every internal test at the +facade boundary. + +## Stage 9 — Update canonical documentation and validate the sprint + +1. Update `docs/config.md` with the two main controls, preparation and bootstrap + boundaries, lookup timing, overwrite precedence, reports/errors, host scope, + and prepared-source reuse/context behavior. Keep source-local controls + distinct from application paths and preserve existing loading semantics. +2. Update `docs/consumers/README.md` and its test map to link the new maintained + example. Update `docs/development.md` only where navigation/test ownership + changes. Explain internal capture and acquisition handoffs in + `docs/internal/overview.md` and relevant import documentation. +3. Update the implemented-behavior paragraphs in architecture policy and retire + its roadmap implementation-status link. Keep ADR 0023's accepted decision + text intact. Preserve the existing bootstrap limits and concurrency/rollback + contract while documenting explicit overwrite as the selected extension. +4. Format changed Go files and run the focused tests for changed owners. Run + `./scripts/check.sh` and `go test -race ./...` on the completed implementation. + Verify edited Markdown links and headings, exported declaration comments, + and the public example. Reuse successful checks for unchanged content. +5. Review the final diff for captured-state aliasing, handles, extra parsing or + discovery, secret disclosure, omitted accounting, and accidental changes to + current configuration processing or helper signatures. Resolve failures and + finish the documented target before marking the sprint complete. + +The final result is the implemented feature with current documentation and +appropriate behavioral coverage. Release preparation follows its separate +procedure when requested. diff --git a/docs/roadmap/secrets-directory.md b/docs/roadmap/secrets-directory.md index 29560db..995a4f1 100644 --- a/docs/roadmap/secrets-directory.md +++ b/docs/roadmap/secrets-directory.md @@ -191,36 +191,19 @@ main-root options. This covers the requested deployment pattern and reuses the native secret-file checks. Caller-supplied filesystem support is a possible later extension requiring decisions about file-type and symlink capabilities. -## Work needed +## Target integration -- Add `config.secrets_dir` and `config.secrets_overwrite` to the main-document - control grammar and consume them during ordinary loading without triggering - bootstrap. Retain source-local positions for diagnostics. -- Factor main-file selection and capture for use by the companion helper and - normal acquisition. Add a prepared-source handoff with retained main contents, - identity, context, and ownership. Supply the preparation filesystem context and - optional Go overwrite choice as described above. Preserve per-load accounting, - diagnose incompatible later contexts, and implement the failure/return contract. -- Preserve the current direct-directory signature and add a separately named - options-bearing helper. Reuse directory bootstrap from all three entry points, - with matching reports and failures. Finalize public names during implementation. -- Verify discovery from a different working directory, canonical main paths, - optional search exhaustion, declaration scope, default and explicit - overwrite choices, Go-over-YAML precedence, and all helper interfaces. Include - assignment of the unchanged helper to its existing function type. - Cover empty values, dot entries, read failures before writes, and partial - write failures. Verify bootstrap includes entries unused by typed fields and - that later load failures preserve bootstrap effects. -- Verify error types, empty and partial reports, and unusable prepared sources on - failure. Cover absent-main success and failed later loads without rollback. -- Verify prepared-source reuse after main-file changes and discovery changes, - live later imports and environment lookup, equivalent and incompatible later - contexts, handle cleanup, normal loading without - environment writes, and agreement with selected-file provenance, placement, - relative imports, and main-relative companions. Use disposable credential - values and deterministic failure seams in line with the - [testing policy](../policy/testing.md). -- Document the startup workflow and add a consumer example. Implement the - ownership decision in ADR 0023 and update the canonical configuration contract, - public source documentation, and affected component documentation when the - behavior lands. Retire the roadmap reference in architecture policy at that time. +Main-file selection and bounded capture are shared with normal acquisition. +The prepared source carries the main contents, identity, and context through +ordinary and layered loading with normal placement, accounting, and source +records. Directory bootstrap has one implementation behind the existing helper, +the options-bearing helper, and the configured companion. + +Canonical configuration and public source documentation describe the two-call +startup flow, capture lifetime, and overwrite policy. Consumer examples and +behavioral tests cover discovery, prepared-source reuse, relative inputs, +failure reporting, environment effects, and ordinary loading's boundaries under +the [testing policy](../policy/testing.md). + +The [implementation plan](implementation.md) owns the selected API names, +execution stages, and validation work needed to reach this target.