393 lines
22 KiB
Markdown
393 lines
22 KiB
Markdown
# Configured secrets bootstrap implementation plan
|
|
|
|
## Scope and execution
|
|
|
|
Implement the [secrets-directory roadmap](secrets-directory.md) and accepted
|
|
[ADR 0023](../adr/0023-explicit-configured-secrets-bootstrap.md). The starting
|
|
revision is `fb0560d`. Read [development](../development.md), the
|
|
[agent policy](../policy/agents.md), and the applicable architecture, testing,
|
|
and documentation policies before implementing a stage.
|
|
|
|
The stages run sequentially in one sprint. Intermediate stages may leave callers
|
|
temporarily uncompilable while an internal handoff changes. Finish the shared
|
|
integration before final validation. Keep tests with their behavior's owning
|
|
stage and reuse existing fixtures and failure seams. Each stage is intended to
|
|
fit one Luna implementation prompt.
|
|
|
|
The completed feature provides explicit bootstrap before ordinary configuration
|
|
loading. Discovery and the captured main document are shared between those two
|
|
calls. Ordinary secret resolution continues to read the environment at its
|
|
existing lifecycle point. Filesystem access and bootstrap finish synchronously.
|
|
|
|
## Decisions for implementation
|
|
|
|
### Public API
|
|
|
|
Keep the existing function and its exact function type. Add these root-owned
|
|
public declarations, following the existing facade-to-engine conversion pattern.
|
|
|
|
```go
|
|
func LoadSecretsIntoEnvironment(directory string) (*SecretsReport, error)
|
|
|
|
type SecretsOptions struct {
|
|
Overwrite bool
|
|
}
|
|
|
|
func LoadSecretsIntoEnvironmentWithOptions(
|
|
directory string, options SecretsOptions,
|
|
) (*SecretsReport, error)
|
|
|
|
type SecretsFromConfigOptions struct {
|
|
Filesystem *FilesystemContext
|
|
Overwrite *bool
|
|
}
|
|
|
|
func LoadSecretsIntoEnvironmentFromConfig(
|
|
main Source, supplied ...SecretsFromConfigOptions,
|
|
) (*SecretsReport, Source, error)
|
|
```
|
|
|
|
The companion accepts zero or one options value, matching the load entry points.
|
|
Multiple values produce `LoadError` with `CodeOptionsInvalid`, an empty non-nil
|
|
report, and zero `Source` before acquisition or writes. Snapshot the supplied
|
|
context and Boolean value at call entry using existing conversion conventions.
|
|
Nil overwrite uses YAML, then false. A supplied false or true wins over YAML.
|
|
|
|
The companion accepts host `File` and `SearchConfig` sources and `NoSource`.
|
|
A prepared host source can also be supplied to an explicit companion call using
|
|
its captured main. Reject other source kinds and native filesystem contexts as
|
|
`LoadError` option failures before consuming them. Preserve constructor and search
|
|
validation diagnostics through the existing helpers. NoSource and optional search
|
|
exhaustion succeed with an empty report and `NoSource()`. Explicit context on an
|
|
absent input follows existing absent-host-context validation; subsequent absent
|
|
loading uses the ordinary caller-supplied context rules.
|
|
|
|
Every failure returns a non-nil report and zero `Source`. Discovery, acquisition,
|
|
parse, control, and preparation-context failures use `LoadError`. Directory
|
|
resolution/inspection, credential reads, and environment writes use
|
|
`SecretsError`. Keep both error wrappers and their existing safe rendering and
|
|
detached records. Successful preparation with no directory returns the prepared
|
|
main and an empty report.
|
|
|
|
### Controls and source roles
|
|
|
|
Recognize root `config.secrets_dir` as a nonempty string containing a relative
|
|
host path and root `config.secrets_overwrite` as a Boolean. Accept ordinary YAML
|
|
Boolean spellings through the parser's existing type handling. Nulls, wrong
|
|
types, empty directory strings, absolute paths, and host volume-qualified paths
|
|
are invalid. Treat path text literally using host path operations. Parent-relative
|
|
paths are usable when their canonical targets stay inside the selected root.
|
|
|
|
Only the main input's document root owns these controls. Imports, underlays,
|
|
and selected overlays containing them produce a source-local control diagnostic.
|
|
Nested application `config` mappings and root application data introduced by
|
|
placement, defaults, CLI, or resolution retain their existing meaning under
|
|
[ADR 0020](../adr/0020-document-root-controls.md).
|
|
|
|
Normal loading checks and consumes the main controls, with imports enabled or
|
|
disabled. Filesystem checks for the declared secrets directory happen in the
|
|
companion helper. Both ordinary loading entry points perform their usual secret
|
|
lookup. Preparation validates the root control mapping and the bootstrap fields,
|
|
while other control grammars and import enablement are checked by the later load.
|
|
Retain the original main tree so imports and collection declarations are still
|
|
available to that load. A valid overwrite declaration without a directory is a
|
|
no-op. Invalid bootstrap fields fail even if a Go overwrite choice is supplied.
|
|
|
|
Use existing diagnostic codes. Use `CodeSecretsInvalidPath` for invalid directory
|
|
syntax, `CodeSecretsInvalidValue` for a non-Boolean overwrite setting, and the
|
|
existing directive-context codes for misplaced controls (`CodeImportInvalid` or
|
|
`CodeOverlayDirective`). Root mapping errors keep their current classification.
|
|
Bootstrap-field diagnostics carry `/config/secrets_dir` or
|
|
`/config/secrets_overwrite` and source-local locations, inside a `LoadError`.
|
|
Avoid raw input, parser errors, or credential contents in messages.
|
|
|
|
### Captured-source ownership and context
|
|
|
|
Represent preparation privately as a captured host-file payload attached to an
|
|
engine `sourceFile`. Keep public `Source` opaque and report `SourceKindFile`.
|
|
Capture the original parsed document, original byte count, canonical main path,
|
|
canonical base, and canonical containment root. Retain the unconsumed controls.
|
|
Keep captured data read-only and give each load an owned document clone with
|
|
fresh source IDs and accounting. Namespace tokens and acquisition state belong
|
|
to each load. Retain passive path facts with the source.
|
|
|
|
Normal acquisition consumes the captured main without discovery, file reading,
|
|
or bootstrap. Apply the usual byte, document, placement, and aggregate accounting
|
|
to the captured input in every load. A preparation-time count never substitutes
|
|
for load-wide admission. Imports and resources continue to be acquired when their
|
|
normal operations run. Main deletion or modification does not change captured
|
|
contents; later filesystem operations may fail under their existing contracts.
|
|
|
|
Use the captured context when later loading supplies none. Resolve an explicit
|
|
later host context against the captured canonical main path. It is compatible
|
|
when its canonical root, base, and main path match the captured facts. Compare
|
|
path facts using the existing host path rules, allowing equivalent spellings
|
|
and root symlink aliases. Native contexts, a supplied file BaseDir, and genuinely
|
|
different roots fail with `CodeOptionsInvalid` in `LoadError`. Avoid re-resolving
|
|
the original main filename to compare contexts, since it may now point elsewhere.
|
|
Use the same check for ordinary options and layered main input contexts before
|
|
import or overlay handling can substitute a context.
|
|
|
|
### Directory bootstrap semantics
|
|
|
|
Reuse `loadSecretsDir` and its environment/access seams for all three public
|
|
helpers. Preservation mode retains both existing-variable checks, including
|
|
empty values and a variable created between preflight and application.
|
|
Overwrite mode reads existing names' eligible files and applies them regardless
|
|
of environment presence. Unrelated environment variables retain their values.
|
|
Use the existing loaded/skipped/failed/not-applied statuses. `ExistingEmpty`
|
|
continues to describe skipped existing values.
|
|
|
|
Configured directory resolution uses the captured base and root. Canonicalize
|
|
the selected directory with the existing filesystem helpers and check containment
|
|
of that canonical target before reads or writes. A directory symlink can resolve
|
|
to a contained target, following the existing bootstrap directory convention.
|
|
Return `SecretsError` with `CodeSecretsInvalidPath` for an outside-root target;
|
|
use existing filesystem classifications for missing or unreadable targets.
|
|
The direct-directory helpers keep their existing path resolution behavior.
|
|
|
|
Bootstrap includes all eligible directory entries, independently of the later
|
|
typed schema. Reuse dot-entry exclusion, regular-file checks, limits, sorted
|
|
order, newline handling, empty-file handling, and NUL rejection. Complete
|
|
metadata checks and required reads before writes. A write failure retains prior
|
|
writes and reports remaining entries as not applied. Later load failures leave
|
|
bootstrap effects in place. Reports and prepared sources retain no credential
|
|
bytes from secret files. Applications own startup ordering and concurrent
|
|
process-environment mutation under the existing bootstrap contract.
|
|
|
|
## Stage 1 — Extend shared directory bootstrap with overwrite
|
|
|
|
**Completed.**
|
|
|
|
Work in `internal/engine/secrets.go` and `internal/engine/secrets_test.go`.
|
|
|
|
1. Add a small engine options value for overwrite and keep `LoadSecretsDir`
|
|
delegating with preservation enabled. Add its options-bearing counterpart.
|
|
Extend the shared private worker and existing test seams accordingly.
|
|
2. Make overwrite affect only staging and applying values. Preserve sorted
|
|
enumeration, all metadata validation before reads, bounded reads before
|
|
writes, skipped-file behavior in preservation mode, and failure reporting.
|
|
3. Add focused behavioral cases for overwriting nonempty and empty existing
|
|
variables, overwriting with empty file content, and leaving names without
|
|
files alone. Extend the existing staged-presence-race and partial-apply-failure
|
|
cases to cover overwrite. Reuse current read-limit, dot-file, NUL, and safe
|
|
error coverage. Process-mutating tests restore state and run serially.
|
|
|
|
Stage completion means both policies work through one bootstrap worker and
|
|
existing preservation behavior remains covered. Run the relevant existing and
|
|
new `LoadSecretsDir` tests once callers compile.
|
|
|
|
## Stage 2 — Add main-scoped bootstrap control parsing
|
|
|
|
**Completed.**
|
|
|
|
Work primarily in `internal/engine/directives.go`, `acquisition.go`, and the
|
|
directive/version parsing handoffs in `source_version.go`.
|
|
|
|
1. Introduce a small private bootstrap-settings value with declaration presence,
|
|
directory text, overwrite value, and source-local diagnostic locations. Share
|
|
its field validation between preparation and normal control extraction.
|
|
2. Give extraction enough source-role information to allow these controls only
|
|
on the actual main document. `rootDirectiveContextSource` and
|
|
`rootDirectiveContextLayered` alone cannot identify that role because those
|
|
contexts also occur on contributions. Carry the role through the relevant
|
|
acquisition/version and import traversal call sites.
|
|
3. Keep bootstrap-only extraction usable before normal imports/sequence grammar
|
|
validation. The companion can prepare a main that declares imports without
|
|
enabling or acquiring them. Normal extraction checks the complete control
|
|
block and removes it before placement and application decoding.
|
|
4. Extend directive-scope and facade-control tests with main declarations,
|
|
disabled imports, wrong types, explicit false, misplaced controls, and nested
|
|
ordinary data. Verify source-version control-overlap checks recognize the
|
|
new fields and that version acceptance itself stays in the load.
|
|
|
|
Stage completion means the normal load accepts and consumes valid main settings
|
|
while leaving environment state untouched. Other source-root controls retain
|
|
their existing grammar, enablement, and error precedence.
|
|
|
|
## Stage 3 — Share host-main selection and capture
|
|
|
|
**Completed.**
|
|
|
|
Work in `internal/engine/source.go`, `parse.go`, `discovery.go`, and the initial
|
|
selection/read portion of `acquisition.go`. A focused `prepared_source.go` may
|
|
hold the new private payload and preparation helpers.
|
|
|
|
1. Factor the existing selection and bounded main parsing handoff for reuse by
|
|
normal acquisition and preparation. Reuse `resolveSearchInput`, search option
|
|
checks, `readSource`, and YAML adapter diagnostics. Keep candidate ordering,
|
|
optional exhaustion, selected-file failure, and regular-file semantics.
|
|
2. Add the captured host payload specified above. Preserve original source line
|
|
locations and all controls. Construct independent documents for subsequent
|
|
loads with `document.Clone` and `setDocumentSourceID`. Count captured bytes
|
|
under the current read limits before returning a document to acquisition.
|
|
3. For capture, resolve and validate the host base/root through existing
|
|
`filesystemOptionDiagnostics`, `resolveFilesystemContext`, and directory
|
|
validation. Store path facts without access handles or a preparation token.
|
|
4. Add internal cases proving captured data stays fixed after the original file
|
|
changes, becomes absent, or another search candidate appears. Retain ordinary
|
|
File/SearchConfig reread behavior. Cover parser/read failures and limits with
|
|
existing narrow test fixtures. Keep sources and payloads private until the
|
|
public wrapper stage.
|
|
|
|
Stage completion means the acquisition front end can consume an ordinary host
|
|
file or its captured payload through the same parsed-source handoff.
|
|
|
|
## Stage 4 — Integrate captured contexts with ordinary loading
|
|
|
|
**Completed.**
|
|
|
|
Work in `options.go`, `filesystem_context.go`, `filesystem.go`, `acquisition.go`,
|
|
`load_input.go`, `selected_overlay.go`, and `source_context.go` as needed.
|
|
|
|
1. Make context resolution select captured path facts and enforce the explicit
|
|
context equivalence rule at the shared acquisition boundary. Ensure both
|
|
imported and standalone branches receive the selected resolved context.
|
|
2. Carry the captured base/root into standalone main acquisition, ordinary
|
|
overlay resolution, and passive source-context completion. Ordinary overlays
|
|
keep their established root-relative lookup convention.
|
|
3. Preserve file-kind records, canonical paths, source-local diagnostics, and
|
|
placement projection. Reuse normal source registration and accounting.
|
|
4. Add focused internal cases for omitted/equivalent/incompatible later contexts,
|
|
explicit wider roots, a symlinked selected main, main removal, destinations,
|
|
and ordinary overlay/resource-context behavior. Compare relevant records and
|
|
values with a normal load of the same unmodified file.
|
|
|
|
Stage completion means an ordinary load uses captured contents and context
|
|
through all its normal paths. Real filesystem failures are handled at the
|
|
existing access boundaries without retaining an open handle in the source.
|
|
|
|
## Stage 5 — Integrate imports and layered main-relative consumers
|
|
|
|
**Completed.**
|
|
|
|
Work in `import_acquisition.go`, `load_request.go`, and layered/context owners
|
|
identified by the shared acquisition handoff.
|
|
|
|
1. Feed the captured main document and resolved context into imported acquisition.
|
|
Charge its raw bytes and nodes through existing accounting exactly once per
|
|
load. Keep placement wrapper accounting and per-tier repetition tracking.
|
|
2. Treat the captured main as the same canonical host file for repeated-file and
|
|
cycle checks. A fresh load gets its own host namespace and source IDs.
|
|
3. Ensure layered acquisition propagates the captured main context to declared
|
|
MainRelative underlays and selected overlays, with existing explicit context
|
|
and fallback precedence. Preserve main-first acquisition and tier precedence.
|
|
4. Verify live imports after main capture, repeat/cycle detection involving the
|
|
captured main, source/version evidence with imports, an explicitly wider root
|
|
permitting contained parent-relative imports, and independent repeated loads.
|
|
Include one integrated layered case with a main-relative underlay and overlay.
|
|
Use existing import/placement/version suites for unchanged semantics.
|
|
|
|
Stage completion means prepared main sources behave as host main inputs across
|
|
ordinary and layered composition, metadata, and main-relative bindings.
|
|
|
|
## Stage 6 — Implement configured bootstrap orchestration
|
|
|
|
**Completed.**
|
|
|
|
Add a focused engine companion in `secrets_from_config.go`, using the capture
|
|
and bootstrap helpers from earlier stages.
|
|
|
|
1. Validate and snapshot options, resolve the main, capture its document and
|
|
context, and extract bootstrap settings. Return the no-directory and absent
|
|
cases under the public return contract. A valid Go overwrite override does
|
|
not bypass validation of malformed YAML bootstrap fields.
|
|
2. Join the declared directory to the captured base. Add an optional internal
|
|
containment-root constraint to shared directory preparation so canonical
|
|
resolution and containment validation occur before the worker uses that
|
|
canonical directory. Direct-directory calls supply no such constraint.
|
|
3. Select overwrite using Go, YAML, then false, and invoke the shared worker.
|
|
Return the prepared source only after successful bootstrap. All branches
|
|
return the non-nil report and appropriate existing error family.
|
|
4. Test no declaration, overwrite without a directory, optional/required missing
|
|
main, unsupported source/context, malformed controls, missing or outside-root
|
|
directory, and directory symlink containment. Assert empty/partial reports and
|
|
zero prepared sources on errors. Use the existing environment seam for
|
|
deterministic apply failures and verify safe error/report output.
|
|
|
|
Stage completion means configured bootstrap can finish independently of schema,
|
|
imports, version hooks, resolution, and validation, returning a reusable main
|
|
only when its own operation succeeds.
|
|
|
|
## Stage 7 — Expose root-owned APIs and convert both error families
|
|
|
|
**Completed.**
|
|
|
|
Work in root `secrets.go`, `source.go`, and existing facade conversion owners.
|
|
|
|
1. Add the two options types and new functions with the signatures above. Keep
|
|
the existing function type intact and route its default behavior through the
|
|
shared bootstrap implementation.
|
|
2. Convert the companion's report, prepared engine Source, and errors to the
|
|
existing public wrappers. Preserve `LoadError` source records as well as
|
|
diagnostics. Preserve `SecretsError` without exposing underlying filesystem
|
|
or environment errors. Add only the small conversion logic this boundary
|
|
needs, reusing existing converters.
|
|
3. Document signatures, options precedence, reports, error types, and source
|
|
ownership on their declarations. Explain captured reuse versus File/SearchConfig
|
|
reuse in Source's documentation. Keep implementation types private.
|
|
4. Add facade tests for both new functions and assignment of the old function to
|
|
`func(string) (*SecretsReport, error)`. Exercise YAML true with explicit Go
|
|
false, YAML false with Go true, omitted choices, and multiple options. Verify
|
|
public `errors.As` behavior for preparation and bootstrap failures.
|
|
|
|
Stage completion means applications can use the complete two-call startup flow
|
|
through ConfigKit's public package with the documented ownership and errors.
|
|
|
|
## Stage 8 — Verify the complete startup handoff
|
|
|
|
Extend public integration coverage and add one maintained consumer startup
|
|
example with ordinary and layered use explained from the same helper.
|
|
|
|
1. Start in a different working directory, discover a main with relative secrets,
|
|
bootstrap all eligible entries including an SDK-only name, and load the
|
|
prepared source. Verify configured overwrite and existing empty-variable
|
|
behavior with disposable environment names.
|
|
2. Change the main, discovery candidates, an import, and an environment value
|
|
after preparation. Verify captured main values, live import values, fresh
|
|
environment lookup, and independent metadata/results across subsequent loads.
|
|
Include selected-main provenance and source resource context assertions.
|
|
3. Exercise the layered handoff with explicit root preparation and companion
|
|
inputs. Confirm compatible later context succeeds and incompatible context
|
|
fails safely after the already-completed bootstrap.
|
|
4. Show a later version/decoding or validation failure preserves the bootstrap
|
|
report and environment effects. Separately verify ordinary Load/LoadLayered
|
|
on a main containing the controls perform no environment writes, even with
|
|
YAML overwrite true. Existing secret-resolution tests continue to own
|
|
credential lookup and redaction behavior.
|
|
5. Verify captured sources retain no handles through successful and failed
|
|
preparation. Use existing resource-lifetime techniques and narrow seams,
|
|
avoiding a new filesystem framework. Exercise independent captured-source
|
|
loads under the race detector without concurrent process mutation.
|
|
|
|
Stage completion means the feature works through normal entry points and its
|
|
integration risks are covered without duplicating every internal test at the
|
|
facade boundary.
|
|
|
|
## Stage 9 — Update canonical documentation and validate the sprint
|
|
|
|
1. Update `docs/config.md` with the two main controls, preparation and bootstrap
|
|
boundaries, lookup timing, overwrite precedence, reports/errors, host scope,
|
|
and prepared-source reuse/context behavior. Keep source-local controls
|
|
distinct from application paths and preserve existing loading semantics.
|
|
2. Update `docs/consumers/README.md` and its test map to link the new maintained
|
|
example. Update `docs/development.md` only where navigation/test ownership
|
|
changes. Explain internal capture and acquisition handoffs in
|
|
`docs/internal/overview.md` and relevant import documentation.
|
|
3. Update the implemented-behavior paragraphs in architecture policy and retire
|
|
its roadmap implementation-status link. Keep ADR 0023's accepted decision
|
|
text intact. Preserve the existing bootstrap limits and concurrency/rollback
|
|
contract while documenting explicit overwrite as the selected extension.
|
|
4. Format changed Go files and run the focused tests for changed owners. Run
|
|
`./scripts/check.sh` and `go test -race ./...` on the completed implementation.
|
|
Verify edited Markdown links and headings, exported declaration comments,
|
|
and the public example. Reuse successful checks for unchanged content.
|
|
5. Review the final diff for captured-state aliasing, handles, extra parsing or
|
|
discovery, secret disclosure, omitted accounting, and accidental changes to
|
|
current configuration processing or helper signatures. Resolve failures and
|
|
finish the documented target before marking the sprint complete.
|
|
|
|
The final result is the implemented feature with current documentation and
|
|
appropriate behavioral coverage. Release preparation follows its separate
|
|
procedure when requested.
|