81 lines
4.8 KiB
Markdown
81 lines
4.8 KiB
Markdown
# Configuration Internals
|
|
|
|
## Purpose
|
|
|
|
`internal/config` defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.
|
|
|
|
## Inputs and outputs
|
|
|
|
Input is a YAML file containing optional `secrets` and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
|
|
|
|
## Loading flow
|
|
|
|
`LoadFile` opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses `DefaultConfigPath` when the CLI does not supply a config path.
|
|
|
|
Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.
|
|
|
|
`LoadFile` does not read secret files. `Run` loads the configured secrets directory after config validation and before backend construction.
|
|
|
|
## Defaults
|
|
|
|
Defaults are applied in `ApplyDefaults`:
|
|
|
|
- pipeline validation defaults `on_digest_mismatch` to `fail`;
|
|
- SSH backend `port` defaults to `22`;
|
|
- SSH backend `host_key_policy` defaults to `accept-new`;
|
|
- destination publish policy defaults to source output only;
|
|
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
|
- `transfer.on_destination_same` defaults to `skip`;
|
|
- `transfer.on_destination_older` defaults to `replace`;
|
|
- `transfer.on_destination_newer` defaults to `skip`;
|
|
- `transfer.on_conflict` defaults to `fail`.
|
|
|
|
## Validation responsibilities
|
|
|
|
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, and valid transfer actions.
|
|
|
|
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
|
|
|
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
|
|
|
|
## Executable support boundary
|
|
|
|
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
|
|
|
|
SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
|
|
|
|
S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
|
|
|
|
## Secrets and credential resolution
|
|
|
|
`secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.
|
|
|
|
The resolver checks the real process environment first and loaded secret values second. Differing process/secret conflicts are reported by variable name only. The resolver does not mutate `os.Environ`; default SDK credential chains continue to see only real process environment values.
|
|
|
|
Credential-consuming backend wiring should resolve explicit credential environment variable references through `Environment.ResolveCredentials` or the same resolver pattern instead of calling `os.Getenv` directly.
|
|
|
|
The user-facing configuration reference is `docs/config.md`; this file documents package behavior for maintainers.
|
|
|
|
## Failure behavior
|
|
|
|
Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.
|
|
|
|
Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.
|
|
|
|
## Tests
|
|
|
|
Before changing config behavior, inspect:
|
|
|
|
- `internal/config/load_test.go`
|
|
- `internal/config/validate_test.go`
|
|
- example-loading coverage in `internal/config`
|
|
- user-facing examples under `examples/`
|
|
|
|
## Invariants
|
|
|
|
- Defaults are applied before validation.
|
|
- Unknown YAML fields are rejected.
|
|
- `docs/config.md` remains the canonical user-facing config reference.
|
|
- Runtime backend execution support is not inferred from config validation support.
|
|
- New user-visible config behavior must be covered by tests and docs in the same change.
|