102 lines
7.0 KiB
Markdown
102 lines
7.0 KiB
Markdown
# Configuration Internals
|
|
|
|
## Purpose
|
|
|
|
`internal/config` defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.
|
|
|
|
## Inputs and outputs
|
|
|
|
Input is a YAML file containing optional `server`, optional `secrets`, and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
|
|
|
|
## Loading flow
|
|
|
|
`LoadFile` opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses `DefaultConfigPath` when the CLI does not supply a config path.
|
|
|
|
Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.
|
|
|
|
`LoadFile` does not read secret files. App entrypoints load the configured
|
|
secrets directory after config validation and before credential-consuming work.
|
|
|
|
## Defaults
|
|
|
|
Defaults are applied in `ApplyDefaults`:
|
|
|
|
- HTTP server `bind` defaults to `127.0.0.1:8080`;
|
|
- HTTP server `staging_root` defaults to `/var/spool/distributor`;
|
|
- HTTP server `max_upload_size` defaults to `20MB`;
|
|
- HTTP server `queue_size` defaults to `16`;
|
|
- HTTP server `max_concurrency` defaults to `1`;
|
|
- HTTP server `retention` defaults to `24h`;
|
|
- `http_upload` source `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`;
|
|
- `http_upload` source `max_upload_size` defaults to `server.http.max_upload_size`;
|
|
- pipeline validation defaults `on_digest_mismatch` to `fail`;
|
|
- SSH backend `port` defaults to `22`;
|
|
- SSH backend `host_key_policy` defaults to `accept-new`;
|
|
- destination publish policy defaults to source output only;
|
|
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
|
- destination path mapping defaults to `preserve_relative`;
|
|
- destination link primary policy defaults to `auto` when a `links` block is present;
|
|
- `transfer.on_destination_same` defaults to `skip`;
|
|
- `transfer.on_destination_older` defaults to `replace`;
|
|
- `transfer.on_destination_newer` defaults to `skip`;
|
|
- `transfer.on_conflict` defaults to `fail`.
|
|
|
|
## Validation responsibilities
|
|
|
|
Validation requires positive HTTP server limits and retention, at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
|
|
|
|
HTTP upload sources require `token_env`, a staging path after defaults, and a positive maximum upload size. Literal token fields are not part of the YAML schema. The `http_upload` backend is accepted only for sources and rejected for destinations.
|
|
|
|
Upload size values are parsed from strings with `B`, `KB`, `MB`, or `GB` suffixes using 1024 multipliers. Retention values are parsed with `time.ParseDuration`. Explicit zero values fail validation; omitted values receive defaults before validation.
|
|
|
|
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
|
|
|
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
|
|
|
|
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
|
|
|
|
Destination links are optional. When a `links` block is present, `base_url` is required, must use `http` or `https`, and must not include a query string or fragment. `primary` accepts `auto`, `html`, and `source`.
|
|
|
|
## Executable support boundary
|
|
|
|
Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime `run`, `validate`, and `inspect` workflows open `local`, `ssh`, and `s3` through `internal/app`. Runtime `serve` workflows execute `http_upload` sources through the app upload coordinator and HTTP server.
|
|
|
|
SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
|
|
|
|
S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
|
|
|
|
HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not authenticate requests, stage uploads, or execute HTTP upload sources. The app layer resolves `token_env` through the config-owned environment resolver before starting the HTTP server.
|
|
|
|
## Secrets and credential resolution
|
|
|
|
`secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.
|
|
|
|
The resolver checks the real process environment first and loaded secret values second. Differing process/secret conflicts are reported by variable name only. The resolver does not mutate `os.Environ`; default SDK credential chains continue to see only real process environment values.
|
|
|
|
Credential-consuming backend wiring should resolve explicit credential environment variable references through `Environment.ResolveCredentials` or the same resolver pattern instead of calling `os.Getenv` directly.
|
|
|
|
The user-facing configuration reference is `docs/config.md`; this file documents package behavior for maintainers.
|
|
|
|
## Failure behavior
|
|
|
|
Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.
|
|
|
|
Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.
|
|
|
|
## Tests
|
|
|
|
Before changing config behavior, inspect:
|
|
|
|
- `internal/config/load_test.go`
|
|
- `internal/config/validate_test.go`
|
|
- example-loading coverage in `internal/config`
|
|
- user-facing examples under `examples/`
|
|
|
|
## Invariants
|
|
|
|
- Defaults are applied before validation.
|
|
- Unknown YAML fields are rejected.
|
|
- `docs/config.md` remains the canonical user-facing config reference.
|
|
- Runtime backend execution support is not inferred from config validation support.
|
|
- New user-visible config behavior must be covered by tests and docs in the same change.
|