distributor/docs/internal/config.md

Configuration Internals

Purpose

internal/config defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.

Inputs and outputs

Input is a YAML file containing optional secrets and required pipelines. Output is a Config value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.

Loading flow

LoadFile opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses DefaultConfigPath when the CLI does not supply a config path.

Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.

LoadFile does not read secret files. Run loads the configured secrets directory after config validation and before backend construction.

Defaults

Defaults are applied in ApplyDefaults:

• pipeline validation defaults on_digest_mismatch to fail;
• SSH backend port defaults to 22;
• SSH backend host_key_policy defaults to accept-new;
• destination publish policy defaults to source output only;
• transfer.on_destination_same defaults to skip;
• transfer.on_destination_older defaults to replace;
• transfer.on_destination_newer defaults to skip;
• transfer.on_conflict defaults to fail.

Validation responsibilities

Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, and valid transfer actions.

ValidatePublishTransformPolicy is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in sidecar mode. A publish policy must select source output, HTML output, or both.

Executable support boundary

Config validation accepts local, ssh, and s3 backend shapes so config files can be validated as schemas. Runtime execution opens local and SSH backends through internal/app; S3 remains accepted by validation but unavailable at execution.

SSH config uses structured fields: host, optional user, optional port, path, optional ssh_key_file, optional known_hosts, and optional host_key_policy. host_key_policy accepts YAML booleans and strings and normalizes true/strict, accept-new, and false/off.

Secrets and credential resolution

secrets.directory points to a directory of credential files. LoadSecretEnvironment reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an Environment resolver plus conflict metadata.

The resolver checks the real process environment first and loaded secret values second. Differing process/secret conflicts are reported by variable name only. The resolver does not mutate os.Environ; default SDK credential chains continue to see only real process environment values.

Future credential-consuming backend code should resolve explicit credential environment variable references through Environment.ResolveCredentials or the same resolver pattern instead of calling os.Getenv directly.

The user-facing configuration reference is docs/config.md; this file documents package behavior for maintainers.

Failure behavior

Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.

Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.

Tests

Before changing config behavior, inspect:

• internal/config/load_test.go
• internal/config/validate_test.go
• example-loading coverage in internal/config
• user-facing examples under examples/

Invariants

• Defaults are applied before validation.
• Unknown YAML fields are rejected.
• docs/config.md remains the canonical user-facing config reference.
• Runtime backend execution support is not inferred from config validation support.
• New user-visible config behavior must be covered by tests and docs in the same change.