distributor/docs/internal/config.md

Configuration Internals

Purpose

internal/config defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.

Inputs and outputs

Input is a YAML file containing optional server, optional secrets, and required pipelines. Output is a Config value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.

Loading flow

LoadFile opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses DefaultConfigPath when the CLI does not supply a config path.

Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.

LoadFile does not read secret files. App entrypoints load the configured secrets directory after config validation and before credential-consuming work.

Defaults

Defaults are applied in ApplyDefaults:

• HTTP server bind defaults to 127.0.0.1:8080;
• HTTP server staging_root defaults to /var/spool/distributor;
• HTTP server max_upload_size defaults to 20MB;
• HTTP server queue_size defaults to 16;
• HTTP server max_concurrency defaults to 1;
• HTTP server retention defaults to 24h;
• http_upload source staging_path defaults to <server.http.staging_root>/<pipeline id>;
• http_upload source max_upload_size defaults to server.http.max_upload_size;
• pipeline validation defaults on_digest_mismatch to fail;
• SSH backend port defaults to 22;
• SSH backend host_key_policy defaults to accept-new;
• destination publish policy defaults to source output only;
• Markdown-to-HTML mode defaults to sidecar when a transform block is present and mode is omitted;
• destination path mapping defaults to preserve_relative;
• destination link primary policy defaults to auto when a links block is present;
• transfer.on_destination_same defaults to skip;
• transfer.on_destination_older defaults to replace;
• transfer.on_destination_newer defaults to skip;
• transfer.on_conflict defaults to fail.

Validation responsibilities

Validation requires positive HTTP server limits and retention, at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.

HTTP upload sources require token_env, a staging path after defaults, and a positive maximum upload size. Literal token fields are not part of the YAML schema. The http_upload backend is accepted only for sources and rejected for destinations.

Upload size values are parsed from strings with B, KB, MB, or GB suffixes using 1024 multipliers. Retention values are parsed with time.ParseDuration. Explicit zero values fail validation; omitted values receive defaults before validation.

Transfer validation accepts replace for on_destination_newer and on_conflict, but publish planning honors those destructive actions only when the current run explicitly requests force.

ValidatePublishTransformPolicy is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in sidecar or index mode. Enabled Markdown-to-HTML config is rejected when publish.html is false. input is accepted only for enabled index mode. A publish policy must select source output, HTML output, or both.

Destination path mapping accepts preserve_relative and fixed. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.

Destination links are optional. When a links block is present, base_url is required, must use http or https, and must not include a query string or fragment. primary accepts auto, html, and source.

Executable support boundary

Config validation accepts local, ssh, s3, and source-only http_upload backend shapes. Runtime run, validate, and inspect workflows open local, ssh, and s3 through internal/app. Runtime serve workflows execute http_upload sources through the app upload coordinator and HTTP server.

SSH config uses structured fields: host, optional user, optional port, path, optional ssh_key_file, optional known_hosts, and optional host_key_policy. host_key_policy accepts YAML booleans and strings and normalizes true/strict, accept-new, and false/off.

S3 config requires endpoint and bucket, normalizes optional prefix, defaults region to us-east-1, and defaults omitted force_path_style to true while preserving explicit false.

HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not authenticate requests, stage uploads, or execute HTTP upload sources. The app layer resolves token_env through the config-owned environment resolver before starting the HTTP server.

Secrets and credential resolution

secrets.directory points to a directory of credential files. LoadSecretEnvironment reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an Environment resolver plus conflict metadata.

The resolver checks the real process environment first and loaded secret values second. Differing process/secret conflicts are reported by variable name only. The resolver does not mutate os.Environ; default SDK credential chains continue to see only real process environment values.

Credential-consuming backend wiring should resolve explicit credential environment variable references through Environment.ResolveCredentials or the same resolver pattern instead of calling os.Getenv directly.

The user-facing configuration reference is docs/config.md; this file documents package behavior for maintainers.

Failure behavior

Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.

Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.

Tests

Before changing config behavior, inspect:

• internal/config/load_test.go
• internal/config/validate_test.go
• example-loading coverage in internal/config
• user-facing examples under examples/

Invariants

• Defaults are applied before validation.
• Unknown YAML fields are rejected.
• docs/config.md remains the canonical user-facing config reference.
• Runtime backend execution support is not inferred from config validation support.
• New user-visible config behavior must be covered by tests and docs in the same change.