Normalize internal component documentation
This commit is contained in:
@@ -1,101 +1,57 @@
|
||||
# Configuration Internals
|
||||
|
||||
Audience: developers and LLM coding agents changing `internal/config`.
|
||||
|
||||
## Purpose
|
||||
|
||||
`internal/config` defines YAML-backed configuration structs, defaulting, and validation for distributor pipelines.
|
||||
`internal/config` owns YAML config structs, config file loading, defaulting, validation, size/duration parsing, SSH/S3 normalization helpers, and the credential environment resolver.
|
||||
|
||||
## Inputs and outputs
|
||||
## Inputs And Outputs
|
||||
|
||||
Input is a YAML file containing optional `server`, optional `secrets`, and required `pipelines`. Output is a `Config` value with defaults applied and validation completed. Load failures include the config path and whether the failure occurred during file loading, YAML parsing, or validation.
|
||||
Inputs are YAML files, YAML scalar values, process environment lookup functions, optional secrets directories, and `Config` values. Outputs are defaulted `Config` values, validation errors, parsed byte sizes and durations, normalized backend options, loaded secret environments, secret conflict metadata, and resolved credentials.
|
||||
|
||||
## Loading flow
|
||||
## Boundaries
|
||||
|
||||
`LoadFile` opens the requested path, decodes YAML with known-field checking enabled, applies defaults, and validates the result. The app uses `DefaultConfigPath` when the CLI does not supply a config path.
|
||||
The package does not open storage backends, authenticate HTTP requests, start servers, publish destinations, or execute transforms. Runtime execution support is wired by `internal/app`.
|
||||
|
||||
Known-field checking rejects misspelled or unknown YAML keys before defaults and validation run.
|
||||
The canonical user-facing config reference is `docs/config.md`.
|
||||
|
||||
`LoadFile` does not read secret files. App entrypoints load the configured
|
||||
secrets directory after config validation and before credential-consuming work.
|
||||
## Config Fields Used
|
||||
|
||||
## Defaults
|
||||
The package defines all user-visible config fields: `server.http`, `secrets`, `pipelines`, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, and transfer policy.
|
||||
|
||||
Defaults are applied in `ApplyDefaults`:
|
||||
## Adapters Used
|
||||
|
||||
- HTTP server `bind` defaults to `127.0.0.1:8080`;
|
||||
- HTTP server `staging_root` defaults to `/var/spool/distributor`;
|
||||
- HTTP server `max_upload_size` defaults to `20MB`;
|
||||
- HTTP server `queue_size` defaults to `16`;
|
||||
- HTTP server `max_concurrency` defaults to `1`;
|
||||
- HTTP server `retention` defaults to `24h`;
|
||||
- `http_upload` source `staging_path` defaults to `<server.http.staging_root>/<pipeline id>`;
|
||||
- `http_upload` source `max_upload_size` defaults to `server.http.max_upload_size`;
|
||||
- pipeline validation defaults `on_digest_mismatch` to `fail`;
|
||||
- SSH backend `port` defaults to `22`;
|
||||
- SSH backend `host_key_policy` defaults to `accept-new`;
|
||||
- destination publish policy defaults to source output only;
|
||||
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
||||
- destination path mapping defaults to `preserve_relative`;
|
||||
- destination link primary policy defaults to `auto` when a `links` block is present;
|
||||
- `transfer.on_destination_same` defaults to `skip`;
|
||||
- `transfer.on_destination_older` defaults to `replace`;
|
||||
- `transfer.on_destination_newer` defaults to `skip`;
|
||||
- `transfer.on_conflict` defaults to `fail`.
|
||||
No external storage adapters are used directly. The package exposes normalized config and credential values consumed by app-level adapter construction.
|
||||
|
||||
## Validation responsibilities
|
||||
## State And Manifest Behavior
|
||||
|
||||
Validation requires positive HTTP server limits and retention, at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, valid destination link config, and valid transfer actions.
|
||||
The package does not parse source manifests or destination state. It validates config values that later affect manifest validation and destination state, such as publish/transform combinations, links, transfer policy, backend roots, S3 prefix shape, and HTTP upload source settings.
|
||||
|
||||
HTTP upload sources require `token_env`, a staging path after defaults, and a positive maximum upload size. Literal token fields are not part of the YAML schema. The `http_upload` backend is accepted only for sources and rejected for destinations.
|
||||
## Skip And Resume Behavior
|
||||
|
||||
Upload size values are parsed from strings with `B`, `KB`, `MB`, or `GB` suffixes using 1024 multipliers. Retention values are parsed with `time.ParseDuration`. Explicit zero values fail validation; omitted values receive defaults before validation.
|
||||
The package has no runtime skip or resume behavior. It provides transfer policy values that publish planning later applies to destination comparison outcomes.
|
||||
|
||||
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
||||
## Failure Behavior
|
||||
|
||||
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
|
||||
`LoadFile` wraps file open, YAML parse, and validation failures with config path context. YAML decoding rejects unknown fields. Validation collects all detected field errors into a single error value.
|
||||
|
||||
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
|
||||
Secret loading fails for unreadable secrets directories, invalid secret filenames, unreadable secret files, and missing or empty required credential values. Secret conflicts are returned as warnings metadata, not secret values.
|
||||
|
||||
Destination links are optional. When a `links` block is present, `base_url` is required, must use `http` or `https`, and must not include a query string or fragment. `primary` accepts `auto`, `html`, and `source`.
|
||||
|
||||
## Executable support boundary
|
||||
|
||||
Config validation accepts `local`, `ssh`, `s3`, and source-only `http_upload` backend shapes. Runtime `run`, `validate`, and `inspect` workflows open `local`, `ssh`, and `s3` through `internal/app`. Runtime `serve` workflows execute `http_upload` sources through the app upload coordinator and HTTP server.
|
||||
|
||||
SSH config uses structured fields: `host`, optional `user`, optional `port`, `path`, optional `ssh_key_file`, optional `known_hosts`, and optional `host_key_policy`. `host_key_policy` accepts YAML booleans and strings and normalizes `true`/`strict`, `accept-new`, and `false`/`off`.
|
||||
|
||||
S3 config requires `endpoint` and `bucket`, normalizes optional `prefix`, defaults `region` to `us-east-1`, and defaults omitted `force_path_style` to `true` while preserving explicit `false`.
|
||||
|
||||
HTTP upload config is source-only. Config owns its YAML shape, defaulting, size and duration parsing, and validation. The config package does not authenticate requests, stage uploads, or execute HTTP upload sources. The app layer resolves `token_env` through the config-owned environment resolver before starting the HTTP server.
|
||||
|
||||
## Secrets and credential resolution
|
||||
|
||||
`secrets.directory` points to a directory of credential files. `LoadSecretEnvironment` reads regular files and symlinks to regular files, rejects invalid filenames, trims exactly one trailing LF or CRLF, and returns an `Environment` resolver plus conflict metadata.
|
||||
|
||||
The resolver checks the real process environment first and loaded secret values second. Differing process/secret conflicts are reported by variable name only. The resolver does not mutate `os.Environ`; default SDK credential chains continue to see only real process environment values.
|
||||
|
||||
Credential-consuming backend wiring should resolve explicit credential environment variable references through `Environment.ResolveCredentials` or the same resolver pattern instead of calling `os.Getenv` directly.
|
||||
|
||||
The user-facing configuration reference is `docs/config.md`; this file documents package behavior for maintainers.
|
||||
|
||||
## Failure behavior
|
||||
|
||||
Load errors wrap the underlying file, YAML, or validation error with context. Validation collects all detected field errors into one error value instead of stopping at the first invalid field.
|
||||
|
||||
Unsupported backend names fail validation. Accepted backend names without runtime execution support fail later during app backend opening.
|
||||
|
||||
## Tests
|
||||
|
||||
Before changing config behavior, inspect:
|
||||
## Tests To Inspect
|
||||
|
||||
- `internal/config/load_test.go`
|
||||
- `internal/config/validate_test.go`
|
||||
- example-loading coverage in `internal/config`
|
||||
- user-facing examples under `examples/`
|
||||
- `internal/config/secrets_test.go`
|
||||
- `internal/config/backend_view_test.go`
|
||||
- `internal/app/runtime_test.go`
|
||||
- example configs under `examples/`
|
||||
|
||||
## Invariants
|
||||
## Architectural Invariants
|
||||
|
||||
- Defaults are applied before validation.
|
||||
- Unknown YAML fields are rejected.
|
||||
- `docs/config.md` remains the canonical user-facing config reference.
|
||||
- Runtime backend execution support is not inferred from config validation support.
|
||||
- New user-visible config behavior must be covered by tests and docs in the same change.
|
||||
- `http_upload` is source-only config.
|
||||
- Credential-consuming runtime code must use the config-owned environment resolver.
|
||||
- Secret values are never printed by config warnings.
|
||||
- New user-visible config behavior must update `docs/config.md` and tests.
|
||||
|
||||
Reference in New Issue
Block a user