Add config loading and dry-run validation

2026-05-31 01:53:13 +00:00
parent 22d0424232
commit 29dbad2967
16 changed files with 928 additions and 17 deletions
--- a/docs/config.md
+++ b/docs/config.md
@@ -0,0 +1,96 @@
+# Distributor Configuration
+
+## Config file location
+
+`distributor run --config <path> --dry-run` loads the YAML config at the path provided by `--config`.
+
+If `--config` is omitted during dry-run, the built-in default path is:
+
+```text
+/usr/local/etc/distributor/config.yml
+```
+
+The current implementation loads and validates configuration only. Bundle discovery and publication are not implemented yet.
+
+## Minimal config
+
+```yaml
+pipelines:
+  - id: reports
+    source:
+      backend: local
+      path: /var/spool/distributor/reports
+    destinations:
+      - id: archive
+        backend: local
+        path: /srv/reports/archive
+```
+
+This uses the default publish policy of source files only and the default transfer policy.
+
+## Production-oriented config
+
+```yaml
+pipelines:
+  - id: reports
+    source:
+      backend: local
+      path: /var/spool/distributor/reports
+    validation:
+      on_digest_mismatch: fail
+    destinations:
+      - id: archive
+        backend: s3
+        endpoint: https://s3.example.com
+        bucket: reports
+        prefix: archive
+        region: us-east-1
+        force_path_style: true
+        credentials:
+          access_key_id_env: DISTRIBUTOR_S3_ACCESS_KEY_ID
+          secret_access_key_env: DISTRIBUTOR_S3_SECRET_ACCESS_KEY
+        publish:
+          source: true
+          html: false
+```
+
+## Reference
+
+Top level:
+
+- `pipelines`: required non-empty list.
+
+Pipeline:
+
+- `id`: required unique identifier.
+- `source`: required backend config.
+- `destinations`: required non-empty destination list.
+- `validation.on_digest_mismatch`: optional, defaults to `fail`; only `fail` is supported.
+
+Backends:
+
+- `local`: requires `path`.
+- `ssh`: requires `uri` and `path`.
+- `s3`: requires `endpoint` and `bucket`; supports optional `prefix`, `region`, `force_path_style`, and `credentials`.
+
+Destination policy:
+
+- `publish.source`: publish source artifacts.
+- `publish.html`: publish generated HTML artifacts.
+- `transfer.on_destination_same`: `skip` or `fail`, defaults to `skip`.
+- `transfer.on_destination_older`: `replace` or `fail`, defaults to `replace`.
+- `transfer.on_destination_newer`: `skip` or `fail`, defaults to `skip`.
+- `transfer.on_conflict`: only `fail`, defaults to `fail`.
+
+When `publish.html` is true, `transform.markdown_to_html.enabled: true` and `transform.markdown_to_html.mode: sidecar` are required.
+
+## Secrets
+
+Do not put literal secrets in config files. S3 credentials may refer to environment variable names with:
+
+- `credentials.access_key_id_env`
+- `credentials.secret_access_key_env`
+
+## Examples
+
+Maintained examples live under [examples/](../examples/).