Add fixed destination path mapping

This commit is contained in:
2026-06-01 21:33:12 +00:00
parent 1a52fdce6f
commit a8564035d3
16 changed files with 816 additions and 67 deletions

View File

@@ -38,6 +38,12 @@ Preview local fan-out publication:
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
```
Preview local archive-plus-latest publication:
```sh
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
```
Preview a run for automation:
```sh
@@ -66,7 +72,11 @@ go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
Destination bundle paths are configured per destination with `path_mapping.mode`.
The default mode, `preserve_relative`, preserves the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at that destination.
The `fixed` mode publishes one selected source bundle at the destination backend root. A fixed destination with local `path: /srv/www/reports/latest` writes outputs and `.distributor.json` directly under `/srv/www/reports/latest`. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp, with the source-root-relative bundle path as the deterministic tie-breaker.
The maintained local examples write under `workspace/`, which is ignored by Git.
@@ -152,6 +162,42 @@ transform:
If `input` is omitted in index mode, the source manifest must list exactly one Markdown file. Generated HTML is recorded in `.distributor.json` with `kind: generated`, `source_path`, `transform: markdown_to_html`, digest, and size metadata.
## Archive And Latest Fan-Out
A pipeline can publish the same source to an archive destination and a stable latest destination:
```yaml
pipelines:
- id: reports
source:
backend: local
path: /var/spool/distributor/reports
destinations:
- id: archive
backend: local
path: /srv/reports/archive
path_mapping:
mode: preserve_relative
publish:
source: true
html: false
- id: latest-html
backend: local
path: /srv/www/reports/latest
path_mapping:
mode: fixed
publish:
source: false
html: true
transform:
markdown_to_html:
enabled: true
mode: index
input: report.md
```
The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
## Source Validation and Inspection
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
@@ -180,9 +226,9 @@ For configured SSH sources, host key and authentication behavior matches `run`.
Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`.
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication.
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, output records, final counters, and partial failure records. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, output records, final counters, and partial failure records. Fixed action records include `path_mapping: "fixed"`, and the summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
## Retry and Replacement Behavior
@@ -205,7 +251,7 @@ go run ./cmd/distributor run --config <config-path> --force
Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require `transfer.on_conflict: replace` plus `--force`. Newer destination state requires `transfer.on_destination_newer: replace` plus `--force`.
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For S3 destinations, deletion is constrained to the configured bucket and prefix plus the destination bundle prefix. Force is per run only and has no config default.
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For fixed destinations, the destination bundle path is the backend root, so forced replacement may clear that configured root but not its parent path, sibling directories, or anything outside the configured S3 bucket and prefix. Force is per run only and has no config default.
## Failure Handling
@@ -235,7 +281,7 @@ S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optiona
When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. For fixed destinations, that prefix is the configured bucket plus optional `prefix`. Distributor does not manage bucket versioning or delete markers.
## Secrets Directory