Add fixed destination path mapping
This commit is contained in:
14
docs/cli.md
14
docs/cli.md
@@ -133,6 +133,12 @@ Preview local fan-out publication:
|
||||
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
|
||||
```
|
||||
|
||||
Preview local archive-plus-latest publication:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
|
||||
```
|
||||
|
||||
Preview a forced replacement before publishing:
|
||||
|
||||
```sh
|
||||
@@ -143,7 +149,7 @@ go run ./cmd/distributor run --config <config-path> --dry-run --force
|
||||
|
||||
Text output is the default and is intended for humans.
|
||||
|
||||
`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the bundle path, destination id, destination backend, action, outputs, and reason. Actions include:
|
||||
`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the source bundle path, destination id, destination backend, action, outputs, and reason. Fixed path destinations also print `path_mapping=fixed target=.` to show that the selected bundle targets the destination backend root. Actions include:
|
||||
|
||||
- `publish_new`: destination has no managed state and is empty.
|
||||
- `replace_older`: destination state is older than the source manifest.
|
||||
@@ -154,7 +160,9 @@ Text output is the default and is intended for humans.
|
||||
|
||||
The command exits non-zero if any destination fails. Independent later destinations are still attempted.
|
||||
|
||||
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, and whether the run was a dry run.
|
||||
Dry-run output for fixed path destinations prints a warning with the candidate count and selected source bundle. If a fixed path dry run plans a destructive replacement, it prints an additional warning that the destination root would be replaced.
|
||||
|
||||
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, whether the run was a dry run, and fixed path destinations.
|
||||
|
||||
JSON output writes exactly one JSON document to stdout:
|
||||
|
||||
@@ -178,7 +186,7 @@ Command-specific JSON results:
|
||||
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
|
||||
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
|
||||
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
|
||||
- `run`: dry-run status, pipeline summaries, destination action records, output records, final counters, warnings, and partial failure records.
|
||||
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, optional fixed path mapping markers, output records, final counters, warnings, and partial failure records.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
|
||||
@@ -83,6 +83,40 @@ transform:
|
||||
|
||||
When `mode: index` omits `input`, the source manifest must list exactly one Markdown file.
|
||||
|
||||
## Destination Path Mapping
|
||||
|
||||
Each destination chooses how source bundle paths map into that destination:
|
||||
|
||||
```yaml
|
||||
path_mapping:
|
||||
mode: preserve_relative
|
||||
```
|
||||
|
||||
`preserve_relative` is the default. It publishes each discovered source bundle at the same path relative to the destination backend root. A source bundle at `daily/2026-06-01` publishes below `daily/2026-06-01` for that destination.
|
||||
|
||||
`fixed` publishes one selected source bundle directly at the destination backend root:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: latest-html
|
||||
backend: local
|
||||
path: /srv/www/reports/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: index
|
||||
input: report.md
|
||||
```
|
||||
|
||||
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple candidates have the same timestamp, the source-root-relative bundle path in ascending order wins. Older candidates are not planned or written for that destination.
|
||||
|
||||
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
|
||||
|
||||
## Reference
|
||||
|
||||
Top level:
|
||||
@@ -121,6 +155,7 @@ Destination:
|
||||
- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
|
||||
- `publish`: optional; defaults to source-only publication.
|
||||
- `transform`: required only for generated HTML publication.
|
||||
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
|
||||
- `transfer`: optional; defaults described below.
|
||||
|
||||
Accepted backend names:
|
||||
@@ -210,6 +245,7 @@ Defaults are applied after YAML decoding and before validation:
|
||||
- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
|
||||
- `publish.source: true`
|
||||
- `publish.html: false`
|
||||
- `path_mapping.mode: preserve_relative`
|
||||
- `transfer.on_destination_same: skip`
|
||||
- `transfer.on_destination_older: replace`
|
||||
- `transfer.on_destination_newer: skip`
|
||||
@@ -242,5 +278,6 @@ Maintained examples live under [examples](../examples/):
|
||||
- `local-html.yml`: runnable local HTML publication.
|
||||
- `local-index.yml`: runnable local `index.html` publication.
|
||||
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
|
||||
- `archive-and-latest.yml`: runnable local fan-out publication to an archive destination and a fixed latest destination.
|
||||
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
|
||||
- `s3-destination.yml`: environment-gated local-to-S3 publication example.
|
||||
|
||||
@@ -25,6 +25,7 @@ Defaults are applied in `ApplyDefaults`:
|
||||
- SSH backend `host_key_policy` defaults to `accept-new`;
|
||||
- destination publish policy defaults to source output only;
|
||||
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
||||
- destination path mapping defaults to `preserve_relative`;
|
||||
- `transfer.on_destination_same` defaults to `skip`;
|
||||
- `transfer.on_destination_older` defaults to `replace`;
|
||||
- `transfer.on_destination_newer` defaults to `skip`;
|
||||
@@ -32,12 +33,14 @@ Defaults are applied in `ApplyDefaults`:
|
||||
|
||||
## Validation responsibilities
|
||||
|
||||
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, and valid transfer actions.
|
||||
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, and valid transfer actions.
|
||||
|
||||
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
||||
|
||||
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
|
||||
|
||||
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
|
||||
|
||||
## Executable support boundary
|
||||
|
||||
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, transformer resolver, transfer policy, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
|
||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
|
||||
|
||||
Output is a plan with an action, reason, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
|
||||
|
||||
@@ -24,7 +24,9 @@ Execution fails if a write, delete, state serialization, or context check fails.
|
||||
|
||||
The current implementation publishes source files and Markdown-to-HTML outputs. Markdown sidecar mode writes same-directory `.html` outputs, and Markdown index mode writes `index.html`. Backend behavior is supplied through `internal/storage`; app runtime currently supplies local, SSH, and S3 backends.
|
||||
|
||||
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags or load config files.
|
||||
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags, load config files, or choose which source bundles a destination receives.
|
||||
|
||||
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
|
||||
|
||||
## Safety
|
||||
|
||||
@@ -37,6 +39,7 @@ Before changing publish behavior, inspect tests under `internal/publish` and run
|
||||
## Invariants
|
||||
|
||||
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
|
||||
- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
|
||||
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
||||
- Forced replacement is explicit per run and deletes only within the destination bundle path.
|
||||
- Publish execution writes destination state after selected outputs are written.
|
||||
|
||||
@@ -38,6 +38,12 @@ Preview local fan-out publication:
|
||||
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
|
||||
```
|
||||
|
||||
Preview local archive-plus-latest publication:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
|
||||
```
|
||||
|
||||
Preview a run for automation:
|
||||
|
||||
```sh
|
||||
@@ -66,7 +72,11 @@ go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline
|
||||
|
||||
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
|
||||
|
||||
Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
|
||||
Destination bundle paths are configured per destination with `path_mapping.mode`.
|
||||
|
||||
The default mode, `preserve_relative`, preserves the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at that destination.
|
||||
|
||||
The `fixed` mode publishes one selected source bundle at the destination backend root. A fixed destination with local `path: /srv/www/reports/latest` writes outputs and `.distributor.json` directly under `/srv/www/reports/latest`. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp, with the source-root-relative bundle path as the deterministic tie-breaker.
|
||||
|
||||
The maintained local examples write under `workspace/`, which is ignored by Git.
|
||||
|
||||
@@ -152,6 +162,42 @@ transform:
|
||||
|
||||
If `input` is omitted in index mode, the source manifest must list exactly one Markdown file. Generated HTML is recorded in `.distributor.json` with `kind: generated`, `source_path`, `transform: markdown_to_html`, digest, and size metadata.
|
||||
|
||||
## Archive And Latest Fan-Out
|
||||
|
||||
A pipeline can publish the same source to an archive destination and a stable latest destination:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/reports
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /srv/reports/archive
|
||||
path_mapping:
|
||||
mode: preserve_relative
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
- id: latest-html
|
||||
backend: local
|
||||
path: /srv/www/reports/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: index
|
||||
input: report.md
|
||||
```
|
||||
|
||||
The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
|
||||
|
||||
## Source Validation and Inspection
|
||||
|
||||
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
|
||||
@@ -180,9 +226,9 @@ For configured SSH sources, host key and authentication behavior matches `run`.
|
||||
|
||||
Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`.
|
||||
|
||||
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication.
|
||||
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
|
||||
|
||||
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, output records, final counters, and partial failure records. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
|
||||
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, output records, final counters, and partial failure records. Fixed action records include `path_mapping: "fixed"`, and the summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
|
||||
|
||||
## Retry and Replacement Behavior
|
||||
|
||||
@@ -205,7 +251,7 @@ go run ./cmd/distributor run --config <config-path> --force
|
||||
|
||||
Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require `transfer.on_conflict: replace` plus `--force`. Newer destination state requires `transfer.on_destination_newer: replace` plus `--force`.
|
||||
|
||||
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For S3 destinations, deletion is constrained to the configured bucket and prefix plus the destination bundle prefix. Force is per run only and has no config default.
|
||||
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For fixed destinations, the destination bundle path is the backend root, so forced replacement may clear that configured root but not its parent path, sibling directories, or anything outside the configured S3 bucket and prefix. Force is per run only and has no config default.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
@@ -235,7 +281,7 @@ S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optiona
|
||||
|
||||
When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
|
||||
|
||||
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
|
||||
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. For fixed destinations, that prefix is the configured bucket plus optional `prefix`. Distributor does not manage bucket versioning or delete markers.
|
||||
|
||||
## Secrets Directory
|
||||
|
||||
|
||||
Reference in New Issue
Block a user