Add fixed destination path mapping
This commit is contained in:
@@ -2,7 +2,7 @@
|
||||
|
||||
`distributor` validates manifested report bundles and publishes selected source or generated artifacts to configured destinations.
|
||||
|
||||
It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars.
|
||||
It is a local-first CLI with SSH/SFTP and S3-compatible storage support: source bundles can be read from local or remote storage, destinations can be local directories or remote paths, and Markdown files can be rendered to HTML sidecars or `index.html`.
|
||||
|
||||
Go producers can use `gitea.maximumdirect.net/eric/distributor/pkg/bundle` to build, write, parse, and validate complete local source bundles with the same manifest contract used by `distributor`.
|
||||
|
||||
|
||||
14
docs/cli.md
14
docs/cli.md
@@ -133,6 +133,12 @@ Preview local fan-out publication:
|
||||
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
|
||||
```
|
||||
|
||||
Preview local archive-plus-latest publication:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
|
||||
```
|
||||
|
||||
Preview a forced replacement before publishing:
|
||||
|
||||
```sh
|
||||
@@ -143,7 +149,7 @@ go run ./cmd/distributor run --config <config-path> --dry-run --force
|
||||
|
||||
Text output is the default and is intended for humans.
|
||||
|
||||
`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the bundle path, destination id, destination backend, action, outputs, and reason. Actions include:
|
||||
`run` text output prints the number of configured pipelines, one line per pipeline, one line per planned destination action, and a final status line. Destination action lines include the source bundle path, destination id, destination backend, action, outputs, and reason. Fixed path destinations also print `path_mapping=fixed target=.` to show that the selected bundle targets the destination backend root. Actions include:
|
||||
|
||||
- `publish_new`: destination has no managed state and is empty.
|
||||
- `replace_older`: destination state is older than the source manifest.
|
||||
@@ -154,7 +160,9 @@ Text output is the default and is intended for humans.
|
||||
|
||||
The command exits non-zero if any destination fails. Independent later destinations are still attempted.
|
||||
|
||||
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, and whether the run was a dry run.
|
||||
Dry-run output for fixed path destinations prints a warning with the candidate count and selected source bundle. If a fixed path dry run plans a destructive replacement, it prints an additional warning that the destination root would be replaced.
|
||||
|
||||
The final status line includes counters for `publish_new`, `replace_older`, `force_replace`, skipped destinations, failures, whether the run was a dry run, and fixed path destinations.
|
||||
|
||||
JSON output writes exactly one JSON document to stdout:
|
||||
|
||||
@@ -178,7 +186,7 @@ Command-specific JSON results:
|
||||
- `validate`: bundle count and discovered bundle identifiers. Configured source results also include pipeline id and source backend.
|
||||
- `inspect`: bundle path, id, created timestamp, digest, file count, total size, and manifest file records. Configured source results also include pipeline id and source backend.
|
||||
- `manifest create`: manifest path, bundle root, id, created timestamp, digest, file count, and file records.
|
||||
- `run`: dry-run status, pipeline summaries, destination action records, output records, final counters, warnings, and partial failure records.
|
||||
- `run`: dry-run status, pipeline summaries, destination action records, destination bundle paths, optional fixed path mapping markers, output records, final counters, warnings, and partial failure records.
|
||||
|
||||
## Diagnostics
|
||||
|
||||
|
||||
@@ -83,6 +83,40 @@ transform:
|
||||
|
||||
When `mode: index` omits `input`, the source manifest must list exactly one Markdown file.
|
||||
|
||||
## Destination Path Mapping
|
||||
|
||||
Each destination chooses how source bundle paths map into that destination:
|
||||
|
||||
```yaml
|
||||
path_mapping:
|
||||
mode: preserve_relative
|
||||
```
|
||||
|
||||
`preserve_relative` is the default. It publishes each discovered source bundle at the same path relative to the destination backend root. A source bundle at `daily/2026-06-01` publishes below `daily/2026-06-01` for that destination.
|
||||
|
||||
`fixed` publishes one selected source bundle directly at the destination backend root:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: latest-html
|
||||
backend: local
|
||||
path: /srv/www/reports/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: index
|
||||
input: report.md
|
||||
```
|
||||
|
||||
Fixed destinations select the newest discovered source bundle by manifest `created` timestamp. If multiple candidates have the same timestamp, the source-root-relative bundle path in ascending order wins. Older candidates are not planned or written for that destination.
|
||||
|
||||
Fixed mapping is useful for stable latest-style paths. It is more destructive than archive-style publication because successive source bundles target the same destination root. Preview fixed destinations with `run --dry-run`, especially before using `--force`.
|
||||
|
||||
## Reference
|
||||
|
||||
Top level:
|
||||
@@ -121,6 +155,7 @@ Destination:
|
||||
- Backend fields: same accepted shape as source backends, with destination fields at the destination level.
|
||||
- `publish`: optional; defaults to source-only publication.
|
||||
- `transform`: required only for generated HTML publication.
|
||||
- `path_mapping.mode`: optional; defaults to `preserve_relative`. Accepted values are `preserve_relative` and `fixed`.
|
||||
- `transfer`: optional; defaults described below.
|
||||
|
||||
Accepted backend names:
|
||||
@@ -210,6 +245,7 @@ Defaults are applied after YAML decoding and before validation:
|
||||
- `transform.markdown_to_html.mode: sidecar` when a Markdown-to-HTML transform block is present and mode is omitted
|
||||
- `publish.source: true`
|
||||
- `publish.html: false`
|
||||
- `path_mapping.mode: preserve_relative`
|
||||
- `transfer.on_destination_same: skip`
|
||||
- `transfer.on_destination_older: replace`
|
||||
- `transfer.on_destination_newer: skip`
|
||||
@@ -242,5 +278,6 @@ Maintained examples live under [examples](../examples/):
|
||||
- `local-html.yml`: runnable local HTML publication.
|
||||
- `local-index.yml`: runnable local `index.html` publication.
|
||||
- `fan-out.yml`: runnable local fan-out publication to source and HTML destinations.
|
||||
- `archive-and-latest.yml`: runnable local fan-out publication to an archive destination and a fixed latest destination.
|
||||
- `ssh-destination.yml`: environment-gated local-to-SSH publication example.
|
||||
- `s3-destination.yml`: environment-gated local-to-S3 publication example.
|
||||
|
||||
@@ -25,6 +25,7 @@ Defaults are applied in `ApplyDefaults`:
|
||||
- SSH backend `host_key_policy` defaults to `accept-new`;
|
||||
- destination publish policy defaults to source output only;
|
||||
- Markdown-to-HTML mode defaults to `sidecar` when a transform block is present and mode is omitted;
|
||||
- destination path mapping defaults to `preserve_relative`;
|
||||
- `transfer.on_destination_same` defaults to `skip`;
|
||||
- `transfer.on_destination_older` defaults to `replace`;
|
||||
- `transfer.on_destination_newer` defaults to `skip`;
|
||||
@@ -32,12 +33,14 @@ Defaults are applied in `ApplyDefaults`:
|
||||
|
||||
## Validation responsibilities
|
||||
|
||||
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, and valid transfer actions.
|
||||
Validation requires at least one pipeline, slug-like unique pipeline ids, one source per pipeline, at least one destination, slug-like unique destination ids within each pipeline, backend-specific required fields, valid validation policy, valid publish and transform combinations, valid destination path mapping mode, and valid transfer actions.
|
||||
|
||||
Transfer validation accepts `replace` for `on_destination_newer` and `on_conflict`, but publish planning honors those destructive actions only when the current run explicitly requests force.
|
||||
|
||||
`ValidatePublishTransformPolicy` is shared with publish planning so destination policy combinations are checked consistently. Publishing HTML requires an enabled Markdown-to-HTML transform in `sidecar` or `index` mode. Enabled Markdown-to-HTML config is rejected when `publish.html` is false. `input` is accepted only for enabled `index` mode. A publish policy must select source output, HTML output, or both.
|
||||
|
||||
Destination path mapping accepts `preserve_relative` and `fixed`. The app layer applies the mapping when it selects destination bundle paths; config owns only YAML shape, defaulting, and validation.
|
||||
|
||||
## Executable support boundary
|
||||
|
||||
Config validation accepts `local`, `ssh`, and `s3` backend shapes. Runtime execution opens all three through `internal/app`.
|
||||
|
||||
@@ -6,7 +6,7 @@
|
||||
|
||||
## Inputs and outputs
|
||||
|
||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, transformer resolver, transfer policy, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
|
||||
Inputs are a source bundle, source backend, destination backend, pipeline id, destination id, publish policy, transform policy, transformer resolver, transfer policy, path mapping mode, destination bundle path, existing destination state, and whether explicit force was requested for the current run.
|
||||
|
||||
Output is a plan with an action, reason, and selected source or generated outputs. Execution writes selected source files, generated files, and `.distributor.json` for publish or replacement actions.
|
||||
|
||||
@@ -24,7 +24,9 @@ Execution fails if a write, delete, state serialization, or context check fails.
|
||||
|
||||
The current implementation publishes source files and Markdown-to-HTML outputs. Markdown sidecar mode writes same-directory `.html` outputs, and Markdown index mode writes `index.html`. Backend behavior is supplied through `internal/storage`; app runtime currently supplies local, SSH, and S3 backends.
|
||||
|
||||
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags or load config files.
|
||||
The package uses `internal/state` for destination comparison, `internal/storage` for IO, and the shared `internal/config` publish/transform policy helper for request validation. It resolves transforms through a narrow resolver supplied by the caller; concrete transform registration is owned by the app layer. It does not parse CLI flags, load config files, or choose which source bundles a destination receives.
|
||||
|
||||
The app layer computes the destination bundle path before planning. `preserve_relative` destinations pass the source-root-relative bundle path. `fixed` destinations pass an empty destination bundle path, which means the destination backend root, and pass only the newest selected source bundle for that destination.
|
||||
|
||||
## Safety
|
||||
|
||||
@@ -37,6 +39,7 @@ Before changing publish behavior, inspect tests under `internal/publish` and run
|
||||
## Invariants
|
||||
|
||||
- Publish planning is deterministic for the same source, destination state, policies, and transform outputs.
|
||||
- Destination bundle paths are caller-supplied and are interpreted relative to the destination backend root.
|
||||
- Normal replacement deletes only managed paths recorded in existing state plus `.distributor.json`.
|
||||
- Forced replacement is explicit per run and deletes only within the destination bundle path.
|
||||
- Publish execution writes destination state after selected outputs are written.
|
||||
|
||||
@@ -38,6 +38,12 @@ Preview local fan-out publication:
|
||||
go run ./cmd/distributor run --config examples/fan-out.yml --dry-run
|
||||
```
|
||||
|
||||
Preview local archive-plus-latest publication:
|
||||
|
||||
```sh
|
||||
go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
|
||||
```
|
||||
|
||||
Preview a run for automation:
|
||||
|
||||
```sh
|
||||
@@ -66,7 +72,11 @@ go run ./cmd/distributor validate --config examples/local-publish.yml --pipeline
|
||||
|
||||
Source bundles are discovered beneath the configured source root. Each bundle is a directory containing `manifest.json`.
|
||||
|
||||
Destination bundle paths preserve the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at each destination.
|
||||
Destination bundle paths are configured per destination with `path_mapping.mode`.
|
||||
|
||||
The default mode, `preserve_relative`, preserves the source bundle path relative to the source root. A source bundle at the source root publishes to the destination root. A source bundle under `daily/` publishes under `daily/` at that destination.
|
||||
|
||||
The `fixed` mode publishes one selected source bundle at the destination backend root. A fixed destination with local `path: /srv/www/reports/latest` writes outputs and `.distributor.json` directly under `/srv/www/reports/latest`. Fixed destinations select the newest discovered source bundle by manifest `created` timestamp, with the source-root-relative bundle path as the deterministic tie-breaker.
|
||||
|
||||
The maintained local examples write under `workspace/`, which is ignored by Git.
|
||||
|
||||
@@ -152,6 +162,42 @@ transform:
|
||||
|
||||
If `input` is omitted in index mode, the source manifest must list exactly one Markdown file. Generated HTML is recorded in `.distributor.json` with `kind: generated`, `source_path`, `transform: markdown_to_html`, digest, and size metadata.
|
||||
|
||||
## Archive And Latest Fan-Out
|
||||
|
||||
A pipeline can publish the same source to an archive destination and a stable latest destination:
|
||||
|
||||
```yaml
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /var/spool/distributor/reports
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /srv/reports/archive
|
||||
path_mapping:
|
||||
mode: preserve_relative
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
- id: latest-html
|
||||
backend: local
|
||||
path: /srv/www/reports/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: index
|
||||
input: report.md
|
||||
```
|
||||
|
||||
The archive destination plans every discovered source bundle at its source-relative path. The fixed latest destination plans only the newest discovered bundle and writes `index.html` plus `.distributor.json` at its backend root.
|
||||
|
||||
## Source Validation and Inspection
|
||||
|
||||
`validate` and `inspect` can operate on a local path or on one configured pipeline source. Configured source mode requires both `--config` and `--pipeline`; it loads the normal config, resolves `secrets.directory`, opens only the selected source backend, and does not open any destinations.
|
||||
@@ -180,9 +226,9 @@ For configured SSH sources, host key and authentication behavior matches `run`.
|
||||
|
||||
Dry-run output is useful before publishing to confirm actions such as `publish_new`, `replace_older`, `force_replace`, `skip_same`, and `skip_destination_newer`.
|
||||
|
||||
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication.
|
||||
Destination action lines include the destination backend, so mixed local, SSH, and S3 fan-out runs can be audited before publication. Fixed path destinations add `path_mapping=fixed target=.` to planned action lines. Dry-run also prints a warning with the fixed destination candidate count and selected source bundle; destructive fixed replacements print an additional warning that the destination root would be replaced.
|
||||
|
||||
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, output records, final counters, and partial failure records. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
|
||||
Use `--format json` when another process needs stable run data. JSON output includes warnings, pipeline summaries, destination actions, destination bundle paths, output records, final counters, and partial failure records. Fixed action records include `path_mapping: "fixed"`, and the summary includes `fixed_path`. If one destination fails after planning or execution begins, JSON output still contains the successful and failed destination records with `ok: false`, and the command exits non-zero.
|
||||
|
||||
## Retry and Replacement Behavior
|
||||
|
||||
@@ -205,7 +251,7 @@ go run ./cmd/distributor run --config <config-path> --force
|
||||
|
||||
Forced replacement can overwrite unmanaged non-empty destination paths. Destination state conflicts require `transfer.on_conflict: replace` plus `--force`. Newer destination state requires `transfer.on_destination_newer: replace` plus `--force`.
|
||||
|
||||
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For S3 destinations, deletion is constrained to the configured bucket and prefix plus the destination bundle prefix. Force is per run only and has no config default.
|
||||
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete above that bundle path. For fixed destinations, the destination bundle path is the backend root, so forced replacement may clear that configured root but not its parent path, sibling directories, or anything outside the configured S3 bucket and prefix. Force is per run only and has no config default.
|
||||
|
||||
## Failure Handling
|
||||
|
||||
@@ -235,7 +281,7 @@ S3 execution uses the AWS SDK for Go v2. Configure `endpoint`, `bucket`, optiona
|
||||
|
||||
When explicit credential env names are configured, both variables must resolve to non-empty values through the real process environment or `secrets.directory`. When they are omitted, the AWS SDK default credential chain is used as-is.
|
||||
|
||||
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
|
||||
Normal replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix. For fixed destinations, that prefix is the configured bucket plus optional `prefix`. Distributor does not manage bucket versioning or delete markers.
|
||||
|
||||
## Secrets Directory
|
||||
|
||||
|
||||
26
examples/archive-and-latest.yml
Normal file
26
examples/archive-and-latest.yml
Normal file
@@ -0,0 +1,26 @@
|
||||
pipelines:
|
||||
- id: example-archive-and-latest
|
||||
source:
|
||||
backend: local
|
||||
path: examples/source-bundle
|
||||
destinations:
|
||||
- id: local-source-archive
|
||||
backend: local
|
||||
path: workspace/published/archive-and-latest/archive
|
||||
path_mapping:
|
||||
mode: preserve_relative
|
||||
publish:
|
||||
source: true
|
||||
html: false
|
||||
- id: local-html-latest
|
||||
backend: local
|
||||
path: workspace/published/archive-and-latest/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
publish:
|
||||
source: false
|
||||
html: true
|
||||
transform:
|
||||
markdown_to_html:
|
||||
enabled: true
|
||||
mode: index
|
||||
@@ -5,6 +5,7 @@ import (
|
||||
"errors"
|
||||
"fmt"
|
||||
"io"
|
||||
"sort"
|
||||
"strings"
|
||||
|
||||
"gitea.maximumdirect.net/eric/distributor/internal/bundle"
|
||||
@@ -112,33 +113,55 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
|
||||
return err
|
||||
}
|
||||
}
|
||||
for _, sourceBundle := range bundles {
|
||||
for _, destination := range pipeline.Destinations {
|
||||
destinationBackend, err := backends.openDestination(ctx, destination)
|
||||
if err != nil {
|
||||
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
|
||||
for _, destination := range pipeline.Destinations {
|
||||
selections := selectDestinationBundles(destination, bundles)
|
||||
if isFixedPathDestination(destination) {
|
||||
summary.recordFixedPath()
|
||||
if options.DryRun {
|
||||
warning := fixedPathSelectionWarning(pipeline.ID, destination.ID, selections, len(bundles))
|
||||
if jsonOutput {
|
||||
warnings = append(warnings, warning)
|
||||
} else if options.Stdout != nil {
|
||||
if err := writeWarnings(options.Stdout, []OutputWarning{warning}); err != nil {
|
||||
closeBackend(sourceBackend)
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if len(selections) == 0 {
|
||||
continue
|
||||
}
|
||||
destinationBackend, err := backends.openDestination(ctx, destination)
|
||||
if err != nil {
|
||||
for _, selection := range selections {
|
||||
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(selection.SourceBundle.RootRelativePath), err)
|
||||
summary.recordFailure()
|
||||
if jsonOutput {
|
||||
result.Actions = append(result.Actions, errorAction(pipeline.ID, destination.ID, destination.Backend, sourceBundle.RootRelativePath, err))
|
||||
result.Actions = append(result.Actions, errorAction(pipeline.ID, destination.ID, destination.Backend, selection.SourceBundle.RootRelativePath, err))
|
||||
} else if options.Stdout != nil {
|
||||
writeErrorLine(options.Stdout, sourceBundle.RootRelativePath, destination.ID, destination.Backend, err)
|
||||
}
|
||||
continue
|
||||
}
|
||||
closeDestination := true
|
||||
deferCloseDestination := func() {
|
||||
if closeDestination {
|
||||
closeBackend(destinationBackend)
|
||||
closeDestination = false
|
||||
writeErrorLine(options.Stdout, selection.SourceBundle.RootRelativePath, destination.ID, destination.Backend, err)
|
||||
}
|
||||
}
|
||||
continue
|
||||
}
|
||||
closeDestination := true
|
||||
deferCloseDestination := func() {
|
||||
if closeDestination {
|
||||
closeBackend(destinationBackend)
|
||||
closeDestination = false
|
||||
}
|
||||
}
|
||||
for _, selection := range selections {
|
||||
sourceBundle := selection.SourceBundle
|
||||
req := publish.Request{
|
||||
PipelineID: pipeline.ID,
|
||||
DestinationID: destination.ID,
|
||||
SourceBundle: sourceBundle,
|
||||
SourceBackend: sourceBackend,
|
||||
DestinationBackend: destinationBackend,
|
||||
DestinationBundlePath: sourceBundle.RootRelativePath,
|
||||
DestinationBundlePath: selection.DestinationBundlePath,
|
||||
PathMapping: destination.PathMap.Mode,
|
||||
Publish: *destination.Publish,
|
||||
Transform: destination.Transform,
|
||||
Transformers: transforms,
|
||||
@@ -160,6 +183,24 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
|
||||
if plan.BundlePath == "" {
|
||||
plan.BundlePath = sourceBundle.RootRelativePath
|
||||
}
|
||||
if plan.DestinationBundlePath == "" {
|
||||
plan.DestinationBundlePath = selection.DestinationBundlePath
|
||||
}
|
||||
}
|
||||
if isFixedPathDestination(destination) {
|
||||
plan.PathMapping = config.PathMappingFixed
|
||||
if options.DryRun && isDestructiveFixedPathAction(plan.Action) {
|
||||
warning := fixedPathReplacementWarning(plan)
|
||||
if jsonOutput {
|
||||
warnings = append(warnings, warning)
|
||||
} else if options.Stdout != nil {
|
||||
if err := writeWarnings(options.Stdout, []OutputWarning{warning}); err != nil {
|
||||
deferCloseDestination()
|
||||
closeBackend(sourceBackend)
|
||||
return err
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
if jsonOutput {
|
||||
result.Actions = append(result.Actions, runActionFromPlan(destination.Backend, plan, err))
|
||||
@@ -167,7 +208,6 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
|
||||
writePlanLine(options.Stdout, destination.Backend, plan, err)
|
||||
}
|
||||
if err != nil {
|
||||
deferCloseDestination()
|
||||
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
|
||||
summary.recordFailure()
|
||||
continue
|
||||
@@ -175,22 +215,20 @@ func runConfigWithBackendFactory(ctx context.Context, cfg config.Config, options
|
||||
summary.recordPlan(plan.Action)
|
||||
if !options.DryRun {
|
||||
if err := publish.Execute(ctx, req, plan); err != nil {
|
||||
deferCloseDestination()
|
||||
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
|
||||
summary.recordFailure()
|
||||
continue
|
||||
}
|
||||
if shouldNotify(plan.Action) {
|
||||
if err := notifier.Notify(ctx, notifyEvent(plan)); err != nil {
|
||||
deferCloseDestination()
|
||||
failures.add(pipeline.ID, destination.ID, destination.Backend, storage.DisplayPath(sourceBundle.RootRelativePath), err)
|
||||
summary.recordFailure()
|
||||
continue
|
||||
}
|
||||
}
|
||||
}
|
||||
deferCloseDestination()
|
||||
}
|
||||
deferCloseDestination()
|
||||
}
|
||||
closeBackend(sourceBackend)
|
||||
}
|
||||
@@ -231,10 +269,17 @@ func writePlanLine(w io.Writer, backend string, plan publish.Plan, planErr error
|
||||
if destinationID == "" {
|
||||
destinationID = "unknown"
|
||||
}
|
||||
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s action=error reason=%q\n", storage.DisplayPath(plan.BundlePath), destinationID, backend, planErr.Error())
|
||||
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=error reason=%q\n", storage.DisplayPath(plan.BundlePath), destinationID, backend, pathMappingSummary(plan), planErr.Error())
|
||||
return
|
||||
}
|
||||
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s action=%s outputs=%s reason=%q\n", storage.DisplayPath(plan.BundlePath), plan.DestinationID, backend, plan.Action, outputSummary(plan.Outputs), plan.Reason)
|
||||
fmt.Fprintf(w, " - bundle=%s destination=%s backend=%s%s action=%s outputs=%s reason=%q\n", storage.DisplayPath(plan.BundlePath), plan.DestinationID, backend, pathMappingSummary(plan), plan.Action, outputSummary(plan.Outputs), plan.Reason)
|
||||
}
|
||||
|
||||
func pathMappingSummary(plan publish.Plan) string {
|
||||
if plan.PathMapping != config.PathMappingFixed {
|
||||
return ""
|
||||
}
|
||||
return fmt.Sprintf(" path_mapping=fixed target=%s", storage.DisplayPath(plan.DestinationBundlePath))
|
||||
}
|
||||
|
||||
func writeErrorLine(w io.Writer, bundlePath, destinationID, backend string, err error) {
|
||||
@@ -255,6 +300,66 @@ func outputSummary(outputs []publish.Output) string {
|
||||
return strings.Join(paths, ",")
|
||||
}
|
||||
|
||||
type destinationBundleSelection struct {
|
||||
SourceBundle bundle.Bundle
|
||||
DestinationBundlePath string
|
||||
}
|
||||
|
||||
func selectDestinationBundles(destination config.Destination, bundles []bundle.Bundle) []destinationBundleSelection {
|
||||
if !isFixedPathDestination(destination) {
|
||||
selections := make([]destinationBundleSelection, 0, len(bundles))
|
||||
for _, sourceBundle := range bundles {
|
||||
selections = append(selections, destinationBundleSelection{
|
||||
SourceBundle: sourceBundle,
|
||||
DestinationBundlePath: sourceBundle.RootRelativePath,
|
||||
})
|
||||
}
|
||||
return selections
|
||||
}
|
||||
if len(bundles) == 0 {
|
||||
return nil
|
||||
}
|
||||
sourceBundle := newestBundle(bundles)
|
||||
return []destinationBundleSelection{{
|
||||
SourceBundle: sourceBundle,
|
||||
DestinationBundlePath: "",
|
||||
}}
|
||||
}
|
||||
|
||||
func newestBundle(bundles []bundle.Bundle) bundle.Bundle {
|
||||
if len(bundles) == 0 {
|
||||
return bundle.Bundle{}
|
||||
}
|
||||
sorted := append([]bundle.Bundle(nil), bundles...)
|
||||
sort.Slice(sorted, func(i, j int) bool {
|
||||
if sorted[i].Manifest.Created.Equal(sorted[j].Manifest.Created) {
|
||||
return sorted[i].RootRelativePath < sorted[j].RootRelativePath
|
||||
}
|
||||
return sorted[i].Manifest.Created.After(sorted[j].Manifest.Created)
|
||||
})
|
||||
return sorted[0]
|
||||
}
|
||||
|
||||
func isFixedPathDestination(destination config.Destination) bool {
|
||||
return destination.PathMap.Mode == config.PathMappingFixed
|
||||
}
|
||||
|
||||
func fixedPathSelectionWarning(pipelineID, destinationID string, selections []destinationBundleSelection, candidateCount int) OutputWarning {
|
||||
selected := "none"
|
||||
if len(selections) > 0 {
|
||||
selected = storage.DisplayPath(selections[0].SourceBundle.RootRelativePath)
|
||||
}
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed candidates=%d selected_bundle=%s destination_bundle=.", pipelineID, destinationID, candidateCount, selected)}
|
||||
}
|
||||
|
||||
func isDestructiveFixedPathAction(action publish.Action) bool {
|
||||
return action == publish.ActionReplaceOlder || action == publish.ActionForceReplace
|
||||
}
|
||||
|
||||
func fixedPathReplacementWarning(plan publish.Plan) OutputWarning {
|
||||
return OutputWarning{Message: fmt.Sprintf("pipeline=%s destination=%s path_mapping=fixed action=%s replaces destination root for selected_bundle=%s", plan.PipelineID, plan.DestinationID, plan.Action, storage.DisplayPath(plan.BundlePath))}
|
||||
}
|
||||
|
||||
func destinationIDs(destinations []config.Destination) []string {
|
||||
ids := make([]string, 0, len(destinations))
|
||||
for _, destination := range destinations {
|
||||
@@ -362,14 +467,16 @@ type runPipelineResult struct {
|
||||
}
|
||||
|
||||
type runActionResult struct {
|
||||
PipelineID string `json:"pipeline_id,omitempty"`
|
||||
DestinationID string `json:"destination_id"`
|
||||
Backend string `json:"backend"`
|
||||
BundleID string `json:"bundle_id,omitempty"`
|
||||
BundlePath string `json:"bundle_path"`
|
||||
Action string `json:"action"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Outputs []runOutputResult `json:"outputs"`
|
||||
PipelineID string `json:"pipeline_id,omitempty"`
|
||||
DestinationID string `json:"destination_id"`
|
||||
Backend string `json:"backend"`
|
||||
BundleID string `json:"bundle_id,omitempty"`
|
||||
BundlePath string `json:"bundle_path"`
|
||||
DestinationPath string `json:"destination_path"`
|
||||
PathMapping string `json:"path_mapping,omitempty"`
|
||||
Action string `json:"action"`
|
||||
Reason string `json:"reason,omitempty"`
|
||||
Outputs []runOutputResult `json:"outputs"`
|
||||
}
|
||||
|
||||
type runOutputResult struct {
|
||||
@@ -388,37 +495,42 @@ func runActionFromPlan(backend string, plan publish.Plan, planErr error) runActi
|
||||
destinationID = "unknown"
|
||||
}
|
||||
return runActionResult{
|
||||
PipelineID: plan.PipelineID,
|
||||
DestinationID: destinationID,
|
||||
Backend: backend,
|
||||
BundleID: plan.BundleID,
|
||||
BundlePath: storage.DisplayPath(plan.BundlePath),
|
||||
Action: "error",
|
||||
Reason: planErr.Error(),
|
||||
Outputs: []runOutputResult{},
|
||||
PipelineID: plan.PipelineID,
|
||||
DestinationID: destinationID,
|
||||
Backend: backend,
|
||||
BundleID: plan.BundleID,
|
||||
BundlePath: storage.DisplayPath(plan.BundlePath),
|
||||
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
||||
PathMapping: plan.PathMapping,
|
||||
Action: "error",
|
||||
Reason: planErr.Error(),
|
||||
Outputs: []runOutputResult{},
|
||||
}
|
||||
}
|
||||
return runActionResult{
|
||||
PipelineID: plan.PipelineID,
|
||||
DestinationID: plan.DestinationID,
|
||||
Backend: backend,
|
||||
BundleID: plan.BundleID,
|
||||
BundlePath: storage.DisplayPath(plan.BundlePath),
|
||||
Action: string(plan.Action),
|
||||
Reason: plan.Reason,
|
||||
Outputs: runOutputsFromPlan(plan.Outputs),
|
||||
PipelineID: plan.PipelineID,
|
||||
DestinationID: plan.DestinationID,
|
||||
Backend: backend,
|
||||
BundleID: plan.BundleID,
|
||||
BundlePath: storage.DisplayPath(plan.BundlePath),
|
||||
DestinationPath: storage.DisplayPath(plan.DestinationBundlePath),
|
||||
PathMapping: plan.PathMapping,
|
||||
Action: string(plan.Action),
|
||||
Reason: plan.Reason,
|
||||
Outputs: runOutputsFromPlan(plan.Outputs),
|
||||
}
|
||||
}
|
||||
|
||||
func errorAction(pipelineID, destinationID, backend, bundlePath string, err error) runActionResult {
|
||||
return runActionResult{
|
||||
PipelineID: pipelineID,
|
||||
DestinationID: destinationID,
|
||||
Backend: backend,
|
||||
BundlePath: storage.DisplayPath(bundlePath),
|
||||
Action: "error",
|
||||
Reason: err.Error(),
|
||||
Outputs: []runOutputResult{},
|
||||
PipelineID: pipelineID,
|
||||
DestinationID: destinationID,
|
||||
Backend: backend,
|
||||
BundlePath: storage.DisplayPath(bundlePath),
|
||||
DestinationPath: storage.DisplayPath(bundlePath),
|
||||
Action: "error",
|
||||
Reason: err.Error(),
|
||||
Outputs: []runOutputResult{},
|
||||
}
|
||||
}
|
||||
|
||||
@@ -445,6 +557,7 @@ type runSummary struct {
|
||||
forceReplace int
|
||||
skipped int
|
||||
failures int
|
||||
fixedPath int
|
||||
}
|
||||
|
||||
func (s *runSummary) recordPlan(action publish.Action) {
|
||||
@@ -465,12 +578,16 @@ func (s *runSummary) recordFailure() {
|
||||
s.failures++
|
||||
}
|
||||
|
||||
func (s *runSummary) recordFixedPath() {
|
||||
s.fixedPath++
|
||||
}
|
||||
|
||||
func (s runSummary) Line() string {
|
||||
status := "ok"
|
||||
if s.failures > 0 {
|
||||
status = "failed"
|
||||
}
|
||||
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d replace_older=%d force_replace=%d skipped=%d failed=%d dry_run=%t", status, s.planned, s.publishNew, s.replaceOlder, s.forceReplace, s.skipped, s.failures, s.dryRun)
|
||||
return fmt.Sprintf("Final status: %s planned=%d publish_new=%d replace_older=%d force_replace=%d skipped=%d failed=%d dry_run=%t fixed_path=%d", status, s.planned, s.publishNew, s.replaceOlder, s.forceReplace, s.skipped, s.failures, s.dryRun, s.fixedPath)
|
||||
}
|
||||
|
||||
type runSummaryResult struct {
|
||||
@@ -482,6 +599,7 @@ type runSummaryResult struct {
|
||||
Skipped int `json:"skipped"`
|
||||
Failed int `json:"failed"`
|
||||
DryRun bool `json:"dry_run"`
|
||||
FixedPath int `json:"fixed_path"`
|
||||
}
|
||||
|
||||
func (s runSummary) Result() runSummaryResult {
|
||||
@@ -498,6 +616,7 @@ func (s runSummary) Result() runSummaryResult {
|
||||
Skipped: s.skipped,
|
||||
Failed: s.failures,
|
||||
DryRun: s.dryRun,
|
||||
FixedPath: s.fixedPath,
|
||||
}
|
||||
}
|
||||
|
||||
|
||||
@@ -217,6 +217,329 @@ func TestRunPublishesNewLocalBundle(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunExplicitPreserveRelativePathMappingMatchesDefault(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "daily/report", testBundleOptions{})
|
||||
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingPreserveRelative)})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
assertFile(t, filepath.Join(destinationRoot, "daily", "report", "report.md"), "# Report\nSunny.\n")
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, "report.md")); !os.IsNotExist(err) {
|
||||
t.Fatalf("root report.md stat error = %v, want not exist", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathPublishesNewestBundleAtDestinationRoot(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{
|
||||
ID: "reports.old",
|
||||
Created: testutil.DefaultCreated,
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nOld.\n"},
|
||||
{Path: "summary.txt", Data: "Old summary\n"},
|
||||
},
|
||||
})
|
||||
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
|
||||
ID: "reports.new",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nNew.\n"},
|
||||
{Path: "summary.txt", Data: "New summary\n"},
|
||||
},
|
||||
})
|
||||
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, "new", "report.md")); !os.IsNotExist(err) {
|
||||
t.Fatalf("nested new report stat error = %v, want not exist", err)
|
||||
}
|
||||
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if destinationState.Source.Manifest.ID != "reports.new" {
|
||||
t.Fatalf("state source id = %q, want reports.new", destinationState.Source.Manifest.ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathTieBreaksByBundlePath(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "b", testBundleOptions{
|
||||
ID: "reports.b",
|
||||
Created: testutil.DefaultCreated,
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nB.\n"},
|
||||
{Path: "summary.txt", Data: "B summary\n"},
|
||||
},
|
||||
})
|
||||
writeSourceBundle(t, sourceRoot, "a", testBundleOptions{
|
||||
ID: "reports.a",
|
||||
Created: testutil.DefaultCreated,
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nA.\n"},
|
||||
{Path: "summary.txt", Data: "A summary\n"},
|
||||
},
|
||||
})
|
||||
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if destinationState.Source.Manifest.ID != "reports.a" {
|
||||
t.Fatalf("state source id = %q, want reports.a", destinationState.Source.Manifest.ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathDryRunReportsSelection(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{ID: "reports.old", Created: testutil.DefaultCreated})
|
||||
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{ID: "reports.new", Created: testutil.DefaultCreated.Add(time.Hour)})
|
||||
var stdout bytes.Buffer
|
||||
|
||||
err := Run(context.Background(), RunOptions{
|
||||
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
|
||||
DryRun: true,
|
||||
Stdout: &stdout,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
output := stdout.String()
|
||||
for _, want := range []string{
|
||||
"Warning: pipeline=reports destination=archive path_mapping=fixed candidates=2 selected_bundle=new destination_bundle=.",
|
||||
"bundle=new destination=archive backend=local path_mapping=fixed target=. action=publish_new",
|
||||
"fixed_path=1",
|
||||
} {
|
||||
if !strings.Contains(output, want) {
|
||||
t.Fatalf("stdout = %q, want substring %q", output, want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(output, "bundle=old destination=archive") {
|
||||
t.Fatalf("stdout = %q, older fixed candidate was planned", output)
|
||||
}
|
||||
if entries, err := os.ReadDir(destinationRoot); err != nil || len(entries) != 0 {
|
||||
t.Fatalf("destination entries = %v err=%v, want empty", entries, err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathDryRunWarnsForReplacement(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{
|
||||
ID: "reports.old",
|
||||
Created: testutil.DefaultCreated,
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nOld.\n"},
|
||||
{Path: "summary.txt", Data: "Old summary\n"},
|
||||
},
|
||||
})
|
||||
configPath := writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
|
||||
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
|
||||
t.Fatalf("first Run() error = %v", err)
|
||||
}
|
||||
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
|
||||
ID: "reports.new",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nNew.\n"},
|
||||
{Path: "summary.txt", Data: "New summary\n"},
|
||||
},
|
||||
})
|
||||
var stdout bytes.Buffer
|
||||
|
||||
err := Run(context.Background(), RunOptions{
|
||||
ConfigPath: configPath,
|
||||
DryRun: true,
|
||||
Stdout: &stdout,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
output := stdout.String()
|
||||
for _, want := range []string{
|
||||
"Warning: pipeline=reports destination=archive path_mapping=fixed action=replace_older replaces destination root for selected_bundle=new",
|
||||
"bundle=new destination=archive backend=local path_mapping=fixed target=. action=replace_older",
|
||||
} {
|
||||
if !strings.Contains(output, want) {
|
||||
t.Fatalf("stdout = %q, want substring %q", output, want)
|
||||
}
|
||||
}
|
||||
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
|
||||
}
|
||||
|
||||
func TestRunFixedPathReplacesOlderManagedState(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "old", testBundleOptions{
|
||||
ID: "reports.old",
|
||||
Created: testutil.DefaultCreated,
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nOld.\n"},
|
||||
{Path: "summary.txt", Data: "Old summary\n"},
|
||||
},
|
||||
})
|
||||
configPath := writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)
|
||||
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
|
||||
t.Fatalf("first Run() error = %v", err)
|
||||
}
|
||||
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nOld.\n")
|
||||
|
||||
writeSourceBundle(t, sourceRoot, "new", testBundleOptions{
|
||||
ID: "reports.new",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nNew.\n"},
|
||||
{Path: "summary.txt", Data: "New summary\n"},
|
||||
},
|
||||
})
|
||||
|
||||
if err := Run(context.Background(), RunOptions{ConfigPath: configPath}); err != nil {
|
||||
t.Fatalf("second Run() error = %v", err)
|
||||
}
|
||||
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nNew.\n")
|
||||
destinationState := readStateFile(t, filepath.Join(destinationRoot, storage.StateFileName))
|
||||
if destinationState.Source.Manifest.ID != "reports.new" {
|
||||
t.Fatalf("state source id = %q, want reports.new", destinationState.Source.Manifest.ID)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathSkipsWhenDestinationStateIsNewer(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
newer := testutil.ValidManifest(testutil.BundleOptions{
|
||||
ID: "reports.newer",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
})
|
||||
writeDestinationState(t, destinationRoot, "", newer)
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "report.md"), []byte("# Report\nExisting.\n"), 0o600); err != nil {
|
||||
t.Fatalf("write existing report: %v", err)
|
||||
}
|
||||
writeSourceBundle(t, sourceRoot, "older", testBundleOptions{
|
||||
ID: "reports.older",
|
||||
Created: testutil.DefaultCreated,
|
||||
})
|
||||
|
||||
var stdout bytes.Buffer
|
||||
err := Run(context.Background(), RunOptions{
|
||||
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
|
||||
Stdout: &stdout,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
if !strings.Contains(stdout.String(), "action=skip_destination_newer") {
|
||||
t.Fatalf("stdout = %q, want skip_destination_newer", stdout.String())
|
||||
}
|
||||
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nExisting.\n")
|
||||
}
|
||||
|
||||
func TestRunFixedPathFailsUnmanagedWithoutForce(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
writeSourceBundle(t, sourceRoot, "bundle", testBundleOptions{})
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "unmanaged.txt"), []byte("data"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged file: %v", err)
|
||||
}
|
||||
|
||||
err := Run(context.Background(), RunOptions{ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed)})
|
||||
if err == nil || !strings.Contains(err.Error(), "fail_unmanaged") {
|
||||
t.Fatalf("Run() error = %v, want unmanaged failure", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathForceReplacementStaysWithinDestinationRoot(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
parent := t.TempDir()
|
||||
destinationRoot := filepath.Join(parent, "latest")
|
||||
if err := os.MkdirAll(destinationRoot, 0o755); err != nil {
|
||||
t.Fatalf("mkdir destination: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(parent, "keep.txt"), []byte("keep"), 0o600); err != nil {
|
||||
t.Fatalf("write sibling: %v", err)
|
||||
}
|
||||
if err := os.WriteFile(filepath.Join(destinationRoot, "unmanaged.txt"), []byte("old"), 0o600); err != nil {
|
||||
t.Fatalf("write unmanaged: %v", err)
|
||||
}
|
||||
writeSourceBundle(t, sourceRoot, "bundle", testBundleOptions{})
|
||||
|
||||
err := Run(context.Background(), RunOptions{
|
||||
ConfigPath: writeLocalConfigWithPathMapping(t, sourceRoot, destinationRoot, config.PathMappingFixed),
|
||||
Force: true,
|
||||
})
|
||||
if err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
assertFile(t, filepath.Join(destinationRoot, "report.md"), "# Report\nSunny.\n")
|
||||
assertFile(t, filepath.Join(parent, "keep.txt"), "keep")
|
||||
if _, err := os.Stat(filepath.Join(destinationRoot, "unmanaged.txt")); !os.IsNotExist(err) {
|
||||
t.Fatalf("unmanaged stat error = %v, want removed", err)
|
||||
}
|
||||
}
|
||||
|
||||
func TestRunFixedPathRemoteBackendsUseBackendRoots(t *testing.T) {
|
||||
localSourceRoot := t.TempDir()
|
||||
writeSourceBundle(t, localSourceRoot, "old", testBundleOptions{
|
||||
ID: "reports.old",
|
||||
Created: testutil.DefaultCreated,
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nOld.\n"},
|
||||
{Path: "summary.txt", Data: "Old summary\n"},
|
||||
},
|
||||
})
|
||||
writeSourceBundle(t, localSourceRoot, "new", testBundleOptions{
|
||||
ID: "reports.new",
|
||||
Created: testutil.DefaultCreated.Add(time.Hour),
|
||||
Files: []testFile{
|
||||
{Path: "report.md", Data: "# Report\nNew.\n"},
|
||||
{Path: "summary.txt", Data: "New summary\n"},
|
||||
},
|
||||
})
|
||||
s3Destination := fake.New()
|
||||
sshDestination := fake.New()
|
||||
cfg := config.Config{Pipelines: []config.Pipeline{{
|
||||
ID: "reports",
|
||||
Source: config.Backend{Backend: config.BackendLocal, Path: localSourceRoot},
|
||||
Destinations: []config.Destination{
|
||||
{
|
||||
ID: "object-latest",
|
||||
Backend: config.BackendS3,
|
||||
Endpoint: "http://s3.test",
|
||||
Bucket: "destination-bucket",
|
||||
PathMap: config.PathMapping{Mode: config.PathMappingFixed},
|
||||
},
|
||||
{
|
||||
ID: "ssh-latest",
|
||||
Backend: config.BackendSSH,
|
||||
Host: "ssh.test",
|
||||
Path: "/latest",
|
||||
PathMap: config.PathMapping{Mode: config.PathMappingFixed},
|
||||
},
|
||||
},
|
||||
}}}
|
||||
config.ApplyDefaults(&cfg)
|
||||
provider := fakeBackendFactoryProvider(t, map[string]storage.Backend{
|
||||
"s3:destination-bucket": s3Destination,
|
||||
"ssh:/latest": sshDestination,
|
||||
})
|
||||
|
||||
if err := runConfigWithBackendFactory(context.Background(), cfg, RunOptions{}, provider); err != nil {
|
||||
t.Fatalf("Run() error = %v", err)
|
||||
}
|
||||
assertFakeFile(t, s3Destination, "report.md", "# Report\nNew.\n")
|
||||
assertFakeFile(t, s3Destination, "summary.txt", "New summary\n")
|
||||
assertFakeMissing(t, s3Destination, "new/report.md")
|
||||
assertFakeFile(t, sshDestination, "report.md", "# Report\nNew.\n")
|
||||
assertFakeFile(t, sshDestination, "summary.txt", "New summary\n")
|
||||
assertFakeMissing(t, sshDestination, "new/report.md")
|
||||
}
|
||||
|
||||
func TestRunNotifiesAfterPublication(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
@@ -924,6 +1247,23 @@ pipelines:
|
||||
`)
|
||||
}
|
||||
|
||||
func writeLocalConfigWithPathMapping(t *testing.T, sourceRoot, destinationRoot, mode string) string {
|
||||
t.Helper()
|
||||
return writeConfigFile(t, `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: `+sourceRoot+`
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: `+destinationRoot+`
|
||||
path_mapping:
|
||||
mode: `+mode+`
|
||||
`)
|
||||
}
|
||||
|
||||
func writeLocalConfigWithMarkdownTransform(t *testing.T, sourceRoot, destinationRoot string, publishSource, publishHTML bool, mode, input string) string {
|
||||
t.Helper()
|
||||
enabled := publishHTML
|
||||
|
||||
@@ -620,6 +620,61 @@ func TestExecuteRunJSONDryRun(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunJSONDryRunReportsFixedPathMapping(t *testing.T) {
|
||||
sourceRoot := t.TempDir()
|
||||
destinationRoot := t.TempDir()
|
||||
testutil.WriteSourceBundle(t, sourceRoot, "", testutil.BundleOptions{})
|
||||
configPath := filepath.Join(t.TempDir(), "config.yml")
|
||||
if err := os.WriteFile(configPath, []byte(`
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: `+sourceRoot+`
|
||||
destinations:
|
||||
- id: latest
|
||||
backend: local
|
||||
path: `+destinationRoot+`
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
`), 0o600); err != nil {
|
||||
t.Fatalf("write config: %v", err)
|
||||
}
|
||||
|
||||
var stdout, stderr bytes.Buffer
|
||||
|
||||
code := Execute(context.Background(), []string{"run", "--config", configPath, "--dry-run", "--format", "json"}, &stdout, &stderr)
|
||||
|
||||
if code != exitOK {
|
||||
t.Fatalf("exit code = %d, want %d; stderr = %q", code, exitOK, stderr.String())
|
||||
}
|
||||
envelope := decodeEnvelope(t, &stdout)
|
||||
warnings, ok := envelope["warnings"].([]any)
|
||||
if !ok || len(warnings) != 1 {
|
||||
t.Fatalf("warnings = %#v, want one fixed-path warning", envelope["warnings"])
|
||||
}
|
||||
warning, ok := warnings[0].(map[string]any)
|
||||
if !ok || !strings.Contains(fmt.Sprint(warning["message"]), "path_mapping=fixed candidates=1 selected_bundle=.") {
|
||||
t.Fatalf("warning = %#v, want fixed-path selection warning", warnings[0])
|
||||
}
|
||||
result := envelopeResult(t, envelope)
|
||||
actions, ok := result["actions"].([]any)
|
||||
if !ok || len(actions) != 1 {
|
||||
t.Fatalf("actions = %#v, want one action", result["actions"])
|
||||
}
|
||||
action, ok := actions[0].(map[string]any)
|
||||
if !ok || action["path_mapping"] != "fixed" || action["destination_path"] != "." || action["action"] != "publish_new" {
|
||||
t.Fatalf("action = %#v, want fixed publish_new at root", actions[0])
|
||||
}
|
||||
summary, ok := result["summary"].(map[string]any)
|
||||
if !ok || summary["fixed_path"] != float64(1) {
|
||||
t.Fatalf("summary = %#v, want fixed_path 1", result["summary"])
|
||||
}
|
||||
if stderr.Len() != 0 {
|
||||
t.Fatalf("stderr = %q, want empty", stderr.String())
|
||||
}
|
||||
}
|
||||
|
||||
func TestExecuteRunJSONWarningsAreStructured(t *testing.T) {
|
||||
name := "DISTRIBUTOR_TEST_CLI_JSON_SECRET"
|
||||
t.Setenv(name, "process-value")
|
||||
|
||||
@@ -32,6 +32,7 @@ type Destination struct {
|
||||
SSH SSH `yaml:",inline"`
|
||||
Publish *PublishPolicy `yaml:"publish"`
|
||||
Transform Transform `yaml:"transform"`
|
||||
PathMap PathMapping `yaml:"path_mapping"`
|
||||
Transfer TransferPolicy `yaml:"transfer"`
|
||||
}
|
||||
|
||||
@@ -80,6 +81,10 @@ type MarkdownToHTML struct {
|
||||
Input string `yaml:"input"`
|
||||
}
|
||||
|
||||
type PathMapping struct {
|
||||
Mode string `yaml:"mode"`
|
||||
}
|
||||
|
||||
type TransferPolicy struct {
|
||||
OnDestinationSame string `yaml:"on_destination_same"`
|
||||
OnDestinationOlder string `yaml:"on_destination_older"`
|
||||
|
||||
@@ -25,6 +25,11 @@ const (
|
||||
TransformModeIndex = transform.MarkdownModeIndex
|
||||
)
|
||||
|
||||
const (
|
||||
PathMappingPreserveRelative = "preserve_relative"
|
||||
PathMappingFixed = "fixed"
|
||||
)
|
||||
|
||||
const DefaultS3Region = "us-east-1"
|
||||
|
||||
func ApplyDefaults(cfg *Config) {
|
||||
@@ -43,6 +48,9 @@ func ApplyDefaults(cfg *Config) {
|
||||
if destination.Transform.MarkdownToHTML != nil && destination.Transform.MarkdownToHTML.Mode == "" {
|
||||
destination.Transform.MarkdownToHTML.Mode = TransformModeSidecar
|
||||
}
|
||||
if destination.PathMap.Mode == "" {
|
||||
destination.PathMap.Mode = PathMappingPreserveRelative
|
||||
}
|
||||
if destination.Transfer.OnDestinationSame == "" {
|
||||
destination.Transfer.OnDestinationSame = TransferActionSkip
|
||||
}
|
||||
|
||||
@@ -146,6 +146,44 @@ pipelines:
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileDefaultsPathMappingToPreserveRelative(t *testing.T) {
|
||||
cfg := loadConfig(t, `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /source
|
||||
destinations:
|
||||
- id: archive
|
||||
backend: local
|
||||
path: /destination
|
||||
`)
|
||||
|
||||
if got, want := cfg.Pipelines[0].Destinations[0].PathMap.Mode, PathMappingPreserveRelative; got != want {
|
||||
t.Fatalf("path mapping mode = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileAcceptsFixedPathMapping(t *testing.T) {
|
||||
cfg := loadConfig(t, `
|
||||
pipelines:
|
||||
- id: reports
|
||||
source:
|
||||
backend: local
|
||||
path: /source
|
||||
destinations:
|
||||
- id: latest
|
||||
backend: local
|
||||
path: /destination/latest
|
||||
path_mapping:
|
||||
mode: fixed
|
||||
`)
|
||||
|
||||
if got, want := cfg.Pipelines[0].Destinations[0].PathMap.Mode, PathMappingFixed; got != want {
|
||||
t.Fatalf("path mapping mode = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestLoadFileValidBackendConfigs(t *testing.T) {
|
||||
tests := map[string]string{
|
||||
"local": `
|
||||
@@ -518,6 +556,7 @@ func TestExampleConfigsLoad(t *testing.T) {
|
||||
"../../examples/local-html.yml",
|
||||
"../../examples/local-index.yml",
|
||||
"../../examples/fan-out.yml",
|
||||
"../../examples/archive-and-latest.yml",
|
||||
"../../examples/ssh-destination.yml",
|
||||
"../../examples/s3-destination.yml",
|
||||
} {
|
||||
|
||||
@@ -58,6 +58,7 @@ func Validate(cfg Config) error {
|
||||
|
||||
errs = validateDestinationBackend(errs, destinationContext, destination)
|
||||
errs = validatePublishTransformPolicy(errs, destinationContext, destination.Publish, destination.Transform)
|
||||
errs = validatePathMapping(errs, destinationContext+".path_mapping", destination.PathMap)
|
||||
errs = validateTransferPolicy(errs, destinationContext+".transfer", destination.Transfer)
|
||||
}
|
||||
}
|
||||
@@ -171,6 +172,13 @@ func ValidatePublishTransformPolicy(publish PublishPolicy, transform Transform)
|
||||
return nil
|
||||
}
|
||||
|
||||
func validatePathMapping(errs ValidationErrors, context string, mapping PathMapping) ValidationErrors {
|
||||
if mapping.Mode != PathMappingPreserveRelative && mapping.Mode != PathMappingFixed {
|
||||
errs = append(errs, context+".mode must be "+PathMappingPreserveRelative+" or "+PathMappingFixed)
|
||||
}
|
||||
return errs
|
||||
}
|
||||
|
||||
func validateTransferPolicy(errs ValidationErrors, context string, policy TransferPolicy) ValidationErrors {
|
||||
if policy.OnDestinationSame != TransferActionSkip && policy.OnDestinationSame != TransferActionFail {
|
||||
errs = append(errs, context+".on_destination_same must be skip or fail")
|
||||
|
||||
@@ -70,6 +70,40 @@ func TestValidateAcceptsForceReplacementTransferActions(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestValidatePathMapping(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
mode string
|
||||
wantErr bool
|
||||
}{
|
||||
{name: "preserve relative", mode: PathMappingPreserveRelative},
|
||||
{name: "fixed", mode: PathMappingFixed},
|
||||
{name: "invalid", mode: "archive", wantErr: true},
|
||||
}
|
||||
for _, tt := range tests {
|
||||
t.Run(tt.name, func(t *testing.T) {
|
||||
cfg := Config{Pipelines: []Pipeline{{
|
||||
ID: "reports",
|
||||
Source: Backend{Backend: BackendLocal, Path: "/source"},
|
||||
Destinations: []Destination{{
|
||||
ID: "archive",
|
||||
Backend: BackendLocal,
|
||||
Path: "/destination",
|
||||
PathMap: PathMapping{Mode: tt.mode},
|
||||
}},
|
||||
}}}
|
||||
ApplyDefaults(&cfg)
|
||||
err := Validate(cfg)
|
||||
if tt.wantErr && err == nil {
|
||||
t.Fatal("Validate() error = nil, want error")
|
||||
}
|
||||
if !tt.wantErr && err != nil {
|
||||
t.Fatalf("Validate() error = %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
type publishTransformPolicyCase struct {
|
||||
name string
|
||||
publish PublishPolicy
|
||||
|
||||
@@ -30,6 +30,7 @@ type Request struct {
|
||||
SourceBackend storage.Backend
|
||||
DestinationBackend storage.Backend
|
||||
DestinationBundlePath string
|
||||
PathMapping string
|
||||
Publish config.PublishPolicy
|
||||
Transform config.Transform
|
||||
Transformers TransformerResolver
|
||||
@@ -48,6 +49,7 @@ type Plan struct {
|
||||
BundleID string
|
||||
BundlePath string
|
||||
DestinationBundlePath string
|
||||
PathMapping string
|
||||
Action Action
|
||||
Reason string
|
||||
Force bool
|
||||
@@ -77,7 +79,7 @@ func Build(ctx context.Context, req Request) (Plan, error) {
|
||||
if err != nil {
|
||||
return Plan{}, err
|
||||
}
|
||||
comparison := state.Compare(req.SourceBundle.Manifest, req.PipelineID, req.DestinationID, status)
|
||||
comparison := compareDestination(req, status)
|
||||
action, reason := actionForComparison(comparison, req.Transfer, req.Force)
|
||||
plan := Plan{
|
||||
PipelineID: req.PipelineID,
|
||||
@@ -85,6 +87,7 @@ func Build(ctx context.Context, req Request) (Plan, error) {
|
||||
BundleID: req.SourceBundle.Manifest.ID,
|
||||
BundlePath: req.SourceBundle.RootRelativePath,
|
||||
DestinationBundlePath: req.DestinationBundlePath,
|
||||
PathMapping: req.PathMapping,
|
||||
Action: action,
|
||||
Reason: reason,
|
||||
Force: action == ActionForceReplace,
|
||||
@@ -116,6 +119,21 @@ func validateRequest(req Request) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
func compareDestination(req Request, status state.DestinationStatus) state.Comparison {
|
||||
comparison := state.Compare(req.SourceBundle.Manifest, req.PipelineID, req.DestinationID, status)
|
||||
if req.PathMapping != config.PathMappingFixed || comparison.Outcome != state.OutcomeDifferentSourceConflict || status.State == nil {
|
||||
return comparison
|
||||
}
|
||||
destinationManifest := status.State.Source.Manifest
|
||||
if destinationManifest.Created.Before(req.SourceBundle.Manifest.Created) {
|
||||
return state.Comparison{Outcome: state.OutcomeDestinationOlder, Reason: "fixed destination source is older than selected source"}
|
||||
}
|
||||
if destinationManifest.Created.After(req.SourceBundle.Manifest.Created) {
|
||||
return state.Comparison{Outcome: state.OutcomeDestinationNewer, Reason: "fixed destination source is newer than selected source"}
|
||||
}
|
||||
return comparison
|
||||
}
|
||||
|
||||
func actionForComparison(comparison state.Comparison, transfer config.TransferPolicy, force bool) (Action, string) {
|
||||
switch comparison.Outcome {
|
||||
case state.OutcomeDestinationAbsent:
|
||||
|
||||
Reference in New Issue
Block a user