eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Implement catalog force replacement

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-19 16:39:01 +00:00
parent 4e673dda76
commit 9a2eaf8e5e
13 changed files with 307 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files

12
docs/operations.md
View File

@@ -166,12 +166,12 @@ Review these action labels before publishing:
- `replace_takeover`: destination state is valid managed state and `takeover.mode` allows replacement across an identity, source, or shared-root output-owner mismatch.
- `skip_same`: destination state already matches the source.
- `skip_destination_newer`: destination state is newer than the source and is skipped.
- `force_replace`: destructive replacement selected because `--force` is present and policy permits it.
- `force_replace`: destructive catalog replacement selected because `--force` is present for unmanaged content, a planned unmanaged path collision, invalid state, or unsupported future state.
- `error`: planning or execution failed for that destination.
Fixed destinations add fixed-path warnings during dry runs, including the selected source bundle and replacement warnings when the destination root would be replaced.
Text and JSON summaries count `publish_new`, `replace_older`, `replace_conflict`, `replace_newer`, `replace_takeover`, `force_replace`, skipped, and failed destinations separately. Takeover action records include the configured takeover mode and the conflict reason. JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.
Text and JSON summaries count `force_replace` separately from normal publish and catalog workflow actions. JSON output includes warnings, pipeline summaries, destination action records, output records, URLs when configured, final counters, and partial failure details. Fatal setup failures such as unreadable config or invalid secrets do not produce a JSON result document.
## Forced Replacement Workflow
@@ -182,11 +182,11 @@ go run ./cmd/distributor run --config <config-path> --dry-run --force
go run ./cmd/distributor run --config <config-path> --force
```
Forced replacement can claim unmanaged non-empty destination paths and is reserved for exceptional destructive replacement. Valid managed newer state and valid managed conflict state use `replace_newer` and `replace_conflict` when the corresponding transfer policy allows replacement; they do not require `--force`.
Forced replacement can claim a non-empty destination path with no valid `.distributor.json`, replace planned output paths that collide with storage content not recorded in valid catalog state, and recover from invalid or unsupported future destination state. It is reserved for exceptional destructive replacement. Valid catalog-managed additive upserts and replacement workflow publishes do not require `--force`.
Forced replacement deletes the current destination bundle path before writing outputs and state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the backend root, so a forced replacement can clear that configured root.
Forced replacement deletes the current destination bundle path before writing planned outputs and schema version `4` catalog state. It does not delete parent paths, sibling paths, or storage outside the destination bundle path. For fixed destinations, the destination bundle path is the configured backend root, so a forced replacement can clear that configured root. Dry-run text and JSON output report that root as `target=.` or `destination_path: "."`.
For shared-root destinations, forced replacement also deletes the configured destination bundle path before writing new shared-root state. This removes unrelated owners inside that destination root. Preview with `--dry-run --force` and confirm the destination path before applying.
Preview with `--dry-run --force` and confirm the destination path before applying.
`--force` applies only to the current invocation. There is no config field that enables forced replacement by default.
@@ -272,7 +272,7 @@ S3 execution uses the AWS SDK for Go v2. See [S3-Compatible Storage Integration]
When explicit S3 credential variable names are configured, both must resolve to non-empty values through the process environment or `secrets.directory`. When omitted, the AWS SDK default credential chain is used as-is.
Normal single-owner replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Shared-root replacement deletes only current-owner omitted output objects and rewrites the shared state object. Same-source merge publication retains omitted managed objects and may overwrite existing managed objects. Takeover replacement does not retain omitted outputs through merge reconciliation. Forced replacement deletes objects under the bounded destination bundle prefix. Distributor does not manage bucket versioning or delete markers.
Normal managed replacement and failed-write cleanup delete only managed output objects recorded in `.distributor.json` plus the state object. Forced replacement deletes objects under the bounded destination bundle prefix and then writes schema version `4` catalog state. Distributor does not manage bucket versioning or delete markers.
## Secrets Operation