Close completed catalog roadmap
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
All checks were successful
ci/woodpecker/tag/release Pipeline was successful
This commit is contained in:
@@ -1,283 +0,0 @@
|
||||
# Catalog State And Destination Workflow Roadmap
|
||||
|
||||
This roadmap records the intended first-class state model and destination
|
||||
workflow configuration for replacement and additive publication workflows.
|
||||
|
||||
Current `single_owner` and `shared_root` state are oriented around comparing a
|
||||
destination owner to one latest source manifest. That works well for replacement
|
||||
workflows, where a producer maintains a curated source of truth and
|
||||
`distributor` makes a destination match it. It is a poor fit for additive
|
||||
workflows, where each run contributes new or updated output paths while
|
||||
preserving unrelated managed outputs from previous runs and other pipelines.
|
||||
|
||||
The target model is a current-state catalog: `.distributor.json` records the
|
||||
currently managed output paths at a destination root, and each output record
|
||||
stores its current owner, compact source identity, content digest, and update
|
||||
metadata. It is not intended to be an audit log.
|
||||
|
||||
Both replacement and additive workflows should use the same catalog state shape.
|
||||
The workflow choice is runtime policy from destination config, not persisted
|
||||
state.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
- Add one destination state mode named `catalog` for both replacement and
|
||||
additive workflows.
|
||||
- Catalog state writes `.distributor.json` schema version `4`.
|
||||
- Add a destination-level `workflow` setting with accepted values
|
||||
`replacement` and `additive`.
|
||||
- Default `workflow` to `additive` because it is the least destructive workflow.
|
||||
- `workflow` replaces the normal user-facing need to combine `state.mode`,
|
||||
`reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the
|
||||
two primary workflows.
|
||||
- `workflow: replacement` treats the current planned outputs as the authoritative
|
||||
managed output set for the destination scope and removes previously managed
|
||||
outputs in that scope when they are no longer planned.
|
||||
- `workflow: additive` writes or overwrites the planned output paths and retains
|
||||
unrelated managed outputs.
|
||||
- If a planned path already exists as a managed output, the new publication may
|
||||
overwrite it and becomes that path's current owner.
|
||||
- If a planned path exists in storage but is not recorded in valid catalog state,
|
||||
it remains unmanaged content and must not be adopted implicitly.
|
||||
- If an output path changes owner, preserve the output record's existing
|
||||
`created_at` and update only `updated_at`.
|
||||
- Catalog state records current ownership only. It does not retain historical
|
||||
owners, historical sources, or old versions of overwritten output records.
|
||||
- `pipeline_id` and `destination_id` are stored separately on each output
|
||||
record. They are not concatenated into one owner string.
|
||||
- Catalog state does not include top-level `owners`; owners are derivable from
|
||||
the output records.
|
||||
- Catalog state does not include top-level `sources`; compact source identity is
|
||||
stored directly on each output record.
|
||||
- Catalog state does not record whether the last run used `replacement` or
|
||||
`additive`. Workflow is execution policy, and persisting it would create
|
||||
drift risk if config changes later.
|
||||
- Legacy destination policy fields should be rejected outright in the new
|
||||
workflow config model. Do not retain aliases for `state`, `reconciliation`,
|
||||
`takeover`, or `transfer`.
|
||||
|
||||
## Destination Workflow Semantics
|
||||
|
||||
The user-facing destination config should express intent directly:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-latest
|
||||
backend: local
|
||||
path: /srv/reports/weather/latest
|
||||
workflow: additive
|
||||
```
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-archive
|
||||
backend: local
|
||||
path: /srv/reports/weather/archive
|
||||
workflow: replacement
|
||||
```
|
||||
|
||||
`workflow` is orthogonal to backend config, path mapping, publish source/HTML
|
||||
selection, transforms, links, and retention.
|
||||
|
||||
### Replacement Workflow
|
||||
|
||||
A replacement destination is for producers that maintain a curated source of
|
||||
truth and expect `distributor` to make the destination's managed scope match the
|
||||
current publication.
|
||||
|
||||
For a destination configured with `workflow: replacement`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each written output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs in the destination scope that are not present in the current
|
||||
plan are deleted and removed from catalog state;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- the catalog state shape remains the same as additive workflow state.
|
||||
|
||||
### Additive Workflow
|
||||
|
||||
An additive destination is for producers that routinely contribute outputs to a
|
||||
shared destination root.
|
||||
|
||||
For a destination configured with `workflow: additive`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each overwritten output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs not present in the current plan are retained;
|
||||
- unrelated managed outputs from other pipelines or destinations are retained;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- pruning can select retained managed outputs by `updated_at` and, when useful,
|
||||
by `pipeline_id` and/or `destination_id`.
|
||||
|
||||
Replacement workflow treats the current publication as the desired managed
|
||||
output set for a destination scope. Additive workflow treats the current
|
||||
publication as a patch to the catalog of currently managed outputs.
|
||||
|
||||
## Destination State Schema Version 4
|
||||
|
||||
Catalog state should use this top-level shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 4,
|
||||
"distributor_version": "dev",
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"state": {
|
||||
"mode": "catalog"
|
||||
},
|
||||
"outputs": []
|
||||
}
|
||||
```
|
||||
|
||||
Top-level fields:
|
||||
|
||||
- `schema_version`: required. Value `4` for catalog state.
|
||||
- `distributor_version`: optional diagnostic version string.
|
||||
- `created_at`: required RFC3339 timestamp for when the catalog state file was
|
||||
first created.
|
||||
- `updated_at`: required RFC3339 timestamp for the latest catalog state update.
|
||||
- `state.mode`: required. Value `catalog`.
|
||||
- `outputs`: required array of currently managed output records.
|
||||
|
||||
Catalog state should not include top-level `pipeline_id`, `destination_id`,
|
||||
`published_at`, `workflow`, `owners`, `sources`, or a full source manifest.
|
||||
Those concepts belong on output records or in configuration.
|
||||
|
||||
## Output Record Schema
|
||||
|
||||
Each output record should be self-contained enough to support current
|
||||
ownership, pruning, repair, bitrot checks, and basic provenance without a
|
||||
separate owner or source catalog.
|
||||
|
||||
```json
|
||||
{
|
||||
"path": "tomorrow/index.html",
|
||||
"pipeline_id": "weatherreporter.daily",
|
||||
"destination_id": "latest-html",
|
||||
"source": {
|
||||
"id": "weatherreporter.tomorrow",
|
||||
"digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
||||
"created": "2026-06-19T12:00:00Z"
|
||||
},
|
||||
"kind": "generated",
|
||||
"source_path": "report.md",
|
||||
"transform": "markdown_to_html",
|
||||
"sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
|
||||
"size": 12345,
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"url": "https://reports.example.com/weather/tomorrow/"
|
||||
}
|
||||
```
|
||||
|
||||
Required output fields:
|
||||
|
||||
- `path`: destination-relative managed output path.
|
||||
- `pipeline_id`: pipeline that most recently wrote this output path.
|
||||
- `destination_id`: destination that most recently wrote this output path.
|
||||
- `source`: compact identity of the source bundle that produced this output.
|
||||
- `source.id`: source manifest id.
|
||||
- `source.digest`: source manifest digest.
|
||||
- `source.created`: source manifest creation timestamp, RFC3339.
|
||||
- `kind`: `source` or `generated`.
|
||||
- `sha256`: digest of the output bytes.
|
||||
- `size`: output byte size.
|
||||
- `created_at`: RFC3339 timestamp for when this output path first became
|
||||
managed in catalog state.
|
||||
- `updated_at`: RFC3339 timestamp for when this output path was most recently
|
||||
written or updated.
|
||||
|
||||
Optional output fields:
|
||||
|
||||
- `source_path`: present only for generated outputs; source manifest path used
|
||||
to produce the generated output.
|
||||
- `transform`: present only for generated outputs; transform name used to
|
||||
produce the generated output.
|
||||
- `url`: present only when destination link configuration produces a public URL
|
||||
for this output.
|
||||
|
||||
For copied source outputs, `source_path` and `transform` should be omitted. The
|
||||
output `path` is already the copied source artifact path.
|
||||
|
||||
## Useful Properties
|
||||
|
||||
The catalog shape supports both primary workflows without unnecessary
|
||||
normalization:
|
||||
|
||||
- current owner of each path is explicit;
|
||||
- current source identity for each path is explicit;
|
||||
- bitrot checks can compare storage bytes to `sha256`;
|
||||
- pruning can use `updated_at`, optionally scoped by `pipeline_id` and
|
||||
`destination_id`;
|
||||
- repair can remove missing managed output records without consulting an owner
|
||||
or source table;
|
||||
- overwriting an output path updates one output record in place;
|
||||
- no orphaned top-level owner/source records need to be maintained.
|
||||
|
||||
The schema intentionally avoids storing a full source manifest for every output.
|
||||
The source manifest remains the producer-to-distributor validation contract, but
|
||||
catalog state only needs compact source identity for currently managed outputs.
|
||||
|
||||
## Relationship To Existing State And Config Modes
|
||||
|
||||
Existing state modes remain current behavior until catalog mode is implemented:
|
||||
|
||||
- `single_owner` schema version `2` supports one owner for a destination bundle
|
||||
path.
|
||||
- `shared_root` schema version `3` supports multiple owners in one destination
|
||||
root but still keeps owner records with latest source manifests.
|
||||
- `catalog` schema version `4` should support replacement and additive
|
||||
current-state ownership without top-level owner or source catalogs.
|
||||
|
||||
The intended user-facing config should move toward `workflow: replacement` and
|
||||
`workflow: additive` instead of requiring ordinary users to combine
|
||||
`state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict
|
||||
settings.
|
||||
|
||||
Because the project is still alpha pre-release, catalog implementation should be
|
||||
a clean break:
|
||||
|
||||
- remove legacy write paths for current `single_owner` and `shared_root` state;
|
||||
- do not implement migration from schema versions `1`, `2`, or `3`;
|
||||
- do not preserve backwards compatibility for legacy destination state;
|
||||
- when a configured destination writes successfully, write schema version `4`
|
||||
catalog state;
|
||||
- if an existing `.distributor.json` has schema version lower than `4`, treat it
|
||||
as superseded legacy state for planning purposes and overwrite it according to
|
||||
the configured workflow, without attempting conversion.
|
||||
|
||||
For the first catalog run against superseded legacy state:
|
||||
|
||||
- `workflow: replacement` may clear the bounded destination root before writing
|
||||
the planned outputs and schema version `4` catalog state.
|
||||
- `workflow: additive` may overwrite only the planned output paths, then write
|
||||
schema version `4` catalog state containing those planned outputs.
|
||||
- unplanned files left behind by an additive run against superseded legacy state
|
||||
are not recorded in catalog state and are treated as unmanaged content by
|
||||
later catalog runs.
|
||||
|
||||
The target direction is that both primary workflows use catalog state.
|
||||
|
||||
## Prune And Reconcile-State Scope
|
||||
|
||||
Catalog maintenance commands should initially keep the existing ownership
|
||||
selector model:
|
||||
|
||||
- `prune --pipeline <id> --destination <id>` operates only on outputs currently
|
||||
owned by that pipeline/destination.
|
||||
- `reconcile-state --pipeline <id> --destination <id>` repairs only outputs
|
||||
currently owned by that pipeline/destination.
|
||||
- `reconcile-state --all-owners` repairs all output records in the catalog.
|
||||
- `prune` remains scoped to the selected pipeline/destination owner only in the
|
||||
initial catalog implementation.
|
||||
|
||||
Do not add catalog-specific selectors in the initial implementation, such as
|
||||
`--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they
|
||||
are not required to make additive workflow first-class.
|
||||
@@ -93,10 +93,10 @@ internal docs.
|
||||
- Current upload status, queue, and idempotency state are memory-only.
|
||||
- Producers submit complete tar or gzip-compressed tar source bundles today.
|
||||
- Producers do not choose destination ids, destination paths, transforms, links,
|
||||
publish policy, transfer policy, storage backends, reconciliation mode, state
|
||||
mode, or retention policy through upload requests.
|
||||
publish policy, destination workflow, storage backends, or retention policy
|
||||
through upload requests.
|
||||
- Source manifests remain free of routing, destination, transform, credential,
|
||||
reconciliation, state, and retention data.
|
||||
workflow, state, and retention data.
|
||||
- Public access policy, TLS termination, and rate limiting belong outside
|
||||
`distributor` unless a future implementation changes that boundary.
|
||||
- `distributor` is not a broad storage synchronization tool unless a future
|
||||
|
||||
@@ -1,399 +0,0 @@
|
||||
# Catalog Follow-Up Implementation Roadmap
|
||||
|
||||
This is the active staged plan for closing the remaining catalog-state
|
||||
implementation gaps found after the schema version `4` catalog workflow landed.
|
||||
It supersedes the earlier catalog implementation plan.
|
||||
|
||||
The current code already supports destination `workflow: additive` and
|
||||
`workflow: replacement`, writes catalog state schema version `4`, rejects legacy
|
||||
destination policy YAML fields, and routes publish execution through catalog
|
||||
planning for normal runs. The remaining work is to make the implemented behavior
|
||||
release-ready: restore a meaningful catalog `--force` path, synchronize current
|
||||
docs and examples, and remove legacy single-owner/shared-root implementation
|
||||
debt.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
- Destination config accepts `workflow: additive` and `workflow: replacement`.
|
||||
- Omitted `workflow` defaults to `additive`.
|
||||
- Destination YAML fields `state`, `reconciliation`, `takeover`, and `transfer`
|
||||
are rejected by strict YAML decoding.
|
||||
- Successful publish writes `.distributor.json` schema version `4` with
|
||||
`state.mode: catalog`.
|
||||
- Schema versions `1`, `2`, and `3` are treated as superseded legacy state for
|
||||
publish planning.
|
||||
- `run --force` remains in the CLI and docs, but catalog publish planning does
|
||||
not currently select `force_replace`.
|
||||
- Current docs and examples still describe legacy config/state behavior in
|
||||
several places.
|
||||
- Legacy single-owner and shared-root state types, tests, and execution helpers
|
||||
remain in the codebase even though normal publish planning no longer uses
|
||||
them.
|
||||
|
||||
## Implementation Principles
|
||||
|
||||
- Preserve the catalog state shape defined in `docs/roadmap/catalog.md`.
|
||||
- Keep `workflow` as runtime config policy; do not persist it in
|
||||
`.distributor.json`.
|
||||
- Keep unmanaged content protected by default.
|
||||
- Keep `--force` explicit, per-run, dry-runnable, and bounded to the configured
|
||||
destination bundle path.
|
||||
- Do not reintroduce legacy config aliases or compatibility migration.
|
||||
- Keep backend adapters unaware of catalog, workflow, and force policy.
|
||||
- Update current-behavior docs in the same stage that makes the described
|
||||
behavior true.
|
||||
|
||||
## Active Implementation Stages
|
||||
|
||||
## Stage 1: Catalog Force Planning And Execution
|
||||
|
||||
Goal: make `run --force` meaningful for catalog workflows while keeping normal
|
||||
catalog safety conservative.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Update `internal/publish` planning so `req.Force` can select
|
||||
`force_replace` for exceptional destructive catalog cases.
|
||||
- Force should apply to these cases:
|
||||
- no valid `.distributor.json` and destination bundle path has unmanaged
|
||||
content;
|
||||
- planned path collision with storage content not recorded in valid catalog
|
||||
state;
|
||||
- invalid destination state JSON or invalid destination state fields;
|
||||
- unsupported future destination state schema, if the operator explicitly
|
||||
chooses force replacement after dry-run review.
|
||||
- Force should not be needed for ordinary valid catalog-managed upserts or
|
||||
replacements. Additive and replacement workflow behavior remains normal managed
|
||||
behavior.
|
||||
- `force_replace` must delete only the bounded destination bundle path through
|
||||
the storage abstraction, then write planned outputs and schema version `4`
|
||||
catalog state.
|
||||
- For fixed-path destinations, the bounded destination bundle path is the
|
||||
configured backend root. Dry-run output must make that clear.
|
||||
- Preserve default behavior without `--force`:
|
||||
- unmanaged planned path collisions fail as `fail_unmanaged`;
|
||||
- invalid or future state fails as `fail_conflict`;
|
||||
- no-state non-empty destinations fail as unmanaged.
|
||||
- Ensure `force_replace` state creation uses the same catalog output projection
|
||||
as normal publish planning.
|
||||
- Keep `Force` out of config. There must be no persistent force default.
|
||||
- Remove or update skipped force tests that were disabled during catalog
|
||||
implementation.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
|
||||
- Update only the force-related sections of:
|
||||
- `docs/cli.md`;
|
||||
- `docs/operations.md`;
|
||||
- `docs/troubleshooting.md`;
|
||||
- `docs/internal/publish.md`;
|
||||
- backend integration docs where they describe forced deletion boundaries.
|
||||
- Document force as an exceptional catalog recovery/replacement workflow, not as
|
||||
a normal way to handle valid managed catalog state.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/publish`
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- `go test ./internal/adapters/local ./internal/storage ./internal/storage/fake`
|
||||
- Planning tests:
|
||||
- no-state non-empty destination returns `fail_unmanaged` without force and
|
||||
`force_replace` with force;
|
||||
- unmanaged planned path collision returns `fail_unmanaged` without force and
|
||||
`force_replace` with force;
|
||||
- invalid state returns `fail_conflict` without force and `force_replace` with
|
||||
force;
|
||||
- future schema returns `fail_conflict` without force and `force_replace` with
|
||||
force;
|
||||
- valid catalog additive and replacement actions do not become
|
||||
`force_replace`.
|
||||
- Execution tests:
|
||||
- `force_replace` clears only the destination bundle path;
|
||||
- sibling paths outside the bundle path survive for local, fake S3, and fake
|
||||
SSH-style backends;
|
||||
- fixed-path force clears the configured backend root and is reported as such;
|
||||
- resulting state is schema version `4` catalog state.
|
||||
- CLI/app tests:
|
||||
- dry-run `--force` reports `force_replace` without writing;
|
||||
- normal `--force` executes and increments only the `force_replace` counter;
|
||||
- JSON output includes stable `force_replace` action and summary fields.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- `run --force` has observable, tested catalog behavior.
|
||||
- Force is still unnecessary for normal managed catalog replacement/upsert.
|
||||
- Unmanaged/invalid/future-state replacement remains impossible without explicit
|
||||
`--force`.
|
||||
|
||||
## Stage 2: Current Documentation And Example Synchronization
|
||||
|
||||
Goal: make all current-behavior docs and copyable examples match catalog schema
|
||||
version `4` and the `workflow` config model.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Rewrite `docs/config.md` so destination behavior is described through:
|
||||
- `workflow: additive`;
|
||||
- `workflow: replacement`;
|
||||
- `publish`;
|
||||
- `transform`;
|
||||
- `path_mapping`;
|
||||
- `links`;
|
||||
- `retention`.
|
||||
- Remove current-behavior documentation for destination config fields:
|
||||
- `state`;
|
||||
- `reconciliation`;
|
||||
- `takeover`;
|
||||
- `transfer`.
|
||||
- Rewrite `docs/integrations/destination-state.md` around schema version `4`
|
||||
catalog state:
|
||||
- top-level catalog fields;
|
||||
- output record fields;
|
||||
- source identity fields;
|
||||
- generated-output-only `source_path` and `transform`;
|
||||
- optional per-output `url`;
|
||||
- superseded legacy schema handling for publish planning.
|
||||
- Update `docs/operations.md`:
|
||||
- additive workflow semantics;
|
||||
- replacement workflow semantics;
|
||||
- catalog `force_replace`;
|
||||
- prune and reconcile-state catalog behavior;
|
||||
- dry-run action labels.
|
||||
- Update `docs/troubleshooting.md`:
|
||||
- replace legacy conflict guidance with workflow/catalog guidance;
|
||||
- describe rejected legacy config fields;
|
||||
- describe force-only exceptional cases.
|
||||
- Update `docs/cli.md`:
|
||||
- run summary counters now use `publish_new`, `upsert_additive`,
|
||||
`replace_catalog`, `skip_same`, `force_replace`, `fail_unmanaged`, and
|
||||
`fail_conflict`;
|
||||
- remove legacy action labels from current command reference.
|
||||
- Update relevant internal docs:
|
||||
- `docs/internal/app.md`;
|
||||
- `docs/internal/config.md`;
|
||||
- `docs/internal/publish.md`;
|
||||
- `docs/internal/state.md`.
|
||||
- Update policy docs where current architectural text still describes
|
||||
single-owner/shared-root behavior as current:
|
||||
- `docs/policy/architecture.md`;
|
||||
- `docs/policy/development.md`.
|
||||
- Update examples:
|
||||
- remove or rewrite `examples/shared-root.yml`;
|
||||
- remove or rewrite `examples/merge-reconciliation.yml`;
|
||||
- ensure examples use `workflow` where workflow intent matters;
|
||||
- keep examples valid, secret-free, and copyable.
|
||||
- Do not document any future catalog selectors, state migration, or compatibility
|
||||
aliases outside `docs/roadmap/`.
|
||||
|
||||
Tests and checks:
|
||||
|
||||
- `go test ./internal/config ./internal/cli`
|
||||
- `go test ./...`
|
||||
- Run the config/example checks already used by the test suite.
|
||||
- Manual smoke checks:
|
||||
- `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`
|
||||
- `go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run`
|
||||
- smoke any rewritten replacement/additive examples.
|
||||
- Consistency searches:
|
||||
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
|
||||
- `rg -n "single_owner|shared_root|schema version \`2\`|schema version \`3\`" docs --glob '!docs/roadmap/**'`
|
||||
- `rg -n "replace_older|replace_newer|replace_conflict|replace_takeover|takeover_mode" docs README.md examples --glob '!docs/roadmap/**'`
|
||||
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Current docs describe implemented catalog behavior only.
|
||||
- Copyable examples load successfully.
|
||||
- No current doc tells users to configure rejected legacy fields.
|
||||
- Destination state documentation is schema version `4` first.
|
||||
|
||||
## Stage 3: Catalog Idempotent Skip
|
||||
|
||||
Goal: make `skip_same` a real catalog no-op optimization instead of a stale
|
||||
legacy action label.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Implement `skip_same` when every planned output is already catalog-managed
|
||||
with matching:
|
||||
- pipeline id;
|
||||
- destination id;
|
||||
- source id;
|
||||
- source digest;
|
||||
- source created timestamp;
|
||||
- output path;
|
||||
- output kind;
|
||||
- output digest;
|
||||
- output size;
|
||||
- generated output `source_path`;
|
||||
- generated output `transform`;
|
||||
- output URL metadata.
|
||||
- Do not read destination bytes for this optimization. Trust valid catalog
|
||||
metadata; bitrot detection remains a separate future concern.
|
||||
- For `workflow: additive`, allow `skip_same` when all planned outputs match and
|
||||
no planned output needs to be written. Unrelated catalog outputs are ignored
|
||||
for the skip decision and remain retained.
|
||||
- For `workflow: replacement`, allow `skip_same` only when all planned outputs
|
||||
match and there are no current-owner catalog outputs that replacement workflow
|
||||
would delete.
|
||||
- Never return `skip_same` for superseded legacy state, invalid state, unmanaged
|
||||
content, future state, or force replacement.
|
||||
- `skip_same` must not write outputs, rewrite state, delete files, or notify.
|
||||
- Count `skip_same` in text and JSON summaries.
|
||||
- Keep behavior deterministic and identical across local, SSH, and S3
|
||||
destinations.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
|
||||
- Update `docs/cli.md`, `docs/operations.md`, and `docs/internal/publish.md` to
|
||||
describe catalog `skip_same`.
|
||||
- Document that `skip_same` is metadata-based and does not validate destination
|
||||
bytes.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/publish ./internal/app ./internal/cli`
|
||||
- Repeated additive publish with identical catalog metadata returns
|
||||
`skip_same`.
|
||||
- Repeated replacement publish with identical catalog metadata and no omitted
|
||||
current-owner outputs returns `skip_same`.
|
||||
- Changed source digest, output digest, URL, transform mode, owner, output size,
|
||||
or generated source path prevents skip.
|
||||
- Replacement workflow does not skip when it would delete omitted current-owner
|
||||
outputs.
|
||||
- `skip_same` does not write outputs or state and does not notify.
|
||||
- `go test ./...`
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Catalog action vocabulary has no stale action label.
|
||||
- Repeated publish behavior is intentional, documented, and tested.
|
||||
|
||||
## Stage 4: Legacy Publish And State Code Removal
|
||||
|
||||
Goal: remove dead or near-dead single-owner/shared-root write and planning code
|
||||
after catalog force behavior, docs, and skip semantics are settled.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Remove legacy publish actions that are no longer planned or documented:
|
||||
- `replace_older`;
|
||||
- `replace_newer`;
|
||||
- `replace_conflict`;
|
||||
- `replace_takeover`;
|
||||
- `skip_destination_newer`.
|
||||
- Keep only catalog-era actions:
|
||||
- `publish_new`;
|
||||
- `upsert_additive`;
|
||||
- `replace_catalog`;
|
||||
- `skip_same`;
|
||||
- `force_replace`;
|
||||
- `fail_unmanaged`;
|
||||
- `fail_conflict`.
|
||||
- Remove legacy fields from `publish.Plan`:
|
||||
- `StateMode`;
|
||||
- `Reconciliation`;
|
||||
- `TakeoverMode`;
|
||||
- `ExistingState`;
|
||||
- `ExistingSharedRoot`;
|
||||
- shared-root owner/output carry fields retained only for old execution.
|
||||
- Remove single-owner and shared-root execution branches from
|
||||
`internal/publish/execute.go`.
|
||||
- Remove unused single-owner/shared-root helper functions from `internal/publish`
|
||||
once no tests or callers use them.
|
||||
- Remove legacy destination config internals from `internal/config`:
|
||||
- `StatePolicy`;
|
||||
- `ReconciliationPolicy`;
|
||||
- `TakeoverPolicy`;
|
||||
- `TransferPolicy`;
|
||||
- legacy constants and defaults that are no longer referenced.
|
||||
- In `internal/state`, keep only what is needed for:
|
||||
- parsing schema version `4` catalog state;
|
||||
- identifying schema versions `1`, `2`, and `3` as superseded legacy by
|
||||
schema number;
|
||||
- validating and writing catalog state;
|
||||
- pruning and reconciling catalog outputs.
|
||||
- Delete or rewrite tests that assert legacy state parsing, validation,
|
||||
comparison, shared-root ownership, or single-owner output behavior.
|
||||
- Preserve test fixtures only where they are used to create superseded legacy
|
||||
state for first catalog-run behavior. Prefer small local helpers over keeping
|
||||
broad legacy state builders.
|
||||
- Remove stale skipped tests that only represent old behavior. If a skipped test
|
||||
still represents current expected behavior, unskip and update it.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
|
||||
- No new user docs should be needed if Stage 2 is complete.
|
||||
- Update internal docs only if removal changes internal package contracts beyond
|
||||
what Stage 2 already documented.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state`
|
||||
- `go test ./internal/publish`
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- `go test ./internal/config`
|
||||
- `go test ./...`
|
||||
- Consistency searches:
|
||||
- `rg -n "ActionReplaceOlder|ActionReplaceNewer|ActionReplaceConflict|ActionReplaceTakeover|ActionSkipDestinationNewer" internal`
|
||||
- `rg -n "StateModeSingleOwner|StateModeSharedRoot|SharedRootState|DistributorState|ParseSharedRoot|ReconciliationPolicy|TakeoverPolicy|TransferPolicy" internal --glob '!**/*_test.go'`
|
||||
- `rg -n "Retained while the executor is migrated to catalog state" internal`
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Production publish execution has one catalog code path plus explicit
|
||||
`force_replace`.
|
||||
- Legacy config policy types are gone from production config structs/defaults.
|
||||
- State package no longer exposes full v2/v3 implementation machinery unless it
|
||||
is required by tests that generate superseded legacy fixtures.
|
||||
- Full test suite passes without skipped tests that mask catalog cleanup work.
|
||||
|
||||
## Stage 5: Roadmap Closeout
|
||||
|
||||
Goal: leave `docs/roadmap/` in a clean state after the follow-up work is
|
||||
implemented.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Remove or rewrite completed roadmap material:
|
||||
- this `implementation.md`;
|
||||
- `docs/roadmap/catalog.md`, if catalog behavior is fully documented in
|
||||
current docs;
|
||||
- any future roadmap entries that still describe completed catalog cleanup as
|
||||
pending work.
|
||||
- Keep only genuinely future work in `docs/roadmap/future.md` or another active
|
||||
roadmap file.
|
||||
- Ensure future work remains out of current behavior docs.
|
||||
|
||||
Tests and checks:
|
||||
|
||||
- `go test ./...`
|
||||
- `git status --short`
|
||||
- `rg -n "future catalog|planned catalog|single_owner|shared_root|takeover|reconciliation|transfer" docs README.md examples`
|
||||
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs README.md examples`
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Current docs, examples, and code describe the same implemented behavior.
|
||||
- Roadmap docs contain only future work.
|
||||
- Full test suite passes.
|
||||
|
||||
## Refactors To Avoid
|
||||
|
||||
- Do not reintroduce `state`, `reconciliation`, `takeover`, or `transfer` YAML
|
||||
fields as deprecated aliases.
|
||||
- Do not migrate legacy destination state into catalog state.
|
||||
- Do not add top-level `owners` or `sources` to catalog state.
|
||||
- Do not persist `workflow` in `.distributor.json`.
|
||||
- Do not add new prune/reconcile selectors as part of this cleanup.
|
||||
- Do not move catalog or force policy into storage adapters.
|
||||
- Do not redesign the run JSON envelope while cleaning action labels.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No open questions remain. This roadmap locks the follow-up decisions:
|
||||
catalog `--force` will be implemented for exceptional destructive replacement,
|
||||
current docs and examples will be synchronized to catalog schema version `4`,
|
||||
catalog `skip_same` will be implemented as a metadata-only no-op optimization,
|
||||
and legacy single-owner/shared-root implementation debt will be removed after
|
||||
the catalog action vocabulary is complete.
|
||||
Reference in New Issue
Block a user