400 lines
16 KiB
Markdown
400 lines
16 KiB
Markdown
# Catalog Follow-Up Implementation Roadmap
|
|
|
|
This is the active staged plan for closing the remaining catalog-state
|
|
implementation gaps found after the schema version `4` catalog workflow landed.
|
|
It supersedes the earlier catalog implementation plan.
|
|
|
|
The current code already supports destination `workflow: additive` and
|
|
`workflow: replacement`, writes catalog state schema version `4`, rejects legacy
|
|
destination policy YAML fields, and routes publish execution through catalog
|
|
planning for normal runs. The remaining work is to make the implemented behavior
|
|
release-ready: restore a meaningful catalog `--force` path, synchronize current
|
|
docs and examples, and remove legacy single-owner/shared-root implementation
|
|
debt.
|
|
|
|
## Current Baseline
|
|
|
|
- Destination config accepts `workflow: additive` and `workflow: replacement`.
|
|
- Omitted `workflow` defaults to `additive`.
|
|
- Destination YAML fields `state`, `reconciliation`, `takeover`, and `transfer`
|
|
are rejected by strict YAML decoding.
|
|
- Successful publish writes `.distributor.json` schema version `4` with
|
|
`state.mode: catalog`.
|
|
- Schema versions `1`, `2`, and `3` are treated as superseded legacy state for
|
|
publish planning.
|
|
- `run --force` remains in the CLI and docs, but catalog publish planning does
|
|
not currently select `force_replace`.
|
|
- Current docs and examples still describe legacy config/state behavior in
|
|
several places.
|
|
- Legacy single-owner and shared-root state types, tests, and execution helpers
|
|
remain in the codebase even though normal publish planning no longer uses
|
|
them.
|
|
|
|
## Implementation Principles
|
|
|
|
- Preserve the catalog state shape defined in `docs/roadmap/catalog.md`.
|
|
- Keep `workflow` as runtime config policy; do not persist it in
|
|
`.distributor.json`.
|
|
- Keep unmanaged content protected by default.
|
|
- Keep `--force` explicit, per-run, dry-runnable, and bounded to the configured
|
|
destination bundle path.
|
|
- Do not reintroduce legacy config aliases or compatibility migration.
|
|
- Keep backend adapters unaware of catalog, workflow, and force policy.
|
|
- Update current-behavior docs in the same stage that makes the described
|
|
behavior true.
|
|
|
|
## Active Implementation Stages
|
|
|
|
## Stage 1: Catalog Force Planning And Execution
|
|
|
|
Goal: make `run --force` meaningful for catalog workflows while keeping normal
|
|
catalog safety conservative.
|
|
|
|
Implementation scope:
|
|
|
|
- Update `internal/publish` planning so `req.Force` can select
|
|
`force_replace` for exceptional destructive catalog cases.
|
|
- Force should apply to these cases:
|
|
- no valid `.distributor.json` and destination bundle path has unmanaged
|
|
content;
|
|
- planned path collision with storage content not recorded in valid catalog
|
|
state;
|
|
- invalid destination state JSON or invalid destination state fields;
|
|
- unsupported future destination state schema, if the operator explicitly
|
|
chooses force replacement after dry-run review.
|
|
- Force should not be needed for ordinary valid catalog-managed upserts or
|
|
replacements. Additive and replacement workflow behavior remains normal managed
|
|
behavior.
|
|
- `force_replace` must delete only the bounded destination bundle path through
|
|
the storage abstraction, then write planned outputs and schema version `4`
|
|
catalog state.
|
|
- For fixed-path destinations, the bounded destination bundle path is the
|
|
configured backend root. Dry-run output must make that clear.
|
|
- Preserve default behavior without `--force`:
|
|
- unmanaged planned path collisions fail as `fail_unmanaged`;
|
|
- invalid or future state fails as `fail_conflict`;
|
|
- no-state non-empty destinations fail as unmanaged.
|
|
- Ensure `force_replace` state creation uses the same catalog output projection
|
|
as normal publish planning.
|
|
- Keep `Force` out of config. There must be no persistent force default.
|
|
- Remove or update skipped force tests that were disabled during catalog
|
|
implementation.
|
|
|
|
Current-behavior documentation updates:
|
|
|
|
- Update only the force-related sections of:
|
|
- `docs/cli.md`;
|
|
- `docs/operations.md`;
|
|
- `docs/troubleshooting.md`;
|
|
- `docs/internal/publish.md`;
|
|
- backend integration docs where they describe forced deletion boundaries.
|
|
- Document force as an exceptional catalog recovery/replacement workflow, not as
|
|
a normal way to handle valid managed catalog state.
|
|
|
|
Tests:
|
|
|
|
- `go test ./internal/publish`
|
|
- `go test ./internal/app ./internal/cli`
|
|
- `go test ./internal/adapters/local ./internal/storage ./internal/storage/fake`
|
|
- Planning tests:
|
|
- no-state non-empty destination returns `fail_unmanaged` without force and
|
|
`force_replace` with force;
|
|
- unmanaged planned path collision returns `fail_unmanaged` without force and
|
|
`force_replace` with force;
|
|
- invalid state returns `fail_conflict` without force and `force_replace` with
|
|
force;
|
|
- future schema returns `fail_conflict` without force and `force_replace` with
|
|
force;
|
|
- valid catalog additive and replacement actions do not become
|
|
`force_replace`.
|
|
- Execution tests:
|
|
- `force_replace` clears only the destination bundle path;
|
|
- sibling paths outside the bundle path survive for local, fake S3, and fake
|
|
SSH-style backends;
|
|
- fixed-path force clears the configured backend root and is reported as such;
|
|
- resulting state is schema version `4` catalog state.
|
|
- CLI/app tests:
|
|
- dry-run `--force` reports `force_replace` without writing;
|
|
- normal `--force` executes and increments only the `force_replace` counter;
|
|
- JSON output includes stable `force_replace` action and summary fields.
|
|
|
|
Completion criteria:
|
|
|
|
- `run --force` has observable, tested catalog behavior.
|
|
- Force is still unnecessary for normal managed catalog replacement/upsert.
|
|
- Unmanaged/invalid/future-state replacement remains impossible without explicit
|
|
`--force`.
|
|
|
|
## Stage 2: Current Documentation And Example Synchronization
|
|
|
|
Goal: make all current-behavior docs and copyable examples match catalog schema
|
|
version `4` and the `workflow` config model.
|
|
|
|
Implementation scope:
|
|
|
|
- Rewrite `docs/config.md` so destination behavior is described through:
|
|
- `workflow: additive`;
|
|
- `workflow: replacement`;
|
|
- `publish`;
|
|
- `transform`;
|
|
- `path_mapping`;
|
|
- `links`;
|
|
- `retention`.
|
|
- Remove current-behavior documentation for destination config fields:
|
|
- `state`;
|
|
- `reconciliation`;
|
|
- `takeover`;
|
|
- `transfer`.
|
|
- Rewrite `docs/integrations/destination-state.md` around schema version `4`
|
|
catalog state:
|
|
- top-level catalog fields;
|
|
- output record fields;
|
|
- source identity fields;
|
|
- generated-output-only `source_path` and `transform`;
|
|
- optional per-output `url`;
|
|
- superseded legacy schema handling for publish planning.
|
|
- Update `docs/operations.md`:
|
|
- additive workflow semantics;
|
|
- replacement workflow semantics;
|
|
- catalog `force_replace`;
|
|
- prune and reconcile-state catalog behavior;
|
|
- dry-run action labels.
|
|
- Update `docs/troubleshooting.md`:
|
|
- replace legacy conflict guidance with workflow/catalog guidance;
|
|
- describe rejected legacy config fields;
|
|
- describe force-only exceptional cases.
|
|
- Update `docs/cli.md`:
|
|
- run summary counters now use `publish_new`, `upsert_additive`,
|
|
`replace_catalog`, `skip_same`, `force_replace`, `fail_unmanaged`, and
|
|
`fail_conflict`;
|
|
- remove legacy action labels from current command reference.
|
|
- Update relevant internal docs:
|
|
- `docs/internal/app.md`;
|
|
- `docs/internal/config.md`;
|
|
- `docs/internal/publish.md`;
|
|
- `docs/internal/state.md`.
|
|
- Update policy docs where current architectural text still describes
|
|
single-owner/shared-root behavior as current:
|
|
- `docs/policy/architecture.md`;
|
|
- `docs/policy/development.md`.
|
|
- Update examples:
|
|
- remove or rewrite `examples/shared-root.yml`;
|
|
- remove or rewrite `examples/merge-reconciliation.yml`;
|
|
- ensure examples use `workflow` where workflow intent matters;
|
|
- keep examples valid, secret-free, and copyable.
|
|
- Do not document any future catalog selectors, state migration, or compatibility
|
|
aliases outside `docs/roadmap/`.
|
|
|
|
Tests and checks:
|
|
|
|
- `go test ./internal/config ./internal/cli`
|
|
- `go test ./...`
|
|
- Run the config/example checks already used by the test suite.
|
|
- Manual smoke checks:
|
|
- `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`
|
|
- `go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run`
|
|
- smoke any rewritten replacement/additive examples.
|
|
- Consistency searches:
|
|
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
|
|
- `rg -n "single_owner|shared_root|schema version \`2\`|schema version \`3\`" docs --glob '!docs/roadmap/**'`
|
|
- `rg -n "replace_older|replace_newer|replace_conflict|replace_takeover|takeover_mode" docs README.md examples --glob '!docs/roadmap/**'`
|
|
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
|
|
|
|
Completion criteria:
|
|
|
|
- Current docs describe implemented catalog behavior only.
|
|
- Copyable examples load successfully.
|
|
- No current doc tells users to configure rejected legacy fields.
|
|
- Destination state documentation is schema version `4` first.
|
|
|
|
## Stage 3: Catalog Idempotent Skip
|
|
|
|
Goal: make `skip_same` a real catalog no-op optimization instead of a stale
|
|
legacy action label.
|
|
|
|
Implementation scope:
|
|
|
|
- Implement `skip_same` when every planned output is already catalog-managed
|
|
with matching:
|
|
- pipeline id;
|
|
- destination id;
|
|
- source id;
|
|
- source digest;
|
|
- source created timestamp;
|
|
- output path;
|
|
- output kind;
|
|
- output digest;
|
|
- output size;
|
|
- generated output `source_path`;
|
|
- generated output `transform`;
|
|
- output URL metadata.
|
|
- Do not read destination bytes for this optimization. Trust valid catalog
|
|
metadata; bitrot detection remains a separate future concern.
|
|
- For `workflow: additive`, allow `skip_same` when all planned outputs match and
|
|
no planned output needs to be written. Unrelated catalog outputs are ignored
|
|
for the skip decision and remain retained.
|
|
- For `workflow: replacement`, allow `skip_same` only when all planned outputs
|
|
match and there are no current-owner catalog outputs that replacement workflow
|
|
would delete.
|
|
- Never return `skip_same` for superseded legacy state, invalid state, unmanaged
|
|
content, future state, or force replacement.
|
|
- `skip_same` must not write outputs, rewrite state, delete files, or notify.
|
|
- Count `skip_same` in text and JSON summaries.
|
|
- Keep behavior deterministic and identical across local, SSH, and S3
|
|
destinations.
|
|
|
|
Current-behavior documentation updates:
|
|
|
|
- Update `docs/cli.md`, `docs/operations.md`, and `docs/internal/publish.md` to
|
|
describe catalog `skip_same`.
|
|
- Document that `skip_same` is metadata-based and does not validate destination
|
|
bytes.
|
|
|
|
Tests:
|
|
|
|
- `go test ./internal/publish ./internal/app ./internal/cli`
|
|
- Repeated additive publish with identical catalog metadata returns
|
|
`skip_same`.
|
|
- Repeated replacement publish with identical catalog metadata and no omitted
|
|
current-owner outputs returns `skip_same`.
|
|
- Changed source digest, output digest, URL, transform mode, owner, output size,
|
|
or generated source path prevents skip.
|
|
- Replacement workflow does not skip when it would delete omitted current-owner
|
|
outputs.
|
|
- `skip_same` does not write outputs or state and does not notify.
|
|
- `go test ./...`
|
|
|
|
Completion criteria:
|
|
|
|
- Catalog action vocabulary has no stale action label.
|
|
- Repeated publish behavior is intentional, documented, and tested.
|
|
|
|
## Stage 4: Legacy Publish And State Code Removal
|
|
|
|
Goal: remove dead or near-dead single-owner/shared-root write and planning code
|
|
after catalog force behavior, docs, and skip semantics are settled.
|
|
|
|
Implementation scope:
|
|
|
|
- Remove legacy publish actions that are no longer planned or documented:
|
|
- `replace_older`;
|
|
- `replace_newer`;
|
|
- `replace_conflict`;
|
|
- `replace_takeover`;
|
|
- `skip_destination_newer`.
|
|
- Keep only catalog-era actions:
|
|
- `publish_new`;
|
|
- `upsert_additive`;
|
|
- `replace_catalog`;
|
|
- `skip_same`;
|
|
- `force_replace`;
|
|
- `fail_unmanaged`;
|
|
- `fail_conflict`.
|
|
- Remove legacy fields from `publish.Plan`:
|
|
- `StateMode`;
|
|
- `Reconciliation`;
|
|
- `TakeoverMode`;
|
|
- `ExistingState`;
|
|
- `ExistingSharedRoot`;
|
|
- shared-root owner/output carry fields retained only for old execution.
|
|
- Remove single-owner and shared-root execution branches from
|
|
`internal/publish/execute.go`.
|
|
- Remove unused single-owner/shared-root helper functions from `internal/publish`
|
|
once no tests or callers use them.
|
|
- Remove legacy destination config internals from `internal/config`:
|
|
- `StatePolicy`;
|
|
- `ReconciliationPolicy`;
|
|
- `TakeoverPolicy`;
|
|
- `TransferPolicy`;
|
|
- legacy constants and defaults that are no longer referenced.
|
|
- In `internal/state`, keep only what is needed for:
|
|
- parsing schema version `4` catalog state;
|
|
- identifying schema versions `1`, `2`, and `3` as superseded legacy by
|
|
schema number;
|
|
- validating and writing catalog state;
|
|
- pruning and reconciling catalog outputs.
|
|
- Delete or rewrite tests that assert legacy state parsing, validation,
|
|
comparison, shared-root ownership, or single-owner output behavior.
|
|
- Preserve test fixtures only where they are used to create superseded legacy
|
|
state for first catalog-run behavior. Prefer small local helpers over keeping
|
|
broad legacy state builders.
|
|
- Remove stale skipped tests that only represent old behavior. If a skipped test
|
|
still represents current expected behavior, unskip and update it.
|
|
|
|
Current-behavior documentation updates:
|
|
|
|
- No new user docs should be needed if Stage 2 is complete.
|
|
- Update internal docs only if removal changes internal package contracts beyond
|
|
what Stage 2 already documented.
|
|
|
|
Tests:
|
|
|
|
- `go test ./internal/state`
|
|
- `go test ./internal/publish`
|
|
- `go test ./internal/app ./internal/cli`
|
|
- `go test ./internal/config`
|
|
- `go test ./...`
|
|
- Consistency searches:
|
|
- `rg -n "ActionReplaceOlder|ActionReplaceNewer|ActionReplaceConflict|ActionReplaceTakeover|ActionSkipDestinationNewer" internal`
|
|
- `rg -n "StateModeSingleOwner|StateModeSharedRoot|SharedRootState|DistributorState|ParseSharedRoot|ReconciliationPolicy|TakeoverPolicy|TransferPolicy" internal --glob '!**/*_test.go'`
|
|
- `rg -n "Retained while the executor is migrated to catalog state" internal`
|
|
|
|
Completion criteria:
|
|
|
|
- Production publish execution has one catalog code path plus explicit
|
|
`force_replace`.
|
|
- Legacy config policy types are gone from production config structs/defaults.
|
|
- State package no longer exposes full v2/v3 implementation machinery unless it
|
|
is required by tests that generate superseded legacy fixtures.
|
|
- Full test suite passes without skipped tests that mask catalog cleanup work.
|
|
|
|
## Stage 5: Roadmap Closeout
|
|
|
|
Goal: leave `docs/roadmap/` in a clean state after the follow-up work is
|
|
implemented.
|
|
|
|
Implementation scope:
|
|
|
|
- Remove or rewrite completed roadmap material:
|
|
- this `implementation.md`;
|
|
- `docs/roadmap/catalog.md`, if catalog behavior is fully documented in
|
|
current docs;
|
|
- any future roadmap entries that still describe completed catalog cleanup as
|
|
pending work.
|
|
- Keep only genuinely future work in `docs/roadmap/future.md` or another active
|
|
roadmap file.
|
|
- Ensure future work remains out of current behavior docs.
|
|
|
|
Tests and checks:
|
|
|
|
- `go test ./...`
|
|
- `git status --short`
|
|
- `rg -n "future catalog|planned catalog|single_owner|shared_root|takeover|reconciliation|transfer" docs README.md examples`
|
|
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs README.md examples`
|
|
|
|
Completion criteria:
|
|
|
|
- Current docs, examples, and code describe the same implemented behavior.
|
|
- Roadmap docs contain only future work.
|
|
- Full test suite passes.
|
|
|
|
## Refactors To Avoid
|
|
|
|
- Do not reintroduce `state`, `reconciliation`, `takeover`, or `transfer` YAML
|
|
fields as deprecated aliases.
|
|
- Do not migrate legacy destination state into catalog state.
|
|
- Do not add top-level `owners` or `sources` to catalog state.
|
|
- Do not persist `workflow` in `.distributor.json`.
|
|
- Do not add new prune/reconcile selectors as part of this cleanup.
|
|
- Do not move catalog or force policy into storage adapters.
|
|
- Do not redesign the run JSON envelope while cleaning action labels.
|
|
|
|
## Open Questions
|
|
|
|
No open questions remain. This roadmap locks the follow-up decisions:
|
|
catalog `--force` will be implemented for exceptional destructive replacement,
|
|
current docs and examples will be synchronized to catalog schema version `4`,
|
|
catalog `skip_same` will be implemented as a metadata-only no-op optimization,
|
|
and legacy single-owner/shared-root implementation debt will be removed after
|
|
the catalog action vocabulary is complete.
|