distributor/docs/roadmap/implementation.md

# Catalog Follow-Up Implementation Roadmap

This is the active staged plan for closing the remaining catalog-state
implementation gaps found after the schema version `4` catalog workflow landed.
It supersedes the earlier catalog implementation plan.

The current code already supports destination `workflow: additive` and
`workflow: replacement`, writes catalog state schema version `4`, rejects legacy
destination policy YAML fields, and routes publish execution through catalog
planning for normal runs. The remaining work is to make the implemented behavior
release-ready: restore a meaningful catalog `--force` path, synchronize current
docs and examples, and remove legacy single-owner/shared-root implementation
debt.

## Current Baseline

- Destination config accepts `workflow: additive` and `workflow: replacement`.
- Omitted `workflow` defaults to `additive`.
- Destination YAML fields `state`, `reconciliation`, `takeover`, and `transfer`
  are rejected by strict YAML decoding.
- Successful publish writes `.distributor.json` schema version `4` with
  `state.mode: catalog`.
- Schema versions `1`, `2`, and `3` are treated as superseded legacy state for
  publish planning.
- `run --force` remains in the CLI and docs, but catalog publish planning does
  not currently select `force_replace`.
- Current docs and examples still describe legacy config/state behavior in
  several places.
- Legacy single-owner and shared-root state types, tests, and execution helpers
  remain in the codebase even though normal publish planning no longer uses
  them.

## Implementation Principles

- Preserve the catalog state shape defined in `docs/roadmap/catalog.md`.
- Keep `workflow` as runtime config policy; do not persist it in
  `.distributor.json`.
- Keep unmanaged content protected by default.
- Keep `--force` explicit, per-run, dry-runnable, and bounded to the configured
  destination bundle path.
- Do not reintroduce legacy config aliases or compatibility migration.
- Keep backend adapters unaware of catalog, workflow, and force policy.
- Update current-behavior docs in the same stage that makes the described
  behavior true.

## Active Implementation Stages

## Stage 1: Catalog Force Planning And Execution

Goal: make `run --force` meaningful for catalog workflows while keeping normal
catalog safety conservative.

Implementation scope:

- Update `internal/publish` planning so `req.Force` can select
  `force_replace` for exceptional destructive catalog cases.
- Force should apply to these cases:
  - no valid `.distributor.json` and destination bundle path has unmanaged
    content;
  - planned path collision with storage content not recorded in valid catalog
    state;
  - invalid destination state JSON or invalid destination state fields;
  - unsupported future destination state schema, if the operator explicitly
    chooses force replacement after dry-run review.
- Force should not be needed for ordinary valid catalog-managed upserts or
  replacements. Additive and replacement workflow behavior remains normal managed
  behavior.
- `force_replace` must delete only the bounded destination bundle path through
  the storage abstraction, then write planned outputs and schema version `4`
  catalog state.
- For fixed-path destinations, the bounded destination bundle path is the
  configured backend root. Dry-run output must make that clear.
- Preserve default behavior without `--force`:
  - unmanaged planned path collisions fail as `fail_unmanaged`;
  - invalid or future state fails as `fail_conflict`;
  - no-state non-empty destinations fail as unmanaged.
- Ensure `force_replace` state creation uses the same catalog output projection
  as normal publish planning.
- Keep `Force` out of config. There must be no persistent force default.
- Remove or update skipped force tests that were disabled during catalog
  implementation.

Current-behavior documentation updates:

- Update only the force-related sections of:
  - `docs/cli.md`;
  - `docs/operations.md`;
  - `docs/troubleshooting.md`;
  - `docs/internal/publish.md`;
  - backend integration docs where they describe forced deletion boundaries.
- Document force as an exceptional catalog recovery/replacement workflow, not as
  a normal way to handle valid managed catalog state.

Tests:

- `go test ./internal/publish`
- `go test ./internal/app ./internal/cli`
- `go test ./internal/adapters/local ./internal/storage ./internal/storage/fake`
- Planning tests:
  - no-state non-empty destination returns `fail_unmanaged` without force and
    `force_replace` with force;
  - unmanaged planned path collision returns `fail_unmanaged` without force and
    `force_replace` with force;
  - invalid state returns `fail_conflict` without force and `force_replace` with
    force;
  - future schema returns `fail_conflict` without force and `force_replace` with
    force;
  - valid catalog additive and replacement actions do not become
    `force_replace`.
- Execution tests:
  - `force_replace` clears only the destination bundle path;
  - sibling paths outside the bundle path survive for local, fake S3, and fake
    SSH-style backends;
  - fixed-path force clears the configured backend root and is reported as such;
  - resulting state is schema version `4` catalog state.
- CLI/app tests:
  - dry-run `--force` reports `force_replace` without writing;
  - normal `--force` executes and increments only the `force_replace` counter;
  - JSON output includes stable `force_replace` action and summary fields.

Completion criteria:

- `run --force` has observable, tested catalog behavior.
- Force is still unnecessary for normal managed catalog replacement/upsert.
- Unmanaged/invalid/future-state replacement remains impossible without explicit
  `--force`.

## Stage 2: Current Documentation And Example Synchronization

Goal: make all current-behavior docs and copyable examples match catalog schema
version `4` and the `workflow` config model.

Implementation scope:

- Rewrite `docs/config.md` so destination behavior is described through:
  - `workflow: additive`;
  - `workflow: replacement`;
  - `publish`;
  - `transform`;
  - `path_mapping`;
  - `links`;
  - `retention`.
- Remove current-behavior documentation for destination config fields:
  - `state`;
  - `reconciliation`;
  - `takeover`;
  - `transfer`.
- Rewrite `docs/integrations/destination-state.md` around schema version `4`
  catalog state:
  - top-level catalog fields;
  - output record fields;
  - source identity fields;
  - generated-output-only `source_path` and `transform`;
  - optional per-output `url`;
  - superseded legacy schema handling for publish planning.
- Update `docs/operations.md`:
  - additive workflow semantics;
  - replacement workflow semantics;
  - catalog `force_replace`;
  - prune and reconcile-state catalog behavior;
  - dry-run action labels.
- Update `docs/troubleshooting.md`:
  - replace legacy conflict guidance with workflow/catalog guidance;
  - describe rejected legacy config fields;
  - describe force-only exceptional cases.
- Update `docs/cli.md`:
  - run summary counters now use `publish_new`, `upsert_additive`,
    `replace_catalog`, `skip_same`, `force_replace`, `fail_unmanaged`, and
    `fail_conflict`;
  - remove legacy action labels from current command reference.
- Update relevant internal docs:
  - `docs/internal/app.md`;
  - `docs/internal/config.md`;
  - `docs/internal/publish.md`;
  - `docs/internal/state.md`.
- Update policy docs where current architectural text still describes
  single-owner/shared-root behavior as current:
  - `docs/policy/architecture.md`;
  - `docs/policy/development.md`.
- Update examples:
  - remove or rewrite `examples/shared-root.yml`;
  - remove or rewrite `examples/merge-reconciliation.yml`;
  - ensure examples use `workflow` where workflow intent matters;
  - keep examples valid, secret-free, and copyable.
- Do not document any future catalog selectors, state migration, or compatibility
  aliases outside `docs/roadmap/`.

Tests and checks:

- `go test ./internal/config ./internal/cli`
- `go test ./...`
- Run the config/example checks already used by the test suite.
- Manual smoke checks:
  - `go run ./cmd/distributor run --config examples/local-publish.yml --dry-run`
  - `go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run`
  - smoke any rewritten replacement/additive examples.
- Consistency searches:
  - `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
  - `rg -n "single_owner|shared_root|schema version \`2\`|schema version \`3\`" docs --glob '!docs/roadmap/**'`
  - `rg -n "replace_older|replace_newer|replace_conflict|replace_takeover|takeover_mode" docs README.md examples --glob '!docs/roadmap/**'`
  - `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`

Completion criteria:

- Current docs describe implemented catalog behavior only.
- Copyable examples load successfully.
- No current doc tells users to configure rejected legacy fields.
- Destination state documentation is schema version `4` first.

## Stage 3: Catalog Idempotent Skip

Goal: make `skip_same` a real catalog no-op optimization instead of a stale
legacy action label.

Implementation scope:

- Implement `skip_same` when every planned output is already catalog-managed
  with matching:
  - pipeline id;
  - destination id;
  - source id;
  - source digest;
  - source created timestamp;
  - output path;
  - output kind;
  - output digest;
  - output size;
  - generated output `source_path`;
  - generated output `transform`;
  - output URL metadata.
- Do not read destination bytes for this optimization. Trust valid catalog
  metadata; bitrot detection remains a separate future concern.
- For `workflow: additive`, allow `skip_same` when all planned outputs match and
  no planned output needs to be written. Unrelated catalog outputs are ignored
  for the skip decision and remain retained.
- For `workflow: replacement`, allow `skip_same` only when all planned outputs
  match and there are no current-owner catalog outputs that replacement workflow
  would delete.
- Never return `skip_same` for superseded legacy state, invalid state, unmanaged
  content, future state, or force replacement.
- `skip_same` must not write outputs, rewrite state, delete files, or notify.
- Count `skip_same` in text and JSON summaries.
- Keep behavior deterministic and identical across local, SSH, and S3
  destinations.

Current-behavior documentation updates:

- Update `docs/cli.md`, `docs/operations.md`, and `docs/internal/publish.md` to
  describe catalog `skip_same`.
- Document that `skip_same` is metadata-based and does not validate destination
  bytes.

Tests:

- `go test ./internal/publish ./internal/app ./internal/cli`
- Repeated additive publish with identical catalog metadata returns
  `skip_same`.
- Repeated replacement publish with identical catalog metadata and no omitted
  current-owner outputs returns `skip_same`.
- Changed source digest, output digest, URL, transform mode, owner, output size,
  or generated source path prevents skip.
- Replacement workflow does not skip when it would delete omitted current-owner
  outputs.
- `skip_same` does not write outputs or state and does not notify.
- `go test ./...`

Completion criteria:

- Catalog action vocabulary has no stale action label.
- Repeated publish behavior is intentional, documented, and tested.

## Stage 4: Legacy Publish And State Code Removal

Goal: remove dead or near-dead single-owner/shared-root write and planning code
after catalog force behavior, docs, and skip semantics are settled.

Implementation scope:

- Remove legacy publish actions that are no longer planned or documented:
  - `replace_older`;
  - `replace_newer`;
  - `replace_conflict`;
  - `replace_takeover`;
  - `skip_destination_newer`.
- Keep only catalog-era actions:
  - `publish_new`;
  - `upsert_additive`;
  - `replace_catalog`;
  - `skip_same`;
  - `force_replace`;
  - `fail_unmanaged`;
  - `fail_conflict`.
- Remove legacy fields from `publish.Plan`:
  - `StateMode`;
  - `Reconciliation`;
  - `TakeoverMode`;
  - `ExistingState`;
  - `ExistingSharedRoot`;
  - shared-root owner/output carry fields retained only for old execution.
- Remove single-owner and shared-root execution branches from
  `internal/publish/execute.go`.
- Remove unused single-owner/shared-root helper functions from `internal/publish`
  once no tests or callers use them.
- Remove legacy destination config internals from `internal/config`:
  - `StatePolicy`;
  - `ReconciliationPolicy`;
  - `TakeoverPolicy`;
  - `TransferPolicy`;
  - legacy constants and defaults that are no longer referenced.
- In `internal/state`, keep only what is needed for:
  - parsing schema version `4` catalog state;
  - identifying schema versions `1`, `2`, and `3` as superseded legacy by
    schema number;
  - validating and writing catalog state;
  - pruning and reconciling catalog outputs.
- Delete or rewrite tests that assert legacy state parsing, validation,
  comparison, shared-root ownership, or single-owner output behavior.
- Preserve test fixtures only where they are used to create superseded legacy
  state for first catalog-run behavior. Prefer small local helpers over keeping
  broad legacy state builders.
- Remove stale skipped tests that only represent old behavior. If a skipped test
  still represents current expected behavior, unskip and update it.

Current-behavior documentation updates:

- No new user docs should be needed if Stage 2 is complete.
- Update internal docs only if removal changes internal package contracts beyond
  what Stage 2 already documented.

Tests:

- `go test ./internal/state`
- `go test ./internal/publish`
- `go test ./internal/app ./internal/cli`
- `go test ./internal/config`
- `go test ./...`
- Consistency searches:
  - `rg -n "ActionReplaceOlder|ActionReplaceNewer|ActionReplaceConflict|ActionReplaceTakeover|ActionSkipDestinationNewer" internal`
  - `rg -n "StateModeSingleOwner|StateModeSharedRoot|SharedRootState|DistributorState|ParseSharedRoot|ReconciliationPolicy|TakeoverPolicy|TransferPolicy" internal --glob '!**/*_test.go'`
  - `rg -n "Retained while the executor is migrated to catalog state" internal`

Completion criteria:

- Production publish execution has one catalog code path plus explicit
  `force_replace`.
- Legacy config policy types are gone from production config structs/defaults.
- State package no longer exposes full v2/v3 implementation machinery unless it
  is required by tests that generate superseded legacy fixtures.
- Full test suite passes without skipped tests that mask catalog cleanup work.

## Stage 5: Roadmap Closeout

Goal: leave `docs/roadmap/` in a clean state after the follow-up work is
implemented.

Implementation scope:

- Remove or rewrite completed roadmap material:
  - this `implementation.md`;
  - `docs/roadmap/catalog.md`, if catalog behavior is fully documented in
    current docs;
  - any future roadmap entries that still describe completed catalog cleanup as
    pending work.
- Keep only genuinely future work in `docs/roadmap/future.md` or another active
  roadmap file.
- Ensure future work remains out of current behavior docs.

Tests and checks:

- `go test ./...`
- `git status --short`
- `rg -n "future catalog|planned catalog|single_owner|shared_root|takeover|reconciliation|transfer" docs README.md examples`
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs README.md examples`

Completion criteria:

- Current docs, examples, and code describe the same implemented behavior.
- Roadmap docs contain only future work.
- Full test suite passes.

## Refactors To Avoid

- Do not reintroduce `state`, `reconciliation`, `takeover`, or `transfer` YAML
  fields as deprecated aliases.
- Do not migrate legacy destination state into catalog state.
- Do not add top-level `owners` or `sources` to catalog state.
- Do not persist `workflow` in `.distributor.json`.
- Do not add new prune/reconcile selectors as part of this cleanup.
- Do not move catalog or force policy into storage adapters.
- Do not redesign the run JSON envelope while cleaning action labels.

## Open Questions

No open questions remain. This roadmap locks the follow-up decisions:
catalog `--force` will be implemented for exceptional destructive replacement,
current docs and examples will be synchronized to catalog schema version `4`,
catalog `skip_same` will be implemented as a metadata-only no-op optimization,
and legacy single-owner/shared-root implementation debt will be removed after
the catalog action vocabulary is complete.