distributor/docs/roadmap/implementation.md

Catalog Follow-Up Implementation Roadmap

This is the active staged plan for closing the remaining catalog-state implementation gaps found after the schema version 4 catalog workflow landed. It supersedes the earlier catalog implementation plan.

The current code already supports destination workflow: additive and workflow: replacement, writes catalog state schema version 4, rejects legacy destination policy YAML fields, and routes publish execution through catalog planning for normal runs. The remaining work is to make the implemented behavior release-ready: restore a meaningful catalog --force path, synchronize current docs and examples, and remove legacy single-owner/shared-root implementation debt.

Current Baseline

• Destination config accepts workflow: additive and workflow: replacement.
• Omitted workflow defaults to additive.
• Destination YAML fields state, reconciliation, takeover, and transfer are rejected by strict YAML decoding.
• Successful publish writes .distributor.json schema version 4 with state.mode: catalog.
• Schema versions 1, 2, and 3 are treated as superseded legacy state for publish planning.
• run --force remains in the CLI and docs, but catalog publish planning does not currently select force_replace.
• Current docs and examples still describe legacy config/state behavior in several places.
• Legacy single-owner and shared-root state types, tests, and execution helpers remain in the codebase even though normal publish planning no longer uses them.

Implementation Principles

• Preserve the catalog state shape defined in docs/roadmap/catalog.md.
• Keep workflow as runtime config policy; do not persist it in .distributor.json.
• Keep unmanaged content protected by default.
• Keep --force explicit, per-run, dry-runnable, and bounded to the configured destination bundle path.
• Do not reintroduce legacy config aliases or compatibility migration.
• Keep backend adapters unaware of catalog, workflow, and force policy.
• Update current-behavior docs in the same stage that makes the described behavior true.

Active Implementation Stages

Stage 1: Catalog Force Planning And Execution

Goal: make run --force meaningful for catalog workflows while keeping normal catalog safety conservative.

Implementation scope:

• Update internal/publish planning so req.Force can select force_replace for exceptional destructive catalog cases.
• Force should apply to these cases:
  • no valid .distributor.json and destination bundle path has unmanaged content;
  • planned path collision with storage content not recorded in valid catalog state;
  • invalid destination state JSON or invalid destination state fields;
  • unsupported future destination state schema, if the operator explicitly chooses force replacement after dry-run review.
• Force should not be needed for ordinary valid catalog-managed upserts or replacements. Additive and replacement workflow behavior remains normal managed behavior.
• force_replace must delete only the bounded destination bundle path through the storage abstraction, then write planned outputs and schema version 4 catalog state.
• For fixed-path destinations, the bounded destination bundle path is the configured backend root. Dry-run output must make that clear.
• Preserve default behavior without --force:
  • unmanaged planned path collisions fail as fail_unmanaged;
  • invalid or future state fails as fail_conflict;
  • no-state non-empty destinations fail as unmanaged.
• Ensure force_replace state creation uses the same catalog output projection as normal publish planning.
• Keep Force out of config. There must be no persistent force default.
• Remove or update skipped force tests that were disabled during catalog implementation.

Current-behavior documentation updates:

• Update only the force-related sections of:
  • docs/cli.md;
  • docs/operations.md;
  • docs/troubleshooting.md;
  • docs/internal/publish.md;
  • backend integration docs where they describe forced deletion boundaries.
• Document force as an exceptional catalog recovery/replacement workflow, not as a normal way to handle valid managed catalog state.

Tests:

• go test ./internal/publish
• go test ./internal/app ./internal/cli
• go test ./internal/adapters/local ./internal/storage ./internal/storage/fake
• Planning tests:
  • no-state non-empty destination returns fail_unmanaged without force and force_replace with force;
  • unmanaged planned path collision returns fail_unmanaged without force and force_replace with force;
  • invalid state returns fail_conflict without force and force_replace with force;
  • future schema returns fail_conflict without force and force_replace with force;
  • valid catalog additive and replacement actions do not become force_replace.
• Execution tests:
  • force_replace clears only the destination bundle path;
  • sibling paths outside the bundle path survive for local, fake S3, and fake SSH-style backends;
  • fixed-path force clears the configured backend root and is reported as such;
  • resulting state is schema version 4 catalog state.
• CLI/app tests:
  • dry-run --force reports force_replace without writing;
  • normal --force executes and increments only the force_replace counter;
  • JSON output includes stable force_replace action and summary fields.

Completion criteria:

• run --force has observable, tested catalog behavior.
• Force is still unnecessary for normal managed catalog replacement/upsert.
• Unmanaged/invalid/future-state replacement remains impossible without explicit --force.

Stage 2: Current Documentation And Example Synchronization

Goal: make all current-behavior docs and copyable examples match catalog schema version 4 and the workflow config model.

Implementation scope:

• Rewrite docs/config.md so destination behavior is described through:
  • workflow: additive;
  • workflow: replacement;
  • publish;
  • transform;
  • path_mapping;
  • links;
  • retention.
• Remove current-behavior documentation for destination config fields:
  • state;
  • reconciliation;
  • takeover;
  • transfer.
• Rewrite docs/integrations/destination-state.md around schema version 4 catalog state:
  • top-level catalog fields;
  • output record fields;
  • source identity fields;
  • generated-output-only source_path and transform;
  • optional per-output url;
  • superseded legacy schema handling for publish planning.
• Update docs/operations.md:
  • additive workflow semantics;
  • replacement workflow semantics;
  • catalog force_replace;
  • prune and reconcile-state catalog behavior;
  • dry-run action labels.
• Update docs/troubleshooting.md:
  • replace legacy conflict guidance with workflow/catalog guidance;
  • describe rejected legacy config fields;
  • describe force-only exceptional cases.
• Update docs/cli.md:
  • run summary counters now use publish_new, upsert_additive, replace_catalog, skip_same, force_replace, fail_unmanaged, and fail_conflict;
  • remove legacy action labels from current command reference.
• Update relevant internal docs:
  • docs/internal/app.md;
  • docs/internal/config.md;
  • docs/internal/publish.md;
  • docs/internal/state.md.
• Update policy docs where current architectural text still describes single-owner/shared-root behavior as current:
  • docs/policy/architecture.md;
  • docs/policy/development.md.
• Update examples:
  • remove or rewrite examples/shared-root.yml;
  • remove or rewrite examples/merge-reconciliation.yml;
  • ensure examples use workflow where workflow intent matters;
  • keep examples valid, secret-free, and copyable.
• Do not document any future catalog selectors, state migration, or compatibility aliases outside docs/roadmap/.

Tests and checks:

• go test ./internal/config ./internal/cli
• go test ./...
• Run the config/example checks already used by the test suite.
• Manual smoke checks:
  • go run ./cmd/distributor run --config examples/local-publish.yml --dry-run
  • go run ./cmd/distributor run --config examples/archive-and-latest.yml --dry-run
  • smoke any rewritten replacement/additive examples.
• Consistency searches:
  • rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'
  • rg -n "single_owner|shared_root|schema version \2`|schema version `3`" docs --glob '!docs/roadmap/**'`
  • rg -n "replace_older|replace_newer|replace_conflict|replace_takeover|takeover_mode" docs README.md examples --glob '!docs/roadmap/**'
  • rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples

Completion criteria:

• Current docs describe implemented catalog behavior only.
• Copyable examples load successfully.
• No current doc tells users to configure rejected legacy fields.
• Destination state documentation is schema version 4 first.

Stage 3: Catalog Idempotent Skip

Goal: make skip_same a real catalog no-op optimization instead of a stale legacy action label.

Implementation scope:

• Implement skip_same when every planned output is already catalog-managed with matching:
  • pipeline id;
  • destination id;
  • source id;
  • source digest;
  • source created timestamp;
  • output path;
  • output kind;
  • output digest;
  • output size;
  • generated output source_path;
  • generated output transform;
  • output URL metadata.
• Do not read destination bytes for this optimization. Trust valid catalog metadata; bitrot detection remains a separate future concern.
• For workflow: additive, allow skip_same when all planned outputs match and no planned output needs to be written. Unrelated catalog outputs are ignored for the skip decision and remain retained.
• For workflow: replacement, allow skip_same only when all planned outputs match and there are no current-owner catalog outputs that replacement workflow would delete.
• Never return skip_same for superseded legacy state, invalid state, unmanaged content, future state, or force replacement.
• skip_same must not write outputs, rewrite state, delete files, or notify.
• Count skip_same in text and JSON summaries.
• Keep behavior deterministic and identical across local, SSH, and S3 destinations.

Current-behavior documentation updates:

• Update docs/cli.md, docs/operations.md, and docs/internal/publish.md to describe catalog skip_same.
• Document that skip_same is metadata-based and does not validate destination bytes.

Tests:

• go test ./internal/publish ./internal/app ./internal/cli
• Repeated additive publish with identical catalog metadata returns skip_same.
• Repeated replacement publish with identical catalog metadata and no omitted current-owner outputs returns skip_same.
• Changed source digest, output digest, URL, transform mode, owner, output size, or generated source path prevents skip.
• Replacement workflow does not skip when it would delete omitted current-owner outputs.
• skip_same does not write outputs or state and does not notify.
• go test ./...

Completion criteria:

• Catalog action vocabulary has no stale action label.
• Repeated publish behavior is intentional, documented, and tested.

Stage 4: Legacy Publish And State Code Removal

Goal: remove dead or near-dead single-owner/shared-root write and planning code after catalog force behavior, docs, and skip semantics are settled.

Implementation scope:

• Remove legacy publish actions that are no longer planned or documented:
  • replace_older;
  • replace_newer;
  • replace_conflict;
  • replace_takeover;
  • skip_destination_newer.
• Keep only catalog-era actions:
  • publish_new;
  • upsert_additive;
  • replace_catalog;
  • skip_same;
  • force_replace;
  • fail_unmanaged;
  • fail_conflict.
• Remove legacy fields from publish.Plan:
  • StateMode;
  • Reconciliation;
  • TakeoverMode;
  • ExistingState;
  • ExistingSharedRoot;
  • shared-root owner/output carry fields retained only for old execution.
• Remove single-owner and shared-root execution branches from internal/publish/execute.go.
• Remove unused single-owner/shared-root helper functions from internal/publish once no tests or callers use them.
• Remove legacy destination config internals from internal/config:
  • StatePolicy;
  • ReconciliationPolicy;
  • TakeoverPolicy;
  • TransferPolicy;
  • legacy constants and defaults that are no longer referenced.
• In internal/state, keep only what is needed for:
  • parsing schema version 4 catalog state;
  • identifying schema versions 1, 2, and 3 as superseded legacy by schema number;
  • validating and writing catalog state;
  • pruning and reconciling catalog outputs.
• Delete or rewrite tests that assert legacy state parsing, validation, comparison, shared-root ownership, or single-owner output behavior.
• Preserve test fixtures only where they are used to create superseded legacy state for first catalog-run behavior. Prefer small local helpers over keeping broad legacy state builders.
• Remove stale skipped tests that only represent old behavior. If a skipped test still represents current expected behavior, unskip and update it.

Current-behavior documentation updates:

• No new user docs should be needed if Stage 2 is complete.
• Update internal docs only if removal changes internal package contracts beyond what Stage 2 already documented.

Tests:

• go test ./internal/state
• go test ./internal/publish
• go test ./internal/app ./internal/cli
• go test ./internal/config
• go test ./...
• Consistency searches:
  • rg -n "ActionReplaceOlder|ActionReplaceNewer|ActionReplaceConflict|ActionReplaceTakeover|ActionSkipDestinationNewer" internal
  • rg -n "StateModeSingleOwner|StateModeSharedRoot|SharedRootState|DistributorState|ParseSharedRoot|ReconciliationPolicy|TakeoverPolicy|TransferPolicy" internal --glob '!**/*_test.go'
  • rg -n "Retained while the executor is migrated to catalog state" internal

Completion criteria:

• Production publish execution has one catalog code path plus explicit force_replace.
• Legacy config policy types are gone from production config structs/defaults.
• State package no longer exposes full v2/v3 implementation machinery unless it is required by tests that generate superseded legacy fixtures.
• Full test suite passes without skipped tests that mask catalog cleanup work.

Stage 5: Roadmap Closeout

Goal: leave docs/roadmap/ in a clean state after the follow-up work is implemented.

Implementation scope:

• Remove or rewrite completed roadmap material:
  • this implementation.md;
  • docs/roadmap/catalog.md, if catalog behavior is fully documented in current docs;
  • any future roadmap entries that still describe completed catalog cleanup as pending work.
• Keep only genuinely future work in docs/roadmap/future.md or another active roadmap file.
• Ensure future work remains out of current behavior docs.

Tests and checks:

• go test ./...
• git status --short
• rg -n "future catalog|planned catalog|single_owner|shared_root|takeover|reconciliation|transfer" docs README.md examples
• rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs README.md examples

Completion criteria:

• Current docs, examples, and code describe the same implemented behavior.
• Roadmap docs contain only future work.
• Full test suite passes.

Refactors To Avoid

• Do not reintroduce state, reconciliation, takeover, or transfer YAML fields as deprecated aliases.
• Do not migrate legacy destination state into catalog state.
• Do not add top-level owners or sources to catalog state.
• Do not persist workflow in .distributor.json.
• Do not add new prune/reconcile selectors as part of this cleanup.
• Do not move catalog or force policy into storage adapters.
• Do not redesign the run JSON envelope while cleaning action labels.

Open Questions

No open questions remain. This roadmap locks the follow-up decisions: catalog --force will be implemented for exceptional destructive replacement, current docs and examples will be synchronized to catalog schema version 4, catalog skip_same will be implemented as a metadata-only no-op optimization, and legacy single-owner/shared-root implementation debt will be removed after the catalog action vocabulary is complete.