distributor/docs/roadmap/implementation.md

Catalog State And Workflow Implementation Roadmap

This is the active staged implementation plan for docs/roadmap/catalog.md. The feature roadmap defines the target state model and policy decisions; this document defines the implementation sequence for an LLM coding agent to follow stage by stage.

Future behavior must remain under docs/roadmap/ until implemented. Update current-behavior docs only in the stage that implements the corresponding behavior.

Current Baseline

distributor currently writes destination .distributor.json using single-owner schema version 2 or shared-root schema version 3. Destination behavior is selected through several low-level knobs: state.mode, reconciliation.mode, takeover.mode, and transfer.

The target behavior is a clean alpha break:

• all newly written destination state uses schema version 4;
• all destinations use state.mode: catalog internally;
• user-facing destination behavior is selected by workflow: additive or workflow: replacement;
• workflow defaults to additive;
• legacy destination policy fields are rejected, not aliased;
• legacy state schema versions 1, 2, and 3 are not migrated.

Implementation Principles

• Keep the source manifest contract unchanged.
• Keep backend adapters unaware of catalog and workflow policy.
• Treat workflow as runtime config, not persisted state.
• Keep unmanaged content protected after catalog state exists.
• Preserve output created_at when a path changes owner; update only updated_at.
• Prefer removing legacy state/config paths over compatibility shims.
• Preserve current public APIs outside destination state/config unless the catalog roadmap explicitly changes them.

Active Implementation Stages

Stage 1: Destination Workflow Config Clean Break

Goal: replace low-level destination policy config with the user-facing workflow switch.

Implementation scope:

• Add destination workflow config with accepted values additive and replacement.
• Default omitted workflow to additive.
• Remove or make unsupported the destination-level config fields state, reconciliation, takeover, and transfer.
• Ensure configs containing those legacy fields fail clearly. Prefer strict YAML unknown-field failure by removing struct fields; add explicit validation only if clearer errors are needed without weakening strict decoding.
• Preserve backend, publish, transform, path mapping, links, retention, and other non-policy destination fields.
• Update example configs only when the implementation stage also updates current docs; otherwise keep this stage focused on config code and tests.

Tests:

• go test ./internal/config
• Omitted workflow defaults to additive.
• workflow: additive and workflow: replacement validate.
• Unknown workflow values fail.
• Legacy state, reconciliation, takeover, and transfer fields fail.
• Existing valid examples are updated or tests are adjusted in the same change if examples currently use legacy fields.

Completion criteria:

• New configs express destination replacement/additive intent with one field.
• No normal config path accepts legacy destination policy knobs.

Stage 2: Catalog State Schema Version 4

Goal: implement the schema version 4 catalog state model in internal/state.

Implementation scope:

• Add catalog state types for top-level schema version 4, state.mode: catalog, and outputs.
• Add compact per-output source identity with id, digest, and created.
• Add catalog output records with required path, pipeline_id, destination_id, source, kind, sha256, size, created_at, and updated_at.
• Add optional source_path, transform, and url, present only where allowed by the catalog roadmap.
• Validate duplicate paths, invalid owner ids, invalid source identities, invalid output paths, invalid digests, negative sizes, invalid timestamps, invalid kind/transform combinations, and invalid URLs.
• Marshal catalog state deterministically with the existing JSON formatting conventions.
• Update ParseDocument so schema version 4 returns catalog state.
• Treat schema versions lower than 4 as superseded legacy state for publish planning, not as readable/migrated active state. Keep enough detection to identify legacy state and avoid treating it as arbitrary invalid JSON.
• Reject unsupported future schema versions.

Tests:

• go test ./internal/state
• Parse/validate/marshal valid catalog state.
• Reject malformed catalog state and invalid output records.
• Detect schema versions 1, 2, and 3 as superseded legacy state.
• Reject future schema versions.
• Prove no top-level owners, sources, workflow, source manifest, pipeline id, destination id, or published timestamp is accepted for catalog state.

Completion criteria:

• State package has one canonical schema version 4 catalog contract for new writes.
• Legacy state is detected but not migrated.

Stage 3: Catalog Publish Planning

Goal: plan publish actions against catalog state and destination workflow.

Implementation scope:

• Replace publish request inputs that consume state/reconciliation/takeover and transfer policy with destination workflow.
• Map workflow to catalog actions:
  • workflow: additive: upsert planned outputs and retain all other managed catalog outputs.
  • workflow: replacement: upsert planned outputs and delete catalog outputs owned by the current pipeline/destination that are omitted from the plan.
• Planned paths that already exist as catalog-managed outputs may be overwritten and become owned by the current pipeline/destination.
• Planned path collisions with storage content not recorded in catalog state fail as unmanaged content.
• Matching planned outputs may skip writes when source identity and output digest already match, if that optimization can be implemented without changing externally visible results; otherwise writing idempotently is acceptable.
• Existing schema < 4 state is superseded:
  • replacement workflow may plan a bounded destination-root clear before writing planned outputs;
  • additive workflow may plan overwrites for planned paths only and leave unplanned files unmanaged.
• Invalid JSON or future schema state remains a conflict, not a superseded legacy state.
• Preserve path mapping, publish policy, transforms, links, fixed-path selection, and output collision checks.

Tests:

• go test ./internal/publish
• Additive workflow publishes new catalog state.
• Additive workflow overwrites an existing managed output and retains unrelated outputs.
• Replacement workflow removes omitted outputs owned by the current pipeline/destination.
• Replacement workflow preserves unrelated catalog outputs owned by other pipeline/destination pairs.
• Managed output ownership moves to the current pipeline/destination when a planned path is overwritten.
• created_at is preserved when an existing path changes owner; updated_at changes.
• Unmanaged path collisions fail.
• Legacy schema < 4 state follows the superseded-state rules above.
• Invalid/future state fails.

Completion criteria:

• Publish planning no longer depends on single-owner/shared-root comparison semantics.
• Workflow behavior is fully driven by workflow.

Stage 4: Catalog Publish Execution

Goal: execute catalog publish plans and write schema version 4 state.

Implementation scope:

• Write only schema version 4 catalog .distributor.json.
• Implement additive execution as managed upsert of planned outputs plus catalog state update.
• Implement replacement execution as managed upsert plus deletion of omitted current-owner catalog outputs.
• For replacement over superseded legacy state, clear the bounded destination root before writing planned outputs and catalog state.
• For additive over superseded legacy state, overwrite planned paths and write a catalog containing only planned outputs; leave unplanned files unmanaged.
• Preserve cleanup behavior on failed writes:
  • additive cleanup removes outputs newly created by the failed attempt where practical;
  • replacement cleanup follows existing managed-replacement safety where practical;
  • state is written only after selected outputs are written.
• Preserve link metadata, generated output digests, source output digests, content type behavior, and backend-safe writes.

Tests:

• go test ./internal/publish ./internal/app
• Additive execution writes planned outputs, overwrites managed planned paths, retains unrelated managed outputs, and writes catalog state.
• Replacement execution deletes omitted current-owner outputs and preserves unrelated owner outputs.
• Superseded legacy replacement clears bounded destination root only.
• Superseded legacy additive leaves unplanned files on disk but out of catalog state.
• Failed writes do not leave misleading catalog state.
• Local, fake-backed SSH, and fake-backed S3 app paths exercise the same publish behavior.

Completion criteria:

• Successful run writes only v4 catalog state.
• Destination outputs match additive/replacement workflow semantics.

Stage 5: Run Reporting, Notifications, And CLI Surface

Goal: update user-visible run behavior to describe workflow/catalog actions instead of legacy replacement/takeover actions.

Implementation scope:

• Replace legacy action labels tied to state/reconciliation/takeover/transfer with these stable workflow-oriented labels:
  • publish_new
  • upsert_additive
  • replace_catalog
  • skip_same
  • force_replace
  • fail_unmanaged
  • fail_conflict
• Ensure text and JSON run summaries include workflow-relevant counters.
• Include workflow in run action records where useful.
• Update fixed-path dry-run warnings to describe additive upsert or replacement clearly.
• Ensure notifications use the new action labels.
• Remove reporting assumptions that depend on replace_older, replace_newer, replace_conflict, or replace_takeover.

Tests:

• go test ./internal/app ./internal/cli
• Text dry-run output distinguishes additive from replacement workflow.
• JSON output includes workflow and stable action labels.
• Summary counters are deterministic.
• Notifications fire for additive and replacement writes.
• Existing CLI commands still parse and execute with the new config shape.

Completion criteria:

• Operators can understand from dry-run output whether a destination will upsert or replace managed catalog outputs.

Stage 6: Catalog Prune And Reconcile-State

Goal: update maintenance commands to operate on catalog state with the agreed selector model.

Implementation scope:

• Prune:
  • selected --pipeline and --destination prune only outputs currently owned by that pipeline/destination;
  • no --all-owners prune mode in the initial catalog implementation;
  • do not add --source-id, --path-prefix, or --kind selectors.
• Reconcile-state:
  • selected --pipeline and --destination repair only outputs currently owned by that pipeline/destination;
  • --all-owners repairs all catalog outputs;
  • unmanaged reporting compares storage entries to all catalog output paths, not just selected owner paths.
• Rewrite repaired/pruned state as schema version 4.
• Remove legacy single-owner/shared-root maintenance branches.

Tests:

• go test ./internal/state ./internal/app ./internal/cli
• Prune selects only current-owner catalog outputs.
• Prune deletes selected outputs and removes their catalog records.
• Reconcile-state selected owner removes only missing outputs for that owner.
• reconcile-state --all-owners removes missing outputs for all owners.
• Unmanaged reporting excludes all catalog-managed paths and reports unrecorded storage entries.
• JSON/text output remains stable and clear.

Completion criteria:

• Maintenance commands operate only on catalog state and respect the locked selector rules.

Stage 7: Clean Break Removal And Documentation

Goal: remove legacy state/config behavior and document the implemented catalog workflow model.

Implementation scope:

• Remove dead code for writing schema version 2 single-owner and schema version 3 shared-root state.
• Remove legacy config structs/constants/defaults/validation for destination state, reconciliation, takeover, and transfer where no longer used.
• Remove or rewrite tests that only assert legacy state/config behavior.
• Update current-behavior docs:
  • docs/config.md
  • docs/cli.md
  • docs/operations.md
  • docs/troubleshooting.md
  • docs/integrations/destination-state.md
  • relevant docs/internal/ docs
  • docs/policy/architecture.md
  • docs/policy/development.md
• Update examples to use workflow and remove legacy fields.
• Keep roadmap-only material out of current docs.
• Once implemented and documented, remove or rewrite docs/roadmap/catalog.md so completed behavior is not described only as future work.

Tests and checks:

• go test ./...
• rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'
• rg -n 'single_owner|shared_root|schema version 2|schema version 3' docs internal
• rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples

Completion criteria:

• Current docs and examples describe the catalog workflow model.
• Legacy destination policy fields and legacy write paths are gone.
• Full test suite passes.

Refactors To Avoid

• Do not add top-level owners or sources catalogs.
• Do not persist workflow in .distributor.json.
• Do not keep deprecated config aliases for legacy destination policy fields.
• Do not implement state migration from schema versions 1, 2, or 3.
• Do not add catalog-specific prune/reconcile selectors beyond the agreed initial ownership scope.
• Do not move catalog policy into storage adapters.

Open Questions

No open questions are known. The catalog roadmap locks the state shape, workflow values, default workflow, clean-break policy, legacy-state behavior, output field selection, created_at preservation, and maintenance command scope.