distributor/docs/roadmap/catalog.md

Catalog State And Destination Workflow Roadmap

This roadmap records the intended first-class state model and destination workflow configuration for replacement and additive publication workflows.

Current single_owner and shared_root state are oriented around comparing a destination owner to one latest source manifest. That works well for replacement workflows, where a producer maintains a curated source of truth and distributor makes a destination match it. It is a poor fit for additive workflows, where each run contributes new or updated output paths while preserving unrelated managed outputs from previous runs and other pipelines.

The target model is a current-state catalog: .distributor.json records the currently managed output paths at a destination root, and each output record stores its current owner, compact source identity, content digest, and update metadata. It is not intended to be an audit log.

Both replacement and additive workflows should use the same catalog state shape. The workflow choice is runtime policy from destination config, not persisted state.

Locked Decisions

• Add one destination state mode named catalog for both replacement and additive workflows.
• Catalog state writes .distributor.json schema version 4.
• Add a destination-level workflow setting with accepted values replacement and additive.
• Default workflow to additive because it is the least destructive workflow.
• workflow replaces the normal user-facing need to combine state.mode, reconciliation.mode, takeover.mode, and transfer conflict settings for the two primary workflows.
• workflow: replacement treats the current planned outputs as the authoritative managed output set for the destination scope and removes previously managed outputs in that scope when they are no longer planned.
• workflow: additive writes or overwrites the planned output paths and retains unrelated managed outputs.
• If a planned path already exists as a managed output, the new publication may overwrite it and becomes that path's current owner.
• If a planned path exists in storage but is not recorded in valid catalog state, it remains unmanaged content and must not be adopted implicitly.
• If an output path changes owner, preserve the output record's existing created_at and update only updated_at.
• Catalog state records current ownership only. It does not retain historical owners, historical sources, or old versions of overwritten output records.
• pipeline_id and destination_id are stored separately on each output record. They are not concatenated into one owner string.
• Catalog state does not include top-level owners; owners are derivable from the output records.
• Catalog state does not include top-level sources; compact source identity is stored directly on each output record.
• Catalog state does not record whether the last run used replacement or additive. Workflow is execution policy, and persisting it would create drift risk if config changes later.
• Legacy destination policy fields should be rejected outright in the new workflow config model. Do not retain aliases for state, reconciliation, takeover, or transfer.

Destination Workflow Semantics

The user-facing destination config should express intent directly:

destinations:
  - id: weather-latest
    backend: local
    path: /srv/reports/weather/latest
    workflow: additive

destinations:
  - id: weather-archive
    backend: local
    path: /srv/reports/weather/archive
    workflow: replacement

workflow is orthogonal to backend config, path mapping, publish source/HTML selection, transforms, links, and retention.

Replacement Workflow

A replacement destination is for producers that maintain a curated source of truth and expect distributor to make the destination's managed scope match the current publication.

For a destination configured with workflow: replacement:

• planned outputs are written to their resolved destination paths;
• planned outputs may overwrite existing managed outputs at the same path;
• each written output record is replaced in-place with the current publication's owner, source identity, hash, size, and timestamps;
• managed outputs in the destination scope that are not present in the current plan are deleted and removed from catalog state;
• unmanaged destination content remains unmanaged and blocks planned path collisions unless an explicit force workflow later chooses otherwise;
• the catalog state shape remains the same as additive workflow state.

Additive Workflow

An additive destination is for producers that routinely contribute outputs to a shared destination root.

For a destination configured with workflow: additive:

• planned outputs are written to their resolved destination paths;
• planned outputs may overwrite existing managed outputs at the same path;
• each overwritten output record is replaced in-place with the current publication's owner, source identity, hash, size, and timestamps;
• managed outputs not present in the current plan are retained;
• unrelated managed outputs from other pipelines or destinations are retained;
• unmanaged destination content remains unmanaged and blocks planned path collisions unless an explicit force workflow later chooses otherwise;
• pruning can select retained managed outputs by updated_at and, when useful, by pipeline_id and/or destination_id.

Replacement workflow treats the current publication as the desired managed output set for a destination scope. Additive workflow treats the current publication as a patch to the catalog of currently managed outputs.

Destination State Schema Version 4

Catalog state should use this top-level shape:

{
  "schema_version": 4,
  "distributor_version": "dev",
  "created_at": "2026-06-19T12:00:00Z",
  "updated_at": "2026-06-19T12:05:00Z",
  "state": {
    "mode": "catalog"
  },
  "outputs": []
}

Top-level fields:

• schema_version: required. Value 4 for catalog state.
• distributor_version: optional diagnostic version string.
• created_at: required RFC3339 timestamp for when the catalog state file was first created.
• updated_at: required RFC3339 timestamp for the latest catalog state update.
• state.mode: required. Value catalog.
• outputs: required array of currently managed output records.

Catalog state should not include top-level pipeline_id, destination_id, published_at, workflow, owners, sources, or a full source manifest. Those concepts belong on output records or in configuration.

Output Record Schema

Each output record should be self-contained enough to support current ownership, pruning, repair, bitrot checks, and basic provenance without a separate owner or source catalog.

{
  "path": "tomorrow/index.html",
  "pipeline_id": "weatherreporter.daily",
  "destination_id": "latest-html",
  "source": {
    "id": "weatherreporter.tomorrow",
    "digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
    "created": "2026-06-19T12:00:00Z"
  },
  "kind": "generated",
  "source_path": "report.md",
  "transform": "markdown_to_html",
  "sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
  "size": 12345,
  "created_at": "2026-06-19T12:00:00Z",
  "updated_at": "2026-06-19T12:05:00Z",
  "url": "https://reports.example.com/weather/tomorrow/"
}

Required output fields:

• path: destination-relative managed output path.
• pipeline_id: pipeline that most recently wrote this output path.
• destination_id: destination that most recently wrote this output path.
• source: compact identity of the source bundle that produced this output.
• source.id: source manifest id.
• source.digest: source manifest digest.
• source.created: source manifest creation timestamp, RFC3339.
• kind: source or generated.
• sha256: digest of the output bytes.
• size: output byte size.
• created_at: RFC3339 timestamp for when this output path first became managed in catalog state.
• updated_at: RFC3339 timestamp for when this output path was most recently written or updated.

Optional output fields:

• source_path: present only for generated outputs; source manifest path used to produce the generated output.
• transform: present only for generated outputs; transform name used to produce the generated output.
• url: present only when destination link configuration produces a public URL for this output.

For copied source outputs, source_path and transform should be omitted. The output path is already the copied source artifact path.

Useful Properties

The catalog shape supports both primary workflows without unnecessary normalization:

• current owner of each path is explicit;
• current source identity for each path is explicit;
• bitrot checks can compare storage bytes to sha256;
• pruning can use updated_at, optionally scoped by pipeline_id and destination_id;
• repair can remove missing managed output records without consulting an owner or source table;
• overwriting an output path updates one output record in place;
• no orphaned top-level owner/source records need to be maintained.

The schema intentionally avoids storing a full source manifest for every output. The source manifest remains the producer-to-distributor validation contract, but catalog state only needs compact source identity for currently managed outputs.

Relationship To Existing State And Config Modes

Existing state modes remain current behavior until catalog mode is implemented:

• single_owner schema version 2 supports one owner for a destination bundle path.
• shared_root schema version 3 supports multiple owners in one destination root but still keeps owner records with latest source manifests.
• catalog schema version 4 should support replacement and additive current-state ownership without top-level owner or source catalogs.

The intended user-facing config should move toward workflow: replacement and workflow: additive instead of requiring ordinary users to combine state.mode, reconciliation.mode, takeover.mode, and transfer conflict settings.

Because the project is still alpha pre-release, catalog implementation should be a clean break:

• remove legacy write paths for current single_owner and shared_root state;
• do not implement migration from schema versions 1, 2, or 3;
• do not preserve backwards compatibility for legacy destination state;
• when a configured destination writes successfully, write schema version 4 catalog state;
• if an existing .distributor.json has schema version lower than 4, treat it as superseded legacy state for planning purposes and overwrite it according to the configured workflow, without attempting conversion.

For the first catalog run against superseded legacy state:

• workflow: replacement may clear the bounded destination root before writing the planned outputs and schema version 4 catalog state.
• workflow: additive may overwrite only the planned output paths, then write schema version 4 catalog state containing those planned outputs.
• unplanned files left behind by an additive run against superseded legacy state are not recorded in catalog state and are treated as unmanaged content by later catalog runs.

The target direction is that both primary workflows use catalog state.

Prune And Reconcile-State Scope

Catalog maintenance commands should initially keep the existing ownership selector model:

• prune --pipeline <id> --destination <id> operates only on outputs currently owned by that pipeline/destination.
• reconcile-state --pipeline <id> --destination <id> repairs only outputs currently owned by that pipeline/destination.
• reconcile-state --all-owners repairs all output records in the catalog.
• prune remains scoped to the selected pipeline/destination owner only in the initial catalog implementation.

Do not add catalog-specific selectors in the initial implementation, such as --source-id, --path-prefix, or --kind. Those may be useful later, but they are not required to make additive workflow first-class.