# Catalog State And Destination Workflow Roadmap This roadmap records the intended first-class state model and destination workflow configuration for replacement and additive publication workflows. Current `single_owner` and `shared_root` state are oriented around comparing a destination owner to one latest source manifest. That works well for replacement workflows, where a producer maintains a curated source of truth and `distributor` makes a destination match it. It is a poor fit for additive workflows, where each run contributes new or updated output paths while preserving unrelated managed outputs from previous runs and other pipelines. The target model is a current-state catalog: `.distributor.json` records the currently managed output paths at a destination root, and each output record stores its current owner, compact source identity, content digest, and update metadata. It is not intended to be an audit log. Both replacement and additive workflows should use the same catalog state shape. The workflow choice is runtime policy from destination config, not persisted state. ## Locked Decisions - Add one destination state mode named `catalog` for both replacement and additive workflows. - Catalog state writes `.distributor.json` schema version `4`. - Add a destination-level `workflow` setting with accepted values `replacement` and `additive`. - Default `workflow` to `additive` because it is the least destructive workflow. - `workflow` replaces the normal user-facing need to combine `state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the two primary workflows. - `workflow: replacement` treats the current planned outputs as the authoritative managed output set for the destination scope and removes previously managed outputs in that scope when they are no longer planned. - `workflow: additive` writes or overwrites the planned output paths and retains unrelated managed outputs. - If a planned path already exists as a managed output, the new publication may overwrite it and becomes that path's current owner. - If a planned path exists in storage but is not recorded in valid catalog state, it remains unmanaged content and must not be adopted implicitly. - If an output path changes owner, preserve the output record's existing `created_at` and update only `updated_at`. - Catalog state records current ownership only. It does not retain historical owners, historical sources, or old versions of overwritten output records. - `pipeline_id` and `destination_id` are stored separately on each output record. They are not concatenated into one owner string. - Catalog state does not include top-level `owners`; owners are derivable from the output records. - Catalog state does not include top-level `sources`; compact source identity is stored directly on each output record. - Catalog state does not record whether the last run used `replacement` or `additive`. Workflow is execution policy, and persisting it would create drift risk if config changes later. - Legacy destination policy fields should be rejected outright in the new workflow config model. Do not retain aliases for `state`, `reconciliation`, `takeover`, or `transfer`. ## Destination Workflow Semantics The user-facing destination config should express intent directly: ```yaml destinations: - id: weather-latest backend: local path: /srv/reports/weather/latest workflow: additive ``` ```yaml destinations: - id: weather-archive backend: local path: /srv/reports/weather/archive workflow: replacement ``` `workflow` is orthogonal to backend config, path mapping, publish source/HTML selection, transforms, links, and retention. ### Replacement Workflow A replacement destination is for producers that maintain a curated source of truth and expect `distributor` to make the destination's managed scope match the current publication. For a destination configured with `workflow: replacement`: - planned outputs are written to their resolved destination paths; - planned outputs may overwrite existing managed outputs at the same path; - each written output record is replaced in-place with the current publication's owner, source identity, hash, size, and timestamps; - managed outputs in the destination scope that are not present in the current plan are deleted and removed from catalog state; - unmanaged destination content remains unmanaged and blocks planned path collisions unless an explicit force workflow later chooses otherwise; - the catalog state shape remains the same as additive workflow state. ### Additive Workflow An additive destination is for producers that routinely contribute outputs to a shared destination root. For a destination configured with `workflow: additive`: - planned outputs are written to their resolved destination paths; - planned outputs may overwrite existing managed outputs at the same path; - each overwritten output record is replaced in-place with the current publication's owner, source identity, hash, size, and timestamps; - managed outputs not present in the current plan are retained; - unrelated managed outputs from other pipelines or destinations are retained; - unmanaged destination content remains unmanaged and blocks planned path collisions unless an explicit force workflow later chooses otherwise; - pruning can select retained managed outputs by `updated_at` and, when useful, by `pipeline_id` and/or `destination_id`. Replacement workflow treats the current publication as the desired managed output set for a destination scope. Additive workflow treats the current publication as a patch to the catalog of currently managed outputs. ## Destination State Schema Version 4 Catalog state should use this top-level shape: ```json { "schema_version": 4, "distributor_version": "dev", "created_at": "2026-06-19T12:00:00Z", "updated_at": "2026-06-19T12:05:00Z", "state": { "mode": "catalog" }, "outputs": [] } ``` Top-level fields: - `schema_version`: required. Value `4` for catalog state. - `distributor_version`: optional diagnostic version string. - `created_at`: required RFC3339 timestamp for when the catalog state file was first created. - `updated_at`: required RFC3339 timestamp for the latest catalog state update. - `state.mode`: required. Value `catalog`. - `outputs`: required array of currently managed output records. Catalog state should not include top-level `pipeline_id`, `destination_id`, `published_at`, `workflow`, `owners`, `sources`, or a full source manifest. Those concepts belong on output records or in configuration. ## Output Record Schema Each output record should be self-contained enough to support current ownership, pruning, repair, bitrot checks, and basic provenance without a separate owner or source catalog. ```json { "path": "tomorrow/index.html", "pipeline_id": "weatherreporter.daily", "destination_id": "latest-html", "source": { "id": "weatherreporter.tomorrow", "digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", "created": "2026-06-19T12:00:00Z" }, "kind": "generated", "source_path": "report.md", "transform": "markdown_to_html", "sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789", "size": 12345, "created_at": "2026-06-19T12:00:00Z", "updated_at": "2026-06-19T12:05:00Z", "url": "https://reports.example.com/weather/tomorrow/" } ``` Required output fields: - `path`: destination-relative managed output path. - `pipeline_id`: pipeline that most recently wrote this output path. - `destination_id`: destination that most recently wrote this output path. - `source`: compact identity of the source bundle that produced this output. - `source.id`: source manifest id. - `source.digest`: source manifest digest. - `source.created`: source manifest creation timestamp, RFC3339. - `kind`: `source` or `generated`. - `sha256`: digest of the output bytes. - `size`: output byte size. - `created_at`: RFC3339 timestamp for when this output path first became managed in catalog state. - `updated_at`: RFC3339 timestamp for when this output path was most recently written or updated. Optional output fields: - `source_path`: present only for generated outputs; source manifest path used to produce the generated output. - `transform`: present only for generated outputs; transform name used to produce the generated output. - `url`: present only when destination link configuration produces a public URL for this output. For copied source outputs, `source_path` and `transform` should be omitted. The output `path` is already the copied source artifact path. ## Useful Properties The catalog shape supports both primary workflows without unnecessary normalization: - current owner of each path is explicit; - current source identity for each path is explicit; - bitrot checks can compare storage bytes to `sha256`; - pruning can use `updated_at`, optionally scoped by `pipeline_id` and `destination_id`; - repair can remove missing managed output records without consulting an owner or source table; - overwriting an output path updates one output record in place; - no orphaned top-level owner/source records need to be maintained. The schema intentionally avoids storing a full source manifest for every output. The source manifest remains the producer-to-distributor validation contract, but catalog state only needs compact source identity for currently managed outputs. ## Relationship To Existing State And Config Modes Existing state modes remain current behavior until catalog mode is implemented: - `single_owner` schema version `2` supports one owner for a destination bundle path. - `shared_root` schema version `3` supports multiple owners in one destination root but still keeps owner records with latest source manifests. - `catalog` schema version `4` should support replacement and additive current-state ownership without top-level owner or source catalogs. The intended user-facing config should move toward `workflow: replacement` and `workflow: additive` instead of requiring ordinary users to combine `state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict settings. Because the project is still alpha pre-release, catalog implementation should be a clean break: - remove legacy write paths for current `single_owner` and `shared_root` state; - do not implement migration from schema versions `1`, `2`, or `3`; - do not preserve backwards compatibility for legacy destination state; - when a configured destination writes successfully, write schema version `4` catalog state; - if an existing `.distributor.json` has schema version lower than `4`, treat it as superseded legacy state for planning purposes and overwrite it according to the configured workflow, without attempting conversion. For the first catalog run against superseded legacy state: - `workflow: replacement` may clear the bounded destination root before writing the planned outputs and schema version `4` catalog state. - `workflow: additive` may overwrite only the planned output paths, then write schema version `4` catalog state containing those planned outputs. - unplanned files left behind by an additive run against superseded legacy state are not recorded in catalog state and are treated as unmanaged content by later catalog runs. The target direction is that both primary workflows use catalog state. ## Prune And Reconcile-State Scope Catalog maintenance commands should initially keep the existing ownership selector model: - `prune --pipeline --destination ` operates only on outputs currently owned by that pipeline/destination. - `reconcile-state --pipeline --destination ` repairs only outputs currently owned by that pipeline/destination. - `reconcile-state --all-owners` repairs all output records in the catalog. - `prune` remains scoped to the selected pipeline/destination owner only in the initial catalog implementation. Do not add catalog-specific selectors in the initial implementation, such as `--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they are not required to make additive workflow first-class.