284 lines
12 KiB
Markdown
284 lines
12 KiB
Markdown
# Catalog State And Destination Workflow Roadmap
|
|
|
|
This roadmap records the intended first-class state model and destination
|
|
workflow configuration for replacement and additive publication workflows.
|
|
|
|
Current `single_owner` and `shared_root` state are oriented around comparing a
|
|
destination owner to one latest source manifest. That works well for replacement
|
|
workflows, where a producer maintains a curated source of truth and
|
|
`distributor` makes a destination match it. It is a poor fit for additive
|
|
workflows, where each run contributes new or updated output paths while
|
|
preserving unrelated managed outputs from previous runs and other pipelines.
|
|
|
|
The target model is a current-state catalog: `.distributor.json` records the
|
|
currently managed output paths at a destination root, and each output record
|
|
stores its current owner, compact source identity, content digest, and update
|
|
metadata. It is not intended to be an audit log.
|
|
|
|
Both replacement and additive workflows should use the same catalog state shape.
|
|
The workflow choice is runtime policy from destination config, not persisted
|
|
state.
|
|
|
|
## Locked Decisions
|
|
|
|
- Add one destination state mode named `catalog` for both replacement and
|
|
additive workflows.
|
|
- Catalog state writes `.distributor.json` schema version `4`.
|
|
- Add a destination-level `workflow` setting with accepted values
|
|
`replacement` and `additive`.
|
|
- Default `workflow` to `additive` because it is the least destructive workflow.
|
|
- `workflow` replaces the normal user-facing need to combine `state.mode`,
|
|
`reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the
|
|
two primary workflows.
|
|
- `workflow: replacement` treats the current planned outputs as the authoritative
|
|
managed output set for the destination scope and removes previously managed
|
|
outputs in that scope when they are no longer planned.
|
|
- `workflow: additive` writes or overwrites the planned output paths and retains
|
|
unrelated managed outputs.
|
|
- If a planned path already exists as a managed output, the new publication may
|
|
overwrite it and becomes that path's current owner.
|
|
- If a planned path exists in storage but is not recorded in valid catalog state,
|
|
it remains unmanaged content and must not be adopted implicitly.
|
|
- If an output path changes owner, preserve the output record's existing
|
|
`created_at` and update only `updated_at`.
|
|
- Catalog state records current ownership only. It does not retain historical
|
|
owners, historical sources, or old versions of overwritten output records.
|
|
- `pipeline_id` and `destination_id` are stored separately on each output
|
|
record. They are not concatenated into one owner string.
|
|
- Catalog state does not include top-level `owners`; owners are derivable from
|
|
the output records.
|
|
- Catalog state does not include top-level `sources`; compact source identity is
|
|
stored directly on each output record.
|
|
- Catalog state does not record whether the last run used `replacement` or
|
|
`additive`. Workflow is execution policy, and persisting it would create
|
|
drift risk if config changes later.
|
|
- Legacy destination policy fields should be rejected outright in the new
|
|
workflow config model. Do not retain aliases for `state`, `reconciliation`,
|
|
`takeover`, or `transfer`.
|
|
|
|
## Destination Workflow Semantics
|
|
|
|
The user-facing destination config should express intent directly:
|
|
|
|
```yaml
|
|
destinations:
|
|
- id: weather-latest
|
|
backend: local
|
|
path: /srv/reports/weather/latest
|
|
workflow: additive
|
|
```
|
|
|
|
```yaml
|
|
destinations:
|
|
- id: weather-archive
|
|
backend: local
|
|
path: /srv/reports/weather/archive
|
|
workflow: replacement
|
|
```
|
|
|
|
`workflow` is orthogonal to backend config, path mapping, publish source/HTML
|
|
selection, transforms, links, and retention.
|
|
|
|
### Replacement Workflow
|
|
|
|
A replacement destination is for producers that maintain a curated source of
|
|
truth and expect `distributor` to make the destination's managed scope match the
|
|
current publication.
|
|
|
|
For a destination configured with `workflow: replacement`:
|
|
|
|
- planned outputs are written to their resolved destination paths;
|
|
- planned outputs may overwrite existing managed outputs at the same path;
|
|
- each written output record is replaced in-place with the current
|
|
publication's owner, source identity, hash, size, and timestamps;
|
|
- managed outputs in the destination scope that are not present in the current
|
|
plan are deleted and removed from catalog state;
|
|
- unmanaged destination content remains unmanaged and blocks planned path
|
|
collisions unless an explicit force workflow later chooses otherwise;
|
|
- the catalog state shape remains the same as additive workflow state.
|
|
|
|
### Additive Workflow
|
|
|
|
An additive destination is for producers that routinely contribute outputs to a
|
|
shared destination root.
|
|
|
|
For a destination configured with `workflow: additive`:
|
|
|
|
- planned outputs are written to their resolved destination paths;
|
|
- planned outputs may overwrite existing managed outputs at the same path;
|
|
- each overwritten output record is replaced in-place with the current
|
|
publication's owner, source identity, hash, size, and timestamps;
|
|
- managed outputs not present in the current plan are retained;
|
|
- unrelated managed outputs from other pipelines or destinations are retained;
|
|
- unmanaged destination content remains unmanaged and blocks planned path
|
|
collisions unless an explicit force workflow later chooses otherwise;
|
|
- pruning can select retained managed outputs by `updated_at` and, when useful,
|
|
by `pipeline_id` and/or `destination_id`.
|
|
|
|
Replacement workflow treats the current publication as the desired managed
|
|
output set for a destination scope. Additive workflow treats the current
|
|
publication as a patch to the catalog of currently managed outputs.
|
|
|
|
## Destination State Schema Version 4
|
|
|
|
Catalog state should use this top-level shape:
|
|
|
|
```json
|
|
{
|
|
"schema_version": 4,
|
|
"distributor_version": "dev",
|
|
"created_at": "2026-06-19T12:00:00Z",
|
|
"updated_at": "2026-06-19T12:05:00Z",
|
|
"state": {
|
|
"mode": "catalog"
|
|
},
|
|
"outputs": []
|
|
}
|
|
```
|
|
|
|
Top-level fields:
|
|
|
|
- `schema_version`: required. Value `4` for catalog state.
|
|
- `distributor_version`: optional diagnostic version string.
|
|
- `created_at`: required RFC3339 timestamp for when the catalog state file was
|
|
first created.
|
|
- `updated_at`: required RFC3339 timestamp for the latest catalog state update.
|
|
- `state.mode`: required. Value `catalog`.
|
|
- `outputs`: required array of currently managed output records.
|
|
|
|
Catalog state should not include top-level `pipeline_id`, `destination_id`,
|
|
`published_at`, `workflow`, `owners`, `sources`, or a full source manifest.
|
|
Those concepts belong on output records or in configuration.
|
|
|
|
## Output Record Schema
|
|
|
|
Each output record should be self-contained enough to support current
|
|
ownership, pruning, repair, bitrot checks, and basic provenance without a
|
|
separate owner or source catalog.
|
|
|
|
```json
|
|
{
|
|
"path": "tomorrow/index.html",
|
|
"pipeline_id": "weatherreporter.daily",
|
|
"destination_id": "latest-html",
|
|
"source": {
|
|
"id": "weatherreporter.tomorrow",
|
|
"digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
|
"created": "2026-06-19T12:00:00Z"
|
|
},
|
|
"kind": "generated",
|
|
"source_path": "report.md",
|
|
"transform": "markdown_to_html",
|
|
"sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
|
|
"size": 12345,
|
|
"created_at": "2026-06-19T12:00:00Z",
|
|
"updated_at": "2026-06-19T12:05:00Z",
|
|
"url": "https://reports.example.com/weather/tomorrow/"
|
|
}
|
|
```
|
|
|
|
Required output fields:
|
|
|
|
- `path`: destination-relative managed output path.
|
|
- `pipeline_id`: pipeline that most recently wrote this output path.
|
|
- `destination_id`: destination that most recently wrote this output path.
|
|
- `source`: compact identity of the source bundle that produced this output.
|
|
- `source.id`: source manifest id.
|
|
- `source.digest`: source manifest digest.
|
|
- `source.created`: source manifest creation timestamp, RFC3339.
|
|
- `kind`: `source` or `generated`.
|
|
- `sha256`: digest of the output bytes.
|
|
- `size`: output byte size.
|
|
- `created_at`: RFC3339 timestamp for when this output path first became
|
|
managed in catalog state.
|
|
- `updated_at`: RFC3339 timestamp for when this output path was most recently
|
|
written or updated.
|
|
|
|
Optional output fields:
|
|
|
|
- `source_path`: present only for generated outputs; source manifest path used
|
|
to produce the generated output.
|
|
- `transform`: present only for generated outputs; transform name used to
|
|
produce the generated output.
|
|
- `url`: present only when destination link configuration produces a public URL
|
|
for this output.
|
|
|
|
For copied source outputs, `source_path` and `transform` should be omitted. The
|
|
output `path` is already the copied source artifact path.
|
|
|
|
## Useful Properties
|
|
|
|
The catalog shape supports both primary workflows without unnecessary
|
|
normalization:
|
|
|
|
- current owner of each path is explicit;
|
|
- current source identity for each path is explicit;
|
|
- bitrot checks can compare storage bytes to `sha256`;
|
|
- pruning can use `updated_at`, optionally scoped by `pipeline_id` and
|
|
`destination_id`;
|
|
- repair can remove missing managed output records without consulting an owner
|
|
or source table;
|
|
- overwriting an output path updates one output record in place;
|
|
- no orphaned top-level owner/source records need to be maintained.
|
|
|
|
The schema intentionally avoids storing a full source manifest for every output.
|
|
The source manifest remains the producer-to-distributor validation contract, but
|
|
catalog state only needs compact source identity for currently managed outputs.
|
|
|
|
## Relationship To Existing State And Config Modes
|
|
|
|
Existing state modes remain current behavior until catalog mode is implemented:
|
|
|
|
- `single_owner` schema version `2` supports one owner for a destination bundle
|
|
path.
|
|
- `shared_root` schema version `3` supports multiple owners in one destination
|
|
root but still keeps owner records with latest source manifests.
|
|
- `catalog` schema version `4` should support replacement and additive
|
|
current-state ownership without top-level owner or source catalogs.
|
|
|
|
The intended user-facing config should move toward `workflow: replacement` and
|
|
`workflow: additive` instead of requiring ordinary users to combine
|
|
`state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict
|
|
settings.
|
|
|
|
Because the project is still alpha pre-release, catalog implementation should be
|
|
a clean break:
|
|
|
|
- remove legacy write paths for current `single_owner` and `shared_root` state;
|
|
- do not implement migration from schema versions `1`, `2`, or `3`;
|
|
- do not preserve backwards compatibility for legacy destination state;
|
|
- when a configured destination writes successfully, write schema version `4`
|
|
catalog state;
|
|
- if an existing `.distributor.json` has schema version lower than `4`, treat it
|
|
as superseded legacy state for planning purposes and overwrite it according to
|
|
the configured workflow, without attempting conversion.
|
|
|
|
For the first catalog run against superseded legacy state:
|
|
|
|
- `workflow: replacement` may clear the bounded destination root before writing
|
|
the planned outputs and schema version `4` catalog state.
|
|
- `workflow: additive` may overwrite only the planned output paths, then write
|
|
schema version `4` catalog state containing those planned outputs.
|
|
- unplanned files left behind by an additive run against superseded legacy state
|
|
are not recorded in catalog state and are treated as unmanaged content by
|
|
later catalog runs.
|
|
|
|
The target direction is that both primary workflows use catalog state.
|
|
|
|
## Prune And Reconcile-State Scope
|
|
|
|
Catalog maintenance commands should initially keep the existing ownership
|
|
selector model:
|
|
|
|
- `prune --pipeline <id> --destination <id>` operates only on outputs currently
|
|
owned by that pipeline/destination.
|
|
- `reconcile-state --pipeline <id> --destination <id>` repairs only outputs
|
|
currently owned by that pipeline/destination.
|
|
- `reconcile-state --all-owners` repairs all output records in the catalog.
|
|
- `prune` remains scoped to the selected pipeline/destination owner only in the
|
|
initial catalog implementation.
|
|
|
|
Do not add catalog-specific selectors in the initial implementation, such as
|
|
`--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they
|
|
are not required to make additive workflow first-class.
|