Added a roadmap and implementation plan for a significant refactor of pipeline destination policy and catalog state
This commit is contained in:
283
docs/roadmap/catalog.md
Normal file
283
docs/roadmap/catalog.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# Catalog State And Destination Workflow Roadmap
|
||||
|
||||
This roadmap records the intended first-class state model and destination
|
||||
workflow configuration for replacement and additive publication workflows.
|
||||
|
||||
Current `single_owner` and `shared_root` state are oriented around comparing a
|
||||
destination owner to one latest source manifest. That works well for replacement
|
||||
workflows, where a producer maintains a curated source of truth and
|
||||
`distributor` makes a destination match it. It is a poor fit for additive
|
||||
workflows, where each run contributes new or updated output paths while
|
||||
preserving unrelated managed outputs from previous runs and other pipelines.
|
||||
|
||||
The target model is a current-state catalog: `.distributor.json` records the
|
||||
currently managed output paths at a destination root, and each output record
|
||||
stores its current owner, compact source identity, content digest, and update
|
||||
metadata. It is not intended to be an audit log.
|
||||
|
||||
Both replacement and additive workflows should use the same catalog state shape.
|
||||
The workflow choice is runtime policy from destination config, not persisted
|
||||
state.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
- Add one destination state mode named `catalog` for both replacement and
|
||||
additive workflows.
|
||||
- Catalog state writes `.distributor.json` schema version `4`.
|
||||
- Add a destination-level `workflow` setting with accepted values
|
||||
`replacement` and `additive`.
|
||||
- Default `workflow` to `additive` because it is the least destructive workflow.
|
||||
- `workflow` replaces the normal user-facing need to combine `state.mode`,
|
||||
`reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the
|
||||
two primary workflows.
|
||||
- `workflow: replacement` treats the current planned outputs as the authoritative
|
||||
managed output set for the destination scope and removes previously managed
|
||||
outputs in that scope when they are no longer planned.
|
||||
- `workflow: additive` writes or overwrites the planned output paths and retains
|
||||
unrelated managed outputs.
|
||||
- If a planned path already exists as a managed output, the new publication may
|
||||
overwrite it and becomes that path's current owner.
|
||||
- If a planned path exists in storage but is not recorded in valid catalog state,
|
||||
it remains unmanaged content and must not be adopted implicitly.
|
||||
- If an output path changes owner, preserve the output record's existing
|
||||
`created_at` and update only `updated_at`.
|
||||
- Catalog state records current ownership only. It does not retain historical
|
||||
owners, historical sources, or old versions of overwritten output records.
|
||||
- `pipeline_id` and `destination_id` are stored separately on each output
|
||||
record. They are not concatenated into one owner string.
|
||||
- Catalog state does not include top-level `owners`; owners are derivable from
|
||||
the output records.
|
||||
- Catalog state does not include top-level `sources`; compact source identity is
|
||||
stored directly on each output record.
|
||||
- Catalog state does not record whether the last run used `replacement` or
|
||||
`additive`. Workflow is execution policy, and persisting it would create
|
||||
drift risk if config changes later.
|
||||
- Legacy destination policy fields should be rejected outright in the new
|
||||
workflow config model. Do not retain aliases for `state`, `reconciliation`,
|
||||
`takeover`, or `transfer`.
|
||||
|
||||
## Destination Workflow Semantics
|
||||
|
||||
The user-facing destination config should express intent directly:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-latest
|
||||
backend: local
|
||||
path: /srv/reports/weather/latest
|
||||
workflow: additive
|
||||
```
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-archive
|
||||
backend: local
|
||||
path: /srv/reports/weather/archive
|
||||
workflow: replacement
|
||||
```
|
||||
|
||||
`workflow` is orthogonal to backend config, path mapping, publish source/HTML
|
||||
selection, transforms, links, and retention.
|
||||
|
||||
### Replacement Workflow
|
||||
|
||||
A replacement destination is for producers that maintain a curated source of
|
||||
truth and expect `distributor` to make the destination's managed scope match the
|
||||
current publication.
|
||||
|
||||
For a destination configured with `workflow: replacement`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each written output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs in the destination scope that are not present in the current
|
||||
plan are deleted and removed from catalog state;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- the catalog state shape remains the same as additive workflow state.
|
||||
|
||||
### Additive Workflow
|
||||
|
||||
An additive destination is for producers that routinely contribute outputs to a
|
||||
shared destination root.
|
||||
|
||||
For a destination configured with `workflow: additive`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each overwritten output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs not present in the current plan are retained;
|
||||
- unrelated managed outputs from other pipelines or destinations are retained;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- pruning can select retained managed outputs by `updated_at` and, when useful,
|
||||
by `pipeline_id` and/or `destination_id`.
|
||||
|
||||
Replacement workflow treats the current publication as the desired managed
|
||||
output set for a destination scope. Additive workflow treats the current
|
||||
publication as a patch to the catalog of currently managed outputs.
|
||||
|
||||
## Destination State Schema Version 4
|
||||
|
||||
Catalog state should use this top-level shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 4,
|
||||
"distributor_version": "dev",
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"state": {
|
||||
"mode": "catalog"
|
||||
},
|
||||
"outputs": []
|
||||
}
|
||||
```
|
||||
|
||||
Top-level fields:
|
||||
|
||||
- `schema_version`: required. Value `4` for catalog state.
|
||||
- `distributor_version`: optional diagnostic version string.
|
||||
- `created_at`: required RFC3339 timestamp for when the catalog state file was
|
||||
first created.
|
||||
- `updated_at`: required RFC3339 timestamp for the latest catalog state update.
|
||||
- `state.mode`: required. Value `catalog`.
|
||||
- `outputs`: required array of currently managed output records.
|
||||
|
||||
Catalog state should not include top-level `pipeline_id`, `destination_id`,
|
||||
`published_at`, `workflow`, `owners`, `sources`, or a full source manifest.
|
||||
Those concepts belong on output records or in configuration.
|
||||
|
||||
## Output Record Schema
|
||||
|
||||
Each output record should be self-contained enough to support current
|
||||
ownership, pruning, repair, bitrot checks, and basic provenance without a
|
||||
separate owner or source catalog.
|
||||
|
||||
```json
|
||||
{
|
||||
"path": "tomorrow/index.html",
|
||||
"pipeline_id": "weatherreporter.daily",
|
||||
"destination_id": "latest-html",
|
||||
"source": {
|
||||
"id": "weatherreporter.tomorrow",
|
||||
"digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
||||
"created": "2026-06-19T12:00:00Z"
|
||||
},
|
||||
"kind": "generated",
|
||||
"source_path": "report.md",
|
||||
"transform": "markdown_to_html",
|
||||
"sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
|
||||
"size": 12345,
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"url": "https://reports.example.com/weather/tomorrow/"
|
||||
}
|
||||
```
|
||||
|
||||
Required output fields:
|
||||
|
||||
- `path`: destination-relative managed output path.
|
||||
- `pipeline_id`: pipeline that most recently wrote this output path.
|
||||
- `destination_id`: destination that most recently wrote this output path.
|
||||
- `source`: compact identity of the source bundle that produced this output.
|
||||
- `source.id`: source manifest id.
|
||||
- `source.digest`: source manifest digest.
|
||||
- `source.created`: source manifest creation timestamp, RFC3339.
|
||||
- `kind`: `source` or `generated`.
|
||||
- `sha256`: digest of the output bytes.
|
||||
- `size`: output byte size.
|
||||
- `created_at`: RFC3339 timestamp for when this output path first became
|
||||
managed in catalog state.
|
||||
- `updated_at`: RFC3339 timestamp for when this output path was most recently
|
||||
written or updated.
|
||||
|
||||
Optional output fields:
|
||||
|
||||
- `source_path`: present only for generated outputs; source manifest path used
|
||||
to produce the generated output.
|
||||
- `transform`: present only for generated outputs; transform name used to
|
||||
produce the generated output.
|
||||
- `url`: present only when destination link configuration produces a public URL
|
||||
for this output.
|
||||
|
||||
For copied source outputs, `source_path` and `transform` should be omitted. The
|
||||
output `path` is already the copied source artifact path.
|
||||
|
||||
## Useful Properties
|
||||
|
||||
The catalog shape supports both primary workflows without unnecessary
|
||||
normalization:
|
||||
|
||||
- current owner of each path is explicit;
|
||||
- current source identity for each path is explicit;
|
||||
- bitrot checks can compare storage bytes to `sha256`;
|
||||
- pruning can use `updated_at`, optionally scoped by `pipeline_id` and
|
||||
`destination_id`;
|
||||
- repair can remove missing managed output records without consulting an owner
|
||||
or source table;
|
||||
- overwriting an output path updates one output record in place;
|
||||
- no orphaned top-level owner/source records need to be maintained.
|
||||
|
||||
The schema intentionally avoids storing a full source manifest for every output.
|
||||
The source manifest remains the producer-to-distributor validation contract, but
|
||||
catalog state only needs compact source identity for currently managed outputs.
|
||||
|
||||
## Relationship To Existing State And Config Modes
|
||||
|
||||
Existing state modes remain current behavior until catalog mode is implemented:
|
||||
|
||||
- `single_owner` schema version `2` supports one owner for a destination bundle
|
||||
path.
|
||||
- `shared_root` schema version `3` supports multiple owners in one destination
|
||||
root but still keeps owner records with latest source manifests.
|
||||
- `catalog` schema version `4` should support replacement and additive
|
||||
current-state ownership without top-level owner or source catalogs.
|
||||
|
||||
The intended user-facing config should move toward `workflow: replacement` and
|
||||
`workflow: additive` instead of requiring ordinary users to combine
|
||||
`state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict
|
||||
settings.
|
||||
|
||||
Because the project is still alpha pre-release, catalog implementation should be
|
||||
a clean break:
|
||||
|
||||
- remove legacy write paths for current `single_owner` and `shared_root` state;
|
||||
- do not implement migration from schema versions `1`, `2`, or `3`;
|
||||
- do not preserve backwards compatibility for legacy destination state;
|
||||
- when a configured destination writes successfully, write schema version `4`
|
||||
catalog state;
|
||||
- if an existing `.distributor.json` has schema version lower than `4`, treat it
|
||||
as superseded legacy state for planning purposes and overwrite it according to
|
||||
the configured workflow, without attempting conversion.
|
||||
|
||||
For the first catalog run against superseded legacy state:
|
||||
|
||||
- `workflow: replacement` may clear the bounded destination root before writing
|
||||
the planned outputs and schema version `4` catalog state.
|
||||
- `workflow: additive` may overwrite only the planned output paths, then write
|
||||
schema version `4` catalog state containing those planned outputs.
|
||||
- unplanned files left behind by an additive run against superseded legacy state
|
||||
are not recorded in catalog state and are treated as unmanaged content by
|
||||
later catalog runs.
|
||||
|
||||
The target direction is that both primary workflows use catalog state.
|
||||
|
||||
## Prune And Reconcile-State Scope
|
||||
|
||||
Catalog maintenance commands should initially keep the existing ownership
|
||||
selector model:
|
||||
|
||||
- `prune --pipeline <id> --destination <id>` operates only on outputs currently
|
||||
owned by that pipeline/destination.
|
||||
- `reconcile-state --pipeline <id> --destination <id>` repairs only outputs
|
||||
currently owned by that pipeline/destination.
|
||||
- `reconcile-state --all-owners` repairs all output records in the catalog.
|
||||
- `prune` remains scoped to the selected pipeline/destination owner only in the
|
||||
initial catalog implementation.
|
||||
|
||||
Do not add catalog-specific selectors in the initial implementation, such as
|
||||
`--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they
|
||||
are not required to make additive workflow first-class.
|
||||
Reference in New Issue
Block a user