Added a roadmap and implementation plan for a significant refactor of pipeline destination policy and catalog state
This commit is contained in:
283
docs/roadmap/catalog.md
Normal file
283
docs/roadmap/catalog.md
Normal file
@@ -0,0 +1,283 @@
|
||||
# Catalog State And Destination Workflow Roadmap
|
||||
|
||||
This roadmap records the intended first-class state model and destination
|
||||
workflow configuration for replacement and additive publication workflows.
|
||||
|
||||
Current `single_owner` and `shared_root` state are oriented around comparing a
|
||||
destination owner to one latest source manifest. That works well for replacement
|
||||
workflows, where a producer maintains a curated source of truth and
|
||||
`distributor` makes a destination match it. It is a poor fit for additive
|
||||
workflows, where each run contributes new or updated output paths while
|
||||
preserving unrelated managed outputs from previous runs and other pipelines.
|
||||
|
||||
The target model is a current-state catalog: `.distributor.json` records the
|
||||
currently managed output paths at a destination root, and each output record
|
||||
stores its current owner, compact source identity, content digest, and update
|
||||
metadata. It is not intended to be an audit log.
|
||||
|
||||
Both replacement and additive workflows should use the same catalog state shape.
|
||||
The workflow choice is runtime policy from destination config, not persisted
|
||||
state.
|
||||
|
||||
## Locked Decisions
|
||||
|
||||
- Add one destination state mode named `catalog` for both replacement and
|
||||
additive workflows.
|
||||
- Catalog state writes `.distributor.json` schema version `4`.
|
||||
- Add a destination-level `workflow` setting with accepted values
|
||||
`replacement` and `additive`.
|
||||
- Default `workflow` to `additive` because it is the least destructive workflow.
|
||||
- `workflow` replaces the normal user-facing need to combine `state.mode`,
|
||||
`reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the
|
||||
two primary workflows.
|
||||
- `workflow: replacement` treats the current planned outputs as the authoritative
|
||||
managed output set for the destination scope and removes previously managed
|
||||
outputs in that scope when they are no longer planned.
|
||||
- `workflow: additive` writes or overwrites the planned output paths and retains
|
||||
unrelated managed outputs.
|
||||
- If a planned path already exists as a managed output, the new publication may
|
||||
overwrite it and becomes that path's current owner.
|
||||
- If a planned path exists in storage but is not recorded in valid catalog state,
|
||||
it remains unmanaged content and must not be adopted implicitly.
|
||||
- If an output path changes owner, preserve the output record's existing
|
||||
`created_at` and update only `updated_at`.
|
||||
- Catalog state records current ownership only. It does not retain historical
|
||||
owners, historical sources, or old versions of overwritten output records.
|
||||
- `pipeline_id` and `destination_id` are stored separately on each output
|
||||
record. They are not concatenated into one owner string.
|
||||
- Catalog state does not include top-level `owners`; owners are derivable from
|
||||
the output records.
|
||||
- Catalog state does not include top-level `sources`; compact source identity is
|
||||
stored directly on each output record.
|
||||
- Catalog state does not record whether the last run used `replacement` or
|
||||
`additive`. Workflow is execution policy, and persisting it would create
|
||||
drift risk if config changes later.
|
||||
- Legacy destination policy fields should be rejected outright in the new
|
||||
workflow config model. Do not retain aliases for `state`, `reconciliation`,
|
||||
`takeover`, or `transfer`.
|
||||
|
||||
## Destination Workflow Semantics
|
||||
|
||||
The user-facing destination config should express intent directly:
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-latest
|
||||
backend: local
|
||||
path: /srv/reports/weather/latest
|
||||
workflow: additive
|
||||
```
|
||||
|
||||
```yaml
|
||||
destinations:
|
||||
- id: weather-archive
|
||||
backend: local
|
||||
path: /srv/reports/weather/archive
|
||||
workflow: replacement
|
||||
```
|
||||
|
||||
`workflow` is orthogonal to backend config, path mapping, publish source/HTML
|
||||
selection, transforms, links, and retention.
|
||||
|
||||
### Replacement Workflow
|
||||
|
||||
A replacement destination is for producers that maintain a curated source of
|
||||
truth and expect `distributor` to make the destination's managed scope match the
|
||||
current publication.
|
||||
|
||||
For a destination configured with `workflow: replacement`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each written output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs in the destination scope that are not present in the current
|
||||
plan are deleted and removed from catalog state;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- the catalog state shape remains the same as additive workflow state.
|
||||
|
||||
### Additive Workflow
|
||||
|
||||
An additive destination is for producers that routinely contribute outputs to a
|
||||
shared destination root.
|
||||
|
||||
For a destination configured with `workflow: additive`:
|
||||
|
||||
- planned outputs are written to their resolved destination paths;
|
||||
- planned outputs may overwrite existing managed outputs at the same path;
|
||||
- each overwritten output record is replaced in-place with the current
|
||||
publication's owner, source identity, hash, size, and timestamps;
|
||||
- managed outputs not present in the current plan are retained;
|
||||
- unrelated managed outputs from other pipelines or destinations are retained;
|
||||
- unmanaged destination content remains unmanaged and blocks planned path
|
||||
collisions unless an explicit force workflow later chooses otherwise;
|
||||
- pruning can select retained managed outputs by `updated_at` and, when useful,
|
||||
by `pipeline_id` and/or `destination_id`.
|
||||
|
||||
Replacement workflow treats the current publication as the desired managed
|
||||
output set for a destination scope. Additive workflow treats the current
|
||||
publication as a patch to the catalog of currently managed outputs.
|
||||
|
||||
## Destination State Schema Version 4
|
||||
|
||||
Catalog state should use this top-level shape:
|
||||
|
||||
```json
|
||||
{
|
||||
"schema_version": 4,
|
||||
"distributor_version": "dev",
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"state": {
|
||||
"mode": "catalog"
|
||||
},
|
||||
"outputs": []
|
||||
}
|
||||
```
|
||||
|
||||
Top-level fields:
|
||||
|
||||
- `schema_version`: required. Value `4` for catalog state.
|
||||
- `distributor_version`: optional diagnostic version string.
|
||||
- `created_at`: required RFC3339 timestamp for when the catalog state file was
|
||||
first created.
|
||||
- `updated_at`: required RFC3339 timestamp for the latest catalog state update.
|
||||
- `state.mode`: required. Value `catalog`.
|
||||
- `outputs`: required array of currently managed output records.
|
||||
|
||||
Catalog state should not include top-level `pipeline_id`, `destination_id`,
|
||||
`published_at`, `workflow`, `owners`, `sources`, or a full source manifest.
|
||||
Those concepts belong on output records or in configuration.
|
||||
|
||||
## Output Record Schema
|
||||
|
||||
Each output record should be self-contained enough to support current
|
||||
ownership, pruning, repair, bitrot checks, and basic provenance without a
|
||||
separate owner or source catalog.
|
||||
|
||||
```json
|
||||
{
|
||||
"path": "tomorrow/index.html",
|
||||
"pipeline_id": "weatherreporter.daily",
|
||||
"destination_id": "latest-html",
|
||||
"source": {
|
||||
"id": "weatherreporter.tomorrow",
|
||||
"digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
|
||||
"created": "2026-06-19T12:00:00Z"
|
||||
},
|
||||
"kind": "generated",
|
||||
"source_path": "report.md",
|
||||
"transform": "markdown_to_html",
|
||||
"sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
|
||||
"size": 12345,
|
||||
"created_at": "2026-06-19T12:00:00Z",
|
||||
"updated_at": "2026-06-19T12:05:00Z",
|
||||
"url": "https://reports.example.com/weather/tomorrow/"
|
||||
}
|
||||
```
|
||||
|
||||
Required output fields:
|
||||
|
||||
- `path`: destination-relative managed output path.
|
||||
- `pipeline_id`: pipeline that most recently wrote this output path.
|
||||
- `destination_id`: destination that most recently wrote this output path.
|
||||
- `source`: compact identity of the source bundle that produced this output.
|
||||
- `source.id`: source manifest id.
|
||||
- `source.digest`: source manifest digest.
|
||||
- `source.created`: source manifest creation timestamp, RFC3339.
|
||||
- `kind`: `source` or `generated`.
|
||||
- `sha256`: digest of the output bytes.
|
||||
- `size`: output byte size.
|
||||
- `created_at`: RFC3339 timestamp for when this output path first became
|
||||
managed in catalog state.
|
||||
- `updated_at`: RFC3339 timestamp for when this output path was most recently
|
||||
written or updated.
|
||||
|
||||
Optional output fields:
|
||||
|
||||
- `source_path`: present only for generated outputs; source manifest path used
|
||||
to produce the generated output.
|
||||
- `transform`: present only for generated outputs; transform name used to
|
||||
produce the generated output.
|
||||
- `url`: present only when destination link configuration produces a public URL
|
||||
for this output.
|
||||
|
||||
For copied source outputs, `source_path` and `transform` should be omitted. The
|
||||
output `path` is already the copied source artifact path.
|
||||
|
||||
## Useful Properties
|
||||
|
||||
The catalog shape supports both primary workflows without unnecessary
|
||||
normalization:
|
||||
|
||||
- current owner of each path is explicit;
|
||||
- current source identity for each path is explicit;
|
||||
- bitrot checks can compare storage bytes to `sha256`;
|
||||
- pruning can use `updated_at`, optionally scoped by `pipeline_id` and
|
||||
`destination_id`;
|
||||
- repair can remove missing managed output records without consulting an owner
|
||||
or source table;
|
||||
- overwriting an output path updates one output record in place;
|
||||
- no orphaned top-level owner/source records need to be maintained.
|
||||
|
||||
The schema intentionally avoids storing a full source manifest for every output.
|
||||
The source manifest remains the producer-to-distributor validation contract, but
|
||||
catalog state only needs compact source identity for currently managed outputs.
|
||||
|
||||
## Relationship To Existing State And Config Modes
|
||||
|
||||
Existing state modes remain current behavior until catalog mode is implemented:
|
||||
|
||||
- `single_owner` schema version `2` supports one owner for a destination bundle
|
||||
path.
|
||||
- `shared_root` schema version `3` supports multiple owners in one destination
|
||||
root but still keeps owner records with latest source manifests.
|
||||
- `catalog` schema version `4` should support replacement and additive
|
||||
current-state ownership without top-level owner or source catalogs.
|
||||
|
||||
The intended user-facing config should move toward `workflow: replacement` and
|
||||
`workflow: additive` instead of requiring ordinary users to combine
|
||||
`state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict
|
||||
settings.
|
||||
|
||||
Because the project is still alpha pre-release, catalog implementation should be
|
||||
a clean break:
|
||||
|
||||
- remove legacy write paths for current `single_owner` and `shared_root` state;
|
||||
- do not implement migration from schema versions `1`, `2`, or `3`;
|
||||
- do not preserve backwards compatibility for legacy destination state;
|
||||
- when a configured destination writes successfully, write schema version `4`
|
||||
catalog state;
|
||||
- if an existing `.distributor.json` has schema version lower than `4`, treat it
|
||||
as superseded legacy state for planning purposes and overwrite it according to
|
||||
the configured workflow, without attempting conversion.
|
||||
|
||||
For the first catalog run against superseded legacy state:
|
||||
|
||||
- `workflow: replacement` may clear the bounded destination root before writing
|
||||
the planned outputs and schema version `4` catalog state.
|
||||
- `workflow: additive` may overwrite only the planned output paths, then write
|
||||
schema version `4` catalog state containing those planned outputs.
|
||||
- unplanned files left behind by an additive run against superseded legacy state
|
||||
are not recorded in catalog state and are treated as unmanaged content by
|
||||
later catalog runs.
|
||||
|
||||
The target direction is that both primary workflows use catalog state.
|
||||
|
||||
## Prune And Reconcile-State Scope
|
||||
|
||||
Catalog maintenance commands should initially keep the existing ownership
|
||||
selector model:
|
||||
|
||||
- `prune --pipeline <id> --destination <id>` operates only on outputs currently
|
||||
owned by that pipeline/destination.
|
||||
- `reconcile-state --pipeline <id> --destination <id>` repairs only outputs
|
||||
currently owned by that pipeline/destination.
|
||||
- `reconcile-state --all-owners` repairs all output records in the catalog.
|
||||
- `prune` remains scoped to the selected pipeline/destination owner only in the
|
||||
initial catalog implementation.
|
||||
|
||||
Do not add catalog-specific selectors in the initial implementation, such as
|
||||
`--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they
|
||||
are not required to make additive workflow first-class.
|
||||
@@ -1,375 +1,345 @@
|
||||
# Managed Destination Takeover Implementation Roadmap
|
||||
# Catalog State And Workflow Implementation Roadmap
|
||||
|
||||
This is the completed staged implementation plan for the takeover feature. The
|
||||
detailed feature roadmap was removed after implementation; current behavior is
|
||||
documented outside `docs/roadmap/`. This document records the implementation
|
||||
sequence used by LLM coding agents stage by stage.
|
||||
This is the active staged implementation plan for
|
||||
`docs/roadmap/catalog.md`. The feature roadmap defines the target state model
|
||||
and policy decisions; this document defines the implementation sequence for an
|
||||
LLM coding agent to follow stage by stage.
|
||||
|
||||
Future behavior must remain under `docs/roadmap/` until implemented.
|
||||
Preparatory internal stages should not update user-facing current docs.
|
||||
Current-behavior docs should be updated when behavior is wired for
|
||||
operator-facing use.
|
||||
Future behavior must remain under `docs/roadmap/` until implemented. Update
|
||||
current-behavior docs only in the stage that implements the corresponding
|
||||
behavior.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
`distributor` already supports local, SSH/SFTP, S3, and HTTP upload source
|
||||
workflows, destination state schemas for single-owner and shared-root state,
|
||||
path mapping, link generation, reconciliation, transfer policy, explicit
|
||||
`--force`, and text/JSON run output.
|
||||
`distributor` currently writes destination `.distributor.json` using
|
||||
single-owner schema version `2` or shared-root schema version `3`. Destination
|
||||
behavior is selected through several low-level knobs: `state.mode`,
|
||||
`reconciliation.mode`, `takeover.mode`, and `transfer`.
|
||||
|
||||
Current destination comparison is strict:
|
||||
The target behavior is a clean alpha break:
|
||||
|
||||
- same source manifest skips;
|
||||
- same source id with older destination state normally replaces;
|
||||
- same source id with newer destination state normally skips;
|
||||
- same source id and same creation time with different digest conflicts;
|
||||
- different source id, pipeline id, destination id, or shared-root output owner
|
||||
conflicts unless explicit force policy applies;
|
||||
- unmanaged content and invalid state do not become normal managed replacement
|
||||
cases.
|
||||
|
||||
The target feature adds destination-level `takeover.mode`, defaulting to
|
||||
`same_pipeline`, so valid managed content can be replaced without `--force` when
|
||||
the configured policy says that ownership/source takeover is expected.
|
||||
- all newly written destination state uses schema version `4`;
|
||||
- all destinations use `state.mode: catalog` internally;
|
||||
- user-facing destination behavior is selected by `workflow: additive` or
|
||||
`workflow: replacement`;
|
||||
- `workflow` defaults to `additive`;
|
||||
- legacy destination policy fields are rejected, not aliased;
|
||||
- legacy state schema versions `1`, `2`, and `3` are not migrated.
|
||||
|
||||
## Implementation Principles
|
||||
|
||||
- Preserve public behavior until the stage that explicitly changes it.
|
||||
- Keep source manifests unchanged; takeover is destination configuration and
|
||||
publish planning policy.
|
||||
- Keep state comparison pure. State comparison may expose structured conflict
|
||||
details, but publish planning decides whether takeover is allowed.
|
||||
- Keep adapters thin. No local, SSH, or S3 adapter should know takeover policy.
|
||||
- Keep unmanaged content and invalid destination state outside normal takeover.
|
||||
- Keep `reconciliation.mode`, `transfer`, `state.mode`, `path_mapping.mode`, and
|
||||
`--force` as separate concepts.
|
||||
- Prefer narrow behavior-preserving refactors over broad publication rewrites.
|
||||
- Update implemented-behavior docs in the same stage as the behavior change.
|
||||
- Keep the source manifest contract unchanged.
|
||||
- Keep backend adapters unaware of catalog and workflow policy.
|
||||
- Treat `workflow` as runtime config, not persisted state.
|
||||
- Keep unmanaged content protected after catalog state exists.
|
||||
- Preserve output `created_at` when a path changes owner; update only
|
||||
`updated_at`.
|
||||
- Prefer removing legacy state/config paths over compatibility shims.
|
||||
- Preserve current public APIs outside destination state/config unless the
|
||||
catalog roadmap explicitly changes them.
|
||||
|
||||
## Active Implementation Stages
|
||||
|
||||
## Stage 1: Config Model And Validation
|
||||
## Stage 1: Destination Workflow Config Clean Break
|
||||
|
||||
Goal: add the destination config field, defaults, and validation without
|
||||
changing publish behavior.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Configuration, Policy Semantics, Safety
|
||||
Rules.
|
||||
Goal: replace low-level destination policy config with the user-facing workflow
|
||||
switch.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add destination-level `TakeoverPolicy` to `internal/config`.
|
||||
- Add constants for:
|
||||
- `same_pipeline`
|
||||
- `same_source`
|
||||
- `any_managed`
|
||||
- `never`
|
||||
- Default `takeover.mode` to `same_pipeline` in config defaults.
|
||||
- Validate accepted values with clear field context such as
|
||||
`pipelines[0].destinations[0].takeover.mode`.
|
||||
- Preserve strict YAML unknown-field behavior.
|
||||
- Thread the defaulted policy into existing destination config views or helper
|
||||
structures if those are used by app/publish request construction.
|
||||
- Do not change publish planning, execution, CLI output, or docs outside
|
||||
roadmap files in this stage.
|
||||
- Add destination `workflow` config with accepted values `additive` and
|
||||
`replacement`.
|
||||
- Default omitted `workflow` to `additive`.
|
||||
- Remove or make unsupported the destination-level config fields `state`,
|
||||
`reconciliation`, `takeover`, and `transfer`.
|
||||
- Ensure configs containing those legacy fields fail clearly. Prefer strict YAML
|
||||
unknown-field failure by removing struct fields; add explicit validation only
|
||||
if clearer errors are needed without weakening strict decoding.
|
||||
- Preserve backend, publish, transform, path mapping, links, retention, and
|
||||
other non-policy destination fields.
|
||||
- Update example configs only when the implementation stage also updates
|
||||
current docs; otherwise keep this stage focused on config code and tests.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/config`
|
||||
- Add config tests for omitted `takeover`, each accepted mode, invalid mode, and
|
||||
unknown nested fields.
|
||||
- Add or update example-loading tests only if examples are touched, which should
|
||||
not be necessary in this stage.
|
||||
- Omitted workflow defaults to `additive`.
|
||||
- `workflow: additive` and `workflow: replacement` validate.
|
||||
- Unknown workflow values fail.
|
||||
- Legacy `state`, `reconciliation`, `takeover`, and `transfer` fields fail.
|
||||
- Existing valid examples are updated or tests are adjusted in the same change if
|
||||
examples currently use legacy fields.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Every destination has a defaulted `takeover.mode`.
|
||||
- Invalid values fail during config validation.
|
||||
- No run behavior changes because publish planning does not consume the policy
|
||||
yet.
|
||||
- New configs express destination replacement/additive intent with one field.
|
||||
- No normal config path accepts legacy destination policy knobs.
|
||||
|
||||
## Stage 2: Structured State Comparison Details
|
||||
## Stage 2: Catalog State Schema Version 4
|
||||
|
||||
Goal: expose enough structured comparison detail for publish planning to decide
|
||||
takeover eligibility without parsing human-readable reason strings.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Policy Semantics, Relationship To Existing
|
||||
Policies.
|
||||
Goal: implement the schema version `4` catalog state model in `internal/state`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Extend `internal/state.Comparison` or add a package-local structured detail
|
||||
type so callers can distinguish:
|
||||
- pipeline id mismatch;
|
||||
- destination id mismatch;
|
||||
- different source id;
|
||||
- same-created digest conflict;
|
||||
- destination newer;
|
||||
- invalid state;
|
||||
- unmanaged content;
|
||||
- absent shared-root owner;
|
||||
- shared-root managed output owner conflicts, if those are currently reported
|
||||
outside `internal/state`.
|
||||
- Keep existing outcome names and reason strings stable where practical.
|
||||
- Keep comparison functions pure. They should report facts about existing state,
|
||||
not consult `takeover.mode`, `transfer`, `reconciliation`, or `force`.
|
||||
- Do not add new persisted state fields.
|
||||
- Do not change publish actions in this stage.
|
||||
- Add catalog state types for top-level schema version `4`, `state.mode:
|
||||
catalog`, and `outputs`.
|
||||
- Add compact per-output source identity with `id`, `digest`, and `created`.
|
||||
- Add catalog output records with required `path`, `pipeline_id`,
|
||||
`destination_id`, `source`, `kind`, `sha256`, `size`, `created_at`, and
|
||||
`updated_at`.
|
||||
- Add optional `source_path`, `transform`, and `url`, present only where allowed
|
||||
by the catalog roadmap.
|
||||
- Validate duplicate paths, invalid owner ids, invalid source identities,
|
||||
invalid output paths, invalid digests, negative sizes, invalid timestamps,
|
||||
invalid kind/transform combinations, and invalid URLs.
|
||||
- Marshal catalog state deterministically with the existing JSON formatting
|
||||
conventions.
|
||||
- Update `ParseDocument` so schema version `4` returns catalog state.
|
||||
- Treat schema versions lower than `4` as superseded legacy state for publish
|
||||
planning, not as readable/migrated active state. Keep enough detection to
|
||||
identify legacy state and avoid treating it as arbitrary invalid JSON.
|
||||
- Reject unsupported future schema versions.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/publish`
|
||||
- Add state tests for structured details on identity mismatch and different
|
||||
source id.
|
||||
- Add shared-root tests for owner absence and output ownership conflict detail,
|
||||
either in `internal/state` or `internal/publish` depending on where the
|
||||
conflict is currently detected.
|
||||
- Preserve existing comparison outcome tests.
|
||||
- `go test ./internal/state`
|
||||
- Parse/validate/marshal valid catalog state.
|
||||
- Reject malformed catalog state and invalid output records.
|
||||
- Detect schema versions `1`, `2`, and `3` as superseded legacy state.
|
||||
- Reject future schema versions.
|
||||
- Prove no top-level `owners`, `sources`, workflow, source manifest, pipeline id,
|
||||
destination id, or published timestamp is accepted for catalog state.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Publish planning can make takeover decisions from structured data.
|
||||
- Existing behavior remains unchanged because no takeover mapping is applied yet.
|
||||
- State package has one canonical schema version `4` catalog contract for new
|
||||
writes.
|
||||
- Legacy state is detected but not migrated.
|
||||
|
||||
## Stage 3: Single-Owner Takeover Planning And Execution
|
||||
## Stage 3: Catalog Publish Planning
|
||||
|
||||
Goal: implement `takeover.mode` for single-owner destination state.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination
|
||||
State Results, Safety Rules.
|
||||
Goal: plan publish actions against catalog state and destination workflow.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add `TakeoverPolicy` or equivalent values to `internal/publish.Request`.
|
||||
- Pass destination takeover policy from app-level run planning into publish.
|
||||
- Add publish action `replace_takeover`.
|
||||
- Map eligible single-owner conflicts to `replace_takeover`:
|
||||
- `same_pipeline`: existing state pipeline id equals current pipeline id,
|
||||
regardless of destination id or source id;
|
||||
- `same_source`: existing state source id equals current source id;
|
||||
- `any_managed`: existing state is valid distributor-managed state;
|
||||
- `never`: no identity/source takeover.
|
||||
- Keep these cases failing by default unless existing force behavior applies:
|
||||
- unmanaged content;
|
||||
- invalid state;
|
||||
- same-created digest conflict;
|
||||
- destination newer for the same source id unless `transfer` plus `--force`
|
||||
already permits replacement;
|
||||
- conflicts not allowed by `takeover.mode`.
|
||||
- Execute `replace_takeover` through bounded managed replacement mechanics.
|
||||
- For cross-source takeover, do not retain omitted outputs through
|
||||
`reconciliation.mode: merge`; treat the affected single-owner state as a
|
||||
managed replacement so old-source outputs are not attributed to the new source.
|
||||
- Preserve existing `replace_older`, `skip_same`, `skip_destination_newer`,
|
||||
`fail_conflict`, `fail_unmanaged`, and `force_replace` behavior.
|
||||
- Keep adapters unchanged.
|
||||
- Replace publish request inputs that consume state/reconciliation/takeover and
|
||||
transfer policy with destination workflow.
|
||||
- Map workflow to catalog actions:
|
||||
- `workflow: additive`: upsert planned outputs and retain all other managed
|
||||
catalog outputs.
|
||||
- `workflow: replacement`: upsert planned outputs and delete catalog outputs
|
||||
owned by the current pipeline/destination that are omitted from the plan.
|
||||
- Planned paths that already exist as catalog-managed outputs may be overwritten
|
||||
and become owned by the current pipeline/destination.
|
||||
- Planned path collisions with storage content not recorded in catalog state
|
||||
fail as unmanaged content.
|
||||
- Matching planned outputs may skip writes when source identity and output digest
|
||||
already match, if that optimization can be implemented without changing
|
||||
externally visible results; otherwise writing idempotently is acceptable.
|
||||
- Existing schema `< 4` state is superseded:
|
||||
- replacement workflow may plan a bounded destination-root clear before
|
||||
writing planned outputs;
|
||||
- additive workflow may plan overwrites for planned paths only and leave
|
||||
unplanned files unmanaged.
|
||||
- Invalid JSON or future schema state remains a conflict, not a superseded
|
||||
legacy state.
|
||||
- Preserve path mapping, publish policy, transforms, links, fixed-path
|
||||
selection, and output collision checks.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
Tests:
|
||||
|
||||
- Do not update user docs yet unless CLI output changes in this stage. Prefer
|
||||
deferring user docs to Stage 5 so the behavior, output, and docs land
|
||||
together.
|
||||
- If this stage changes visible dry-run or run output enough that tests require
|
||||
new wording, document only the implemented single-owner behavior and clearly
|
||||
leave shared-root takeover out until Stage 4.
|
||||
- `go test ./internal/publish`
|
||||
- Additive workflow publishes new catalog state.
|
||||
- Additive workflow overwrites an existing managed output and retains unrelated
|
||||
outputs.
|
||||
- Replacement workflow removes omitted outputs owned by the current
|
||||
pipeline/destination.
|
||||
- Replacement workflow preserves unrelated catalog outputs owned by other
|
||||
pipeline/destination pairs.
|
||||
- Managed output ownership moves to the current pipeline/destination when a
|
||||
planned path is overwritten.
|
||||
- `created_at` is preserved when an existing path changes owner; `updated_at`
|
||||
changes.
|
||||
- Unmanaged path collisions fail.
|
||||
- Legacy schema `< 4` state follows the superseded-state rules above.
|
||||
- Invalid/future state fails.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Publish planning no longer depends on single-owner/shared-root comparison
|
||||
semantics.
|
||||
- Workflow behavior is fully driven by `workflow`.
|
||||
|
||||
## Stage 4: Catalog Publish Execution
|
||||
|
||||
Goal: execute catalog publish plans and write schema version `4` state.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Write only schema version `4` catalog `.distributor.json`.
|
||||
- Implement additive execution as managed upsert of planned outputs plus catalog
|
||||
state update.
|
||||
- Implement replacement execution as managed upsert plus deletion of omitted
|
||||
current-owner catalog outputs.
|
||||
- For replacement over superseded legacy state, clear the bounded destination
|
||||
root before writing planned outputs and catalog state.
|
||||
- For additive over superseded legacy state, overwrite planned paths and write a
|
||||
catalog containing only planned outputs; leave unplanned files unmanaged.
|
||||
- Preserve cleanup behavior on failed writes:
|
||||
- additive cleanup removes outputs newly created by the failed attempt where
|
||||
practical;
|
||||
- replacement cleanup follows existing managed-replacement safety where
|
||||
practical;
|
||||
- state is written only after selected outputs are written.
|
||||
- Preserve link metadata, generated output digests, source output digests,
|
||||
content type behavior, and backend-safe writes.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/publish ./internal/app`
|
||||
- Default `same_pipeline` replaces a different source id from the same pipeline.
|
||||
- Default `same_pipeline` replaces state with a different destination id under
|
||||
the same pipeline.
|
||||
- Default `same_pipeline` refuses a different pipeline.
|
||||
- `same_source` allows same source id and refuses different source id.
|
||||
- `any_managed` replaces valid state from a different pipeline.
|
||||
- `never` refuses identity/source takeover.
|
||||
- Invalid state and unmanaged content still fail without force.
|
||||
- Cross-source takeover with `reconciliation.mode: merge` does not retain
|
||||
omitted outputs from the previous source.
|
||||
- Dry-run plans `replace_takeover` without writing.
|
||||
- Existing force tests still pass.
|
||||
- Additive execution writes planned outputs, overwrites managed planned paths,
|
||||
retains unrelated managed outputs, and writes catalog state.
|
||||
- Replacement execution deletes omitted current-owner outputs and preserves
|
||||
unrelated owner outputs.
|
||||
- Superseded legacy replacement clears bounded destination root only.
|
||||
- Superseded legacy additive leaves unplanned files on disk but out of catalog
|
||||
state.
|
||||
- Failed writes do not leave misleading catalog state.
|
||||
- Local, fake-backed SSH, and fake-backed S3 app paths exercise the same publish
|
||||
behavior.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Single-owner latest-style destinations can be updated by different bundle ids
|
||||
from the same configured pipeline without `--force`.
|
||||
- No unmanaged or invalid-state path becomes a normal takeover path.
|
||||
- Successful `run` writes only v4 catalog state.
|
||||
- Destination outputs match additive/replacement workflow semantics.
|
||||
|
||||
## Stage 4: Shared-Root Takeover Planning And Execution
|
||||
## Stage 5: Run Reporting, Notifications, And CLI Surface
|
||||
|
||||
Goal: apply the same takeover vocabulary to shared-root owner and output-path
|
||||
conflicts.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination
|
||||
State Results, Safety Rules.
|
||||
Goal: update user-visible run behavior to describe workflow/catalog actions
|
||||
instead of legacy replacement/takeover actions.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Extend shared-root planning so planned output collisions with existing managed
|
||||
owners are eligible for takeover according to `takeover.mode`.
|
||||
- Apply policy as follows:
|
||||
- `same_pipeline`: current owner may take over output paths owned by another
|
||||
destination under the same pipeline;
|
||||
- `same_source`: current owner may take over output paths whose owner records
|
||||
the same source id as the current source;
|
||||
- `any_managed`: current owner may take over output paths owned by any valid
|
||||
shared-root owner;
|
||||
- `never`: preserve current owner conflict behavior.
|
||||
- Preserve unrelated owner records and non-conflicting output records.
|
||||
- Keep unmanaged path collisions failing without force.
|
||||
- Keep compatible single-owner-to-shared-root migration behavior intact.
|
||||
- For cross-source takeover, do not retain omitted outputs from the previous
|
||||
source under the taking-over owner when `reconciliation.mode: merge` is set.
|
||||
- Keep shared-root forced replacement behavior explicit and bounded as it is
|
||||
today.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
|
||||
- Defer broad documentation updates to Stage 5 unless shared-root behavior must
|
||||
be documented immediately to keep tests or generated docs consistent.
|
||||
- Replace legacy action labels tied to state/reconciliation/takeover/transfer
|
||||
with these stable workflow-oriented labels:
|
||||
- `publish_new`
|
||||
- `upsert_additive`
|
||||
- `replace_catalog`
|
||||
- `skip_same`
|
||||
- `force_replace`
|
||||
- `fail_unmanaged`
|
||||
- `fail_conflict`
|
||||
- Ensure text and JSON run summaries include workflow-relevant counters.
|
||||
- Include `workflow` in run action records where useful.
|
||||
- Update fixed-path dry-run warnings to describe additive upsert or replacement
|
||||
clearly.
|
||||
- Ensure notifications use the new action labels.
|
||||
- Remove reporting assumptions that depend on `replace_older`,
|
||||
`replace_newer`, `replace_conflict`, or `replace_takeover`.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/publish ./internal/app`
|
||||
- `same_pipeline` permits taking over a path owned by another destination in the
|
||||
same pipeline.
|
||||
- `same_pipeline` refuses a path owned by another pipeline.
|
||||
- `same_source` permits only matching source-id ownership transfer.
|
||||
- `any_managed` permits cross-pipeline managed ownership transfer.
|
||||
- `never` preserves current shared-root conflict behavior.
|
||||
- Reconciliation `replace` and `merge` handle omitted outputs according to the
|
||||
feature roadmap, including the cross-source no-retain rule.
|
||||
- Shared-root migration from compatible single-owner state still works.
|
||||
- Unmanaged collisions still fail without force.
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- Text dry-run output distinguishes additive from replacement workflow.
|
||||
- JSON output includes workflow and stable action labels.
|
||||
- Summary counters are deterministic.
|
||||
- Notifications fire for additive and replacement writes.
|
||||
- Existing CLI commands still parse and execute with the new config shape.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Single-owner and shared-root destinations use one consistent takeover policy.
|
||||
- Shared-root takeover changes only affected owners/outputs and preserves
|
||||
unrelated managed state.
|
||||
- Operators can understand from dry-run output whether a destination will upsert
|
||||
or replace managed catalog outputs.
|
||||
|
||||
## Stage 5: Run Output, Summaries, And Documentation
|
||||
## Stage 6: Catalog Prune And Reconcile-State
|
||||
|
||||
Goal: make takeover behavior visible to operators and document the implemented
|
||||
feature.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Documentation Impact, Publish Planning,
|
||||
Relationship To Existing Policies.
|
||||
Goal: update maintenance commands to operate on catalog state with the agreed
|
||||
selector model.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add `replace_takeover` to text run output.
|
||||
- Add `replace_takeover` to JSON run action output.
|
||||
- Add a distinct summary counter for takeover replacements in text and JSON run
|
||||
summaries, using the field name `replace_takeover` for JSON.
|
||||
- Include takeover mode and conflict reason in action projection where useful
|
||||
and consistent with existing output style.
|
||||
- Update fixed-path dry-run warnings when takeover will replace the destination
|
||||
root.
|
||||
- Prune:
|
||||
- selected `--pipeline` and `--destination` prune only outputs currently owned
|
||||
by that pipeline/destination;
|
||||
- no `--all-owners` prune mode in the initial catalog implementation;
|
||||
- do not add `--source-id`, `--path-prefix`, or `--kind` selectors.
|
||||
- Reconcile-state:
|
||||
- selected `--pipeline` and `--destination` repair only outputs currently
|
||||
owned by that pipeline/destination;
|
||||
- `--all-owners` repairs all catalog outputs;
|
||||
- unmanaged reporting compares storage entries to all catalog output paths,
|
||||
not just selected owner paths.
|
||||
- Rewrite repaired/pruned state as schema version `4`.
|
||||
- Remove legacy single-owner/shared-root maintenance branches.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/app ./internal/cli`
|
||||
- Prune selects only current-owner catalog outputs.
|
||||
- Prune deletes selected outputs and removes their catalog records.
|
||||
- Reconcile-state selected owner removes only missing outputs for that owner.
|
||||
- `reconcile-state --all-owners` removes missing outputs for all owners.
|
||||
- Unmanaged reporting excludes all catalog-managed paths and reports unrecorded
|
||||
storage entries.
|
||||
- JSON/text output remains stable and clear.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Maintenance commands operate only on catalog state and respect the locked
|
||||
selector rules.
|
||||
|
||||
## Stage 7: Clean Break Removal And Documentation
|
||||
|
||||
Goal: remove legacy state/config behavior and document the implemented catalog
|
||||
workflow model.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Remove dead code for writing schema version `2` single-owner and schema
|
||||
version `3` shared-root state.
|
||||
- Remove legacy config structs/constants/defaults/validation for destination
|
||||
`state`, `reconciliation`, `takeover`, and `transfer` where no longer used.
|
||||
- Remove or rewrite tests that only assert legacy state/config behavior.
|
||||
- Update current-behavior docs:
|
||||
- `docs/config.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/integrations/destination-state.md`
|
||||
- `docs/internal/publish.md`
|
||||
- `docs/internal/state.md`
|
||||
- `docs/internal/config.md`
|
||||
- relevant `docs/internal/` docs
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/development.md`
|
||||
- Keep docs concise and link to canonical references rather than duplicating
|
||||
full state semantics in every file.
|
||||
- Do not document any unimplemented future takeover extensions outside
|
||||
`docs/roadmap/`.
|
||||
- Update examples to use `workflow` and remove legacy fields.
|
||||
- Keep roadmap-only material out of current docs.
|
||||
- Once implemented and documented, remove or rewrite
|
||||
`docs/roadmap/catalog.md` so completed behavior is not described only as
|
||||
future work.
|
||||
|
||||
Tests:
|
||||
Tests and checks:
|
||||
|
||||
- `go test ./internal/app ./internal/cli ./internal/config`
|
||||
- Text dry-run output includes `replace_takeover`.
|
||||
- JSON output records `replace_takeover` with stable field names.
|
||||
- Summary counters are deterministic.
|
||||
- Fixed-path warnings remain deterministic.
|
||||
- Config docs examples, if changed or added, load through existing config tests.
|
||||
- Troubleshooting text matches actual error/action wording.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Operators can distinguish ordinary same-source replacement, takeover
|
||||
replacement, and explicit forced replacement.
|
||||
- Current docs describe implemented takeover behavior outside roadmap files.
|
||||
|
||||
## Stage 6: Cross-Backend Regression And Roadmap Closeout
|
||||
|
||||
Goal: prove takeover behavior is backend-agnostic and clean up roadmap state
|
||||
after implementation.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Goals, Non-Goals, Safety Rules.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add app-level coverage showing takeover planning/execution works with storage
|
||||
abstraction rather than adapter-specific logic.
|
||||
- Cover local destinations directly.
|
||||
- Cover SSH and S3 through fake-backed app tests where possible; add adapter
|
||||
integration tests only if existing test infrastructure already supports
|
||||
opt-in remote credentials.
|
||||
- Add tests for `path_mapping.mode: fixed` with different source ids under the
|
||||
same pipeline.
|
||||
- Add tests for archive-style `preserve_relative` destinations using
|
||||
`takeover.mode: same_source` where strict source identity is desired.
|
||||
- Run final consistency searches.
|
||||
- Remove the completed detailed takeover roadmap when no future takeover work
|
||||
remains.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/config`
|
||||
- `go test ./internal/state`
|
||||
- `go test ./internal/publish`
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- `go test ./...`
|
||||
|
||||
Documentation and consistency checks:
|
||||
|
||||
```sh
|
||||
rg -n "takeover|replace_takeover|same_pipeline|same_source|any_managed" docs README.md examples
|
||||
rg -n "destination source id differs from source|pipeline id .* does not match|destination id .* does not match" docs
|
||||
rg -n "Managed Destination Takeover Roadmap|future takeover|planned takeover" docs README.md examples --glob '!docs/roadmap/**'
|
||||
```
|
||||
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
|
||||
- `rg -n 'single_owner|shared_root|schema version `2`|schema version `3`' docs internal`
|
||||
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Current docs and examples describe the catalog workflow model.
|
||||
- Legacy destination policy fields and legacy write paths are gone.
|
||||
- Full test suite passes.
|
||||
- The default `same_pipeline` behavior applies only to valid
|
||||
distributor-managed takeover cases.
|
||||
- Unmanaged content and invalid state remain protected.
|
||||
- Current docs no longer describe implemented takeover behavior as future work.
|
||||
|
||||
## Refactors To Avoid
|
||||
|
||||
- Do not build a generic ownership engine outside `internal/state` and
|
||||
`internal/publish`.
|
||||
- Do not move destination comparison policy into storage adapters.
|
||||
- Do not change the source manifest schema.
|
||||
- Do not add takeover fields to HTTP upload requests or producer APIs.
|
||||
- Do not make `--force` persistent config.
|
||||
- Do not merge `takeover`, `transfer`, and `reconciliation` into one broad
|
||||
policy object.
|
||||
- Do not silently adopt unmanaged content.
|
||||
- Do not add top-level `owners` or `sources` catalogs.
|
||||
- Do not persist workflow in `.distributor.json`.
|
||||
- Do not keep deprecated config aliases for legacy destination policy fields.
|
||||
- Do not implement state migration from schema versions `1`, `2`, or `3`.
|
||||
- Do not add catalog-specific prune/reconcile selectors beyond the agreed
|
||||
initial ownership scope.
|
||||
- Do not move catalog policy into storage adapters.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No open questions are known. The feature roadmap selects the default mode
|
||||
(`same_pipeline`), the accepted modes, the safety boundary, and the relationship
|
||||
to existing state, reconciliation, transfer, and force policies.
|
||||
No open questions are known. The catalog roadmap locks the state shape, workflow
|
||||
values, default workflow, clean-break policy, legacy-state behavior, output
|
||||
field selection, `created_at` preservation, and maintenance command scope.
|
||||
|
||||
Reference in New Issue
Block a user