eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Added a roadmap and implementation plan for a significant refactor of pipeline destination policy and catalog state

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-19 10:10:38 -05:00
parent 69043801d0
commit d23c624179
2 changed files with 543 additions and 290 deletions
Download Patch File Download Diff File Expand all files Collapse all files

283
docs/roadmap/catalog.md Normal file
View File

@@ -0,0 +1,283 @@
# Catalog State And Destination Workflow Roadmap
This roadmap records the intended first-class state model and destination
workflow configuration for replacement and additive publication workflows.
Current `single_owner` and `shared_root` state are oriented around comparing a
destination owner to one latest source manifest. That works well for replacement
workflows, where a producer maintains a curated source of truth and
`distributor` makes a destination match it. It is a poor fit for additive
workflows, where each run contributes new or updated output paths while
preserving unrelated managed outputs from previous runs and other pipelines.
The target model is a current-state catalog: `.distributor.json` records the
currently managed output paths at a destination root, and each output record
stores its current owner, compact source identity, content digest, and update
metadata. It is not intended to be an audit log.
Both replacement and additive workflows should use the same catalog state shape.
The workflow choice is runtime policy from destination config, not persisted
state.
## Locked Decisions
- Add one destination state mode named `catalog` for both replacement and
additive workflows.
- Catalog state writes `.distributor.json` schema version `4`.
- Add a destination-level `workflow` setting with accepted values
`replacement` and `additive`.
- Default `workflow` to `additive` because it is the least destructive workflow.
- `workflow` replaces the normal user-facing need to combine `state.mode`,
`reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the
two primary workflows.
- `workflow: replacement` treats the current planned outputs as the authoritative
managed output set for the destination scope and removes previously managed
outputs in that scope when they are no longer planned.
- `workflow: additive` writes or overwrites the planned output paths and retains
unrelated managed outputs.
- If a planned path already exists as a managed output, the new publication may
overwrite it and becomes that path's current owner.
- If a planned path exists in storage but is not recorded in valid catalog state,
it remains unmanaged content and must not be adopted implicitly.
- If an output path changes owner, preserve the output record's existing
`created_at` and update only `updated_at`.
- Catalog state records current ownership only. It does not retain historical
owners, historical sources, or old versions of overwritten output records.
- `pipeline_id` and `destination_id` are stored separately on each output
record. They are not concatenated into one owner string.
- Catalog state does not include top-level `owners`; owners are derivable from
the output records.
- Catalog state does not include top-level `sources`; compact source identity is
stored directly on each output record.
- Catalog state does not record whether the last run used `replacement` or
`additive`. Workflow is execution policy, and persisting it would create
drift risk if config changes later.
- Legacy destination policy fields should be rejected outright in the new
workflow config model. Do not retain aliases for `state`, `reconciliation`,
`takeover`, or `transfer`.
## Destination Workflow Semantics
The user-facing destination config should express intent directly:
```yaml
destinations:
- id: weather-latest
backend: local
path: /srv/reports/weather/latest
workflow: additive
```
```yaml
destinations:
- id: weather-archive
backend: local
path: /srv/reports/weather/archive
workflow: replacement
```
`workflow` is orthogonal to backend config, path mapping, publish source/HTML
selection, transforms, links, and retention.
### Replacement Workflow
A replacement destination is for producers that maintain a curated source of
truth and expect `distributor` to make the destination's managed scope match the
current publication.
For a destination configured with `workflow: replacement`:
- planned outputs are written to their resolved destination paths;
- planned outputs may overwrite existing managed outputs at the same path;
- each written output record is replaced in-place with the current
publication's owner, source identity, hash, size, and timestamps;
- managed outputs in the destination scope that are not present in the current
plan are deleted and removed from catalog state;
- unmanaged destination content remains unmanaged and blocks planned path
collisions unless an explicit force workflow later chooses otherwise;
- the catalog state shape remains the same as additive workflow state.
### Additive Workflow
An additive destination is for producers that routinely contribute outputs to a
shared destination root.
For a destination configured with `workflow: additive`:
- planned outputs are written to their resolved destination paths;
- planned outputs may overwrite existing managed outputs at the same path;
- each overwritten output record is replaced in-place with the current
publication's owner, source identity, hash, size, and timestamps;
- managed outputs not present in the current plan are retained;
- unrelated managed outputs from other pipelines or destinations are retained;
- unmanaged destination content remains unmanaged and blocks planned path
collisions unless an explicit force workflow later chooses otherwise;
- pruning can select retained managed outputs by `updated_at` and, when useful,
by `pipeline_id` and/or `destination_id`.
Replacement workflow treats the current publication as the desired managed
output set for a destination scope. Additive workflow treats the current
publication as a patch to the catalog of currently managed outputs.
## Destination State Schema Version 4
Catalog state should use this top-level shape:
```json
{
"schema_version": 4,
"distributor_version": "dev",
"created_at": "2026-06-19T12:00:00Z",
"updated_at": "2026-06-19T12:05:00Z",
"state": {
"mode": "catalog"
},
"outputs": []
}
```
Top-level fields:
- `schema_version`: required. Value `4` for catalog state.
- `distributor_version`: optional diagnostic version string.
- `created_at`: required RFC3339 timestamp for when the catalog state file was
first created.
- `updated_at`: required RFC3339 timestamp for the latest catalog state update.
- `state.mode`: required. Value `catalog`.
- `outputs`: required array of currently managed output records.
Catalog state should not include top-level `pipeline_id`, `destination_id`,
`published_at`, `workflow`, `owners`, `sources`, or a full source manifest.
Those concepts belong on output records or in configuration.
## Output Record Schema
Each output record should be self-contained enough to support current
ownership, pruning, repair, bitrot checks, and basic provenance without a
separate owner or source catalog.
```json
{
"path": "tomorrow/index.html",
"pipeline_id": "weatherreporter.daily",
"destination_id": "latest-html",
"source": {
"id": "weatherreporter.tomorrow",
"digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef",
"created": "2026-06-19T12:00:00Z"
},
"kind": "generated",
"source_path": "report.md",
"transform": "markdown_to_html",
"sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789",
"size": 12345,
"created_at": "2026-06-19T12:00:00Z",
"updated_at": "2026-06-19T12:05:00Z",
"url": "https://reports.example.com/weather/tomorrow/"
}
```
Required output fields:
- `path`: destination-relative managed output path.
- `pipeline_id`: pipeline that most recently wrote this output path.
- `destination_id`: destination that most recently wrote this output path.
- `source`: compact identity of the source bundle that produced this output.
- `source.id`: source manifest id.
- `source.digest`: source manifest digest.
- `source.created`: source manifest creation timestamp, RFC3339.
- `kind`: `source` or `generated`.
- `sha256`: digest of the output bytes.
- `size`: output byte size.
- `created_at`: RFC3339 timestamp for when this output path first became
managed in catalog state.
- `updated_at`: RFC3339 timestamp for when this output path was most recently
written or updated.
Optional output fields:
- `source_path`: present only for generated outputs; source manifest path used
to produce the generated output.
- `transform`: present only for generated outputs; transform name used to
produce the generated output.
- `url`: present only when destination link configuration produces a public URL
for this output.
For copied source outputs, `source_path` and `transform` should be omitted. The
output `path` is already the copied source artifact path.
## Useful Properties
The catalog shape supports both primary workflows without unnecessary
normalization:
- current owner of each path is explicit;
- current source identity for each path is explicit;
- bitrot checks can compare storage bytes to `sha256`;
- pruning can use `updated_at`, optionally scoped by `pipeline_id` and
`destination_id`;
- repair can remove missing managed output records without consulting an owner
or source table;
- overwriting an output path updates one output record in place;
- no orphaned top-level owner/source records need to be maintained.
The schema intentionally avoids storing a full source manifest for every output.
The source manifest remains the producer-to-distributor validation contract, but
catalog state only needs compact source identity for currently managed outputs.
## Relationship To Existing State And Config Modes
Existing state modes remain current behavior until catalog mode is implemented:
- `single_owner` schema version `2` supports one owner for a destination bundle
path.
- `shared_root` schema version `3` supports multiple owners in one destination
root but still keeps owner records with latest source manifests.
- `catalog` schema version `4` should support replacement and additive
current-state ownership without top-level owner or source catalogs.
The intended user-facing config should move toward `workflow: replacement` and
`workflow: additive` instead of requiring ordinary users to combine
`state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict
settings.
Because the project is still alpha pre-release, catalog implementation should be
a clean break:
- remove legacy write paths for current `single_owner` and `shared_root` state;
- do not implement migration from schema versions `1`, `2`, or `3`;
- do not preserve backwards compatibility for legacy destination state;
- when a configured destination writes successfully, write schema version `4`
catalog state;
- if an existing `.distributor.json` has schema version lower than `4`, treat it
as superseded legacy state for planning purposes and overwrite it according to
the configured workflow, without attempting conversion.
For the first catalog run against superseded legacy state:
- `workflow: replacement` may clear the bounded destination root before writing
the planned outputs and schema version `4` catalog state.
- `workflow: additive` may overwrite only the planned output paths, then write
schema version `4` catalog state containing those planned outputs.
- unplanned files left behind by an additive run against superseded legacy state
are not recorded in catalog state and are treated as unmanaged content by
later catalog runs.
The target direction is that both primary workflows use catalog state.
## Prune And Reconcile-State Scope
Catalog maintenance commands should initially keep the existing ownership
selector model:
- `prune --pipeline <id> --destination <id>` operates only on outputs currently
owned by that pipeline/destination.
- `reconcile-state --pipeline <id> --destination <id>` repairs only outputs
currently owned by that pipeline/destination.
- `reconcile-state --all-owners` repairs all output records in the catalog.
- `prune` remains scoped to the selected pipeline/destination owner only in the
initial catalog implementation.
Do not add catalog-specific selectors in the initial implementation, such as
`--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they
are not required to make additive workflow first-class.

550
docs/roadmap/implementation.md
View File

@@ -1,375 +1,345 @@
# Managed Destination Takeover Implementation Roadmap # Catalog State And Workflow Implementation Roadmap
This is the completed staged implementation plan for the takeover feature. The This is the active staged implementation plan for
detailed feature roadmap was removed after implementation; current behavior is `docs/roadmap/catalog.md`. The feature roadmap defines the target state model
documented outside `docs/roadmap/`. This document records the implementation and policy decisions; this document defines the implementation sequence for an
sequence used by LLM coding agents stage by stage. LLM coding agent to follow stage by stage.
Future behavior must remain under `docs/roadmap/` until implemented. Future behavior must remain under `docs/roadmap/` until implemented. Update
Preparatory internal stages should not update user-facing current docs. current-behavior docs only in the stage that implements the corresponding
Current-behavior docs should be updated when behavior is wired for behavior.
operator-facing use.
## Current Baseline ## Current Baseline
`distributor` already supports local, SSH/SFTP, S3, and HTTP upload source `distributor` currently writes destination `.distributor.json` using
workflows, destination state schemas for single-owner and shared-root state, single-owner schema version `2` or shared-root schema version `3`. Destination
path mapping, link generation, reconciliation, transfer policy, explicit behavior is selected through several low-level knobs: `state.mode`,
`--force`, and text/JSON run output. `reconciliation.mode`, `takeover.mode`, and `transfer`.
Current destination comparison is strict: The target behavior is a clean alpha break:
- same source manifest skips; - all newly written destination state uses schema version `4`;
- same source id with older destination state normally replaces; - all destinations use `state.mode: catalog` internally;
- same source id with newer destination state normally skips; - user-facing destination behavior is selected by `workflow: additive` or
- same source id and same creation time with different digest conflicts; `workflow: replacement`;
- different source id, pipeline id, destination id, or shared-root output owner - `workflow` defaults to `additive`;
conflicts unless explicit force policy applies; - legacy destination policy fields are rejected, not aliased;
- unmanaged content and invalid state do not become normal managed replacement - legacy state schema versions `1`, `2`, and `3` are not migrated.
cases.
The target feature adds destination-level `takeover.mode`, defaulting to
`same_pipeline`, so valid managed content can be replaced without `--force` when
the configured policy says that ownership/source takeover is expected.
## Implementation Principles ## Implementation Principles
- Preserve public behavior until the stage that explicitly changes it. - Keep the source manifest contract unchanged.
- Keep source manifests unchanged; takeover is destination configuration and - Keep backend adapters unaware of catalog and workflow policy.
publish planning policy. - Treat `workflow` as runtime config, not persisted state.
- Keep state comparison pure. State comparison may expose structured conflict - Keep unmanaged content protected after catalog state exists.
details, but publish planning decides whether takeover is allowed. - Preserve output `created_at` when a path changes owner; update only
- Keep adapters thin. No local, SSH, or S3 adapter should know takeover policy. `updated_at`.
- Keep unmanaged content and invalid destination state outside normal takeover. - Prefer removing legacy state/config paths over compatibility shims.
- Keep `reconciliation.mode`, `transfer`, `state.mode`, `path_mapping.mode`, and - Preserve current public APIs outside destination state/config unless the
`--force` as separate concepts. catalog roadmap explicitly changes them.
- Prefer narrow behavior-preserving refactors over broad publication rewrites.
- Update implemented-behavior docs in the same stage as the behavior change.
## Active Implementation Stages ## Active Implementation Stages
## Stage 1: Config Model And Validation ## Stage 1: Destination Workflow Config Clean Break
Goal: add the destination config field, defaults, and validation without Goal: replace low-level destination policy config with the user-facing workflow
changing publish behavior. switch.
Source roadmap reference:
- Completed takeover feature roadmap: Configuration, Policy Semantics, Safety
Rules.
Implementation scope: Implementation scope:
- Add destination-level `TakeoverPolicy` to `internal/config`. - Add destination `workflow` config with accepted values `additive` and
- Add constants for: `replacement`.
- `same_pipeline` - Default omitted `workflow` to `additive`.
- `same_source` - Remove or make unsupported the destination-level config fields `state`,
- `any_managed` `reconciliation`, `takeover`, and `transfer`.
- `never` - Ensure configs containing those legacy fields fail clearly. Prefer strict YAML
- Default `takeover.mode` to `same_pipeline` in config defaults. unknown-field failure by removing struct fields; add explicit validation only
- Validate accepted values with clear field context such as if clearer errors are needed without weakening strict decoding.
`pipelines[0].destinations[0].takeover.mode`. - Preserve backend, publish, transform, path mapping, links, retention, and
- Preserve strict YAML unknown-field behavior. other non-policy destination fields.
- Thread the defaulted policy into existing destination config views or helper - Update example configs only when the implementation stage also updates
structures if those are used by app/publish request construction. current docs; otherwise keep this stage focused on config code and tests.
- Do not change publish planning, execution, CLI output, or docs outside
roadmap files in this stage.
Tests: Tests:
- `go test ./internal/config` - `go test ./internal/config`
- Add config tests for omitted `takeover`, each accepted mode, invalid mode, and - Omitted workflow defaults to `additive`.
unknown nested fields. - `workflow: additive` and `workflow: replacement` validate.
- Add or update example-loading tests only if examples are touched, which should - Unknown workflow values fail.
not be necessary in this stage. - Legacy `state`, `reconciliation`, `takeover`, and `transfer` fields fail.
- Existing valid examples are updated or tests are adjusted in the same change if
examples currently use legacy fields.
Completion criteria: Completion criteria:
- Every destination has a defaulted `takeover.mode`. - New configs express destination replacement/additive intent with one field.
- Invalid values fail during config validation. - No normal config path accepts legacy destination policy knobs.
- No run behavior changes because publish planning does not consume the policy
yet.
## Stage 2: Structured State Comparison Details ## Stage 2: Catalog State Schema Version 4
Goal: expose enough structured comparison detail for publish planning to decide Goal: implement the schema version `4` catalog state model in `internal/state`.
takeover eligibility without parsing human-readable reason strings.
Source roadmap reference:
- Completed takeover feature roadmap: Policy Semantics, Relationship To Existing
Policies.
Implementation scope: Implementation scope:
- Extend `internal/state.Comparison` or add a package-local structured detail - Add catalog state types for top-level schema version `4`, `state.mode:
type so callers can distinguish: catalog`, and `outputs`.
- pipeline id mismatch; - Add compact per-output source identity with `id`, `digest`, and `created`.
- destination id mismatch; - Add catalog output records with required `path`, `pipeline_id`,
- different source id; `destination_id`, `source`, `kind`, `sha256`, `size`, `created_at`, and
- same-created digest conflict; `updated_at`.
- destination newer; - Add optional `source_path`, `transform`, and `url`, present only where allowed
- invalid state; by the catalog roadmap.
- unmanaged content; - Validate duplicate paths, invalid owner ids, invalid source identities,
- absent shared-root owner; invalid output paths, invalid digests, negative sizes, invalid timestamps,
- shared-root managed output owner conflicts, if those are currently reported invalid kind/transform combinations, and invalid URLs.
outside `internal/state`. - Marshal catalog state deterministically with the existing JSON formatting
- Keep existing outcome names and reason strings stable where practical. conventions.
- Keep comparison functions pure. They should report facts about existing state, - Update `ParseDocument` so schema version `4` returns catalog state.
not consult `takeover.mode`, `transfer`, `reconciliation`, or `force`. - Treat schema versions lower than `4` as superseded legacy state for publish
- Do not add new persisted state fields. planning, not as readable/migrated active state. Keep enough detection to
- Do not change publish actions in this stage. identify legacy state and avoid treating it as arbitrary invalid JSON.
- Reject unsupported future schema versions.
Tests: Tests:
- `go test ./internal/state ./internal/publish` - `go test ./internal/state`
- Add state tests for structured details on identity mismatch and different - Parse/validate/marshal valid catalog state.
source id. - Reject malformed catalog state and invalid output records.
- Add shared-root tests for owner absence and output ownership conflict detail, - Detect schema versions `1`, `2`, and `3` as superseded legacy state.
either in `internal/state` or `internal/publish` depending on where the - Reject future schema versions.
conflict is currently detected. - Prove no top-level `owners`, `sources`, workflow, source manifest, pipeline id,
- Preserve existing comparison outcome tests. destination id, or published timestamp is accepted for catalog state.
Completion criteria: Completion criteria:
- Publish planning can make takeover decisions from structured data. - State package has one canonical schema version `4` catalog contract for new
- Existing behavior remains unchanged because no takeover mapping is applied yet. writes.
- Legacy state is detected but not migrated.
## Stage 3: Single-Owner Takeover Planning And Execution ## Stage 3: Catalog Publish Planning
Goal: implement `takeover.mode` for single-owner destination state. Goal: plan publish actions against catalog state and destination workflow.
Source roadmap reference:
- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination
State Results, Safety Rules.
Implementation scope: Implementation scope:
- Add `TakeoverPolicy` or equivalent values to `internal/publish.Request`. - Replace publish request inputs that consume state/reconciliation/takeover and
- Pass destination takeover policy from app-level run planning into publish. transfer policy with destination workflow.
- Add publish action `replace_takeover`. - Map workflow to catalog actions:
- Map eligible single-owner conflicts to `replace_takeover`: - `workflow: additive`: upsert planned outputs and retain all other managed
- `same_pipeline`: existing state pipeline id equals current pipeline id, catalog outputs.
regardless of destination id or source id; - `workflow: replacement`: upsert planned outputs and delete catalog outputs
- `same_source`: existing state source id equals current source id; owned by the current pipeline/destination that are omitted from the plan.
- `any_managed`: existing state is valid distributor-managed state; - Planned paths that already exist as catalog-managed outputs may be overwritten
- `never`: no identity/source takeover. and become owned by the current pipeline/destination.
- Keep these cases failing by default unless existing force behavior applies: - Planned path collisions with storage content not recorded in catalog state
- unmanaged content; fail as unmanaged content.
- invalid state; - Matching planned outputs may skip writes when source identity and output digest
- same-created digest conflict; already match, if that optimization can be implemented without changing
- destination newer for the same source id unless `transfer` plus `--force` externally visible results; otherwise writing idempotently is acceptable.
already permits replacement; - Existing schema `< 4` state is superseded:
- conflicts not allowed by `takeover.mode`. - replacement workflow may plan a bounded destination-root clear before
- Execute `replace_takeover` through bounded managed replacement mechanics. writing planned outputs;
- For cross-source takeover, do not retain omitted outputs through - additive workflow may plan overwrites for planned paths only and leave
`reconciliation.mode: merge`; treat the affected single-owner state as a unplanned files unmanaged.
managed replacement so old-source outputs are not attributed to the new source. - Invalid JSON or future schema state remains a conflict, not a superseded
- Preserve existing `replace_older`, `skip_same`, `skip_destination_newer`, legacy state.
`fail_conflict`, `fail_unmanaged`, and `force_replace` behavior. - Preserve path mapping, publish policy, transforms, links, fixed-path
- Keep adapters unchanged. selection, and output collision checks.
Current-behavior documentation updates: Tests:
- Do not update user docs yet unless CLI output changes in this stage. Prefer - `go test ./internal/publish`
deferring user docs to Stage 5 so the behavior, output, and docs land - Additive workflow publishes new catalog state.
together. - Additive workflow overwrites an existing managed output and retains unrelated
- If this stage changes visible dry-run or run output enough that tests require outputs.
new wording, document only the implemented single-owner behavior and clearly - Replacement workflow removes omitted outputs owned by the current
leave shared-root takeover out until Stage 4. pipeline/destination.
- Replacement workflow preserves unrelated catalog outputs owned by other
pipeline/destination pairs.
- Managed output ownership moves to the current pipeline/destination when a
planned path is overwritten.
- `created_at` is preserved when an existing path changes owner; `updated_at`
changes.
- Unmanaged path collisions fail.
- Legacy schema `< 4` state follows the superseded-state rules above.
- Invalid/future state fails.
Completion criteria:
- Publish planning no longer depends on single-owner/shared-root comparison
semantics.
- Workflow behavior is fully driven by `workflow`.
## Stage 4: Catalog Publish Execution
Goal: execute catalog publish plans and write schema version `4` state.
Implementation scope:
- Write only schema version `4` catalog `.distributor.json`.
- Implement additive execution as managed upsert of planned outputs plus catalog
state update.
- Implement replacement execution as managed upsert plus deletion of omitted
current-owner catalog outputs.
- For replacement over superseded legacy state, clear the bounded destination
root before writing planned outputs and catalog state.
- For additive over superseded legacy state, overwrite planned paths and write a
catalog containing only planned outputs; leave unplanned files unmanaged.
- Preserve cleanup behavior on failed writes:
- additive cleanup removes outputs newly created by the failed attempt where
practical;
- replacement cleanup follows existing managed-replacement safety where
practical;
- state is written only after selected outputs are written.
- Preserve link metadata, generated output digests, source output digests,
content type behavior, and backend-safe writes.
Tests: Tests:
- `go test ./internal/publish ./internal/app` - `go test ./internal/publish ./internal/app`
- Default `same_pipeline` replaces a different source id from the same pipeline. - Additive execution writes planned outputs, overwrites managed planned paths,
- Default `same_pipeline` replaces state with a different destination id under retains unrelated managed outputs, and writes catalog state.
the same pipeline. - Replacement execution deletes omitted current-owner outputs and preserves
- Default `same_pipeline` refuses a different pipeline. unrelated owner outputs.
- `same_source` allows same source id and refuses different source id. - Superseded legacy replacement clears bounded destination root only.
- `any_managed` replaces valid state from a different pipeline. - Superseded legacy additive leaves unplanned files on disk but out of catalog
- `never` refuses identity/source takeover. state.
- Invalid state and unmanaged content still fail without force. - Failed writes do not leave misleading catalog state.
- Cross-source takeover with `reconciliation.mode: merge` does not retain - Local, fake-backed SSH, and fake-backed S3 app paths exercise the same publish
omitted outputs from the previous source. behavior.
- Dry-run plans `replace_takeover` without writing.
- Existing force tests still pass.
Completion criteria: Completion criteria:
- Single-owner latest-style destinations can be updated by different bundle ids - Successful `run` writes only v4 catalog state.
from the same configured pipeline without `--force`. - Destination outputs match additive/replacement workflow semantics.
- No unmanaged or invalid-state path becomes a normal takeover path.
## Stage 4: Shared-Root Takeover Planning And Execution ## Stage 5: Run Reporting, Notifications, And CLI Surface
Goal: apply the same takeover vocabulary to shared-root owner and output-path Goal: update user-visible run behavior to describe workflow/catalog actions
conflicts. instead of legacy replacement/takeover actions.
Source roadmap reference:
- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination
State Results, Safety Rules.
Implementation scope: Implementation scope:
- Extend shared-root planning so planned output collisions with existing managed - Replace legacy action labels tied to state/reconciliation/takeover/transfer
owners are eligible for takeover according to `takeover.mode`. with these stable workflow-oriented labels:
- Apply policy as follows: - `publish_new`
- `same_pipeline`: current owner may take over output paths owned by another - `upsert_additive`
destination under the same pipeline; - `replace_catalog`
- `same_source`: current owner may take over output paths whose owner records - `skip_same`
the same source id as the current source; - `force_replace`
- `any_managed`: current owner may take over output paths owned by any valid - `fail_unmanaged`
shared-root owner; - `fail_conflict`
- `never`: preserve current owner conflict behavior. - Ensure text and JSON run summaries include workflow-relevant counters.
- Preserve unrelated owner records and non-conflicting output records. - Include `workflow` in run action records where useful.
- Keep unmanaged path collisions failing without force. - Update fixed-path dry-run warnings to describe additive upsert or replacement
- Keep compatible single-owner-to-shared-root migration behavior intact. clearly.
- For cross-source takeover, do not retain omitted outputs from the previous - Ensure notifications use the new action labels.
source under the taking-over owner when `reconciliation.mode: merge` is set. - Remove reporting assumptions that depend on `replace_older`,
- Keep shared-root forced replacement behavior explicit and bounded as it is `replace_newer`, `replace_conflict`, or `replace_takeover`.
today.
Current-behavior documentation updates:
- Defer broad documentation updates to Stage 5 unless shared-root behavior must
be documented immediately to keep tests or generated docs consistent.
Tests: Tests:
- `go test ./internal/state ./internal/publish ./internal/app` - `go test ./internal/app ./internal/cli`
- `same_pipeline` permits taking over a path owned by another destination in the - Text dry-run output distinguishes additive from replacement workflow.
same pipeline. - JSON output includes workflow and stable action labels.
- `same_pipeline` refuses a path owned by another pipeline. - Summary counters are deterministic.
- `same_source` permits only matching source-id ownership transfer. - Notifications fire for additive and replacement writes.
- `any_managed` permits cross-pipeline managed ownership transfer. - Existing CLI commands still parse and execute with the new config shape.
- `never` preserves current shared-root conflict behavior.
- Reconciliation `replace` and `merge` handle omitted outputs according to the
feature roadmap, including the cross-source no-retain rule.
- Shared-root migration from compatible single-owner state still works.
- Unmanaged collisions still fail without force.
Completion criteria: Completion criteria:
- Single-owner and shared-root destinations use one consistent takeover policy. - Operators can understand from dry-run output whether a destination will upsert
- Shared-root takeover changes only affected owners/outputs and preserves or replace managed catalog outputs.
unrelated managed state.
## Stage 5: Run Output, Summaries, And Documentation ## Stage 6: Catalog Prune And Reconcile-State
Goal: make takeover behavior visible to operators and document the implemented Goal: update maintenance commands to operate on catalog state with the agreed
feature. selector model.
Source roadmap reference:
- Completed takeover feature roadmap: Documentation Impact, Publish Planning,
Relationship To Existing Policies.
Implementation scope: Implementation scope:
- Add `replace_takeover` to text run output. - Prune:
- Add `replace_takeover` to JSON run action output. - selected `--pipeline` and `--destination` prune only outputs currently owned
- Add a distinct summary counter for takeover replacements in text and JSON run by that pipeline/destination;
summaries, using the field name `replace_takeover` for JSON. - no `--all-owners` prune mode in the initial catalog implementation;
- Include takeover mode and conflict reason in action projection where useful - do not add `--source-id`, `--path-prefix`, or `--kind` selectors.
and consistent with existing output style. - Reconcile-state:
- Update fixed-path dry-run warnings when takeover will replace the destination - selected `--pipeline` and `--destination` repair only outputs currently
root. owned by that pipeline/destination;
- `--all-owners` repairs all catalog outputs;
- unmanaged reporting compares storage entries to all catalog output paths,
not just selected owner paths.
- Rewrite repaired/pruned state as schema version `4`.
- Remove legacy single-owner/shared-root maintenance branches.
Tests:
- `go test ./internal/state ./internal/app ./internal/cli`
- Prune selects only current-owner catalog outputs.
- Prune deletes selected outputs and removes their catalog records.
- Reconcile-state selected owner removes only missing outputs for that owner.
- `reconcile-state --all-owners` removes missing outputs for all owners.
- Unmanaged reporting excludes all catalog-managed paths and reports unrecorded
storage entries.
- JSON/text output remains stable and clear.
Completion criteria:
- Maintenance commands operate only on catalog state and respect the locked
selector rules.
## Stage 7: Clean Break Removal And Documentation
Goal: remove legacy state/config behavior and document the implemented catalog
workflow model.
Implementation scope:
- Remove dead code for writing schema version `2` single-owner and schema
version `3` shared-root state.
- Remove legacy config structs/constants/defaults/validation for destination
`state`, `reconciliation`, `takeover`, and `transfer` where no longer used.
- Remove or rewrite tests that only assert legacy state/config behavior.
- Update current-behavior docs: - Update current-behavior docs:
- `docs/config.md` - `docs/config.md`
- `docs/cli.md`
- `docs/operations.md` - `docs/operations.md`
- `docs/troubleshooting.md` - `docs/troubleshooting.md`
- `docs/integrations/destination-state.md` - `docs/integrations/destination-state.md`
- `docs/internal/publish.md` - relevant `docs/internal/` docs
- `docs/internal/state.md`
- `docs/internal/config.md`
- `docs/policy/architecture.md` - `docs/policy/architecture.md`
- `docs/policy/development.md` - `docs/policy/development.md`
- Keep docs concise and link to canonical references rather than duplicating - Update examples to use `workflow` and remove legacy fields.
full state semantics in every file. - Keep roadmap-only material out of current docs.
- Do not document any unimplemented future takeover extensions outside - Once implemented and documented, remove or rewrite
`docs/roadmap/`. `docs/roadmap/catalog.md` so completed behavior is not described only as
future work.
Tests: Tests and checks:
- `go test ./internal/app ./internal/cli ./internal/config`
- Text dry-run output includes `replace_takeover`.
- JSON output records `replace_takeover` with stable field names.
- Summary counters are deterministic.
- Fixed-path warnings remain deterministic.
- Config docs examples, if changed or added, load through existing config tests.
- Troubleshooting text matches actual error/action wording.
Completion criteria:
- Operators can distinguish ordinary same-source replacement, takeover
replacement, and explicit forced replacement.
- Current docs describe implemented takeover behavior outside roadmap files.
## Stage 6: Cross-Backend Regression And Roadmap Closeout
Goal: prove takeover behavior is backend-agnostic and clean up roadmap state
after implementation.
Source roadmap reference:
- Completed takeover feature roadmap: Goals, Non-Goals, Safety Rules.
Implementation scope:
- Add app-level coverage showing takeover planning/execution works with storage
abstraction rather than adapter-specific logic.
- Cover local destinations directly.
- Cover SSH and S3 through fake-backed app tests where possible; add adapter
integration tests only if existing test infrastructure already supports
opt-in remote credentials.
- Add tests for `path_mapping.mode: fixed` with different source ids under the
same pipeline.
- Add tests for archive-style `preserve_relative` destinations using
`takeover.mode: same_source` where strict source identity is desired.
- Run final consistency searches.
- Remove the completed detailed takeover roadmap when no future takeover work
remains.
Tests:
- `go test ./internal/config`
- `go test ./internal/state`
- `go test ./internal/publish`
- `go test ./internal/app ./internal/cli`
- `go test ./...` - `go test ./...`
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
Documentation and consistency checks: - `rg -n 'single_owner|shared_root|schema version `2`|schema version `3`' docs internal`
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
```sh
rg -n "takeover|replace_takeover|same_pipeline|same_source|any_managed" docs README.md examples
rg -n "destination source id differs from source|pipeline id .* does not match|destination id .* does not match" docs
rg -n "Managed Destination Takeover Roadmap|future takeover|planned takeover" docs README.md examples --glob '!docs/roadmap/**'
```
Completion criteria: Completion criteria:
- Current docs and examples describe the catalog workflow model.
- Legacy destination policy fields and legacy write paths are gone.
- Full test suite passes. - Full test suite passes.
- The default `same_pipeline` behavior applies only to valid
distributor-managed takeover cases.
- Unmanaged content and invalid state remain protected.
- Current docs no longer describe implemented takeover behavior as future work.
## Refactors To Avoid ## Refactors To Avoid
- Do not build a generic ownership engine outside `internal/state` and - Do not add top-level `owners` or `sources` catalogs.
`internal/publish`. - Do not persist workflow in `.distributor.json`.
- Do not move destination comparison policy into storage adapters. - Do not keep deprecated config aliases for legacy destination policy fields.
- Do not change the source manifest schema. - Do not implement state migration from schema versions `1`, `2`, or `3`.
- Do not add takeover fields to HTTP upload requests or producer APIs. - Do not add catalog-specific prune/reconcile selectors beyond the agreed
- Do not make `--force` persistent config. initial ownership scope.
- Do not merge `takeover`, `transfer`, and `reconciliation` into one broad - Do not move catalog policy into storage adapters.
policy object.
- Do not silently adopt unmanaged content.
## Open Questions ## Open Questions
No open questions are known. The feature roadmap selects the default mode No open questions are known. The catalog roadmap locks the state shape, workflow
(`same_pipeline`), the accepted modes, the safety boundary, and the relationship values, default workflow, clean-break policy, legacy-state behavior, output
to existing state, reconciliation, transfer, and force policies. field selection, `created_at` preservation, and maintenance command scope.