From d23c624179b7ee7b0913877f2df3d2aa3d47791c Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Fri, 19 Jun 2026 10:10:38 -0500 Subject: [PATCH] Added a roadmap and implementation plan for a significant refactor of pipeline destination policy and catalog state --- docs/roadmap/catalog.md | 283 +++++++++++++++++ docs/roadmap/implementation.md | 550 ++++++++++++++++----------------- 2 files changed, 543 insertions(+), 290 deletions(-) create mode 100644 docs/roadmap/catalog.md diff --git a/docs/roadmap/catalog.md b/docs/roadmap/catalog.md new file mode 100644 index 0000000..e95d524 --- /dev/null +++ b/docs/roadmap/catalog.md @@ -0,0 +1,283 @@ +# Catalog State And Destination Workflow Roadmap + +This roadmap records the intended first-class state model and destination +workflow configuration for replacement and additive publication workflows. + +Current `single_owner` and `shared_root` state are oriented around comparing a +destination owner to one latest source manifest. That works well for replacement +workflows, where a producer maintains a curated source of truth and +`distributor` makes a destination match it. It is a poor fit for additive +workflows, where each run contributes new or updated output paths while +preserving unrelated managed outputs from previous runs and other pipelines. + +The target model is a current-state catalog: `.distributor.json` records the +currently managed output paths at a destination root, and each output record +stores its current owner, compact source identity, content digest, and update +metadata. It is not intended to be an audit log. + +Both replacement and additive workflows should use the same catalog state shape. +The workflow choice is runtime policy from destination config, not persisted +state. + +## Locked Decisions + +- Add one destination state mode named `catalog` for both replacement and + additive workflows. +- Catalog state writes `.distributor.json` schema version `4`. +- Add a destination-level `workflow` setting with accepted values + `replacement` and `additive`. +- Default `workflow` to `additive` because it is the least destructive workflow. +- `workflow` replaces the normal user-facing need to combine `state.mode`, + `reconciliation.mode`, `takeover.mode`, and transfer conflict settings for the + two primary workflows. +- `workflow: replacement` treats the current planned outputs as the authoritative + managed output set for the destination scope and removes previously managed + outputs in that scope when they are no longer planned. +- `workflow: additive` writes or overwrites the planned output paths and retains + unrelated managed outputs. +- If a planned path already exists as a managed output, the new publication may + overwrite it and becomes that path's current owner. +- If a planned path exists in storage but is not recorded in valid catalog state, + it remains unmanaged content and must not be adopted implicitly. +- If an output path changes owner, preserve the output record's existing + `created_at` and update only `updated_at`. +- Catalog state records current ownership only. It does not retain historical + owners, historical sources, or old versions of overwritten output records. +- `pipeline_id` and `destination_id` are stored separately on each output + record. They are not concatenated into one owner string. +- Catalog state does not include top-level `owners`; owners are derivable from + the output records. +- Catalog state does not include top-level `sources`; compact source identity is + stored directly on each output record. +- Catalog state does not record whether the last run used `replacement` or + `additive`. Workflow is execution policy, and persisting it would create + drift risk if config changes later. +- Legacy destination policy fields should be rejected outright in the new + workflow config model. Do not retain aliases for `state`, `reconciliation`, + `takeover`, or `transfer`. + +## Destination Workflow Semantics + +The user-facing destination config should express intent directly: + +```yaml +destinations: + - id: weather-latest + backend: local + path: /srv/reports/weather/latest + workflow: additive +``` + +```yaml +destinations: + - id: weather-archive + backend: local + path: /srv/reports/weather/archive + workflow: replacement +``` + +`workflow` is orthogonal to backend config, path mapping, publish source/HTML +selection, transforms, links, and retention. + +### Replacement Workflow + +A replacement destination is for producers that maintain a curated source of +truth and expect `distributor` to make the destination's managed scope match the +current publication. + +For a destination configured with `workflow: replacement`: + +- planned outputs are written to their resolved destination paths; +- planned outputs may overwrite existing managed outputs at the same path; +- each written output record is replaced in-place with the current + publication's owner, source identity, hash, size, and timestamps; +- managed outputs in the destination scope that are not present in the current + plan are deleted and removed from catalog state; +- unmanaged destination content remains unmanaged and blocks planned path + collisions unless an explicit force workflow later chooses otherwise; +- the catalog state shape remains the same as additive workflow state. + +### Additive Workflow + +An additive destination is for producers that routinely contribute outputs to a +shared destination root. + +For a destination configured with `workflow: additive`: + +- planned outputs are written to their resolved destination paths; +- planned outputs may overwrite existing managed outputs at the same path; +- each overwritten output record is replaced in-place with the current + publication's owner, source identity, hash, size, and timestamps; +- managed outputs not present in the current plan are retained; +- unrelated managed outputs from other pipelines or destinations are retained; +- unmanaged destination content remains unmanaged and blocks planned path + collisions unless an explicit force workflow later chooses otherwise; +- pruning can select retained managed outputs by `updated_at` and, when useful, + by `pipeline_id` and/or `destination_id`. + +Replacement workflow treats the current publication as the desired managed +output set for a destination scope. Additive workflow treats the current +publication as a patch to the catalog of currently managed outputs. + +## Destination State Schema Version 4 + +Catalog state should use this top-level shape: + +```json +{ + "schema_version": 4, + "distributor_version": "dev", + "created_at": "2026-06-19T12:00:00Z", + "updated_at": "2026-06-19T12:05:00Z", + "state": { + "mode": "catalog" + }, + "outputs": [] +} +``` + +Top-level fields: + +- `schema_version`: required. Value `4` for catalog state. +- `distributor_version`: optional diagnostic version string. +- `created_at`: required RFC3339 timestamp for when the catalog state file was + first created. +- `updated_at`: required RFC3339 timestamp for the latest catalog state update. +- `state.mode`: required. Value `catalog`. +- `outputs`: required array of currently managed output records. + +Catalog state should not include top-level `pipeline_id`, `destination_id`, +`published_at`, `workflow`, `owners`, `sources`, or a full source manifest. +Those concepts belong on output records or in configuration. + +## Output Record Schema + +Each output record should be self-contained enough to support current +ownership, pruning, repair, bitrot checks, and basic provenance without a +separate owner or source catalog. + +```json +{ + "path": "tomorrow/index.html", + "pipeline_id": "weatherreporter.daily", + "destination_id": "latest-html", + "source": { + "id": "weatherreporter.tomorrow", + "digest": "sha256:0123456789abcdef0123456789abcdef0123456789abcdef0123456789abcdef", + "created": "2026-06-19T12:00:00Z" + }, + "kind": "generated", + "source_path": "report.md", + "transform": "markdown_to_html", + "sha256": "sha256:abcdef0123456789abcdef0123456789abcdef0123456789abcdef0123456789", + "size": 12345, + "created_at": "2026-06-19T12:00:00Z", + "updated_at": "2026-06-19T12:05:00Z", + "url": "https://reports.example.com/weather/tomorrow/" +} +``` + +Required output fields: + +- `path`: destination-relative managed output path. +- `pipeline_id`: pipeline that most recently wrote this output path. +- `destination_id`: destination that most recently wrote this output path. +- `source`: compact identity of the source bundle that produced this output. +- `source.id`: source manifest id. +- `source.digest`: source manifest digest. +- `source.created`: source manifest creation timestamp, RFC3339. +- `kind`: `source` or `generated`. +- `sha256`: digest of the output bytes. +- `size`: output byte size. +- `created_at`: RFC3339 timestamp for when this output path first became + managed in catalog state. +- `updated_at`: RFC3339 timestamp for when this output path was most recently + written or updated. + +Optional output fields: + +- `source_path`: present only for generated outputs; source manifest path used + to produce the generated output. +- `transform`: present only for generated outputs; transform name used to + produce the generated output. +- `url`: present only when destination link configuration produces a public URL + for this output. + +For copied source outputs, `source_path` and `transform` should be omitted. The +output `path` is already the copied source artifact path. + +## Useful Properties + +The catalog shape supports both primary workflows without unnecessary +normalization: + +- current owner of each path is explicit; +- current source identity for each path is explicit; +- bitrot checks can compare storage bytes to `sha256`; +- pruning can use `updated_at`, optionally scoped by `pipeline_id` and + `destination_id`; +- repair can remove missing managed output records without consulting an owner + or source table; +- overwriting an output path updates one output record in place; +- no orphaned top-level owner/source records need to be maintained. + +The schema intentionally avoids storing a full source manifest for every output. +The source manifest remains the producer-to-distributor validation contract, but +catalog state only needs compact source identity for currently managed outputs. + +## Relationship To Existing State And Config Modes + +Existing state modes remain current behavior until catalog mode is implemented: + +- `single_owner` schema version `2` supports one owner for a destination bundle + path. +- `shared_root` schema version `3` supports multiple owners in one destination + root but still keeps owner records with latest source manifests. +- `catalog` schema version `4` should support replacement and additive + current-state ownership without top-level owner or source catalogs. + +The intended user-facing config should move toward `workflow: replacement` and +`workflow: additive` instead of requiring ordinary users to combine +`state.mode`, `reconciliation.mode`, `takeover.mode`, and transfer conflict +settings. + +Because the project is still alpha pre-release, catalog implementation should be +a clean break: + +- remove legacy write paths for current `single_owner` and `shared_root` state; +- do not implement migration from schema versions `1`, `2`, or `3`; +- do not preserve backwards compatibility for legacy destination state; +- when a configured destination writes successfully, write schema version `4` + catalog state; +- if an existing `.distributor.json` has schema version lower than `4`, treat it + as superseded legacy state for planning purposes and overwrite it according to + the configured workflow, without attempting conversion. + +For the first catalog run against superseded legacy state: + +- `workflow: replacement` may clear the bounded destination root before writing + the planned outputs and schema version `4` catalog state. +- `workflow: additive` may overwrite only the planned output paths, then write + schema version `4` catalog state containing those planned outputs. +- unplanned files left behind by an additive run against superseded legacy state + are not recorded in catalog state and are treated as unmanaged content by + later catalog runs. + +The target direction is that both primary workflows use catalog state. + +## Prune And Reconcile-State Scope + +Catalog maintenance commands should initially keep the existing ownership +selector model: + +- `prune --pipeline --destination ` operates only on outputs currently + owned by that pipeline/destination. +- `reconcile-state --pipeline --destination ` repairs only outputs + currently owned by that pipeline/destination. +- `reconcile-state --all-owners` repairs all output records in the catalog. +- `prune` remains scoped to the selected pipeline/destination owner only in the + initial catalog implementation. + +Do not add catalog-specific selectors in the initial implementation, such as +`--source-id`, `--path-prefix`, or `--kind`. Those may be useful later, but they +are not required to make additive workflow first-class. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index f5815a1..2e052bd 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,375 +1,345 @@ -# Managed Destination Takeover Implementation Roadmap +# Catalog State And Workflow Implementation Roadmap -This is the completed staged implementation plan for the takeover feature. The -detailed feature roadmap was removed after implementation; current behavior is -documented outside `docs/roadmap/`. This document records the implementation -sequence used by LLM coding agents stage by stage. +This is the active staged implementation plan for +`docs/roadmap/catalog.md`. The feature roadmap defines the target state model +and policy decisions; this document defines the implementation sequence for an +LLM coding agent to follow stage by stage. -Future behavior must remain under `docs/roadmap/` until implemented. -Preparatory internal stages should not update user-facing current docs. -Current-behavior docs should be updated when behavior is wired for -operator-facing use. +Future behavior must remain under `docs/roadmap/` until implemented. Update +current-behavior docs only in the stage that implements the corresponding +behavior. ## Current Baseline -`distributor` already supports local, SSH/SFTP, S3, and HTTP upload source -workflows, destination state schemas for single-owner and shared-root state, -path mapping, link generation, reconciliation, transfer policy, explicit -`--force`, and text/JSON run output. +`distributor` currently writes destination `.distributor.json` using +single-owner schema version `2` or shared-root schema version `3`. Destination +behavior is selected through several low-level knobs: `state.mode`, +`reconciliation.mode`, `takeover.mode`, and `transfer`. -Current destination comparison is strict: +The target behavior is a clean alpha break: -- same source manifest skips; -- same source id with older destination state normally replaces; -- same source id with newer destination state normally skips; -- same source id and same creation time with different digest conflicts; -- different source id, pipeline id, destination id, or shared-root output owner - conflicts unless explicit force policy applies; -- unmanaged content and invalid state do not become normal managed replacement - cases. - -The target feature adds destination-level `takeover.mode`, defaulting to -`same_pipeline`, so valid managed content can be replaced without `--force` when -the configured policy says that ownership/source takeover is expected. +- all newly written destination state uses schema version `4`; +- all destinations use `state.mode: catalog` internally; +- user-facing destination behavior is selected by `workflow: additive` or + `workflow: replacement`; +- `workflow` defaults to `additive`; +- legacy destination policy fields are rejected, not aliased; +- legacy state schema versions `1`, `2`, and `3` are not migrated. ## Implementation Principles -- Preserve public behavior until the stage that explicitly changes it. -- Keep source manifests unchanged; takeover is destination configuration and - publish planning policy. -- Keep state comparison pure. State comparison may expose structured conflict - details, but publish planning decides whether takeover is allowed. -- Keep adapters thin. No local, SSH, or S3 adapter should know takeover policy. -- Keep unmanaged content and invalid destination state outside normal takeover. -- Keep `reconciliation.mode`, `transfer`, `state.mode`, `path_mapping.mode`, and - `--force` as separate concepts. -- Prefer narrow behavior-preserving refactors over broad publication rewrites. -- Update implemented-behavior docs in the same stage as the behavior change. +- Keep the source manifest contract unchanged. +- Keep backend adapters unaware of catalog and workflow policy. +- Treat `workflow` as runtime config, not persisted state. +- Keep unmanaged content protected after catalog state exists. +- Preserve output `created_at` when a path changes owner; update only + `updated_at`. +- Prefer removing legacy state/config paths over compatibility shims. +- Preserve current public APIs outside destination state/config unless the + catalog roadmap explicitly changes them. ## Active Implementation Stages -## Stage 1: Config Model And Validation +## Stage 1: Destination Workflow Config Clean Break -Goal: add the destination config field, defaults, and validation without -changing publish behavior. - -Source roadmap reference: - -- Completed takeover feature roadmap: Configuration, Policy Semantics, Safety - Rules. +Goal: replace low-level destination policy config with the user-facing workflow +switch. Implementation scope: -- Add destination-level `TakeoverPolicy` to `internal/config`. -- Add constants for: - - `same_pipeline` - - `same_source` - - `any_managed` - - `never` -- Default `takeover.mode` to `same_pipeline` in config defaults. -- Validate accepted values with clear field context such as - `pipelines[0].destinations[0].takeover.mode`. -- Preserve strict YAML unknown-field behavior. -- Thread the defaulted policy into existing destination config views or helper - structures if those are used by app/publish request construction. -- Do not change publish planning, execution, CLI output, or docs outside - roadmap files in this stage. +- Add destination `workflow` config with accepted values `additive` and + `replacement`. +- Default omitted `workflow` to `additive`. +- Remove or make unsupported the destination-level config fields `state`, + `reconciliation`, `takeover`, and `transfer`. +- Ensure configs containing those legacy fields fail clearly. Prefer strict YAML + unknown-field failure by removing struct fields; add explicit validation only + if clearer errors are needed without weakening strict decoding. +- Preserve backend, publish, transform, path mapping, links, retention, and + other non-policy destination fields. +- Update example configs only when the implementation stage also updates + current docs; otherwise keep this stage focused on config code and tests. Tests: - `go test ./internal/config` -- Add config tests for omitted `takeover`, each accepted mode, invalid mode, and - unknown nested fields. -- Add or update example-loading tests only if examples are touched, which should - not be necessary in this stage. +- Omitted workflow defaults to `additive`. +- `workflow: additive` and `workflow: replacement` validate. +- Unknown workflow values fail. +- Legacy `state`, `reconciliation`, `takeover`, and `transfer` fields fail. +- Existing valid examples are updated or tests are adjusted in the same change if + examples currently use legacy fields. Completion criteria: -- Every destination has a defaulted `takeover.mode`. -- Invalid values fail during config validation. -- No run behavior changes because publish planning does not consume the policy - yet. +- New configs express destination replacement/additive intent with one field. +- No normal config path accepts legacy destination policy knobs. -## Stage 2: Structured State Comparison Details +## Stage 2: Catalog State Schema Version 4 -Goal: expose enough structured comparison detail for publish planning to decide -takeover eligibility without parsing human-readable reason strings. - -Source roadmap reference: - -- Completed takeover feature roadmap: Policy Semantics, Relationship To Existing - Policies. +Goal: implement the schema version `4` catalog state model in `internal/state`. Implementation scope: -- Extend `internal/state.Comparison` or add a package-local structured detail - type so callers can distinguish: - - pipeline id mismatch; - - destination id mismatch; - - different source id; - - same-created digest conflict; - - destination newer; - - invalid state; - - unmanaged content; - - absent shared-root owner; - - shared-root managed output owner conflicts, if those are currently reported - outside `internal/state`. -- Keep existing outcome names and reason strings stable where practical. -- Keep comparison functions pure. They should report facts about existing state, - not consult `takeover.mode`, `transfer`, `reconciliation`, or `force`. -- Do not add new persisted state fields. -- Do not change publish actions in this stage. +- Add catalog state types for top-level schema version `4`, `state.mode: + catalog`, and `outputs`. +- Add compact per-output source identity with `id`, `digest`, and `created`. +- Add catalog output records with required `path`, `pipeline_id`, + `destination_id`, `source`, `kind`, `sha256`, `size`, `created_at`, and + `updated_at`. +- Add optional `source_path`, `transform`, and `url`, present only where allowed + by the catalog roadmap. +- Validate duplicate paths, invalid owner ids, invalid source identities, + invalid output paths, invalid digests, negative sizes, invalid timestamps, + invalid kind/transform combinations, and invalid URLs. +- Marshal catalog state deterministically with the existing JSON formatting + conventions. +- Update `ParseDocument` so schema version `4` returns catalog state. +- Treat schema versions lower than `4` as superseded legacy state for publish + planning, not as readable/migrated active state. Keep enough detection to + identify legacy state and avoid treating it as arbitrary invalid JSON. +- Reject unsupported future schema versions. Tests: -- `go test ./internal/state ./internal/publish` -- Add state tests for structured details on identity mismatch and different - source id. -- Add shared-root tests for owner absence and output ownership conflict detail, - either in `internal/state` or `internal/publish` depending on where the - conflict is currently detected. -- Preserve existing comparison outcome tests. +- `go test ./internal/state` +- Parse/validate/marshal valid catalog state. +- Reject malformed catalog state and invalid output records. +- Detect schema versions `1`, `2`, and `3` as superseded legacy state. +- Reject future schema versions. +- Prove no top-level `owners`, `sources`, workflow, source manifest, pipeline id, + destination id, or published timestamp is accepted for catalog state. Completion criteria: -- Publish planning can make takeover decisions from structured data. -- Existing behavior remains unchanged because no takeover mapping is applied yet. +- State package has one canonical schema version `4` catalog contract for new + writes. +- Legacy state is detected but not migrated. -## Stage 3: Single-Owner Takeover Planning And Execution +## Stage 3: Catalog Publish Planning -Goal: implement `takeover.mode` for single-owner destination state. - -Source roadmap reference: - -- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination - State Results, Safety Rules. +Goal: plan publish actions against catalog state and destination workflow. Implementation scope: -- Add `TakeoverPolicy` or equivalent values to `internal/publish.Request`. -- Pass destination takeover policy from app-level run planning into publish. -- Add publish action `replace_takeover`. -- Map eligible single-owner conflicts to `replace_takeover`: - - `same_pipeline`: existing state pipeline id equals current pipeline id, - regardless of destination id or source id; - - `same_source`: existing state source id equals current source id; - - `any_managed`: existing state is valid distributor-managed state; - - `never`: no identity/source takeover. -- Keep these cases failing by default unless existing force behavior applies: - - unmanaged content; - - invalid state; - - same-created digest conflict; - - destination newer for the same source id unless `transfer` plus `--force` - already permits replacement; - - conflicts not allowed by `takeover.mode`. -- Execute `replace_takeover` through bounded managed replacement mechanics. -- For cross-source takeover, do not retain omitted outputs through - `reconciliation.mode: merge`; treat the affected single-owner state as a - managed replacement so old-source outputs are not attributed to the new source. -- Preserve existing `replace_older`, `skip_same`, `skip_destination_newer`, - `fail_conflict`, `fail_unmanaged`, and `force_replace` behavior. -- Keep adapters unchanged. +- Replace publish request inputs that consume state/reconciliation/takeover and + transfer policy with destination workflow. +- Map workflow to catalog actions: + - `workflow: additive`: upsert planned outputs and retain all other managed + catalog outputs. + - `workflow: replacement`: upsert planned outputs and delete catalog outputs + owned by the current pipeline/destination that are omitted from the plan. +- Planned paths that already exist as catalog-managed outputs may be overwritten + and become owned by the current pipeline/destination. +- Planned path collisions with storage content not recorded in catalog state + fail as unmanaged content. +- Matching planned outputs may skip writes when source identity and output digest + already match, if that optimization can be implemented without changing + externally visible results; otherwise writing idempotently is acceptable. +- Existing schema `< 4` state is superseded: + - replacement workflow may plan a bounded destination-root clear before + writing planned outputs; + - additive workflow may plan overwrites for planned paths only and leave + unplanned files unmanaged. +- Invalid JSON or future schema state remains a conflict, not a superseded + legacy state. +- Preserve path mapping, publish policy, transforms, links, fixed-path + selection, and output collision checks. -Current-behavior documentation updates: +Tests: -- Do not update user docs yet unless CLI output changes in this stage. Prefer - deferring user docs to Stage 5 so the behavior, output, and docs land - together. -- If this stage changes visible dry-run or run output enough that tests require - new wording, document only the implemented single-owner behavior and clearly - leave shared-root takeover out until Stage 4. +- `go test ./internal/publish` +- Additive workflow publishes new catalog state. +- Additive workflow overwrites an existing managed output and retains unrelated + outputs. +- Replacement workflow removes omitted outputs owned by the current + pipeline/destination. +- Replacement workflow preserves unrelated catalog outputs owned by other + pipeline/destination pairs. +- Managed output ownership moves to the current pipeline/destination when a + planned path is overwritten. +- `created_at` is preserved when an existing path changes owner; `updated_at` + changes. +- Unmanaged path collisions fail. +- Legacy schema `< 4` state follows the superseded-state rules above. +- Invalid/future state fails. + +Completion criteria: + +- Publish planning no longer depends on single-owner/shared-root comparison + semantics. +- Workflow behavior is fully driven by `workflow`. + +## Stage 4: Catalog Publish Execution + +Goal: execute catalog publish plans and write schema version `4` state. + +Implementation scope: + +- Write only schema version `4` catalog `.distributor.json`. +- Implement additive execution as managed upsert of planned outputs plus catalog + state update. +- Implement replacement execution as managed upsert plus deletion of omitted + current-owner catalog outputs. +- For replacement over superseded legacy state, clear the bounded destination + root before writing planned outputs and catalog state. +- For additive over superseded legacy state, overwrite planned paths and write a + catalog containing only planned outputs; leave unplanned files unmanaged. +- Preserve cleanup behavior on failed writes: + - additive cleanup removes outputs newly created by the failed attempt where + practical; + - replacement cleanup follows existing managed-replacement safety where + practical; + - state is written only after selected outputs are written. +- Preserve link metadata, generated output digests, source output digests, + content type behavior, and backend-safe writes. Tests: - `go test ./internal/publish ./internal/app` -- Default `same_pipeline` replaces a different source id from the same pipeline. -- Default `same_pipeline` replaces state with a different destination id under - the same pipeline. -- Default `same_pipeline` refuses a different pipeline. -- `same_source` allows same source id and refuses different source id. -- `any_managed` replaces valid state from a different pipeline. -- `never` refuses identity/source takeover. -- Invalid state and unmanaged content still fail without force. -- Cross-source takeover with `reconciliation.mode: merge` does not retain - omitted outputs from the previous source. -- Dry-run plans `replace_takeover` without writing. -- Existing force tests still pass. +- Additive execution writes planned outputs, overwrites managed planned paths, + retains unrelated managed outputs, and writes catalog state. +- Replacement execution deletes omitted current-owner outputs and preserves + unrelated owner outputs. +- Superseded legacy replacement clears bounded destination root only. +- Superseded legacy additive leaves unplanned files on disk but out of catalog + state. +- Failed writes do not leave misleading catalog state. +- Local, fake-backed SSH, and fake-backed S3 app paths exercise the same publish + behavior. Completion criteria: -- Single-owner latest-style destinations can be updated by different bundle ids - from the same configured pipeline without `--force`. -- No unmanaged or invalid-state path becomes a normal takeover path. +- Successful `run` writes only v4 catalog state. +- Destination outputs match additive/replacement workflow semantics. -## Stage 4: Shared-Root Takeover Planning And Execution +## Stage 5: Run Reporting, Notifications, And CLI Surface -Goal: apply the same takeover vocabulary to shared-root owner and output-path -conflicts. - -Source roadmap reference: - -- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination - State Results, Safety Rules. +Goal: update user-visible run behavior to describe workflow/catalog actions +instead of legacy replacement/takeover actions. Implementation scope: -- Extend shared-root planning so planned output collisions with existing managed - owners are eligible for takeover according to `takeover.mode`. -- Apply policy as follows: - - `same_pipeline`: current owner may take over output paths owned by another - destination under the same pipeline; - - `same_source`: current owner may take over output paths whose owner records - the same source id as the current source; - - `any_managed`: current owner may take over output paths owned by any valid - shared-root owner; - - `never`: preserve current owner conflict behavior. -- Preserve unrelated owner records and non-conflicting output records. -- Keep unmanaged path collisions failing without force. -- Keep compatible single-owner-to-shared-root migration behavior intact. -- For cross-source takeover, do not retain omitted outputs from the previous - source under the taking-over owner when `reconciliation.mode: merge` is set. -- Keep shared-root forced replacement behavior explicit and bounded as it is - today. - -Current-behavior documentation updates: - -- Defer broad documentation updates to Stage 5 unless shared-root behavior must - be documented immediately to keep tests or generated docs consistent. +- Replace legacy action labels tied to state/reconciliation/takeover/transfer + with these stable workflow-oriented labels: + - `publish_new` + - `upsert_additive` + - `replace_catalog` + - `skip_same` + - `force_replace` + - `fail_unmanaged` + - `fail_conflict` +- Ensure text and JSON run summaries include workflow-relevant counters. +- Include `workflow` in run action records where useful. +- Update fixed-path dry-run warnings to describe additive upsert or replacement + clearly. +- Ensure notifications use the new action labels. +- Remove reporting assumptions that depend on `replace_older`, + `replace_newer`, `replace_conflict`, or `replace_takeover`. Tests: -- `go test ./internal/state ./internal/publish ./internal/app` -- `same_pipeline` permits taking over a path owned by another destination in the - same pipeline. -- `same_pipeline` refuses a path owned by another pipeline. -- `same_source` permits only matching source-id ownership transfer. -- `any_managed` permits cross-pipeline managed ownership transfer. -- `never` preserves current shared-root conflict behavior. -- Reconciliation `replace` and `merge` handle omitted outputs according to the - feature roadmap, including the cross-source no-retain rule. -- Shared-root migration from compatible single-owner state still works. -- Unmanaged collisions still fail without force. +- `go test ./internal/app ./internal/cli` +- Text dry-run output distinguishes additive from replacement workflow. +- JSON output includes workflow and stable action labels. +- Summary counters are deterministic. +- Notifications fire for additive and replacement writes. +- Existing CLI commands still parse and execute with the new config shape. Completion criteria: -- Single-owner and shared-root destinations use one consistent takeover policy. -- Shared-root takeover changes only affected owners/outputs and preserves - unrelated managed state. +- Operators can understand from dry-run output whether a destination will upsert + or replace managed catalog outputs. -## Stage 5: Run Output, Summaries, And Documentation +## Stage 6: Catalog Prune And Reconcile-State -Goal: make takeover behavior visible to operators and document the implemented -feature. - -Source roadmap reference: - -- Completed takeover feature roadmap: Documentation Impact, Publish Planning, - Relationship To Existing Policies. +Goal: update maintenance commands to operate on catalog state with the agreed +selector model. Implementation scope: -- Add `replace_takeover` to text run output. -- Add `replace_takeover` to JSON run action output. -- Add a distinct summary counter for takeover replacements in text and JSON run - summaries, using the field name `replace_takeover` for JSON. -- Include takeover mode and conflict reason in action projection where useful - and consistent with existing output style. -- Update fixed-path dry-run warnings when takeover will replace the destination - root. +- Prune: + - selected `--pipeline` and `--destination` prune only outputs currently owned + by that pipeline/destination; + - no `--all-owners` prune mode in the initial catalog implementation; + - do not add `--source-id`, `--path-prefix`, or `--kind` selectors. +- Reconcile-state: + - selected `--pipeline` and `--destination` repair only outputs currently + owned by that pipeline/destination; + - `--all-owners` repairs all catalog outputs; + - unmanaged reporting compares storage entries to all catalog output paths, + not just selected owner paths. +- Rewrite repaired/pruned state as schema version `4`. +- Remove legacy single-owner/shared-root maintenance branches. + +Tests: + +- `go test ./internal/state ./internal/app ./internal/cli` +- Prune selects only current-owner catalog outputs. +- Prune deletes selected outputs and removes their catalog records. +- Reconcile-state selected owner removes only missing outputs for that owner. +- `reconcile-state --all-owners` removes missing outputs for all owners. +- Unmanaged reporting excludes all catalog-managed paths and reports unrecorded + storage entries. +- JSON/text output remains stable and clear. + +Completion criteria: + +- Maintenance commands operate only on catalog state and respect the locked + selector rules. + +## Stage 7: Clean Break Removal And Documentation + +Goal: remove legacy state/config behavior and document the implemented catalog +workflow model. + +Implementation scope: + +- Remove dead code for writing schema version `2` single-owner and schema + version `3` shared-root state. +- Remove legacy config structs/constants/defaults/validation for destination + `state`, `reconciliation`, `takeover`, and `transfer` where no longer used. +- Remove or rewrite tests that only assert legacy state/config behavior. - Update current-behavior docs: - `docs/config.md` + - `docs/cli.md` - `docs/operations.md` - `docs/troubleshooting.md` - `docs/integrations/destination-state.md` - - `docs/internal/publish.md` - - `docs/internal/state.md` - - `docs/internal/config.md` + - relevant `docs/internal/` docs - `docs/policy/architecture.md` - `docs/policy/development.md` -- Keep docs concise and link to canonical references rather than duplicating - full state semantics in every file. -- Do not document any unimplemented future takeover extensions outside - `docs/roadmap/`. +- Update examples to use `workflow` and remove legacy fields. +- Keep roadmap-only material out of current docs. +- Once implemented and documented, remove or rewrite + `docs/roadmap/catalog.md` so completed behavior is not described only as + future work. -Tests: +Tests and checks: -- `go test ./internal/app ./internal/cli ./internal/config` -- Text dry-run output includes `replace_takeover`. -- JSON output records `replace_takeover` with stable field names. -- Summary counters are deterministic. -- Fixed-path warnings remain deterministic. -- Config docs examples, if changed or added, load through existing config tests. -- Troubleshooting text matches actual error/action wording. - -Completion criteria: - -- Operators can distinguish ordinary same-source replacement, takeover - replacement, and explicit forced replacement. -- Current docs describe implemented takeover behavior outside roadmap files. - -## Stage 6: Cross-Backend Regression And Roadmap Closeout - -Goal: prove takeover behavior is backend-agnostic and clean up roadmap state -after implementation. - -Source roadmap reference: - -- Completed takeover feature roadmap: Goals, Non-Goals, Safety Rules. - -Implementation scope: - -- Add app-level coverage showing takeover planning/execution works with storage - abstraction rather than adapter-specific logic. -- Cover local destinations directly. -- Cover SSH and S3 through fake-backed app tests where possible; add adapter - integration tests only if existing test infrastructure already supports - opt-in remote credentials. -- Add tests for `path_mapping.mode: fixed` with different source ids under the - same pipeline. -- Add tests for archive-style `preserve_relative` destinations using - `takeover.mode: same_source` where strict source identity is desired. -- Run final consistency searches. -- Remove the completed detailed takeover roadmap when no future takeover work - remains. - -Tests: - -- `go test ./internal/config` -- `go test ./internal/state` -- `go test ./internal/publish` -- `go test ./internal/app ./internal/cli` - `go test ./...` - -Documentation and consistency checks: - -```sh -rg -n "takeover|replace_takeover|same_pipeline|same_source|any_managed" docs README.md examples -rg -n "destination source id differs from source|pipeline id .* does not match|destination id .* does not match" docs -rg -n "Managed Destination Takeover Roadmap|future takeover|planned takeover" docs README.md examples --glob '!docs/roadmap/**' -``` +- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'` +- `rg -n 'single_owner|shared_root|schema version `2`|schema version `3`' docs internal` +- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples` Completion criteria: +- Current docs and examples describe the catalog workflow model. +- Legacy destination policy fields and legacy write paths are gone. - Full test suite passes. -- The default `same_pipeline` behavior applies only to valid - distributor-managed takeover cases. -- Unmanaged content and invalid state remain protected. -- Current docs no longer describe implemented takeover behavior as future work. ## Refactors To Avoid -- Do not build a generic ownership engine outside `internal/state` and - `internal/publish`. -- Do not move destination comparison policy into storage adapters. -- Do not change the source manifest schema. -- Do not add takeover fields to HTTP upload requests or producer APIs. -- Do not make `--force` persistent config. -- Do not merge `takeover`, `transfer`, and `reconciliation` into one broad - policy object. -- Do not silently adopt unmanaged content. +- Do not add top-level `owners` or `sources` catalogs. +- Do not persist workflow in `.distributor.json`. +- Do not keep deprecated config aliases for legacy destination policy fields. +- Do not implement state migration from schema versions `1`, `2`, or `3`. +- Do not add catalog-specific prune/reconcile selectors beyond the agreed + initial ownership scope. +- Do not move catalog policy into storage adapters. ## Open Questions -No open questions are known. The feature roadmap selects the default mode -(`same_pipeline`), the accepted modes, the safety boundary, and the relationship -to existing state, reconciliation, transfer, and force policies. +No open questions are known. The catalog roadmap locks the state shape, workflow +values, default workflow, clean-break policy, legacy-state behavior, output +field selection, `created_at` preservation, and maintenance command scope.