Added a roadmap and implementation plan for a significant refactor of pipeline destination policy and catalog state
This commit is contained in:
@@ -1,375 +1,345 @@
|
||||
# Managed Destination Takeover Implementation Roadmap
|
||||
# Catalog State And Workflow Implementation Roadmap
|
||||
|
||||
This is the completed staged implementation plan for the takeover feature. The
|
||||
detailed feature roadmap was removed after implementation; current behavior is
|
||||
documented outside `docs/roadmap/`. This document records the implementation
|
||||
sequence used by LLM coding agents stage by stage.
|
||||
This is the active staged implementation plan for
|
||||
`docs/roadmap/catalog.md`. The feature roadmap defines the target state model
|
||||
and policy decisions; this document defines the implementation sequence for an
|
||||
LLM coding agent to follow stage by stage.
|
||||
|
||||
Future behavior must remain under `docs/roadmap/` until implemented.
|
||||
Preparatory internal stages should not update user-facing current docs.
|
||||
Current-behavior docs should be updated when behavior is wired for
|
||||
operator-facing use.
|
||||
Future behavior must remain under `docs/roadmap/` until implemented. Update
|
||||
current-behavior docs only in the stage that implements the corresponding
|
||||
behavior.
|
||||
|
||||
## Current Baseline
|
||||
|
||||
`distributor` already supports local, SSH/SFTP, S3, and HTTP upload source
|
||||
workflows, destination state schemas for single-owner and shared-root state,
|
||||
path mapping, link generation, reconciliation, transfer policy, explicit
|
||||
`--force`, and text/JSON run output.
|
||||
`distributor` currently writes destination `.distributor.json` using
|
||||
single-owner schema version `2` or shared-root schema version `3`. Destination
|
||||
behavior is selected through several low-level knobs: `state.mode`,
|
||||
`reconciliation.mode`, `takeover.mode`, and `transfer`.
|
||||
|
||||
Current destination comparison is strict:
|
||||
The target behavior is a clean alpha break:
|
||||
|
||||
- same source manifest skips;
|
||||
- same source id with older destination state normally replaces;
|
||||
- same source id with newer destination state normally skips;
|
||||
- same source id and same creation time with different digest conflicts;
|
||||
- different source id, pipeline id, destination id, or shared-root output owner
|
||||
conflicts unless explicit force policy applies;
|
||||
- unmanaged content and invalid state do not become normal managed replacement
|
||||
cases.
|
||||
|
||||
The target feature adds destination-level `takeover.mode`, defaulting to
|
||||
`same_pipeline`, so valid managed content can be replaced without `--force` when
|
||||
the configured policy says that ownership/source takeover is expected.
|
||||
- all newly written destination state uses schema version `4`;
|
||||
- all destinations use `state.mode: catalog` internally;
|
||||
- user-facing destination behavior is selected by `workflow: additive` or
|
||||
`workflow: replacement`;
|
||||
- `workflow` defaults to `additive`;
|
||||
- legacy destination policy fields are rejected, not aliased;
|
||||
- legacy state schema versions `1`, `2`, and `3` are not migrated.
|
||||
|
||||
## Implementation Principles
|
||||
|
||||
- Preserve public behavior until the stage that explicitly changes it.
|
||||
- Keep source manifests unchanged; takeover is destination configuration and
|
||||
publish planning policy.
|
||||
- Keep state comparison pure. State comparison may expose structured conflict
|
||||
details, but publish planning decides whether takeover is allowed.
|
||||
- Keep adapters thin. No local, SSH, or S3 adapter should know takeover policy.
|
||||
- Keep unmanaged content and invalid destination state outside normal takeover.
|
||||
- Keep `reconciliation.mode`, `transfer`, `state.mode`, `path_mapping.mode`, and
|
||||
`--force` as separate concepts.
|
||||
- Prefer narrow behavior-preserving refactors over broad publication rewrites.
|
||||
- Update implemented-behavior docs in the same stage as the behavior change.
|
||||
- Keep the source manifest contract unchanged.
|
||||
- Keep backend adapters unaware of catalog and workflow policy.
|
||||
- Treat `workflow` as runtime config, not persisted state.
|
||||
- Keep unmanaged content protected after catalog state exists.
|
||||
- Preserve output `created_at` when a path changes owner; update only
|
||||
`updated_at`.
|
||||
- Prefer removing legacy state/config paths over compatibility shims.
|
||||
- Preserve current public APIs outside destination state/config unless the
|
||||
catalog roadmap explicitly changes them.
|
||||
|
||||
## Active Implementation Stages
|
||||
|
||||
## Stage 1: Config Model And Validation
|
||||
## Stage 1: Destination Workflow Config Clean Break
|
||||
|
||||
Goal: add the destination config field, defaults, and validation without
|
||||
changing publish behavior.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Configuration, Policy Semantics, Safety
|
||||
Rules.
|
||||
Goal: replace low-level destination policy config with the user-facing workflow
|
||||
switch.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add destination-level `TakeoverPolicy` to `internal/config`.
|
||||
- Add constants for:
|
||||
- `same_pipeline`
|
||||
- `same_source`
|
||||
- `any_managed`
|
||||
- `never`
|
||||
- Default `takeover.mode` to `same_pipeline` in config defaults.
|
||||
- Validate accepted values with clear field context such as
|
||||
`pipelines[0].destinations[0].takeover.mode`.
|
||||
- Preserve strict YAML unknown-field behavior.
|
||||
- Thread the defaulted policy into existing destination config views or helper
|
||||
structures if those are used by app/publish request construction.
|
||||
- Do not change publish planning, execution, CLI output, or docs outside
|
||||
roadmap files in this stage.
|
||||
- Add destination `workflow` config with accepted values `additive` and
|
||||
`replacement`.
|
||||
- Default omitted `workflow` to `additive`.
|
||||
- Remove or make unsupported the destination-level config fields `state`,
|
||||
`reconciliation`, `takeover`, and `transfer`.
|
||||
- Ensure configs containing those legacy fields fail clearly. Prefer strict YAML
|
||||
unknown-field failure by removing struct fields; add explicit validation only
|
||||
if clearer errors are needed without weakening strict decoding.
|
||||
- Preserve backend, publish, transform, path mapping, links, retention, and
|
||||
other non-policy destination fields.
|
||||
- Update example configs only when the implementation stage also updates
|
||||
current docs; otherwise keep this stage focused on config code and tests.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/config`
|
||||
- Add config tests for omitted `takeover`, each accepted mode, invalid mode, and
|
||||
unknown nested fields.
|
||||
- Add or update example-loading tests only if examples are touched, which should
|
||||
not be necessary in this stage.
|
||||
- Omitted workflow defaults to `additive`.
|
||||
- `workflow: additive` and `workflow: replacement` validate.
|
||||
- Unknown workflow values fail.
|
||||
- Legacy `state`, `reconciliation`, `takeover`, and `transfer` fields fail.
|
||||
- Existing valid examples are updated or tests are adjusted in the same change if
|
||||
examples currently use legacy fields.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Every destination has a defaulted `takeover.mode`.
|
||||
- Invalid values fail during config validation.
|
||||
- No run behavior changes because publish planning does not consume the policy
|
||||
yet.
|
||||
- New configs express destination replacement/additive intent with one field.
|
||||
- No normal config path accepts legacy destination policy knobs.
|
||||
|
||||
## Stage 2: Structured State Comparison Details
|
||||
## Stage 2: Catalog State Schema Version 4
|
||||
|
||||
Goal: expose enough structured comparison detail for publish planning to decide
|
||||
takeover eligibility without parsing human-readable reason strings.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Policy Semantics, Relationship To Existing
|
||||
Policies.
|
||||
Goal: implement the schema version `4` catalog state model in `internal/state`.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Extend `internal/state.Comparison` or add a package-local structured detail
|
||||
type so callers can distinguish:
|
||||
- pipeline id mismatch;
|
||||
- destination id mismatch;
|
||||
- different source id;
|
||||
- same-created digest conflict;
|
||||
- destination newer;
|
||||
- invalid state;
|
||||
- unmanaged content;
|
||||
- absent shared-root owner;
|
||||
- shared-root managed output owner conflicts, if those are currently reported
|
||||
outside `internal/state`.
|
||||
- Keep existing outcome names and reason strings stable where practical.
|
||||
- Keep comparison functions pure. They should report facts about existing state,
|
||||
not consult `takeover.mode`, `transfer`, `reconciliation`, or `force`.
|
||||
- Do not add new persisted state fields.
|
||||
- Do not change publish actions in this stage.
|
||||
- Add catalog state types for top-level schema version `4`, `state.mode:
|
||||
catalog`, and `outputs`.
|
||||
- Add compact per-output source identity with `id`, `digest`, and `created`.
|
||||
- Add catalog output records with required `path`, `pipeline_id`,
|
||||
`destination_id`, `source`, `kind`, `sha256`, `size`, `created_at`, and
|
||||
`updated_at`.
|
||||
- Add optional `source_path`, `transform`, and `url`, present only where allowed
|
||||
by the catalog roadmap.
|
||||
- Validate duplicate paths, invalid owner ids, invalid source identities,
|
||||
invalid output paths, invalid digests, negative sizes, invalid timestamps,
|
||||
invalid kind/transform combinations, and invalid URLs.
|
||||
- Marshal catalog state deterministically with the existing JSON formatting
|
||||
conventions.
|
||||
- Update `ParseDocument` so schema version `4` returns catalog state.
|
||||
- Treat schema versions lower than `4` as superseded legacy state for publish
|
||||
planning, not as readable/migrated active state. Keep enough detection to
|
||||
identify legacy state and avoid treating it as arbitrary invalid JSON.
|
||||
- Reject unsupported future schema versions.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/publish`
|
||||
- Add state tests for structured details on identity mismatch and different
|
||||
source id.
|
||||
- Add shared-root tests for owner absence and output ownership conflict detail,
|
||||
either in `internal/state` or `internal/publish` depending on where the
|
||||
conflict is currently detected.
|
||||
- Preserve existing comparison outcome tests.
|
||||
- `go test ./internal/state`
|
||||
- Parse/validate/marshal valid catalog state.
|
||||
- Reject malformed catalog state and invalid output records.
|
||||
- Detect schema versions `1`, `2`, and `3` as superseded legacy state.
|
||||
- Reject future schema versions.
|
||||
- Prove no top-level `owners`, `sources`, workflow, source manifest, pipeline id,
|
||||
destination id, or published timestamp is accepted for catalog state.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Publish planning can make takeover decisions from structured data.
|
||||
- Existing behavior remains unchanged because no takeover mapping is applied yet.
|
||||
- State package has one canonical schema version `4` catalog contract for new
|
||||
writes.
|
||||
- Legacy state is detected but not migrated.
|
||||
|
||||
## Stage 3: Single-Owner Takeover Planning And Execution
|
||||
## Stage 3: Catalog Publish Planning
|
||||
|
||||
Goal: implement `takeover.mode` for single-owner destination state.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination
|
||||
State Results, Safety Rules.
|
||||
Goal: plan publish actions against catalog state and destination workflow.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add `TakeoverPolicy` or equivalent values to `internal/publish.Request`.
|
||||
- Pass destination takeover policy from app-level run planning into publish.
|
||||
- Add publish action `replace_takeover`.
|
||||
- Map eligible single-owner conflicts to `replace_takeover`:
|
||||
- `same_pipeline`: existing state pipeline id equals current pipeline id,
|
||||
regardless of destination id or source id;
|
||||
- `same_source`: existing state source id equals current source id;
|
||||
- `any_managed`: existing state is valid distributor-managed state;
|
||||
- `never`: no identity/source takeover.
|
||||
- Keep these cases failing by default unless existing force behavior applies:
|
||||
- unmanaged content;
|
||||
- invalid state;
|
||||
- same-created digest conflict;
|
||||
- destination newer for the same source id unless `transfer` plus `--force`
|
||||
already permits replacement;
|
||||
- conflicts not allowed by `takeover.mode`.
|
||||
- Execute `replace_takeover` through bounded managed replacement mechanics.
|
||||
- For cross-source takeover, do not retain omitted outputs through
|
||||
`reconciliation.mode: merge`; treat the affected single-owner state as a
|
||||
managed replacement so old-source outputs are not attributed to the new source.
|
||||
- Preserve existing `replace_older`, `skip_same`, `skip_destination_newer`,
|
||||
`fail_conflict`, `fail_unmanaged`, and `force_replace` behavior.
|
||||
- Keep adapters unchanged.
|
||||
- Replace publish request inputs that consume state/reconciliation/takeover and
|
||||
transfer policy with destination workflow.
|
||||
- Map workflow to catalog actions:
|
||||
- `workflow: additive`: upsert planned outputs and retain all other managed
|
||||
catalog outputs.
|
||||
- `workflow: replacement`: upsert planned outputs and delete catalog outputs
|
||||
owned by the current pipeline/destination that are omitted from the plan.
|
||||
- Planned paths that already exist as catalog-managed outputs may be overwritten
|
||||
and become owned by the current pipeline/destination.
|
||||
- Planned path collisions with storage content not recorded in catalog state
|
||||
fail as unmanaged content.
|
||||
- Matching planned outputs may skip writes when source identity and output digest
|
||||
already match, if that optimization can be implemented without changing
|
||||
externally visible results; otherwise writing idempotently is acceptable.
|
||||
- Existing schema `< 4` state is superseded:
|
||||
- replacement workflow may plan a bounded destination-root clear before
|
||||
writing planned outputs;
|
||||
- additive workflow may plan overwrites for planned paths only and leave
|
||||
unplanned files unmanaged.
|
||||
- Invalid JSON or future schema state remains a conflict, not a superseded
|
||||
legacy state.
|
||||
- Preserve path mapping, publish policy, transforms, links, fixed-path
|
||||
selection, and output collision checks.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
Tests:
|
||||
|
||||
- Do not update user docs yet unless CLI output changes in this stage. Prefer
|
||||
deferring user docs to Stage 5 so the behavior, output, and docs land
|
||||
together.
|
||||
- If this stage changes visible dry-run or run output enough that tests require
|
||||
new wording, document only the implemented single-owner behavior and clearly
|
||||
leave shared-root takeover out until Stage 4.
|
||||
- `go test ./internal/publish`
|
||||
- Additive workflow publishes new catalog state.
|
||||
- Additive workflow overwrites an existing managed output and retains unrelated
|
||||
outputs.
|
||||
- Replacement workflow removes omitted outputs owned by the current
|
||||
pipeline/destination.
|
||||
- Replacement workflow preserves unrelated catalog outputs owned by other
|
||||
pipeline/destination pairs.
|
||||
- Managed output ownership moves to the current pipeline/destination when a
|
||||
planned path is overwritten.
|
||||
- `created_at` is preserved when an existing path changes owner; `updated_at`
|
||||
changes.
|
||||
- Unmanaged path collisions fail.
|
||||
- Legacy schema `< 4` state follows the superseded-state rules above.
|
||||
- Invalid/future state fails.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Publish planning no longer depends on single-owner/shared-root comparison
|
||||
semantics.
|
||||
- Workflow behavior is fully driven by `workflow`.
|
||||
|
||||
## Stage 4: Catalog Publish Execution
|
||||
|
||||
Goal: execute catalog publish plans and write schema version `4` state.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Write only schema version `4` catalog `.distributor.json`.
|
||||
- Implement additive execution as managed upsert of planned outputs plus catalog
|
||||
state update.
|
||||
- Implement replacement execution as managed upsert plus deletion of omitted
|
||||
current-owner catalog outputs.
|
||||
- For replacement over superseded legacy state, clear the bounded destination
|
||||
root before writing planned outputs and catalog state.
|
||||
- For additive over superseded legacy state, overwrite planned paths and write a
|
||||
catalog containing only planned outputs; leave unplanned files unmanaged.
|
||||
- Preserve cleanup behavior on failed writes:
|
||||
- additive cleanup removes outputs newly created by the failed attempt where
|
||||
practical;
|
||||
- replacement cleanup follows existing managed-replacement safety where
|
||||
practical;
|
||||
- state is written only after selected outputs are written.
|
||||
- Preserve link metadata, generated output digests, source output digests,
|
||||
content type behavior, and backend-safe writes.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/publish ./internal/app`
|
||||
- Default `same_pipeline` replaces a different source id from the same pipeline.
|
||||
- Default `same_pipeline` replaces state with a different destination id under
|
||||
the same pipeline.
|
||||
- Default `same_pipeline` refuses a different pipeline.
|
||||
- `same_source` allows same source id and refuses different source id.
|
||||
- `any_managed` replaces valid state from a different pipeline.
|
||||
- `never` refuses identity/source takeover.
|
||||
- Invalid state and unmanaged content still fail without force.
|
||||
- Cross-source takeover with `reconciliation.mode: merge` does not retain
|
||||
omitted outputs from the previous source.
|
||||
- Dry-run plans `replace_takeover` without writing.
|
||||
- Existing force tests still pass.
|
||||
- Additive execution writes planned outputs, overwrites managed planned paths,
|
||||
retains unrelated managed outputs, and writes catalog state.
|
||||
- Replacement execution deletes omitted current-owner outputs and preserves
|
||||
unrelated owner outputs.
|
||||
- Superseded legacy replacement clears bounded destination root only.
|
||||
- Superseded legacy additive leaves unplanned files on disk but out of catalog
|
||||
state.
|
||||
- Failed writes do not leave misleading catalog state.
|
||||
- Local, fake-backed SSH, and fake-backed S3 app paths exercise the same publish
|
||||
behavior.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Single-owner latest-style destinations can be updated by different bundle ids
|
||||
from the same configured pipeline without `--force`.
|
||||
- No unmanaged or invalid-state path becomes a normal takeover path.
|
||||
- Successful `run` writes only v4 catalog state.
|
||||
- Destination outputs match additive/replacement workflow semantics.
|
||||
|
||||
## Stage 4: Shared-Root Takeover Planning And Execution
|
||||
## Stage 5: Run Reporting, Notifications, And CLI Surface
|
||||
|
||||
Goal: apply the same takeover vocabulary to shared-root owner and output-path
|
||||
conflicts.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Policy Semantics, Publish Planning, Destination
|
||||
State Results, Safety Rules.
|
||||
Goal: update user-visible run behavior to describe workflow/catalog actions
|
||||
instead of legacy replacement/takeover actions.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Extend shared-root planning so planned output collisions with existing managed
|
||||
owners are eligible for takeover according to `takeover.mode`.
|
||||
- Apply policy as follows:
|
||||
- `same_pipeline`: current owner may take over output paths owned by another
|
||||
destination under the same pipeline;
|
||||
- `same_source`: current owner may take over output paths whose owner records
|
||||
the same source id as the current source;
|
||||
- `any_managed`: current owner may take over output paths owned by any valid
|
||||
shared-root owner;
|
||||
- `never`: preserve current owner conflict behavior.
|
||||
- Preserve unrelated owner records and non-conflicting output records.
|
||||
- Keep unmanaged path collisions failing without force.
|
||||
- Keep compatible single-owner-to-shared-root migration behavior intact.
|
||||
- For cross-source takeover, do not retain omitted outputs from the previous
|
||||
source under the taking-over owner when `reconciliation.mode: merge` is set.
|
||||
- Keep shared-root forced replacement behavior explicit and bounded as it is
|
||||
today.
|
||||
|
||||
Current-behavior documentation updates:
|
||||
|
||||
- Defer broad documentation updates to Stage 5 unless shared-root behavior must
|
||||
be documented immediately to keep tests or generated docs consistent.
|
||||
- Replace legacy action labels tied to state/reconciliation/takeover/transfer
|
||||
with these stable workflow-oriented labels:
|
||||
- `publish_new`
|
||||
- `upsert_additive`
|
||||
- `replace_catalog`
|
||||
- `skip_same`
|
||||
- `force_replace`
|
||||
- `fail_unmanaged`
|
||||
- `fail_conflict`
|
||||
- Ensure text and JSON run summaries include workflow-relevant counters.
|
||||
- Include `workflow` in run action records where useful.
|
||||
- Update fixed-path dry-run warnings to describe additive upsert or replacement
|
||||
clearly.
|
||||
- Ensure notifications use the new action labels.
|
||||
- Remove reporting assumptions that depend on `replace_older`,
|
||||
`replace_newer`, `replace_conflict`, or `replace_takeover`.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/publish ./internal/app`
|
||||
- `same_pipeline` permits taking over a path owned by another destination in the
|
||||
same pipeline.
|
||||
- `same_pipeline` refuses a path owned by another pipeline.
|
||||
- `same_source` permits only matching source-id ownership transfer.
|
||||
- `any_managed` permits cross-pipeline managed ownership transfer.
|
||||
- `never` preserves current shared-root conflict behavior.
|
||||
- Reconciliation `replace` and `merge` handle omitted outputs according to the
|
||||
feature roadmap, including the cross-source no-retain rule.
|
||||
- Shared-root migration from compatible single-owner state still works.
|
||||
- Unmanaged collisions still fail without force.
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- Text dry-run output distinguishes additive from replacement workflow.
|
||||
- JSON output includes workflow and stable action labels.
|
||||
- Summary counters are deterministic.
|
||||
- Notifications fire for additive and replacement writes.
|
||||
- Existing CLI commands still parse and execute with the new config shape.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Single-owner and shared-root destinations use one consistent takeover policy.
|
||||
- Shared-root takeover changes only affected owners/outputs and preserves
|
||||
unrelated managed state.
|
||||
- Operators can understand from dry-run output whether a destination will upsert
|
||||
or replace managed catalog outputs.
|
||||
|
||||
## Stage 5: Run Output, Summaries, And Documentation
|
||||
## Stage 6: Catalog Prune And Reconcile-State
|
||||
|
||||
Goal: make takeover behavior visible to operators and document the implemented
|
||||
feature.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Documentation Impact, Publish Planning,
|
||||
Relationship To Existing Policies.
|
||||
Goal: update maintenance commands to operate on catalog state with the agreed
|
||||
selector model.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add `replace_takeover` to text run output.
|
||||
- Add `replace_takeover` to JSON run action output.
|
||||
- Add a distinct summary counter for takeover replacements in text and JSON run
|
||||
summaries, using the field name `replace_takeover` for JSON.
|
||||
- Include takeover mode and conflict reason in action projection where useful
|
||||
and consistent with existing output style.
|
||||
- Update fixed-path dry-run warnings when takeover will replace the destination
|
||||
root.
|
||||
- Prune:
|
||||
- selected `--pipeline` and `--destination` prune only outputs currently owned
|
||||
by that pipeline/destination;
|
||||
- no `--all-owners` prune mode in the initial catalog implementation;
|
||||
- do not add `--source-id`, `--path-prefix`, or `--kind` selectors.
|
||||
- Reconcile-state:
|
||||
- selected `--pipeline` and `--destination` repair only outputs currently
|
||||
owned by that pipeline/destination;
|
||||
- `--all-owners` repairs all catalog outputs;
|
||||
- unmanaged reporting compares storage entries to all catalog output paths,
|
||||
not just selected owner paths.
|
||||
- Rewrite repaired/pruned state as schema version `4`.
|
||||
- Remove legacy single-owner/shared-root maintenance branches.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/state ./internal/app ./internal/cli`
|
||||
- Prune selects only current-owner catalog outputs.
|
||||
- Prune deletes selected outputs and removes their catalog records.
|
||||
- Reconcile-state selected owner removes only missing outputs for that owner.
|
||||
- `reconcile-state --all-owners` removes missing outputs for all owners.
|
||||
- Unmanaged reporting excludes all catalog-managed paths and reports unrecorded
|
||||
storage entries.
|
||||
- JSON/text output remains stable and clear.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Maintenance commands operate only on catalog state and respect the locked
|
||||
selector rules.
|
||||
|
||||
## Stage 7: Clean Break Removal And Documentation
|
||||
|
||||
Goal: remove legacy state/config behavior and document the implemented catalog
|
||||
workflow model.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Remove dead code for writing schema version `2` single-owner and schema
|
||||
version `3` shared-root state.
|
||||
- Remove legacy config structs/constants/defaults/validation for destination
|
||||
`state`, `reconciliation`, `takeover`, and `transfer` where no longer used.
|
||||
- Remove or rewrite tests that only assert legacy state/config behavior.
|
||||
- Update current-behavior docs:
|
||||
- `docs/config.md`
|
||||
- `docs/cli.md`
|
||||
- `docs/operations.md`
|
||||
- `docs/troubleshooting.md`
|
||||
- `docs/integrations/destination-state.md`
|
||||
- `docs/internal/publish.md`
|
||||
- `docs/internal/state.md`
|
||||
- `docs/internal/config.md`
|
||||
- relevant `docs/internal/` docs
|
||||
- `docs/policy/architecture.md`
|
||||
- `docs/policy/development.md`
|
||||
- Keep docs concise and link to canonical references rather than duplicating
|
||||
full state semantics in every file.
|
||||
- Do not document any unimplemented future takeover extensions outside
|
||||
`docs/roadmap/`.
|
||||
- Update examples to use `workflow` and remove legacy fields.
|
||||
- Keep roadmap-only material out of current docs.
|
||||
- Once implemented and documented, remove or rewrite
|
||||
`docs/roadmap/catalog.md` so completed behavior is not described only as
|
||||
future work.
|
||||
|
||||
Tests:
|
||||
Tests and checks:
|
||||
|
||||
- `go test ./internal/app ./internal/cli ./internal/config`
|
||||
- Text dry-run output includes `replace_takeover`.
|
||||
- JSON output records `replace_takeover` with stable field names.
|
||||
- Summary counters are deterministic.
|
||||
- Fixed-path warnings remain deterministic.
|
||||
- Config docs examples, if changed or added, load through existing config tests.
|
||||
- Troubleshooting text matches actual error/action wording.
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Operators can distinguish ordinary same-source replacement, takeover
|
||||
replacement, and explicit forced replacement.
|
||||
- Current docs describe implemented takeover behavior outside roadmap files.
|
||||
|
||||
## Stage 6: Cross-Backend Regression And Roadmap Closeout
|
||||
|
||||
Goal: prove takeover behavior is backend-agnostic and clean up roadmap state
|
||||
after implementation.
|
||||
|
||||
Source roadmap reference:
|
||||
|
||||
- Completed takeover feature roadmap: Goals, Non-Goals, Safety Rules.
|
||||
|
||||
Implementation scope:
|
||||
|
||||
- Add app-level coverage showing takeover planning/execution works with storage
|
||||
abstraction rather than adapter-specific logic.
|
||||
- Cover local destinations directly.
|
||||
- Cover SSH and S3 through fake-backed app tests where possible; add adapter
|
||||
integration tests only if existing test infrastructure already supports
|
||||
opt-in remote credentials.
|
||||
- Add tests for `path_mapping.mode: fixed` with different source ids under the
|
||||
same pipeline.
|
||||
- Add tests for archive-style `preserve_relative` destinations using
|
||||
`takeover.mode: same_source` where strict source identity is desired.
|
||||
- Run final consistency searches.
|
||||
- Remove the completed detailed takeover roadmap when no future takeover work
|
||||
remains.
|
||||
|
||||
Tests:
|
||||
|
||||
- `go test ./internal/config`
|
||||
- `go test ./internal/state`
|
||||
- `go test ./internal/publish`
|
||||
- `go test ./internal/app ./internal/cli`
|
||||
- `go test ./...`
|
||||
|
||||
Documentation and consistency checks:
|
||||
|
||||
```sh
|
||||
rg -n "takeover|replace_takeover|same_pipeline|same_source|any_managed" docs README.md examples
|
||||
rg -n "destination source id differs from source|pipeline id .* does not match|destination id .* does not match" docs
|
||||
rg -n "Managed Destination Takeover Roadmap|future takeover|planned takeover" docs README.md examples --glob '!docs/roadmap/**'
|
||||
```
|
||||
- `rg -n "state:|reconciliation:|takeover:|transfer:" examples docs --glob '!docs/roadmap/**'`
|
||||
- `rg -n 'single_owner|shared_root|schema version `2`|schema version `3`' docs internal`
|
||||
- `rg -n "workflow: additive|workflow: replacement|schema_version.*4" docs examples`
|
||||
|
||||
Completion criteria:
|
||||
|
||||
- Current docs and examples describe the catalog workflow model.
|
||||
- Legacy destination policy fields and legacy write paths are gone.
|
||||
- Full test suite passes.
|
||||
- The default `same_pipeline` behavior applies only to valid
|
||||
distributor-managed takeover cases.
|
||||
- Unmanaged content and invalid state remain protected.
|
||||
- Current docs no longer describe implemented takeover behavior as future work.
|
||||
|
||||
## Refactors To Avoid
|
||||
|
||||
- Do not build a generic ownership engine outside `internal/state` and
|
||||
`internal/publish`.
|
||||
- Do not move destination comparison policy into storage adapters.
|
||||
- Do not change the source manifest schema.
|
||||
- Do not add takeover fields to HTTP upload requests or producer APIs.
|
||||
- Do not make `--force` persistent config.
|
||||
- Do not merge `takeover`, `transfer`, and `reconciliation` into one broad
|
||||
policy object.
|
||||
- Do not silently adopt unmanaged content.
|
||||
- Do not add top-level `owners` or `sources` catalogs.
|
||||
- Do not persist workflow in `.distributor.json`.
|
||||
- Do not keep deprecated config aliases for legacy destination policy fields.
|
||||
- Do not implement state migration from schema versions `1`, `2`, or `3`.
|
||||
- Do not add catalog-specific prune/reconcile selectors beyond the agreed
|
||||
initial ownership scope.
|
||||
- Do not move catalog policy into storage adapters.
|
||||
|
||||
## Open Questions
|
||||
|
||||
No open questions are known. The feature roadmap selects the default mode
|
||||
(`same_pipeline`), the accepted modes, the safety boundary, and the relationship
|
||||
to existing state, reconciliation, transfer, and force policies.
|
||||
No open questions are known. The catalog roadmap locks the state shape, workflow
|
||||
values, default workflow, clean-break policy, legacy-state behavior, output
|
||||
field selection, `created_at` preservation, and maintenance command scope.
|
||||
|
||||
Reference in New Issue
Block a user