Add a feature roadmap and staged implementation plan for configurable levels of managed file overwrite protection

docs/roadmap/implementation.md

@@ -0,0 +1,375 @@
+# Managed Destination Takeover Implementation Roadmap
+
+This is the active staged implementation plan for
+`docs/roadmap/takeover.md`. The feature roadmap defines the target policy and
+end-state semantics; this document defines the implementation sequence for an
+LLM coding agent to follow stage by stage.
+
+Future behavior must remain under `docs/roadmap/` until each stage is
+implemented. Preparatory internal stages should not update user-facing current
+docs. Current-behavior docs should be updated when takeover behavior is wired
+for operator-facing use.
+
+## Current Baseline
+
+`distributor` already supports local, SSH/SFTP, S3, and HTTP upload source
+workflows, destination state schemas for single-owner and shared-root state,
+path mapping, link generation, reconciliation, transfer policy, explicit
+`--force`, and text/JSON run output.
+
+Current destination comparison is strict:
+
+- same source manifest skips;
+- same source id with older destination state normally replaces;
+- same source id with newer destination state normally skips;
+- same source id and same creation time with different digest conflicts;
+- different source id, pipeline id, destination id, or shared-root output owner
+  conflicts unless explicit force policy applies;
+- unmanaged content and invalid state do not become normal managed replacement
+  cases.
+
+The target feature adds destination-level `takeover.mode`, defaulting to
+`same_pipeline`, so valid managed content can be replaced without `--force` when
+the configured policy says that ownership/source takeover is expected.
+
+## Implementation Principles
+
+- Preserve public behavior until the stage that explicitly changes it.
+- Keep source manifests unchanged; takeover is destination configuration and
+  publish planning policy.
+- Keep state comparison pure. State comparison may expose structured conflict
+  details, but publish planning decides whether takeover is allowed.
+- Keep adapters thin. No local, SSH, or S3 adapter should know takeover policy.
+- Keep unmanaged content and invalid destination state outside normal takeover.
+- Keep `reconciliation.mode`, `transfer`, `state.mode`, `path_mapping.mode`, and
+  `--force` as separate concepts.
+- Prefer narrow behavior-preserving refactors over broad publication rewrites.
+- Update implemented-behavior docs in the same stage as the behavior change.
+
+## Active Implementation Stages
+
+## Stage 1: Config Model And Validation
+
+Goal: add the destination config field, defaults, and validation without
+changing publish behavior.
+
+Source roadmap reference:
+
+- `docs/roadmap/takeover.md`: Configuration, Policy Semantics, Safety Rules.
+
+Implementation scope:
+
+- Add destination-level `TakeoverPolicy` to `internal/config`.
+- Add constants for:
+  - `same_pipeline`
+  - `same_source`
+  - `any_managed`
+  - `never`
+- Default `takeover.mode` to `same_pipeline` in config defaults.
+- Validate accepted values with clear field context such as
+  `pipelines[0].destinations[0].takeover.mode`.
+- Preserve strict YAML unknown-field behavior.
+- Thread the defaulted policy into existing destination config views or helper
+  structures if those are used by app/publish request construction.
+- Do not change publish planning, execution, CLI output, or docs outside
+  roadmap files in this stage.
+
+Tests:
+
+- `go test ./internal/config`
+- Add config tests for omitted `takeover`, each accepted mode, invalid mode, and
+  unknown nested fields.
+- Add or update example-loading tests only if examples are touched, which should
+  not be necessary in this stage.
+
+Completion criteria:
+
+- Every destination has a defaulted `takeover.mode`.
+- Invalid values fail during config validation.
+- No run behavior changes because publish planning does not consume the policy
+  yet.
+
+## Stage 2: Structured State Comparison Details
+
+Goal: expose enough structured comparison detail for publish planning to decide
+takeover eligibility without parsing human-readable reason strings.
+
+Source roadmap reference:
+
+- `docs/roadmap/takeover.md`: Policy Semantics, Relationship To Existing
+  Policies.
+
+Implementation scope:
+
+- Extend `internal/state.Comparison` or add a package-local structured detail
+  type so callers can distinguish:
+  - pipeline id mismatch;
+  - destination id mismatch;
+  - different source id;
+  - same-created digest conflict;
+  - destination newer;
+  - invalid state;
+  - unmanaged content;
+  - absent shared-root owner;
+  - shared-root managed output owner conflicts, if those are currently reported
+    outside `internal/state`.
+- Keep existing outcome names and reason strings stable where practical.
+- Keep comparison functions pure. They should report facts about existing state,
+  not consult `takeover.mode`, `transfer`, `reconciliation`, or `force`.
+- Do not add new persisted state fields.
+- Do not change publish actions in this stage.
+
+Tests:
+
+- `go test ./internal/state ./internal/publish`
+- Add state tests for structured details on identity mismatch and different
+  source id.
+- Add shared-root tests for owner absence and output ownership conflict detail,
+  either in `internal/state` or `internal/publish` depending on where the
+  conflict is currently detected.
+- Preserve existing comparison outcome tests.
+
+Completion criteria:
+
+- Publish planning can make takeover decisions from structured data.
+- Existing behavior remains unchanged because no takeover mapping is applied yet.
+
+## Stage 3: Single-Owner Takeover Planning And Execution
+
+Goal: implement `takeover.mode` for single-owner destination state.
+
+Source roadmap reference:
+
+- `docs/roadmap/takeover.md`: Policy Semantics, Publish Planning, Destination
+  State Results, Safety Rules.
+
+Implementation scope:
+
+- Add `TakeoverPolicy` or equivalent values to `internal/publish.Request`.
+- Pass destination takeover policy from app-level run planning into publish.
+- Add publish action `replace_takeover`.
+- Map eligible single-owner conflicts to `replace_takeover`:
+  - `same_pipeline`: existing state pipeline id equals current pipeline id,
+    regardless of destination id or source id;
+  - `same_source`: existing state source id equals current source id;
+  - `any_managed`: existing state is valid distributor-managed state;
+  - `never`: no identity/source takeover.
+- Keep these cases failing by default unless existing force behavior applies:
+  - unmanaged content;
+  - invalid state;
+  - same-created digest conflict;
+  - destination newer for the same source id unless `transfer` plus `--force`
+    already permits replacement;
+  - conflicts not allowed by `takeover.mode`.
+- Execute `replace_takeover` through bounded managed replacement mechanics.
+- For cross-source takeover, do not retain omitted outputs through
+  `reconciliation.mode: merge`; treat the affected single-owner state as a
+  managed replacement so old-source outputs are not attributed to the new source.
+- Preserve existing `replace_older`, `skip_same`, `skip_destination_newer`,
+  `fail_conflict`, `fail_unmanaged`, and `force_replace` behavior.
+- Keep adapters unchanged.
+
+Current-behavior documentation updates:
+
+- Do not update user docs yet unless CLI output changes in this stage. Prefer
+  deferring user docs to Stage 5 so the behavior, output, and docs land
+  together.
+- If this stage changes visible dry-run or run output enough that tests require
+  new wording, document only the implemented single-owner behavior and clearly
+  leave shared-root takeover out until Stage 4.
+
+Tests:
+
+- `go test ./internal/publish ./internal/app`
+- Default `same_pipeline` replaces a different source id from the same pipeline.
+- Default `same_pipeline` replaces state with a different destination id under
+  the same pipeline.
+- Default `same_pipeline` refuses a different pipeline.
+- `same_source` allows same source id and refuses different source id.
+- `any_managed` replaces valid state from a different pipeline.
+- `never` refuses identity/source takeover.
+- Invalid state and unmanaged content still fail without force.
+- Cross-source takeover with `reconciliation.mode: merge` does not retain
+  omitted outputs from the previous source.
+- Dry-run plans `replace_takeover` without writing.
+- Existing force tests still pass.
+
+Completion criteria:
+
+- Single-owner latest-style destinations can be updated by different bundle ids
+  from the same configured pipeline without `--force`.
+- No unmanaged or invalid-state path becomes a normal takeover path.
+
+## Stage 4: Shared-Root Takeover Planning And Execution
+
+Goal: apply the same takeover vocabulary to shared-root owner and output-path
+conflicts.
+
+Source roadmap reference:
+
+- `docs/roadmap/takeover.md`: Policy Semantics, Publish Planning, Destination
+  State Results, Safety Rules.
+
+Implementation scope:
+
+- Extend shared-root planning so planned output collisions with existing managed
+  owners are eligible for takeover according to `takeover.mode`.
+- Apply policy as follows:
+  - `same_pipeline`: current owner may take over output paths owned by another
+    destination under the same pipeline;
+  - `same_source`: current owner may take over output paths whose owner records
+    the same source id as the current source;
+  - `any_managed`: current owner may take over output paths owned by any valid
+    shared-root owner;
+  - `never`: preserve current owner conflict behavior.
+- Preserve unrelated owner records and non-conflicting output records.
+- Keep unmanaged path collisions failing without force.
+- Keep compatible single-owner-to-shared-root migration behavior intact.
+- For cross-source takeover, do not retain omitted outputs from the previous
+  source under the taking-over owner when `reconciliation.mode: merge` is set.
+- Keep shared-root forced replacement behavior explicit and bounded as it is
+  today.
+
+Current-behavior documentation updates:
+
+- Defer broad documentation updates to Stage 5 unless shared-root behavior must
+  be documented immediately to keep tests or generated docs consistent.
+
+Tests:
+
+- `go test ./internal/state ./internal/publish ./internal/app`
+- `same_pipeline` permits taking over a path owned by another destination in the
+  same pipeline.
+- `same_pipeline` refuses a path owned by another pipeline.
+- `same_source` permits only matching source-id ownership transfer.
+- `any_managed` permits cross-pipeline managed ownership transfer.
+- `never` preserves current shared-root conflict behavior.
+- Reconciliation `replace` and `merge` handle omitted outputs according to the
+  feature roadmap, including the cross-source no-retain rule.
+- Shared-root migration from compatible single-owner state still works.
+- Unmanaged collisions still fail without force.
+
+Completion criteria:
+
+- Single-owner and shared-root destinations use one consistent takeover policy.
+- Shared-root takeover changes only affected owners/outputs and preserves
+  unrelated managed state.
+
+## Stage 5: Run Output, Summaries, And Documentation
+
+Goal: make takeover behavior visible to operators and document the implemented
+feature.
+
+Source roadmap reference:
+
+- `docs/roadmap/takeover.md`: Documentation Impact, Publish Planning,
+  Relationship To Existing Policies.
+
+Implementation scope:
+
+- Add `replace_takeover` to text run output.
+- Add `replace_takeover` to JSON run action output.
+- Add a distinct summary counter for takeover replacements in text and JSON run
+  summaries, using the field name `replace_takeover` for JSON.
+- Include takeover mode and conflict reason in action projection where useful
+  and consistent with existing output style.
+- Update fixed-path dry-run warnings when takeover will replace the destination
+  root.
+- Update current-behavior docs:
+  - `docs/config.md`
+  - `docs/operations.md`
+  - `docs/troubleshooting.md`
+  - `docs/integrations/destination-state.md`
+  - `docs/internal/publish.md`
+  - `docs/internal/state.md`
+  - `docs/internal/config.md`
+  - `docs/policy/architecture.md`
+  - `docs/policy/development.md`
+- Keep docs concise and link to canonical references rather than duplicating
+  full state semantics in every file.
+- Do not document any unimplemented future takeover extensions outside
+  `docs/roadmap/`.
+
+Tests:
+
+- `go test ./internal/app ./internal/cli ./internal/config`
+- Text dry-run output includes `replace_takeover`.
+- JSON output records `replace_takeover` with stable field names.
+- Summary counters are deterministic.
+- Fixed-path warnings remain deterministic.
+- Config docs examples, if changed or added, load through existing config tests.
+- Troubleshooting text matches actual error/action wording.
+
+Completion criteria:
+
+- Operators can distinguish ordinary same-source replacement, takeover
+  replacement, and explicit forced replacement.
+- Current docs describe implemented takeover behavior outside roadmap files.
+
+## Stage 6: Cross-Backend Regression And Roadmap Closeout
+
+Goal: prove takeover behavior is backend-agnostic and clean up roadmap state
+after implementation.
+
+Source roadmap reference:
+
+- `docs/roadmap/takeover.md`: Goals, Non-Goals, Safety Rules.
+
+Implementation scope:
+
+- Add app-level coverage showing takeover planning/execution works with storage
+  abstraction rather than adapter-specific logic.
+- Cover local destinations directly.
+- Cover SSH and S3 through fake-backed app tests where possible; add adapter
+  integration tests only if existing test infrastructure already supports
+  opt-in remote credentials.
+- Add tests for `path_mapping.mode: fixed` with different source ids under the
+  same pipeline.
+- Add tests for archive-style `preserve_relative` destinations using
+  `takeover.mode: same_source` where strict source identity is desired.
+- Run final consistency searches.
+- Once behavior and docs are complete, either remove `docs/roadmap/takeover.md`
+  or reduce it to future-only material according to the documentation policy.
+  If no future takeover work remains, remove the roadmap file.
+
+Tests:
+
+- `go test ./internal/config`
+- `go test ./internal/state`
+- `go test ./internal/publish`
+- `go test ./internal/app ./internal/cli`
+- `go test ./...`
+
+Documentation and consistency checks:
+
+```sh
+rg -n "takeover|replace_takeover|same_pipeline|same_source|any_managed" docs README.md examples
+rg -n "destination source id differs from source|pipeline id .* does not match|destination id .* does not match" docs
+rg -n "Managed Destination Takeover Roadmap|future takeover|planned takeover" docs README.md examples --glob '!docs/roadmap/**'
+```
+
+Completion criteria:
+
+- Full test suite passes.
+- The default `same_pipeline` behavior applies only to valid
+  distributor-managed takeover cases.
+- Unmanaged content and invalid state remain protected.
+- Current docs no longer describe implemented takeover behavior as future work.
+
+## Refactors To Avoid
+
+- Do not build a generic ownership engine outside `internal/state` and
+  `internal/publish`.
+- Do not move destination comparison policy into storage adapters.
+- Do not change the source manifest schema.
+- Do not add takeover fields to HTTP upload requests or producer APIs.
+- Do not make `--force` persistent config.
+- Do not merge `takeover`, `transfer`, and `reconciliation` into one broad
+  policy object.
+- Do not silently adopt unmanaged content.
+
+## Open Questions
+
+No open questions are known. The feature roadmap selects the default mode
+(`same_pipeline`), the accepted modes, the safety boundary, and the relationship
+to existing state, reconciliation, transfer, and force policies.

docs/roadmap/takeover.md

@@ -0,0 +1,229 @@
+# Managed Destination Takeover Roadmap
+
+This roadmap defines planned replacement policy for distributor-managed
+destination content when the existing destination state does not describe the
+same source bundle as the current publication.
+
+Current behavior is intentionally conservative. A destination may be replaced
+automatically when its existing state describes the same source id and an older
+source creation time. If the existing state belongs to a different source id,
+pipeline id, destination id, or shared-root owner, `distributor` reports a
+conflict unless an explicit forced replacement workflow is selected where
+supported.
+
+That strict default was useful for the initial implementation, but it is too
+rigid for producers that intentionally publish newer generated content to the
+same destination path from different producer tasks or bundle ids. `distributor`
+is primarily a distribution tool: by default, a valid producer request should
+publish the requested bundle unless doing so would claim unmanaged content or
+cross an ownership boundary that is likely to surprise the operator.
+
+## Goals
+
+- Add a destination-level policy that defines when normal, non-force publication
+  may replace existing distributor-managed content whose source or owner
+  identity differs from the current source.
+- Default the policy to `same_pipeline`, so a pipeline may distribute the latest
+  valid bundle to its configured destination paths without requiring every
+  producer task to reuse one source manifest id.
+- Preserve strong safety boundaries for unmanaged content, invalid destination
+  state, and cross-pipeline ownership by default.
+- Keep reconciliation policy focused on how managed output sets are updated
+  after replacement is allowed.
+- Keep transfer policy focused on same-source comparison outcomes and explicit
+  forced replacement behavior.
+- Preserve clear dry-run and JSON output so takeover behavior is visible before
+  files are changed.
+
+## Non-Goals
+
+- Do not allow automatic adoption of unmanaged destination files.
+- Do not allow invalid `.distributor.json` state to be overwritten without the
+  existing explicit force workflow.
+- Do not make producers select destination ownership or takeover policy through
+  source manifests or HTTP upload requests.
+- Do not replace `state.mode`, `reconciliation.mode`, `path_mapping.mode`, or
+  `transfer` policy.
+- Do not introduce broad synchronization behavior outside configured
+  destination bundle paths.
+
+## Configuration
+
+Add destination-level takeover policy:
+
+```yaml
+destinations:
+  - id: latest
+    backend: local
+    path: /srv/reports/weather/latest/tomorrow
+    path_mapping:
+      mode: fixed
+    takeover:
+      mode: same_pipeline
+```
+
+Accepted `takeover.mode` values:
+
+- `same_pipeline`: default. Normal publication may replace valid
+  distributor-managed state or managed output paths owned by the same pipeline,
+  even when the previous source id or destination id differs.
+- `same_source`: normal publication may replace only when the destination state
+  has the same source manifest id as the current source. This is closest to the
+  current strict managed replacement behavior.
+- `any_managed`: normal publication may replace any valid distributor-managed
+  state or managed output path in the selected destination bundle path,
+  regardless of pipeline id, destination id, or source id.
+- `never`: normal publication never performs takeover replacement for identity
+  or source conflicts. Same-source older/newer behavior remains governed by
+  `transfer`, and explicit forced replacement remains governed by `--force` and
+  `transfer.on_conflict`.
+
+The field is destination-local because different destinations from the same
+pipeline can have different ownership expectations. Archive destinations often
+want stricter behavior, while latest-style fixed destinations often want
+`same_pipeline` or, after explicit operator review, `any_managed`.
+
+## Policy Semantics
+
+Takeover policy applies only after destination state has been parsed and
+validated as distributor-managed state.
+
+For single-owner state:
+
+- `same_pipeline` permits replacement when the existing state `pipeline_id`
+  matches the current pipeline id. The previous `destination_id` and source id
+  may differ.
+- `same_source` permits replacement only when the existing source manifest id
+  matches the current source manifest id.
+- `any_managed` permits replacement of any valid single-owner destination state.
+- `never` does not permit takeover replacement.
+
+For shared-root state:
+
+- Takeover is evaluated per managed output path that collides with the current
+  plan and per current owner record when one already exists.
+- `same_pipeline` permits the current destination owner to take over output
+  paths owned by another destination under the same pipeline.
+- `same_source` permits takeover only when the colliding output owner records
+  the same source manifest id as the current source.
+- `any_managed` permits takeover of colliding managed output paths owned by any
+  owner in the shared root.
+- `never` preserves the current owner-scoped conflict behavior.
+
+Takeover never applies to:
+
+- destination content with no valid `.distributor.json`;
+- invalid destination state;
+- planned paths that exist in storage but are not recorded as managed;
+- source digest mismatches or invalid source manifests;
+- destination newer/same-created digest comparisons for the same source id,
+  except where existing transfer policy already permits replacement.
+
+## Relationship To Existing Policies
+
+`takeover.mode` answers this question:
+
+> May this destination normally replace valid managed content whose owner or
+> source identity differs from the current publication?
+
+`reconciliation.mode` continues to answer:
+
+> Once replacement is allowed, should omitted managed outputs be removed
+> (`replace`) or retained (`merge`)?
+
+`transfer` continues to answer:
+
+> What should happen for same-source comparisons, destination-newer state, and
+> explicit forced conflict replacement?
+
+`--force` remains an operator workflow for exceptional cases. Takeover policy is
+not a replacement for forced replacement of unmanaged content or invalid state.
+
+## Publish Planning
+
+Add a new planned action for successful takeover replacement, tentatively named
+`replace_takeover`.
+
+`replace_takeover` should execute through the same bounded managed replacement
+machinery as other managed replacements, with one additional source-identity
+safety rule: when takeover crosses source id or owner identity, omitted outputs
+from the previous source must not be retained merely because the destination is
+configured with `reconciliation.mode: merge`. Retaining omitted outputs is valid
+only when those outputs still belong to the same source identity being
+published. This keeps destination state from claiming that outputs derived from
+an old source belong to the new source manifest.
+
+- single-owner `replace` deletes only managed outputs recorded in existing
+  state plus `.distributor.json`, then writes the new outputs and state;
+- single-owner `merge` retains omitted managed outputs only for same-source
+  updates; cross-source takeover behaves as a managed replacement for the
+  affected owner;
+- shared-root `replace` deletes only output records taken over by the current
+  owner or omitted from the current owner according to the shared-root plan;
+- shared-root `merge` preserves unrelated non-conflicting managed outputs and
+  updates ownership for paths explicitly taken over, but does not retain omitted
+  outputs from a different source under the taking-over owner.
+
+Text output, JSON output, dry-run output, and summary counters should identify
+takeover replacements separately from `replace_older` and `force_replace`.
+
+## Destination State Results
+
+After a successful takeover, destination state must describe the current
+pipeline, destination, source manifest, outputs, links, reconciliation policy,
+and state mode.
+
+For single-owner state, the destination bundle path has one current owner after
+publication: the publishing pipeline and destination.
+
+For shared-root state, only the affected owner records and output records are
+changed. Unrelated owners and non-conflicting outputs remain recorded.
+
+No source manifest schema change is required. No destination state schema change
+is expected unless implementation discovers that takeover metadata must be
+persisted for recovery or audit. The preferred initial design is to make
+takeover a planning decision, not a new persisted state concept.
+
+## Documentation Impact
+
+When this feature is implemented, current-behavior docs should explain the new
+policy where operators and maintainers already look for publication safety,
+state comparison, and configuration behavior:
+
+- `docs/config.md`: document `takeover.mode`, default `same_pipeline`, accepted
+  values, examples for archive and latest destinations, and interaction with
+  `transfer`, `reconciliation`, and `--force`.
+- `docs/operations.md`: explain takeover dry-runs, normal replacement safety,
+  shared-root behavior, and recovery guidance.
+- `docs/troubleshooting.md`: update conflict guidance for
+  `destination source id differs from source`, pipeline/destination mismatch,
+  and shared-root ownership conflicts.
+- `docs/integrations/destination-state.md`: describe how takeover affects
+  comparison and state rewriting without changing the source manifest contract.
+- `docs/internal/publish.md`, `docs/internal/state.md`, and
+  `docs/internal/config.md`: document package responsibilities and invariants.
+- `docs/policy/architecture.md`: clarify that the default posture is to
+  distribute valid producer bundles through configured destinations while
+  protecting unmanaged content and cross-pipeline ownership by default.
+- `docs/policy/development.md`: add `takeover.mode` to the list of config
+  changes that must be validated and documented when publish safety changes.
+
+## Safety Rules
+
+- Takeover must be dry-runnable.
+- Takeover must operate only inside the resolved destination bundle path.
+- Takeover must never delete parent paths, sibling paths, or unmanaged files.
+- Takeover must never claim storage content that is not recorded in valid
+  destination state.
+- `any_managed` should be documented as an intentional cross-pipeline ownership
+  policy for tightly controlled destinations, not as the recommended default for
+  archives.
+- Archive-style destinations should generally use `same_source` if each bundle
+  id represents a distinct immutable artifact.
+
+## Implementation Reference
+
+The staged implementation plan for this feature lives in
+`docs/roadmap/implementation.md`. This document remains the feature contract and
+policy reference; implementation sequencing, package-by-package work, and test
+commands belong in the implementation roadmap.