diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md new file mode 100644 index 0000000..d9a98cd --- /dev/null +++ b/docs/roadmap/implementation.md @@ -0,0 +1,375 @@ +# Managed Destination Takeover Implementation Roadmap + +This is the active staged implementation plan for +`docs/roadmap/takeover.md`. The feature roadmap defines the target policy and +end-state semantics; this document defines the implementation sequence for an +LLM coding agent to follow stage by stage. + +Future behavior must remain under `docs/roadmap/` until each stage is +implemented. Preparatory internal stages should not update user-facing current +docs. Current-behavior docs should be updated when takeover behavior is wired +for operator-facing use. + +## Current Baseline + +`distributor` already supports local, SSH/SFTP, S3, and HTTP upload source +workflows, destination state schemas for single-owner and shared-root state, +path mapping, link generation, reconciliation, transfer policy, explicit +`--force`, and text/JSON run output. + +Current destination comparison is strict: + +- same source manifest skips; +- same source id with older destination state normally replaces; +- same source id with newer destination state normally skips; +- same source id and same creation time with different digest conflicts; +- different source id, pipeline id, destination id, or shared-root output owner + conflicts unless explicit force policy applies; +- unmanaged content and invalid state do not become normal managed replacement + cases. + +The target feature adds destination-level `takeover.mode`, defaulting to +`same_pipeline`, so valid managed content can be replaced without `--force` when +the configured policy says that ownership/source takeover is expected. + +## Implementation Principles + +- Preserve public behavior until the stage that explicitly changes it. +- Keep source manifests unchanged; takeover is destination configuration and + publish planning policy. +- Keep state comparison pure. State comparison may expose structured conflict + details, but publish planning decides whether takeover is allowed. +- Keep adapters thin. No local, SSH, or S3 adapter should know takeover policy. +- Keep unmanaged content and invalid destination state outside normal takeover. +- Keep `reconciliation.mode`, `transfer`, `state.mode`, `path_mapping.mode`, and + `--force` as separate concepts. +- Prefer narrow behavior-preserving refactors over broad publication rewrites. +- Update implemented-behavior docs in the same stage as the behavior change. + +## Active Implementation Stages + +## Stage 1: Config Model And Validation + +Goal: add the destination config field, defaults, and validation without +changing publish behavior. + +Source roadmap reference: + +- `docs/roadmap/takeover.md`: Configuration, Policy Semantics, Safety Rules. + +Implementation scope: + +- Add destination-level `TakeoverPolicy` to `internal/config`. +- Add constants for: + - `same_pipeline` + - `same_source` + - `any_managed` + - `never` +- Default `takeover.mode` to `same_pipeline` in config defaults. +- Validate accepted values with clear field context such as + `pipelines[0].destinations[0].takeover.mode`. +- Preserve strict YAML unknown-field behavior. +- Thread the defaulted policy into existing destination config views or helper + structures if those are used by app/publish request construction. +- Do not change publish planning, execution, CLI output, or docs outside + roadmap files in this stage. + +Tests: + +- `go test ./internal/config` +- Add config tests for omitted `takeover`, each accepted mode, invalid mode, and + unknown nested fields. +- Add or update example-loading tests only if examples are touched, which should + not be necessary in this stage. + +Completion criteria: + +- Every destination has a defaulted `takeover.mode`. +- Invalid values fail during config validation. +- No run behavior changes because publish planning does not consume the policy + yet. + +## Stage 2: Structured State Comparison Details + +Goal: expose enough structured comparison detail for publish planning to decide +takeover eligibility without parsing human-readable reason strings. + +Source roadmap reference: + +- `docs/roadmap/takeover.md`: Policy Semantics, Relationship To Existing + Policies. + +Implementation scope: + +- Extend `internal/state.Comparison` or add a package-local structured detail + type so callers can distinguish: + - pipeline id mismatch; + - destination id mismatch; + - different source id; + - same-created digest conflict; + - destination newer; + - invalid state; + - unmanaged content; + - absent shared-root owner; + - shared-root managed output owner conflicts, if those are currently reported + outside `internal/state`. +- Keep existing outcome names and reason strings stable where practical. +- Keep comparison functions pure. They should report facts about existing state, + not consult `takeover.mode`, `transfer`, `reconciliation`, or `force`. +- Do not add new persisted state fields. +- Do not change publish actions in this stage. + +Tests: + +- `go test ./internal/state ./internal/publish` +- Add state tests for structured details on identity mismatch and different + source id. +- Add shared-root tests for owner absence and output ownership conflict detail, + either in `internal/state` or `internal/publish` depending on where the + conflict is currently detected. +- Preserve existing comparison outcome tests. + +Completion criteria: + +- Publish planning can make takeover decisions from structured data. +- Existing behavior remains unchanged because no takeover mapping is applied yet. + +## Stage 3: Single-Owner Takeover Planning And Execution + +Goal: implement `takeover.mode` for single-owner destination state. + +Source roadmap reference: + +- `docs/roadmap/takeover.md`: Policy Semantics, Publish Planning, Destination + State Results, Safety Rules. + +Implementation scope: + +- Add `TakeoverPolicy` or equivalent values to `internal/publish.Request`. +- Pass destination takeover policy from app-level run planning into publish. +- Add publish action `replace_takeover`. +- Map eligible single-owner conflicts to `replace_takeover`: + - `same_pipeline`: existing state pipeline id equals current pipeline id, + regardless of destination id or source id; + - `same_source`: existing state source id equals current source id; + - `any_managed`: existing state is valid distributor-managed state; + - `never`: no identity/source takeover. +- Keep these cases failing by default unless existing force behavior applies: + - unmanaged content; + - invalid state; + - same-created digest conflict; + - destination newer for the same source id unless `transfer` plus `--force` + already permits replacement; + - conflicts not allowed by `takeover.mode`. +- Execute `replace_takeover` through bounded managed replacement mechanics. +- For cross-source takeover, do not retain omitted outputs through + `reconciliation.mode: merge`; treat the affected single-owner state as a + managed replacement so old-source outputs are not attributed to the new source. +- Preserve existing `replace_older`, `skip_same`, `skip_destination_newer`, + `fail_conflict`, `fail_unmanaged`, and `force_replace` behavior. +- Keep adapters unchanged. + +Current-behavior documentation updates: + +- Do not update user docs yet unless CLI output changes in this stage. Prefer + deferring user docs to Stage 5 so the behavior, output, and docs land + together. +- If this stage changes visible dry-run or run output enough that tests require + new wording, document only the implemented single-owner behavior and clearly + leave shared-root takeover out until Stage 4. + +Tests: + +- `go test ./internal/publish ./internal/app` +- Default `same_pipeline` replaces a different source id from the same pipeline. +- Default `same_pipeline` replaces state with a different destination id under + the same pipeline. +- Default `same_pipeline` refuses a different pipeline. +- `same_source` allows same source id and refuses different source id. +- `any_managed` replaces valid state from a different pipeline. +- `never` refuses identity/source takeover. +- Invalid state and unmanaged content still fail without force. +- Cross-source takeover with `reconciliation.mode: merge` does not retain + omitted outputs from the previous source. +- Dry-run plans `replace_takeover` without writing. +- Existing force tests still pass. + +Completion criteria: + +- Single-owner latest-style destinations can be updated by different bundle ids + from the same configured pipeline without `--force`. +- No unmanaged or invalid-state path becomes a normal takeover path. + +## Stage 4: Shared-Root Takeover Planning And Execution + +Goal: apply the same takeover vocabulary to shared-root owner and output-path +conflicts. + +Source roadmap reference: + +- `docs/roadmap/takeover.md`: Policy Semantics, Publish Planning, Destination + State Results, Safety Rules. + +Implementation scope: + +- Extend shared-root planning so planned output collisions with existing managed + owners are eligible for takeover according to `takeover.mode`. +- Apply policy as follows: + - `same_pipeline`: current owner may take over output paths owned by another + destination under the same pipeline; + - `same_source`: current owner may take over output paths whose owner records + the same source id as the current source; + - `any_managed`: current owner may take over output paths owned by any valid + shared-root owner; + - `never`: preserve current owner conflict behavior. +- Preserve unrelated owner records and non-conflicting output records. +- Keep unmanaged path collisions failing without force. +- Keep compatible single-owner-to-shared-root migration behavior intact. +- For cross-source takeover, do not retain omitted outputs from the previous + source under the taking-over owner when `reconciliation.mode: merge` is set. +- Keep shared-root forced replacement behavior explicit and bounded as it is + today. + +Current-behavior documentation updates: + +- Defer broad documentation updates to Stage 5 unless shared-root behavior must + be documented immediately to keep tests or generated docs consistent. + +Tests: + +- `go test ./internal/state ./internal/publish ./internal/app` +- `same_pipeline` permits taking over a path owned by another destination in the + same pipeline. +- `same_pipeline` refuses a path owned by another pipeline. +- `same_source` permits only matching source-id ownership transfer. +- `any_managed` permits cross-pipeline managed ownership transfer. +- `never` preserves current shared-root conflict behavior. +- Reconciliation `replace` and `merge` handle omitted outputs according to the + feature roadmap, including the cross-source no-retain rule. +- Shared-root migration from compatible single-owner state still works. +- Unmanaged collisions still fail without force. + +Completion criteria: + +- Single-owner and shared-root destinations use one consistent takeover policy. +- Shared-root takeover changes only affected owners/outputs and preserves + unrelated managed state. + +## Stage 5: Run Output, Summaries, And Documentation + +Goal: make takeover behavior visible to operators and document the implemented +feature. + +Source roadmap reference: + +- `docs/roadmap/takeover.md`: Documentation Impact, Publish Planning, + Relationship To Existing Policies. + +Implementation scope: + +- Add `replace_takeover` to text run output. +- Add `replace_takeover` to JSON run action output. +- Add a distinct summary counter for takeover replacements in text and JSON run + summaries, using the field name `replace_takeover` for JSON. +- Include takeover mode and conflict reason in action projection where useful + and consistent with existing output style. +- Update fixed-path dry-run warnings when takeover will replace the destination + root. +- Update current-behavior docs: + - `docs/config.md` + - `docs/operations.md` + - `docs/troubleshooting.md` + - `docs/integrations/destination-state.md` + - `docs/internal/publish.md` + - `docs/internal/state.md` + - `docs/internal/config.md` + - `docs/policy/architecture.md` + - `docs/policy/development.md` +- Keep docs concise and link to canonical references rather than duplicating + full state semantics in every file. +- Do not document any unimplemented future takeover extensions outside + `docs/roadmap/`. + +Tests: + +- `go test ./internal/app ./internal/cli ./internal/config` +- Text dry-run output includes `replace_takeover`. +- JSON output records `replace_takeover` with stable field names. +- Summary counters are deterministic. +- Fixed-path warnings remain deterministic. +- Config docs examples, if changed or added, load through existing config tests. +- Troubleshooting text matches actual error/action wording. + +Completion criteria: + +- Operators can distinguish ordinary same-source replacement, takeover + replacement, and explicit forced replacement. +- Current docs describe implemented takeover behavior outside roadmap files. + +## Stage 6: Cross-Backend Regression And Roadmap Closeout + +Goal: prove takeover behavior is backend-agnostic and clean up roadmap state +after implementation. + +Source roadmap reference: + +- `docs/roadmap/takeover.md`: Goals, Non-Goals, Safety Rules. + +Implementation scope: + +- Add app-level coverage showing takeover planning/execution works with storage + abstraction rather than adapter-specific logic. +- Cover local destinations directly. +- Cover SSH and S3 through fake-backed app tests where possible; add adapter + integration tests only if existing test infrastructure already supports + opt-in remote credentials. +- Add tests for `path_mapping.mode: fixed` with different source ids under the + same pipeline. +- Add tests for archive-style `preserve_relative` destinations using + `takeover.mode: same_source` where strict source identity is desired. +- Run final consistency searches. +- Once behavior and docs are complete, either remove `docs/roadmap/takeover.md` + or reduce it to future-only material according to the documentation policy. + If no future takeover work remains, remove the roadmap file. + +Tests: + +- `go test ./internal/config` +- `go test ./internal/state` +- `go test ./internal/publish` +- `go test ./internal/app ./internal/cli` +- `go test ./...` + +Documentation and consistency checks: + +```sh +rg -n "takeover|replace_takeover|same_pipeline|same_source|any_managed" docs README.md examples +rg -n "destination source id differs from source|pipeline id .* does not match|destination id .* does not match" docs +rg -n "Managed Destination Takeover Roadmap|future takeover|planned takeover" docs README.md examples --glob '!docs/roadmap/**' +``` + +Completion criteria: + +- Full test suite passes. +- The default `same_pipeline` behavior applies only to valid + distributor-managed takeover cases. +- Unmanaged content and invalid state remain protected. +- Current docs no longer describe implemented takeover behavior as future work. + +## Refactors To Avoid + +- Do not build a generic ownership engine outside `internal/state` and + `internal/publish`. +- Do not move destination comparison policy into storage adapters. +- Do not change the source manifest schema. +- Do not add takeover fields to HTTP upload requests or producer APIs. +- Do not make `--force` persistent config. +- Do not merge `takeover`, `transfer`, and `reconciliation` into one broad + policy object. +- Do not silently adopt unmanaged content. + +## Open Questions + +No open questions are known. The feature roadmap selects the default mode +(`same_pipeline`), the accepted modes, the safety boundary, and the relationship +to existing state, reconciliation, transfer, and force policies. diff --git a/docs/roadmap/takeover.md b/docs/roadmap/takeover.md new file mode 100644 index 0000000..02f6fcc --- /dev/null +++ b/docs/roadmap/takeover.md @@ -0,0 +1,229 @@ +# Managed Destination Takeover Roadmap + +This roadmap defines planned replacement policy for distributor-managed +destination content when the existing destination state does not describe the +same source bundle as the current publication. + +Current behavior is intentionally conservative. A destination may be replaced +automatically when its existing state describes the same source id and an older +source creation time. If the existing state belongs to a different source id, +pipeline id, destination id, or shared-root owner, `distributor` reports a +conflict unless an explicit forced replacement workflow is selected where +supported. + +That strict default was useful for the initial implementation, but it is too +rigid for producers that intentionally publish newer generated content to the +same destination path from different producer tasks or bundle ids. `distributor` +is primarily a distribution tool: by default, a valid producer request should +publish the requested bundle unless doing so would claim unmanaged content or +cross an ownership boundary that is likely to surprise the operator. + +## Goals + +- Add a destination-level policy that defines when normal, non-force publication + may replace existing distributor-managed content whose source or owner + identity differs from the current source. +- Default the policy to `same_pipeline`, so a pipeline may distribute the latest + valid bundle to its configured destination paths without requiring every + producer task to reuse one source manifest id. +- Preserve strong safety boundaries for unmanaged content, invalid destination + state, and cross-pipeline ownership by default. +- Keep reconciliation policy focused on how managed output sets are updated + after replacement is allowed. +- Keep transfer policy focused on same-source comparison outcomes and explicit + forced replacement behavior. +- Preserve clear dry-run and JSON output so takeover behavior is visible before + files are changed. + +## Non-Goals + +- Do not allow automatic adoption of unmanaged destination files. +- Do not allow invalid `.distributor.json` state to be overwritten without the + existing explicit force workflow. +- Do not make producers select destination ownership or takeover policy through + source manifests or HTTP upload requests. +- Do not replace `state.mode`, `reconciliation.mode`, `path_mapping.mode`, or + `transfer` policy. +- Do not introduce broad synchronization behavior outside configured + destination bundle paths. + +## Configuration + +Add destination-level takeover policy: + +```yaml +destinations: + - id: latest + backend: local + path: /srv/reports/weather/latest/tomorrow + path_mapping: + mode: fixed + takeover: + mode: same_pipeline +``` + +Accepted `takeover.mode` values: + +- `same_pipeline`: default. Normal publication may replace valid + distributor-managed state or managed output paths owned by the same pipeline, + even when the previous source id or destination id differs. +- `same_source`: normal publication may replace only when the destination state + has the same source manifest id as the current source. This is closest to the + current strict managed replacement behavior. +- `any_managed`: normal publication may replace any valid distributor-managed + state or managed output path in the selected destination bundle path, + regardless of pipeline id, destination id, or source id. +- `never`: normal publication never performs takeover replacement for identity + or source conflicts. Same-source older/newer behavior remains governed by + `transfer`, and explicit forced replacement remains governed by `--force` and + `transfer.on_conflict`. + +The field is destination-local because different destinations from the same +pipeline can have different ownership expectations. Archive destinations often +want stricter behavior, while latest-style fixed destinations often want +`same_pipeline` or, after explicit operator review, `any_managed`. + +## Policy Semantics + +Takeover policy applies only after destination state has been parsed and +validated as distributor-managed state. + +For single-owner state: + +- `same_pipeline` permits replacement when the existing state `pipeline_id` + matches the current pipeline id. The previous `destination_id` and source id + may differ. +- `same_source` permits replacement only when the existing source manifest id + matches the current source manifest id. +- `any_managed` permits replacement of any valid single-owner destination state. +- `never` does not permit takeover replacement. + +For shared-root state: + +- Takeover is evaluated per managed output path that collides with the current + plan and per current owner record when one already exists. +- `same_pipeline` permits the current destination owner to take over output + paths owned by another destination under the same pipeline. +- `same_source` permits takeover only when the colliding output owner records + the same source manifest id as the current source. +- `any_managed` permits takeover of colliding managed output paths owned by any + owner in the shared root. +- `never` preserves the current owner-scoped conflict behavior. + +Takeover never applies to: + +- destination content with no valid `.distributor.json`; +- invalid destination state; +- planned paths that exist in storage but are not recorded as managed; +- source digest mismatches or invalid source manifests; +- destination newer/same-created digest comparisons for the same source id, + except where existing transfer policy already permits replacement. + +## Relationship To Existing Policies + +`takeover.mode` answers this question: + +> May this destination normally replace valid managed content whose owner or +> source identity differs from the current publication? + +`reconciliation.mode` continues to answer: + +> Once replacement is allowed, should omitted managed outputs be removed +> (`replace`) or retained (`merge`)? + +`transfer` continues to answer: + +> What should happen for same-source comparisons, destination-newer state, and +> explicit forced conflict replacement? + +`--force` remains an operator workflow for exceptional cases. Takeover policy is +not a replacement for forced replacement of unmanaged content or invalid state. + +## Publish Planning + +Add a new planned action for successful takeover replacement, tentatively named +`replace_takeover`. + +`replace_takeover` should execute through the same bounded managed replacement +machinery as other managed replacements, with one additional source-identity +safety rule: when takeover crosses source id or owner identity, omitted outputs +from the previous source must not be retained merely because the destination is +configured with `reconciliation.mode: merge`. Retaining omitted outputs is valid +only when those outputs still belong to the same source identity being +published. This keeps destination state from claiming that outputs derived from +an old source belong to the new source manifest. + +- single-owner `replace` deletes only managed outputs recorded in existing + state plus `.distributor.json`, then writes the new outputs and state; +- single-owner `merge` retains omitted managed outputs only for same-source + updates; cross-source takeover behaves as a managed replacement for the + affected owner; +- shared-root `replace` deletes only output records taken over by the current + owner or omitted from the current owner according to the shared-root plan; +- shared-root `merge` preserves unrelated non-conflicting managed outputs and + updates ownership for paths explicitly taken over, but does not retain omitted + outputs from a different source under the taking-over owner. + +Text output, JSON output, dry-run output, and summary counters should identify +takeover replacements separately from `replace_older` and `force_replace`. + +## Destination State Results + +After a successful takeover, destination state must describe the current +pipeline, destination, source manifest, outputs, links, reconciliation policy, +and state mode. + +For single-owner state, the destination bundle path has one current owner after +publication: the publishing pipeline and destination. + +For shared-root state, only the affected owner records and output records are +changed. Unrelated owners and non-conflicting outputs remain recorded. + +No source manifest schema change is required. No destination state schema change +is expected unless implementation discovers that takeover metadata must be +persisted for recovery or audit. The preferred initial design is to make +takeover a planning decision, not a new persisted state concept. + +## Documentation Impact + +When this feature is implemented, current-behavior docs should explain the new +policy where operators and maintainers already look for publication safety, +state comparison, and configuration behavior: + +- `docs/config.md`: document `takeover.mode`, default `same_pipeline`, accepted + values, examples for archive and latest destinations, and interaction with + `transfer`, `reconciliation`, and `--force`. +- `docs/operations.md`: explain takeover dry-runs, normal replacement safety, + shared-root behavior, and recovery guidance. +- `docs/troubleshooting.md`: update conflict guidance for + `destination source id differs from source`, pipeline/destination mismatch, + and shared-root ownership conflicts. +- `docs/integrations/destination-state.md`: describe how takeover affects + comparison and state rewriting without changing the source manifest contract. +- `docs/internal/publish.md`, `docs/internal/state.md`, and + `docs/internal/config.md`: document package responsibilities and invariants. +- `docs/policy/architecture.md`: clarify that the default posture is to + distribute valid producer bundles through configured destinations while + protecting unmanaged content and cross-pipeline ownership by default. +- `docs/policy/development.md`: add `takeover.mode` to the list of config + changes that must be validated and documented when publish safety changes. + +## Safety Rules + +- Takeover must be dry-runnable. +- Takeover must operate only inside the resolved destination bundle path. +- Takeover must never delete parent paths, sibling paths, or unmanaged files. +- Takeover must never claim storage content that is not recorded in valid + destination state. +- `any_managed` should be documented as an intentional cross-pipeline ownership + policy for tightly controlled destinations, not as the recommended default for + archives. +- Archive-style destinations should generally use `same_source` if each bundle + id represents a distinct immutable artifact. + +## Implementation Reference + +The staged implementation plan for this feature lives in +`docs/roadmap/implementation.md`. This document remains the feature contract and +policy reference; implementation sequencing, package-by-package work, and test +commands belong in the implementation roadmap.