From c804fd604a694fdfc26160bc7944dac7fa06bcc1 Mon Sep 17 00:00:00 2001 From: Eric Rakestraw Date: Mon, 8 Jun 2026 19:42:08 +0000 Subject: [PATCH] Refresh feature roadmap documentation --- docs/config.md | 4 +- docs/roadmap/implementation.md | 661 +++----------------------------- docs/roadmap/multipipeline.md | 210 ++-------- docs/roadmap/prune.md | 204 ++-------- docs/roadmap/reconcile-state.md | 188 ++------- docs/roadmap/reconciliation.md | 188 +-------- 6 files changed, 155 insertions(+), 1300 deletions(-) diff --git a/docs/config.md b/docs/config.md index 3e65b39..348a909 100644 --- a/docs/config.md +++ b/docs/config.md @@ -35,7 +35,7 @@ pipelines: path: /srv/reports/archive ``` -This config publishes source files only. It uses default validation, destination path mapping, publish, reconciliation, transfer, and HTTP server values. +This config publishes source files only. It uses default validation, destination path mapping, publish, state, reconciliation, retention, transfer, and HTTP server values. ## Production-Oriented Local Config @@ -255,7 +255,7 @@ Source bundle digest mismatches fail validation before destination writes occur. ## Destination Fields -Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, state, reconciliation, and transfer behavior. +Each destination embeds a backend config at the destination level and may also configure publishing, transforms, path mapping, links, state, reconciliation, retention, and transfer behavior. ```yaml destinations: diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index bcbbeca..18df97b 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,628 +1,67 @@ -# Reconciliation, Shared-Root, Reconcile-State, And Prune Implementation Plan +# Reconciliation, Shared-Root, Reconcile-State, And Prune Status -This document is the staged implementation plan for the feature roadmaps in: +This roadmap file tracks the status and remaining boundaries for the feature +group that includes destination reconciliation, shared-root destination state, +state repair, and managed-output pruning. -- `docs/roadmap/reconciliation.md` -- `docs/roadmap/multipipeline.md` -- `docs/roadmap/reconcile-state.md` -- `docs/roadmap/prune.md` +Current behavior is documented in: -Audience: LLM coding agents and maintainers implementing these features in -order. +- `docs/config.md` +- `docs/cli.md` +- `docs/operations.md` +- `docs/troubleshooting.md` +- `docs/integrations/destination-state.md` +- `docs/internal/app.md` +- `docs/internal/publish.md` +- `docs/internal/state.md` +- `docs/internal/storage.md` -This is roadmap guidance, not current behavior documentation. Do not describe -any stage as implemented outside `docs/roadmap/` until that stage has landed. +## Current Scope -## Global Decisions +Current scope includes: -- Implement in this order: - 1. destination reconciliation modes; - 2. shared-root multi-pipeline state; - 3. reconcile-state tooling; - 4. pruning. -- Keep reconciliation mode per destination as `reconciliation.mode`. -- Use reconciliation modes `replace` and `merge`; default is `replace`. -- Persist reconciliation metadata in `.distributor.json`. -- Keep shared-root behavior as an explicit per-destination opt-in using - `state.mode: single_owner|shared_root`. -- Key shared-root output ownership by both `pipeline_id` and `destination_id`. -- Do not allow implicit takeover of another owner path. -- Store compact per-output source identity fields, and keep the latest full - source manifest once per owner where publish comparison needs it. -- `reconcile-state` applies repairs by default and uses `--dry-run` for - read-only inspection. -- `prune` is opt-in, uses configured retention policy, and requires `--apply` - for deletion. -- Use per-output `updated_at` for initial prune policies. -- Support both `older_than` and `keep_latest` pruning in the initial prune - implementation. -- Write single-owner reconciliation-capable state as schema version `2`. -- Write shared-root collection state as schema version `3`. -- Do not adopt unmanaged files, validate output digests, or perform - whole-config reconciliation in the initial reconcile-state command. -- Do not run pruning automatically after publish in the initial implementation. +- destination `reconciliation.mode: replace|merge`; +- destination `state.mode: single_owner|shared_root`; +- schema version `2` single-owner destination state; +- schema version `3` shared-root destination state; +- owner-scoped shared-root publishing; +- `reconcile-state` for removing missing managed-output records from valid + destination state; +- destination `retention.prune` policy with `older_than` and `keep_latest`; +- `prune --dry-run` and `prune --apply` for configured managed-output pruning. -## Cross-Cutting Constraints +## Current Boundaries -- Follow `docs/policy/architecture.md`, `docs/policy/development.md`, and - `docs/policy/documentation.md`. -- Keep producer manifests free of destination routing, reconciliation, state, - retention, transform, link, and backend policy. -- Keep adapter packages thin. State comparison, reconciliation, pruning, and - retention decisions belong in `internal/state`, `internal/publish`, and - `internal/app`, not concrete backends. -- Keep public packages `pkg/bundle` and `pkg/upload` unaffected unless an - implementation stage explicitly requires consumer-facing docs updates. -- Use dry-run behavior for every workflow that may write state or delete files. -- Never delete unmanaged content. -- Use `storage.StateFileName`, `storage.StatePath`, - `storage.ManagedBundleTargets`, and storage path validation helpers instead - of duplicating path rules. -- Update implemented-behavior docs and examples in the same change as the - corresponding behavior. Keep future behavior in roadmap docs only. -- A feature group is not complete until its paired documentation stage lands. - Do not move the roadmap item out of future/planned status until the behavior, - tests, current-behavior docs, and examples for that group are complete. -- Run the stage-specific tests before moving to the next stage. Run - `go test ./...` after each completed feature group. +- Producer manifests and upload requests do not carry destination routing, + reconciliation, state mode, retention, transform, link, or backend policy. +- Reconcile-state repairs state records only. It does not adopt unmanaged + files, validate output digests, repair invalid JSON, or reconcile an entire + config at once. +- Pruning is explicit operator action. It does not run automatically after + publish and does not accept one-off retention overrides. +- Pruning deletes only managed output paths recorded in destination state. It + does not delete unmanaged content or `.distributor.json`. +- Shared-root publishing preserves unrelated owners and rejects implicit + takeover of another owner path. -## Stage 1: Reconciliation Config And State Foundation +## Deferred Work -Goal: add destination-level reconciliation configuration and a state model that -can persist reconciliation metadata and cumulative outputs for single-owner -destinations. +Deferred or future work belongs in the other roadmap files under +`docs/roadmap/`. Current deferred items include: -Implementation: +- unmanaged-file adoption workflow; +- digest-audit mode for state repair; +- whole-config state repair; +- automatic post-publish pruning; +- path/date parsing retention policies; +- producer-facing retention, reconciliation, or destination routing APIs; +- broad storage synchronization behavior. -- Add config types: - - `Destination.Reconciliation ReconciliationPolicy` - - `ReconciliationPolicy.Mode string` -- Add constants in `internal/config/defaults.go`: - - `ReconciliationModeReplace = "replace"` - - `ReconciliationModeMerge = "merge"` -- Default `reconciliation.mode` to `replace` for every destination. -- Validate `reconciliation.mode` as `replace` or `merge`. -- Add focused config tests for defaults, explicit `replace`, explicit `merge`, - and invalid modes. -- Extend `internal/state` for a new single-owner state schema that writes: - - `schema_version`; - - `distributor_version`; - - `pipeline_id`; - - `destination_id`; - - `published_at`; - - `created_at`; - - `updated_at`; - - `state.mode: "single_owner"`; - - `reconciliation.mode`; - - `source.manifest`; - - `links`; - - cumulative `outputs`. -- Add per-output `created_at` and `updated_at` fields now. For current - single-owner output records written by this stage, set both timestamps when a - path is first created and preserve `created_at` when a path is updated. -- Keep parsing current schema version 1 state for compatibility. When v1 state - is read, infer: - - `state.mode = "single_owner"`; - - `reconciliation.mode = "replace"`; - - `created_at = published_at`; - - `updated_at = published_at`; - - each output `created_at` and `updated_at` from `published_at`. -- Write only the new schema for newly written state after this stage. -- Keep validation strict: required timestamps, valid reconciliation mode, unique - output paths, valid links, valid digests, clean paths, and valid embedded - source manifest. -- Add state helper functions for: - - finding output records by path; - - merging retained and newly planned output records; - - computing owner-scoped managed output paths for single-owner state; - - projecting publish outputs into state outputs with timestamps. +## Verification -Tests: +Before changing this feature group, run the focused tests for affected +packages and finish with: ```sh -go test ./internal/config -go test ./internal/state -``` - -Keep publish semantics unchanged in this stage. Only adapt current state -construction to write schema version `2` with default `replace` metadata and -timestamps. - -## Stage 2: Single-Owner Replace And Merge Publish Execution - -Goal: implement `reconciliation.mode` for current single-owner destinations. - -Implementation: - -- Add `Reconciliation config.ReconciliationPolicy` to `publish.Request` and - `publish.Plan`. -- Thread destination reconciliation config from `internal/app` into - `internal/publish`. -- Keep `replace` behavior equivalent to current behavior: - - delete prior managed outputs absent from the new plan; - - write the new outputs; - - write state whose `outputs` are exactly the new planned output set. -- Implement `merge` behavior: - - inspect existing state before writing; - - retain previous managed outputs not overwritten by the new plan; - - allow overwrite only when the destination path is already recorded in - existing state as managed; - - fail if a newly planned path exists in storage but is not recorded as - managed; - - write new or overwritten outputs; - - write state with the cumulative managed output set; - - derive `links.primary_url` from newly planned outputs only; - - clean up only outputs written by the failed attempt. -- For `merge`, update per-output timestamps: - - new output path: `created_at = updated_at = now`; - - overwritten managed path: preserve existing `created_at`, set - `updated_at = now`; - - retained output: preserve both timestamps. -- Ensure state writes use overwrite semantics appropriate for replacing the - existing `.distributor.json`, while output writes remain narrow and explicit. -- Preserve existing force behavior. Do not add unmanaged-content adoption. - -Tests: - -```sh -go test ./internal/publish ./internal/state -go test ./internal/app ./internal/cli go test ./... ``` - -Required coverage: - -- `replace` deletes omitted managed outputs. -- `merge` retains omitted managed outputs. -- `merge` overwrites an existing managed path. -- `merge` fails on unmanaged destination path collision. -- failed merge cleanup deletes only attempted new writes. -- fixed-path destinations support both modes. -- source-only, generated-only, and source-plus-generated policies honor both - modes. - -## Stage 3: Reconciliation Documentation And Examples - -Goal: make implemented reconciliation behavior visible outside roadmap docs. -The reconciliation feature group is not complete until this stage lands. - -Implementation: - -- Update `docs/config.md` with `reconciliation.mode`, defaults, and examples. -- Update `docs/operations.md` with replace versus merge behavior, dry-run - expectations, failed-write cleanup, and recovery notes. -- Update `docs/integrations/destination-state.md` with the new state schema, - reconciliation metadata, output timestamps, and v1 compatibility behavior. -- Update `docs/internal/state.md` and `docs/internal/publish.md`. -- Update relevant examples under `examples/` only if they are valid and - executable for implemented behavior. - -Tests: - -```sh -go test ./internal/config -go test ./... -``` - -## Stage 4: Shared-Root State Model And Config - -Goal: add explicit shared-root destination state support without changing -publish behavior yet. - -Implementation: - -- Add destination state config: - - `Destination.State StatePolicy` - - `StatePolicy.Mode string` - - accepted values: `single_owner`, `shared_root` - - default: `single_owner` -- Validate state mode values. -- Add a shared-root state schema in `internal/state`. -- The shared-root JSON must include: - - `schema_version`; - - `distributor_version`; - - `created_at`; - - `updated_at`; - - `state.mode: "shared_root"`; - - owner records keyed by `pipeline_id` and `destination_id`; - - latest full source manifest per owner for comparison; - - owner-level reconciliation metadata; - - owner-level latest `links.primary_url`, if present; - - output records for every managed path in the root. -- Shared-root output records must include: - - path; - - kind; - - source path; - - transform, when generated; - - URL, when present; - - SHA-256; - - size; - - owner `pipeline_id`; - - owner `destination_id`; - - source manifest id; - - source manifest digest; - - source manifest created timestamp; - - `created_at`; - - `updated_at`. -- Keep compact per-output source identity fields. Do not repeat full source - manifests on every output. -- Add state helpers that can: - - parse both single-owner and shared-root state; - - identify the current owner scope; - - return the latest source manifest for one owner; - - list managed paths for one owner or all owners; - - detect path ownership conflicts; - - merge one owner's planned output state while preserving unrelated owners; - - remove one owner's omitted outputs for owner-scoped `replace`. -- Do not automatically convert unrelated single-owner state to shared-root - state. If shared-root is configured and existing single-owner state matches - the current owner, convert it during the first successful shared-root publish. - Otherwise fail with an actionable conflict. - -Tests: - -```sh -go test ./internal/config -go test ./internal/state -``` - -Do not enable shared-root publish behavior until Stage 5. - -## Stage 5: Shared-Root Publish Planning And Comparison - -Goal: teach publish planning to reason about shared-root owner scopes without -writing shared-root outputs yet. - -Implementation: - -- Thread destination state mode from `internal/app` into `internal/publish`. -- Update destination inspection and comparison so: - - `single_owner` destinations preserve current single-owner comparison - semantics; - - `shared_root` destinations compare only the current - `pipeline_id`/`destination_id` owner scope; - - absent owner state behaves like destination absent for that owner, unless - unmanaged storage content collides with a planned output; - - outputs owned by other owners are preserved. -- Enforce ownership conflict rules: - - same `pipeline_id` and `destination_id` may overwrite its own managed path; - - different owner writing the same path fails as a conflict; - - unmanaged storage path collision fails by default; - - no implicit takeover. -- Compose reconciliation within the owner scope: - - owner-scoped `replace` removes prior outputs for that owner that are absent - from the new plan; - - owner-scoped `merge` retains prior outputs for that owner that are absent - from the new plan. -- Return plans that include the current owner scope, retained owner outputs, - owner outputs to delete, and owner outputs to write. -- Keep execution disabled for shared-root write actions until Stage 6. - -Tests: - -```sh -go test ./internal/publish ./internal/state -go test ./internal/app -``` - -Required coverage: - -- planning treats absent owner state as publishable for that owner; -- planning preserves other owners in the planned retained state; -- path conflict with another owner plans or returns a conflict failure; -- unmanaged path collision plans or returns an unmanaged-content failure; -- owner-scoped `replace` selects only that owner's omitted outputs for deletion; -- owner-scoped `merge` retains that owner's omitted outputs; -- current single-owner planning remains covered. - -## Stage 6: Shared-Root Publish Execution - -Goal: allow multiple pipelines to publish disjoint managed paths into one -shared destination root. - -Implementation: - -- Preserve unrelated owners and their outputs exactly. -- Preserve and update timestamps: - - top-level `created_at` remains the original root creation time; - - top-level `updated_at` changes on successful state writes; - - output `created_at` is stable; - - output `updated_at` changes only when that path is rewritten; - - owner metadata updates when that owner publishes. -- Keep failed-write cleanup limited to outputs written by the failed attempt. -- Keep forced replacement bounded and explicitly reported. Forced replacement - for a shared-root destination may delete the configured destination root, so - retain existing dry-run visibility and do not broaden its scope. - -Tests: - -```sh -go test ./internal/publish ./internal/state -go test ./internal/app ./internal/cli -go test ./... -``` - -Required coverage: - -- two pipelines publish disjoint paths into the same shared root; -- one pipeline replaces its own output without deleting another owner's output; -- one pipeline merges new outputs while preserving its own retained outputs and - other owners' outputs; -- path conflict with another owner fails; -- unmanaged path collision fails; -- shared-root dry-run writes no outputs or state; -- current single-owner behavior remains covered. - -## Stage 7: Shared-Root Documentation And Examples - -Goal: document implemented shared-root behavior. -The shared-root feature group is not complete until this stage lands. - -Implementation: - -- Update `docs/config.md` with destination state mode config and defaults. -- Update `docs/integrations/destination-state.md` with shared-root schema, - owners, output ownership, timestamps, and migration rules. -- Update `docs/operations.md` with shared-root publishing, conflicts, dry-run, - and recovery behavior. -- Update `docs/troubleshooting.md` with ownership conflict and unmanaged - collision entries. -- Update `docs/internal/state.md` and `docs/internal/publish.md`. -- Add maintained examples only when they are valid and load-tested. - -Tests: - -```sh -go test ./internal/config -go test ./... -``` - -## Stage 8: Reconcile-State Planning Core - -Goal: add app-level planning and state helpers for reconcile-state without -adding CLI execution first. - -Implementation: - -- Add `internal/app` reconcile-state use case types: - - options with config path, pipeline id, destination id, all-owners flag, - dry-run flag, stdout, and output format; - - report struct with destination identity, backend, root path, state schema, - owner scope, checked count, missing managed outputs, unmanaged entries, and - changed/would-change status. -- Add state helpers for removing missing managed output records: - - single-owner scope; - - shared-root current-owner scope; - - shared-root all-owner scope. -- Use storage `Stat` for managed output existence checks. -- Use bounded `Walk` under the selected destination root to report unmanaged - existing files. Exclude `.distributor.json` and all paths already recorded as - managed. -- Do not validate output digests. -- Do not delete destination files. -- Do not adopt unmanaged files. -- Do not rewrite invalid or ambiguous state. -- Require enough scope to identify one destination root. For current - single-owner state, require pipeline and destination. For shared-root - all-owner mode, still require a pipeline/destination selector to identify the - configured destination root, then apply `--all-owners` inside that root. -- Default behavior applies state repair. `DryRun` reports without writing. - -Tests: - -```sh -go test ./internal/app ./internal/state ./internal/storage/fake -``` - -Required coverage: - -- dry-run reports missing managed outputs without rewriting state; -- apply removes missing records and writes valid state; -- unmanaged files are reported but not adopted or deleted; -- invalid state fails without rewrite; -- shared-root owner-scope and all-owner repair both work. - -## Stage 9: Reconcile-State CLI And Docs - -Goal: expose reconcile-state as an operator command. -The reconcile-state feature group is not complete until this stage lands. - -Implementation: - -- Add `reconcile-state` to `internal/cli/root.go` help and dispatch. -- Add CLI flags: - - `--config `; - - `--pipeline `; - - `--destination `; - - `--all-owners`; - - `--dry-run`; - - `--format text|json`. -- Without `--dry-run`, apply state repairs. -- Text output should clearly report whether state was changed or would be - changed. -- JSON output should use the existing app JSON envelope pattern. -- Update `docs/cli.md`, `docs/operations.md`, `docs/troubleshooting.md`, - `docs/integrations/destination-state.md`, `docs/internal/state.md`, - `docs/internal/storage.md`, and `docs/internal/app.md`. - -Tests: - -```sh -go test ./internal/cli ./internal/app -go test ./... -``` - -Required coverage: - -- flag validation; -- default apply behavior; -- `--dry-run` behavior; -- text and JSON reports; -- no deletion of destination files. - -## Stage 10: Prune Config And Planning Core - -Goal: add retention config and pure prune planning. - -Implementation: - -- Add destination retention config: - - `Destination.Retention RetentionPolicy` - - `RetentionPolicy.Prune PrunePolicy` - - `PrunePolicy.Enabled bool` - - `PrunePolicy.OlderThan *Duration` - - `PrunePolicy.KeepLatest *int` -- Default pruning disabled. -- Validate: - - enabled prune policy must set at least one of `older_than` or - `keep_latest`; - - `older_than` must be positive; - - `keep_latest` must be zero or greater; - - documented duration units must parse consistently with existing duration - config behavior. -- Add app/state prune planning helpers: - - select owner-scoped managed outputs; - - sort deterministically by `updated_at`, then path; - - preserve the newest `keep_latest` outputs; - - delete remaining outputs older than `older_than`; - - if both policies are set, first preserve `keep_latest`, then delete - remaining outputs older than `older_than`. -- Use per-output `updated_at` as the timestamp basis. -- Plan only; do not delete storage files in this stage. - -Tests: - -```sh -go test ./internal/config -go test ./internal/app ./internal/state -``` - -Required coverage: - -- config defaults and validation; -- `older_than` planning; -- `keep_latest` planning; -- combined policy planning; -- deterministic tie-breaking; -- shared-root owner scope preservation. - -## Stage 11: Prune Execution Core - -Goal: implement safe managed-output pruning below the CLI layer. - -Implementation: - -- Add app prune execution: - - load config; - - resolve selected destination and owner scope; - - read and validate state; - - build prune plan from configured retention policy; - - in dry-run, write no files and delete nothing; - - in apply mode, delete only planned managed output paths; - - after confirmed deletes, remove deleted records from state and update state - timestamps; - - preserve state records for failed deletes so retry remains accurate; - - never delete unmanaged files; - - do not delete `.distributor.json` unless a later roadmap explicitly permits - empty-state removal. -- Do not add CLI dispatch in this stage. -- Return a prune report that can later be rendered by text and JSON CLI output. -- Do not add one-off retention overrides in the initial implementation. -- Let local and SSH/SFTP backend helpers prune empty directories only where - they already do so safely. Preserve object-storage prefix markers unless they - are managed output records. - -Tests: - -```sh -go test ./internal/config -go test ./internal/app ./internal/state -go test ./internal/storage/fake -``` - -Required coverage: - -- dry-run reports deletes without deleting files or rewriting state; -- apply deletes only managed output paths; -- unmanaged files under the destination root are preserved; -- partial delete failure preserves accurate state for confirmed deleted and - undeleted outputs; -- shared-root pruning preserves other owners when scoped to one owner. - -## Stage 12: Prune CLI And Docs - -Goal: expose safe managed-output pruning as an operator command. The prune -feature group is not complete until this stage lands. - -Implementation: - -- Add `prune` CLI dispatch and help. -- Add CLI flags: - - `--config `; - - `--pipeline `; - - `--destination `; - - `--dry-run`; - - `--apply`; - - `--format text|json`. -- Require exactly one of `--dry-run` or `--apply`. `--dry-run` plans and - reports without writing. `--apply` deletes planned managed outputs and - rewrites state after confirmed deletes. -- JSON output should use the existing app JSON envelope pattern. -- Update `docs/config.md`, `docs/cli.md`, `docs/operations.md`, - `docs/troubleshooting.md`, `docs/integrations/destination-state.md`, - `docs/internal/state.md`, `docs/internal/publish.md`, and - `docs/internal/app.md`. - -Tests: - -```sh -go test ./internal/config -go test ./internal/app ./internal/cli ./internal/state -go test ./internal/storage/fake -go test ./... -``` - -Required coverage: - -- CLI requires exactly one of `--dry-run` or `--apply`; -- text and JSON reports; -- CLI apply gating and report output. - -## Final Verification - -After all stages land: - -```sh -go test ./internal/config -go test ./internal/state ./internal/publish -go test ./internal/app ./internal/cli -go test ./internal/storage/fake -go test ./pkg/bundle ./pkg/upload -go test ./... -``` - -Also run: - -```sh -rg -n "reconciliation|shared_root|single_owner|reconcile-state|retention|prune|older_than|keep_latest" README.md docs examples -``` - -Confirm: - -- current-behavior docs outside `docs/roadmap/` describe only implemented - behavior; -- roadmap docs no longer describe completed behavior as future work, or are - revised to mark only remaining deferred work; -- examples load successfully where config tests cover them; -- producer-facing docs still keep destination routing, reconciliation, state, - and retention policy out of producer manifests and upload requests. - -## Non-Goals For This Plan - -- No unmanaged-file adoption workflow. -- No digest-audit mode for reconcile-state. -- No whole-config reconcile-state command. -- No automatic post-publish pruning. -- No path/date parsing retention policy. -- No producer-facing retention, reconciliation, or destination routing API. -- No broad storage synchronization behavior. diff --git a/docs/roadmap/multipipeline.md b/docs/roadmap/multipipeline.md index 6f1cf58..15d8c17 100644 --- a/docs/roadmap/multipipeline.md +++ b/docs/roadmap/multipipeline.md @@ -1,195 +1,35 @@ # Multi-Pipeline Shared Root Roadmap -This document records planned shared-root destination state work that is not -part of the current implementation. Current destination state behavior is -documented in `docs/integrations/destination-state.md`. +Current shared-root destination state behavior is documented in: -This work should not be implemented until the reconciliation work in -`docs/roadmap/reconciliation.md` is present and complete. Shared-root -multi-pipeline publishing should then be implemented before reconcile-state -tooling (`docs/roadmap/reconcile-state.md`) and pruning -(`docs/roadmap/prune.md`) because both tools should operate on the final shared -state model. +- `docs/config.md` +- `docs/operations.md` +- `docs/troubleshooting.md` +- `docs/integrations/destination-state.md` +- `docs/internal/publish.md` +- `docs/internal/state.md` -## Problem +## Current Scope -Current `.distributor.json` state is a single-owner destination bundle record. -It stores one `pipeline_id`, one `destination_id`, one latest source manifest, -and one output set. That works when each pipeline writes to a separate -destination bundle path. +Current scope includes `state.mode: shared_root` for destinations that +intentionally share one destination root across multiple pipeline/destination +owners. -It does not work well when multiple pipelines intentionally publish into one -shared destination root, for example: +Shared-root state: -```text -2026-06-08/daily/report.md -2026-06-08/event/watch-123.md -``` +- records owner identity by both `pipeline_id` and `destination_id`; +- records each managed output with its owner and compact source identity; +- preserves unrelated owners during owner-scoped publish, state repair, and + pruning; +- rejects implicit takeover of another owner path; +- treats unmanaged storage content as unmanaged unless explicit forced + replacement is selected where supported. -If both pipelines resolve to the same destination bundle path, the second -pipeline encounters a state identity mismatch even when its output paths are -disjoint. Separating roots avoids the conflict, but can make retention, -cleanup, linking, and human browsing less natural. +## Deferred Work -## Goal +No additional shared-root publishing work is planned here. Related deferred work +is tracked in: -Support multiple configured pipelines writing to one shared destination root -while preserving distributor ownership, unmanaged-content safety, dry-run -behavior, and deterministic conflict handling. - -The desired shape is a collection-oriented `.distributor.json` that records -managed outputs from multiple owners under one state file. Producers still -submit source bundles only. Producers must not choose destinations, -reconciliation mode, retention policy, transforms, links, or storage backends. - -Shared-root behavior should be an explicit per-destination state mode, for -example `state.mode: single_owner|shared_root` or a similarly named field under -the destination. Shared-root behavior changes the meaning of `.distributor.json`, -so it should be visible at the destination boundary rather than inferred from -other configured pipelines. - -## State Model - -Add a new destination state schema that can represent a managed collection -rather than one latest source bundle. - -State shape: - -- top-level `schema_version`; -- top-level `distributor_version`; -- top-level `created_at` for the first time this managed root was created; -- top-level `updated_at` for the last successful state update; -- top-level collection metadata identifying the state as shared-root capable; -- an `outputs` array containing all managed output records in the shared root; -- per-output owner metadata: - - `pipeline_id`; - - `destination_id`; - - source manifest id; - - source manifest digest; - - source manifest created timestamp; - - output kind, source path, transform, URL, SHA-256, and size; - - `created_at` for first publication of that output path; - - `updated_at` for the most recent write of that output path. -- optional source-manifest history keyed by source digest, if compact per-output - source identity fields are not enough for later diagnostics or tooling. - -Output ownership is keyed by both `pipeline_id` and `destination_id`. The -architecture treats destinations as independent policy boundaries, and the same -pipeline may publish different output sets to different destinations. - -The state should be able to answer these questions without scanning the whole -destination: - -- Which paths are distributor-managed? -- Which pipeline/destination/source currently owns each managed path? -- Which outputs belong to a specific pipeline/destination publish operation? -- Which outputs are old enough, or least recently updated enough, for later - pruning decisions? -- Which paths are absent from storage and should be removed by reconcile-state - tooling? - -## Ownership And Conflict Rules - -Shared-root support must keep ownership explicit. - -Required rules: - -- A publish may write only output paths planned from its configured pipeline and - destination. -- A publish may overwrite an existing managed output path only when ownership - rules allow it. -- A new output path that exists in storage but is not recorded in - `.distributor.json` remains unmanaged content and fails by default. -- Disjoint outputs from different pipelines can coexist in one state file. -- State updates must preserve unrelated managed outputs from other pipelines. -- Failed-write cleanup must delete only paths written by the failed attempt. -- Forced replacement must remain explicit, dry-runnable, and bounded to the - configured destination root. - -Overwrite rule: an output path may be overwritten by the same `pipeline_id` and -`destination_id`; a different pipeline or destination writing the same path -fails as an ownership conflict unless a later explicit operator takeover -workflow is implemented. - -## Planning And Execution Work - -Implementation work: - -- Introduce a new state schema and parser that can represent shared ownership. -- Keep current schema parsing available long enough to migrate or reject old - state with clear errors. -- Add explicit per-destination configuration that opts a destination into - shared-root behavior. -- Thread shared-root state semantics through destination inspection, - comparison, publish planning, and execution. -- Make comparison operate on the subset of state owned by the current - pipeline/destination where appropriate. -- Make output collision detection check both newly planned outputs and existing - managed outputs from other owners. -- Write state atomically with preserved unrelated owners and updated records for - the current publish. -- Preserve merge and replace semantics within the current owner scope: - - owner-scoped `replace` removes prior outputs for that owner that are absent - from the new plan; - - owner-scoped `merge` retains prior outputs for that owner that are absent - from the new plan. -- Ensure local, SSH/SFTP, S3, and fake backends support the necessary state read, - write, overwrite, and cleanup paths. - -## Migration And Compatibility - -The initial implementation should prefer explicit safety over automatic -conversion. - -Migration behavior: - -- A destination configured for shared-root behavior may read current single-owner - state and convert it only when the existing state identity matches the current - pipeline/destination or when an explicit migration command/flag is used. -- A destination not configured for shared-root behavior continues to use the - current single-owner state semantics. -- Invalid or ambiguous state fails with an actionable error and a dry-run report - path. - -## Testing - -Important tests: - -- Two pipelines publish disjoint paths into the same destination root. -- One pipeline can replace its own older output without deleting another - pipeline's output. -- One pipeline can merge new outputs while retaining its prior outputs and - another pipeline's outputs. -- A pipeline cannot silently overwrite another pipeline's managed path. -- A pipeline cannot claim unmanaged storage content by default. -- State timestamps are set on creation and updated on subsequent writes. -- Per-output timestamps preserve original `created_at` and update `updated_at` - only when that output path is rewritten. -- Failed writes clean up only attempted outputs. -- Dry-run reports shared-root actions without writing outputs or state. -- Forced replacement remains bounded and explicitly reported. -- Current single-owner state behavior remains covered for non-shared - destinations. - -## Documentation Work - -When implemented, update current-behavior docs in the same change: - -- `docs/integrations/destination-state.md`: document the new schema and - ownership rules. -- `docs/config.md`: document the shared-root opt-in configuration. -- `docs/operations.md`: explain shared-root publishing, conflicts, and recovery. -- `docs/troubleshooting.md`: add shared-root ownership conflict entries. -- `docs/internal/state.md` and `docs/internal/publish.md`: record the new - planning and state invariants. -- Maintained examples that demonstrate date-first shared roots only after the - behavior is implemented. - -## Boundaries - -- Do not let producers route to destinations or choose shared-root behavior. -- Do not add prune behavior in this work. -- Do not add broad unmanaged-content adoption in this work. -- Do not make shared-root behavior implicit for all destinations. -- Do not use storage listings as the primary source of truth during normal - publishing; `.distributor.json` remains the managed ownership index. +- `docs/roadmap/future.md` +- `docs/roadmap/reconcile-state.md` +- `docs/roadmap/prune.md` diff --git a/docs/roadmap/prune.md b/docs/roadmap/prune.md index cf12aad..18d8f51 100644 --- a/docs/roadmap/prune.md +++ b/docs/roadmap/prune.md @@ -1,183 +1,45 @@ # Prune Roadmap -This document records planned destination pruning work that is not part of the -current implementation. Current cleanup and forced replacement behavior is -documented in `docs/operations.md`. +Current managed-output pruning behavior is documented in: -This work should be implemented after the reconciliation work in -`docs/roadmap/reconciliation.md`, shared-root state work in -`docs/roadmap/multipipeline.md`, and reconcile-state tooling in -`docs/roadmap/reconcile-state.md`. Prune decisions should rely on accurate -managed state, explicit ownership, and timestamp metadata. +- `docs/config.md` +- `docs/cli.md` +- `docs/operations.md` +- `docs/troubleshooting.md` +- `docs/integrations/destination-state.md` +- `docs/internal/app.md` +- `docs/internal/publish.md` +- `docs/internal/state.md` -## Problem +## Current Scope -Merge reconciliation and shared-root publishing allow long-lived destinations -to accumulate distributor-managed outputs. That is useful for archives and -date-first report layouts, but those destinations can grow without bound. +Current scope includes: -Operators need a safe way to delete old distributor-managed outputs according -to explicit retention policy without touching unmanaged files or outputs owned -by unrelated scopes. +- destination `retention.prune.enabled`; +- `retention.prune.older_than`; +- `retention.prune.keep_latest`; +- `distributor prune --dry-run`; +- `distributor prune --apply`; +- text and JSON reports; +- owner-scoped pruning for shared-root state. -## Goal +Pruning uses managed output `updated_at` timestamps from destination state. +When both `keep_latest` and `older_than` are configured, the newest +`keep_latest` outputs are preserved before age-based pruning is applied to the +remaining managed outputs. -Add pruning support for managed destination outputs. +Apply mode deletes only planned managed output paths, preserves unmanaged +content, preserves `.distributor.json`, and rewrites state only for confirmed +deletions. -Initial pruning should support both of these policy families: +## Deferred Work -- keep only the most recently updated N files in a configured scope; -- delete managed files older than a configured duration. +Deferred pruning work: -Pruning must be based on `.distributor.json` ownership records. It should never -delete files merely because they exist under a configured prefix. +- one-off retention overrides; +- automatic post-publish pruning; +- path/date parsing retention policies; +- group-level pruning by source publication; +- removal of empty state files. -## Policy Model - -Configuration shape: - -```yaml -destinations: - - id: archive - backend: s3 - bucket: reports - prefix: weather - reconciliation: - mode: merge - retention: - prune: - enabled: true - older_than: 90d - keep_latest: 500 -``` - -Policy semantics: - -- `enabled` defaults to `false`. -- `older_than` deletes managed outputs whose selected timestamp is older than - the duration. -- `keep_latest` keeps the most recent N managed outputs in scope and deletes - older managed outputs beyond that count. -- If both are set, preserve outputs that must be kept by `keep_latest`, then - delete remaining outputs older than `older_than`. -- Retention scope is the configured destination owner scope unless - shared-root configuration explicitly chooses a broader scope. - -Timestamp basis: use per-output `updated_at` for both policies, with -`created_at` available for future created-time retention. Updated-time -retention best matches recurring reports where a stable path is overwritten. - -## CLI Shape - -Command shape: - -```sh -distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run -distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --apply -``` - -The command should use configured retention policy by default. A later explicit -override mode can allow one-off retention values if that is useful, but the -initial implementation should avoid ad hoc deletion flags that bypass reviewed -config. - -## Behavior - -Planned behavior: - -- Load and validate config normally. -- Resolve the selected destination and retention policy. -- Read and validate `.distributor.json`. -- Select managed output records in the requested owner scope. -- Sort candidates deterministically by selected timestamp and path. -- Build a delete plan from `older_than`, `keep_latest`, or both. -- In dry-run mode, report planned deletions and state changes without writing. -- In apply mode, delete only planned managed output paths. -- After successful deletes, remove deleted outputs from state and update state - timestamps. -- If a delete fails, report partial failure and preserve enough state for safe - retry. Prefer removing records only for confirmed deleted outputs. -- Never delete unmanaged storage entries. -- Delete only managed output paths initially. Local and SSH/SFTP backend helpers - may prune empty directories only where they already do so safely. Object - storage prefix markers are preserved unless the prefix marker is itself a - managed output record. - -Prune should operate on output records, not source manifests. If a future state -model groups outputs into logical source publications, group-level pruning can -be added later as an explicit policy. - -## Safety Rules - -Required safety behavior: - -- Prune is opt-in. -- Apply mode is explicit; dry-run is the safe default. -- Delete only paths recorded as managed in `.distributor.json`. -- Do not delete `.distributor.json` unless every managed output in scope is - deleted and a later design explicitly allows removing empty state. -- Do not delete unmanaged files, unknown paths, parent directories, or sibling - prefixes. -- Do not run prune against invalid or ambiguous state. -- Preserve outputs outside the selected owner scope. -- Report all destructive actions with backend and path context. - -## Implementation Work - -Implementation work: - -- Add retention config structs, defaults, parsing, and validation. -- Add duration parsing that supports documented minute/hour/day units. -- Add app-level prune planning and execution use cases. -- Add state helpers for selecting owner-scoped output records, sorting by - timestamp, and removing deleted records. -- Add storage delete support or reuse existing managed-delete helpers. -- Add CLI parsing, help text, dry-run output, and apply output. -- Ensure local, SSH/SFTP, S3, and fake backends preserve bounded deletion - semantics. -- Ensure pruning composes with merge mode and shared-root state. - -## Testing - -Important tests: - -- Config defaults retention disabled. -- Config rejects invalid durations, negative counts, and enabled policies with - no retention rule. -- Dry-run reports deletes without deleting files or rewriting state. -- `older_than` deletes only managed outputs older than the threshold. -- `keep_latest` preserves the newest N outputs and deletes older managed - outputs. -- Combined `older_than` and `keep_latest` behavior is deterministic. -- Shared-root pruning preserves outputs from other owners when scoped to one - owner. -- Unmanaged files under the destination root are never deleted. -- Partial delete failure preserves accurate state for confirmed deleted and - undeleted outputs. -- CLI tests cover required flags, apply gating, and report output. -- Backend tests cover local, SSH/SFTP, S3, and fake deletion behavior used by - prune. - -## Documentation Work - -When implemented, update current-behavior docs in the same change: - -- `docs/config.md`: retention configuration, defaults, and examples. -- `docs/cli.md`: prune command syntax and workflows. -- `docs/operations.md`: pruning safety, dry-run, apply, and recovery behavior. -- `docs/troubleshooting.md`: prune conflicts and partial failures. -- `docs/integrations/destination-state.md`: timestamp fields used by pruning. -- `docs/internal/state.md`, `docs/internal/publish.md`, and - `docs/internal/app.md`: implementation boundaries and invariants. - -## Boundaries - -- Do not let producers choose retention policy. -- Do not use prune as a general storage cleanup tool. -- Do not delete unmanaged content. -- Do not infer retention from filenames unless a later roadmap explicitly adds - path/date parsing. -- Do not depend on backend-native lifecycle policies for correctness; those can - coexist operationally but are not distributor's managed state. -- Do not run pruning automatically after publish in the initial implementation; - pruning should start as an explicit CLI command that operators can schedule. +Pruning remains an operator command rather than a general storage cleanup tool. diff --git a/docs/roadmap/reconcile-state.md b/docs/roadmap/reconcile-state.md index ab97f3c..5cc1bad 100644 --- a/docs/roadmap/reconcile-state.md +++ b/docs/roadmap/reconcile-state.md @@ -1,171 +1,37 @@ # Reconcile-State Roadmap -This document records planned destination state repair tooling that is not part -of the current implementation. Current recovery behavior is documented in -`docs/operations.md`. +Current destination state repair behavior is documented in: -This work should be implemented after the reconciliation work in -`docs/roadmap/reconciliation.md` and the shared-root state work in -`docs/roadmap/multipipeline.md`. Reconcile-state tooling should understand the -final managed state model before prune behavior is added in -`docs/roadmap/prune.md`. +- `docs/cli.md` +- `docs/operations.md` +- `docs/troubleshooting.md` +- `docs/integrations/destination-state.md` +- `docs/internal/app.md` +- `docs/internal/state.md` +- `docs/internal/storage.md` -## Problem +## Current Scope -`.distributor.json` is the managed ownership index for destination outputs. In -long-lived destinations, operators may delete files manually, lifecycle rules -may remove objects, failed external operations may leave state stale, or a -future shared-root destination may accumulate many independently managed -outputs. +Current scope includes: -Without a first-class repair command, operators must inspect and edit state by -hand. That is risky because destination state drives replacement, merge, -conflict, and cleanup decisions. +- `distributor reconcile-state --config --pipeline + --destination `; +- read-only inspection with `--dry-run`; +- default apply behavior when `--dry-run` is omitted; +- shared-root owner-scoped repair; +- shared-root all-owner repair with `--all-owners`; +- text and JSON reports. -## Goal +Reconcile-state removes records for missing managed outputs from valid +destination state. It reports unmanaged entries but does not delete destination +files, adopt unmanaged entries, validate output digests, or repair invalid +state JSON. -Add an operator-facing reconcile-state workflow that compares -`.distributor.json` to actual destination storage and repairs stale managed -state safely. +## Deferred Work -The initial command should focus on removing state records for managed outputs -that no longer exist. It should report unmanaged files, but it should not adopt -them by default. +Deferred state repair work: -## CLI Shape - -Command shape: - -```sh -distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run -distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive -``` - -For shared-root destinations, the command should also support reconciling all -owners in one selected destination root. The final selector should be explicit -enough to avoid relying on destination ids being globally unique: - -```sh -distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --all-owners --dry-run -``` - -The exact flags should follow existing CLI conventions. Dry-run should be the -explicit `--dry-run` flag; without `--dry-run`, the command applies state -repairs. - -## Behavior - -Planned behavior: - -- Load and validate config normally. -- Open the configured destination backend. -- Resolve the destination root or destination bundle path using the same path - mapping semantics as publish where applicable. -- Read and validate `.distributor.json`. -- For each managed output record, check whether the storage object/file exists. -- Remove state records for managed outputs that are missing unless `--dry-run` - is set. -- Preserve state records for managed outputs that exist. -- Report storage entries that exist but are not recorded as managed. -- Preserve unmanaged entries by default. -- Update state timestamps when state is changed. -- Write state atomically where the backend supports atomic writes. -- Never delete storage objects as part of the initial reconcile-state command. -- Do not validate output digests against storage contents in the initial - command; digest verification can be added later as a separate audit mode. - -For current single-owner state, reconciliation removes missing records from the -single `outputs` array. For future shared-root state, reconciliation removes -missing output records across the selected owner scope or all owners, depending -on command flags. - -## Reporting - -Dry-run and apply output should be explicit enough for operators and LLM agents -to understand the result. - -Reports should include: - -- destination backend and root; -- state schema version; -- owner scope, if filtered; -- count of managed outputs checked; -- count and paths of missing managed outputs; -- count and paths of unmanaged existing entries, subject to sensible output - limits; -- whether state would be changed or was changed; -- errors with enough backend/path context to recover safely. - -Machine-readable output should be considered if existing CLI report formats can -support it without broad refactoring. - -## Safety Rules - -Required safety behavior: - -- Do not delete destination files. -- Do not adopt unmanaged destination files by default. -- Do not rewrite invalid state except through a future explicit repair workflow. -- Do not repair state when the destination identity or ownership scope is - ambiguous. -- Do not scan outside the configured backend root or destination root. -- Apply state repairs by default and make `--dry-run` the explicit read-only - inspection mode. -- Keep secrets out of reports and errors. - -## Implementation Work - -Implementation work: - -- Add CLI parsing and help text for the reconcile-state command. -- Add an app-level use case that loads config, opens destinations, and builds a - reconcile-state report. -- Add state package helpers for removing missing outputs without weakening - validation. -- Add storage-facing existence/listing helpers only if current backend - interfaces are insufficient. -- Implement local, SSH/SFTP, S3, and fake backend support needed by the command. -- Preserve shared-root ownership metadata and timestamps when rewriting state. -- Add dry-run and applied-change output in the existing CLI reporting style. - -## Testing - -Important tests: - -- Dry-run reports missing managed outputs and does not write state. -- Apply removes missing managed output records and writes valid state. -- Existing managed outputs are preserved. -- Unmanaged existing files are reported but not adopted or deleted. -- Invalid state fails without rewriting. -- Shared-root state can be reconciled for one owner scope. -- Shared-root state can be reconciled for all owner scopes. -- Local, SSH/SFTP, S3, and fake backend behavior is covered for the existence - and listing operations used. -- CLI tests cover flag validation, default apply behavior, `--dry-run` - behavior, and report output. - -## Documentation Work - -When implemented, update current-behavior docs in the same change: - -- `docs/cli.md`: command syntax, flags, and examples. -- `docs/operations.md`: when and how to run reconcile-state safely. -- `docs/troubleshooting.md`: stale state and missing managed file entries. -- `docs/integrations/destination-state.md`: explain how tools may rewrite - state. -- `docs/internal/state.md`, `docs/internal/storage.md`, and - `docs/internal/app.md`: record implementation boundaries. - -## Boundaries - -- Do not implement prune behavior in this work. -- Do not let reconcile-state infer producer source data or regenerate outputs. -- Do not repair invalid JSON or manually edited state unless a later roadmap - defines an explicit recovery flow. -- Do not claim unmanaged files by default. -- Do not add unmanaged-file adoption or digest validation in the initial - command. -- Do not add whole-config reconciliation in the initial command; require enough - scope to identify one destination root, with all-owner reconciliation allowed - inside a configured shared-root destination. -- Do not expose a producer-facing API for this workflow. +- unmanaged-file adoption workflow; +- digest-audit mode; +- explicit invalid-state repair workflow; +- whole-config repair command. diff --git a/docs/roadmap/reconciliation.md b/docs/roadmap/reconciliation.md index 914c8c6..79f1e55 100644 --- a/docs/roadmap/reconciliation.md +++ b/docs/roadmap/reconciliation.md @@ -1,180 +1,28 @@ # Reconciliation Roadmap -This document records planned destination reconciliation work that is not part -of the current implementation. Current destination replacement, state, and -recovery behavior is documented in `docs/operations.md`, -`docs/config.md`, and `docs/integrations/destination-state.md`. +Current destination reconciliation behavior is documented in: -This work should be implemented before the shared-root multi-pipeline state -work described in `docs/roadmap/multipipeline.md`. The merge mode described -here is still scoped to one pipeline/destination owner for a destination bundle -path; it does not by itself allow multiple pipelines to share one -`.distributor.json`. +- `docs/config.md` +- `docs/operations.md` +- `docs/integrations/destination-state.md` +- `docs/internal/publish.md` +- `docs/internal/state.md` -## Destination Reconciliation Modes +## Current Scope -Current destination replacement behavior treats each publish as the complete -desired managed output set for the destination bundle path. When a newer source -replaces an older destination state, `distributor` deletes prior managed outputs -that are no longer present in the new plan, then writes the new outputs and -state. +Current scope includes destination `reconciliation.mode` values: -That behavior is useful for snapshot-style producers, but it is not ideal for -incremental producers that publish new reports without retaining prior report -state. +- `replace`: the managed output set is reconciled to the newly planned outputs. +- `merge`: omitted managed outputs are retained, and planned paths may overwrite + only paths already recorded as managed. -Planned work: +`replace` is the default. Merge mode does not adopt unmanaged content and does +not let producers choose destination reconciliation policy. -- Add a per-destination reconciliation mode. -- Keep current behavior as the default mode. -- Add a mode that writes new outputs while retaining prior managed outputs that - are not present in the new plan. -- Keep reconciliation mode per destination. Replacement and merge behavior are - destination safety policies, and one pipeline may need both a snapshot-style - `latest` destination and an accumulating `archive` destination from the same - source bundle. +## Deferred Work -Proposed configuration: +No additional reconciliation-specific work is planned here. Related deferred +work is tracked in: -```yaml -destinations: - - id: archive - backend: s3 - bucket: reports - prefix: weather/morning/archive - reconciliation: - mode: merge - - - id: latest - backend: s3 - bucket: reports - prefix: weather/morning/latest - path_mapping: - mode: fixed - reconciliation: - mode: replace -``` - -Accepted modes: - -- `replace`: current behavior. The destination bundle path is reconciled to the - new planned output set. Prior managed outputs not present in the new plan are - removed. -- `merge`: new planned outputs are written into the managed destination set. - Prior managed outputs not present in the new plan are retained. - -## State Model - -Destination state needs to represent cumulative managed outputs for merge mode. -The current state file already records an `outputs` array, but that array -currently represents the outputs from the most recent publication. - -Planned state semantics: - -- `source.manifest` continues to record the latest source manifest used for - destination comparison. -- `outputs` records the complete currently managed output set for the - destination bundle path. -- In `replace` mode, the new state `outputs` are exactly the newly planned - outputs. -- In `merge` mode, the new state `outputs` are: - - previous managed outputs not overwritten by the new plan; - - plus newly planned outputs. -- Persist reconciliation metadata so state interpretation remains explicit after - config changes and so later migration, reconcile-state, and prune tooling can - explain why historical outputs were retained, for example: - -```json -{ - "reconciliation": { - "mode": "merge" - } -} -``` - -If state schema changes are required, bump the destination state schema version -and keep parsing/validation rules explicit. - -## Safety Rules - -Merge mode must not become an unsafe overwrite path. - -Required safety behavior: - -- A new output may overwrite a prior output only when that path is already - recorded as distributor-managed in the existing `.distributor.json`. -- A new output path that exists in storage but is not recorded as managed should - fail as unmanaged content by default. -- Prior managed outputs not included in the new plan are retained in merge mode. -- Prior managed outputs not included in the new plan are deleted in replace - mode. -- Failed-write cleanup should delete only outputs written by the failed attempt, - not retained prior managed outputs. -- Forced replacement behavior remains explicit, bounded, dry-runnable, and - constrained to the destination bundle path. - -## Planning And Execution Work - -Implementation work: - -- Add destination config type `reconciliation.mode`. -- Default `reconciliation.mode` to `replace`. -- Validate mode values as `replace` or `merge`. -- Thread reconciliation mode into publish planning and execution requests. -- During replace execution, preserve current managed cleanup behavior. -- During merge execution: - - inspect existing state outputs before writing; - - allow overwrites only for paths already managed by existing state; - - write new outputs with overwrite enabled only for managed existing paths; - - retain previous managed outputs that are not overwritten; - - write destination state with the cumulative managed output set. -- Ensure generated HTML outputs and copied source outputs use the same - reconciliation semantics. -- Ensure link metadata and primary URL behavior remain deterministic when - retained outputs exist. Derive `links.primary_url` from the newly planned - outputs for the current publish, not from retained historical outputs. - -## Testing - -Important tests: - -- Config defaults `reconciliation.mode` to `replace`. -- Config rejects unknown reconciliation modes. -- Replace mode deletes prior managed outputs omitted from the new bundle. -- Merge mode retains prior managed outputs omitted from the new bundle. -- Merge mode overwrites a path already recorded as managed. -- Merge mode fails when a new output path collides with unmanaged destination - content. -- Merge mode state records the cumulative managed output set. -- Failed merge writes clean up only outputs from the failed attempt. -- Fixed-path destinations support both modes. -- Source-only, generated-HTML-only, and source-plus-HTML publish policies all - honor reconciliation mode. -- S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or - cleanup behavior that changes. - -## Documentation Work - -When implemented, update current-behavior docs in the same change: - -- `docs/config.md`: document `reconciliation.mode`, defaults, and examples. -- `docs/operations.md`: explain replace versus merge behavior and recovery. -- `docs/integrations/destination-state.md`: document any state schema or - semantic changes. -- `docs/internal/publish.md` and related internal docs: record planning and - execution invariants. -- Relevant examples under `examples/`, especially archive/latest style - configurations. - -## Boundaries - -- Do not add lifecycle retention policies such as keep-latest-N or delete older - than a duration as part of this work. -- Do not let producers choose reconciliation mode through source manifests or - upload requests. -- Do not weaken unmanaged-content safety checks. -- Do not make merge mode the default; preserve current replacement behavior for - existing configurations. -- Do not claim unmanaged colliding files in merge mode. Claiming unmanaged - content should remain an explicit force or repair workflow, not normal publish - behavior. +- `docs/roadmap/future.md` +- `docs/roadmap/prune.md`