distributor/docs/roadmap/implementation.md

Reconciliation, Shared-Root, Reconcile-State, And Prune Implementation Plan

This document is the staged implementation plan for the feature roadmaps in:

• docs/roadmap/reconciliation.md
• docs/roadmap/multipipeline.md
• docs/roadmap/reconcile-state.md
• docs/roadmap/prune.md

Audience: LLM coding agents and maintainers implementing these features in order.

This is roadmap guidance, not current behavior documentation. Do not describe any stage as implemented outside docs/roadmap/ until that stage has landed.

Global Decisions

• Implement in this order:
  1. destination reconciliation modes;
  2. shared-root multi-pipeline state;
  3. reconcile-state tooling;
  4. pruning.
• Keep reconciliation mode per destination as reconciliation.mode.
• Use reconciliation modes replace and merge; default is replace.
• Persist reconciliation metadata in .distributor.json.
• Keep shared-root behavior as an explicit per-destination opt-in using state.mode: single_owner|shared_root.
• Key shared-root output ownership by both pipeline_id and destination_id.
• Do not allow implicit takeover of another owner path.
• Store compact per-output source identity fields, and keep the latest full source manifest once per owner where publish comparison needs it.
• reconcile-state applies repairs by default and uses --dry-run for read-only inspection.
• prune is opt-in, uses configured retention policy, and requires --apply for deletion.
• Use per-output updated_at for initial prune policies.
• Support both older_than and keep_latest pruning in the initial prune implementation.
• Write single-owner reconciliation-capable state as schema version 2.
• Write shared-root collection state as schema version 3.
• Do not adopt unmanaged files, validate output digests, or perform whole-config reconciliation in the initial reconcile-state command.
• Do not run pruning automatically after publish in the initial implementation.

Cross-Cutting Constraints

• Follow docs/policy/architecture.md, docs/policy/development.md, and docs/policy/documentation.md.
• Keep producer manifests free of destination routing, reconciliation, state, retention, transform, link, and backend policy.
• Keep adapter packages thin. State comparison, reconciliation, pruning, and retention decisions belong in internal/state, internal/publish, and internal/app, not concrete backends.
• Keep public packages pkg/bundle and pkg/upload unaffected unless an implementation stage explicitly requires consumer-facing docs updates.
• Use dry-run behavior for every workflow that may write state or delete files.
• Never delete unmanaged content.
• Use storage.StateFileName, storage.StatePath, storage.ManagedBundleTargets, and storage path validation helpers instead of duplicating path rules.
• Update implemented-behavior docs and examples in the same change as the corresponding behavior. Keep future behavior in roadmap docs only.
• A feature group is not complete until its paired documentation stage lands. Do not move the roadmap item out of future/planned status until the behavior, tests, current-behavior docs, and examples for that group are complete.
• Run the stage-specific tests before moving to the next stage. Run go test ./... after each completed feature group.

Stage 1: Reconciliation Config And State Foundation

Goal: add destination-level reconciliation configuration and a state model that can persist reconciliation metadata and cumulative outputs for single-owner destinations.

Implementation:

• Add config types:
  • Destination.Reconciliation ReconciliationPolicy
  • ReconciliationPolicy.Mode string
• Add constants in internal/config/defaults.go:
  • ReconciliationModeReplace = "replace"
  • ReconciliationModeMerge = "merge"
• Default reconciliation.mode to replace for every destination.
• Validate reconciliation.mode as replace or merge.
• Add focused config tests for defaults, explicit replace, explicit merge, and invalid modes.
• Extend internal/state for a new single-owner state schema that writes:
  • schema_version;
  • distributor_version;
  • pipeline_id;
  • destination_id;
  • published_at;
  • created_at;
  • updated_at;
  • state.mode: "single_owner";
  • reconciliation.mode;
  • source.manifest;
  • links;
  • cumulative outputs.
• Add per-output created_at and updated_at fields now. For current single-owner output records written by this stage, set both timestamps when a path is first created and preserve created_at when a path is updated.
• Keep parsing current schema version 1 state for compatibility. When v1 state is read, infer:
  • state.mode = "single_owner";
  • reconciliation.mode = "replace";
  • created_at = published_at;
  • updated_at = published_at;
  • each output created_at and updated_at from published_at.
• Write only the new schema for newly written state after this stage.
• Keep validation strict: required timestamps, valid reconciliation mode, unique output paths, valid links, valid digests, clean paths, and valid embedded source manifest.
• Add state helper functions for:
  • finding output records by path;
  • merging retained and newly planned output records;
  • computing owner-scoped managed output paths for single-owner state;
  • projecting publish outputs into state outputs with timestamps.

Tests:

go test ./internal/config
go test ./internal/state

Keep publish semantics unchanged in this stage. Only adapt current state construction to write schema version 2 with default replace metadata and timestamps.

Stage 2: Single-Owner Replace And Merge Publish Execution

Goal: implement reconciliation.mode for current single-owner destinations.

Implementation:

• Add Reconciliation config.ReconciliationPolicy to publish.Request and publish.Plan.
• Thread destination reconciliation config from internal/app into internal/publish.
• Keep replace behavior equivalent to current behavior:
  • delete prior managed outputs absent from the new plan;
  • write the new outputs;
  • write state whose outputs are exactly the new planned output set.
• Implement merge behavior:
  • inspect existing state before writing;
  • retain previous managed outputs not overwritten by the new plan;
  • allow overwrite only when the destination path is already recorded in existing state as managed;
  • fail if a newly planned path exists in storage but is not recorded as managed;
  • write new or overwritten outputs;
  • write state with the cumulative managed output set;
  • derive links.primary_url from newly planned outputs only;
  • clean up only outputs written by the failed attempt.
• For merge, update per-output timestamps:
  • new output path: created_at = updated_at = now;
  • overwritten managed path: preserve existing created_at, set updated_at = now;
  • retained output: preserve both timestamps.
• Ensure state writes use overwrite semantics appropriate for replacing the existing .distributor.json, while output writes remain narrow and explicit.
• Preserve existing force behavior. Do not add unmanaged-content adoption.

Tests:

go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...

Required coverage:

• replace deletes omitted managed outputs.
• merge retains omitted managed outputs.
• merge overwrites an existing managed path.
• merge fails on unmanaged destination path collision.
• failed merge cleanup deletes only attempted new writes.
• fixed-path destinations support both modes.
• source-only, generated-only, and source-plus-generated policies honor both modes.

Stage 3: Reconciliation Documentation And Examples

Goal: make implemented reconciliation behavior visible outside roadmap docs. The reconciliation feature group is not complete until this stage lands.

Implementation:

• Update docs/config.md with reconciliation.mode, defaults, and examples.
• Update docs/operations.md with replace versus merge behavior, dry-run expectations, failed-write cleanup, and recovery notes.
• Update docs/integrations/destination-state.md with the new state schema, reconciliation metadata, output timestamps, and v1 compatibility behavior.
• Update docs/internal/state.md and docs/internal/publish.md.
• Update relevant examples under examples/ only if they are valid and executable for implemented behavior.

Tests:

go test ./internal/config
go test ./...

Stage 4: Shared-Root State Model And Config

Goal: add explicit shared-root destination state support without changing publish behavior yet.

Implementation:

• Add destination state config:
  • Destination.State StatePolicy
  • StatePolicy.Mode string
  • accepted values: single_owner, shared_root
  • default: single_owner
• Validate state mode values.
• Add a shared-root state schema in internal/state.
• The shared-root JSON must include:
  • schema_version;
  • distributor_version;
  • created_at;
  • updated_at;
  • state.mode: "shared_root";
  • owner records keyed by pipeline_id and destination_id;
  • latest full source manifest per owner for comparison;
  • owner-level reconciliation metadata;
  • owner-level latest links.primary_url, if present;
  • output records for every managed path in the root.
• Shared-root output records must include:
  • path;
  • kind;
  • source path;
  • transform, when generated;
  • URL, when present;
  • SHA-256;
  • size;
  • owner pipeline_id;
  • owner destination_id;
  • source manifest id;
  • source manifest digest;
  • source manifest created timestamp;
  • created_at;
  • updated_at.
• Keep compact per-output source identity fields. Do not repeat full source manifests on every output.
• Add state helpers that can:
  • parse both single-owner and shared-root state;
  • identify the current owner scope;
  • return the latest source manifest for one owner;
  • list managed paths for one owner or all owners;
  • detect path ownership conflicts;
  • merge one owner's planned output state while preserving unrelated owners;
  • remove one owner's omitted outputs for owner-scoped replace.
• Do not automatically convert unrelated single-owner state to shared-root state. If shared-root is configured and existing single-owner state matches the current owner, convert it during the first successful shared-root publish. Otherwise fail with an actionable conflict.

Tests:

go test ./internal/config
go test ./internal/state

Do not enable shared-root publish behavior until Stage 5.

Stage 5: Shared-Root Publish Planning And Comparison

Goal: teach publish planning to reason about shared-root owner scopes without writing shared-root outputs yet.

Implementation:

• Thread destination state mode from internal/app into internal/publish.
• Update destination inspection and comparison so:
  • single_owner destinations preserve current single-owner comparison semantics;
  • shared_root destinations compare only the current pipeline_id/destination_id owner scope;
  • absent owner state behaves like destination absent for that owner, unless unmanaged storage content collides with a planned output;
  • outputs owned by other owners are preserved.
• Enforce ownership conflict rules:
  • same pipeline_id and destination_id may overwrite its own managed path;
  • different owner writing the same path fails as a conflict;
  • unmanaged storage path collision fails by default;
  • no implicit takeover.
• Compose reconciliation within the owner scope:
  • owner-scoped replace removes prior outputs for that owner that are absent from the new plan;
  • owner-scoped merge retains prior outputs for that owner that are absent from the new plan.
• Return plans that include the current owner scope, retained owner outputs, owner outputs to delete, and owner outputs to write.
• Keep execution disabled for shared-root write actions until Stage 6.

Tests:

go test ./internal/publish ./internal/state
go test ./internal/app

Required coverage:

• planning treats absent owner state as publishable for that owner;
• planning preserves other owners in the planned retained state;
• path conflict with another owner plans or returns a conflict failure;
• unmanaged path collision plans or returns an unmanaged-content failure;
• owner-scoped replace selects only that owner's omitted outputs for deletion;
• owner-scoped merge retains that owner's omitted outputs;
• current single-owner planning remains covered.

Stage 6: Shared-Root Publish Execution

Goal: allow multiple pipelines to publish disjoint managed paths into one shared destination root.

Implementation:

• Preserve unrelated owners and their outputs exactly.
• Preserve and update timestamps:
  • top-level created_at remains the original root creation time;
  • top-level updated_at changes on successful state writes;
  • output created_at is stable;
  • output updated_at changes only when that path is rewritten;
  • owner metadata updates when that owner publishes.
• Keep failed-write cleanup limited to outputs written by the failed attempt.
• Keep forced replacement bounded and explicitly reported. Forced replacement for a shared-root destination may delete the configured destination root, so retain existing dry-run visibility and do not broaden its scope.

Tests:

go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...

Required coverage:

• two pipelines publish disjoint paths into the same shared root;
• one pipeline replaces its own output without deleting another owner's output;
• one pipeline merges new outputs while preserving its own retained outputs and other owners' outputs;
• path conflict with another owner fails;
• unmanaged path collision fails;
• shared-root dry-run writes no outputs or state;
• current single-owner behavior remains covered.

Stage 7: Shared-Root Documentation And Examples

Goal: document implemented shared-root behavior. The shared-root feature group is not complete until this stage lands.

Implementation:

• Update docs/config.md with destination state mode config and defaults.
• Update docs/integrations/destination-state.md with shared-root schema, owners, output ownership, timestamps, and migration rules.
• Update docs/operations.md with shared-root publishing, conflicts, dry-run, and recovery behavior.
• Update docs/troubleshooting.md with ownership conflict and unmanaged collision entries.
• Update docs/internal/state.md and docs/internal/publish.md.
• Add maintained examples only when they are valid and load-tested.

Tests:

go test ./internal/config
go test ./...

Stage 8: Reconcile-State Planning Core

Goal: add app-level planning and state helpers for reconcile-state without adding CLI execution first.

Implementation:

• Add internal/app reconcile-state use case types:
  • options with config path, pipeline id, destination id, all-owners flag, dry-run flag, stdout, and output format;
  • report struct with destination identity, backend, root path, state schema, owner scope, checked count, missing managed outputs, unmanaged entries, and changed/would-change status.
• Add state helpers for removing missing managed output records:
  • single-owner scope;
  • shared-root current-owner scope;
  • shared-root all-owner scope.
• Use storage Stat for managed output existence checks.
• Use bounded Walk under the selected destination root to report unmanaged existing files. Exclude .distributor.json and all paths already recorded as managed.
• Do not validate output digests.
• Do not delete destination files.
• Do not adopt unmanaged files.
• Do not rewrite invalid or ambiguous state.
• Require enough scope to identify one destination root. For current single-owner state, require pipeline and destination. For shared-root all-owner mode, still require a pipeline/destination selector to identify the configured destination root, then apply --all-owners inside that root.
• Default behavior applies state repair. DryRun reports without writing.

Tests:

go test ./internal/app ./internal/state ./internal/storage/fake

Required coverage:

• dry-run reports missing managed outputs without rewriting state;
• apply removes missing records and writes valid state;
• unmanaged files are reported but not adopted or deleted;
• invalid state fails without rewrite;
• shared-root owner-scope and all-owner repair both work.

Stage 9: Reconcile-State CLI And Docs

Goal: expose reconcile-state as an operator command. The reconcile-state feature group is not complete until this stage lands.

Implementation:

• Add reconcile-state to internal/cli/root.go help and dispatch.
• Add CLI flags:
  • --config <path>;
  • --pipeline <id>;
  • --destination <id>;
  • --all-owners;
  • --dry-run;
  • --format text|json.
• Without --dry-run, apply state repairs.
• Text output should clearly report whether state was changed or would be changed.
• JSON output should use the existing app JSON envelope pattern.
• Update docs/cli.md, docs/operations.md, docs/troubleshooting.md, docs/integrations/destination-state.md, docs/internal/state.md, docs/internal/storage.md, and docs/internal/app.md.

Tests:

go test ./internal/cli ./internal/app
go test ./...

Required coverage:

• flag validation;
• default apply behavior;
• --dry-run behavior;
• text and JSON reports;
• no deletion of destination files.

Stage 10: Prune Config And Planning Core

Goal: add retention config and pure prune planning.

Implementation:

• Add destination retention config:
  • Destination.Retention RetentionPolicy
  • RetentionPolicy.Prune PrunePolicy
  • PrunePolicy.Enabled bool
  • PrunePolicy.OlderThan *Duration
  • PrunePolicy.KeepLatest *int
• Default pruning disabled.
• Validate:
  • enabled prune policy must set at least one of older_than or keep_latest;
  • older_than must be positive;
  • keep_latest must be zero or greater;
  • documented duration units must parse consistently with existing duration config behavior.
• Add app/state prune planning helpers:
  • select owner-scoped managed outputs;
  • sort deterministically by updated_at, then path;
  • preserve the newest keep_latest outputs;
  • delete remaining outputs older than older_than;
  • if both policies are set, first preserve keep_latest, then delete remaining outputs older than older_than.
• Use per-output updated_at as the timestamp basis.
• Plan only; do not delete storage files in this stage.

Tests:

go test ./internal/config
go test ./internal/app ./internal/state

Required coverage:

• config defaults and validation;
• older_than planning;
• keep_latest planning;
• combined policy planning;
• deterministic tie-breaking;
• shared-root owner scope preservation.

Stage 11: Prune Execution Core

Goal: implement safe managed-output pruning below the CLI layer.

Implementation:

• Add app prune execution:
  • load config;
  • resolve selected destination and owner scope;
  • read and validate state;
  • build prune plan from configured retention policy;
  • in dry-run, write no files and delete nothing;
  • in apply mode, delete only planned managed output paths;
  • after confirmed deletes, remove deleted records from state and update state timestamps;
  • preserve state records for failed deletes so retry remains accurate;
  • never delete unmanaged files;
  • do not delete .distributor.json unless a later roadmap explicitly permits empty-state removal.
• Do not add CLI dispatch in this stage.
• Return a prune report that can later be rendered by text and JSON CLI output.
• Do not add one-off retention overrides in the initial implementation.
• Let local and SSH/SFTP backend helpers prune empty directories only where they already do so safely. Preserve object-storage prefix markers unless they are managed output records.

Tests:

go test ./internal/config
go test ./internal/app ./internal/state
go test ./internal/storage/fake

Required coverage:

• dry-run reports deletes without deleting files or rewriting state;
• apply deletes only managed output paths;
• unmanaged files under the destination root are preserved;
• partial delete failure preserves accurate state for confirmed deleted and undeleted outputs;
• shared-root pruning preserves other owners when scoped to one owner.

Stage 12: Prune CLI And Docs

Goal: expose safe managed-output pruning as an operator command. The prune feature group is not complete until this stage lands.

Implementation:

• Add prune CLI dispatch and help.
• Add CLI flags:
  • --config <path>;
  • --pipeline <id>;
  • --destination <id>;
  • --dry-run;
  • --apply;
  • --format text|json.
• Require exactly one of --dry-run or --apply. --dry-run plans and reports without writing. --apply deletes planned managed outputs and rewrites state after confirmed deletes.
• JSON output should use the existing app JSON envelope pattern.
• Update docs/config.md, docs/cli.md, docs/operations.md, docs/troubleshooting.md, docs/integrations/destination-state.md, docs/internal/state.md, docs/internal/publish.md, and docs/internal/app.md.

Tests:

go test ./internal/config
go test ./internal/app ./internal/cli ./internal/state
go test ./internal/storage/fake
go test ./...

Required coverage:

• CLI requires exactly one of --dry-run or --apply;
• text and JSON reports;
• CLI apply gating and report output.

Final Verification

After all stages land:

go test ./internal/config
go test ./internal/state ./internal/publish
go test ./internal/app ./internal/cli
go test ./internal/storage/fake
go test ./pkg/bundle ./pkg/upload
go test ./...

Also run:

rg -n "reconciliation|shared_root|single_owner|reconcile-state|retention|prune|older_than|keep_latest" README.md docs examples

Confirm:

• current-behavior docs outside docs/roadmap/ describe only implemented behavior;
• roadmap docs no longer describe completed behavior as future work, or are revised to mark only remaining deferred work;
• examples load successfully where config tests cover them;
• producer-facing docs still keep destination routing, reconciliation, state, and retention policy out of producer manifests and upload requests.

Non-Goals For This Plan

• No unmanaged-file adoption workflow.
• No digest-audit mode for reconcile-state.
• No whole-config reconcile-state command.
• No automatic post-publish pruning.
• No path/date parsing retention policy.
• No producer-facing retention, reconciliation, or destination routing API.
• No broad storage synchronization behavior.