eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Added four new roadmaps related to state management and a corresponding implementation plan

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-08 13:09:00 -05:00
parent ee6a351960
commit 8f3ef33f18
5 changed files with 1295 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

582
docs/roadmap/implementation.md
View File

@@ -1,22 +1,572 @@
# Upload API Implementation Notes
# Reconciliation, Shared-Root, Reconcile-State, And Prune Implementation Plan
This file has no active implementation tasks for the pipeline-scoped upload API.
This document is the staged implementation plan for the feature roadmaps in:
Current behavior is documented in:
- `docs/roadmap/reconciliation.md`
- `docs/roadmap/multipipeline.md`
- `docs/roadmap/reconcile-state.md`
- `docs/roadmap/prune.md`
- `docs/config.md`
- `docs/integrations/http-upload.md`
- `docs/consumers/api.md`
- `docs/consumers/pkg-upload.md`
- `docs/operations.md`
- `docs/troubleshooting.md`
Audience: LLM coding agents and maintainers implementing these features in
order.
Deferred API work is tracked in `docs/roadmap/api.md` and broader deferred work is tracked in `docs/roadmap/future.md`.
This is roadmap guidance, not current behavior documentation. Do not describe
any stage as implemented outside `docs/roadmap/` until that stage has landed.
## Boundaries
## Global Decisions
- Keep producer routing, destination selection, transform policy, publish policy, transfer policy, and backend credentials out of source manifests.
- Keep `pkg/upload` focused on producer-side bundle submission and status polling.
- Keep server-side upload authentication, authorization, queueing, status, and publish orchestration in `internal/app`.
- Keep archive extraction and staged bundle validation in `internal/ingest`.
- Keep durable status, durable idempotency, retry endpoints, cancellation endpoints, and wait helpers out of the current implementation until a new roadmap item defines them.
- Implement in this order:
1. destination reconciliation modes;
2. shared-root multi-pipeline state;
3. reconcile-state tooling;
4. pruning.
- Keep reconciliation mode per destination as `reconciliation.mode`.
- Use reconciliation modes `replace` and `merge`; default is `replace`.
- Persist reconciliation metadata in `.distributor.json`.
- Keep shared-root behavior as an explicit per-destination opt-in using
`state.mode: single_owner|shared_root`.
- Key shared-root output ownership by both `pipeline_id` and `destination_id`.
- Do not allow implicit takeover of another owner path.
- Store compact per-output source identity fields, and keep the latest full
source manifest once per owner where publish comparison needs it.
- `reconcile-state` applies repairs by default and uses `--dry-run` for
read-only inspection.
- `prune` is opt-in, uses configured retention policy, and requires `--apply`
for deletion.
- Use per-output `updated_at` for initial prune policies.
- Support both `older_than` and `keep_latest` pruning in the initial prune
implementation.
- Write single-owner reconciliation-capable state as schema version `2`.
- Write shared-root collection state as schema version `3`.
- Do not adopt unmanaged files, validate output digests, or perform
whole-config reconciliation in the initial reconcile-state command.
- Do not run pruning automatically after publish in the initial implementation.
## Cross-Cutting Constraints
- Follow `docs/policy/architecture.md`, `docs/policy/development.md`, and
`docs/policy/documentation.md`.
- Keep producer manifests free of destination routing, reconciliation, state,
retention, transform, link, and backend policy.
- Keep adapter packages thin. State comparison, reconciliation, pruning, and
retention decisions belong in `internal/state`, `internal/publish`, and
`internal/app`, not concrete backends.
- Keep public packages `pkg/bundle` and `pkg/upload` unaffected unless an
implementation stage explicitly requires consumer-facing docs updates.
- Use dry-run behavior for every workflow that may write state or delete files.
- Never delete unmanaged content.
- Use `storage.StateFileName`, `storage.StatePath`,
`storage.ManagedBundleTargets`, and storage path validation helpers instead
of duplicating path rules.
- Update implemented-behavior docs and examples in the same change as the
corresponding behavior. Keep future behavior in roadmap docs only.
- Run the stage-specific tests before moving to the next stage. Run
`go test ./...` after each completed feature group.
## Stage 1: Reconciliation Config And State Foundation
Goal: add destination-level reconciliation configuration and a state model that
can persist reconciliation metadata and cumulative outputs for single-owner
destinations.
Implementation:
- Add config types:
- `Destination.Reconciliation ReconciliationPolicy`
- `ReconciliationPolicy.Mode string`
- Add constants in `internal/config/defaults.go`:
- `ReconciliationModeReplace = "replace"`
- `ReconciliationModeMerge = "merge"`
- Default `reconciliation.mode` to `replace` for every destination.
- Validate `reconciliation.mode` as `replace` or `merge`.
- Add focused config tests for defaults, explicit `replace`, explicit `merge`,
and invalid modes.
- Extend `internal/state` for a new single-owner state schema that writes:
- `schema_version`;
- `distributor_version`;
- `pipeline_id`;
- `destination_id`;
- `published_at`;
- `created_at`;
- `updated_at`;
- `state.mode: "single_owner"`;
- `reconciliation.mode`;
- `source.manifest`;
- `links`;
- cumulative `outputs`.
- Add per-output `created_at` and `updated_at` fields now. For current
single-owner output records written by this stage, set both timestamps when a
path is first created and preserve `created_at` when a path is updated.
- Keep parsing current schema version 1 state for compatibility. When v1 state
is read, infer:
- `state.mode = "single_owner"`;
- `reconciliation.mode = "replace"`;
- `created_at = published_at`;
- `updated_at = published_at`;
- each output `created_at` and `updated_at` from `published_at`.
- Write only the new schema for newly written state after this stage.
- Keep validation strict: required timestamps, valid reconciliation mode, unique
output paths, valid links, valid digests, clean paths, and valid embedded
source manifest.
- Add state helper functions for:
- finding output records by path;
- merging retained and newly planned output records;
- computing owner-scoped managed output paths for single-owner state;
- projecting publish outputs into state outputs with timestamps.
Tests:
```sh
go test ./internal/config
go test ./internal/state
```
Keep publish semantics unchanged in this stage. Only adapt current state
construction to write schema version `2` with default `replace` metadata and
timestamps.
## Stage 2: Single-Owner Replace And Merge Publish Execution
Goal: implement `reconciliation.mode` for current single-owner destinations.
Implementation:
- Add `Reconciliation config.ReconciliationPolicy` to `publish.Request` and
`publish.Plan`.
- Thread destination reconciliation config from `internal/app` into
`internal/publish`.
- Keep `replace` behavior equivalent to current behavior:
- delete prior managed outputs absent from the new plan;
- write the new outputs;
- write state whose `outputs` are exactly the new planned output set.
- Implement `merge` behavior:
- inspect existing state before writing;
- retain previous managed outputs not overwritten by the new plan;
- allow overwrite only when the destination path is already recorded in
existing state as managed;
- fail if a newly planned path exists in storage but is not recorded as
managed;
- write new or overwritten outputs;
- write state with the cumulative managed output set;
- derive `links.primary_url` from newly planned outputs only;
- clean up only outputs written by the failed attempt.
- For `merge`, update per-output timestamps:
- new output path: `created_at = updated_at = now`;
- overwritten managed path: preserve existing `created_at`, set
`updated_at = now`;
- retained output: preserve both timestamps.
- Ensure state writes use overwrite semantics appropriate for replacing the
existing `.distributor.json`, while output writes remain narrow and explicit.
- Preserve existing force behavior. Do not add unmanaged-content adoption.
Tests:
```sh
go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...
```
Required coverage:
- `replace` deletes omitted managed outputs.
- `merge` retains omitted managed outputs.
- `merge` overwrites an existing managed path.
- `merge` fails on unmanaged destination path collision.
- failed merge cleanup deletes only attempted new writes.
- fixed-path destinations support both modes.
- source-only, generated-only, and source-plus-generated policies honor both
modes.
## Stage 3: Reconciliation Documentation And Examples
Goal: make implemented reconciliation behavior visible outside roadmap docs.
Implementation:
- Update `docs/config.md` with `reconciliation.mode`, defaults, and examples.
- Update `docs/operations.md` with replace versus merge behavior, dry-run
expectations, failed-write cleanup, and recovery notes.
- Update `docs/integrations/destination-state.md` with the new state schema,
reconciliation metadata, output timestamps, and v1 compatibility behavior.
- Update `docs/internal/state.md` and `docs/internal/publish.md`.
- Update relevant examples under `examples/` only if they are valid and
executable for implemented behavior.
Tests:
```sh
go test ./internal/config
go test ./...
```
## Stage 4: Shared-Root State Model And Config
Goal: add explicit shared-root destination state support without changing
publish behavior yet.
Implementation:
- Add destination state config:
- `Destination.State StatePolicy`
- `StatePolicy.Mode string`
- accepted values: `single_owner`, `shared_root`
- default: `single_owner`
- Validate state mode values.
- Add a shared-root state schema in `internal/state`.
- The shared-root JSON must include:
- `schema_version`;
- `distributor_version`;
- `created_at`;
- `updated_at`;
- `state.mode: "shared_root"`;
- owner records keyed by `pipeline_id` and `destination_id`;
- latest full source manifest per owner for comparison;
- owner-level reconciliation metadata;
- owner-level latest `links.primary_url`, if present;
- output records for every managed path in the root.
- Shared-root output records must include:
- path;
- kind;
- source path;
- transform, when generated;
- URL, when present;
- SHA-256;
- size;
- owner `pipeline_id`;
- owner `destination_id`;
- source manifest id;
- source manifest digest;
- source manifest created timestamp;
- `created_at`;
- `updated_at`.
- Keep compact per-output source identity fields. Do not repeat full source
manifests on every output.
- Add state helpers that can:
- parse both single-owner and shared-root state;
- identify the current owner scope;
- return the latest source manifest for one owner;
- list managed paths for one owner or all owners;
- detect path ownership conflicts;
- merge one owner's planned output state while preserving unrelated owners;
- remove one owner's omitted outputs for owner-scoped `replace`.
- Do not automatically convert unrelated single-owner state to shared-root
state. If shared-root is configured and existing single-owner state matches
the current owner, convert it during the first successful shared-root publish.
Otherwise fail with an actionable conflict.
Tests:
```sh
go test ./internal/config
go test ./internal/state
```
Do not enable shared-root publish behavior until Stage 5.
## Stage 5: Shared-Root Publish Planning And Execution
Goal: allow multiple pipelines to publish disjoint managed paths into one
shared destination root.
Implementation:
- Thread destination state mode from `internal/app` into `internal/publish`.
- Update destination inspection and comparison so:
- `single_owner` destinations preserve current single-owner comparison
semantics;
- `shared_root` destinations compare only the current
`pipeline_id`/`destination_id` owner scope;
- absent owner state behaves like destination absent for that owner, unless
unmanaged storage content collides with a planned output;
- outputs owned by other owners are preserved.
- Enforce ownership conflict rules:
- same `pipeline_id` and `destination_id` may overwrite its own managed path;
- different owner writing the same path fails as a conflict;
- unmanaged storage path collision fails by default;
- no implicit takeover.
- Compose reconciliation within the owner scope:
- owner-scoped `replace` removes prior outputs for that owner that are absent
from the new plan;
- owner-scoped `merge` retains prior outputs for that owner that are absent
from the new plan.
- Preserve unrelated owners and their outputs exactly.
- Preserve and update timestamps:
- top-level `created_at` remains the original root creation time;
- top-level `updated_at` changes on successful state writes;
- output `created_at` is stable;
- output `updated_at` changes only when that path is rewritten;
- owner metadata updates when that owner publishes.
- Keep failed-write cleanup limited to outputs written by the failed attempt.
- Keep forced replacement bounded and explicitly reported. Forced replacement
for a shared-root destination may delete the configured destination root, so
retain existing dry-run visibility and do not broaden its scope.
Tests:
```sh
go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...
```
Required coverage:
- two pipelines publish disjoint paths into the same shared root;
- one pipeline replaces its own output without deleting another owner's output;
- one pipeline merges new outputs while preserving its own retained outputs and
other owners' outputs;
- path conflict with another owner fails;
- unmanaged path collision fails;
- shared-root dry-run writes no outputs or state;
- current single-owner behavior remains covered.
## Stage 6: Shared-Root Documentation And Examples
Goal: document implemented shared-root behavior.
Implementation:
- Update `docs/config.md` with destination state mode config and defaults.
- Update `docs/integrations/destination-state.md` with shared-root schema,
owners, output ownership, timestamps, and migration rules.
- Update `docs/operations.md` with shared-root publishing, conflicts, dry-run,
and recovery behavior.
- Update `docs/troubleshooting.md` with ownership conflict and unmanaged
collision entries.
- Update `docs/internal/state.md` and `docs/internal/publish.md`.
- Add maintained examples only when they are valid and load-tested.
Tests:
```sh
go test ./internal/config
go test ./...
```
## Stage 7: Reconcile-State Planning Core
Goal: add app-level planning and state helpers for reconcile-state without
adding CLI execution first.
Implementation:
- Add `internal/app` reconcile-state use case types:
- options with config path, pipeline id, destination id, all-owners flag,
dry-run flag, stdout, and output format if the existing format pattern is
reused;
- report struct with destination identity, backend, root path, state schema,
owner scope, checked count, missing managed outputs, unmanaged entries, and
changed/would-change status.
- Add state helpers for removing missing managed output records:
- single-owner scope;
- shared-root current-owner scope;
- shared-root all-owner scope.
- Use storage `Stat` for managed output existence checks.
- Use bounded `Walk` under the selected destination root to report unmanaged
existing files. Exclude `.distributor.json` and all paths already recorded as
managed.
- Do not validate output digests.
- Do not delete destination files.
- Do not adopt unmanaged files.
- Do not rewrite invalid or ambiguous state.
- Require enough scope to identify one destination root. For current
single-owner state, require pipeline and destination. For shared-root
all-owner mode, still require a pipeline/destination selector to identify the
configured destination root, then apply `--all-owners` inside that root.
- Default behavior applies state repair. `DryRun` reports without writing.
Tests:
```sh
go test ./internal/app ./internal/state ./internal/storage/fake
```
Required coverage:
- dry-run reports missing managed outputs without rewriting state;
- apply removes missing records and writes valid state;
- unmanaged files are reported but not adopted or deleted;
- invalid state fails without rewrite;
- shared-root owner-scope and all-owner repair both work.
## Stage 8: Reconcile-State CLI And Docs
Goal: expose reconcile-state as an operator command.
Implementation:
- Add `reconcile-state` to `internal/cli/root.go` help and dispatch.
- Add CLI flags:
- `--config <path>`;
- `--pipeline <id>`;
- `--destination <id>`;
- `--all-owners`;
- `--dry-run`;
- `--format text|json`.
- Without `--dry-run`, apply state repairs.
- Text output should clearly report whether state was changed or would be
changed.
- JSON output should use the existing app JSON envelope pattern.
- Update `docs/cli.md`, `docs/operations.md`, `docs/troubleshooting.md`,
`docs/integrations/destination-state.md`, `docs/internal/state.md`,
`docs/internal/storage.md`, and `docs/internal/app.md`.
Tests:
```sh
go test ./internal/cli ./internal/app
go test ./...
```
Required coverage:
- flag validation;
- default apply behavior;
- `--dry-run` behavior;
- text and JSON reports;
- no deletion of destination files.
## Stage 9: Prune Config And Planning Core
Goal: add retention config and pure prune planning.
Implementation:
- Add destination retention config:
- `Destination.Retention RetentionPolicy`
- `RetentionPolicy.Prune PrunePolicy`
- `PrunePolicy.Enabled bool`
- `PrunePolicy.OlderThan *Duration`
- `PrunePolicy.KeepLatest *int`
- Default pruning disabled.
- Validate:
- enabled prune policy must set at least one of `older_than` or
`keep_latest`;
- `older_than` must be positive;
- `keep_latest` must be zero or greater;
- documented duration units must parse consistently with existing duration
config behavior.
- Add app/state prune planning helpers:
- select owner-scoped managed outputs;
- sort deterministically by `updated_at`, then path;
- preserve the newest `keep_latest` outputs;
- delete remaining outputs older than `older_than`;
- if both policies are set, first preserve `keep_latest`, then delete
remaining outputs older than `older_than`.
- Use per-output `updated_at` as the timestamp basis.
- Plan only; do not delete storage files in this stage.
Tests:
```sh
go test ./internal/config
go test ./internal/app ./internal/state
```
Required coverage:
- config defaults and validation;
- `older_than` planning;
- `keep_latest` planning;
- combined policy planning;
- deterministic tie-breaking;
- shared-root owner scope preservation.
## Stage 10: Prune Execution, CLI, And Docs
Goal: expose safe managed-output pruning.
Implementation:
- Add app prune execution:
- load config;
- resolve selected destination and owner scope;
- read and validate state;
- build prune plan from configured retention policy;
- in dry-run, write no files and delete nothing;
- in apply mode, delete only planned managed output paths;
- after confirmed deletes, remove deleted records from state and update state
timestamps;
- preserve state records for failed deletes so retry remains accurate;
- never delete unmanaged files;
- do not delete `.distributor.json` unless a later roadmap explicitly permits
empty-state removal.
- Add `prune` CLI dispatch and help.
- Add CLI flags:
- `--config <path>`;
- `--pipeline <id>`;
- `--destination <id>`;
- `--dry-run`;
- `--apply`;
- `--format text|json`.
- Require exactly one of `--dry-run` or `--apply`. `--dry-run` plans and
reports without writing. `--apply` deletes planned managed outputs and
rewrites state after confirmed deletes.
- JSON output should use the existing app JSON envelope pattern.
- Do not add one-off retention overrides in the initial implementation.
- Let local and SSH/SFTP backend helpers prune empty directories only where
they already do so safely. Preserve object-storage prefix markers unless they
are managed output records.
- Update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
`docs/troubleshooting.md`, `docs/integrations/destination-state.md`,
`docs/internal/state.md`, `docs/internal/publish.md`, and
`docs/internal/app.md`.
Tests:
```sh
go test ./internal/config
go test ./internal/app ./internal/cli ./internal/state
go test ./internal/storage/fake
go test ./...
```
Required coverage:
- dry-run reports deletes without deleting files or rewriting state;
- apply deletes only managed output paths;
- unmanaged files under the destination root are preserved;
- partial delete failure preserves accurate state for confirmed deleted and
undeleted outputs;
- shared-root pruning preserves other owners when scoped to one owner;
- CLI apply gating and report output.
## Final Verification
After all stages land:
```sh
go test ./internal/config
go test ./internal/state ./internal/publish
go test ./internal/app ./internal/cli
go test ./internal/storage/fake
go test ./pkg/bundle ./pkg/upload
go test ./...
```
Also run:
```sh
rg -n "reconciliation|shared_root|single_owner|reconcile-state|retention|prune|older_than|keep_latest" README.md docs examples
```
Confirm:
- current-behavior docs outside `docs/roadmap/` describe only implemented
behavior;
- roadmap docs no longer describe completed behavior as future work, or are
revised to mark only remaining deferred work;
- examples load successfully where config tests cover them;
- producer-facing docs still keep destination routing, reconciliation, state,
and retention policy out of producer manifests and upload requests.
## Non-Goals For This Plan
- No unmanaged-file adoption workflow.
- No digest-audit mode for reconcile-state.
- No whole-config reconcile-state command.
- No automatic post-publish pruning.
- No path/date parsing retention policy.
- No producer-facing retention, reconciliation, or destination routing API.
- No broad storage synchronization behavior.