eric/distributor
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 7 Wiki Activity

Added four new roadmaps related to state management and a corresponding implementation plan

Browse Source
This commit is contained in:
Eric Rakestraw
2026-06-08 13:09:00 -05:00
parent ee6a351960
commit 8f3ef33f18
5 changed files with 1295 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files

582
docs/roadmap/implementation.md
View File

@@ -1,22 +1,572 @@
# Upload API Implementation Notes
# Reconciliation, Shared-Root, Reconcile-State, And Prune Implementation Plan
This file has no active implementation tasks for the pipeline-scoped upload API.
This document is the staged implementation plan for the feature roadmaps in:
Current behavior is documented in:
- `docs/roadmap/reconciliation.md`
- `docs/roadmap/multipipeline.md`
- `docs/roadmap/reconcile-state.md`
- `docs/roadmap/prune.md`
- `docs/config.md`
- `docs/integrations/http-upload.md`
- `docs/consumers/api.md`
- `docs/consumers/pkg-upload.md`
- `docs/operations.md`
- `docs/troubleshooting.md`
Audience: LLM coding agents and maintainers implementing these features in
order.
Deferred API work is tracked in `docs/roadmap/api.md` and broader deferred work is tracked in `docs/roadmap/future.md`.
This is roadmap guidance, not current behavior documentation. Do not describe
any stage as implemented outside `docs/roadmap/` until that stage has landed.
## Boundaries
## Global Decisions
- Keep producer routing, destination selection, transform policy, publish policy, transfer policy, and backend credentials out of source manifests.
- Keep `pkg/upload` focused on producer-side bundle submission and status polling.
- Keep server-side upload authentication, authorization, queueing, status, and publish orchestration in `internal/app`.
- Keep archive extraction and staged bundle validation in `internal/ingest`.
- Keep durable status, durable idempotency, retry endpoints, cancellation endpoints, and wait helpers out of the current implementation until a new roadmap item defines them.
- Implement in this order:
1. destination reconciliation modes;
2. shared-root multi-pipeline state;
3. reconcile-state tooling;
4. pruning.
- Keep reconciliation mode per destination as `reconciliation.mode`.
- Use reconciliation modes `replace` and `merge`; default is `replace`.
- Persist reconciliation metadata in `.distributor.json`.
- Keep shared-root behavior as an explicit per-destination opt-in using
`state.mode: single_owner|shared_root`.
- Key shared-root output ownership by both `pipeline_id` and `destination_id`.
- Do not allow implicit takeover of another owner path.
- Store compact per-output source identity fields, and keep the latest full
source manifest once per owner where publish comparison needs it.
- `reconcile-state` applies repairs by default and uses `--dry-run` for
read-only inspection.
- `prune` is opt-in, uses configured retention policy, and requires `--apply`
for deletion.
- Use per-output `updated_at` for initial prune policies.
- Support both `older_than` and `keep_latest` pruning in the initial prune
implementation.
- Write single-owner reconciliation-capable state as schema version `2`.
- Write shared-root collection state as schema version `3`.
- Do not adopt unmanaged files, validate output digests, or perform
whole-config reconciliation in the initial reconcile-state command.
- Do not run pruning automatically after publish in the initial implementation.
## Cross-Cutting Constraints
- Follow `docs/policy/architecture.md`, `docs/policy/development.md`, and
`docs/policy/documentation.md`.
- Keep producer manifests free of destination routing, reconciliation, state,
retention, transform, link, and backend policy.
- Keep adapter packages thin. State comparison, reconciliation, pruning, and
retention decisions belong in `internal/state`, `internal/publish`, and
`internal/app`, not concrete backends.
- Keep public packages `pkg/bundle` and `pkg/upload` unaffected unless an
implementation stage explicitly requires consumer-facing docs updates.
- Use dry-run behavior for every workflow that may write state or delete files.
- Never delete unmanaged content.
- Use `storage.StateFileName`, `storage.StatePath`,
`storage.ManagedBundleTargets`, and storage path validation helpers instead
of duplicating path rules.
- Update implemented-behavior docs and examples in the same change as the
corresponding behavior. Keep future behavior in roadmap docs only.
- Run the stage-specific tests before moving to the next stage. Run
`go test ./...` after each completed feature group.
## Stage 1: Reconciliation Config And State Foundation
Goal: add destination-level reconciliation configuration and a state model that
can persist reconciliation metadata and cumulative outputs for single-owner
destinations.
Implementation:
- Add config types:
- `Destination.Reconciliation ReconciliationPolicy`
- `ReconciliationPolicy.Mode string`
- Add constants in `internal/config/defaults.go`:
- `ReconciliationModeReplace = "replace"`
- `ReconciliationModeMerge = "merge"`
- Default `reconciliation.mode` to `replace` for every destination.
- Validate `reconciliation.mode` as `replace` or `merge`.
- Add focused config tests for defaults, explicit `replace`, explicit `merge`,
and invalid modes.
- Extend `internal/state` for a new single-owner state schema that writes:
- `schema_version`;
- `distributor_version`;
- `pipeline_id`;
- `destination_id`;
- `published_at`;
- `created_at`;
- `updated_at`;
- `state.mode: "single_owner"`;
- `reconciliation.mode`;
- `source.manifest`;
- `links`;
- cumulative `outputs`.
- Add per-output `created_at` and `updated_at` fields now. For current
single-owner output records written by this stage, set both timestamps when a
path is first created and preserve `created_at` when a path is updated.
- Keep parsing current schema version 1 state for compatibility. When v1 state
is read, infer:
- `state.mode = "single_owner"`;
- `reconciliation.mode = "replace"`;
- `created_at = published_at`;
- `updated_at = published_at`;
- each output `created_at` and `updated_at` from `published_at`.
- Write only the new schema for newly written state after this stage.
- Keep validation strict: required timestamps, valid reconciliation mode, unique
output paths, valid links, valid digests, clean paths, and valid embedded
source manifest.
- Add state helper functions for:
- finding output records by path;
- merging retained and newly planned output records;
- computing owner-scoped managed output paths for single-owner state;
- projecting publish outputs into state outputs with timestamps.
Tests:
```sh
go test ./internal/config
go test ./internal/state
```
Keep publish semantics unchanged in this stage. Only adapt current state
construction to write schema version `2` with default `replace` metadata and
timestamps.
## Stage 2: Single-Owner Replace And Merge Publish Execution
Goal: implement `reconciliation.mode` for current single-owner destinations.
Implementation:
- Add `Reconciliation config.ReconciliationPolicy` to `publish.Request` and
`publish.Plan`.
- Thread destination reconciliation config from `internal/app` into
`internal/publish`.
- Keep `replace` behavior equivalent to current behavior:
- delete prior managed outputs absent from the new plan;
- write the new outputs;
- write state whose `outputs` are exactly the new planned output set.
- Implement `merge` behavior:
- inspect existing state before writing;
- retain previous managed outputs not overwritten by the new plan;
- allow overwrite only when the destination path is already recorded in
existing state as managed;
- fail if a newly planned path exists in storage but is not recorded as
managed;
- write new or overwritten outputs;
- write state with the cumulative managed output set;
- derive `links.primary_url` from newly planned outputs only;
- clean up only outputs written by the failed attempt.
- For `merge`, update per-output timestamps:
- new output path: `created_at = updated_at = now`;
- overwritten managed path: preserve existing `created_at`, set
`updated_at = now`;
- retained output: preserve both timestamps.
- Ensure state writes use overwrite semantics appropriate for replacing the
existing `.distributor.json`, while output writes remain narrow and explicit.
- Preserve existing force behavior. Do not add unmanaged-content adoption.
Tests:
```sh
go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...
```
Required coverage:
- `replace` deletes omitted managed outputs.
- `merge` retains omitted managed outputs.
- `merge` overwrites an existing managed path.
- `merge` fails on unmanaged destination path collision.
- failed merge cleanup deletes only attempted new writes.
- fixed-path destinations support both modes.
- source-only, generated-only, and source-plus-generated policies honor both
modes.
## Stage 3: Reconciliation Documentation And Examples
Goal: make implemented reconciliation behavior visible outside roadmap docs.
Implementation:
- Update `docs/config.md` with `reconciliation.mode`, defaults, and examples.
- Update `docs/operations.md` with replace versus merge behavior, dry-run
expectations, failed-write cleanup, and recovery notes.
- Update `docs/integrations/destination-state.md` with the new state schema,
reconciliation metadata, output timestamps, and v1 compatibility behavior.
- Update `docs/internal/state.md` and `docs/internal/publish.md`.
- Update relevant examples under `examples/` only if they are valid and
executable for implemented behavior.
Tests:
```sh
go test ./internal/config
go test ./...
```
## Stage 4: Shared-Root State Model And Config
Goal: add explicit shared-root destination state support without changing
publish behavior yet.
Implementation:
- Add destination state config:
- `Destination.State StatePolicy`
- `StatePolicy.Mode string`
- accepted values: `single_owner`, `shared_root`
- default: `single_owner`
- Validate state mode values.
- Add a shared-root state schema in `internal/state`.
- The shared-root JSON must include:
- `schema_version`;
- `distributor_version`;
- `created_at`;
- `updated_at`;
- `state.mode: "shared_root"`;
- owner records keyed by `pipeline_id` and `destination_id`;
- latest full source manifest per owner for comparison;
- owner-level reconciliation metadata;
- owner-level latest `links.primary_url`, if present;
- output records for every managed path in the root.
- Shared-root output records must include:
- path;
- kind;
- source path;
- transform, when generated;
- URL, when present;
- SHA-256;
- size;
- owner `pipeline_id`;
- owner `destination_id`;
- source manifest id;
- source manifest digest;
- source manifest created timestamp;
- `created_at`;
- `updated_at`.
- Keep compact per-output source identity fields. Do not repeat full source
manifests on every output.
- Add state helpers that can:
- parse both single-owner and shared-root state;
- identify the current owner scope;
- return the latest source manifest for one owner;
- list managed paths for one owner or all owners;
- detect path ownership conflicts;
- merge one owner's planned output state while preserving unrelated owners;
- remove one owner's omitted outputs for owner-scoped `replace`.
- Do not automatically convert unrelated single-owner state to shared-root
state. If shared-root is configured and existing single-owner state matches
the current owner, convert it during the first successful shared-root publish.
Otherwise fail with an actionable conflict.
Tests:
```sh
go test ./internal/config
go test ./internal/state
```
Do not enable shared-root publish behavior until Stage 5.
## Stage 5: Shared-Root Publish Planning And Execution
Goal: allow multiple pipelines to publish disjoint managed paths into one
shared destination root.
Implementation:
- Thread destination state mode from `internal/app` into `internal/publish`.
- Update destination inspection and comparison so:
- `single_owner` destinations preserve current single-owner comparison
semantics;
- `shared_root` destinations compare only the current
`pipeline_id`/`destination_id` owner scope;
- absent owner state behaves like destination absent for that owner, unless
unmanaged storage content collides with a planned output;
- outputs owned by other owners are preserved.
- Enforce ownership conflict rules:
- same `pipeline_id` and `destination_id` may overwrite its own managed path;
- different owner writing the same path fails as a conflict;
- unmanaged storage path collision fails by default;
- no implicit takeover.
- Compose reconciliation within the owner scope:
- owner-scoped `replace` removes prior outputs for that owner that are absent
from the new plan;
- owner-scoped `merge` retains prior outputs for that owner that are absent
from the new plan.
- Preserve unrelated owners and their outputs exactly.
- Preserve and update timestamps:
- top-level `created_at` remains the original root creation time;
- top-level `updated_at` changes on successful state writes;
- output `created_at` is stable;
- output `updated_at` changes only when that path is rewritten;
- owner metadata updates when that owner publishes.
- Keep failed-write cleanup limited to outputs written by the failed attempt.
- Keep forced replacement bounded and explicitly reported. Forced replacement
for a shared-root destination may delete the configured destination root, so
retain existing dry-run visibility and do not broaden its scope.
Tests:
```sh
go test ./internal/publish ./internal/state
go test ./internal/app ./internal/cli
go test ./...
```
Required coverage:
- two pipelines publish disjoint paths into the same shared root;
- one pipeline replaces its own output without deleting another owner's output;
- one pipeline merges new outputs while preserving its own retained outputs and
other owners' outputs;
- path conflict with another owner fails;
- unmanaged path collision fails;
- shared-root dry-run writes no outputs or state;
- current single-owner behavior remains covered.
## Stage 6: Shared-Root Documentation And Examples
Goal: document implemented shared-root behavior.
Implementation:
- Update `docs/config.md` with destination state mode config and defaults.
- Update `docs/integrations/destination-state.md` with shared-root schema,
owners, output ownership, timestamps, and migration rules.
- Update `docs/operations.md` with shared-root publishing, conflicts, dry-run,
and recovery behavior.
- Update `docs/troubleshooting.md` with ownership conflict and unmanaged
collision entries.
- Update `docs/internal/state.md` and `docs/internal/publish.md`.
- Add maintained examples only when they are valid and load-tested.
Tests:
```sh
go test ./internal/config
go test ./...
```
## Stage 7: Reconcile-State Planning Core
Goal: add app-level planning and state helpers for reconcile-state without
adding CLI execution first.
Implementation:
- Add `internal/app` reconcile-state use case types:
- options with config path, pipeline id, destination id, all-owners flag,
dry-run flag, stdout, and output format if the existing format pattern is
reused;
- report struct with destination identity, backend, root path, state schema,
owner scope, checked count, missing managed outputs, unmanaged entries, and
changed/would-change status.
- Add state helpers for removing missing managed output records:
- single-owner scope;
- shared-root current-owner scope;
- shared-root all-owner scope.
- Use storage `Stat` for managed output existence checks.
- Use bounded `Walk` under the selected destination root to report unmanaged
existing files. Exclude `.distributor.json` and all paths already recorded as
managed.
- Do not validate output digests.
- Do not delete destination files.
- Do not adopt unmanaged files.
- Do not rewrite invalid or ambiguous state.
- Require enough scope to identify one destination root. For current
single-owner state, require pipeline and destination. For shared-root
all-owner mode, still require a pipeline/destination selector to identify the
configured destination root, then apply `--all-owners` inside that root.
- Default behavior applies state repair. `DryRun` reports without writing.
Tests:
```sh
go test ./internal/app ./internal/state ./internal/storage/fake
```
Required coverage:
- dry-run reports missing managed outputs without rewriting state;
- apply removes missing records and writes valid state;
- unmanaged files are reported but not adopted or deleted;
- invalid state fails without rewrite;
- shared-root owner-scope and all-owner repair both work.
## Stage 8: Reconcile-State CLI And Docs
Goal: expose reconcile-state as an operator command.
Implementation:
- Add `reconcile-state` to `internal/cli/root.go` help and dispatch.
- Add CLI flags:
- `--config <path>`;
- `--pipeline <id>`;
- `--destination <id>`;
- `--all-owners`;
- `--dry-run`;
- `--format text|json`.
- Without `--dry-run`, apply state repairs.
- Text output should clearly report whether state was changed or would be
changed.
- JSON output should use the existing app JSON envelope pattern.
- Update `docs/cli.md`, `docs/operations.md`, `docs/troubleshooting.md`,
`docs/integrations/destination-state.md`, `docs/internal/state.md`,
`docs/internal/storage.md`, and `docs/internal/app.md`.
Tests:
```sh
go test ./internal/cli ./internal/app
go test ./...
```
Required coverage:
- flag validation;
- default apply behavior;
- `--dry-run` behavior;
- text and JSON reports;
- no deletion of destination files.
## Stage 9: Prune Config And Planning Core
Goal: add retention config and pure prune planning.
Implementation:
- Add destination retention config:
- `Destination.Retention RetentionPolicy`
- `RetentionPolicy.Prune PrunePolicy`
- `PrunePolicy.Enabled bool`
- `PrunePolicy.OlderThan *Duration`
- `PrunePolicy.KeepLatest *int`
- Default pruning disabled.
- Validate:
- enabled prune policy must set at least one of `older_than` or
`keep_latest`;
- `older_than` must be positive;
- `keep_latest` must be zero or greater;
- documented duration units must parse consistently with existing duration
config behavior.
- Add app/state prune planning helpers:
- select owner-scoped managed outputs;
- sort deterministically by `updated_at`, then path;
- preserve the newest `keep_latest` outputs;
- delete remaining outputs older than `older_than`;
- if both policies are set, first preserve `keep_latest`, then delete
remaining outputs older than `older_than`.
- Use per-output `updated_at` as the timestamp basis.
- Plan only; do not delete storage files in this stage.
Tests:
```sh
go test ./internal/config
go test ./internal/app ./internal/state
```
Required coverage:
- config defaults and validation;
- `older_than` planning;
- `keep_latest` planning;
- combined policy planning;
- deterministic tie-breaking;
- shared-root owner scope preservation.
## Stage 10: Prune Execution, CLI, And Docs
Goal: expose safe managed-output pruning.
Implementation:
- Add app prune execution:
- load config;
- resolve selected destination and owner scope;
- read and validate state;
- build prune plan from configured retention policy;
- in dry-run, write no files and delete nothing;
- in apply mode, delete only planned managed output paths;
- after confirmed deletes, remove deleted records from state and update state
timestamps;
- preserve state records for failed deletes so retry remains accurate;
- never delete unmanaged files;
- do not delete `.distributor.json` unless a later roadmap explicitly permits
empty-state removal.
- Add `prune` CLI dispatch and help.
- Add CLI flags:
- `--config <path>`;
- `--pipeline <id>`;
- `--destination <id>`;
- `--dry-run`;
- `--apply`;
- `--format text|json`.
- Require exactly one of `--dry-run` or `--apply`. `--dry-run` plans and
reports without writing. `--apply` deletes planned managed outputs and
rewrites state after confirmed deletes.
- JSON output should use the existing app JSON envelope pattern.
- Do not add one-off retention overrides in the initial implementation.
- Let local and SSH/SFTP backend helpers prune empty directories only where
they already do so safely. Preserve object-storage prefix markers unless they
are managed output records.
- Update `docs/config.md`, `docs/cli.md`, `docs/operations.md`,
`docs/troubleshooting.md`, `docs/integrations/destination-state.md`,
`docs/internal/state.md`, `docs/internal/publish.md`, and
`docs/internal/app.md`.
Tests:
```sh
go test ./internal/config
go test ./internal/app ./internal/cli ./internal/state
go test ./internal/storage/fake
go test ./...
```
Required coverage:
- dry-run reports deletes without deleting files or rewriting state;
- apply deletes only managed output paths;
- unmanaged files under the destination root are preserved;
- partial delete failure preserves accurate state for confirmed deleted and
undeleted outputs;
- shared-root pruning preserves other owners when scoped to one owner;
- CLI apply gating and report output.
## Final Verification
After all stages land:
```sh
go test ./internal/config
go test ./internal/state ./internal/publish
go test ./internal/app ./internal/cli
go test ./internal/storage/fake
go test ./pkg/bundle ./pkg/upload
go test ./...
```
Also run:
```sh
rg -n "reconciliation|shared_root|single_owner|reconcile-state|retention|prune|older_than|keep_latest" README.md docs examples
```
Confirm:
- current-behavior docs outside `docs/roadmap/` describe only implemented
behavior;
- roadmap docs no longer describe completed behavior as future work, or are
revised to mark only remaining deferred work;
- examples load successfully where config tests cover them;
- producer-facing docs still keep destination routing, reconciliation, state,
and retention policy out of producer manifests and upload requests.
## Non-Goals For This Plan
- No unmanaged-file adoption workflow.
- No digest-audit mode for reconcile-state.
- No whole-config reconcile-state command.
- No automatic post-publish pruning.
- No path/date parsing retention policy.
- No producer-facing retention, reconciliation, or destination routing API.
- No broad storage synchronization behavior.

195
docs/roadmap/multipipeline.md Normal file
View File

@@ -0,0 +1,195 @@
# Multi-Pipeline Shared Root Roadmap
This document records planned shared-root destination state work that is not
part of the current implementation. Current destination state behavior is
documented in `docs/integrations/destination-state.md`.
This work should not be implemented until the reconciliation work in
`docs/roadmap/reconciliation.md` is present and complete. Shared-root
multi-pipeline publishing should then be implemented before reconcile-state
tooling (`docs/roadmap/reconcile-state.md`) and pruning
(`docs/roadmap/prune.md`) because both tools should operate on the final shared
state model.
## Problem
Current `.distributor.json` state is a single-owner destination bundle record.
It stores one `pipeline_id`, one `destination_id`, one latest source manifest,
and one output set. That works when each pipeline writes to a separate
destination bundle path.
It does not work well when multiple pipelines intentionally publish into one
shared destination root, for example:
```text
2026-06-08/daily/report.md
2026-06-08/event/watch-123.md
```
If both pipelines resolve to the same destination bundle path, the second
pipeline encounters a state identity mismatch even when its output paths are
disjoint. Separating roots avoids the conflict, but can make retention,
cleanup, linking, and human browsing less natural.
## Goal
Support multiple configured pipelines writing to one shared destination root
while preserving distributor ownership, unmanaged-content safety, dry-run
behavior, and deterministic conflict handling.
The desired shape is a collection-oriented `.distributor.json` that records
managed outputs from multiple owners under one state file. Producers still
submit source bundles only. Producers must not choose destinations,
reconciliation mode, retention policy, transforms, links, or storage backends.
Shared-root behavior should be an explicit per-destination state mode, for
example `state.mode: single_owner|shared_root` or a similarly named field under
the destination. Shared-root behavior changes the meaning of `.distributor.json`,
so it should be visible at the destination boundary rather than inferred from
other configured pipelines.
## State Model
Add a new destination state schema that can represent a managed collection
rather than one latest source bundle.
State shape:
- top-level `schema_version`;
- top-level `distributor_version`;
- top-level `created_at` for the first time this managed root was created;
- top-level `updated_at` for the last successful state update;
- top-level collection metadata identifying the state as shared-root capable;
- an `outputs` array containing all managed output records in the shared root;
- per-output owner metadata:
- `pipeline_id`;
- `destination_id`;
- source manifest id;
- source manifest digest;
- source manifest created timestamp;
- output kind, source path, transform, URL, SHA-256, and size;
- `created_at` for first publication of that output path;
- `updated_at` for the most recent write of that output path.
- optional source-manifest history keyed by source digest, if compact per-output
source identity fields are not enough for later diagnostics or tooling.
Output ownership is keyed by both `pipeline_id` and `destination_id`. The
architecture treats destinations as independent policy boundaries, and the same
pipeline may publish different output sets to different destinations.
The state should be able to answer these questions without scanning the whole
destination:
- Which paths are distributor-managed?
- Which pipeline/destination/source currently owns each managed path?
- Which outputs belong to a specific pipeline/destination publish operation?
- Which outputs are old enough, or least recently updated enough, for later
pruning decisions?
- Which paths are absent from storage and should be removed by reconcile-state
tooling?
## Ownership And Conflict Rules
Shared-root support must keep ownership explicit.
Required rules:
- A publish may write only output paths planned from its configured pipeline and
destination.
- A publish may overwrite an existing managed output path only when ownership
rules allow it.
- A new output path that exists in storage but is not recorded in
`.distributor.json` remains unmanaged content and fails by default.
- Disjoint outputs from different pipelines can coexist in one state file.
- State updates must preserve unrelated managed outputs from other pipelines.
- Failed-write cleanup must delete only paths written by the failed attempt.
- Forced replacement must remain explicit, dry-runnable, and bounded to the
configured destination root.
Overwrite rule: an output path may be overwritten by the same `pipeline_id` and
`destination_id`; a different pipeline or destination writing the same path
fails as an ownership conflict unless a later explicit operator takeover
workflow is implemented.
## Planning And Execution Work
Implementation work:
- Introduce a new state schema and parser that can represent shared ownership.
- Keep current schema parsing available long enough to migrate or reject old
state with clear errors.
- Add explicit per-destination configuration that opts a destination into
shared-root behavior.
- Thread shared-root state semantics through destination inspection,
comparison, publish planning, and execution.
- Make comparison operate on the subset of state owned by the current
pipeline/destination where appropriate.
- Make output collision detection check both newly planned outputs and existing
managed outputs from other owners.
- Write state atomically with preserved unrelated owners and updated records for
the current publish.
- Preserve merge and replace semantics within the current owner scope:
- owner-scoped `replace` removes prior outputs for that owner that are absent
from the new plan;
- owner-scoped `merge` retains prior outputs for that owner that are absent
from the new plan.
- Ensure local, SSH/SFTP, S3, and fake backends support the necessary state read,
write, overwrite, and cleanup paths.
## Migration And Compatibility
The initial implementation should prefer explicit safety over automatic
conversion.
Migration behavior:
- A destination configured for shared-root behavior may read current single-owner
state and convert it only when the existing state identity matches the current
pipeline/destination or when an explicit migration command/flag is used.
- A destination not configured for shared-root behavior continues to use the
current single-owner state semantics.
- Invalid or ambiguous state fails with an actionable error and a dry-run report
path.
## Testing
Important tests:
- Two pipelines publish disjoint paths into the same destination root.
- One pipeline can replace its own older output without deleting another
pipeline's output.
- One pipeline can merge new outputs while retaining its prior outputs and
another pipeline's outputs.
- A pipeline cannot silently overwrite another pipeline's managed path.
- A pipeline cannot claim unmanaged storage content by default.
- State timestamps are set on creation and updated on subsequent writes.
- Per-output timestamps preserve original `created_at` and update `updated_at`
only when that output path is rewritten.
- Failed writes clean up only attempted outputs.
- Dry-run reports shared-root actions without writing outputs or state.
- Forced replacement remains bounded and explicitly reported.
- Current single-owner state behavior remains covered for non-shared
destinations.
## Documentation Work
When implemented, update current-behavior docs in the same change:
- `docs/integrations/destination-state.md`: document the new schema and
ownership rules.
- `docs/config.md`: document the shared-root opt-in configuration.
- `docs/operations.md`: explain shared-root publishing, conflicts, and recovery.
- `docs/troubleshooting.md`: add shared-root ownership conflict entries.
- `docs/internal/state.md` and `docs/internal/publish.md`: record the new
planning and state invariants.
- Maintained examples that demonstrate date-first shared roots only after the
behavior is implemented.
## Boundaries
- Do not let producers route to destinations or choose shared-root behavior.
- Do not add prune behavior in this work.
- Do not add broad unmanaged-content adoption in this work.
- Do not make shared-root behavior implicit for all destinations.
- Do not use storage listings as the primary source of truth during normal
publishing; `.distributor.json` remains the managed ownership index.

183
docs/roadmap/prune.md Normal file
View File

@@ -0,0 +1,183 @@
# Prune Roadmap
This document records planned destination pruning work that is not part of the
current implementation. Current cleanup and forced replacement behavior is
documented in `docs/operations.md`.
This work should be implemented after the reconciliation work in
`docs/roadmap/reconciliation.md`, shared-root state work in
`docs/roadmap/multipipeline.md`, and reconcile-state tooling in
`docs/roadmap/reconcile-state.md`. Prune decisions should rely on accurate
managed state, explicit ownership, and timestamp metadata.
## Problem
Merge reconciliation and shared-root publishing allow long-lived destinations
to accumulate distributor-managed outputs. That is useful for archives and
date-first report layouts, but those destinations can grow without bound.
Operators need a safe way to delete old distributor-managed outputs according
to explicit retention policy without touching unmanaged files or outputs owned
by unrelated scopes.
## Goal
Add pruning support for managed destination outputs.
Initial pruning should support both of these policy families:
- keep only the most recently updated N files in a configured scope;
- delete managed files older than a configured duration.
Pruning must be based on `.distributor.json` ownership records. It should never
delete files merely because they exist under a configured prefix.
## Policy Model
Configuration shape:
```yaml
destinations:
- id: archive
backend: s3
bucket: reports
prefix: weather
reconciliation:
mode: merge
retention:
prune:
enabled: true
older_than: 90d
keep_latest: 500
```
Policy semantics:
- `enabled` defaults to `false`.
- `older_than` deletes managed outputs whose selected timestamp is older than
the duration.
- `keep_latest` keeps the most recent N managed outputs in scope and deletes
older managed outputs beyond that count.
- If both are set, preserve outputs that must be kept by `keep_latest`, then
delete remaining outputs older than `older_than`.
- Retention scope is the configured destination owner scope unless
shared-root configuration explicitly chooses a broader scope.
Timestamp basis: use per-output `updated_at` for both policies, with
`created_at` available for future created-time retention. Updated-time
retention best matches recurring reports where a stable path is overwritten.
## CLI Shape
Command shape:
```sh
distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --apply
```
The command should use configured retention policy by default. A later explicit
override mode can allow one-off retention values if that is useful, but the
initial implementation should avoid ad hoc deletion flags that bypass reviewed
config.
## Behavior
Planned behavior:
- Load and validate config normally.
- Resolve the selected destination and retention policy.
- Read and validate `.distributor.json`.
- Select managed output records in the requested owner scope.
- Sort candidates deterministically by selected timestamp and path.
- Build a delete plan from `older_than`, `keep_latest`, or both.
- In dry-run mode, report planned deletions and state changes without writing.
- In apply mode, delete only planned managed output paths.
- After successful deletes, remove deleted outputs from state and update state
timestamps.
- If a delete fails, report partial failure and preserve enough state for safe
retry. Prefer removing records only for confirmed deleted outputs.
- Never delete unmanaged storage entries.
- Delete only managed output paths initially. Local and SSH/SFTP backend helpers
may prune empty directories only where they already do so safely. Object
storage prefix markers are preserved unless the prefix marker is itself a
managed output record.
Prune should operate on output records, not source manifests. If a future state
model groups outputs into logical source publications, group-level pruning can
be added later as an explicit policy.
## Safety Rules
Required safety behavior:
- Prune is opt-in.
- Apply mode is explicit; dry-run is the safe default.
- Delete only paths recorded as managed in `.distributor.json`.
- Do not delete `.distributor.json` unless every managed output in scope is
deleted and a later design explicitly allows removing empty state.
- Do not delete unmanaged files, unknown paths, parent directories, or sibling
prefixes.
- Do not run prune against invalid or ambiguous state.
- Preserve outputs outside the selected owner scope.
- Report all destructive actions with backend and path context.
## Implementation Work
Implementation work:
- Add retention config structs, defaults, parsing, and validation.
- Add duration parsing that supports documented minute/hour/day units.
- Add app-level prune planning and execution use cases.
- Add state helpers for selecting owner-scoped output records, sorting by
timestamp, and removing deleted records.
- Add storage delete support or reuse existing managed-delete helpers.
- Add CLI parsing, help text, dry-run output, and apply output.
- Ensure local, SSH/SFTP, S3, and fake backends preserve bounded deletion
semantics.
- Ensure pruning composes with merge mode and shared-root state.
## Testing
Important tests:
- Config defaults retention disabled.
- Config rejects invalid durations, negative counts, and enabled policies with
no retention rule.
- Dry-run reports deletes without deleting files or rewriting state.
- `older_than` deletes only managed outputs older than the threshold.
- `keep_latest` preserves the newest N outputs and deletes older managed
outputs.
- Combined `older_than` and `keep_latest` behavior is deterministic.
- Shared-root pruning preserves outputs from other owners when scoped to one
owner.
- Unmanaged files under the destination root are never deleted.
- Partial delete failure preserves accurate state for confirmed deleted and
undeleted outputs.
- CLI tests cover required flags, apply gating, and report output.
- Backend tests cover local, SSH/SFTP, S3, and fake deletion behavior used by
prune.
## Documentation Work
When implemented, update current-behavior docs in the same change:
- `docs/config.md`: retention configuration, defaults, and examples.
- `docs/cli.md`: prune command syntax and workflows.
- `docs/operations.md`: pruning safety, dry-run, apply, and recovery behavior.
- `docs/troubleshooting.md`: prune conflicts and partial failures.
- `docs/integrations/destination-state.md`: timestamp fields used by pruning.
- `docs/internal/state.md`, `docs/internal/publish.md`, and
`docs/internal/app.md`: implementation boundaries and invariants.
## Boundaries
- Do not let producers choose retention policy.
- Do not use prune as a general storage cleanup tool.
- Do not delete unmanaged content.
- Do not infer retention from filenames unless a later roadmap explicitly adds
path/date parsing.
- Do not depend on backend-native lifecycle policies for correctness; those can
coexist operationally but are not distributor's managed state.
- Do not run pruning automatically after publish in the initial implementation;
pruning should start as an explicit CLI command that operators can schedule.

171
docs/roadmap/reconcile-state.md Normal file
View File

@@ -0,0 +1,171 @@
# Reconcile-State Roadmap
This document records planned destination state repair tooling that is not part
of the current implementation. Current recovery behavior is documented in
`docs/operations.md`.
This work should be implemented after the reconciliation work in
`docs/roadmap/reconciliation.md` and the shared-root state work in
`docs/roadmap/multipipeline.md`. Reconcile-state tooling should understand the
final managed state model before prune behavior is added in
`docs/roadmap/prune.md`.
## Problem
`.distributor.json` is the managed ownership index for destination outputs. In
long-lived destinations, operators may delete files manually, lifecycle rules
may remove objects, failed external operations may leave state stale, or a
future shared-root destination may accumulate many independently managed
outputs.
Without a first-class repair command, operators must inspect and edit state by
hand. That is risky because destination state drives replacement, merge,
conflict, and cleanup decisions.
## Goal
Add an operator-facing reconcile-state workflow that compares
`.distributor.json` to actual destination storage and repairs stale managed
state safely.
The initial command should focus on removing state records for managed outputs
that no longer exist. It should report unmanaged files, but it should not adopt
them by default.
## CLI Shape
Command shape:
```sh
distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run
distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive
```
For shared-root destinations, the command should also support reconciling all
owners in one selected destination root. The final selector should be explicit
enough to avoid relying on destination ids being globally unique:
```sh
distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --all-owners --dry-run
```
The exact flags should follow existing CLI conventions. Dry-run should be the
explicit `--dry-run` flag; without `--dry-run`, the command applies state
repairs.
## Behavior
Planned behavior:
- Load and validate config normally.
- Open the configured destination backend.
- Resolve the destination root or destination bundle path using the same path
mapping semantics as publish where applicable.
- Read and validate `.distributor.json`.
- For each managed output record, check whether the storage object/file exists.
- Remove state records for managed outputs that are missing unless `--dry-run`
is set.
- Preserve state records for managed outputs that exist.
- Report storage entries that exist but are not recorded as managed.
- Preserve unmanaged entries by default.
- Update state timestamps when state is changed.
- Write state atomically where the backend supports atomic writes.
- Never delete storage objects as part of the initial reconcile-state command.
- Do not validate output digests against storage contents in the initial
command; digest verification can be added later as a separate audit mode.
For current single-owner state, reconciliation removes missing records from the
single `outputs` array. For future shared-root state, reconciliation removes
missing output records across the selected owner scope or all owners, depending
on command flags.
## Reporting
Dry-run and apply output should be explicit enough for operators and LLM agents
to understand the result.
Reports should include:
- destination backend and root;
- state schema version;
- owner scope, if filtered;
- count of managed outputs checked;
- count and paths of missing managed outputs;
- count and paths of unmanaged existing entries, subject to sensible output
limits;
- whether state would be changed or was changed;
- errors with enough backend/path context to recover safely.
Machine-readable output should be considered if existing CLI report formats can
support it without broad refactoring.
## Safety Rules
Required safety behavior:
- Do not delete destination files.
- Do not adopt unmanaged destination files by default.
- Do not rewrite invalid state except through a future explicit repair workflow.
- Do not repair state when the destination identity or ownership scope is
ambiguous.
- Do not scan outside the configured backend root or destination root.
- Apply state repairs by default and make `--dry-run` the explicit read-only
inspection mode.
- Keep secrets out of reports and errors.
## Implementation Work
Implementation work:
- Add CLI parsing and help text for the reconcile-state command.
- Add an app-level use case that loads config, opens destinations, and builds a
reconcile-state report.
- Add state package helpers for removing missing outputs without weakening
validation.
- Add storage-facing existence/listing helpers only if current backend
interfaces are insufficient.
- Implement local, SSH/SFTP, S3, and fake backend support needed by the command.
- Preserve shared-root ownership metadata and timestamps when rewriting state.
- Add dry-run and applied-change output in the existing CLI reporting style.
## Testing
Important tests:
- Dry-run reports missing managed outputs and does not write state.
- Apply removes missing managed output records and writes valid state.
- Existing managed outputs are preserved.
- Unmanaged existing files are reported but not adopted or deleted.
- Invalid state fails without rewriting.
- Shared-root state can be reconciled for one owner scope.
- Shared-root state can be reconciled for all owner scopes.
- Local, SSH/SFTP, S3, and fake backend behavior is covered for the existence
and listing operations used.
- CLI tests cover flag validation, default apply behavior, `--dry-run`
behavior, and report output.
## Documentation Work
When implemented, update current-behavior docs in the same change:
- `docs/cli.md`: command syntax, flags, and examples.
- `docs/operations.md`: when and how to run reconcile-state safely.
- `docs/troubleshooting.md`: stale state and missing managed file entries.
- `docs/integrations/destination-state.md`: explain how tools may rewrite
state.
- `docs/internal/state.md`, `docs/internal/storage.md`, and
`docs/internal/app.md`: record implementation boundaries.
## Boundaries
- Do not implement prune behavior in this work.
- Do not let reconcile-state infer producer source data or regenerate outputs.
- Do not repair invalid JSON or manually edited state unless a later roadmap
defines an explicit recovery flow.
- Do not claim unmanaged files by default.
- Do not add unmanaged-file adoption or digest validation in the initial
command.
- Do not add whole-config reconciliation in the initial command; require enough
scope to identify one destination root, with all-owner reconciliation allowed
inside a configured shared-root destination.
- Do not expose a producer-facing API for this workflow.

180
docs/roadmap/reconciliation.md Normal file
View File

@@ -0,0 +1,180 @@
# Reconciliation Roadmap
This document records planned destination reconciliation work that is not part
of the current implementation. Current destination replacement, state, and
recovery behavior is documented in `docs/operations.md`,
`docs/config.md`, and `docs/integrations/destination-state.md`.
This work should be implemented before the shared-root multi-pipeline state
work described in `docs/roadmap/multipipeline.md`. The merge mode described
here is still scoped to one pipeline/destination owner for a destination bundle
path; it does not by itself allow multiple pipelines to share one
`.distributor.json`.
## Destination Reconciliation Modes
Current destination replacement behavior treats each publish as the complete
desired managed output set for the destination bundle path. When a newer source
replaces an older destination state, `distributor` deletes prior managed outputs
that are no longer present in the new plan, then writes the new outputs and
state.
That behavior is useful for snapshot-style producers, but it is not ideal for
incremental producers that publish new reports without retaining prior report
state.
Planned work:
- Add a per-destination reconciliation mode.
- Keep current behavior as the default mode.
- Add a mode that writes new outputs while retaining prior managed outputs that
are not present in the new plan.
- Keep reconciliation mode per destination. Replacement and merge behavior are
destination safety policies, and one pipeline may need both a snapshot-style
`latest` destination and an accumulating `archive` destination from the same
source bundle.
Proposed configuration:
```yaml
destinations:
- id: archive
backend: s3
bucket: reports
prefix: weather/morning/archive
reconciliation:
mode: merge
- id: latest
backend: s3
bucket: reports
prefix: weather/morning/latest
path_mapping:
mode: fixed
reconciliation:
mode: replace
```
Accepted modes:
- `replace`: current behavior. The destination bundle path is reconciled to the
new planned output set. Prior managed outputs not present in the new plan are
removed.
- `merge`: new planned outputs are written into the managed destination set.
Prior managed outputs not present in the new plan are retained.
## State Model
Destination state needs to represent cumulative managed outputs for merge mode.
The current state file already records an `outputs` array, but that array
currently represents the outputs from the most recent publication.
Planned state semantics:
- `source.manifest` continues to record the latest source manifest used for
destination comparison.
- `outputs` records the complete currently managed output set for the
destination bundle path.
- In `replace` mode, the new state `outputs` are exactly the newly planned
outputs.
- In `merge` mode, the new state `outputs` are:
- previous managed outputs not overwritten by the new plan;
- plus newly planned outputs.
- Persist reconciliation metadata so state interpretation remains explicit after
config changes and so later migration, reconcile-state, and prune tooling can
explain why historical outputs were retained, for example:
```json
{
"reconciliation": {
"mode": "merge"
}
}
```
If state schema changes are required, bump the destination state schema version
and keep parsing/validation rules explicit.
## Safety Rules
Merge mode must not become an unsafe overwrite path.
Required safety behavior:
- A new output may overwrite a prior output only when that path is already
recorded as distributor-managed in the existing `.distributor.json`.
- A new output path that exists in storage but is not recorded as managed should
fail as unmanaged content by default.
- Prior managed outputs not included in the new plan are retained in merge mode.
- Prior managed outputs not included in the new plan are deleted in replace
mode.
- Failed-write cleanup should delete only outputs written by the failed attempt,
not retained prior managed outputs.
- Forced replacement behavior remains explicit, bounded, dry-runnable, and
constrained to the destination bundle path.
## Planning And Execution Work
Implementation work:
- Add destination config type `reconciliation.mode`.
- Default `reconciliation.mode` to `replace`.
- Validate mode values as `replace` or `merge`.
- Thread reconciliation mode into publish planning and execution requests.
- During replace execution, preserve current managed cleanup behavior.
- During merge execution:
- inspect existing state outputs before writing;
- allow overwrites only for paths already managed by existing state;
- write new outputs with overwrite enabled only for managed existing paths;
- retain previous managed outputs that are not overwritten;
- write destination state with the cumulative managed output set.
- Ensure generated HTML outputs and copied source outputs use the same
reconciliation semantics.
- Ensure link metadata and primary URL behavior remain deterministic when
retained outputs exist. Derive `links.primary_url` from the newly planned
outputs for the current publish, not from retained historical outputs.
## Testing
Important tests:
- Config defaults `reconciliation.mode` to `replace`.
- Config rejects unknown reconciliation modes.
- Replace mode deletes prior managed outputs omitted from the new bundle.
- Merge mode retains prior managed outputs omitted from the new bundle.
- Merge mode overwrites a path already recorded as managed.
- Merge mode fails when a new output path collides with unmanaged destination
content.
- Merge mode state records the cumulative managed output set.
- Failed merge writes clean up only outputs from the failed attempt.
- Fixed-path destinations support both modes.
- Source-only, generated-HTML-only, and source-plus-HTML publish policies all
honor reconciliation mode.
- S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or
cleanup behavior that changes.
## Documentation Work
When implemented, update current-behavior docs in the same change:
- `docs/config.md`: document `reconciliation.mode`, defaults, and examples.
- `docs/operations.md`: explain replace versus merge behavior and recovery.
- `docs/integrations/destination-state.md`: document any state schema or
semantic changes.
- `docs/internal/publish.md` and related internal docs: record planning and
execution invariants.
- Relevant examples under `examples/`, especially archive/latest style
configurations.
## Boundaries
- Do not add lifecycle retention policies such as keep-latest-N or delete older
than a duration as part of this work.
- Do not let producers choose reconciliation mode through source manifests or
upload requests.
- Do not weaken unmanaged-content safety checks.
- Do not make merge mode the default; preserve current replacement behavior for
existing configurations.
- Do not claim unmanaged colliding files in merge mode. Claiming unmanaged
content should remain an explicit force or repair workflow, not normal publish
behavior.