diff --git a/docs/roadmap/api.md b/docs/roadmap/api.md deleted file mode 100644 index aae84f0..0000000 --- a/docs/roadmap/api.md +++ /dev/null @@ -1,25 +0,0 @@ -# API Roadmap - -This document records API work that is not part of the current implementation. Current HTTP upload behavior is documented in `docs/integrations/http-upload.md`, current configuration behavior is documented in `docs/config.md`, and current producer package usage is documented under `docs/consumers/`. - -## Deferred Upload API Work - -- Durable upload status storage. -- Durable idempotency records across server restarts. -- Run listing, retry, and cancellation endpoints. -- Long-polling or wait-for-completion workflows. -- Multipart, resumable, or streaming upload protocols. -- Additional archive content negotiation beyond tar and gzip-compressed tar. -- URL-token authentication for constrained clients. -- Upload token lifecycle tooling. -- Mutual TLS or other in-app identity mechanisms. -- In-app TLS termination. -- In-app public exposure policy. -- In-app upload rate limiting. - -## Boundaries - -- Producers do not choose destination ids, destination paths, transforms, links, publish policy, transfer policy, or storage backends through upload requests. -- Source manifests remain free of routing, destination, transform, and credential data. -- `http_upload` remains source-only unless a future design changes that contract. -- Public access policy, TLS termination, and rate limiting belong in deployment infrastructure unless a future design changes that boundary. diff --git a/docs/roadmap/future.md b/docs/roadmap/future.md index cb732d9..2fa4044 100644 --- a/docs/roadmap/future.md +++ b/docs/roadmap/future.md @@ -33,6 +33,21 @@ internal docs. - Resumable upload support. - Streaming upload protocols. +## Destination State Repair + +- Unmanaged-file adoption workflow for destination state repair. +- Digest-audit mode for managed destination outputs. +- Explicit invalid-state repair workflow. +- Whole-config state repair command. + +## Destination Retention And Pruning + +- One-off retention overrides for `prune`. +- Automatic post-publish pruning. +- Path/date parsing retention policies. +- Group-level pruning by source publication. +- Removal of empty state files. + ## Destination Backends - GitHub Gist destination backend support. @@ -68,6 +83,7 @@ internal docs. - Mutual TLS or other in-app identity mechanisms. - In-app TLS. - Public exposure defaults. +- In-app upload rate limiting. - Browser UI. ## Boundaries @@ -76,5 +92,12 @@ internal docs. contract. - Current upload status, queue, and idempotency state are memory-only. - Producers submit complete tar or gzip-compressed tar source bundles today. +- Producers do not choose destination ids, destination paths, transforms, links, + publish policy, transfer policy, storage backends, reconciliation mode, state + mode, or retention policy through upload requests. +- Source manifests remain free of routing, destination, transform, credential, + reconciliation, state, and retention data. - Public access policy, TLS termination, and rate limiting belong outside `distributor` unless a future implementation changes that boundary. +- `distributor` is not a broad storage synchronization tool unless a future + implementation explicitly changes that non-goal. diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md deleted file mode 100644 index 18df97b..0000000 --- a/docs/roadmap/implementation.md +++ /dev/null @@ -1,67 +0,0 @@ -# Reconciliation, Shared-Root, Reconcile-State, And Prune Status - -This roadmap file tracks the status and remaining boundaries for the feature -group that includes destination reconciliation, shared-root destination state, -state repair, and managed-output pruning. - -Current behavior is documented in: - -- `docs/config.md` -- `docs/cli.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- `docs/integrations/destination-state.md` -- `docs/internal/app.md` -- `docs/internal/publish.md` -- `docs/internal/state.md` -- `docs/internal/storage.md` - -## Current Scope - -Current scope includes: - -- destination `reconciliation.mode: replace|merge`; -- destination `state.mode: single_owner|shared_root`; -- schema version `2` single-owner destination state; -- schema version `3` shared-root destination state; -- owner-scoped shared-root publishing; -- `reconcile-state` for removing missing managed-output records from valid - destination state; -- destination `retention.prune` policy with `older_than` and `keep_latest`; -- `prune --dry-run` and `prune --apply` for configured managed-output pruning. - -## Current Boundaries - -- Producer manifests and upload requests do not carry destination routing, - reconciliation, state mode, retention, transform, link, or backend policy. -- Reconcile-state repairs state records only. It does not adopt unmanaged - files, validate output digests, repair invalid JSON, or reconcile an entire - config at once. -- Pruning is explicit operator action. It does not run automatically after - publish and does not accept one-off retention overrides. -- Pruning deletes only managed output paths recorded in destination state. It - does not delete unmanaged content or `.distributor.json`. -- Shared-root publishing preserves unrelated owners and rejects implicit - takeover of another owner path. - -## Deferred Work - -Deferred or future work belongs in the other roadmap files under -`docs/roadmap/`. Current deferred items include: - -- unmanaged-file adoption workflow; -- digest-audit mode for state repair; -- whole-config state repair; -- automatic post-publish pruning; -- path/date parsing retention policies; -- producer-facing retention, reconciliation, or destination routing APIs; -- broad storage synchronization behavior. - -## Verification - -Before changing this feature group, run the focused tests for affected -packages and finish with: - -```sh -go test ./... -``` diff --git a/docs/roadmap/multipipeline.md b/docs/roadmap/multipipeline.md deleted file mode 100644 index 15d8c17..0000000 --- a/docs/roadmap/multipipeline.md +++ /dev/null @@ -1,35 +0,0 @@ -# Multi-Pipeline Shared Root Roadmap - -Current shared-root destination state behavior is documented in: - -- `docs/config.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- `docs/integrations/destination-state.md` -- `docs/internal/publish.md` -- `docs/internal/state.md` - -## Current Scope - -Current scope includes `state.mode: shared_root` for destinations that -intentionally share one destination root across multiple pipeline/destination -owners. - -Shared-root state: - -- records owner identity by both `pipeline_id` and `destination_id`; -- records each managed output with its owner and compact source identity; -- preserves unrelated owners during owner-scoped publish, state repair, and - pruning; -- rejects implicit takeover of another owner path; -- treats unmanaged storage content as unmanaged unless explicit forced - replacement is selected where supported. - -## Deferred Work - -No additional shared-root publishing work is planned here. Related deferred work -is tracked in: - -- `docs/roadmap/future.md` -- `docs/roadmap/reconcile-state.md` -- `docs/roadmap/prune.md` diff --git a/docs/roadmap/prune.md b/docs/roadmap/prune.md deleted file mode 100644 index 18d8f51..0000000 --- a/docs/roadmap/prune.md +++ /dev/null @@ -1,45 +0,0 @@ -# Prune Roadmap - -Current managed-output pruning behavior is documented in: - -- `docs/config.md` -- `docs/cli.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- `docs/integrations/destination-state.md` -- `docs/internal/app.md` -- `docs/internal/publish.md` -- `docs/internal/state.md` - -## Current Scope - -Current scope includes: - -- destination `retention.prune.enabled`; -- `retention.prune.older_than`; -- `retention.prune.keep_latest`; -- `distributor prune --dry-run`; -- `distributor prune --apply`; -- text and JSON reports; -- owner-scoped pruning for shared-root state. - -Pruning uses managed output `updated_at` timestamps from destination state. -When both `keep_latest` and `older_than` are configured, the newest -`keep_latest` outputs are preserved before age-based pruning is applied to the -remaining managed outputs. - -Apply mode deletes only planned managed output paths, preserves unmanaged -content, preserves `.distributor.json`, and rewrites state only for confirmed -deletions. - -## Deferred Work - -Deferred pruning work: - -- one-off retention overrides; -- automatic post-publish pruning; -- path/date parsing retention policies; -- group-level pruning by source publication; -- removal of empty state files. - -Pruning remains an operator command rather than a general storage cleanup tool. diff --git a/docs/roadmap/reconcile-state.md b/docs/roadmap/reconcile-state.md deleted file mode 100644 index 5cc1bad..0000000 --- a/docs/roadmap/reconcile-state.md +++ /dev/null @@ -1,37 +0,0 @@ -# Reconcile-State Roadmap - -Current destination state repair behavior is documented in: - -- `docs/cli.md` -- `docs/operations.md` -- `docs/troubleshooting.md` -- `docs/integrations/destination-state.md` -- `docs/internal/app.md` -- `docs/internal/state.md` -- `docs/internal/storage.md` - -## Current Scope - -Current scope includes: - -- `distributor reconcile-state --config --pipeline - --destination `; -- read-only inspection with `--dry-run`; -- default apply behavior when `--dry-run` is omitted; -- shared-root owner-scoped repair; -- shared-root all-owner repair with `--all-owners`; -- text and JSON reports. - -Reconcile-state removes records for missing managed outputs from valid -destination state. It reports unmanaged entries but does not delete destination -files, adopt unmanaged entries, validate output digests, or repair invalid -state JSON. - -## Deferred Work - -Deferred state repair work: - -- unmanaged-file adoption workflow; -- digest-audit mode; -- explicit invalid-state repair workflow; -- whole-config repair command. diff --git a/docs/roadmap/reconciliation.md b/docs/roadmap/reconciliation.md deleted file mode 100644 index 79f1e55..0000000 --- a/docs/roadmap/reconciliation.md +++ /dev/null @@ -1,28 +0,0 @@ -# Reconciliation Roadmap - -Current destination reconciliation behavior is documented in: - -- `docs/config.md` -- `docs/operations.md` -- `docs/integrations/destination-state.md` -- `docs/internal/publish.md` -- `docs/internal/state.md` - -## Current Scope - -Current scope includes destination `reconciliation.mode` values: - -- `replace`: the managed output set is reconciled to the newly planned outputs. -- `merge`: omitted managed outputs are retained, and planned paths may overwrite - only paths already recorded as managed. - -`replace` is the default. Merge mode does not adopt unmanaged content and does -not let producers choose destination reconciliation policy. - -## Deferred Work - -No additional reconciliation-specific work is planned here. Related deferred -work is tracked in: - -- `docs/roadmap/future.md` -- `docs/roadmap/prune.md`