diff --git a/docs/roadmap/implementation.md b/docs/roadmap/implementation.md index fcb65f3..6a6af51 100644 --- a/docs/roadmap/implementation.md +++ b/docs/roadmap/implementation.md @@ -1,22 +1,572 @@ -# Upload API Implementation Notes +# Reconciliation, Shared-Root, Reconcile-State, And Prune Implementation Plan -This file has no active implementation tasks for the pipeline-scoped upload API. +This document is the staged implementation plan for the feature roadmaps in: -Current behavior is documented in: +- `docs/roadmap/reconciliation.md` +- `docs/roadmap/multipipeline.md` +- `docs/roadmap/reconcile-state.md` +- `docs/roadmap/prune.md` -- `docs/config.md` -- `docs/integrations/http-upload.md` -- `docs/consumers/api.md` -- `docs/consumers/pkg-upload.md` -- `docs/operations.md` -- `docs/troubleshooting.md` +Audience: LLM coding agents and maintainers implementing these features in +order. -Deferred API work is tracked in `docs/roadmap/api.md` and broader deferred work is tracked in `docs/roadmap/future.md`. +This is roadmap guidance, not current behavior documentation. Do not describe +any stage as implemented outside `docs/roadmap/` until that stage has landed. -## Boundaries +## Global Decisions -- Keep producer routing, destination selection, transform policy, publish policy, transfer policy, and backend credentials out of source manifests. -- Keep `pkg/upload` focused on producer-side bundle submission and status polling. -- Keep server-side upload authentication, authorization, queueing, status, and publish orchestration in `internal/app`. -- Keep archive extraction and staged bundle validation in `internal/ingest`. -- Keep durable status, durable idempotency, retry endpoints, cancellation endpoints, and wait helpers out of the current implementation until a new roadmap item defines them. +- Implement in this order: + 1. destination reconciliation modes; + 2. shared-root multi-pipeline state; + 3. reconcile-state tooling; + 4. pruning. +- Keep reconciliation mode per destination as `reconciliation.mode`. +- Use reconciliation modes `replace` and `merge`; default is `replace`. +- Persist reconciliation metadata in `.distributor.json`. +- Keep shared-root behavior as an explicit per-destination opt-in using + `state.mode: single_owner|shared_root`. +- Key shared-root output ownership by both `pipeline_id` and `destination_id`. +- Do not allow implicit takeover of another owner path. +- Store compact per-output source identity fields, and keep the latest full + source manifest once per owner where publish comparison needs it. +- `reconcile-state` applies repairs by default and uses `--dry-run` for + read-only inspection. +- `prune` is opt-in, uses configured retention policy, and requires `--apply` + for deletion. +- Use per-output `updated_at` for initial prune policies. +- Support both `older_than` and `keep_latest` pruning in the initial prune + implementation. +- Write single-owner reconciliation-capable state as schema version `2`. +- Write shared-root collection state as schema version `3`. +- Do not adopt unmanaged files, validate output digests, or perform + whole-config reconciliation in the initial reconcile-state command. +- Do not run pruning automatically after publish in the initial implementation. + +## Cross-Cutting Constraints + +- Follow `docs/policy/architecture.md`, `docs/policy/development.md`, and + `docs/policy/documentation.md`. +- Keep producer manifests free of destination routing, reconciliation, state, + retention, transform, link, and backend policy. +- Keep adapter packages thin. State comparison, reconciliation, pruning, and + retention decisions belong in `internal/state`, `internal/publish`, and + `internal/app`, not concrete backends. +- Keep public packages `pkg/bundle` and `pkg/upload` unaffected unless an + implementation stage explicitly requires consumer-facing docs updates. +- Use dry-run behavior for every workflow that may write state or delete files. +- Never delete unmanaged content. +- Use `storage.StateFileName`, `storage.StatePath`, + `storage.ManagedBundleTargets`, and storage path validation helpers instead + of duplicating path rules. +- Update implemented-behavior docs and examples in the same change as the + corresponding behavior. Keep future behavior in roadmap docs only. +- Run the stage-specific tests before moving to the next stage. Run + `go test ./...` after each completed feature group. + +## Stage 1: Reconciliation Config And State Foundation + +Goal: add destination-level reconciliation configuration and a state model that +can persist reconciliation metadata and cumulative outputs for single-owner +destinations. + +Implementation: + +- Add config types: + - `Destination.Reconciliation ReconciliationPolicy` + - `ReconciliationPolicy.Mode string` +- Add constants in `internal/config/defaults.go`: + - `ReconciliationModeReplace = "replace"` + - `ReconciliationModeMerge = "merge"` +- Default `reconciliation.mode` to `replace` for every destination. +- Validate `reconciliation.mode` as `replace` or `merge`. +- Add focused config tests for defaults, explicit `replace`, explicit `merge`, + and invalid modes. +- Extend `internal/state` for a new single-owner state schema that writes: + - `schema_version`; + - `distributor_version`; + - `pipeline_id`; + - `destination_id`; + - `published_at`; + - `created_at`; + - `updated_at`; + - `state.mode: "single_owner"`; + - `reconciliation.mode`; + - `source.manifest`; + - `links`; + - cumulative `outputs`. +- Add per-output `created_at` and `updated_at` fields now. For current + single-owner output records written by this stage, set both timestamps when a + path is first created and preserve `created_at` when a path is updated. +- Keep parsing current schema version 1 state for compatibility. When v1 state + is read, infer: + - `state.mode = "single_owner"`; + - `reconciliation.mode = "replace"`; + - `created_at = published_at`; + - `updated_at = published_at`; + - each output `created_at` and `updated_at` from `published_at`. +- Write only the new schema for newly written state after this stage. +- Keep validation strict: required timestamps, valid reconciliation mode, unique + output paths, valid links, valid digests, clean paths, and valid embedded + source manifest. +- Add state helper functions for: + - finding output records by path; + - merging retained and newly planned output records; + - computing owner-scoped managed output paths for single-owner state; + - projecting publish outputs into state outputs with timestamps. + +Tests: + +```sh +go test ./internal/config +go test ./internal/state +``` + +Keep publish semantics unchanged in this stage. Only adapt current state +construction to write schema version `2` with default `replace` metadata and +timestamps. + +## Stage 2: Single-Owner Replace And Merge Publish Execution + +Goal: implement `reconciliation.mode` for current single-owner destinations. + +Implementation: + +- Add `Reconciliation config.ReconciliationPolicy` to `publish.Request` and + `publish.Plan`. +- Thread destination reconciliation config from `internal/app` into + `internal/publish`. +- Keep `replace` behavior equivalent to current behavior: + - delete prior managed outputs absent from the new plan; + - write the new outputs; + - write state whose `outputs` are exactly the new planned output set. +- Implement `merge` behavior: + - inspect existing state before writing; + - retain previous managed outputs not overwritten by the new plan; + - allow overwrite only when the destination path is already recorded in + existing state as managed; + - fail if a newly planned path exists in storage but is not recorded as + managed; + - write new or overwritten outputs; + - write state with the cumulative managed output set; + - derive `links.primary_url` from newly planned outputs only; + - clean up only outputs written by the failed attempt. +- For `merge`, update per-output timestamps: + - new output path: `created_at = updated_at = now`; + - overwritten managed path: preserve existing `created_at`, set + `updated_at = now`; + - retained output: preserve both timestamps. +- Ensure state writes use overwrite semantics appropriate for replacing the + existing `.distributor.json`, while output writes remain narrow and explicit. +- Preserve existing force behavior. Do not add unmanaged-content adoption. + +Tests: + +```sh +go test ./internal/publish ./internal/state +go test ./internal/app ./internal/cli +go test ./... +``` + +Required coverage: + +- `replace` deletes omitted managed outputs. +- `merge` retains omitted managed outputs. +- `merge` overwrites an existing managed path. +- `merge` fails on unmanaged destination path collision. +- failed merge cleanup deletes only attempted new writes. +- fixed-path destinations support both modes. +- source-only, generated-only, and source-plus-generated policies honor both + modes. + +## Stage 3: Reconciliation Documentation And Examples + +Goal: make implemented reconciliation behavior visible outside roadmap docs. + +Implementation: + +- Update `docs/config.md` with `reconciliation.mode`, defaults, and examples. +- Update `docs/operations.md` with replace versus merge behavior, dry-run + expectations, failed-write cleanup, and recovery notes. +- Update `docs/integrations/destination-state.md` with the new state schema, + reconciliation metadata, output timestamps, and v1 compatibility behavior. +- Update `docs/internal/state.md` and `docs/internal/publish.md`. +- Update relevant examples under `examples/` only if they are valid and + executable for implemented behavior. + +Tests: + +```sh +go test ./internal/config +go test ./... +``` + +## Stage 4: Shared-Root State Model And Config + +Goal: add explicit shared-root destination state support without changing +publish behavior yet. + +Implementation: + +- Add destination state config: + - `Destination.State StatePolicy` + - `StatePolicy.Mode string` + - accepted values: `single_owner`, `shared_root` + - default: `single_owner` +- Validate state mode values. +- Add a shared-root state schema in `internal/state`. +- The shared-root JSON must include: + - `schema_version`; + - `distributor_version`; + - `created_at`; + - `updated_at`; + - `state.mode: "shared_root"`; + - owner records keyed by `pipeline_id` and `destination_id`; + - latest full source manifest per owner for comparison; + - owner-level reconciliation metadata; + - owner-level latest `links.primary_url`, if present; + - output records for every managed path in the root. +- Shared-root output records must include: + - path; + - kind; + - source path; + - transform, when generated; + - URL, when present; + - SHA-256; + - size; + - owner `pipeline_id`; + - owner `destination_id`; + - source manifest id; + - source manifest digest; + - source manifest created timestamp; + - `created_at`; + - `updated_at`. +- Keep compact per-output source identity fields. Do not repeat full source + manifests on every output. +- Add state helpers that can: + - parse both single-owner and shared-root state; + - identify the current owner scope; + - return the latest source manifest for one owner; + - list managed paths for one owner or all owners; + - detect path ownership conflicts; + - merge one owner's planned output state while preserving unrelated owners; + - remove one owner's omitted outputs for owner-scoped `replace`. +- Do not automatically convert unrelated single-owner state to shared-root + state. If shared-root is configured and existing single-owner state matches + the current owner, convert it during the first successful shared-root publish. + Otherwise fail with an actionable conflict. + +Tests: + +```sh +go test ./internal/config +go test ./internal/state +``` + +Do not enable shared-root publish behavior until Stage 5. + +## Stage 5: Shared-Root Publish Planning And Execution + +Goal: allow multiple pipelines to publish disjoint managed paths into one +shared destination root. + +Implementation: + +- Thread destination state mode from `internal/app` into `internal/publish`. +- Update destination inspection and comparison so: + - `single_owner` destinations preserve current single-owner comparison + semantics; + - `shared_root` destinations compare only the current + `pipeline_id`/`destination_id` owner scope; + - absent owner state behaves like destination absent for that owner, unless + unmanaged storage content collides with a planned output; + - outputs owned by other owners are preserved. +- Enforce ownership conflict rules: + - same `pipeline_id` and `destination_id` may overwrite its own managed path; + - different owner writing the same path fails as a conflict; + - unmanaged storage path collision fails by default; + - no implicit takeover. +- Compose reconciliation within the owner scope: + - owner-scoped `replace` removes prior outputs for that owner that are absent + from the new plan; + - owner-scoped `merge` retains prior outputs for that owner that are absent + from the new plan. +- Preserve unrelated owners and their outputs exactly. +- Preserve and update timestamps: + - top-level `created_at` remains the original root creation time; + - top-level `updated_at` changes on successful state writes; + - output `created_at` is stable; + - output `updated_at` changes only when that path is rewritten; + - owner metadata updates when that owner publishes. +- Keep failed-write cleanup limited to outputs written by the failed attempt. +- Keep forced replacement bounded and explicitly reported. Forced replacement + for a shared-root destination may delete the configured destination root, so + retain existing dry-run visibility and do not broaden its scope. + +Tests: + +```sh +go test ./internal/publish ./internal/state +go test ./internal/app ./internal/cli +go test ./... +``` + +Required coverage: + +- two pipelines publish disjoint paths into the same shared root; +- one pipeline replaces its own output without deleting another owner's output; +- one pipeline merges new outputs while preserving its own retained outputs and + other owners' outputs; +- path conflict with another owner fails; +- unmanaged path collision fails; +- shared-root dry-run writes no outputs or state; +- current single-owner behavior remains covered. + +## Stage 6: Shared-Root Documentation And Examples + +Goal: document implemented shared-root behavior. + +Implementation: + +- Update `docs/config.md` with destination state mode config and defaults. +- Update `docs/integrations/destination-state.md` with shared-root schema, + owners, output ownership, timestamps, and migration rules. +- Update `docs/operations.md` with shared-root publishing, conflicts, dry-run, + and recovery behavior. +- Update `docs/troubleshooting.md` with ownership conflict and unmanaged + collision entries. +- Update `docs/internal/state.md` and `docs/internal/publish.md`. +- Add maintained examples only when they are valid and load-tested. + +Tests: + +```sh +go test ./internal/config +go test ./... +``` + +## Stage 7: Reconcile-State Planning Core + +Goal: add app-level planning and state helpers for reconcile-state without +adding CLI execution first. + +Implementation: + +- Add `internal/app` reconcile-state use case types: + - options with config path, pipeline id, destination id, all-owners flag, + dry-run flag, stdout, and output format if the existing format pattern is + reused; + - report struct with destination identity, backend, root path, state schema, + owner scope, checked count, missing managed outputs, unmanaged entries, and + changed/would-change status. +- Add state helpers for removing missing managed output records: + - single-owner scope; + - shared-root current-owner scope; + - shared-root all-owner scope. +- Use storage `Stat` for managed output existence checks. +- Use bounded `Walk` under the selected destination root to report unmanaged + existing files. Exclude `.distributor.json` and all paths already recorded as + managed. +- Do not validate output digests. +- Do not delete destination files. +- Do not adopt unmanaged files. +- Do not rewrite invalid or ambiguous state. +- Require enough scope to identify one destination root. For current + single-owner state, require pipeline and destination. For shared-root + all-owner mode, still require a pipeline/destination selector to identify the + configured destination root, then apply `--all-owners` inside that root. +- Default behavior applies state repair. `DryRun` reports without writing. + +Tests: + +```sh +go test ./internal/app ./internal/state ./internal/storage/fake +``` + +Required coverage: + +- dry-run reports missing managed outputs without rewriting state; +- apply removes missing records and writes valid state; +- unmanaged files are reported but not adopted or deleted; +- invalid state fails without rewrite; +- shared-root owner-scope and all-owner repair both work. + +## Stage 8: Reconcile-State CLI And Docs + +Goal: expose reconcile-state as an operator command. + +Implementation: + +- Add `reconcile-state` to `internal/cli/root.go` help and dispatch. +- Add CLI flags: + - `--config `; + - `--pipeline `; + - `--destination `; + - `--all-owners`; + - `--dry-run`; + - `--format text|json`. +- Without `--dry-run`, apply state repairs. +- Text output should clearly report whether state was changed or would be + changed. +- JSON output should use the existing app JSON envelope pattern. +- Update `docs/cli.md`, `docs/operations.md`, `docs/troubleshooting.md`, + `docs/integrations/destination-state.md`, `docs/internal/state.md`, + `docs/internal/storage.md`, and `docs/internal/app.md`. + +Tests: + +```sh +go test ./internal/cli ./internal/app +go test ./... +``` + +Required coverage: + +- flag validation; +- default apply behavior; +- `--dry-run` behavior; +- text and JSON reports; +- no deletion of destination files. + +## Stage 9: Prune Config And Planning Core + +Goal: add retention config and pure prune planning. + +Implementation: + +- Add destination retention config: + - `Destination.Retention RetentionPolicy` + - `RetentionPolicy.Prune PrunePolicy` + - `PrunePolicy.Enabled bool` + - `PrunePolicy.OlderThan *Duration` + - `PrunePolicy.KeepLatest *int` +- Default pruning disabled. +- Validate: + - enabled prune policy must set at least one of `older_than` or + `keep_latest`; + - `older_than` must be positive; + - `keep_latest` must be zero or greater; + - documented duration units must parse consistently with existing duration + config behavior. +- Add app/state prune planning helpers: + - select owner-scoped managed outputs; + - sort deterministically by `updated_at`, then path; + - preserve the newest `keep_latest` outputs; + - delete remaining outputs older than `older_than`; + - if both policies are set, first preserve `keep_latest`, then delete + remaining outputs older than `older_than`. +- Use per-output `updated_at` as the timestamp basis. +- Plan only; do not delete storage files in this stage. + +Tests: + +```sh +go test ./internal/config +go test ./internal/app ./internal/state +``` + +Required coverage: + +- config defaults and validation; +- `older_than` planning; +- `keep_latest` planning; +- combined policy planning; +- deterministic tie-breaking; +- shared-root owner scope preservation. + +## Stage 10: Prune Execution, CLI, And Docs + +Goal: expose safe managed-output pruning. + +Implementation: + +- Add app prune execution: + - load config; + - resolve selected destination and owner scope; + - read and validate state; + - build prune plan from configured retention policy; + - in dry-run, write no files and delete nothing; + - in apply mode, delete only planned managed output paths; + - after confirmed deletes, remove deleted records from state and update state + timestamps; + - preserve state records for failed deletes so retry remains accurate; + - never delete unmanaged files; + - do not delete `.distributor.json` unless a later roadmap explicitly permits + empty-state removal. +- Add `prune` CLI dispatch and help. +- Add CLI flags: + - `--config `; + - `--pipeline `; + - `--destination `; + - `--dry-run`; + - `--apply`; + - `--format text|json`. +- Require exactly one of `--dry-run` or `--apply`. `--dry-run` plans and + reports without writing. `--apply` deletes planned managed outputs and + rewrites state after confirmed deletes. +- JSON output should use the existing app JSON envelope pattern. +- Do not add one-off retention overrides in the initial implementation. +- Let local and SSH/SFTP backend helpers prune empty directories only where + they already do so safely. Preserve object-storage prefix markers unless they + are managed output records. +- Update `docs/config.md`, `docs/cli.md`, `docs/operations.md`, + `docs/troubleshooting.md`, `docs/integrations/destination-state.md`, + `docs/internal/state.md`, `docs/internal/publish.md`, and + `docs/internal/app.md`. + +Tests: + +```sh +go test ./internal/config +go test ./internal/app ./internal/cli ./internal/state +go test ./internal/storage/fake +go test ./... +``` + +Required coverage: + +- dry-run reports deletes without deleting files or rewriting state; +- apply deletes only managed output paths; +- unmanaged files under the destination root are preserved; +- partial delete failure preserves accurate state for confirmed deleted and + undeleted outputs; +- shared-root pruning preserves other owners when scoped to one owner; +- CLI apply gating and report output. + +## Final Verification + +After all stages land: + +```sh +go test ./internal/config +go test ./internal/state ./internal/publish +go test ./internal/app ./internal/cli +go test ./internal/storage/fake +go test ./pkg/bundle ./pkg/upload +go test ./... +``` + +Also run: + +```sh +rg -n "reconciliation|shared_root|single_owner|reconcile-state|retention|prune|older_than|keep_latest" README.md docs examples +``` + +Confirm: + +- current-behavior docs outside `docs/roadmap/` describe only implemented + behavior; +- roadmap docs no longer describe completed behavior as future work, or are + revised to mark only remaining deferred work; +- examples load successfully where config tests cover them; +- producer-facing docs still keep destination routing, reconciliation, state, + and retention policy out of producer manifests and upload requests. + +## Non-Goals For This Plan + +- No unmanaged-file adoption workflow. +- No digest-audit mode for reconcile-state. +- No whole-config reconcile-state command. +- No automatic post-publish pruning. +- No path/date parsing retention policy. +- No producer-facing retention, reconciliation, or destination routing API. +- No broad storage synchronization behavior. diff --git a/docs/roadmap/multipipeline.md b/docs/roadmap/multipipeline.md new file mode 100644 index 0000000..6f1cf58 --- /dev/null +++ b/docs/roadmap/multipipeline.md @@ -0,0 +1,195 @@ +# Multi-Pipeline Shared Root Roadmap + +This document records planned shared-root destination state work that is not +part of the current implementation. Current destination state behavior is +documented in `docs/integrations/destination-state.md`. + +This work should not be implemented until the reconciliation work in +`docs/roadmap/reconciliation.md` is present and complete. Shared-root +multi-pipeline publishing should then be implemented before reconcile-state +tooling (`docs/roadmap/reconcile-state.md`) and pruning +(`docs/roadmap/prune.md`) because both tools should operate on the final shared +state model. + +## Problem + +Current `.distributor.json` state is a single-owner destination bundle record. +It stores one `pipeline_id`, one `destination_id`, one latest source manifest, +and one output set. That works when each pipeline writes to a separate +destination bundle path. + +It does not work well when multiple pipelines intentionally publish into one +shared destination root, for example: + +```text +2026-06-08/daily/report.md +2026-06-08/event/watch-123.md +``` + +If both pipelines resolve to the same destination bundle path, the second +pipeline encounters a state identity mismatch even when its output paths are +disjoint. Separating roots avoids the conflict, but can make retention, +cleanup, linking, and human browsing less natural. + +## Goal + +Support multiple configured pipelines writing to one shared destination root +while preserving distributor ownership, unmanaged-content safety, dry-run +behavior, and deterministic conflict handling. + +The desired shape is a collection-oriented `.distributor.json` that records +managed outputs from multiple owners under one state file. Producers still +submit source bundles only. Producers must not choose destinations, +reconciliation mode, retention policy, transforms, links, or storage backends. + +Shared-root behavior should be an explicit per-destination state mode, for +example `state.mode: single_owner|shared_root` or a similarly named field under +the destination. Shared-root behavior changes the meaning of `.distributor.json`, +so it should be visible at the destination boundary rather than inferred from +other configured pipelines. + +## State Model + +Add a new destination state schema that can represent a managed collection +rather than one latest source bundle. + +State shape: + +- top-level `schema_version`; +- top-level `distributor_version`; +- top-level `created_at` for the first time this managed root was created; +- top-level `updated_at` for the last successful state update; +- top-level collection metadata identifying the state as shared-root capable; +- an `outputs` array containing all managed output records in the shared root; +- per-output owner metadata: + - `pipeline_id`; + - `destination_id`; + - source manifest id; + - source manifest digest; + - source manifest created timestamp; + - output kind, source path, transform, URL, SHA-256, and size; + - `created_at` for first publication of that output path; + - `updated_at` for the most recent write of that output path. +- optional source-manifest history keyed by source digest, if compact per-output + source identity fields are not enough for later diagnostics or tooling. + +Output ownership is keyed by both `pipeline_id` and `destination_id`. The +architecture treats destinations as independent policy boundaries, and the same +pipeline may publish different output sets to different destinations. + +The state should be able to answer these questions without scanning the whole +destination: + +- Which paths are distributor-managed? +- Which pipeline/destination/source currently owns each managed path? +- Which outputs belong to a specific pipeline/destination publish operation? +- Which outputs are old enough, or least recently updated enough, for later + pruning decisions? +- Which paths are absent from storage and should be removed by reconcile-state + tooling? + +## Ownership And Conflict Rules + +Shared-root support must keep ownership explicit. + +Required rules: + +- A publish may write only output paths planned from its configured pipeline and + destination. +- A publish may overwrite an existing managed output path only when ownership + rules allow it. +- A new output path that exists in storage but is not recorded in + `.distributor.json` remains unmanaged content and fails by default. +- Disjoint outputs from different pipelines can coexist in one state file. +- State updates must preserve unrelated managed outputs from other pipelines. +- Failed-write cleanup must delete only paths written by the failed attempt. +- Forced replacement must remain explicit, dry-runnable, and bounded to the + configured destination root. + +Overwrite rule: an output path may be overwritten by the same `pipeline_id` and +`destination_id`; a different pipeline or destination writing the same path +fails as an ownership conflict unless a later explicit operator takeover +workflow is implemented. + +## Planning And Execution Work + +Implementation work: + +- Introduce a new state schema and parser that can represent shared ownership. +- Keep current schema parsing available long enough to migrate or reject old + state with clear errors. +- Add explicit per-destination configuration that opts a destination into + shared-root behavior. +- Thread shared-root state semantics through destination inspection, + comparison, publish planning, and execution. +- Make comparison operate on the subset of state owned by the current + pipeline/destination where appropriate. +- Make output collision detection check both newly planned outputs and existing + managed outputs from other owners. +- Write state atomically with preserved unrelated owners and updated records for + the current publish. +- Preserve merge and replace semantics within the current owner scope: + - owner-scoped `replace` removes prior outputs for that owner that are absent + from the new plan; + - owner-scoped `merge` retains prior outputs for that owner that are absent + from the new plan. +- Ensure local, SSH/SFTP, S3, and fake backends support the necessary state read, + write, overwrite, and cleanup paths. + +## Migration And Compatibility + +The initial implementation should prefer explicit safety over automatic +conversion. + +Migration behavior: + +- A destination configured for shared-root behavior may read current single-owner + state and convert it only when the existing state identity matches the current + pipeline/destination or when an explicit migration command/flag is used. +- A destination not configured for shared-root behavior continues to use the + current single-owner state semantics. +- Invalid or ambiguous state fails with an actionable error and a dry-run report + path. + +## Testing + +Important tests: + +- Two pipelines publish disjoint paths into the same destination root. +- One pipeline can replace its own older output without deleting another + pipeline's output. +- One pipeline can merge new outputs while retaining its prior outputs and + another pipeline's outputs. +- A pipeline cannot silently overwrite another pipeline's managed path. +- A pipeline cannot claim unmanaged storage content by default. +- State timestamps are set on creation and updated on subsequent writes. +- Per-output timestamps preserve original `created_at` and update `updated_at` + only when that output path is rewritten. +- Failed writes clean up only attempted outputs. +- Dry-run reports shared-root actions without writing outputs or state. +- Forced replacement remains bounded and explicitly reported. +- Current single-owner state behavior remains covered for non-shared + destinations. + +## Documentation Work + +When implemented, update current-behavior docs in the same change: + +- `docs/integrations/destination-state.md`: document the new schema and + ownership rules. +- `docs/config.md`: document the shared-root opt-in configuration. +- `docs/operations.md`: explain shared-root publishing, conflicts, and recovery. +- `docs/troubleshooting.md`: add shared-root ownership conflict entries. +- `docs/internal/state.md` and `docs/internal/publish.md`: record the new + planning and state invariants. +- Maintained examples that demonstrate date-first shared roots only after the + behavior is implemented. + +## Boundaries + +- Do not let producers route to destinations or choose shared-root behavior. +- Do not add prune behavior in this work. +- Do not add broad unmanaged-content adoption in this work. +- Do not make shared-root behavior implicit for all destinations. +- Do not use storage listings as the primary source of truth during normal + publishing; `.distributor.json` remains the managed ownership index. diff --git a/docs/roadmap/prune.md b/docs/roadmap/prune.md new file mode 100644 index 0000000..cf12aad --- /dev/null +++ b/docs/roadmap/prune.md @@ -0,0 +1,183 @@ +# Prune Roadmap + +This document records planned destination pruning work that is not part of the +current implementation. Current cleanup and forced replacement behavior is +documented in `docs/operations.md`. + +This work should be implemented after the reconciliation work in +`docs/roadmap/reconciliation.md`, shared-root state work in +`docs/roadmap/multipipeline.md`, and reconcile-state tooling in +`docs/roadmap/reconcile-state.md`. Prune decisions should rely on accurate +managed state, explicit ownership, and timestamp metadata. + +## Problem + +Merge reconciliation and shared-root publishing allow long-lived destinations +to accumulate distributor-managed outputs. That is useful for archives and +date-first report layouts, but those destinations can grow without bound. + +Operators need a safe way to delete old distributor-managed outputs according +to explicit retention policy without touching unmanaged files or outputs owned +by unrelated scopes. + +## Goal + +Add pruning support for managed destination outputs. + +Initial pruning should support both of these policy families: + +- keep only the most recently updated N files in a configured scope; +- delete managed files older than a configured duration. + +Pruning must be based on `.distributor.json` ownership records. It should never +delete files merely because they exist under a configured prefix. + +## Policy Model + +Configuration shape: + +```yaml +destinations: + - id: archive + backend: s3 + bucket: reports + prefix: weather + reconciliation: + mode: merge + retention: + prune: + enabled: true + older_than: 90d + keep_latest: 500 +``` + +Policy semantics: + +- `enabled` defaults to `false`. +- `older_than` deletes managed outputs whose selected timestamp is older than + the duration. +- `keep_latest` keeps the most recent N managed outputs in scope and deletes + older managed outputs beyond that count. +- If both are set, preserve outputs that must be kept by `keep_latest`, then + delete remaining outputs older than `older_than`. +- Retention scope is the configured destination owner scope unless + shared-root configuration explicitly chooses a broader scope. + +Timestamp basis: use per-output `updated_at` for both policies, with +`created_at` available for future created-time retention. Updated-time +retention best matches recurring reports where a stable path is overwritten. + +## CLI Shape + +Command shape: + +```sh +distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run +distributor prune --config /path/to/config.yml --pipeline weather.daily --destination archive --apply +``` + +The command should use configured retention policy by default. A later explicit +override mode can allow one-off retention values if that is useful, but the +initial implementation should avoid ad hoc deletion flags that bypass reviewed +config. + +## Behavior + +Planned behavior: + +- Load and validate config normally. +- Resolve the selected destination and retention policy. +- Read and validate `.distributor.json`. +- Select managed output records in the requested owner scope. +- Sort candidates deterministically by selected timestamp and path. +- Build a delete plan from `older_than`, `keep_latest`, or both. +- In dry-run mode, report planned deletions and state changes without writing. +- In apply mode, delete only planned managed output paths. +- After successful deletes, remove deleted outputs from state and update state + timestamps. +- If a delete fails, report partial failure and preserve enough state for safe + retry. Prefer removing records only for confirmed deleted outputs. +- Never delete unmanaged storage entries. +- Delete only managed output paths initially. Local and SSH/SFTP backend helpers + may prune empty directories only where they already do so safely. Object + storage prefix markers are preserved unless the prefix marker is itself a + managed output record. + +Prune should operate on output records, not source manifests. If a future state +model groups outputs into logical source publications, group-level pruning can +be added later as an explicit policy. + +## Safety Rules + +Required safety behavior: + +- Prune is opt-in. +- Apply mode is explicit; dry-run is the safe default. +- Delete only paths recorded as managed in `.distributor.json`. +- Do not delete `.distributor.json` unless every managed output in scope is + deleted and a later design explicitly allows removing empty state. +- Do not delete unmanaged files, unknown paths, parent directories, or sibling + prefixes. +- Do not run prune against invalid or ambiguous state. +- Preserve outputs outside the selected owner scope. +- Report all destructive actions with backend and path context. + +## Implementation Work + +Implementation work: + +- Add retention config structs, defaults, parsing, and validation. +- Add duration parsing that supports documented minute/hour/day units. +- Add app-level prune planning and execution use cases. +- Add state helpers for selecting owner-scoped output records, sorting by + timestamp, and removing deleted records. +- Add storage delete support or reuse existing managed-delete helpers. +- Add CLI parsing, help text, dry-run output, and apply output. +- Ensure local, SSH/SFTP, S3, and fake backends preserve bounded deletion + semantics. +- Ensure pruning composes with merge mode and shared-root state. + +## Testing + +Important tests: + +- Config defaults retention disabled. +- Config rejects invalid durations, negative counts, and enabled policies with + no retention rule. +- Dry-run reports deletes without deleting files or rewriting state. +- `older_than` deletes only managed outputs older than the threshold. +- `keep_latest` preserves the newest N outputs and deletes older managed + outputs. +- Combined `older_than` and `keep_latest` behavior is deterministic. +- Shared-root pruning preserves outputs from other owners when scoped to one + owner. +- Unmanaged files under the destination root are never deleted. +- Partial delete failure preserves accurate state for confirmed deleted and + undeleted outputs. +- CLI tests cover required flags, apply gating, and report output. +- Backend tests cover local, SSH/SFTP, S3, and fake deletion behavior used by + prune. + +## Documentation Work + +When implemented, update current-behavior docs in the same change: + +- `docs/config.md`: retention configuration, defaults, and examples. +- `docs/cli.md`: prune command syntax and workflows. +- `docs/operations.md`: pruning safety, dry-run, apply, and recovery behavior. +- `docs/troubleshooting.md`: prune conflicts and partial failures. +- `docs/integrations/destination-state.md`: timestamp fields used by pruning. +- `docs/internal/state.md`, `docs/internal/publish.md`, and + `docs/internal/app.md`: implementation boundaries and invariants. + +## Boundaries + +- Do not let producers choose retention policy. +- Do not use prune as a general storage cleanup tool. +- Do not delete unmanaged content. +- Do not infer retention from filenames unless a later roadmap explicitly adds + path/date parsing. +- Do not depend on backend-native lifecycle policies for correctness; those can + coexist operationally but are not distributor's managed state. +- Do not run pruning automatically after publish in the initial implementation; + pruning should start as an explicit CLI command that operators can schedule. diff --git a/docs/roadmap/reconcile-state.md b/docs/roadmap/reconcile-state.md new file mode 100644 index 0000000..ab97f3c --- /dev/null +++ b/docs/roadmap/reconcile-state.md @@ -0,0 +1,171 @@ +# Reconcile-State Roadmap + +This document records planned destination state repair tooling that is not part +of the current implementation. Current recovery behavior is documented in +`docs/operations.md`. + +This work should be implemented after the reconciliation work in +`docs/roadmap/reconciliation.md` and the shared-root state work in +`docs/roadmap/multipipeline.md`. Reconcile-state tooling should understand the +final managed state model before prune behavior is added in +`docs/roadmap/prune.md`. + +## Problem + +`.distributor.json` is the managed ownership index for destination outputs. In +long-lived destinations, operators may delete files manually, lifecycle rules +may remove objects, failed external operations may leave state stale, or a +future shared-root destination may accumulate many independently managed +outputs. + +Without a first-class repair command, operators must inspect and edit state by +hand. That is risky because destination state drives replacement, merge, +conflict, and cleanup decisions. + +## Goal + +Add an operator-facing reconcile-state workflow that compares +`.distributor.json` to actual destination storage and repairs stale managed +state safely. + +The initial command should focus on removing state records for managed outputs +that no longer exist. It should report unmanaged files, but it should not adopt +them by default. + +## CLI Shape + +Command shape: + +```sh +distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --dry-run +distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive +``` + +For shared-root destinations, the command should also support reconciling all +owners in one selected destination root. The final selector should be explicit +enough to avoid relying on destination ids being globally unique: + +```sh +distributor reconcile-state --config /path/to/config.yml --pipeline weather.daily --destination archive --all-owners --dry-run +``` + +The exact flags should follow existing CLI conventions. Dry-run should be the +explicit `--dry-run` flag; without `--dry-run`, the command applies state +repairs. + +## Behavior + +Planned behavior: + +- Load and validate config normally. +- Open the configured destination backend. +- Resolve the destination root or destination bundle path using the same path + mapping semantics as publish where applicable. +- Read and validate `.distributor.json`. +- For each managed output record, check whether the storage object/file exists. +- Remove state records for managed outputs that are missing unless `--dry-run` + is set. +- Preserve state records for managed outputs that exist. +- Report storage entries that exist but are not recorded as managed. +- Preserve unmanaged entries by default. +- Update state timestamps when state is changed. +- Write state atomically where the backend supports atomic writes. +- Never delete storage objects as part of the initial reconcile-state command. +- Do not validate output digests against storage contents in the initial + command; digest verification can be added later as a separate audit mode. + +For current single-owner state, reconciliation removes missing records from the +single `outputs` array. For future shared-root state, reconciliation removes +missing output records across the selected owner scope or all owners, depending +on command flags. + +## Reporting + +Dry-run and apply output should be explicit enough for operators and LLM agents +to understand the result. + +Reports should include: + +- destination backend and root; +- state schema version; +- owner scope, if filtered; +- count of managed outputs checked; +- count and paths of missing managed outputs; +- count and paths of unmanaged existing entries, subject to sensible output + limits; +- whether state would be changed or was changed; +- errors with enough backend/path context to recover safely. + +Machine-readable output should be considered if existing CLI report formats can +support it without broad refactoring. + +## Safety Rules + +Required safety behavior: + +- Do not delete destination files. +- Do not adopt unmanaged destination files by default. +- Do not rewrite invalid state except through a future explicit repair workflow. +- Do not repair state when the destination identity or ownership scope is + ambiguous. +- Do not scan outside the configured backend root or destination root. +- Apply state repairs by default and make `--dry-run` the explicit read-only + inspection mode. +- Keep secrets out of reports and errors. + +## Implementation Work + +Implementation work: + +- Add CLI parsing and help text for the reconcile-state command. +- Add an app-level use case that loads config, opens destinations, and builds a + reconcile-state report. +- Add state package helpers for removing missing outputs without weakening + validation. +- Add storage-facing existence/listing helpers only if current backend + interfaces are insufficient. +- Implement local, SSH/SFTP, S3, and fake backend support needed by the command. +- Preserve shared-root ownership metadata and timestamps when rewriting state. +- Add dry-run and applied-change output in the existing CLI reporting style. + +## Testing + +Important tests: + +- Dry-run reports missing managed outputs and does not write state. +- Apply removes missing managed output records and writes valid state. +- Existing managed outputs are preserved. +- Unmanaged existing files are reported but not adopted or deleted. +- Invalid state fails without rewriting. +- Shared-root state can be reconciled for one owner scope. +- Shared-root state can be reconciled for all owner scopes. +- Local, SSH/SFTP, S3, and fake backend behavior is covered for the existence + and listing operations used. +- CLI tests cover flag validation, default apply behavior, `--dry-run` + behavior, and report output. + +## Documentation Work + +When implemented, update current-behavior docs in the same change: + +- `docs/cli.md`: command syntax, flags, and examples. +- `docs/operations.md`: when and how to run reconcile-state safely. +- `docs/troubleshooting.md`: stale state and missing managed file entries. +- `docs/integrations/destination-state.md`: explain how tools may rewrite + state. +- `docs/internal/state.md`, `docs/internal/storage.md`, and + `docs/internal/app.md`: record implementation boundaries. + +## Boundaries + +- Do not implement prune behavior in this work. +- Do not let reconcile-state infer producer source data or regenerate outputs. +- Do not repair invalid JSON or manually edited state unless a later roadmap + defines an explicit recovery flow. +- Do not claim unmanaged files by default. +- Do not add unmanaged-file adoption or digest validation in the initial + command. +- Do not add whole-config reconciliation in the initial command; require enough + scope to identify one destination root, with all-owner reconciliation allowed + inside a configured shared-root destination. +- Do not expose a producer-facing API for this workflow. diff --git a/docs/roadmap/reconciliation.md b/docs/roadmap/reconciliation.md new file mode 100644 index 0000000..914c8c6 --- /dev/null +++ b/docs/roadmap/reconciliation.md @@ -0,0 +1,180 @@ +# Reconciliation Roadmap + +This document records planned destination reconciliation work that is not part +of the current implementation. Current destination replacement, state, and +recovery behavior is documented in `docs/operations.md`, +`docs/config.md`, and `docs/integrations/destination-state.md`. + +This work should be implemented before the shared-root multi-pipeline state +work described in `docs/roadmap/multipipeline.md`. The merge mode described +here is still scoped to one pipeline/destination owner for a destination bundle +path; it does not by itself allow multiple pipelines to share one +`.distributor.json`. + +## Destination Reconciliation Modes + +Current destination replacement behavior treats each publish as the complete +desired managed output set for the destination bundle path. When a newer source +replaces an older destination state, `distributor` deletes prior managed outputs +that are no longer present in the new plan, then writes the new outputs and +state. + +That behavior is useful for snapshot-style producers, but it is not ideal for +incremental producers that publish new reports without retaining prior report +state. + +Planned work: + +- Add a per-destination reconciliation mode. +- Keep current behavior as the default mode. +- Add a mode that writes new outputs while retaining prior managed outputs that + are not present in the new plan. +- Keep reconciliation mode per destination. Replacement and merge behavior are + destination safety policies, and one pipeline may need both a snapshot-style + `latest` destination and an accumulating `archive` destination from the same + source bundle. + +Proposed configuration: + +```yaml +destinations: + - id: archive + backend: s3 + bucket: reports + prefix: weather/morning/archive + reconciliation: + mode: merge + + - id: latest + backend: s3 + bucket: reports + prefix: weather/morning/latest + path_mapping: + mode: fixed + reconciliation: + mode: replace +``` + +Accepted modes: + +- `replace`: current behavior. The destination bundle path is reconciled to the + new planned output set. Prior managed outputs not present in the new plan are + removed. +- `merge`: new planned outputs are written into the managed destination set. + Prior managed outputs not present in the new plan are retained. + +## State Model + +Destination state needs to represent cumulative managed outputs for merge mode. +The current state file already records an `outputs` array, but that array +currently represents the outputs from the most recent publication. + +Planned state semantics: + +- `source.manifest` continues to record the latest source manifest used for + destination comparison. +- `outputs` records the complete currently managed output set for the + destination bundle path. +- In `replace` mode, the new state `outputs` are exactly the newly planned + outputs. +- In `merge` mode, the new state `outputs` are: + - previous managed outputs not overwritten by the new plan; + - plus newly planned outputs. +- Persist reconciliation metadata so state interpretation remains explicit after + config changes and so later migration, reconcile-state, and prune tooling can + explain why historical outputs were retained, for example: + +```json +{ + "reconciliation": { + "mode": "merge" + } +} +``` + +If state schema changes are required, bump the destination state schema version +and keep parsing/validation rules explicit. + +## Safety Rules + +Merge mode must not become an unsafe overwrite path. + +Required safety behavior: + +- A new output may overwrite a prior output only when that path is already + recorded as distributor-managed in the existing `.distributor.json`. +- A new output path that exists in storage but is not recorded as managed should + fail as unmanaged content by default. +- Prior managed outputs not included in the new plan are retained in merge mode. +- Prior managed outputs not included in the new plan are deleted in replace + mode. +- Failed-write cleanup should delete only outputs written by the failed attempt, + not retained prior managed outputs. +- Forced replacement behavior remains explicit, bounded, dry-runnable, and + constrained to the destination bundle path. + +## Planning And Execution Work + +Implementation work: + +- Add destination config type `reconciliation.mode`. +- Default `reconciliation.mode` to `replace`. +- Validate mode values as `replace` or `merge`. +- Thread reconciliation mode into publish planning and execution requests. +- During replace execution, preserve current managed cleanup behavior. +- During merge execution: + - inspect existing state outputs before writing; + - allow overwrites only for paths already managed by existing state; + - write new outputs with overwrite enabled only for managed existing paths; + - retain previous managed outputs that are not overwritten; + - write destination state with the cumulative managed output set. +- Ensure generated HTML outputs and copied source outputs use the same + reconciliation semantics. +- Ensure link metadata and primary URL behavior remain deterministic when + retained outputs exist. Derive `links.primary_url` from the newly planned + outputs for the current publish, not from retained historical outputs. + +## Testing + +Important tests: + +- Config defaults `reconciliation.mode` to `replace`. +- Config rejects unknown reconciliation modes. +- Replace mode deletes prior managed outputs omitted from the new bundle. +- Merge mode retains prior managed outputs omitted from the new bundle. +- Merge mode overwrites a path already recorded as managed. +- Merge mode fails when a new output path collides with unmanaged destination + content. +- Merge mode state records the cumulative managed output set. +- Failed merge writes clean up only outputs from the failed attempt. +- Fixed-path destinations support both modes. +- Source-only, generated-HTML-only, and source-plus-HTML publish policies all + honor reconciliation mode. +- S3, SSH/SFTP, local, and fake backend tests cover any backend overwrite or + cleanup behavior that changes. + +## Documentation Work + +When implemented, update current-behavior docs in the same change: + +- `docs/config.md`: document `reconciliation.mode`, defaults, and examples. +- `docs/operations.md`: explain replace versus merge behavior and recovery. +- `docs/integrations/destination-state.md`: document any state schema or + semantic changes. +- `docs/internal/publish.md` and related internal docs: record planning and + execution invariants. +- Relevant examples under `examples/`, especially archive/latest style + configurations. + +## Boundaries + +- Do not add lifecycle retention policies such as keep-latest-N or delete older + than a duration as part of this work. +- Do not let producers choose reconciliation mode through source manifests or + upload requests. +- Do not weaken unmanaged-content safety checks. +- Do not make merge mode the default; preserve current replacement behavior for + existing configurations. +- Do not claim unmanaged colliding files in merge mode. Claiming unmanaged + content should remain an explicit force or repair workflow, not normal publish + behavior.