Add prune retention planning

This commit is contained in:
2026-06-08 19:26:48 +00:00
parent 2abd09bde3
commit c67ecf86a9
12 changed files with 628 additions and 7 deletions

View File

@@ -4,13 +4,13 @@ Audience: developers and LLM coding agents changing `internal/app`.
## Purpose
`internal/app` owns top-level application use cases: run, single-pipeline run, staged-source run, validate, inspect, manifest creation, reconcile-state planning/repair, and HTTP upload serving. It coordinates config loading, secret resolution, backend construction, source discovery, destination selection, publish planning/execution, state repair reporting, notification handoff, output projection, and upload coordination.
`internal/app` owns top-level application use cases: run, single-pipeline run, staged-source run, validate, inspect, manifest creation, reconcile-state planning/repair, prune planning, and HTTP upload serving. It coordinates config loading, secret resolution, backend construction, source discovery, destination selection, publish planning/execution, state repair reporting, retention plan reporting, notification handoff, output projection, and upload coordination.
## Inputs And Outputs
Inputs include app option structs, contexts, config paths, pipeline ids, local source roots, dry-run/force flags, output format, stdout writers, HTTP requests, and optional notifier implementations.
Outputs include `RunReport`, `ReconcileStateReport`, validate/inspect/manifest results, CLI text/JSON projections, HTTP upload responses, upload status records, and errors. Destination-scoped failures can return a partial run report plus an aggregated error; fatal setup failures return before a complete report exists.
Outputs include `RunReport`, `ReconcileStateReport`, `PrunePlanReport`, validate/inspect/manifest results, CLI text/JSON projections, HTTP upload responses, upload status records, and errors. Destination-scoped failures can return a partial run report plus an aggregated error; fatal setup failures return before a complete report exists.
## Boundaries
@@ -20,7 +20,7 @@ User-facing command parsing stays in `internal/cli`, including `reconcile-state`
## Config Fields Used
The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, reconciliation policy, and transfer policy.
The package consumes the loaded `config.Config`: `server.http`, `secrets.directory`, pipeline ids, source and destination backend fields, validation policy, publish policy, transform policy, path mapping, links, state policy, reconciliation policy, retention policy, and transfer policy.
Config fields are validated and defaulted by `internal/config` before app workflows use them.
@@ -36,6 +36,8 @@ Run workflows discover and validate source bundles through `internal/bundle`. De
Reconcile-state workflows load one configured pipeline/destination selector, open that destination root, parse the root `.distributor.json`, and report missing managed output records plus unmanaged storage entries. Managed output existence checks use storage `Stat`; unmanaged reporting uses bounded storage `Walk` and excludes `.distributor.json` plus all paths already recorded as managed. Apply mode removes missing managed output records from state and rewrites valid state only; dry-run reports the same repair without writing. Text output reports `changed`, `would_change`, or `unchanged`; JSON output uses the shared app envelope. It does not validate output digests, delete destination files, adopt unmanaged files, or rewrite invalid or mismatched state.
Prune planning consumes a parsed destination state document and a validated retention prune policy, then returns owner-scoped managed output records that would be pruned or preserved. Planning uses output `updated_at` timestamps, applies `keep_latest` before `older_than` when both are configured, and does not open storage, delete files, or rewrite state.
HTTP uploads stage and validate archives before enqueueing a pipeline run with a local staged source root. Go producers can use the public `pkg/upload` package to create client-side gzip tar uploads for this server contract; `internal/app` remains the server-side orchestration boundary and does not import that producer package.
Upload idempotency is owned by the upload coordinator. Optional `Idempotency-Key` values are scoped to token id, pipeline id, and key. The coordinator reserves a key while staging is in progress, records the accepted run id with the validated source manifest identity after staging succeeds, returns the original accepted record for the same scoped key and same manifest, and rejects the same scoped key with a different manifest as a conflict.
@@ -57,6 +59,7 @@ HTTP upload startup fails if upload tokens are missing, empty, or duplicated. Up
## Tests To Inspect
- `internal/app/*_test.go`
- `internal/app/prune_test.go`
- `internal/cli/reconcile_state_test.go`
- `internal/cli/root_test.go`
- `internal/config/*_test.go`
@@ -73,3 +76,4 @@ HTTP upload startup fails if upload tokens are missing, empty, or duplicated. Up
- Idempotent upload retries compare normalized source manifest identity, not archive bytes.
- Runtime backend registration remains app-owned.
- Reconcile-state repairs state records only; it never deletes or adopts destination files.
- Prune planning is report-only until execution code applies a plan.