Add post-archive cleanup policies

This commit is contained in:
2026-05-16 23:09:39 +00:00
parent 6fbefb9867
commit d5a9ad38f8
11 changed files with 639 additions and 5 deletions

View File

@@ -20,11 +20,13 @@ Implemented:
- archive uploads configured promoted outputs to session-level keys.
- archive uploads `current/manifest.json`.
- archive uploads `current/run_id.txt` last as the effective commit marker.
- optional post-archive local cleanup:
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
- tests use fake storage and do not require live S3.
Future work:
- spool audio cleanup/deletion behavior
- `notify` stage behavior
- stale detection
- optional future source-audio upload mode
@@ -89,6 +91,7 @@ Archive writes:
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
Cleanup runs only after this commit-marker write has succeeded.
## Audio Upload Policy
@@ -99,6 +102,7 @@ Original audio is expected at the session-level audio prefix and is not duplicat
- `archive.enabled: false` skips archive cleanly.
- `archive.upload_run: false` skips run upload cleanly.
- both skip cases also skip post-archive local cleanup.
## Metadata

View File

@@ -120,10 +120,13 @@ Implemented in repository:
- `current/run_id.txt` is uploaded last as the effective commit marker
- current pointer content is `{run_id}` plus trailing newline
- if promotion/current manifest upload fails, current pointer is not written
- post-archive local cleanup behavior:
- `pipeline.spool.delete_audio_after_archive` removes run-scoped spool audio only after successful archive commit
- `pipeline.workspace.cleanup_after_archive` removes run-scoped workdir only after successful archive commit
- cleanup is skipped for failed/incomplete/skipped/unarchived runs
Not implemented yet:
- spool audio cleanup / deletion behavior
- `notify` stage behavior
- generic stale detection based on input/config checksums
- optional future mode for uploading source audio from local workspace/spool

View File

@@ -12,7 +12,6 @@ Implemented:
Not implemented:
- spool cleanup/deletion behavior
- uploads of failed runs
## Required Configuration