Add post-archive cleanup policies
This commit is contained in:
@@ -20,11 +20,13 @@ Implemented:
|
||||
- archive uploads configured promoted outputs to session-level keys.
|
||||
- archive uploads `current/manifest.json`.
|
||||
- archive uploads `current/run_id.txt` last as the effective commit marker.
|
||||
- optional post-archive local cleanup:
|
||||
- `pipeline.spool.delete_audio_after_archive: true` removes only the run-scoped spool audio directory
|
||||
- `pipeline.workspace.cleanup_after_archive: true` removes only the run-scoped local workdir
|
||||
- tests use fake storage and do not require live S3.
|
||||
|
||||
Future work:
|
||||
|
||||
- spool audio cleanup/deletion behavior
|
||||
- `notify` stage behavior
|
||||
- stale detection
|
||||
- optional future source-audio upload mode
|
||||
@@ -89,6 +91,7 @@ Archive writes:
|
||||
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
|
||||
|
||||
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
|
||||
Cleanup runs only after this commit-marker write has succeeded.
|
||||
|
||||
## Audio Upload Policy
|
||||
|
||||
@@ -99,6 +102,7 @@ Original audio is expected at the session-level audio prefix and is not duplicat
|
||||
|
||||
- `archive.enabled: false` skips archive cleanly.
|
||||
- `archive.upload_run: false` skips run upload cleanly.
|
||||
- both skip cases also skip post-archive local cleanup.
|
||||
|
||||
## Metadata
|
||||
|
||||
|
||||
@@ -120,10 +120,13 @@ Implemented in repository:
|
||||
- `current/run_id.txt` is uploaded last as the effective commit marker
|
||||
- current pointer content is `{run_id}` plus trailing newline
|
||||
- if promotion/current manifest upload fails, current pointer is not written
|
||||
- post-archive local cleanup behavior:
|
||||
- `pipeline.spool.delete_audio_after_archive` removes run-scoped spool audio only after successful archive commit
|
||||
- `pipeline.workspace.cleanup_after_archive` removes run-scoped workdir only after successful archive commit
|
||||
- cleanup is skipped for failed/incomplete/skipped/unarchived runs
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- spool audio cleanup / deletion behavior
|
||||
- `notify` stage behavior
|
||||
- generic stale detection based on input/config checksums
|
||||
- optional future mode for uploading source audio from local workspace/spool
|
||||
|
||||
@@ -12,7 +12,6 @@ Implemented:
|
||||
|
||||
Not implemented:
|
||||
|
||||
- spool cleanup/deletion behavior
|
||||
- uploads of failed runs
|
||||
|
||||
## Required Configuration
|
||||
|
||||
Reference in New Issue
Block a user