Upload successful run records to storage

This commit is contained in:
2026-05-16 14:43:29 +00:00
parent ed4dcf1ef7
commit d40c91acde
11 changed files with 716 additions and 34 deletions

73
docs/archive-storage.md Normal file
View File

@@ -0,0 +1,73 @@
# Archive Storage
This document describes implemented archive-stage storage behavior.
## Scope
Implemented:
- archive uploads successful run records to remote object storage through the storage backend abstraction.
- upload destination is the run prefix:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
- tests use fake storage and do not require live S3.
Not implemented:
- promotion uploads to top-level `transcripts/` or `artifacts/`
- `current/manifest.json` write
- `current/run_id.txt` write
- audio re-upload from local workdir
- failed/incomplete run uploads
## Prerequisites
Archive verifies the following stages succeeded before upload:
- `prepare`
- `transcribe`
- `merge`
- `polish`
- `normalize`
- `trim`
- `analyze`
If any prerequisite is missing or not succeeded, archive fails and does not upload.
## Uploaded Run Record
Archive uploads existing files from the run workdir when present:
- `inputs/`
- `transcripts/`
- `artifacts/`
- `reports/` (optional)
- `config/`
- `logs/`
- `manifest.json`
Archive preserves relative file paths under `runs/{run_id}/`.
Example:
- local `transcripts/raw/speaker.json`
- remote `runs/{run_id}/transcripts/raw/speaker.json`
## Audio Upload Policy
Archive does not upload local `audio/` by default.
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
## Config Controls
- `archive.enabled: false` skips archive cleanly.
- `archive.upload_run: false` skips run upload cleanly.
## Metadata
Archive stage metadata includes non-secret upload context:
- `s3_bucket`
- `s3_run_prefix`
- `files_uploaded`
- `uploaded_paths`
- `audio_upload_skipped`

View File

@@ -100,10 +100,16 @@ Implemented in repository:
- fake storage backend for deterministic, no-network testing
- S3-compatible backend built from `storage.s3` config
- backend construction helper from resolved config
- archive run upload behavior:
- archive validates required prior stage success before uploading
- archive uploads successful run records under `runs/{run_id}/`
- upload set includes run-record files (`inputs`, `transcripts`, `artifacts`, optional `reports`, `config`, `logs`, `manifest.json`)
- local audio is not uploaded by default
- upload uses storage backend abstraction and deterministic ordering
- archive skips cleanly when `archive.enabled` or `archive.upload_run` is false
Not implemented yet:
- archive-stage S3 upload behavior
- promotion uploads
- writing `current/manifest.json` and `current/run_id.txt` to S3