Upload successful run records to storage
This commit is contained in:
73
docs/archive-storage.md
Normal file
73
docs/archive-storage.md
Normal file
@@ -0,0 +1,73 @@
|
||||
# Archive Storage
|
||||
|
||||
This document describes implemented archive-stage storage behavior.
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- archive uploads successful run records to remote object storage through the storage backend abstraction.
|
||||
- upload destination is the run prefix:
|
||||
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
||||
- tests use fake storage and do not require live S3.
|
||||
|
||||
Not implemented:
|
||||
|
||||
- promotion uploads to top-level `transcripts/` or `artifacts/`
|
||||
- `current/manifest.json` write
|
||||
- `current/run_id.txt` write
|
||||
- audio re-upload from local workdir
|
||||
- failed/incomplete run uploads
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Archive verifies the following stages succeeded before upload:
|
||||
|
||||
- `prepare`
|
||||
- `transcribe`
|
||||
- `merge`
|
||||
- `polish`
|
||||
- `normalize`
|
||||
- `trim`
|
||||
- `analyze`
|
||||
|
||||
If any prerequisite is missing or not succeeded, archive fails and does not upload.
|
||||
|
||||
## Uploaded Run Record
|
||||
|
||||
Archive uploads existing files from the run workdir when present:
|
||||
|
||||
- `inputs/`
|
||||
- `transcripts/`
|
||||
- `artifacts/`
|
||||
- `reports/` (optional)
|
||||
- `config/`
|
||||
- `logs/`
|
||||
- `manifest.json`
|
||||
|
||||
Archive preserves relative file paths under `runs/{run_id}/`.
|
||||
|
||||
Example:
|
||||
|
||||
- local `transcripts/raw/speaker.json`
|
||||
- remote `runs/{run_id}/transcripts/raw/speaker.json`
|
||||
|
||||
## Audio Upload Policy
|
||||
|
||||
Archive does not upload local `audio/` by default.
|
||||
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
|
||||
|
||||
## Config Controls
|
||||
|
||||
- `archive.enabled: false` skips archive cleanly.
|
||||
- `archive.upload_run: false` skips run upload cleanly.
|
||||
|
||||
## Metadata
|
||||
|
||||
Archive stage metadata includes non-secret upload context:
|
||||
|
||||
- `s3_bucket`
|
||||
- `s3_run_prefix`
|
||||
- `files_uploaded`
|
||||
- `uploaded_paths`
|
||||
- `audio_upload_skipped`
|
||||
@@ -100,10 +100,16 @@ Implemented in repository:
|
||||
- fake storage backend for deterministic, no-network testing
|
||||
- S3-compatible backend built from `storage.s3` config
|
||||
- backend construction helper from resolved config
|
||||
- archive run upload behavior:
|
||||
- archive validates required prior stage success before uploading
|
||||
- archive uploads successful run records under `runs/{run_id}/`
|
||||
- upload set includes run-record files (`inputs`, `transcripts`, `artifacts`, optional `reports`, `config`, `logs`, `manifest.json`)
|
||||
- local audio is not uploaded by default
|
||||
- upload uses storage backend abstraction and deterministic ordering
|
||||
- archive skips cleanly when `archive.enabled` or `archive.upload_run` is false
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- archive-stage S3 upload behavior
|
||||
- promotion uploads
|
||||
- writing `current/manifest.json` and `current/run_id.txt` to S3
|
||||
|
||||
|
||||
Reference in New Issue
Block a user