116 lines
2.8 KiB
Markdown
116 lines
2.8 KiB
Markdown
# Archive Storage
|
|
|
|
This document describes implemented archive-stage publish behavior.
|
|
|
|
## S3 Paths
|
|
|
|
Session root:
|
|
|
|
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
|
|
|
Run prefix:
|
|
|
|
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
|
|
|
|
## Scope
|
|
|
|
Implemented:
|
|
|
|
- archive uploads successful run records to remote object storage through the storage backend abstraction.
|
|
- archive uploads configured promoted outputs to session-level keys.
|
|
- archive uploads `current/manifest.json`.
|
|
- archive uploads `current/run_id.txt` last as the effective commit marker.
|
|
- tests use fake storage and do not require live S3.
|
|
|
|
Future work:
|
|
|
|
- spool audio cleanup/deletion behavior
|
|
- `notify` stage behavior
|
|
- stale detection
|
|
- optional future source-audio upload mode
|
|
- additional artifact generation beyond current implemented set
|
|
|
|
## Prerequisites
|
|
|
|
Archive verifies these stages succeeded before upload:
|
|
|
|
- `prepare`
|
|
- `transcribe`
|
|
- `merge`
|
|
- `polish`
|
|
- `normalize`
|
|
- `trim`
|
|
- `analyze`
|
|
|
|
If any prerequisite is missing or not succeeded, archive fails and does not upload.
|
|
Failed or incomplete runs remain local only.
|
|
|
|
## Run Upload
|
|
|
|
Archive uploads existing files from the run workdir when present:
|
|
|
|
- `inputs/`
|
|
- `transcripts/`
|
|
- `artifacts/`
|
|
- `reports/` (optional)
|
|
- `config/`
|
|
- `logs/`
|
|
- `manifest.json`
|
|
|
|
Relative paths are preserved under `runs/{run_id}/`.
|
|
|
|
## Promotion Rules
|
|
|
|
Archive applies `archive.promote_artifacts` in config order.
|
|
|
|
Rule behavior:
|
|
|
|
- `from`: local workdir-relative source path
|
|
- `to`: session-root-relative destination key
|
|
- `required: true`: missing source fails archive
|
|
- `required: false`: missing source is skipped and recorded
|
|
|
|
Default promoted outputs:
|
|
|
|
- `transcripts/trimmed.json`
|
|
- `artifacts/session_recap.md`
|
|
|
|
## Current Pointers
|
|
|
|
Archive writes:
|
|
|
|
1. `current/manifest.json` (after run upload + promotions)
|
|
2. `current/run_id.txt` last
|
|
|
|
`current/run_id.txt` contains exactly:
|
|
|
|
- `{run_id}` plus trailing newline
|
|
|
|
Writing `current/run_id.txt` last makes it the effective commit marker for published session state.
|
|
|
|
If any required run upload, promotion upload, or current-manifest upload fails, archive returns failure and does not write `current/run_id.txt`.
|
|
|
|
## Audio Upload Policy
|
|
|
|
Archive does not upload local `audio/` by default.
|
|
Original audio is expected at the session-level audio prefix and is not duplicated under `runs/{run_id}/`.
|
|
|
|
## Config Controls
|
|
|
|
- `archive.enabled: false` skips archive cleanly.
|
|
- `archive.upload_run: false` skips run upload cleanly.
|
|
|
|
## Metadata
|
|
|
|
Archive stage metadata includes non-secret upload context (for example):
|
|
|
|
- `s3_bucket`
|
|
- `s3_run_prefix`
|
|
- run upload counts/paths
|
|
- promoted upload counts/paths
|
|
- skipped optional promotions
|
|
- `current_manifest_key`
|
|
- `current_run_id_key`
|
|
- `current_pointer_written`
|
|
- `audio_upload_skipped`
|