Download S3 audio during prepare

This commit is contained in:
2026-05-16 14:33:42 +00:00
parent 1e6db89dd4
commit 24cce49a70
10 changed files with 429 additions and 31 deletions

View File

@@ -103,7 +103,6 @@ Implemented in repository:
Not implemented yet:
- prepare-stage S3 list/download behavior
- archive-stage S3 upload behavior
- promotion uploads
- writing `current/manifest.json` and `current/run_id.txt` to S3
@@ -314,7 +313,7 @@ work/audio
transcribe
```
The `prepare` stage should:
Implemented `prepare` behavior:
1. list `.flac` objects under the configured S3 audio prefix,
2. fail clearly if none are found,
@@ -331,7 +330,11 @@ spool:
delete_audio_after_archive: true
```
For v1, prefer retaining local workdir audio until the run has successfully archived. The source of truth remains S3, but local diagnostics are valuable during development.
Current boundary:
- downloaded audio is retained in spool/workdir
- spool cleanup policy remains future work
- archive does not upload source audio by default
## 6. Configuration Design

82
docs/s3-audio-input.md Normal file
View File

@@ -0,0 +1,82 @@
# S3 Audio Input
This document describes implemented S3 audio input behavior in `prepare`.
## Scope
Implemented:
- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
- object listing and download go through the storage backend abstraction.
- tests use fake storage; no live S3 service is required for test runs.
Not implemented:
- archive uploads (run uploads, promotion uploads, current pointer writes)
- spool cleanup/deletion behavior
- uploads of failed runs
## Required Configuration
`pipeline.yml`:
- `storage.s3.bucket` must be set when S3 audio input is used.
- `storage.s3.root_prefix` defaults to `dnd`.
- `spool.root` defaults to `/var/spool/narratio`.
`session.yml`:
- configure `session.campaign` and `session.session_id`.
- configure `session.inputs.audio_s3.prefix` for S3 audio input.
- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
## Prefix Shape
Session S3 root:
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
Audio prefix:
`{session_root}/{audio_s3.prefix}`
Example:
`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
Audio files must already exist in S3 before running Narratio.
## Prepare Behavior
When `inputs.audio_s3.prefix` is configured, `prepare`:
1. lists objects under the computed S3 audio prefix
2. filters to `.flac` objects
3. fails when no `.flac` objects are found
4. downloads selected objects to spool audio:
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
5. materializes audio into workdir audio:
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
Notes:
- `.flac` filtering is case-insensitive.
- ETag is recorded as provider metadata only and is not treated as a checksum.
## Local Audio Development
Local audio workflows remain supported:
- `inputs.audio_dir`
- `inputs.audio_files`
These options are mutually exclusive with `inputs.audio_s3`.
## Archive Boundary
Current archive behavior remains unimplemented in this area:
- no upload of source audio by default
- no upload of failed runs
- no `current/manifest.json` or `current/run_id.txt` writes