Download S3 audio during prepare
This commit is contained in:
@@ -103,7 +103,6 @@ Implemented in repository:
|
||||
|
||||
Not implemented yet:
|
||||
|
||||
- prepare-stage S3 list/download behavior
|
||||
- archive-stage S3 upload behavior
|
||||
- promotion uploads
|
||||
- writing `current/manifest.json` and `current/run_id.txt` to S3
|
||||
@@ -314,7 +313,7 @@ work/audio
|
||||
transcribe
|
||||
```
|
||||
|
||||
The `prepare` stage should:
|
||||
Implemented `prepare` behavior:
|
||||
|
||||
1. list `.flac` objects under the configured S3 audio prefix,
|
||||
2. fail clearly if none are found,
|
||||
@@ -331,7 +330,11 @@ spool:
|
||||
delete_audio_after_archive: true
|
||||
```
|
||||
|
||||
For v1, prefer retaining local workdir audio until the run has successfully archived. The source of truth remains S3, but local diagnostics are valuable during development.
|
||||
Current boundary:
|
||||
|
||||
- downloaded audio is retained in spool/workdir
|
||||
- spool cleanup policy remains future work
|
||||
- archive does not upload source audio by default
|
||||
|
||||
## 6. Configuration Design
|
||||
|
||||
|
||||
82
docs/s3-audio-input.md
Normal file
82
docs/s3-audio-input.md
Normal file
@@ -0,0 +1,82 @@
|
||||
# S3 Audio Input
|
||||
|
||||
This document describes implemented S3 audio input behavior in `prepare`.
|
||||
|
||||
## Scope
|
||||
|
||||
Implemented:
|
||||
|
||||
- `prepare` can acquire source audio from S3 when `session.inputs.audio_s3.prefix` is configured.
|
||||
- object listing and download go through the storage backend abstraction.
|
||||
- tests use fake storage; no live S3 service is required for test runs.
|
||||
|
||||
Not implemented:
|
||||
|
||||
- archive uploads (run uploads, promotion uploads, current pointer writes)
|
||||
- spool cleanup/deletion behavior
|
||||
- uploads of failed runs
|
||||
|
||||
## Required Configuration
|
||||
|
||||
`pipeline.yml`:
|
||||
|
||||
- `storage.s3.bucket` must be set when S3 audio input is used.
|
||||
- `storage.s3.root_prefix` defaults to `dnd`.
|
||||
- `spool.root` defaults to `/var/spool/narratio`.
|
||||
|
||||
`session.yml`:
|
||||
|
||||
- configure `session.campaign` and `session.session_id`.
|
||||
- configure `session.inputs.audio_s3.prefix` for S3 audio input.
|
||||
- do not configure `inputs.audio_dir` or `inputs.audio_files` at the same time as `inputs.audio_s3`.
|
||||
|
||||
## Prefix Shape
|
||||
|
||||
Session S3 root:
|
||||
|
||||
`{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
|
||||
Audio prefix:
|
||||
|
||||
`{session_root}/{audio_s3.prefix}`
|
||||
|
||||
Example:
|
||||
|
||||
`dnd/campaigns/forsaken/sessions/2026-04-19/audio/`
|
||||
|
||||
Audio files must already exist in S3 before running Narratio.
|
||||
|
||||
## Prepare Behavior
|
||||
|
||||
When `inputs.audio_s3.prefix` is configured, `prepare`:
|
||||
|
||||
1. lists objects under the computed S3 audio prefix
|
||||
2. filters to `.flac` objects
|
||||
3. fails when no `.flac` objects are found
|
||||
4. downloads selected objects to spool audio:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
5. materializes audio into workdir audio:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
6. records input provenance in the manifest (bucket, key, metadata, local paths, checksum)
|
||||
|
||||
Notes:
|
||||
|
||||
- `.flac` filtering is case-insensitive.
|
||||
- ETag is recorded as provider metadata only and is not treated as a checksum.
|
||||
|
||||
## Local Audio Development
|
||||
|
||||
Local audio workflows remain supported:
|
||||
|
||||
- `inputs.audio_dir`
|
||||
- `inputs.audio_files`
|
||||
|
||||
These options are mutually exclusive with `inputs.audio_s3`.
|
||||
|
||||
## Archive Boundary
|
||||
|
||||
Current archive behavior remains unimplemented in this area:
|
||||
|
||||
- no upload of source audio by default
|
||||
- no upload of failed runs
|
||||
- no `current/manifest.json` or `current/run_id.txt` writes
|
||||
Reference in New Issue
Block a user