Download S3 audio during prepare

This commit is contained in:
2026-05-16 14:33:42 +00:00
parent 1e6db89dd4
commit 24cce49a70
10 changed files with 429 additions and 31 deletions

View File

@@ -41,7 +41,6 @@ Implemented:
Still placeholder/future:
- `archive` stage behavior
- prepare-stage S3 audio download behavior
- archive-stage S3 upload/promotion behavior
- `notify` stage behavior
- additional Scriptorium artifact types beyond `session_recap`
@@ -152,9 +151,9 @@ Session input foundations:
- `session.campaign` is required
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
- optional S3 audio input shape is modeled as `session.inputs.audio_s3.prefix`
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
- S3 input execution (object listing/downloading) is not implemented yet
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend
Cross-config validation scope:
@@ -168,7 +167,21 @@ Remote object-store backend scope:
- S3 key/session path semantics remain outside the backend, with this invariant:
- callers pass full bucket-relative object keys
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
- the backend layer is available for future prepare/archive usage, but no stage currently invokes `List`/`Download`/`Upload`/`Exists`
- `prepare` now uses object-store `List` and `Download` for S3 audio input
- archive upload/promotion behavior using `Upload`/`Exists` remains future work
Prepare S3 audio behavior (implemented):
- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
- resolve `session.inputs.audio_s3.prefix` under that session prefix
- list objects under the computed audio prefix and filter `.flac` keys
- fail clearly when no `.flac` objects are found
- download selected objects to spool audio path:
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
- materialize audio files into workdir audio path:
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.