Download S3 audio during prepare
This commit is contained in:
@@ -41,7 +41,6 @@ Implemented:
|
||||
Still placeholder/future:
|
||||
|
||||
- `archive` stage behavior
|
||||
- prepare-stage S3 audio download behavior
|
||||
- archive-stage S3 upload/promotion behavior
|
||||
- `notify` stage behavior
|
||||
- additional Scriptorium artifact types beyond `session_recap`
|
||||
@@ -152,9 +151,9 @@ Session input foundations:
|
||||
|
||||
- `session.campaign` is required
|
||||
- local audio remains supported through `session.inputs.audio_dir` or `session.inputs.audio_files`
|
||||
- optional S3 audio input shape is modeled as `session.inputs.audio_s3.prefix`
|
||||
- optional S3 audio input shape is `session.inputs.audio_s3.prefix`
|
||||
- `audio_dir`/`audio_files` and `audio_s3` are mutually exclusive
|
||||
- S3 input execution (object listing/downloading) is not implemented yet
|
||||
- when `audio_s3` is configured, `prepare` lists and downloads `.flac` objects through the object-store backend
|
||||
|
||||
Cross-config validation scope:
|
||||
|
||||
@@ -168,7 +167,21 @@ Remote object-store backend scope:
|
||||
- S3 key/session path semantics remain outside the backend, with this invariant:
|
||||
- callers pass full bucket-relative object keys
|
||||
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
|
||||
- the backend layer is available for future prepare/archive usage, but no stage currently invokes `List`/`Download`/`Upload`/`Exists`
|
||||
- `prepare` now uses object-store `List` and `Download` for S3 audio input
|
||||
- archive upload/promotion behavior using `Upload`/`Exists` remains future work
|
||||
|
||||
Prepare S3 audio behavior (implemented):
|
||||
|
||||
- compute session prefix as `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/`
|
||||
- resolve `session.inputs.audio_s3.prefix` under that session prefix
|
||||
- list objects under the computed audio prefix and filter `.flac` keys
|
||||
- fail clearly when no `.flac` objects are found
|
||||
- download selected objects to spool audio path:
|
||||
- `{spool.root}/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- materialize audio files into workdir audio path:
|
||||
- `{workspace.root}/work/{campaign}/{session_id}/{run_id}/audio/`
|
||||
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
|
||||
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
|
||||
|
||||
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user