Upload successful run records to storage

This commit is contained in:
2026-05-16 14:43:29 +00:00
parent ed4dcf1ef7
commit d40c91acde
11 changed files with 716 additions and 34 deletions

View File

@@ -40,8 +40,7 @@ Implemented:
Still placeholder/future:
- `archive` stage behavior
- archive-stage S3 upload/promotion behavior
- archive-stage promotion behavior
- `notify` stage behavior
- additional Scriptorium artifact types beyond `session_recap`
- artifact-to-artifact workflows beyond the initial single-artifact implementation
@@ -168,7 +167,8 @@ Remote object-store backend scope:
- callers pass full bucket-relative object keys
- backend methods do not prepend `root_prefix` or infer campaign/session/run paths
- `prepare` now uses object-store `List` and `Download` for S3 audio input
- archive upload/promotion behavior using `Upload`/`Exists` remains future work
- `archive` now uses object-store `Upload` for successful run-record upload under the run prefix
- archive promotion behavior using top-level writes and current pointers remains future work
Prepare S3 audio behavior (implemented):
@@ -183,6 +183,25 @@ Prepare S3 audio behavior (implemented):
- record S3 provenance in manifest input records (bucket/key/metadata/local paths/checksum)
- no AWS SDK types are used in stage code; storage implementation details stay in storage adapter packages
Archive run upload behavior (implemented):
- `archive` verifies prerequisite stage success before upload:
- `prepare`, `transcribe`, `merge`, `polish`, `normalize`, `trim`, `analyze`
- only successful/completed runs are uploaded
- uploaded run record destination is:
- `{root_prefix}/campaigns/{campaign}/sessions/{session_id}/runs/{run_id}/`
- uploaded existing local paths include:
- `inputs/`, `transcripts/`, `artifacts/`, optional `reports/`, `config/`, `logs/`, and `manifest.json`
- local `audio/` is intentionally excluded from upload by default
- file upload order is deterministic (sorted relative paths)
- `archive.enabled: false` and `archive.upload_run: false` skip upload cleanly
- stage metadata records non-secret upload context:
- `s3_bucket`, `s3_run_prefix`, `files_uploaded`, `uploaded_paths`, `audio_upload_skipped`
- archive does not yet:
- write top-level promoted outputs
- write `current/manifest.json`
- write `current/run_id.txt`
`pipeline.scriptorium` is optional. Existing pipelines without Scriptorium continue to work.
`pipeline.trim` is optional. Existing pipelines without trim config continue to work.